Data analytics solution for a SaaS platform
Client
The customer is a US eLearning provider for K-12 focused on professional development solutions, STEM curriculum & programs.
- Location: USA
- Industry: Education
Key achievements
The Aristek team empowered the US eLearning provider’s SaaS platform with a robust data analytics system.
- 1,5years of fruitful work
- > 100 Tbof data
Challenge
The client owns a SaaS platform specializing in STEM (Science, Technology, Engineering, and Mathematics). The competition in the client industry has been fuelled by the demand for robust analytics tools, as these tools provide critical insights that help companies make informed decisions.
Therefore, it was decided to augment the platform with a data analytics system from scratch that will:
Augment the client’s product with advanced analytics features and dashboards for school admins and teachers to streamline administrative tasks, track student progress, pinpoint gaps in learning content, and more.
Provide comprehensive analytics for internal stakeholders to make proactive decisions across sales, enable forward-thinking product pricing, forecast platform features usage, and more.
The company had a few other data-related challenges to be addressed:
- Data was scattered across multiple systems, including LMS, SIS, and various digital tools used for assignments and assessments. Integrating these sources manually was time-consuming and prone to human errors;
- The existing data processing workflows were heavily manual, requiring educators and administrators to spend countless hours preparing data for analysis;
- Teachers and administrators lacked a unified view of student performance across different subjects and activities, such as which materials students excelled in, where they faced difficulties, and how they progressed through learning paths.
Solution
To enable the management of all data types, workloads, and analytics, the Aristek Systems team suggested developing a powerful and flexible data management solution.
Key features were to include:
Centralized data system to simplify the management of all data types, featuring a data lake for unstructured data and a data warehouse for structured data;
Statistical analysis, reporting, and data mining capabilities to reduce manual work and errors;
Analytical dashboards with visualizations directly into the application so that users can analyze data easily and clearly;
Role-based data access so that each user could see only his/her own data, thus ensuring confidentiality and security of information.
Team
Project manager
Data engineers
Data analytics
QA engineer
Business analyst
Project scope
The project used a state-of-the-art technology stack, including dbt for data transformation and Pulumi for Infrastructure as Code (IaC) management. This significantly accelerated the development and deployment processes, ensuring the flexibility and scalability of the solution.
Aristek experts utilized AWS data security tools such as encryption, access control, and activity monitoring to comply with OWASP and FERPA standards.
Here’s how the team delivered the solution step by step:
The initial step was to gather all the requirements, define the deadlines, and establish methods for data integration;
The client had their own data and data from third-party content sources integrated with the client’s product. From these and other sources, data were aggregated into a data lake;
In the next step, unstructured data from the lake was smoothly transformed into the structured data warehouse. It utilized a Star Schema for easy querying and high-performance data analysis, enabling quick access to actionable insights and reducing the time spent on data retrieval;
After that, the developers’ team deployed the data analytics dashboards into a production environment. Continuous updates and refinements were applied based on user feedback and performance metrics;
Upon project completion, the Aristek team provided comprehensive user training. The experts organized the ongoing support to gather feedback and make the necessary changes for continuous improvement.
During the development process, product owners and the tech lead were joining the team during SCRUM ceremonies. The stakeholders were shown the progress for each sprint. Such a сontinuous collaboration has made it possible to track the subtasks’ completeness and prioritize effectively.
In addition, the data security specialist was also present and co-operated with the team. He regularly audited the development process and shared valuable security recommendations.
Within the analytics section for teaching staff, the teacher can see a variety of analytical tools and dashboards with just a few clicks. They allow to filter data by the following criteria:
- Student results for a specific time period;
- Student’s grade;
- By topics covered, etc.
Tools & technologies
- Python
- AWS Glue
- AWS EventBridge
- AWS Kinesis
- AWS Athena
- AWS QuickSight
- AWS S3
- AWS Lambda
- Dbt framework
- Pulumi
- SQL
- Git
Results
The customer received a fully GDPR and FERPA-compliant data analytics solution that improved operational efficiency and provided actionable insights for informed decision-making. An important project feature within the administrative analytics was that internal data analysts can create year-over-year ad-hoc reports on the following topics:
Curriculum effectiveness;
Platform usage and engagement metrics;
District and school performance, etc.
Creating a robust data warehouse was the cornerstone of this project. This critical infrastructure not only laid the foundation for the entire system but also enabled the introduction of new features for end users. Here are the opportunities it opened up for business:
- The opportunity to augment the platform with custom analytical tools based on cleansed and thoroughly prepared data;
- Self-service data ingestion and transformation services for employees;
- ML-based tools for data-driven predictions or decisions, and many more.
Overall, the main long-term advantage of such a well-organized system is ensuring all data components work together efficiently and provide timely data insights that can be an unrivaled competitive advantage in any industry.