A scalable data platform and governance model.

01

Executive summary

Led the implementation of Databricks as a core platform, processing complex GDS data with PySpark and managing 250+ nested JSON arrays alongside global PostgreSQL data.

02

Role and scope

Organization

StudentUniverseWaltham, MA · Dec 2020 – Dec 2022

Role

Business Intelligence Manager

Scope

Databricks as a core platform, processing complex GDS data alongside global PostgreSQL data.

03

System architecture

An abstract view of the transformation shape: disorder on the left resolving into governed structure on the right. It carries no proprietary schema, screen, or client detail.

04

Outcomes and evidence

Delivered a scalable data platform and governance model following a nine-month implementation.

20 TB

Processed in Databricks and PySpark

250+

Nested JSON arrays managed

05

Technology

• Databricks
• PySpark
• AWS S3
• PostgreSQL
• Tableau
• ETL/ELT
• Data Governance