Media Data Lake and Big Data Computing

Business Objective

Nielsen, a global leader audience measurement, media data and analytics, collecting and processing data from millions of devices, sought to streamline information delivery to its clients. The existing solution architecture and toolset were unable to efficiently process spikes and surges in datasets. Key information was delivered to customers in a manual, labor-intensive manner.

The Transformation

• Exadel architects were part of the organizational transformation to MDL (Media Data Lake) approach. MDL became a unified clean source of truth for the entire organization with the possibility to setup reporting and ML on top.

• Exadel Big Data Engineers refactored and created hundreds of data pipelines on AWS cloud to decrease execution time from hours to minutes, and developed a number of cleansing rules, transformations and aggregations on hundreds of GB(s) of raw data. The Exadel team utilized Airflow as a tool implementing traceability and notification processes for data pipelines.

• Exadel engineers created an API layer allowing integration with MDL and delivering data to the final consumers using APIs versus a partial manual process before.

Enablers

• Storage, Master Data Management, Data QA
• Infrastructure, Big Data Integration
• Data Connectors, APIs
• BI Reporting, Custom Visualizations
• Monitoring & Alerting, Availability & Recovery
• Incidents & Change Requests Management