Building Scalable Pipelines for Processing Data from Devices to the Cloud DataLake

Building Scalable Pipelines for Processing Data from Devices to the Cloud DataLake

We are going to take a closer look at the data journey for user tracking data, starting from user devices to our cloud DataLake and the complex issues that come with this. A story about huge amounts of data, presenting us with an interesting challenge at every stage of the system – starting with scalable client-facing services able to process over 10 000 HTTP requests per second from over one hundred million active monthly users, big data jobs processing daily terabytes of data both in real-time and batch job and the required optimizations in order to balance costs in a cloud environment.