Efficient Data Ingestion and SQL Generation with BigQuery and JSON Schema
Join Razvan Mantu, Senior Data Engineer and Alex Giurgiu, Data Engineering Manager at AdoreMe, as they delve into their innovative approach to integrating microservice data into a BigQuery Data Warehouse (DWH). They’ll uncover how they streamline the process, allowing backend engineers to publish new data seamlessly through JSON schema files and pull requests, ensuring precise and automated data flow.
Discover their strategy involving the utilization of Pub/Sub for sending create/update events, Cloud Function to ingest them into BigQuery, and the implementation of stored procedures, views, and user-defined functions for optimal data manipulation and partitioning. Learn how CI/CD facilitates automatic provisioning from GitHub and how documentation and code are autogenerated, reducing manual intervention and enhancing efficiency.
- Seamless Integration of Microservice Data into BigQuery DWH.
- Utilization of Pub/Sub and Cloud Functions for Efficient Data Ingestion.
- Advanced Data Manipulation using Stored Procedures and User-Defined Functions.
- Automatic Code and Documentation Generation from JSON Schema.
- Streamlined CI/CD for Automatic Provisioning from GitHub.