Migrating approximately 150 ETL pipelines, which encompassed multiple data sources and involved four different databases, to AWS Glue was a pivotal step. Our template-based approach expedited and enhanced the workflow. The entire process, including script creation, job configuration, and workflow establishment from scratch, was executed based on existing rules. A meticulously planned switch-over process – encompassing stopping, starting, and backfilling – ensured a seamless transition. The databases involved in this migration spanned both RDS and RedShift.

Recognizing that the data schema had remained unchanged in the last 3-4 months, we optimized the AWS Glue jobs by reducing unnecessary Crawler execution. This was achieved by modifying the cadence from daily to monthly and triggering execution only when required, such as in the event of a process failure. Additionally, to enhance efficiency, Glue jobs were divided into two distinct parts: DynamoDB to S3 and S3 to RedShift upload. The former utilized data dump features in DynamoDB, while the latter leveraged the Copy command within RedShift. This optimization resulted in a remarkable reduction of unnecessary DPUs from 10 to 0.0625 which in turn resulted in a major cost reduction.

To address the lag in sending contact details to AtData and receiving traffic information due to prolonged processes (12 hours of sending and 12 hours of processing), we implemented a strategic solution. The process was dissected into smaller, more manageable pieces – processing 50 chunks of data per hour. This not only facilitated the receipt of traffic/activity details within an acceptable time range when the subscriber is online but also incorporated a feature to decline information from the AtData when the contact was no longer active. These improvements were seamlessly integrated using AWS Lambda functions, contributing to enhanced efficiency and real-time data processing.