Build production-grade AWS data pipelines with Spark and PySpark as a Senior Data Engineer supporting Middle Office dataset integration and delivery. This subcontractor role covers dataset onboarding, integration, and operationalization for analytics, reporting, and downstream applications. You will own the lifecycle from source analysis and solution design through implementation, deployment, and operational support.
The work requires scalable data engineering across AWS and modern data lake technologies. Key Responsibilities:
Analyze source systems and onboard recent datasets Develop AWS pipelines using EMR, Glue, Athena, and S3 Automate workflows with Airflow scheduling and monitoring Design schemas, partitioning, and data quality controls Troubleshoot production data and performance issues Requirements:
Strong AWS data platform and pipeline experience Advanced Apache Spark and PySpark expertise Apache Hudi or Apache Iceberg experience Strong Python, testing, Git, and software engineering practices Terraform, CI/CD, and workflow orchestration experience Apply Senior Data Engineering expertise across AWS, Spark, Airflow, and production data solutions for Middle Office functions.