Build production-grade AWS data pipelines with Spark and PySpark as a Senior Data Engineer supporting Middle Office dataset integration and delivery.This subcontractor role covers dataset onboarding, integration, and operationalization for analytics, reporting, and downstream applications. You will own the lifecycle from source analysis and solution design through implementation, deployment, and operational support. The work requires scalable data engineering across AWS and up-to-date data lake technologies.Key Responsibilities:
Analyze source systems and onboard new datasetsDevelop AWS pipelines using EMR, Glue, Athena, and S3Automate workflows with Airflow scheduling and monitoringDesign schemas, partitioning, and data quality controlsTroubleshoot production data and performance issuesRequirements:
Strong AWS data platform and pipeline experienceAdvanced Apache Spark and PySpark expertiseApache Hudi or Apache Iceberg experienceStrong Python, testing, Git, and software engineering practicesTerraform, CI/CD, and workflow orchestration experienceApply Senior Data Engineering expertise across AWS, Spark, Airflow, and production data solutions for Middle Office functions.