19 Nov
|
Virtusa
|
Halifax Regional Municipality
19 Nov
Virtusa
Halifax Regional Municipality
AWS Data Engineer with Databricks and DBT Responsibilities
• Design and Development of Data Pipelines:
• Design, build, and optimize robust ETL/ELT pipelines using AWS services (S3, Glue, Lambda) and the Databricks platform (Spark, Delta Lake, DLT).
• Ingest and process large volumes of structured and semi-structured data from various sources (APIs, databases, streaming platforms like Kafka or Kinesis) into a centralized data lake or lakehouse.
• Data Transformation and Modeling:
• Develop and maintain data models (e.g., star/snowflake schemas, medallion architecture) optimized for analytics and BI tools using dbt (Data Build Tool).
• Write complex and productive SQL queries and Python/PySpark code for data manipulation, transformation, and validation within the Databricks environment.
• Implement data quality checks, tests, and documentation as part of the dbt workflow, enforcing data governance and security standards.
• Orchestration and Automation:
• Orchestrate and monitor data workflows using Databricks Jobs or external tools like AWS MWAA (Managed Workflows for Apache Airflow).
• Implement CI/CD pipelines and version control (Git)
for all data engineering artifacts (code, configurations, dbt models) to ensure reliable and consistent deployments.
• Performance Optimization and Operations:
• Monitor, troubleshoot, and resolve issues in production data pipelines and environments to ensure high performance, reliability, and cost-efficiency.
• Tune Spark jobs and optimize Delta Lake features (Z-Order, partitioning) to handle growing data volumes and complexity.
• Collaboration and Support:
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.
• Provide expertise and guidance on data best practices, promoting a culture of data quality and governance. Must Have skills
• SQL
• DBT core and DBT Cloud
• AWS (redshift)
• Data bricks with AWS
• SQL server DB
• Stone branch scheduling tool
• Should understand CICD, GIT
• Work in Agile environment with JIRA Other Skills Required/ Good To Have
• Tableau experience Harness devops Proficient in Linux / Unix environments
📌 Architect (Halifax Regional Municipality)
🏢 Virtusa
📍 Halifax Regional Municipality