Architect (Halifax Regional Municipality)

Architect (Halifax Regional Municipality)

19 Nov
|
Virtusa
|
Halifax Regional Municipality

19 Nov

Virtusa

Halifax Regional Municipality

AWS Data Engineer with Databricks and DBT

Responsibilities

- Design and Development of Data Pipelines:
- Design, build, and optimize robust ETL/ELT pipelines using AWS services (S3, Glue, Lambda) and the Databricks platform (Spark, Delta Lake, DLT).
- Ingest and process large volumes of structured and semi-structured data from various sources (APIs, databases, streaming platforms like Kafka or Kinesis) into a centralized data lake or lakehouse.
- Data Transformation and Modeling:
- Develop and maintain data models (e.g., star/snowflake schemas, medallion architecture) optimized for analytics and BI tools using dbt (Data Build Tool).
- Write complex and productive SQL queries and Python/PySpark code for data manipulation, transformation, and validation within the Databricks environment.
- Implement data quality checks, tests, and documentation as part of the dbt workflow, enforcing data governance and security standards.
- Orchestration and Automation:
- Orchestrate and monitor data workflows using Databricks Jobs or external tools like AWS MWAA (Managed Workflows for Apache Airflow).
- Implement CI/CD pipelines and version control (Git)



for all data engineering artifacts (code, configurations, dbt models) to ensure reliable and consistent deployments.
- Performance Optimization and Operations:
- Monitor, troubleshoot, and resolve issues in production data pipelines and environments to ensure high performance, reliability, and cost-efficiency.
- Tune Spark jobs and optimize Delta Lake features (Z-Order, partitioning) to handle growing data volumes and complexity.
- Collaboration and Support:
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.
- Provide expertise and guidance on data best practices, promoting a culture of data quality and governance.

Must Have skills
- SQL
- DBT core and DBT Cloud
- AWS (redshift)
- Data bricks with AWS
- SQL server DB
- Stone branch scheduling tool
- Should understand CICD, GIT
- Work in Agile environment with JIRA

Other Skills Required/ Good To Have

- Tableau experience

Harness devops Proficient in Linux / Unix environments

📌 Architect (Halifax Regional Municipality)
🏢 Virtusa
📍 Halifax Regional Municipality

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: architect (halifax regional municipality) / halifax regional municipality