Role: Data Engineer

Location: Halifax, Canada (ONSITE)

Duration: Long Term

Key Responsibilities

- Design and implement complex ETL/ELT workflows for diverse data sources both on-prem and Cloud platform (AWS)

- Build and optimize data lake architectures on Cloudera Data Platform (CDP)

- Develop data models in Snowflake cloud warehouse and CDP environments

- Implement CI/CD pipelines using Helios for data infrastructure automation

- Collaborate with analysts to enable advanced analytics on data lake assets

- Ensure security/governance compliance for sensitive data at scale

- Technical Expertise Required:

- Python: Advanced scripting for data processing, API integration, and automation

- AWS : Amazon S3,AWS Glue

- Containerization: Docker image management; orchestration via Kubernetes/OCP4

- Data Platforms:

- Cloudera Data Platform (CDP) for Hadoop/Spark ecosystems





- Snowflake cloud data warehouse architecture

- Storage Solutions: Data lake design (S3/ADLS), structured/unstructured data optimization

- Pipeline Tools: Airflow, Kafka, Helios CI/CD

- Version Control: GitHub collaboration

- Cloud Services: AWS/Azure storage integration with CDP/Snowflake

Soft Skills:

- Problem-solving for hybrid cloud data challenges

- Cross-functional collaboration with data scientists and analysts

- Explicit communication of complex technical solutions

- Nice-to-Have Qualifications:

- Certifications: Cloudera CCP, SnowPro, Red Hat OpenShift

- Experience with streaming platforms (Flink, Kafka Streams)

- MLOps patterns for model deployment

📌 Data Engineer (Halifax Regional Municipality)
🏢 VySystems
📍 Halifax Regional Municipality