24 Sep
|
Iris Software
|
Toronto
24 Sep
Iris Software
Toronto
Iris's direct client, one of the Top 5 Bank in Canada, is looking to hire a
Sr Data Engineer - Apache Iceberg a long-term opportunityat
Toronto, ON (Hybrid).
Our client is a Canadian multinational financial services company and the largest bank in Canada by market capitalization. The bank serves over 17 million clients and has more than 89,000 employees worldwide. Bank is serving individual consumers, small and middle market businesses and large corporations with a full range of banking, investing, asset management and other financial and risk-management products and services.
Location: Toronto, ON (Hybrid) Duration: Long Term (12+ Months) Role summary
Looking for experienced data engineers/data modelers to support large-scale data modernization initiatives using Apache Iceberg. The role will work with architecture and application teams to design, build and optimize governed analytical datasets and reusable data patterns on a modern lakehouse/data platform.
Key responsibilities
Design and build production-grade data pipelines and data products using Apache Iceberg.
Design logical and physical data models for large analytical and regulatory datasets.
Define Iceberg table structures, partitioning strategies, schema evolution and data lifecycle patterns.
Build batch and/or streaming ingestion pipelines into Iceberg tables.
Implement merge/upsert, CDC, incremental processing and historical data handling patterns.
Tune tables and queries for performance, including file sizing, compaction, partition evolution and metadata management.
Work with architects to establish reusable Iceberg engineering and modeling standards.
Implement data quality, reconciliation, lineage, security and governance controls.
Troubleshoot performance, concurrency and data consistency issues across large datasets.
Partner with business/data SMEs to translate source data into well-defined canonical/consumption models.
Support CI/CD, automated testing and operational monitoring of data pipelines.
Core skills
Solid hands-on experience with Apache Iceberg in production.
Strong SQL and data modeling skills.
Hands-on experience with Spark / PySpark or equivalent distributed processing technologies.
Experience with cloud object storage and modern lakehouse architectures.
Deep understanding of
Iceberg snapshots and time travel schema and partition evolution
ACID transactions compaction and small-file management
Experience designing dimensional, normalized and/or canonical enterprise data models.
Experience handling high-volume and complex enterprise datasets.
Good understanding of data governance, lineage, security and data quality.
Good to have
Experience with one or more of AWS, Databricks, Snowflake, Athena, Trino/Presto, EMR, Glue Catalog, Hive Metastore or similar technologies.
Kafka or other streaming/CDC technologies.
Experience modernizing legacy warehouse or Netezza/Hadoop-style workloads onto a Lakehouse architecture.
Exposure to regulatory, risk, finance or capital-markets datasets.
📌 AI Solution Architect / Partner (Banking) (Toronto)
🏢 Iris Software
📍 Toronto