Job Type: Full time Permanent
Design and develop Big Data pipelines using Spark and Scala to process large volumes of raw data stored in Hive/Hadoop.
Analyze raw Hive tables, source schemas, data dictionaries and source-to-target mappings to understand business data.
Develop Spark/Scala transformation frameworks to convert raw source data into standardized conformed data views/tables.
Implement complex business transformation rules including joins, filters, aggregations, derivations and reference-data lookups.
Standardize data across multiple source systems to create a consistent enterprise/conformed data model.
Implement incremental processing, partition-based processing and historical data handling for high-volume datasets.
Develop reusable Spark/Scala components for data enrichment, transformation and validation.
Implement data-quality controls for completeness, uniqueness, validity, referential integrity and business-rule validation.
📌 Big Data Engineer Toronto
🏢 Zodiac Solutions
📍 Toronto
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.