06 Nov
|
TD Synnex
|
Canada
Hyve Company Overview Hyve Solutions transforms complex engineering challenges into production reality for technology innovators building AI, cloud, and connected infrastructure. As a US-based manufacturing partner, the company rapidly delivers fast, agile execution through dep technical partnerships and co-innovation, and supply chain clarity. Hyve’s integrated ODM, CM, and SI capabilities eliminate vendor complexity while accelerating time-to-market with single-partner accountability from design through scale. The company co-innovates with deep engineering expertise, treating customer success as its own while building tomorrow's digital infrastructure. Data Warehouse SE (Data Platform Engineer)
Job Description
We are seeking an experienced Data Warehouse SE (Data Platform Engineer) to lead the evolution of our enterprise data platform from a traditional Hadoop/Hive architecture to a contemporary cloud-native Lakehouse platform built on Apache Iceberg and object storage. This role focuses on platform engineering, building and operating large-scale data infrastructure rather than developing business-specific data pipelines. The ideal candidate will have deep expertise in distributed data systems, cloud-native architectures, and large-scale data processing technologies.
Key Responsibilities
- Design, build, and operate an enterprise-scale Lakehouse platform based on Apache Iceberg to support batch, streaming, and incremental data workloads.
- Drive the migration from Hadoop/HDFS to cloud-native object storage architectures and implement compute-storage separation at scale.
- Develop and enhance core platform services, including metadata management, catalog systems, governance frameworks, and multi-engine data access capabilities.
- Optimize distributed processing engines such as Spark to improve performance,
scalability, reliability, and cost efficiency in PB-scale environments.
- Build and operate cloud-native data platforms on Kubernetes, including workload orchestration, resource management, and operational automation.
- Integrate the platform with cloud ecosystems such as Azure, Databricks, and Snowflake to enable seamless data flow and interoperability.
- Develop internal platform components and engineering tools for data ingestion, orchestration, quality management, diagnostics, and AI-driven automation. Required Skills
- Expert-level knowledge of Apache Spark, including execution planning, shuffle processing, memory management, and performance optimization.
- Strong experience with Apache Iceberg, Hudi, or Delta Lake, including metadata architecture, snapshots, manifests, schema evolution, and partition evolution.
- Hands-on experience deploying and managing data workloads on Kubernetes, including Spark on Kubernetes, Operators, and resource scheduling.
- Solid understanding of cloud object storage platforms such as Azure Blob Storage, Amazon S3, or MinIO, including storage consistency and performance characteristics.
- Deep understanding of modern Lakehouse architecture, including Iceberg, Spark, catalog services, and object storage ecosystems.
- Experience with at least one cloud data platform such as Azure, Databricks, or Snowflake.
- Strong software engineering and programming skills in Java, Scala, or Python.
Preferred Qualifications
- Experience designing and delivering enterprise-level data platform solutions rather than single business applications.
- Hands-on experience with Lakehouse implementation, data lake modernization, or platform migration initiatives.
- Proven experience deploying and operating large-scale data platforms on Kubernetes.
- Strong track record of diagnosing and resolving complex performance issues in distributed systems.
- Experience reading, debugging, or contributing to open-source projects such as Spark, Iceberg, Flink, or Trino.
- Practical experience applying AI technologies to data platforms, including Data Agents, intelligent automation, or AI-assisted analytics. Educational background and Working Experience
- Bachelor's or Master's degree in Computer Science, Software Engineering, Information Systems, or a related technical field.
- Minimum 3 years of experience in large-scale data platform engineering or distributed data systems development.
- Proven experience building platform-level solutions rather than business-specific data engineering applications.
- Experience supporting PB-scale data environments or large distributed computing clusters is highly preferred.
- Experience in the direction or participation of Lakehouse modernization, Hadoop migration, or cloud-native data platform transformation projects is preferred.
- Experience in the migration of Hadoop/HDFS-based platforms to cloud-native architectures and object storage solutions is a strong plus.
- Fluent in Mandarin Chinese is a plus, particularly for collaboration with global engineering teams across multiple regions. Compensation The anticipated base salary range for this position is $70,000
📌 Data Warehouse System Engineer (Canada)
🏢 TD Synnex
📍 Canada