Software Engineering Data, Lakehouse and AI Data Platform Engineer

Job Description

Role Overview:As a Data Engineer in the Lakehouse and AI Data Platform team, your primary responsibility will be to design, build, test, and support data pipelines and curated datasets on the firms modern data platform. You will be involved in various aspects such as ingestion, transformation, modeling, optimization, and data quality, with the aim of delivering reliable, scalable, and fit-for-purpose data products. Additionally, you may also contribute to enhancing shared tooling or framework components to improve platform usability and operations.Key Responsibilities:- Pipeline Engineering - Build, enhance, and support batch and streaming data pipelines on the Lakehouse and AI data platform. - Refactor or modernize existing data flows to enhance reliability, performance, and maintainability. - Develop reusable tooling to improve delivery, consistency, and operational support. - Ensure data pipelines are production-ready, well tested, and operationally supportable.- Data Modelling and Curation - Develop raw, refined, and curated datasets to support analytics, reporting, and AI use cases. - Apply sound data modeling principles to accurately represent business entities, relationships, and historical changes. - Collaborate with consumers to shape data products that are usable, well-documented, and aligned with business needs.- Data Quality and Reconciliation - Implement controls to validate the completeness, accuracy, and consistency of data across pipelines and datasets. - Use reconciliation approaches to build confidence in production outputs and investigate breaks. - Contribute to clear standards for testing, monitoring, and issue resolution. - Drive practical improvements in testing, monitoring, or reconciliation tooling to enhance platform reliability and day-to-day delivery.- Delivery and Partnership - Collaborate with engineers, platform teams, and data consumers to deliver agreed outcomes within set timelines and quality standards. - Communicate progress, risks, dependencies, and design choices effectively, including suggesting improvements to shared platform tooling. - For more experienced candidates, take on a broader role in technical leadership, task breakdown, and support for junior engineers.Qualification Required:- 7-12 years of experience- Bachelors or masters degree in a relevant discipline or equivalent practical experience with strong quantitative skills or data engineering expertise.- Strong hands-on programming experience in Python or Java.- Proficiency in SQL, including troubleshooting, optimization, and data analysis.- Ability to quickly learn new tools, internal platforms, and delivery workflows.- Familiarity with software engineering fundamentals such as version control, testing, release discipline, and CI/CD practices.Additional Company Details:The role involves working with a modern and evolving data stack. Candidates are not expected to have deep expertise in every tool from day one but should bring relevant experience and the ability to work across comparable technologies. The technology environment includes data processing and logic (such as ANSI SQL, Apache Spark, Kafka), data formats (JSON, Avro, Parquet), platforms and storage (Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ), and engineering and deployment tools (CI/CD tooling, containerized or Kubernetes-based deployment approaches where relevant). You will also work with internal data management and platform tooling, requiring a practical and adaptable engineering mindset.Note: The job description does not include any additional company details. Role Overview:As a Data Engineer in the Lakehouse and AI Data Platform team, your primary responsibility will be to design, build, test, and support data pipelines and curated datasets on the firms modern data platform. You will be involved in various aspects such as ingestion, transformation, modeling, optimization, and data quality, with the aim of delivering reliable, scalable, and fit-for-purpose data products. Additionally, you may also contribute to enhancing shared tooling or framework components to improve platform usability and operations.Key Responsibilities:- Pipeline Engineering - Build, enhance, and support batch and streaming data pipelines on the Lakehouse and AI data platform. - Refactor or modernize existing data flows to enhance reliability, performance, and maintainability. - Develop reusable tooling to improve delivery, consistency, and operational support. - Ensure data pipelines are production-ready, well tested, and operationally supportable.- Data Modelling and Curation - Develop raw, refined, and curated datasets to support analytics, reporting, and AI use cases. - Apply sound data modeling principles to accurately represent business entities, relationships, and historical changes. - Collaborate with consumers to shape data products that are usable, wel

Keyskills :
PythonJavaSQLData QualityData ReconciliationApache SparkJSONAvroANSI SQLKafkaSnowflakeSybase IQData ModellingData PipelinesParquetCICDApache IcebergDatabricks