Data Engineer

Job Description

Key Responsibilities:
• Deliverables:
o Document foundational data tables to establish a robust data architecture.o Migrate and validate test tables to ensure data accuracy and integrity.o Develop self-serve ETL frameworks for both streaming and batch processing, empowering teams to access and analyze data efficiently.o Integrate schema registry and establish data lineage to enhance data governance and compliance.
• Collaboration:
o Work closely with onshore teams to understand data needs and translate them into technical solutions.o Collaborate with stakeholders to gather requirements and provide actionable business insights.
• Data Transformation:
o Transform raw data into governed and lucid datasets that meet business standards.o Design and deploy production-quality data pipelines to ensure timely data availability.
• Technical Expertise:
o Write advanced SQL and develop data processing workflows using tools such as Spark, AWS Glue, EMR, Airflow, and Python.o Utilize MPP database systems like Snowflake, AWS Redshift, or Teradata to optimize data storage and retrieval.
• Monitoring and Metrics:
o Understand key metrics for data pipelines and build solutions to provide visibility to partner teams, ensuring the performanceand reliability of data solutions.Qualifications:
• 3 years of experience working with onshore teams in a collaborative environment.
• 5 years of experience in transforming data into governed and lucid datasets.
• 5 years of hands-on experience in building and deploying production-quality data pipelines.
• 5 years of experience working with stakeholders to derive business insights from data.
• 5 years of advanced proficiency in SQL, Spark, AWS Glue, EMR, Airflow, Python, Snowflake, and AWS Redshift.
• 5 years of hands-on experience with MPP database systems such as Snowflake, AWS Redshift, or Teradata.
• Hands-on experience with Python and SQL is a must
• Strong Python knowledge with hands-on experience in using Python for data processing and pipeline development.
• Strong understanding of key metrics for data pipelines and experience in building visibility solutions for partner teams.
Preferred Skills:
• Excellent problem-solving skills and the ability to work in a fast-paced environment.
• Strong communication skills to effectively collaborate with technical and non-technical stakeholders.
• Strong SQL, snowflake/redshift/BQ, Spark & Python skill.
• Experience with data governance practices and data lineage tracking.

Keyskills :
data modelingsnow flakepythonsqlaws