Airflow Data Engineer - Big Data

Job Description

As a Senior Data Engineer, you will be responsible for designing, developing, and optimizing scalable data pipelines and data processing frameworks to handle rapidly growing data volume and complexity. Your deep expertise in Big Data technologies, Azure cloud services, and data pipeline orchestration tools like Airflow and Kafka will be crucial in ensuring the accuracy, reliability, and accessibility of enterprise data for advanced analytics and data-driven decision-making.Key Responsibilities:- Design, build, and maintain efficient ETL/ELT data pipelines using Apache Spark, Python, and SQL within Hadoop and Azure environments.- Develop solutions for large-scale data processing from various sources such as APIs, databases, data lakes, and streaming platforms.- Continuously enhance existing pipelines for performance optimization, scalability, and data quality assurance.- Collaborate with business analysts and data architects to design data models supporting business intelligence and analytics tools.- Apply advanced data modeling techniques like Kimball star schema and dimensional modeling for analytical and reporting use cases.- Implement data integration strategies ensuring consistency and traceability across disparate systems.- Establish data quality frameworks, validation mechanisms, and monitoring systems for accurate and consistent data.- Contribute to data governance efforts by documenting lineage, data flows, and transformation logic.Qualifications & Technical Skills:Required Technical Expertise:- 7 years of Data Engineering experience focusing on Big Data and Cloud-based solutions.- Hands-on experience with Hadoop ecosystem tools like Hive, Spark, and HDFS for at least 6 years.- Proficiency in Python (preferably PySpark), SQL, and data transformation scripting.- Strong knowledge of data modeling, ETL/ELT design patterns, and data warehousing best practices.- Expertise in Azure cloud services such as ADF, Databricks, Azure Data Lake, and Cosmos DB.- Proficient in Apache Airflow for DAG creation, scheduling, and dependency management.- Working understanding of Kafka or similar data streaming technologies for real-time data processing.- Familiarity with CI/CD practices, Git, and deployment automation tools.- Experience with relational and NoSQL databases for analytical and operational workloads.Preferred Skills:- Experience in containerization and orchestration using Docker and Kubernetes.- Familiarity with data governance and cataloging tools like Azure Purview or Collibra.- Exposure to machine learning data pipelines and feature engineering frameworks.- Knowledge of Delta Lake / Lakehouse architectures for unified batch and stream data management. As a Senior Data Engineer, you will be responsible for designing, developing, and optimizing scalable data pipelines and data processing frameworks to handle rapidly growing data volume and complexity. Your deep expertise in Big Data technologies, Azure cloud services, and data pipeline orchestration tools like Airflow and Kafka will be crucial in ensuring the accuracy, reliability, and accessibility of enterprise data for advanced analytics and data-driven decision-making.Key Responsibilities:- Design, build, and maintain efficient ETL/ELT data pipelines using Apache Spark, Python, and SQL within Hadoop and Azure environments.- Develop solutions for large-scale data processing from various sources such as APIs, databases, data lakes, and streaming platforms.- Continuously enhance existing pipelines for performance optimization, scalability, and data quality assurance.- Collaborate with business analysts and data architects to design data models supporting business intelligence and analytics tools.- Apply advanced data modeling techniques like Kimball star schema and dimensional modeling for analytical and reporting use cases.- Implement data integration strategies ensuring consistency and traceability across disparate systems.- Establish data quality frameworks, validation mechanisms, and monitoring systems for accurate and consistent data.- Contribute to data governance efforts by documenting lineage, data flows, and transformation logic.Qualifications & Technical Skills:Required Technical Expertise:- 7 years of Data Engineering experience focusing on Big Data and Cloud-based solutions.- Hands-on experience with Hadoop ecosystem tools like Hive, Spark, and HDFS for at least 6 years.- Proficiency in Python (preferably PySpark), SQL, and data transformation scripting.- Strong knowledge of data modeling, ETL/ELT design patterns, and data warehousing best practices.- Expertise in Azure cloud services such as ADF, Databricks, Azure Data Lake, and Cosmos DB.- Proficient in Apache Airflow for DAG creation, scheduling, and dependency management.- Working understanding of Kafka or similar data streaming technologies for real-time data processing.- Familiarity with CI/CD practices, Git, and deployment automat

Keyskills :
AirflowKafkaApache SparkPythonSQLHadoopData ModelingData WarehousingGitRelational databasesBig Data technologiesAzure cloud servicesETLELT design patternsAzure Data FactoryAzure DatabricksAzure Blob StorageData Lake StorageCICD pipelines