Senior Data Engineer

Job Description

As a Data Engineer at this company, you will be responsible for designing and implementing highly performant data ingestion pipelines from multiple sources. You will deliver and present proofs of concept of key technology components to project stakeholders and develop scalable and re-usable frameworks for ingesting data sets. Your role will involve integrating the end-to-end data pipeline to ensure the quality and consistency of data is maintained at all times. You will work with event-based and streaming technologies to ingest and process data, do analysis of new requirements, and propose solutions.Key Responsibilities:- Hands-on experience with PySpark, Python, Spark, Scala for data validation, delivery, quality, and integrity- Snowflake/Databricks Cloud Data warehouse implementation experience, handling various file formats (XML, JSON, Avro, Parquet)- Experience in Extraction, Transformation, and Loading (ETL) data from multiple sources & formats- Working with distributed data systems such as Hadoop, Spark, Kafka- Experience in business intelligence, reporting, and visualization solutions (Tableau, Power BI)- Exposure to cloud technologies of AWS - EC2, S3, Lambda, SQS, SNS- Strong programming experience using PL/SQL for packages, procedures, views, triggers, indexes- Understanding of Data Warehousing concepts using Fact, Dimension tables and data loading methods- Hands-on experience with Big Data model techniques using Python/Spark- Experience in CI/CD domain using Jenkins, Terraform, Docker, AWS, KubernetesQualifications Required:- At least 3 years of relevant experience in the field- Experience in Pharma industry is a plusThe company, Labcorp, values inclusion and belonging in the workforce and does not tolerate harassment or discrimination of any kind. Qualified applicants will receive equal consideration for employment without regard to various legally protected characteristics. For more information about Labcorps approach to privacy and data handling, you can refer to their Privacy Statement. As a Data Engineer at this company, you will be responsible for designing and implementing highly performant data ingestion pipelines from multiple sources. You will deliver and present proofs of concept of key technology components to project stakeholders and develop scalable and re-usable frameworks for ingesting data sets. Your role will involve integrating the end-to-end data pipeline to ensure the quality and consistency of data is maintained at all times. You will work with event-based and streaming technologies to ingest and process data, do analysis of new requirements, and propose solutions.Key Responsibilities:- Hands-on experience with PySpark, Python, Spark, Scala for data validation, delivery, quality, and integrity- Snowflake/Databricks Cloud Data warehouse implementation experience, handling various file formats (XML, JSON, Avro, Parquet)- Experience in Extraction, Transformation, and Loading (ETL) data from multiple sources & formats- Working with distributed data systems such as Hadoop, Spark, Kafka- Experience in business intelligence, reporting, and visualization solutions (Tableau, Power BI)- Exposure to cloud technologies of AWS - EC2, S3, Lambda, SQS, SNS- Strong programming experience using PL/SQL for packages, procedures, views, triggers, indexes- Understanding of Data Warehousing concepts using Fact, Dimension tables and data loading methods- Hands-on experience with Big Data model techniques using Python/Spark- Experience in CI/CD domain using Jenkins, Terraform, Docker, AWS, KubernetesQualifications Required:- At least 3 years of relevant experience in the field- Experience in Pharma industry is a plusThe company, Labcorp, values inclusion and belonging in the workforce and does not tolerate harassment or discrimination of any kind. Qualified applicants will receive equal consideration for employment without regard to various legally protected characteristics. For more information about Labcorps approach to privacy and data handling, you can refer to their Privacy Statement.

Keyskills :
PythonSparkScalaSnowflakeETLHadoopKafkaTableauPower BIAWSPLSQLData WarehousingBig DataJenkinsDockerKubernetesPySparkDatabricksCICD