Big Data Developer

Job Description

Key Responsibilities
• Design, develop, and maintain large-scale distributed data pipelines using Apache Spark and Scala.
• Write clean, efficient, and maintainable Scala code adhering to industry best practices.
• Implement Spark Core, Spark SQL, and Spark Streaming modules for real-time and batch data processing.
• Collaborate with cross-functional teams to gather and understand data processing requirements.
• Optimize performance of complex queries and processing logic in Hadoop ecosystems.
• Develop and manage workflows using UNIX shell scripting, Hive, Sqoop, and Impala.
• Participate actively in Agile/Scrum ceremoniesdaily stand-ups, sprint planning, retrospectives, etc.
• Provide production support and maintenance for existing applications, ensuring high availability and performance.
• Conduct root cause analysis and resolve data pipeline issues in collaboration with upstream/downstream teams.
• Stay updated with the latest Big Data technologies and contribute to continuous improvement initiatives.
Mandatory Skills
• Strong proficiency in Scala with at least 8 years of hands-on experience.
• Solid experience with Apache Spark for building distributed data processing applications.
• In-depth understanding of data structures, algorithms, and design patterns.
• Strong command over SQL and working knowledge of NoSQL databases.
Desired Skills
• Spark Streaming
• Hadoop and HDFS
• Hive, Sqoop, Impala
• UNIX shell scripting
• Familiarity with ETL concepts, Data Warehousing

Keyskills :
sqlspark