Senior Data Engineer

Job Description

This engineering service focused position includes those who develop and maintain the information technologies, functional excellence or product information change management that enables the engineering practitioners to work more efficiently and effectively.Investigates technical information and knowledge based infrastructure opportunities and challenges; understands causal mechanisms, recommends appropriate action, owns problem resolution and documents results.Designs and/or provides support of information processes such as Engineering Standard Work (ESW), Value Package Introduction (VPI, change management and data repositories such as technical information libraries, all required to meet the needs of practitioners within the technical function.Obtains input and negotiates with a broad variety of technical practitioners to deliver new and revised processes and tools that advance the capability of the function and its ability to deliver high quality drawings and specifications.Makes decisions, plans and implements new and revised processes, work instructions, software and/or computing solutions that improves the technical quality, efficiency and/or effectiveness of the technical function.Provides independent leadership of smaller business impact projects or ownership of complex systems or services with full accountability to the project team.Delivers independent execution of established and emerging work processes and systems, while still developing technology or product knowledge.Leads the development and improvement of work processes and systems across a functional discipline area within a business unit site.Coordinates and directs work amongst technicians and temporary student employees, assists in the transfer of knowledge to lesser experienced engineers through either indirect (scope of influence) or direct management of a small, local group of engineers.Provides support and guidance to influence technical direction within a project team and continues to develop proficiency in the competency areas critical to success in the role., SkillsDevelopment and Management of Technical Function Standards and Practices - Creates and modifies standardized work processes, instructions, tasks or tools utilizing new learning and knowledge gained as part of product development, process improvement or other technical development activities; advises stakeholders with the goal of enabling improved technical productivity and effective knowledge transfer to others who were not originally part of the initial learning.Solution Assessment and Validation - Assesses the proposed solution using the requirements traceability through the life cycle of the requirements to determine if they deliver the desired business outcome.Requirements Elicitation - Discovers and documents functional and non-functional requirements using the Business Analysis Toolkit (modeling the five aspects and creating use cases) to effectively communicate the business needs and solution characteristics for solution design.Knowledge Management - Ensures perspectives, ideas, experiences and information are available in the right place and the right time using the required processes and tools to enable informed decisions.Troubleshooting - Diagnoses and resolves the customer-identified issue by utilizing appropriate department standards to satisfy customer expectations.Optimizes work processes - Knowing the most effective and efficient processes to get things done, with a focus on continuous improvement.Decision quality - Making good and timely decisions that keep the organization moving forward.Drives results - Consistently achieving results, even under tough circumstances.Communicates effectively - Developing and delivering multi-mode communications that convey a clear understanding of the unique needs of different audiences.Builds networks - Effectively building formal and informal relationship networks inside and outside the organization.Resourcefulness - Securing and deploying resources effectively and efficiently.Manages complexity - Making sense of complex, high quantity, and sometimes contradictory information to effectively solve problems.Education, Licenses, CertificationsCollege, university, or equivalent Bachelors degree in Engineering, Computer Science or Information Technology required.ExperiencePrior Senior or Lead Engineer equivalent work experience in a relevant discipline area is required with a demonstrated track record of technical problem solving and quality decision making. Years of experience 6-8 Technical Skills required MicMicrosoft Azure Platform (ADLS, Event Hub, Stream Analytics, Data Factory, SQL DB)SpaSpark v2.0+, Scala, Hive, SQL, HDFS, Kafka, Shell script, Oozie, Appworks, data modeling Responsibilities

• Design and implement End-to-end pipelines for Data Ingestion, Transformation and Storage for various data sources
• Write performance optimized Spark code and Hive/SQL queries for TB-scale data
• Mentoring junior developers
• Production deployment and post-production support till applications are handed over to production support team

Keyskills :
content managementemail sqlanalytics apachelife cycle shell scriptpost production compu