Big Data and Cloud Data Engineer - Manager

Job Description

As a Big Data and Cloud Data Engineer at our company, you will play a crucial role in designing, building, and managing scalable data infrastructure and pipelines in the cloud environment. Your primary responsibility will be to ensure that data is accessible, reliable, and processed efficiently to meet various business needs. This role will require a blend of software engineering, data warehousing, and big data technologies expertise.Key Responsibilities:- Develop, construct, test, and maintain data architectures such as databases and large-scale processing systems.- Align data architectures with business requirements, ensuring data quality, governance, and implementing data security measures.- Collaborate closely with data scientists and analysts to understand their needs and provide necessary data for analysis.Qualifications Required:- Bachelors degree in computer science or engineering.- Minimum 7 years of hands-on experience with data-focused application development, including ETL and data engineering.- Proficiency in advanced Bigdata technologies like Hadoop, Spark, Hive, SQL, and NoSQL.- Strong programming skills with 3 years of hands-on experience in Scala and/or Python.- 2 years of experience with public cloud technologies such as AWS and Data Bricks.- Experience in building and optimizing Spark jobs for performance and cost efficiency.- Prior experience in working with CI/CD pipelines and orchestration tools.- Ability to monitor and troubleshoot production data pipelines.- Knowledge of ETL Tools, DEVOPS, and GitHub is preferable.- Exposure to Elasticsearch, Graph DB, and machine learning is a plus.- Knowledge of Finance, custody services, data privacy, AML, and related compliance domain is advantageous.In addition to the technical qualifications, you will need prior experience with Finance and compliance domains, as well as certifications in Databricks and Cloud environments. The work requirement includes a hybrid work mode with 3-4 days in the office per week and working hours from 11 AM to 8 PM IST.About State Street:At State Street, we are dedicated to helping institutional investors manage risk, respond to challenges, and drive performance and profitability globally. We prioritize our clients and understand that engaged employees are vital to our success. We are committed to creating an inclusive environment where every employee is valued and empowered to reach their full potential. Join us in shaping the future together.As an Equal Opportunity Employer, we consider all qualified applicants for positions without discrimination based on various characteristics protected by applicable law.For more information on job opportunities at State Street, visit StateStreet.com/careers. As a Big Data and Cloud Data Engineer at our company, you will play a crucial role in designing, building, and managing scalable data infrastructure and pipelines in the cloud environment. Your primary responsibility will be to ensure that data is accessible, reliable, and processed efficiently to meet various business needs. This role will require a blend of software engineering, data warehousing, and big data technologies expertise.Key Responsibilities:- Develop, construct, test, and maintain data architectures such as databases and large-scale processing systems.- Align data architectures with business requirements, ensuring data quality, governance, and implementing data security measures.- Collaborate closely with data scientists and analysts to understand their needs and provide necessary data for analysis.Qualifications Required:- Bachelors degree in computer science or engineering.- Minimum 7 years of hands-on experience with data-focused application development, including ETL and data engineering.- Proficiency in advanced Bigdata technologies like Hadoop, Spark, Hive, SQL, and NoSQL.- Strong programming skills with 3 years of hands-on experience in Scala and/or Python.- 2 years of experience with public cloud technologies such as AWS and Data Bricks.- Experience in building and optimizing Spark jobs for performance and cost efficiency.- Prior experience in working with CI/CD pipelines and orchestration tools.- Ability to monitor and troubleshoot production data pipelines.- Knowledge of ETL Tools, DEVOPS, and GitHub is preferable.- Exposure to Elasticsearch, Graph DB, and machine learning is a plus.- Knowledge of Finance, custody services, data privacy, AML, and related compliance domain is advantageous.In addition to the technical qualifications, you will need prior experience with Finance and compliance domains, as well as certifications in Databricks and Cloud environments. The work requirement includes a hybrid work mode with 3-4 days in the office per week and working hours from 11 AM to 8 PM IST.About State Street:At State Street, we are dedicated to helping institutional investors manage risk, respond to challenges, and drive performance and profitabilit

Keyskills :
ETLData EngineeringHadoopSparkHiveSQLNoSQLScalaPythonAWSETL ToolsDEVOPSGitHubElasticsearchMachine LearningData BricksCICD