Senior Data Engineer

Job Description

About the Company Our Purpose: Minfy is on a BIONIC JOURNEY TO BUILD A BETTER WORLD. Bionic signifies that both humans and deep tech are essential for success. Humans stand for natural intelligence, consciousness, conscience, freewill, empathy and compassion. Deeptech implies use of modern technologies including cloud, artificial intelligence (AI), artificial general intelligence and quantum computing. We believe and promote a symbiotic relationship between humans and advanced tech for ultimate success. Journey While we are focused on delivering the outcome, we are mindful about the journey. The path is as important as the outcome. We strive to make the journey a pleasant experience for all our stakeholders including customers, partners, employees and shareholders. Build a Better World We want to contribute to building a better world which is all encompassing. Better economic prospects, Better Health, Better Education and Better Governance are a few examples of our goals. Minfys Nine. Cultural Values that guide our conduct. 1. Harness Peoples Talent 2. Customer Centricity 3. Growth Mindset 4. Excellence with High Velocity 5. Ownership 6. Think & Execute Big 7. Simplification is the Ultimate Sophistication 8. Deliver Outcome 9. Mutual Trust & Respect Opportunity We are seeking an experienced Senior AWS Data Engineer to join our data engineering team. The ideal candidate will have a strong background in designing and implementing data pipelines, data warehouses, and data lakes on the AWS cloud platform. You will be responsible for building and maintaining robust, scalable, and efficient data solutions that support our data analytics and machine learning initiatives. Responsibilities: Data Architecture: Design and implement data architecture solutions on AWS, including data lakes, data warehouses, and streaming data pipelines. ETL Development: Develop and maintain ETL (Extract, Transform, Load) processes to ingest, process, and transform data from various sources into a usable format for analytics and reporting. Data Modeling: Create and maintain data models that support analytical and reporting requirements. Optimize data structures for performance and scalability. AWS Services: Leverage a wide range of AWS services such as S3, Redshift, Glue, EMR, Athena, Kinesis, Lambda, and more to build data solutions. Data Quality: Implement data quality checks and monitoring to ensure data accuracy, completeness, and reliability. Performance Optimization: Continuously optimize data pipelines and infrastructure for improved performance, cost efficiency, and scalability. Security and Compliance: Implement data security and compliance best practices, including data encryption, access controls, and auditing. Documentation: Maintain comprehensive documentation of data engineering processes, configurations, and workflows. Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand their data requirements and provide data engineering support. Troubleshooting: Diagnose and resolve data engineering-related issues in a timely manner. Qualifications: Bachelors degree in Computer Science, Information Technology, or a related field. Masters degree is a plus. Proven experience as a Data Engineer with a strong focus on AWS cloud platform. In-depth knowledge of AWS services and tools for data engineering and analytics. Proficiency in programming languages such as Python, Java, or Scala. Strong SQL skills and experience with relational databases and data warehousing concepts. Experience with big data technologies such as Hadoop, Spark, and Kafka. Understanding of data governance, data quality, and data privacy. Excellent problem-solving and troubleshooting skills. Strong communication and collaboration skills. AWS certifications (e.g., AWS Certified Data Analytics, AWS Certified Data Warehousing) are a plus. 7 years of experience in data engineering / Data integrations. Formative years may be in development projects in data warehouses, DataMarts, ETL, data modelling, data rich middlewares, and distributed computing solutions. 4 years in Cloud services with at least 2 implementation/development project Last 5 years need to be in big data engineering Cloud Integrations, Serverless, S3/BLOB, Hadoop, HBase, Hive, Spark, Spark streaming, Kafka in-memory database systems, Databases (NoSQL and SQL) Should have programmed extensively in python, Pyspark, scala, java or .net. Should have managed an industry standard program in cloud building data pipelines, data migrations or analytics pipelines. Should have worked as software solution provider at the level of solution architect, technology evangelist in medium to large ISV. Direct experience of at least 3 years in design and delivery of bigdata solutions in healthcare vertical Preferred Skills: Experience with serverless computing using AWS Lambda. Familiarity with containerization and orchestration using Docker and Kubernetes. Knowledge of DevOps practices and CI/CD pipelines. Experience with data orchestration tools like Apache Airflow. Minfys Focus areas. We help enterprises accelerate Digital Transformation, Cloud Adoption, and Innovation with transformative services around Cloud Consulting, Migration & Legacy Modernizations, and Cloud-Native Application Development. Credentials Minfy has consistently featured among Top Global Cloud MSP including being listed in the Top 100 by Channel E2E 2022 List and FT High Growth Companies Asia-Pacific. Having built synergies as an AWS Premier Consulting Partner, Azure Gold Partner and Google Consulting Partner, Minfy serves clients across Asia, Europe and USA, extending into multiple industries including BFSI, Education, Manufacturing, Healthcare and Public Sector. Experience Minfy has diverse set of customers from Digital Natives, New Age companies in gaming, crypto, startups in healthtech, fintech, edtechs. Minfy has a credible track record with 20 large customers in the Public Sector, 15 Large Data Center Migrations and several multi-year contracts with large enterprises. We have managed services for 6000 servers, executed 450 migrations, supported 100s of SMBs. Minfy has over 500 employees and is also selected in the Stanford Seed program 2022. We are consistently growing with a CAGR of 100% YOY, since inception (2016) and expect to continue on this growth trajectory in the coming years,

Keyskills :
AWSData ArchitectureData ModelingData QualityPerformance OptimizationDocumentationCollaborationTroubleshootingPythonJavaScalaSQLHadoopSparkKafkaData GovernanceData PrivacyCommunicationCloud ServicesContainerizationOrchestrationDevOps