Kafka Streaming Developer

Job Description

As a Kafka Streaming Developer in Kolkata with 4 to 6 years of experience, your role will involve designing, developing, and maintaining robust and scalable real-time data streaming pipelines using Apache Kafka. You will also be responsible for integrating these pipelines with Snowflake for data warehousing and analytics. Your expertise in Kafka ecosystem components, streaming data ingestion, and optimizing data flow into Snowflake will be crucial for this role.**Key Responsibilities:**- Design, build, and maintain real-time data streaming pipelines using Apache Kafka, Kafka Connect, Kafka Streams, and ksqlDB.- Develop and implement Kafka producers and consumers for various data sources and targets.- Integrate Kafka with Snowflake using Kafka Connectors (e.g., Snowflake Sink Connector), Snowpipe Streaming, or custom solutions for efficient data ingestion.- Configure and manage Kafka clusters, topics, partitions, and security settings (SSL, SASL, RBAC).- Optimize Kafka cluster performance, ensuring high availability, scalability, and fault tolerance.- Develop and implement data transformations and processing logic within Kafka Streams or other streaming frameworks.- Design and implement scalable Snowflake data warehouse solutions, including tables, views, and materialized views, optimized for analytical queries.- Monitor and troubleshoot Kafka and Snowflake data pipelines, identifying and resolving performance bottlenecks and data discrepancies.- Collaborate with data engineers, data scientists, and software developers to understand data requirements and deliver integrated solutions.- Ensure data quality, integrity, and security across the streaming and warehousing solutions.- Document technical designs, configurations, and operational procedures.- Participate in code reviews and promote best practices in Kafka and Snowflake development.This role offers the opportunity to work on cutting-edge technologies and be part of a collaborative team that focuses on delivering high-quality solutions in the domain of real-time data streaming and data warehousing. As a Kafka Streaming Developer in Kolkata with 4 to 6 years of experience, your role will involve designing, developing, and maintaining robust and scalable real-time data streaming pipelines using Apache Kafka. You will also be responsible for integrating these pipelines with Snowflake for data warehousing and analytics. Your expertise in Kafka ecosystem components, streaming data ingestion, and optimizing data flow into Snowflake will be crucial for this role.**Key Responsibilities:**- Design, build, and maintain real-time data streaming pipelines using Apache Kafka, Kafka Connect, Kafka Streams, and ksqlDB.- Develop and implement Kafka producers and consumers for various data sources and targets.- Integrate Kafka with Snowflake using Kafka Connectors (e.g., Snowflake Sink Connector), Snowpipe Streaming, or custom solutions for efficient data ingestion.- Configure and manage Kafka clusters, topics, partitions, and security settings (SSL, SASL, RBAC).- Optimize Kafka cluster performance, ensuring high availability, scalability, and fault tolerance.- Develop and implement data transformations and processing logic within Kafka Streams or other streaming frameworks.- Design and implement scalable Snowflake data warehouse solutions, including tables, views, and materialized views, optimized for analytical queries.- Monitor and troubleshoot Kafka and Snowflake data pipelines, identifying and resolving performance bottlenecks and data discrepancies.- Collaborate with data engineers, data scientists, and software developers to understand data requirements and deliver integrated solutions.- Ensure data quality, integrity, and security across the streaming and warehousing solutions.- Document technical designs, configurations, and operational procedures.- Participate in code reviews and promote best practices in Kafka and Snowflake development.This role offers the opportunity to work on cutting-edge technologies and be part of a collaborative team that focuses on delivering high-quality solutions in the domain of real-time data streaming and data warehousing.

Keyskills :
Apache KafkaSnowflakeData WarehousingData StreamingSSLRBACKafka ConnectKafka StreamsksqlDB