Enterprise Databricks Platform Production Support

Job Description

As a Production Support Engineer for the Databricks platform with over 3 years of experience, your role will involve implementing and maintaining the Databricks platform on AWS. You will be responsible for developing and maintaining automation scripts and tools for platform provisioning and routine tasks using Terraform. Your tasks will include installing, configuring, and maintaining Databricks clusters and workspaces. Additionally, you will be expected to implement and maintain the Databricks platform, including workspace setup, user and group management, access control, and security configurations.Key Responsibilities:- Provide technical support to data engineering, data science, and application teams.- Perform restores/recoveries, troubleshoot service issues, and resolve platform-related problems.- Monitor and manage cluster performance, resource utilization, and platform costs.- Implement and manage access controls and security policies to protect sensitive data.- Ensure compliance with relevant data governance and regulatory requirements.- Manage and maintain connections between Databricks and other data sources like Snowflake.- Optimize data pipelines and workflows.- Manage streaming and batch pipelines - utilizing SQL and PySpark skills to debug and support.Qualifications Required:- Minimum of 3 years of experience in production support of the Databricks platform.- Proficiency in implementing and maintaining the Databricks platform on AWS.- Strong experience in developing automation scripts and tools for platform provisioning using Terraform.- Ability to troubleshoot and resolve platform-related issues effectively.- Knowledge of managing access controls, security policies, and compliance requirements.- Experience in optimizing data pipelines and workflows.- Proficiency in SQL and PySpark for managing streaming and batch pipelines.This job also requires you to have a deep understanding of data engineering concepts and experience in working with data science and application teams. By excelling in this role, you will play a crucial part in ensuring the smooth operation and optimization of the Databricks platform within the organization. As a Production Support Engineer for the Databricks platform with over 3 years of experience, your role will involve implementing and maintaining the Databricks platform on AWS. You will be responsible for developing and maintaining automation scripts and tools for platform provisioning and routine tasks using Terraform. Your tasks will include installing, configuring, and maintaining Databricks clusters and workspaces. Additionally, you will be expected to implement and maintain the Databricks platform, including workspace setup, user and group management, access control, and security configurations.Key Responsibilities:- Provide technical support to data engineering, data science, and application teams.- Perform restores/recoveries, troubleshoot service issues, and resolve platform-related problems.- Monitor and manage cluster performance, resource utilization, and platform costs.- Implement and manage access controls and security policies to protect sensitive data.- Ensure compliance with relevant data governance and regulatory requirements.- Manage and maintain connections between Databricks and other data sources like Snowflake.- Optimize data pipelines and workflows.- Manage streaming and batch pipelines - utilizing SQL and PySpark skills to debug and support.Qualifications Required:- Minimum of 3 years of experience in production support of the Databricks platform.- Proficiency in implementing and maintaining the Databricks platform on AWS.- Strong experience in developing automation scripts and tools for platform provisioning using Terraform.- Ability to troubleshoot and resolve platform-related issues effectively.- Knowledge of managing access controls, security policies, and compliance requirements.- Experience in optimizing data pipelines and workflows.- Proficiency in SQL and PySpark for managing streaming and batch pipelines.This job also requires you to have a deep understanding of data engineering concepts and experience in working with data science and application teams. By excelling in this role, you will play a crucial part in ensuring the smooth operation and optimization of the Databricks platform within the organization.

Keyskills :
AWSAccess controlData engineeringData scienceTroubleshootingResource utilizationData governanceRegulatory requirementsSnowflakeSQLDatabricks platformTerraformInstallconfigure Databricks clustersWorkspace setupUsergroup managementSecurity