Senior Observability Engineer

Job Description

Senior Observability Engineer (GCP & OpenTelemetry)We are looking for an Observability specialist to lead the design and implementation of our telemetry pipeline usingOpenTelemetry (OTel) Collectorsto monitor ourGCP infrastructure. You wont just turn onmonitoring; you will curate a high-signal environment by identifying value-addmetrics and implementing sophisticated label enrichment strategies to ensure our data is actionable, cost-effective, and context-rich.KEY RESPONSIBILITIES
• OTel Collector Architecture:Design, deploy, and maintain OTel Collectors (Sidecars, DaemonSets, and Gateway clusters) across GKE and GCE environments.
• Pipeline Optimization:Configurereceivers(Google Cloud Monitoring, OTLP, Host Metrics),processors(Batch, Memory Limiter, Resource Detection), andexporters(GoogleCloud, Prometheus).
• GCP Metric Curation:Distinguish between noiseand signalby identifying and collecting high-value GCP metrics (e.g.,compute.googleapis.com/instance/cpu/scheduler_wait_timevs. simple utilization).
• Metadata & Label Enrichment:Use theResource Detection ProcessorandTransform Processorto automatically inject GCP-specific metadata (Project ID, Zone, Instance ID, Custom Labels) into all telemetry signals.
• Cost Management:Implement filtering and dropped-label strategies to manage cardinality explosionsand optimize Google Cloud Observability (Stackdriver) costs.
TECHNICAL SKILLS REQUIRED
• GCP Expertise:Deep understanding of GCP resource hierarchies and theCloud Monitoring API (v3).
• OpenTelemetry Proficiency:Advanced configuration of theotel-collector-contribdistribution, specifically for infrastructure monitoring.
• Metric Strategy:Knowledge of which Golden Signals(Latency, Errors, Saturation, Traffic) are most relevant for specific GCP services likeCloud SQL,GKE, andPub/Sub.
• Contextual Enrichment:Experience using OTel to bridge the gap between infra metrics and application context (e.g., mapping a GCE instance ID to a specific Business Unit via labels).
• Infrastructure as Code:Proficiency in Terraform or Helm for deploying observability as a standard part of the landing zone.
THE IDEAL CANDIDATEWere mostly looking for aData Curatorrather than just a Systems Admin.Most people can install a collector; very few know how to make the data coming out of it actually useful and cost-efficient.We need anObservability Engineerwho acts like a filter. They should know which specific GCP metrics actually matter (so we dont drown in noise) and how to tag (enrich) those metrics using OpenTelemetry so that when an alert goes off, we know exactly which team, project, and environment it belongs to.3 GREEN FLAGSTO LOOK FORKeep an eye out for these on a resume or listen for them in a screening:
1. The Optimizer :They talk aboutCost ManagementorCardinality.
   1. Why:Storing every single metric in GCP is expensive. A good candidate mentions filteringor droppinguseless metrics to save money.
2. The Context King :They mentionResource DetectionorAttribute Mapping.
   1. Why:This is the label enrichmentpart. It means they know how to automatically attach metadata (likeOwner:Payments-Team) to a raw metric.
1. The Contribution Pro :They mention using the Contribversionof the OTel Collector.
   1. Why:The standard OTel collector is basic; the Contribversion contains the specific Google Cloud processors needed for high-level monitoring.
TOP 5 TECH STACK NEEDS
• OpenTelemetry (OTel) Collector Contrib:This is the primary engine where the magichappens; it contains the specific processors required to automatically enrich metrics with GCP metadata and transform raw data into businessready signals.
• Google Cloud Observability (Stackdriver):The candidate must deeply understand the destinations proprietary data model and billing logic to ensure that the metrics they collect are formatted correctly and dont cause a cardinality explosionthat spikes your monthly bill.
• Kubernetes (GKE) & Helm:Since the collector typically runs as a DaemonSet or Sidecar in GCP, mastery here ensures the monitoring pipeline is resilient, scales with your clusters, and is easily updated via standardized charts.
• Terraform / Terragrunt:High-quality observability must be codifiedrather than manual; using IaC ensures that every new GCP project or resource is automatically onboarded with the correct labels and alert policies from day one.
• PromQL & MQL (Query Languages):Collecting the data is only half the battle; the candidate needs these languages to build the complex dashboards and SLO (Service Level Objective) ratios that actually prove the value-addto your engineering teams.

Keyskills :
cloudgcpsql