AI-Factory Observability Principal

AI-Factory Observability Principal We are operating large-scale AI training and inference data centers, and we need an expert who can see the entire stack at once — from the chiller plant and switchgear to the GPU fabric and the Kubernetes scheduler. This role spans facilities/OT telemetry (cooling, power) and IT/AI infrastructure observability (compute, network, accelerators), unified by a single goal: complete, real-time, predictive visibility into how AI infrastructure consumes power, generates heat, moves data, and delivers compute. You will design the observability platform that ingests signals from building and electrical systems, server and network fabrics, Kubernetes, and GPU/accelerator clusters — then apply AI/ML models on top of that telemetry to optimize utilization, predict failures, reduce energy cost, and surface insights operators can act on. You are equally comfortable reading a BACnet point list and a GPU NVLink topology, and you can explain to both facilities and platform teams why their data belongs in the same system. Observability architecture & strategy Define and own the end-to-end observability architecture covering metrics, logs, traces, and events across facilities and IT domains. Establish standards for instrumentation, telemetry pipelines, data retention, cardinality management, and a unified data model that lets power, thermal, network, and compute signals be correlated in one place. Design for scale: hundreds of thousands of time series per site, high-frequency power and thermal sampling, and GPU-cluster-level granularity. Facilities & OT integration (BMS / EPMS) Integrate Building Management System (BMS) telemetry — CRAC/CRAH units, chillers, cooling loops, airflow, temperature/humidity, leak detection — into the central observability platform (BACnet, Modbus, MQTT, OPC-UA). Integrate Electrical Power Monitoring System (EPMS) data — switchgear, UPS, PDUs, busways, branch-circuit metering, generators — for real-time power draw, capacity, and quality monitoring (Modbus, DNP3, IEC 61850). Build correlated views of power and thermal behavior against compute workload so operators understand cause and effect (e.g., a training job's effect on rack power and inlet temperatures). Partner with facilities engineering on PUE, capacity planning, stranded-power recovery, and thermal optimization. AI cluster & Kubernetes observability Architect observability for AI/GPU clusters — accelerator utilization, memory pressure, thermals, ECC/Xid errors, power capping, and job-level efficiency (e.g., via NVIDIA DCGM, accelerator telemetry exporters). Instrument Kubernetes environments running AI/ML workloads: cluster, node, pod, and workload metrics, scheduler behavior, GPU/accelerator allocation, and operator health. Provide visibility into training and inference pipelines — throughput, queue depth, checkpoint behavior, straggler detection, and cost-per-token / cost-per-training-step metrics. Surface noisy-neighbor, fragmentation, and underutilization patterns across multi-tenant clusters. Network observability Design monitoring for high-performance data center fabrics, including the AI back-end network (RDMA, InfiniBand and/or RoCE Ethernet) and front-end/management networks. Capture fabric health, congestion, link errors, latency, and bandwidth utilization using streaming telemetry, SNMP, gNMI/gRPC, NetFlow/sFlow, and fabric managers (e.g., InfiniBand UFM). Correlate network behavior with distributed training performance to diagnose collective-communication bottlenecks. AI/ML-driven optimization & insight (AIOps) Apply ML and AI models to the telemetry estate for anomaly detection, predictive maintenance, capacity forecasting, and automated root-cause analysis. Build models and pipelines that recommend (or automate) actions: dynamic cooling and power optimization, workload placement, power capping under thermal/electrical constraints, and failure pre-emption. Leverage LLMs and modern AI techniques to summarize incidents, accelerate root-cause investigation, query telemetry in natural language, and generate operator-facing insights from large volumes of logs and metrics. Establish the feedback loop where observability data trains the models that, in turn, optimize the infrastructure being observed. Cross-functional leadership Act as the technical authority connecting facilities, network, platform, SRE, and AI/ML teams around a shared observability practice. Define SLOs, alerting strategy, and on-call signal quality; drive down alert noise and mean-time-to-resolution. Mentor engineers and set the technical direction for the observability roadmap. Required Qualifications 8+ years in infrastructure, SRE, observability, or data center engineering, with 3+ years in an architect or principal-level role. Demonstrated experience designing and operating observability platforms at scale (metrics, logs, traces). Expertise in Datadog, Dynatrace, Grafana, Prometheus and Grafana. Hands-on experience integrating BMS and EPMS data, and a working understanding of data center mechanical and electrical systems (cooling topologies, power distribution, redundancy, capacity). Strong systems monitoring background — Linux/server fleets, hardware health, baseboard management (IPMI/Redfish). Strong network monitoring background, including high-performance / low-latency fabrics relevant to AI workloads. Expertise in SNMP, WMI. Production experience with Kubernetes and observability of containerized workloads. Experience operating or monitoring GPU / AI-accelerator clusters and understanding of distributed training/inference behavior. Practical experience applying AI/ML models to operational data (anomaly detection, forecasting, or AIOps), and comfort using LLMs to derive insights and automate analysis. Proficiency in at least one language for data/automation work (Python preferred), and infrastructure-as-code practices.

Preferred Qualifications

Experience with tooling such as OpenTelemetry, VictoriaMetrics/Thanos, Loki, Tempo, Elastic, Splunk. Familiarity with OT/industrial protocols: BACnet, Modbus, OPC-UA, DNP3, IEC 61850, MQTT. Familiarity with GPU/accelerator telemetry (NVIDIA DCGM and exporters) and InfiniBand/RDMA monitoring (e.g., UFM). Experience with network telemetry: gNMI/OpenConfig streaming, SNMP, NetFlow/sFlow. Experience with time-series data at high cardinality, stream processing, and data lake/warehouse patterns for telemetry. Background in MLOps, model deployment, or building data/feature pipelines for operational ML. Exposure to power and cooling optimization, PUE improvement, or sustainability/energy-efficiency initiatives. Relevant certifications (e.g., data center facilities, Kubernetes/CKA, cloud or networking) are a plus.