[Remote] Principal Back-End Network Engineer - AI Infrastructure Operations
Note: The job is a remote job and is open to candidates in USA. Nscale is a GPU cloud provider engineered for AI, offering high-performance infrastructure for AI start-ups and enterprises. They are seeking a Principal Network Engineer to lead the technical direction and operational strategy for their AI interconnect networks, focusing on Infiniband and RDMA-based network fabrics.
Responsibilities
- Owning the technical direction and operational strategy for Nscale’s AI interconnect networks
- Designing, reviewing, and evolving large-scale Infiniband and RoCE fabric architectures to support future growth and workload demands
- Acting as the senior escalation point for the most complex network incidents, guiding deep technical investigations and systemic fixes
- Driving cross-team initiatives to improve fabric reliability, performance predictability, and operational maturity
- Defining standards for hardware configuration, congestion control, routing, firmware lifecycle management, and change safety
- Partnering with SRE, Compute Platform, and Network Architecture teams to influence end-to-end system design
- Mentoring senior and mid-level network engineers, raising the bar for operational rigor and technical excellence
- Driving measurable improvements in uptime, latency consistency, capacity efficiency, and incident reduction
Skills
- 10+ years of experience in network engineering, with deep focus on HPC, AI, or hyperscale data centre networking
- Expert-level operational and architectural experience with Infiniband and/or large-scale RoCE fabrics
- Deep understanding of RDMA internals, congestion management, and fabric-level failure modes
- Strong expertise in modern data centre routing and control planes (BGP, OSPF, ECMP)
- Proven ability to debug and resolve cross-layer issues spanning hardware, firmware, kernel, and application communication libraries
- Demonstrated ability to lead complex technical initiatives across teams without direct authority
- A systems-level mindset, balancing performance, reliability, scalability, and operational cost
- Extensive experience with NVIDIA/Mellanox networking platforms in production AI or HPC environments
- Deep familiarity with distributed training frameworks and GPU communication patterns
- Experience designing network observability systems for high-cardinality, high-throughput environments
- Prior experience influencing platform or infrastructure strategy at scale
Benefits
- Highly competitive package (base + equity) with reviews every 12 months.
- Expect a dynamic progression plan tailored to your ambitions.
- Human-First Flexibility: We treat you as humans first. 🫶🏽 Our flexible workplace trusts Nscalers to deliver, giving you the autonomy to shape your day around life's moments.
- Join our thriving remote-first team.
- In addition to base salary, this role may be eligible for bonus, equity, and/or commission programs.
- Nscale may offer a competitive benefits package including medical, dental, vision, flexible paid time off, parental leave, and retirement plan participation.
Company Overview