Senior Distributed Systems Engineer

About the Institute of Foundation Models The Institute of Foundation Models (IFM) designs and operates ultra-scale GPU supercomputing systems to train next-generation foundation models. We believe performance, fault tolerance, and scalability are co-designed across model architecture, communication systems, runtime, and hardware topology. This role sits at the core of that effort — driving communication performance, distributed reliability, and cross-layer optimization for large-scale training workloads. The Mission We are looking for a deeply technical engineer to co-design and optimize the communication stack for large-scale distributed training, including hybrid parallelism and Mixture-of-Experts (MoE) workloads. This is not a network operations role. This is a systems-level engineering position focused on performance engineering, distributed debugging, and communication-runtime co-design. ·       Design and optimize expert-parallel and hybrid-parallel communication patterns ·       Drive high-performance hierarchical collectives for MoE workloads ·       Co-design runtime orchestration with communication topology awareness ·       Reduce tail latency and improve determinism across thousands of GPUs ·       Architect fault-tolerant distributed execution under real-world cluster failures Core Technical Scope ·       Communication-compute overlap and topology-aware collective optimization ·       Deep debugging of NCCL, RDMA, and custom communication layers ·       Hybrid expert parallel strategies in modern large-scale MoE systems ·       Elastic and resilient distributed job orchestration concepts ·       Congestion analysis and routing optimization across InfiniBand/RoCE fabrics ·       Microbenchmarking and performance modeling for communication-heavy workloads Expected Technical Depth ·       Hybrid expert parallel communication for Mixture-of-Experts training ·       Scaling behavior under network pressure ·       Distributed orchestration for elastic, large-scale training ·       Fault detection and recovery in distributed GPU workloads ·       Cross-layer bottlenecks: GPU ↔ NIC ↔ PCIe ↔ NVSwitch ↔ Fabric ↔ Scheduler Required Background ·       Experience optimizing distributed training at 1,000+ GPU scale (or equivalent depth) ·       Hands-on expertise with RDMA, InfiniBand, RoCE, and GPUDirect RDMA ·       Deep familiarity with NCCL and/or UCX internals ·       Strong systems programming ability (C/C++, Rust, or Go) ·       Strong familiarity with modern model training frameworks such as PyTorch ·       Ability to troubleshoot and profile training performance issues related to communication bottlenecks ·       Ability to translate research ideas into production-grade optimizations ·       Experience debugging distributed hangs, desynchronization, and performance regressions What We Mean by "Hardcore" ·       You can explain why an communication degrades at scale and how to fix it ·       You have improved real cluster throughput via communication redesign ·       You can trace a distributed hang across ranks and identify the root cause ·       You are comfortable working at the boundary between hardware and runtime Application Requirements ·       Include a link to your GitHub (required) ·       Provide links to relevant distributed systems, HPC, or large-scale training projects ·       Include a list of publications and/or public technical reports (if applicable) ·       Describe the hardest distributed debugging problem you solved ·       Include measurable performance improvements you have delivered Academic Qualifications Master’s, or Bachelor’s + 1 year of relevant experience. Visa Sponsorship This position is eligible for visa sponsorship. Benefits Include *Comprehensive medical, dental, and vision benefits  *Bonus *401K Plan *Generous paid time off, sick leave and holidays *Paid Parental Leave *Employee Assistance Program *Life insurance and disability

{
  "content_hash": "254fa5160b2a63eec7c5438cbc459d6448b6ad65d43cc9ed90ae15c1ccd11dd9",
  "source_hash": "6a884e6406a9d316395900d7cdb2a3e81ab4719b41cbb0ae7514b715e091dd2c",
  "last_changed_at": "2026-06-02T10:41:24.749Z",
  "active_status": "active"
}

{
  "dedupe": null,
  "language": "en",
  "location": {
    "raw": "Sunnyvale, CA",
    "city": "Sunnyvale",
    "region": "CA",
    "country": "United States",
    "is_remote": false,
    "confidence": 0.9
  },
  "salary_max": 400000,
  "salary_min": 200000,
  "inferred_at": "2026-06-23T07:55:59.802Z",
  "launch_scope": {
    "reason": "english_us_canada",
    "included": true,
    "language": "en",
    "location": {
      "raw": "Sunnyvale, CA",
      "city": "Sunnyvale",
      "region": "CA",
      "country": "United States",
      "is_remote": false,
      "confidence": 0.9
    },
    "countries": [
      "United States"
    ]
  },
  "remote_policy": null,
  "salary_period": "year",
  "workplace_type": "on_site",
  "salary_currency": "USD"
}

{}

{
  "lists": [],
  "country": "US",
  "createdAt": 1772500202035,
  "updatedAt": null,
  "categories": {
    "team": "Engineering",
    "location": "Sunnyvale, CA",
    "allLocations": [
      "Sunnyvale, CA"
    ]
  },
  "salaryRange": {
    "max": 400000,
    "min": 200000,
    "currency": "USD",
    "interval": "per-year-salary"
  },
  "workplaceType": "onsite"
}