ML Inference / GPU Engineer

About the Role

We're looking for a Senior ML Inference Engineer to join our partner companies where the GPU bill is one of the biggest lines in the budget. You'll take trained PyTorch models and make them serve faster and cheaper: quantize them, batch requests well, tune the serving stack, and write a custom kernel when the profiler says it's worth it. Latency and throughput are your numbers. You'll measure both, move them, and explain the trade-off when improving one costs the other.

What You'll Do

• Optimize model serving for latency and throughput with vLLM, TensorRT-LLM, and Triton Inference Server
• Apply quantization (INT8, FP8, AWQ, GPTQ) and measure the accuracy cost of each change
• Profile GPU workloads with Nsight and the PyTorch profiler and fix the bottlenecks
• Write or tune CUDA and Triton kernels for hot operations
• Tune batching, KV-cache, and parallelism settings across GPU fleets
• Track cost per token or per request and cut it without breaking SLAs

Requirements

• 5+ years of software engineering, 2+ focused on ML inference or GPU performance
• Strong Python and PyTorch, plus working C++
• Hands-on CUDA experience: memory hierarchy, kernels, and streams
• Production experience with at least one serving stack (vLLM, TensorRT-LLM, Triton, TGI, SGLang)
• Experience with quantization and model compression
• Excellent written and verbal English (B2+)

Nice to Have

• Experience with multi-GPU and multi-node serving (tensor and pipeline parallelism)
• Knowledge of speculative decoding, continuous batching, or paged attention internals
• Experience with Kubernetes GPU scheduling and autoscaling
• Contributions to vLLM, TensorRT-LLM, or similar open-source projects

Tech Stack

Benefits & Perks