LM Studio logo

Software Engineer, Inference Runtime

LM Studio

New York, NY · HybridJobPosted 1mo agoStill listed 3w ago

Evaluate my fit

Most applications go out cold — see where you stand first. No sign-up to start.

Watch jobs like this.

Don't just apply. Show up ready.

Olive works from this exact posting.

At a glance

Compensation
No compensation found
Location
New York, NYHybrid
Work Authorization
Not specified

Olive lists jobs from US employers, including remote roles you can work from the United States.

Job overview

LM Studio is hiring a Software Engineer, Inference Runtime. LM Studio seeks a curious, self‑motivated Software Engineer to advance its inference runtime on‑device and in the cloud, integrating new engines, optimizing model execution across CPU and GPU targets, and contributing to open‑source projects, while upholding high technical intensity and personal responsibility.

Key focus areas include Maintain and push forward inference stack on-device and in the cloud, Bring up new model architectures and multimodal models, and Improve latency, throughput, memory use, and reliability across runtimes.

Important skills include Machine Learning, Debugging, Memory Management, Python, C++, and Transformer Architectures.

Skills & qualifications

RequiredNice to have

Skills

Machine LearningDebuggingMemory ManagementPythonC++Transformer ArchitecturesCPUGPUProfiling CPU or GPU WorkloadsPyTorchLlama.cppMLXExecuTorchvLLMSGLangTensorRT-LLMCUDAMetalVulkanROCmModel LoadingBatchingSchedulingCachingDistributed ExecutionBenchmarkingDiagnosing Performance Problems

Qualifications

Significant Experience Building Production ML Systems, Inference Runtimes, or Performance-Sensitive InfrastructureStrong Programming Ability in Python and C++Deep Understanding of Transformer Architectures and Model Inference MechanicsExperience Profiling CPU or GPU Workloads and Reasoning About Compute Memory Synchronization and Data MovementExperience With PyTorch and Inference Systems Such as Llama.cpp, MLX, ExecuTorch, VLLM, SGLang, or TensorRT-LLMStrong Debugging Instincts Across Model Code, Runtime Internals, Operating Systems, and CPU or GPU ExecutionTakes Personal Responsibility for Correctness and Performance of WorkPast Contributions to Open-Source Inference Runtime Projects Such as Llama.cpp, MLX, ExecuTorch, VLLM, SGLang, or TensorRT-LLM

Benefits

Medical Insurance
Vision Insurance
Dental Insurance
Paid Time Off

Full job description

LM Studio is used by millions of people around the world to run AI on their own computers, and now with Bionic - also in the cloud. Our values prioritize putting the human in the center, and creating tools that we want to use ourselves, and recommend to our friends and family.

As a team, we work with high technical intensity and personal responsibility. We are looking for curious, self-motivated, creative, and technically excellent teammates to join us and build the future of human-AI interactions in software.

The Role

We are looking for an Inference Runtime Software Engineer to push forward LM Studio's inference stack on-device and in the cloud. You will integrate new inference engines and runtime capabilities, bring up new open-weight models and modalities, and optimize model execution for a wide range of CPU and GPU targets. You will also contribute improvements to the open-source projects we build on.

Qualifications

  • Significant experience building production ML systems, inference runtimes, or performance-sensitive infrastructure

  • Strong programming ability in Python and C++

  • Deep understanding of transformer architectures and the mechanics of model inference

  • Experience profiling CPU or GPU workloads and reasoning about compute, memory, synchronization, and data movement

  • Experience with PyTorch and inference systems such as llama.cpp, MLX, ExecuTorch, vLLM, SGLang, or TensorRT-LLM

  • Strong debugging instincts across model code, runtime internals, operating systems, and CPU or GPU execution

  • Takes personal responsibility for the correctness and performance of their work

Bonus Qualifications

  • Past contributions to open-source inference runtime projects such as llama.cpp, MLX, ExecuTorch, vLLM, SGLang, or TensorRT-LLM

Responsibilities

  • Maintain and push forward our inference stack on-device and in the cloud

  • Bring up new model architectures and multimodal models

  • Improve latency, throughput, memory use, and reliability across CPU, CUDA, Metal, Vulkan, and ROCm runtimes

  • Build runtime capabilities for model loading, batching, scheduling, caching, and distributed execution

  • Benchmark and diagnose correctness and performance problems across the inference stack

  • Contribute upstream to open-source projects such as llama.cpp and MLX

Benefits

  • Competitive salary and equity grants

  • Great medical, vision, dental healthcare plans

  • Catered team lunch / expensed dinners in the office

  • Flexible PTO

  • Flexible WFH

  • Sun-drenched office in SoHo in NYC

Similar jobs, posted recently

Open roles like this one, listed in the last 30 days.

You've read the whole posting — now see how you match it.

Evaluate my fit