Deep Learning Systems · LLM Inference
Deep Learning Algorithm Engineer at NVIDIA, working on SGLang & TensorRT-LLM — pushing LLM inference fast, stable, and correct at scale on GB200 / GB300.
Previously CMU INI & Tongji CS. I care about the intersection of engineering, optimization, and research with real-world impact.
01 — About
I’m a Deep Learning Algorithm Engineer at NVIDIA on the TensorRT-LLM / Inference team, where I focus on making large language models run faster, cheaper, and correctly at scale — most recently on Blackwell-generation hardware (GB200 / GB300 NVL72). A lot of my day-to-day lives inside SGLang: Qwen3.5 / Qwen3-Next bring-up, DeepSeek-V4 performance, heterogeneous-TP KV transfer, DP-Attention, and long-context disaggregated serving.
On the side, I collaborate with LMSYS and UC Berkeley’s Sky Computing Lab on multi-LLM serving (Prism, kvcached), and with CMU’s AI Common Sense Lab under Deepak Pathak on physics-grounded reasoning (Sim2Reason) — turning physics simulators into infinite training environments for LLMs.
Before NVIDIA full-time, I was at CMU INI (M.S., 2023–2024), and Tongji University for my B.E. in CS. I’ve been lucky to publish first-author papers at CIKM and IEEE TAFFC along the way.
02 — Now
Updated 2026
Helping bring up Qwen3.5 / Qwen3-Next in SGLang from the NVIDIA side — working on heterogeneous-TP KV transfer for GQA, Mamba state slice transfer in disaggregation, a GPU staging-buffer with dynamic ring allocator, and fused QKV-Z-BA reshape on prefill.
Working on DeepSeek-V4 / V4-Pro performance on GB200 / GB300 NVL72 alongside the team — disaggregated 1P1D recipes, mega-MoE serving knobs, sa-bench tokenizer support, and PP / TP / EP / DP sweeps to map throughput × latency × memory Pareto frontiers.
Chipping away at SGLang inference critical paths: DP-Attention with reduce_scatterv in place of all-reduce + dp_scatter, an adaptive queue-based prefill delayer, TBO MLP-sync fixes, and NIXL transport for SWA / NSA / Mamba state buffers.
Continuing to contribute to Prism & kvcached — tighter SGLang / vLLM integration, cluster-level deployment, and Kubernetes-friendly multi-model serving.
03 — Selected Work
Contributing from the NVIDIA side: heterogeneous-TP KV transfer for GQA, Mamba state slice transfer in disaggregation, a GPU staging-buffer with dynamic ring allocator, fused QKV-Z-BA prefill, GDN on SM100+. End-to-end Qwen3.5 disagg serving.
Working with the team on disaggregated 1P1D recipes for GB300, mega-MoE serving knobs, sa-bench tokenizer support, and PP / TP / EP / DP sweeps mapping throughput–latency–memory Pareto frontiers for production serving.
Two-tier SLO-aware scheduler (global KV-pressure placement + GPU-local priority admission). 50+ LLMs concurrently on shared infra, 2× cost reduction & 3.3× SLO attainment.
OS-style virtual memory for LLM KV caches. CUDA virtual-memory APIs decouple physical from virtual GPU memory, enabling elastic allocation and cross-model cache sharing. Integrated with SGLang & vLLM.
Procedurally generate MuJoCo scenes & QA pairs through a DSL, then RL-finetune LLMs on synthetic supervision. +10–15pp on IPhO Mechanics across model scales (Qwen2.5-32B: 19.8%→25.2%). With Mihir Prabhudesai, Aryan Satpathy, Deepak Pathak, Katerina Fragkiadaki.
Semi-supervised model with modality-correlation contrastive learning for audio / image / text fusion. Strong results on CMU-MOSEI. First-author, IEEE Transactions on Affective Computing 2025.
04 — Publications
IEEE TAFFC · 2025
Yangmin Li, et al. · IEEE Transactions on Affective Computing, 2025 · first author
CIKM · 2022
Yangmin Li, et al. · ACM CIKM 2022 · first author · ACM DL
Preprint · 2025
With Sky Computing Lab & LMSYS · arXiv 2505.04021
Project · 2024–
With CMU AI Common Sense Lab · project page
05 — Experience
Jan 2025 — Present
TensorRT-LLM · Deep Learning Inference · Santa Clara, CA
SGLang DeepSeek-V3.2 NSA, Spec-V2/EAGLE, GB300 long-context, multi-node Pareto sweeps.
May 2024 — Aug 2024
Autonomous Vehicle ML & Driveworks Platform · Santa Clara, CA
RoadCast 2.1 — distributed data logging with dwProto serialization, batched I/O, NVSci/TCP/DFS.
Feb 2023 — May 2023
Cloud Architecture & MLOps · Beijing
TB-scale data lake (Kinesis → S3 → Glue/Athena), SageMaker MLOps with CodePipeline + EKS/ECS.
Sep 2022 — Dec 2022
Distributed Video Quality DL Evaluation · Shanghai
Multimodal DL/ML video QoE evaluation (D-VQA, multimodal Transformer), Spark on HDFS.
2023 — 2024
M.S. Information Technology · Information Networking Institute · GPA 3.84/4.00
2019 — 2023
B.E. Computer Science & Technology · GPA 90.8/100 · First-class Scholarship
06 — Writing
A few longer reflections on projects I’ve been part of — currently lives on LinkedIn.
SGLang · GB300 · 2025
A few rewarding months making SGLang fast, stable, and correct at scale on GB300 with the LMSYS & NVIDIA folks.
linkedin →
Sim2Reason · CMU · 2026
Why simulators may be where language models finally learn how the world works.
linkedin →
Prism · LMSYS × Berkeley · 2025
From AWS & ByteDance to NVIDIA & SGLang — and how Prism became my favorite collaboration so far.
linkedin →