Paper

{
  "raw_xml": "<entry>\n    <id>http://arxiv.org/abs/2603.17456v1</id>\n    <title>Multi-stage Flow Scheduling for LLM Serving</title>\n    <updated>2026-03-18T07:53:28Z</updated>\n    <link href='https://arxiv.org/abs/2603.17456v1' rel='alternate' type='text/html'/>\n    <link href='https://arxiv.org/pdf/2603.17456v1' rel='related' title='pdf' type='application/pdf'/>\n    <summary>Meeting stringent Time-To-First-Token (TTFT) requirements is crucial for LLM applications. To improve efficiency, modern LLM serving systems adopt disaggregated architectures with diverse parallelisms, introducing complex multi-stage workflows involving reusable KV-block retrieval, collective communication, and P2D transfer. Flows from dependent stages overlap within and across requests on shared bottleneck links, making TTFT highly susceptible to network contention and necessitating stage-aware scheduling. Unfortunately, most existing works schedule flows in a stage-agnostic manner, leading to uncoordinated contention that constitutes a primary cause of SLO violations.\n  In this paper, we present MFS, a holistic multi-stage flow scheduling mechanism designed to maximize TTFT SLO attainment. At its core, MFS approximates the Least-Laxity-First (LLF) scheduling policy without requiring precise knowledge of a request's remaining slack. It achieves this through a Defer-and-Promote principle implemented through a Reverse Multi-Level Queue (RMLQ) structure. By dynamically promoting task precedence as effective laxity diminishes, MFS prioritizes flows with less laxity while preventing requests with loose SLOs from prematurely consuming network bandwidth. We implement MFS as a pluggable module integrated into vLLM, and evaluate it on a 8-server, 32-GPU testbed as well as through large-scale simulations. Our results demonstrate that MFS effectively outperforms state-of-the-art baselines, improving the TTFT SLO attainment by 1.2x--2.4x.</summary>\n    <category scheme='http://arxiv.org/schemas/atom' term='cs.NI'/>\n    <category scheme='http://arxiv.org/schemas/atom' term='cs.DC'/>\n    <published>2026-03-18T07:53:28Z</published>\n    <arxiv:comment>18 pages, 14 figures</arxiv:comment>\n    <arxiv:primary_category term='cs.NI'/>\n    <author>\n      <name>Yijun Sun</name>\n      <arxiv:affiliation>Hong Kong University of Science and Technology</arxiv:affiliation>\n    </author>\n    <author>\n      <name>Xudong Liao</name>\n      <arxiv:affiliation>Hong Kong University of Science and Technology</arxiv:affiliation>\n    </author>\n    <author>\n      <name>Songrun Xie</name>\n      <arxiv:affiliation>Hong Kong University of Science and Technology</arxiv:affiliation>\n    </author>\n    <author>\n      <name>Hao Chen</name>\n      <arxiv:affiliation>Shanghai Jiao Tong University</arxiv:affiliation>\n    </author>\n    <author>\n      <name>Han Tian</name>\n      <arxiv:affiliation>University of Science and Technology of China</arxiv:affiliation>\n    </author>\n    <author>\n      <name>Wenxue Li</name>\n      <arxiv:affiliation>Hong Kong University of Science and Technology</arxiv:affiliation>\n    </author>\n    <author>\n      <name>Yiming Zhang</name>\n      <arxiv:affiliation>Shanghai Jiao Tong University</arxiv:affiliation>\n    </author>\n    <author>\n      <name>Kai Chen</name>\n      <arxiv:affiliation>Hong Kong University of Science and Technology</arxiv:affiliation>\n    </author>\n  </entry>"
}