Skip to content

[BugFix] Fix Memory Leak #17567

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
May 2, 2025
Merged

[BugFix] Fix Memory Leak #17567

merged 4 commits into from
May 2, 2025

Conversation

robertgshaw2-redhat
Copy link
Collaborator

@robertgshaw2-redhat robertgshaw2-redhat commented May 2, 2025 •
edited by github-actions bot
Loading

SUMMARY:

@robertgshaw2-redhat
updated
1e4e107 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
@robertgshaw2-redhat
updated
1d9f88e 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented May 2, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@robertgshaw2-redhat
updated
c00f3e7 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
@mergify mergify bot added the v1 label May 2, 2025
comaniac
comaniac reviewed May 2, 2025
View reviewed changes
Copy link
Collaborator

@comaniac comaniac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for locating the memory leak and the fix!

vllm/v1/core/sched/scheduler.py Outdated
Comment on lines 767 to 770
# NOTE(rob): since we free stopped reqs above, adding stopped reqs
# to _cached_reqs_data will cause a memory leak.
if req_data.req_id not in stopped_set:
self._cached_reqs_data[req_data.req_id].append(req_data)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just do the following so that we don't need to introduce stopped_set?

if req_data.req_id in self._cached_reqs_data:
    self._cached_reqs_data[req_data.req_id].append(req_data)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that would work unless line 541 is changed from self._cached_reqs_data.get(request.request_id) to self._cached_reqs_data[request.request_id].

Alternatively I think you could use if req_data.req_id not in self.finished_req_ids to avoid having a new set.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea

@robertgshaw2-redhat
address nicks comment
bc53d27 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label May 2, 2025
@vllm-bot vllm-bot merged commit c777df7 into vllm-project:main May 2, 2025
66 of 68 checks passed
radeksm pushed a commit to radeksm/vllm that referenced this pull request May 2, 2025
@robertgshaw2-redhat @radeksm
[BugFix] Fix Memory Leak (vllm-project#17567)
1434b45 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
simon-mo pushed a commit that referenced this pull request May 2, 2025
@robertgshaw2-redhat @simon-mo
[BugFix] Fix Memory Leak (#17567)
edb5286 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
@nightflight-dk
Copy link

any plans to fix in V0? batching with phi mini (3.5 and 4), lora_enabled - V0 showed degraded performance yesterday, responses showing signs of context from other prompts

@nFunctor
Copy link
Contributor

nFunctor commented May 5, 2025

@nightflight-dk sorry for a sidetrack, but I wanted to clarify since I don't understand the issue well here, do you mean that the chunked prefill, if enabled in V0, potentially leads to prompt mixing? There was a phenomenon like that for prefix caching due to hash function's behaviour in python 3.12 #12621 , but it got fixed for me in 0.7.2.

@xiamuyingu
Copy link

xiamuyingu commented May 7, 2025 •
edited
Loading

Is it means that if I upgrade vllm to v0.8.5.post1, and use option "--enable-prefix-caching" when start a model service, it will not have memory leak, right?

RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
@robertgshaw2-redhat
[BugFix] Fix Memory Leak (vllm-project#17567)
7e318ba 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
dtrifiro pushed a commit to red-hat-data-services/vllm that referenced this pull request May 13, 2025
@robertgshaw2-redhat @heyselbi
[BugFix] Fix Memory Leak (vllm-project#17567)
e335c34 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
dtrifiro pushed a commit to red-hat-data-services/vllm that referenced this pull request May 13, 2025
@andy-neuma
Sync midstream to upstream v0.8.5.post1 (opendatahub-io#218)
7db80a0 
Syncing midstream NM fork to Upstream tag of
[v0.8.5.post1](https://github.com/vllm-project/vllm/tree/v0.8.5.post1) +
cherry pick of
vllm-project@be633fb
needed for benchmarks +
[CP](neuralmagic/nm-vllm-ent@1fe447d)
for compressed tensor bump +
[CP](vllm-project#17677) for lora on AMD +
[CP](vllm-project#17315) for llama4 w/ pure
dense layers

```
commit 31c73ba (HEAD -> upstream-v0.8.5, nm-fork/upstream-v0.8.5)
Author: Chauncey <chaunceyjiang@gmail.com>
Date:   Wed Apr 30 15:11:04 2025 +0800

    [Bugfix] Fix AttributeError: 'State' object has no attribute 'engine_client' (vllm-project#17434)
    
    Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

commit f8db0bd
Author: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Date:   Fri May 2 14:01:38 2025 -0400

    [BugFix][Attention] Fix sliding window attention in V1 giving incorrect results (vllm-project#17574)
    
    Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

commit e335c34
Author: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Date:   Fri May 2 04:07:03 2025 -0400

    [BugFix] Fix Memory Leak (vllm-project#17567)
    
    Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>

commit cc463fe
Merge: 1e358ff ba41cc9
Author: Selbi Nuryyeva <selbi@redhat.com>
Date:   Tue Apr 29 12:34:57 2025 -0400

    Merge branch 'tag-upstream-v0.8.5' into upstream-v0.8.5

commit ba41cc9 (tag: v0.8.5, tag-upstream-v0.8.5)
Author: Michael Goin <mgoin64@gmail.com>
Date:   Mon Apr 28 16:20:24 2025 -0600

    [Model] Add tuned triton fused_moe configs for Qwen3Moe (vllm-project#17328)
    
    Signed-off-by: mgoin <mgoin64@gmail.com>

commit dcbac4c
Author: Simon Mo <simon.mo@hey.com>
Date:   Mon Apr 28 14:12:01 2025 -0700

    [Model] Qwen3 Dense FP8 Compat Fixes (vllm-project#17318)
    
    Signed-off-by: simon-mo <xmo@berkeley.edu>
[...]
```

Commands
```
git fetch upstream
git checkout -b upstream-v0.8.5
git merge upstream/v0.8.5
git cherry-pick be633fb
```

TEST PLAN
accept sync:
https://github.com/neuralmagic/nm-cicd/actions/runs/14841223552
related PR in cicd: neuralmagic/nm-cicd#99
release workflow:
https://github.com/neuralmagic/nm-cicd/actions/runs/14845693864
mawong-amd pushed a commit to ROCm/vllm that referenced this pull request May 14, 2025
@robertgshaw2-redhat @mawong-amd
[BugFix] Fix Memory Leak (vllm-project#17567)
0570648 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
@ckhordiasma ckhordiasma mentioned this pull request May 14, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@comaniac comaniac comaniac left review comments

@njhill njhill njhill approved these changes

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed v1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@robertgshaw2-redhat @nightflight-dk @nFunctor @xiamuyingu @comaniac @njhill @DarkLight1337 @vllm-bot