Skip to content

feat: make new bench creation script check for free space on build server and report if under 5GB or x% #2573

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
0spinboson opened this issue Mar 30, 2025 · 2 comments

Comments

@0spinboson
Copy link

ran into the issue of a build server being out of space today, wasn't warned anywhere that this was an issue.

Alternatively/additionally you might consider invoking a script that cleans up (docker) build cache files etc., or add other routines to tell you how much space is currently consumed by same, as that's difficult information to ferret out given how docker works.

@tanmoysrt
Copy link
Member

There are already cleanup scripts which get triggerred once your build server's disk become 90% full.

For this you need to configure your prometheus and grafana instance. Because the disk full events get trigerred via the prometheus alert webhooks.

@0spinboson
Copy link
Author

okay, good to know, thanks. But right now when you don't have those set up you get no warning from the bench build scripts when they fail that it's related to lack of space, and that to me seems kind of unnecessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants