fix: don't trash optional dependencies that fail due to incompatible platform #8127

paulrutter · 2025-02-23T08:36:05Z

This PR is a proposed solution for multiple reported issues regarding failed optional dependencies that are platform specific.
See #4828, #7961 and possibly related PR #8077.

It can be reproduced as follows:

mkdir test && cd test
npm init -y
npm i --verbose rollup
Read package-lock.json - observe 18 OS/CPU@rollup/* package variants are listed
rm package-lock.json
npm i --verbose
Read package-lock.json - observe only 1 OS/CPU @rollup/* package variant is now listed for the current platform

With this PR, the package-lock.json will remain to contain the initial 18 rollup related dependencies, as any optional failure due to incompatible platform (EBADPLATFORM) will not lead to trashing the node.

The side-effect of this is that all platform specific dependencies will be present on all platforms (just an empty directory where the platform doesn't match). But it does solve the issue at hand.

I would need help writing a proper unit test for this scenario though. With the steps below, i did manage to test my PR and see that it has the desired effect:

node . i --verbose rollup
Read package-lock.json - observe 18 OS/CPU@rollup/* package variants are listed
rm package-lock.json
node . i --verbose
Read package-lock.json - observe 18 OS/CPU@rollup/* package variants are listed still

It will show this in the logging (on my Windows based machine):

npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-win32-ia32-msvc
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-win32-arm64-msvc
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-linux-x64-musl
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-linux-x64-gnu
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-linux-s390x-gnu
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-linux-riscv64-gnu
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-linux-powerpc64le-gnu
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-linux-loongarch64-gnu
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-linux-arm64-musl
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-linux-arm64-gnu
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-linux-arm-musleabihf
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-linux-arm-gnueabihf
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-freebsd-x64
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-freebsd-arm64
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-darwin-x64
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-darwin-arm64
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-android-arm64
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\@rollup\rollup-android-arm-eabi
npm verbose reify skip trashing optional dependency due to platform mismatch C:\git\npm-cli\node_modules\fsevents

…n incompatible platform, to prevent issues when either the lockfile is removed

… pruning is skipped

workspaces/arborist/lib/arborist/build-ideal-tree.js

paulrutter · 2025-02-24T21:23:03Z

@wraithgar would you be able to take a look at this PR? Thanks 👍

paulrutter · 2025-02-25T17:09:00Z

@owlstronaut thanks for running the test suite. So, in this case, would it make sense to adjust the tests? Seems like these are incorrect, if we want to retain the entries in the lock file.

About code coverage, where would i need to add or extend the tests for build-ideal-tree? See also #8127 (comment)

owlstronaut · 2025-02-25T18:33:13Z

@owlstronaut thanks for running the test suite. So, in this case, would it make sense to adjust the tests? Seems like these are incorrect, if we want to retain the entries in the lock file.

About code coverage, where would i need to add or extend the tests for build-ideal-tree? See also #8127 (comment)

Sure! Yeah, in this case we would need to update the snapshot for the new expectations from reify, and for the isolated-mode, the 2 failing tests probably need a helper similar to setupRequire that will expect to find the folders, but find them empty.

workspaces/arborist/test/arborist/build-ideal-tree.js will need a test added for the code coverage.

I still have to leave your questions about the "right thing" to do open to @wraithgar, I've been on the team less than a month 😸

paulrutter · 2025-02-25T19:49:21Z

Working through the tests and comments now, hope to push something this evening.

- fix tests by adjusting snapshots for reify, now that it doesn't trash optional dependencies that fail due to a platform mismatch - fix tests for isolated-mode by adjusting the assertions to not expect failure, but expect an empty directory though in case of a platform mismatch

- Remove unneeded code in build-ideal-tree, as it currently doesn't seem to ever hit this code - Without this, the original issue is still fixed

paulrutter · 2025-02-25T20:21:06Z

@owlstronaut Could you run the test suite again? I fixed all the snapshot and failing tests.

Besides, i removed the code in build-ideal-tree, as it never seems to hit (in unit tests, but also not when building locally). The code in build-ideal-tree is not entirely clear to me, as i cannot figure out if loadFailures could ever contain a EBADPLATFORM error or not. In unit tests, it's always E404 or ETARGET, never EBADPLATFORM. I tried adding a testcase for this, but still didn't get EBADPLATFORM. Hence i removed the code in build-ideal-tree.

With these changes, the original issue is still verified to be fixed, see the changed snapshots.
The remaining question is if this is the desired approach to this issue.

jakebailey · 2025-02-25T20:22:46Z

workspaces/arborist/test/isolated-mode.js

@@ -552,9 +552,10 @@ tap.test('failing optional deps are not installed', async t => {
  const arborist = new Arborist({ path: dir, registry, packumentCache: new Map(), cache })
  await arborist.reify({ installStrategy: 'linked' })

-  t.notOk(setupRequire(dir)('which'), 'Failing optional deps should not be installed')
+  t.ok(setupRequire(dir)('which'), 'Failing optional deps should not be installed, but the empty directory is there')


Am I right to say that the new behavior would be to leave empty directories for all of the optional deps that did not succeed? Hopefully there's a way to avoid this, since that would leave a load of empty dirs for a lot of native packages, and potentially confuse code which is scanning for these dirs to know which one to load for the current platform...

That is correct indeed. This is used to rebuild the proper package-json.lock file based on the node_modules directory.
I don't know enough of NPM to surface a better solution, but maybe others have ideas on how this can prevented?

Instead of the empty directory, it could also keep a list of failed optionals in a separate file or something, but that would require far more changes i would assume.

That's really unfortunate; I thought that npm stored a file inside node_modules that described the layout? It seems like a bad idea to rely on an empty dir existing, and I really do think it'll cause something to break.

Yes, i can imagine breakage if packages rely on directory scanning without verifying the contents.

If this is not the desired approach, i would need guidance from the maintainers on what would be the best way forward.

Yep, we'll have @wraithgar weigh in

For what it's worth, i checked out bindings which i use myself in https://github.com/blueconic/node-oom-heapdump, and that library handles this just fine. Haven't checked how nodejs itself handles it, as it's able to load native modules out of the box nowadays.

https://github.com/TooTallNate/node-bindings/blob/c8033dcfc04c34397384e23f7399a30e6c13830d/bindings.js#L111

Also, the rollup package doesn't rely on directory scanning: https://github.com/rollup/rollup/blob/d3e79bd79e6b53b01ef7d6535d6ea9ad9be714dd/native.js#L77

sharp seems safe as well: https://github.com/lovell/sharp/blob/1533bf995acda779313fc178d2b9d46791349961/lib/sharp.js#L23

The main ones to check out are rollup and esbuild (the bulk of the people in the issue having this problem), but there's also swc, oxc, dprint, who have this same problem too.

eslint https://github.com/evanw/esbuild/blob/b914dd30294346aa15fcc04278f4b4b51b8b43b5/lib/npm/node-platform.ts#L150 is safe for the happy path, but the fallback for fixing the platform issue does use directory scanning.

swc is safe: https://github.com/swc-project/swc/blob/b65387ac570c4bbb3b776a69bda810be862b434e/packages/core/binding.js#L4

oxc is safe: https://github.com/oxc-project/oxc/blob/bff83c96a21d582775b65d97123d3a2ffabf30be/napi/parser/bindings.js#L4

dprint is safe: https://github.com/dprint/dprint/blob/00e8f5e9895147b20fe70a0e4e5437bd54d928e8/deployment/npm/install_api.js#L23

paulrutter · 2025-02-26T08:26:43Z

To summarize the discussion above:

with this PR @jakebailey is concerned about breaking libraries that rely on directory scanning for loading native modules
the table below shows the common referenced libraries from [BUG] Platform-specific optional dependencies not being included in package-lock.json when reinstalling with node_modules present #4828 and if these would be affected by this PR

Library	Is affected by PR	git reference
`bindings`	No	git
`rollup`	No	git
`esbuild`	Possibly for the fallback behavior, happy path is safe	git
`swc`	No	git
`oxc`	No	git
`dprint`	No	git
`sharp`	No	git

jakebailey · 2025-02-26T08:30:08Z

I assume you meant esbuild in the table above; the other bit is just the "cleanliness" of having dirs for an unbounded number of platforms and the annoyance of having to find the nonempty one when browsing the tree.

jsolly · 2025-03-06T20:15:26Z

Hi all. I am having the issue outlined in this closed ticket which was closed as a dup.

It sounds like there's a fix that's coming soon? This is the workaround I am using in GitHub Actions with the new arm architecture runner, runs-on: ubuntu-24.04-arm

      - name: Install dependencies
        run: npm install
      - name: WORKAROUND (Install biome manually)
        run: npm install --no-save @biomejs/cli-linux-arm64

…nal-dependencies-due-to-incompatible-platform

paulrutter · 2025-03-07T14:10:38Z

Hi all. I am having the issue outlined in this closed ticket which was closed as a dup.

It sounds like there's a fix that's coming soon? This is the workaround I am using in GitHub Actions with the new arm architecture runner, runs-on: ubuntu-24.04-arm
      - name: Install dependencies
        run: npm install
      - name: WORKAROUND (Install biome manually)
        run: npm install --no-save @biomejs/cli-linux-arm64

Let's try to keep this PR clean, as the related issue #4828 already contains a lot of similar reports.
Thanks!

restjohn · 2025-03-07T16:33:27Z

Could this also fix #7622? 🙏

paulrutter · 2025-03-07T16:37:43Z

Could this also fix #7622? 🙏

I doubt it will, as this PR won't change anything to the lockfile, except making sure it's consistent across different platforms.

moiz-frost · 2025-03-10T08:52:58Z

Meanwhile what can I do to for a workaround please?

paulrutter · 2025-03-11T10:44:55Z

@owlstronaut @wraithgar. Sorry for asking again, but is there any timeline on getting feedback on this PR? Thanks!

owlstronaut · 2025-03-12T16:08:50Z

@paulrutter sorry for the delay, I was on vacation last week!

After some discussion with @wraithgar, we don't think this is quite what we want to do. It is clever, but feels like a bit of a workaround as it is really using side-effects to get the desired outcome.

The issue stems from the fact that, when the lockfile is missing, the build-ideal-tree first builds from what already exists in the node_modules, and attempts to change it as little as possible https://github.com/npm/cli/blob/latest/workspaces/arborist/lib/arborist/build-ideal-tree.js#L291-L312

To have subsequent package‑lock generations exactly match the original, you need to "reintroduce" into the ideal tree any optional dependencies that were recorded in the original lockfile, even if they weren't installed on the current platform. Rather than building the ideal tree solely from what's physically present in node_modules (which is why a platform‑mismatched dependency gets dropped), you'd modify the tree‐building process so that it merges in the missing optional dependency metadata.

What probably needs to happen is in here. Traverse to find any optional dependencies that are not in the tree. When one is found, a new Node is created and added to the ideal tree's inventory. This way, when you regenerate the package‑lock.json from the ideal tree, it will include both the installed package and the optional dependency that wasn't physically reified.

paulrutter · 2025-03-12T16:17:11Z

Thanks for the elaborate feedback @owlstronaut. It makes sense that this is not the desired approach, i kinda thought so already. But since it did fix the issue at hand, it was a good way of getting this issue some attention.

Not sure if i have the knowledge to pull of the suggestions made, but given all the related reports, shouldn't this be something that should be handled by the maintainers?
Happy to assist where i can, just not sure if i know enough from the npm ecosystem to get this through.

owlstronaut · 2025-03-12T19:04:32Z

@paulrutter Yes, totally understand the desire to just get something working. Ideally this is something we, the maintainers, should handle - we just have limited resources to fund all of the priorities we face.

paulrutter · 2025-03-12T19:23:04Z

Understood👍🏻i contribute to other open source projects, so i recognize the challenge.
Let's see how we can move forward with this then; i will think about the outlined desired solution a bit further as well.
When i'm feeling strong, i might do another attempt. Otherwise we need to wait until someone else can do the work required.

zkat · 2025-03-21T16:27:20Z

@wraithgar sorry, I got pulled in because typescript stuff but:

How would you feel about keeping "pruned" optional deps in the physical tree (which I assume is what gets written to node_modules/.package-lock.json, and just flagging them with a bool or something instead of removing them from the tree entirely? You could get identical install behavior that way, but when the package-lock.json gets removed, you could still just grab node_modules/.package-lock.json and rebuild the ideal tree from that directly?

Arborist is (mostly) after my time, but I can't imagine this being a very big change to implement, and I don't mind sending a patch myself in the next couple of days if y'all consent to it. And hey, I'm on the payroll for now, so :)

… but keep them 'in the tree' Fixes: npm#4828 Fixes: npm#7961 Replaces: npm#8127 Replaces: npm#8077 When optional dependencies fail, we don't want to install them, for whatever reason. The way this "uninstallation" is done right now is by simply removing the dependencies from the ideal tree during reification. Unfortunately, this means that what gets saved to the "hidden lockfile" is the edited ideal tree as well, and when NPM runs next, it'll use that when regenerating the "real" package-lock. This PR tags failed optional deps such that they're still added to the "trash list" (and thus have their directories cleaned up by the reifier), but prevents Arborist from removing them altogether from the ideal tree (which is usually done by setting their parent to `null`, making them unreachable, and letting them get GC'd).

zkat · 2025-03-22T00:30:30Z

Hi. Just dropping a note that I'm trying to tackle this over at #8177 and I'm pretty confident in the approach but I don't be able to test in earnest until Monday. Some details aren't in place yet

paulrutter · 2025-03-22T07:16:26Z

Closing this one in favor of #8177

… but keep them 'in the tree' Fixes: npm#4828 Fixes: npm#7961 Replaces: npm#8127 Replaces: npm#8077 When optional dependencies fail, we don't want to install them, for whatever reason. The way this "uninstallation" is done right now is by simply removing the dependencies from the ideal tree during reification. Unfortunately, this means that what gets saved to the "hidden lockfile" is the edited ideal tree as well, and when NPM runs next, it'll use that when regenerating the "real" package-lock. This PR tags failed optional deps such that they're still added to the "trash list" (and thus have their directories cleaned up by the reifier), but prevents Arborist from removing them altogether from the ideal tree (which is usually done by setting their parent to `null`, making them unreachable, and letting them get GC'd).

… but keep them 'in the tree' Fixes: #4828 Fixes: #7961 Replaces: #8127 Replaces: #8077 When optional dependencies fail, we don't want to install them, for whatever reason. The way this "uninstallation" is done right now is by simply removing the dependencies from the ideal tree during reification. Unfortunately, this means that what gets saved to the "hidden lockfile" is the edited ideal tree as well, and when NPM runs next, it'll use that when regenerating the "real" package-lock. This PR tags failed optional deps such that they're still added to the "trash list" (and thus have their directories cleaned up by the reifier), but prevents Arborist from removing them altogether from the ideal tree (which is usually done by setting their parent to `null`, making them unreachable, and letting them get GC'd).

@zkat

…8184) … but keep them 'in the tree' This PR was authored by @zkat Fixes: #4828 Fixes: #7961 Replaces: #8127 Replaces: #8077 When optional dependencies fail, we don't want to install them, for whatever reason. The way this "uninstallation" is done right now is by simply removing the dependencies from the ideal tree during reification. Unfortunately, this means that what gets saved to the "hidden lockfile" is the edited ideal tree as well, and when npm runs next, it'll use that when regenerating the "real" package-lock. This PR tags failed optional deps such that they're still added to the "trash list" (and thus have their directories cleaned up by the reifier), but prevents Arborist from removing them altogether from the ideal tree (which is usually done by setting their parent to `null`, making them unreachable, and letting them get GC'd). PS: It's Friday, this is what I managed to get done together. I'm making this a draft PR for now so folks can look at it, but I want to make sure both reifiers work well, fix some messaging issues, and clean stuff up (I'm pretty sure I'm guarding `ideallyInert` in more places than I need to, because I was trying to find the right spot). Still, feel free to talk about the approach. I'll get back to this on Monday. PPS: also hi Co-authored-by: Kat Marchán <kzm@zkat.tech>

fix for npm#4828: don't trash optional dependencies that fail to to a…

e9f27cc

…n incompatible platform, to prevent issues when either the lockfile is removed

paulrutter requested a review from a team as a code owner February 23, 2025 08:36

paulrutter mentioned this pull request Feb 23, 2025

[BUG] Platform-specific optional dependencies not being included in package-lock.json when reinstalling with node_modules present #4828

Closed

2 tasks

paulrutter added 2 commits February 23, 2025 09:42

fix for npm#4828: move check to before loop

e77fa11

fix for npm#4828: logging improvement

6128824

paulrutter changed the title ~~fix for #4828: don't trash optional dependencies that fail to install due to incompatible platform~~ fix: don't trash optional dependencies that fail to install due to incompatible platform Feb 23, 2025

paulrutter mentioned this pull request Feb 23, 2025

[BUG] optional dependencies for OS/CPU package variants are being pruned unexpectedly #7961

Closed

2 tasks

paulrutter added 3 commits February 24, 2025 08:59

fix for npm#4828: apply the same fix to build-ideal-tree and log when…

ba20f4d

… pruning is skipped

fix for npm#4828: logging improvement

feba0f2

fix for npm#4828: simplify code

49e7ffc

paulrutter commented Feb 24, 2025

View reviewed changes

workspaces/arborist/lib/arborist/build-ideal-tree.js Outdated Show resolved Hide resolved

paulrutter changed the title ~~fix: don't trash optional dependencies that fail to install due to incompatible platform~~ fix: don't trash optional dependencies that fail due to incompatible platform Feb 25, 2025

owlstronaut self-assigned this Feb 25, 2025

paulrutter added 2 commits February 25, 2025 20:54

fix for npm#4828:

c7e07a5

- Remove unneeded code in build-ideal-tree, as it currently doesn't seem to ever hit this code - Without this, the original issue is still fixed

jakebailey reviewed Feb 25, 2025

View reviewed changes

Merge branch 'npm:latest' into maintenance/fix-4828-dont-thrash-optio…

03d2938

…nal-dependencies-due-to-incompatible-platform

zkat mentioned this pull request Mar 22, 2025

fix(arborist): omit failed optional dependencies from installed deps,… #8177

Closed

paulrutter closed this Mar 22, 2025

wraithgar mentioned this pull request Mar 31, 2025

fix(arborist): omit failed optional dependencies from installed deps #8184

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: don't trash optional dependencies that fail due to incompatible platform #8127

fix: don't trash optional dependencies that fail due to incompatible platform #8127

paulrutter commented Feb 23, 2025 •

edited

Loading

paulrutter commented Feb 24, 2025

paulrutter commented Feb 25, 2025 •

edited

Loading

owlstronaut commented Feb 25, 2025

paulrutter commented Feb 25, 2025

paulrutter commented Feb 25, 2025

jakebailey Feb 25, 2025

paulrutter Feb 25, 2025 •

edited

Loading

jakebailey Feb 25, 2025

paulrutter Feb 25, 2025 •

edited

Loading

owlstronaut Feb 25, 2025

paulrutter Feb 25, 2025 •

edited

Loading

jakebailey Feb 25, 2025

paulrutter Feb 25, 2025 •

edited

Loading

paulrutter commented Feb 26, 2025 •

edited

Loading

jakebailey commented Feb 26, 2025

jsolly commented Mar 6, 2025 •

edited

Loading

paulrutter commented Mar 7, 2025

restjohn commented Mar 7, 2025

paulrutter commented Mar 7, 2025

moiz-frost commented Mar 10, 2025

paulrutter commented Mar 11, 2025

owlstronaut commented Mar 12, 2025

paulrutter commented Mar 12, 2025 •

edited

Loading

owlstronaut commented Mar 12, 2025

paulrutter commented Mar 12, 2025

zkat commented Mar 21, 2025

zkat commented Mar 22, 2025

paulrutter commented Mar 22, 2025

fix: don't trash optional dependencies that fail due to incompatible platform #8127

fix: don't trash optional dependencies that fail due to incompatible platform #8127

Conversation

paulrutter commented Feb 23, 2025 • edited Loading

paulrutter commented Feb 24, 2025

paulrutter commented Feb 25, 2025 • edited Loading

owlstronaut commented Feb 25, 2025

paulrutter commented Feb 25, 2025

paulrutter commented Feb 25, 2025

jakebailey Feb 25, 2025

Choose a reason for hiding this comment

paulrutter Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

jakebailey Feb 25, 2025

Choose a reason for hiding this comment

paulrutter Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

owlstronaut Feb 25, 2025

Choose a reason for hiding this comment

paulrutter Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

jakebailey Feb 25, 2025

Choose a reason for hiding this comment

paulrutter Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

paulrutter commented Feb 26, 2025 • edited Loading

jakebailey commented Feb 26, 2025

jsolly commented Mar 6, 2025 • edited Loading

paulrutter commented Mar 7, 2025

restjohn commented Mar 7, 2025

paulrutter commented Mar 7, 2025

moiz-frost commented Mar 10, 2025

paulrutter commented Mar 11, 2025

owlstronaut commented Mar 12, 2025

paulrutter commented Mar 12, 2025 • edited Loading

owlstronaut commented Mar 12, 2025

paulrutter commented Mar 12, 2025

zkat commented Mar 21, 2025

zkat commented Mar 22, 2025

paulrutter commented Mar 22, 2025

paulrutter commented Feb 23, 2025 •

edited

Loading

paulrutter commented Feb 25, 2025 •

edited

Loading

paulrutter Feb 25, 2025 •

edited

Loading

paulrutter Feb 25, 2025 •

edited

Loading

paulrutter Feb 25, 2025 •

edited

Loading

paulrutter Feb 25, 2025 •

edited

Loading

paulrutter commented Feb 26, 2025 •

edited

Loading

jsolly commented Mar 6, 2025 •

edited

Loading

paulrutter commented Mar 12, 2025 •

edited

Loading