Blocks used internally by pinner are provided to network by bitswap

[dirkmc]

Problem

The pinner uses the BlockStore to store the sets of direct and recursive pins.
At the moment any block put to the BlockStore is sent to Bitswap, so these internal blocks are being unnecessarily provided to the network.

In the code we

• create an IPLD instance with a BlockService.
• on start, set the BlockService's exchange to be Bitswap.
• in the dag component use the IPLD instance to store DAG nodes.
• in the pinner use the dag component to store objects

Solution

I suggest we

• create an offline IPLD instance, which only talks to the local BlockService (not to Bitswap)
• create an offline dag component, which uses the offline IPLD instance
• in the pinner use the offline dag component

[dirkmc]

@vasco-santos @alanshaw please let me know your thoughts about this approach

[vasco-santos]

Hey @dirkmc

I have been thinking after our conversation today, and despite not liking the solution of having 2 instances of ipld, I am not seeing a nice design for this, as ipld should be agnostic from the network perspective, and an option for not providing is odd. In this way, I would go with that proposal for now

[dirkmc]

Thanks @vasco-santos

The go implementation creates the second instance of ipld in a context where it can only be used by the pinner, so maybe it would make sense for us to do the same thing.

[dirkmc]

Just making a note here that @alanshaw suggested we should make sure there aren't any issues with locking or memory usage. In particular check that when IPLD lazy loads formats, it doesn't unnecessarily load formats twice.

[dirkmc]

Fixed by #2196

[SgtPooki]

js-ipfs is being deprecated in favor of Helia. You can learn more about this deprecation and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterwards (see #4336).

This issue is most likely resolved in Helia, please try it out!