Skip to content
This repository was archived by the owner on Apr 29, 2020. It is now read-only.

Commit 7119a09

Browse files
achingbrainachingbrain
authored
feat: convert to async/await (#21)
* feat: convert to async/await * chore: readme update * chore: fix linting * feat: convert internals to be async/await * test: increase test coverage * chore: remove uncessary await * feat: add export depth and recursive exports * chore: address PR comments * chore: update ipld formats * chore: PR comments * chore: standardise error codes
1 parent bf8bad2 commit 7119a09

26 files changed

+1636
-2306
lines changed

‎.aegir.js

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
'use strict'
2+
3+
module.exports = {
4+
karma: {
5+
browserNoActivityTimeout: 1000 * 1000,
6+
}
7+
}

‎README.md

+170-127
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,22 @@
1919

2020
## Table of Contents
2121

22-
- [Install](#install)
23-
- [Usage](#usage)
24-
- [Example](#example)
25-
- [API](#api)
26-
- [exporter(cid, ipld)](#exportercid-ipld-options)
27-
- [Contribute](#contribute)
28-
- [License](#license)
22+
- [ipfs-unixfs-exporter](#ipfs-unixfs-exporter)
23+
- [Lead Maintainer](#lead-maintainer)
24+
- [Table of Contents](#table-of-contents)
25+
- [Install](#install)
26+
- [Usage](#usage)
27+
- [Example](#example)
28+
- [API](#api)
29+
- [`exporter(cid, ipld)`](#exportercid-ipld)
30+
- [UnixFS V1 entries](#unixfs-v1-entries)
31+
- [Raw entries](#raw-entries)
32+
- [CBOR entries](#cbor-entries)
33+
- [`entry.content({ offset, length })`](#entrycontent-offset-length)
34+
- [`exporter.path(cid, ipld)`](#exporterpathcid-ipld)
35+
- [`exporter.recursive(cid, ipld)`](#exporterrecursivecid-ipld)
36+
- [Contribute](#contribute)
37+
- [License](#license)
2938

3039
## Install
3140

@@ -38,29 +47,41 @@
3847
### Example
3948

4049
```js
41-
// Create an export source pull-stream cid or ipfs path you want to export and a
42-
// <dag or ipld-resolver instance> to fetch the file from
50+
// import a file and export it again
51+
const importer = require('ipfs-unixfs-importer')
4352
const exporter = require('ipfs-unixfs-exporter')
44-
const pull = require('pull-stream/pull')
45-
const { stdout } = require('pull-stdio')
46-
47-
const options = {}
48-
49-
pull(
50-
exporter(cid, ipld, options),
51-
collect((error, files) => {
52-
if (error) {
53-
// ...handle error
54-
}
55-
56-
// Set up a pull stream that sends the file content to process.stdout
57-
pull(
58-
// files[0].content is a pull-stream that contains the bytes of the file
59-
files[0].content,
60-
stdout()
61-
)
62-
})
63-
)
53+
54+
const files = []
55+
56+
for await (const file of importer([{
57+
path: '/foo/bar.txt',
58+
content: Buffer.from(0, 1, 2, 3)
59+
}], ipld)) {
60+
files.push(file)
61+
}
62+
63+
console.info(files[0].cid) // Qmbaz
64+
65+
const entry = await exporter(files[0].cid, ipld)
66+
67+
console.info(entry.cid) // Qmqux
68+
console.info(entry.path) // Qmbaz/foo/bar.txt
69+
console.info(entry.name) // bar.txt
70+
console.info(entry.unixfs.fileSize()) // 4
71+
72+
// stream content from unixfs node
73+
const bytes = []
74+
75+
for await (const buf of entry.content({
76+
offset: 0, // optional offset
77+
length: 4 // optional length
78+
})) {
79+
bytes.push(buf)
80+
}
81+
82+
const content = Buffer.concat(bytes)
83+
84+
console.info(content) // 0, 1, 2, 3
6485
```
6586

6687
#### API
@@ -69,124 +90,146 @@ pull(
6990
const exporter = require('ipfs-unixfs-exporter')
7091
```
7192

72-
### exporter(cid, ipld, options)
93+
### `exporter(cid, ipld)`
7394

74-
Uses the given [dag API][] or an [ipld-resolver instance][] to fetch an IPFS [UnixFS][] object(s) by their CID.
95+
Uses the given [js-ipld instance][] to fetch an IPFS node by it's CID.
7596

76-
Creates a new pull stream that outputs objects of the form
97+
Returns a Promise which resolves to an `entry`.
7798

78-
```js
99+
#### UnixFS V1 entries
100+
101+
Entries with a `dag-pb` codec `CID` return UnixFS V1 entries:
102+
103+
```javascript
79104
{
80-
path: 'a name',
81-
content: <pull stream>
105+
name: 'foo.txt',
106+
path: 'Qmbar/foo.txt',
107+
cid: CID, // see https://github.com/multiformats/js-cid
108+
node: DAGNode, // see https://github.com/ipld/js-ipld-dag-pb
109+
content: function, // returns an async iterator
110+
unixfs: UnixFS // see https://github.com/ipfs/js-ipfs-unixfs
82111
}
83112
```
84113

85-
#### `offset` and `length`
114+
If the entry is a file, `entry.content()` returns an async iterator that yields one or more buffers containing the file content:
86115

87-
`offset` and `length` arguments can optionally be passed to the exporter function. These will cause the returned stream to only emit bytes starting at `offset` and with length of `length`.
116+
```javascript
117+
if (entry.unixfs.type === 'file') {
118+
for await (const chunk of entry.content()) {
119+
// chunk is a Buffer
120+
}
121+
}
122+
```
88123
89-
See [the tests](test/exporter.js) for examples of using these arguments.
124+
If the entry is a directory or hamt shard, `entry.content()` returns further `entry` objects:
90125
91-
```js
92-
const exporter = require('ipfs-unixfs-exporter')
93-
const pull = require('pull-stream')
94-
const drain = require('pull-stream/sinks/drain')
95-
96-
pull(
97-
exporter(cid, ipld, {
98-
offset: 0,
99-
length: 10
100-
})
101-
drain((file) => {
102-
// file.content is a pull stream containing only the first 10 bytes of the file
103-
})
104-
)
126+
```javascript
127+
if (entry.unixfs.type.includes('directory')) { // can be 'directory' or 'hamt-sharded-directory'
128+
for await (const entry of dir.content()) {
129+
console.info(entry.name)
130+
}
131+
}
105132
```
106133
107-
### `fullPath`
134+
#### Raw entries
108135
109-
If specified the exporter will emit an entry for every path component encountered.
136+
Entries with a `raw` codec `CID` return raw entries:
110137
111138
```javascript
112-
const exporter = require('ipfs-unixfs-exporter')
113-
const pull = require('pull-stream')
114-
const collect = require('pull-stream/sinks/collect')
115-
116-
pull(
117-
exporter('QmFoo.../bar/baz.txt', ipld, {
118-
fullPath: true
119-
})
120-
collect((err, files) => {
121-
console.info(files)
122-
123-
// [{
124-
// depth: 0,
125-
// name: 'QmFoo...',
126-
// path: 'QmFoo...',
127-
// size: ...
128-
// cid: CID
129-
// content: undefined
130-
// type: 'dir'
131-
// }, {
132-
// depth: 1,
133-
// name: 'bar',
134-
// path: 'QmFoo.../bar',
135-
// size: ...
136-
// cid: CID
137-
// content: undefined
138-
// type: 'dir'
139-
// }, {
140-
// depth: 2,
141-
// name: 'baz.txt',
142-
// path: 'QmFoo.../bar/baz.txt',
143-
// size: ...
144-
// cid: CID
145-
// content: <Pull stream>
146-
// type: 'file'
147-
// }]
148-
//
149-
})
150-
)
139+
{
140+
name: 'foo.txt',
141+
path: 'Qmbar/foo.txt',
142+
cid: CID, // see https://github.com/multiformats/js-cid
143+
node: Buffer, // see https://nodejs.org/api/buffer.html
144+
content: function, // returns an async iterator
145+
}
151146
```
152147

153-
### `maxDepth`
148+
`entry.content()` returns an async iterator that yields a buffer containing the node content:
154149

155-
If specified the exporter will only emit entries up to the specified depth.
150+
```javascript
151+
for await (const chunk of entry.content()) {
152+
// chunk is a Buffer
153+
}
154+
```
155+
156+
Unless you an options object containing `offset` and `length` keys as an argument to `entry.content()`, `chunk` will be equal to `entry.node`.
157+
158+
#### CBOR entries
159+
160+
Entries with a `dag-cbor` codec `CID` return JavaScript object entries:
156161
157162
```javascript
158-
const exporter = require('ipfs-unixfs-exporter')
159-
const pull = require('pull-stream')
160-
const collect = require('pull-stream/sinks/collect')
161-
162-
pull(
163-
exporter('QmFoo.../bar/baz.txt', ipld, {
164-
fullPath: true,
165-
maxDepth: 1
166-
})
167-
collect((err, files) => {
168-
console.info(files)
169-
170-
// [{
171-
// depth: 0,
172-
// name: 'QmFoo...',
173-
// path: 'QmFoo...',
174-
// size: ...
175-
// cid: CID
176-
// content: undefined
177-
// type: 'dir'
178-
// }, {
179-
// depth: 1,
180-
// name: 'bar',
181-
// path: 'QmFoo.../bar',
182-
// size: ...
183-
// cid: CID
184-
// content: undefined
185-
// type: 'dir'
186-
// }]
187-
//
188-
})
189-
)
163+
{
164+
name: 'foo.txt',
165+
path: 'Qmbar/foo.txt',
166+
cid: CID, // see https://github.com/multiformats/js-cid
167+
node: Object, // see https://github.com/ipld/js-ipld-dag-cbor
168+
}
169+
```
170+
171+
There is no `content` function for a `CBOR` node.
172+
173+
174+
#### `entry.content({ offset, length })`
175+
176+
When `entry` is a file or a `raw` node, `offset` and/or `length` arguments can be passed to `entry.content()` to return slices of data:
177+
178+
```javascript
179+
const bufs = []
180+
181+
for await (const chunk of entry.content({
182+
offset: 0,
183+
length: 5
184+
})) {
185+
bufs.push(chunk)
186+
}
187+
188+
// `data` contains the first 5 bytes of the file
189+
const data = Buffer.concat(bufs)
190+
```
191+
192+
If `entry` is a directory or hamt shard, passing `offset` and/or `length` to `entry.content()` will limit the number of files returned from the directory.
193+
194+
```javascript
195+
const entries = []
196+
197+
for await (const entry of dir.content({
198+
offset: 0,
199+
length: 5
200+
})) {
201+
entries.push(entry)
202+
}
203+
204+
// `entries` contains the first 5 files/directories in the directory
205+
```
206+
207+
### `exporter.path(cid, ipld)`
208+
209+
`exporter.path` will return an async iterator that yields entries for all segments in a path:
210+
211+
```javascript
212+
const entries = []
213+
214+
for await (const entry of exporter.path('Qmfoo/foo/bar/baz.txt', ipld)) {
215+
entries.push(entry)
216+
}
217+
218+
// entries contains 4x `entry` objects
219+
```
220+
221+
### `exporter.recursive(cid, ipld)`
222+
223+
`exporter.recursive` will return an async iterator that yields all entries beneath a given CID or IPFS path, as well as the containing directory.
224+
225+
```javascript
226+
const entries = []
227+
228+
for await (const child of exporter.recursive('Qmfoo/foo/bar', ipld)) {
229+
entries.push(entry)
230+
}
231+
232+
// entries contains all children of the `Qmfoo/foo/bar` directory and it's children
190233
```
191234
192235
[dag API]: https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/DAG.md

0 commit comments

Comments
 (0)