1 | # make-fetch-happen
|
---|
2 | [![npm version](https://img.shields.io/npm/v/make-fetch-happen.svg)](https://npm.im/make-fetch-happen) [![license](https://img.shields.io/npm/l/make-fetch-happen.svg)](https://npm.im/make-fetch-happen) [![Travis](https://img.shields.io/travis/npm/make-fetch-happen.svg)](https://travis-ci.org/npm/make-fetch-happen) [![Coverage Status](https://coveralls.io/repos/github/npm/make-fetch-happen/badge.svg?branch=latest)](https://coveralls.io/github/npm/make-fetch-happen?branch=latest)
|
---|
3 |
|
---|
4 | [`make-fetch-happen`](https://github.com/npm/make-fetch-happen) is a Node.js
|
---|
5 | library that wraps [`minipass-fetch`](https://github.com/npm/minipass-fetch) with additional
|
---|
6 | features [`minipass-fetch`](https://github.com/npm/minipass-fetch) doesn't intend to include, including HTTP Cache support, request
|
---|
7 | pooling, proxies, retries, [and more](#features)!
|
---|
8 |
|
---|
9 | ## Install
|
---|
10 |
|
---|
11 | `$ npm install --save make-fetch-happen`
|
---|
12 |
|
---|
13 | ## Table of Contents
|
---|
14 |
|
---|
15 | * [Example](#example)
|
---|
16 | * [Features](#features)
|
---|
17 | * [Contributing](#contributing)
|
---|
18 | * [API](#api)
|
---|
19 | * [`fetch`](#fetch)
|
---|
20 | * [`fetch.defaults`](#fetch-defaults)
|
---|
21 | * [`minipass-fetch` options](#minipass-fetch-options)
|
---|
22 | * [`make-fetch-happen` options](#extra-options)
|
---|
23 | * [`opts.cachePath`](#opts-cache-path)
|
---|
24 | * [`opts.cache`](#opts-cache)
|
---|
25 | * [`opts.proxy`](#opts-proxy)
|
---|
26 | * [`opts.noProxy`](#opts-no-proxy)
|
---|
27 | * [`opts.ca, opts.cert, opts.key`](#https-opts)
|
---|
28 | * [`opts.maxSockets`](#opts-max-sockets)
|
---|
29 | * [`opts.retry`](#opts-retry)
|
---|
30 | * [`opts.onRetry`](#opts-onretry)
|
---|
31 | * [`opts.integrity`](#opts-integrity)
|
---|
32 | * [Message From Our Sponsors](#wow)
|
---|
33 |
|
---|
34 | ### Example
|
---|
35 |
|
---|
36 | ```javascript
|
---|
37 | const fetch = require('make-fetch-happen').defaults({
|
---|
38 | cachePath: './my-cache' // path where cache will be written (and read)
|
---|
39 | })
|
---|
40 |
|
---|
41 | fetch('https://registry.npmjs.org/make-fetch-happen').then(res => {
|
---|
42 | return res.json() // download the body as JSON
|
---|
43 | }).then(body => {
|
---|
44 | console.log(`got ${body.name} from web`)
|
---|
45 | return fetch('https://registry.npmjs.org/make-fetch-happen', {
|
---|
46 | cache: 'no-cache' // forces a conditional request
|
---|
47 | })
|
---|
48 | }).then(res => {
|
---|
49 | console.log(res.status) // 304! cache validated!
|
---|
50 | return res.json().then(body => {
|
---|
51 | console.log(`got ${body.name} from cache`)
|
---|
52 | })
|
---|
53 | })
|
---|
54 | ```
|
---|
55 |
|
---|
56 | ### Features
|
---|
57 |
|
---|
58 | * Builds around [`minipass-fetch`](https://npm.im/minipass-fetch) for the core [`fetch` API](https://fetch.spec.whatwg.org) implementation
|
---|
59 | * Request pooling out of the box
|
---|
60 | * Quite fast, really
|
---|
61 | * Automatic HTTP-semantics-aware request retries
|
---|
62 | * Cache-fallback automatic "offline mode"
|
---|
63 | * Proxy support (http, https, socks, socks4, socks5)
|
---|
64 | * Built-in request caching following full HTTP caching rules (`Cache-Control`, `ETag`, `304`s, cache fallback on error, etc).
|
---|
65 | * Customize cache storage with any [Cache API](https://developer.mozilla.org/en-US/docs/Web/API/Cache)-compliant `Cache` instance. Cache to Redis!
|
---|
66 | * Node.js Stream support
|
---|
67 | * Transparent gzip and deflate support
|
---|
68 | * [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) support
|
---|
69 | * Literally punches nazis
|
---|
70 | * (PENDING) Range request caching and resuming
|
---|
71 |
|
---|
72 | ### Contributing
|
---|
73 |
|
---|
74 | The make-fetch-happen team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! The [Contributor Guide](https://github.com/npm/cli/blob/latest/CONTRIBUTING.md) outlines the process for community interaction and contribution. Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear.
|
---|
75 |
|
---|
76 | All participants and maintainers in this project are expected to follow the [npm Code of Conduct](https://www.npmjs.com/policies/conduct), and just generally be excellent to each other.
|
---|
77 |
|
---|
78 | Please refer to the [Changelog](CHANGELOG.md) for project history details, too.
|
---|
79 |
|
---|
80 | Happy hacking!
|
---|
81 |
|
---|
82 | ### API
|
---|
83 |
|
---|
84 | #### <a name="fetch"></a> `> fetch(uriOrRequest, [opts]) -> Promise<Response>`
|
---|
85 |
|
---|
86 | This function implements most of the [`fetch` API](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch): given a `uri` string or a `Request` instance, it will fire off an http request and return a Promise containing the relevant response.
|
---|
87 |
|
---|
88 | If `opts` is provided, the [`minipass-fetch`-specific options](#minipass-fetch-options) will be passed to that library. There are also [additional options](#extra-options) specific to make-fetch-happen that add various features, such as HTTP caching, integrity verification, proxy support, and more.
|
---|
89 |
|
---|
90 | ##### Example
|
---|
91 |
|
---|
92 | ```javascript
|
---|
93 | fetch('https://google.com').then(res => res.buffer())
|
---|
94 | ```
|
---|
95 |
|
---|
96 | #### <a name="fetch-defaults"></a> `> fetch.defaults([defaultUrl], [defaultOpts])`
|
---|
97 |
|
---|
98 | Returns a new `fetch` function that will call `make-fetch-happen` using `defaultUrl` and `defaultOpts` as default values to any calls.
|
---|
99 |
|
---|
100 | A defaulted `fetch` will also have a `.defaults()` method, so they can be chained.
|
---|
101 |
|
---|
102 | ##### Example
|
---|
103 |
|
---|
104 | ```javascript
|
---|
105 | const fetch = require('make-fetch-happen').defaults({
|
---|
106 | cachePath: './my-local-cache'
|
---|
107 | })
|
---|
108 |
|
---|
109 | fetch('https://registry.npmjs.org/make-fetch-happen') // will always use the cache
|
---|
110 | ```
|
---|
111 |
|
---|
112 | #### <a name="minipass-fetch-options"></a> `> minipass-fetch options`
|
---|
113 |
|
---|
114 | The following options for `minipass-fetch` are used as-is:
|
---|
115 |
|
---|
116 | * method
|
---|
117 | * body
|
---|
118 | * redirect
|
---|
119 | * follow
|
---|
120 | * timeout
|
---|
121 | * compress
|
---|
122 | * size
|
---|
123 |
|
---|
124 | These other options are modified or augmented by make-fetch-happen:
|
---|
125 |
|
---|
126 | * headers - Default `User-Agent` set to make-fetch happen. `Connection` is set to `keep-alive` or `close` automatically depending on `opts.agent`.
|
---|
127 | * agent
|
---|
128 | * If agent is null, an http or https Agent will be automatically used. By default, these will be `http.globalAgent` and `https.globalAgent`.
|
---|
129 | * If [`opts.proxy`](#opts-proxy) is provided and `opts.agent` is null, the agent will be set to an appropriate proxy-handling agent.
|
---|
130 | * If `opts.agent` is an object, it will be used as the request-pooling agent argument for this request.
|
---|
131 | * If `opts.agent` is `false`, it will be passed as-is to the underlying request library. This causes a new Agent to be spawned for every request.
|
---|
132 |
|
---|
133 | For more details, see [the documentation for `minipass-fetch` itself](https://github.com/npm/minipass-fetch#options).
|
---|
134 |
|
---|
135 | #### <a name="extra-options"></a> `> make-fetch-happen options`
|
---|
136 |
|
---|
137 | make-fetch-happen augments the `minipass-fetch` API with additional features available through extra options. The following extra options are available:
|
---|
138 |
|
---|
139 | * [`opts.cachePath`](#opts-cache-path) - Cache target to read/write
|
---|
140 | * [`opts.cache`](#opts-cache) - `fetch` cache mode. Controls cache *behavior*.
|
---|
141 | * [`opts.proxy`](#opts-proxy) - Proxy agent
|
---|
142 | * [`opts.noProxy`](#opts-no-proxy) - Domain segments to disable proxying for.
|
---|
143 | * [`opts.ca, opts.cert, opts.key, opts.strictSSL`](#https-opts)
|
---|
144 | * [`opts.localAddress`](#opts-local-address)
|
---|
145 | * [`opts.maxSockets`](#opts-max-sockets)
|
---|
146 | * [`opts.retry`](#opts-retry) - Request retry settings
|
---|
147 | * [`opts.onRetry`](#opts-onretry) - a function called whenever a retry is attempted
|
---|
148 | * [`opts.integrity`](#opts-integrity) - [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) metadata.
|
---|
149 |
|
---|
150 | #### <a name="opts-cache-path"></a> `> opts.cachePath`
|
---|
151 |
|
---|
152 | A string `Path` to be used as the cache root for [`cacache`](https://npm.im/cacache).
|
---|
153 |
|
---|
154 | **NOTE**: Requests will not be cached unless their response bodies are consumed. You will need to use one of the `res.json()`, `res.buffer()`, etc methods on the response, or drain the `res.body` stream, in order for it to be written.
|
---|
155 |
|
---|
156 | The default cache manager also adds the following headers to cached responses:
|
---|
157 |
|
---|
158 | * `X-Local-Cache`: Path to the cache the content was found in
|
---|
159 | * `X-Local-Cache-Key`: Unique cache entry key for this response
|
---|
160 | * `X-Local-Cache-Mode`: Either `stream` or `buffer` to indicate how the response was read from cacache
|
---|
161 | * `X-Local-Cache-Hash`: Specific integrity hash for the cached entry
|
---|
162 | * `X-Local-Cache-Status`: One of `miss`, `hit`, `stale`, `revalidated`, `updated`, or `skip` to signal how the response was created
|
---|
163 | * `X-Local-Cache-Time`: UTCString of the cache insertion time for the entry
|
---|
164 |
|
---|
165 | Using [`cacache`](https://npm.im/cacache), a call like this may be used to
|
---|
166 | manually fetch the cached entry:
|
---|
167 |
|
---|
168 | ```javascript
|
---|
169 | const h = response.headers
|
---|
170 | cacache.get(h.get('x-local-cache'), h.get('x-local-cache-key'))
|
---|
171 |
|
---|
172 | // grab content only, directly:
|
---|
173 | cacache.get.byDigest(h.get('x-local-cache'), h.get('x-local-cache-hash'))
|
---|
174 | ```
|
---|
175 |
|
---|
176 | ##### Example
|
---|
177 |
|
---|
178 | ```javascript
|
---|
179 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
---|
180 | cachePath: './my-local-cache'
|
---|
181 | }) // -> 200-level response will be written to disk
|
---|
182 | ```
|
---|
183 |
|
---|
184 | A possible (minimal) implementation for `MyCustomRedisCache`:
|
---|
185 |
|
---|
186 | ```javascript
|
---|
187 | const bluebird = require('bluebird')
|
---|
188 | const redis = require("redis")
|
---|
189 | bluebird.promisifyAll(redis.RedisClient.prototype)
|
---|
190 | class MyCustomRedisCache {
|
---|
191 | constructor (opts) {
|
---|
192 | this.redis = redis.createClient(opts)
|
---|
193 | }
|
---|
194 | match (req) {
|
---|
195 | return this.redis.getAsync(req.url).then(res => {
|
---|
196 | if (res) {
|
---|
197 | const parsed = JSON.parse(res)
|
---|
198 | return new fetch.Response(parsed.body, {
|
---|
199 | url: req.url,
|
---|
200 | headers: parsed.headers,
|
---|
201 | status: 200
|
---|
202 | })
|
---|
203 | }
|
---|
204 | })
|
---|
205 | }
|
---|
206 | put (req, res) {
|
---|
207 | return res.buffer().then(body => {
|
---|
208 | return this.redis.setAsync(req.url, JSON.stringify({
|
---|
209 | body: body,
|
---|
210 | headers: res.headers.raw()
|
---|
211 | }))
|
---|
212 | }).then(() => {
|
---|
213 | // return the response itself
|
---|
214 | return res
|
---|
215 | })
|
---|
216 | }
|
---|
217 | 'delete' (req) {
|
---|
218 | return this.redis.unlinkAsync(req.url)
|
---|
219 | }
|
---|
220 | }
|
---|
221 | ```
|
---|
222 |
|
---|
223 | #### <a name="opts-cache"></a> `> opts.cache`
|
---|
224 |
|
---|
225 | This option follows the standard `fetch` API cache option. This option will do nothing if [`opts.cachePath`](#opts-cache-path) is null. The following values are accepted (as strings):
|
---|
226 |
|
---|
227 | * `default` - Fetch will inspect the HTTP cache on the way to the network. If there is a fresh response it will be used. If there is a stale response a conditional request will be created, and a normal request otherwise. It then updates the HTTP cache with the response. If the revalidation request fails (for example, on a 500 or if you're offline), the stale response will be returned.
|
---|
228 | * `no-store` - Fetch behaves as if there is no HTTP cache at all.
|
---|
229 | * `reload` - Fetch behaves as if there is no HTTP cache on the way to the network. Ergo, it creates a normal request and updates the HTTP cache with the response.
|
---|
230 | * `no-cache` - Fetch creates a conditional request if there is a response in the HTTP cache and a normal request otherwise. It then updates the HTTP cache with the response.
|
---|
231 | * `force-cache` - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it creates a normal request and updates the HTTP cache with the response.
|
---|
232 | * `only-if-cached` - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it returns a network error. (Can only be used when request’s mode is "same-origin". Any cached redirects will be followed assuming request’s redirect mode is "follow" and the redirects do not violate request’s mode.)
|
---|
233 |
|
---|
234 | (Note: option descriptions are taken from https://fetch.spec.whatwg.org/#http-network-or-cache-fetch)
|
---|
235 |
|
---|
236 | ##### Example
|
---|
237 |
|
---|
238 | ```javascript
|
---|
239 | const fetch = require('make-fetch-happen').defaults({
|
---|
240 | cachePath: './my-cache'
|
---|
241 | })
|
---|
242 |
|
---|
243 | // Will error with ENOTCACHED if we haven't already cached this url
|
---|
244 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
---|
245 | cache: 'only-if-cached'
|
---|
246 | })
|
---|
247 |
|
---|
248 | // Will refresh any local content and cache the new response
|
---|
249 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
---|
250 | cache: 'reload'
|
---|
251 | })
|
---|
252 |
|
---|
253 | // Will use any local data, even if stale. Otherwise, will hit network.
|
---|
254 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
---|
255 | cache: 'force-cache'
|
---|
256 | })
|
---|
257 | ```
|
---|
258 |
|
---|
259 | #### <a name="opts-proxy"></a> `> opts.proxy`
|
---|
260 |
|
---|
261 | A string or `new url.URL()`-d URI to proxy through. Different Proxy handlers will be
|
---|
262 | used depending on the proxy's protocol.
|
---|
263 |
|
---|
264 | Additionally, `process.env.HTTP_PROXY`, `process.env.HTTPS_PROXY`, and
|
---|
265 | `process.env.PROXY` are used if present and no `opts.proxy` value is provided.
|
---|
266 |
|
---|
267 | (Pending) `process.env.NO_PROXY` may also be configured to skip proxying requests for all, or specific domains.
|
---|
268 |
|
---|
269 | ##### Example
|
---|
270 |
|
---|
271 | ```javascript
|
---|
272 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
---|
273 | proxy: 'https://corporate.yourcompany.proxy:4445'
|
---|
274 | })
|
---|
275 |
|
---|
276 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
---|
277 | proxy: {
|
---|
278 | protocol: 'https:',
|
---|
279 | hostname: 'corporate.yourcompany.proxy',
|
---|
280 | port: 4445
|
---|
281 | }
|
---|
282 | })
|
---|
283 | ```
|
---|
284 |
|
---|
285 | #### <a name="opts-no-proxy"></a> `> opts.noProxy`
|
---|
286 |
|
---|
287 | If present, should be a comma-separated string or an array of domain extensions
|
---|
288 | that a proxy should _not_ be used for.
|
---|
289 |
|
---|
290 | This option may also be provided through `process.env.NO_PROXY`.
|
---|
291 |
|
---|
292 | #### <a name="https-opts"></a> `> opts.ca, opts.cert, opts.key, opts.strictSSL`
|
---|
293 |
|
---|
294 | These values are passed in directly to the HTTPS agent and will be used for both
|
---|
295 | proxied and unproxied outgoing HTTPS requests. They mostly correspond to the
|
---|
296 | same options the `https` module accepts, which will be themselves passed to
|
---|
297 | `tls.connect()`. `opts.strictSSL` corresponds to `rejectUnauthorized`.
|
---|
298 |
|
---|
299 | #### <a name="opts-local-address"></a> `> opts.localAddress`
|
---|
300 |
|
---|
301 | Passed directly to `http` and `https` request calls. Determines the local
|
---|
302 | address to bind to.
|
---|
303 |
|
---|
304 | #### <a name="opts-max-sockets"></a> `> opts.maxSockets`
|
---|
305 |
|
---|
306 | Default: 15
|
---|
307 |
|
---|
308 | Maximum number of active concurrent sockets to use for the underlying
|
---|
309 | Http/Https/Proxy agents. This setting applies once per spawned agent.
|
---|
310 |
|
---|
311 | 15 is probably a _pretty good value_ for most use-cases, and balances speed
|
---|
312 | with, uh, not knocking out people's routers. 🤓
|
---|
313 |
|
---|
314 | #### <a name="opts-retry"></a> `> opts.retry`
|
---|
315 |
|
---|
316 | An object that can be used to tune request retry settings. Retries will only be attempted on the following conditions:
|
---|
317 |
|
---|
318 | * Request method is NOT `POST` AND
|
---|
319 | * Request status is one of: `408`, `420`, `429`, or any status in the 500-range. OR
|
---|
320 | * Request errored with `ECONNRESET`, `ECONNREFUSED`, `EADDRINUSE`, `ETIMEDOUT`, or the `fetch` error `request-timeout`.
|
---|
321 |
|
---|
322 | The following are worth noting as explicitly not retried:
|
---|
323 |
|
---|
324 | * `getaddrinfo ENOTFOUND` and will be assumed to be either an unreachable domain or the user will be assumed offline. If a response is cached, it will be returned immediately.
|
---|
325 |
|
---|
326 | If `opts.retry` is `false`, it is equivalent to `{retries: 0}`
|
---|
327 |
|
---|
328 | If `opts.retry` is a number, it is equivalent to `{retries: num}`
|
---|
329 |
|
---|
330 | The following retry options are available if you want more control over it:
|
---|
331 |
|
---|
332 | * retries
|
---|
333 | * factor
|
---|
334 | * minTimeout
|
---|
335 | * maxTimeout
|
---|
336 | * randomize
|
---|
337 |
|
---|
338 | For details on what each of these do, refer to the [`retry`](https://npm.im/retry) documentation.
|
---|
339 |
|
---|
340 | ##### Example
|
---|
341 |
|
---|
342 | ```javascript
|
---|
343 | fetch('https://flaky.site.com', {
|
---|
344 | retry: {
|
---|
345 | retries: 10,
|
---|
346 | randomize: true
|
---|
347 | }
|
---|
348 | })
|
---|
349 |
|
---|
350 | fetch('http://reliable.site.com', {
|
---|
351 | retry: false
|
---|
352 | })
|
---|
353 |
|
---|
354 | fetch('http://one-more.site.com', {
|
---|
355 | retry: 3
|
---|
356 | })
|
---|
357 | ```
|
---|
358 |
|
---|
359 | #### <a name="opts-onretry"></a> `> opts.onRetry`
|
---|
360 |
|
---|
361 | A function called whenever a retry is attempted.
|
---|
362 |
|
---|
363 | ##### Example
|
---|
364 |
|
---|
365 | ```javascript
|
---|
366 | fetch('https://flaky.site.com', {
|
---|
367 | onRetry() {
|
---|
368 | console.log('we will retry!')
|
---|
369 | }
|
---|
370 | })
|
---|
371 | ```
|
---|
372 |
|
---|
373 | #### <a name="opts-integrity"></a> `> opts.integrity`
|
---|
374 |
|
---|
375 | Matches the response body against the given [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) metadata. If verification fails, the request will fail with an `EINTEGRITY` error.
|
---|
376 |
|
---|
377 | `integrity` may either be a string or an [`ssri`](https://npm.im/ssri) `Integrity`-like.
|
---|
378 |
|
---|
379 | ##### Example
|
---|
380 |
|
---|
381 | ```javascript
|
---|
382 | fetch('https://registry.npmjs.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
|
---|
383 | integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
|
---|
384 | }) // -> ok
|
---|
385 |
|
---|
386 | fetch('https://malicious-registry.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
|
---|
387 | integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
|
---|
388 | }) // Error: EINTEGRITY
|
---|
389 | ```
|
---|
390 |
|
---|
391 | ### <a name="wow"></a> Message From Our Sponsors
|
---|
392 |
|
---|
393 | ![](stop.gif)
|
---|
394 |
|
---|
395 | ![](happening.gif)
|
---|