You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

634 lines
21 KiB

3 months ago
  1. node-fetch
  2. ==========
  3. [![npm version][npm-image]][npm-url]
  4. [![build status][travis-image]][travis-url]
  5. [![coverage status][codecov-image]][codecov-url]
  6. [![install size][install-size-image]][install-size-url]
  7. [![Discord][discord-image]][discord-url]
  8. A light-weight module that brings `window.fetch` to Node.js
  9. (We are looking for [v2 maintainers and collaborators](https://github.com/bitinn/node-fetch/issues/567))
  10. [![Backers][opencollective-image]][opencollective-url]
  11. <!-- TOC -->
  12. - [Motivation](#motivation)
  13. - [Features](#features)
  14. - [Difference from client-side fetch](#difference-from-client-side-fetch)
  15. - [Installation](#installation)
  16. - [Loading and configuring the module](#loading-and-configuring-the-module)
  17. - [Common Usage](#common-usage)
  18. - [Plain text or HTML](#plain-text-or-html)
  19. - [JSON](#json)
  20. - [Simple Post](#simple-post)
  21. - [Post with JSON](#post-with-json)
  22. - [Post with form parameters](#post-with-form-parameters)
  23. - [Handling exceptions](#handling-exceptions)
  24. - [Handling client and server errors](#handling-client-and-server-errors)
  25. - [Advanced Usage](#advanced-usage)
  26. - [Streams](#streams)
  27. - [Buffer](#buffer)
  28. - [Accessing Headers and other Meta data](#accessing-headers-and-other-meta-data)
  29. - [Extract Set-Cookie Header](#extract-set-cookie-header)
  30. - [Post data using a file stream](#post-data-using-a-file-stream)
  31. - [Post with form-data (detect multipart)](#post-with-form-data-detect-multipart)
  32. - [Request cancellation with AbortSignal](#request-cancellation-with-abortsignal)
  33. - [API](#api)
  34. - [fetch(url[, options])](#fetchurl-options)
  35. - [Options](#options)
  36. - [Class: Request](#class-request)
  37. - [Class: Response](#class-response)
  38. - [Class: Headers](#class-headers)
  39. - [Interface: Body](#interface-body)
  40. - [Class: FetchError](#class-fetcherror)
  41. - [License](#license)
  42. - [Acknowledgement](#acknowledgement)
  43. <!-- /TOC -->
  44. ## Motivation
  45. Instead of implementing `XMLHttpRequest` in Node.js to run browser-specific [Fetch polyfill](https://github.com/github/fetch), why not go from native `http` to `fetch` API directly? Hence, `node-fetch`, minimal code for a `window.fetch` compatible API on Node.js runtime.
  46. See Matt Andrews' [isomorphic-fetch](https://github.com/matthew-andrews/isomorphic-fetch) or Leonardo Quixada's [cross-fetch](https://github.com/lquixada/cross-fetch) for isomorphic usage (exports `node-fetch` for server-side, `whatwg-fetch` for client-side).
  47. ## Features
  48. - Stay consistent with `window.fetch` API.
  49. - Make conscious trade-off when following [WHATWG fetch spec][whatwg-fetch] and [stream spec](https://streams.spec.whatwg.org/) implementation details, document known differences.
  50. - Use native promise but allow substituting it with [insert your favorite promise library].
  51. - Use native Node streams for body on both request and response.
  52. - Decode content encoding (gzip/deflate) properly and convert string output (such as `res.text()` and `res.json()`) to UTF-8 automatically.
  53. - Useful extensions such as timeout, redirect limit, response size limit, [explicit errors](ERROR-HANDLING.md) for troubleshooting.
  54. ## Difference from client-side fetch
  55. - See [Known Differences](LIMITS.md) for details.
  56. - If you happen to use a missing feature that `window.fetch` offers, feel free to open an issue.
  57. - Pull requests are welcomed too!
  58. ## Installation
  59. Current stable release (`2.x`)
  60. ```sh
  61. $ npm install node-fetch
  62. ```
  63. ## Loading and configuring the module
  64. We suggest you load the module via `require` until the stabilization of ES modules in node:
  65. ```js
  66. const fetch = require('node-fetch');
  67. ```
  68. If you are using a Promise library other than native, set it through `fetch.Promise`:
  69. ```js
  70. const Bluebird = require('bluebird');
  71. fetch.Promise = Bluebird;
  72. ```
  73. ## Common Usage
  74. NOTE: The documentation below is up-to-date with `2.x` releases; see the [`1.x` readme](https://github.com/bitinn/node-fetch/blob/1.x/README.md), [changelog](https://github.com/bitinn/node-fetch/blob/1.x/CHANGELOG.md) and [2.x upgrade guide](UPGRADE-GUIDE.md) for the differences.
  75. #### Plain text or HTML
  76. ```js
  77. fetch('https://github.com/')
  78. .then(res => res.text())
  79. .then(body => console.log(body));
  80. ```
  81. #### JSON
  82. ```js
  83. fetch('https://api.github.com/users/github')
  84. .then(res => res.json())
  85. .then(json => console.log(json));
  86. ```
  87. #### Simple Post
  88. ```js
  89. fetch('https://httpbin.org/post', { method: 'POST', body: 'a=1' })
  90. .then(res => res.json()) // expecting a json response
  91. .then(json => console.log(json));
  92. ```
  93. #### Post with JSON
  94. ```js
  95. const body = { a: 1 };
  96. fetch('https://httpbin.org/post', {
  97. method: 'post',
  98. body: JSON.stringify(body),
  99. headers: { 'Content-Type': 'application/json' },
  100. })
  101. .then(res => res.json())
  102. .then(json => console.log(json));
  103. ```
  104. #### Post with form parameters
  105. `URLSearchParams` is available in Node.js as of v7.5.0. See [official documentation](https://nodejs.org/api/url.html#url_class_urlsearchparams) for more usage methods.
  106. NOTE: The `Content-Type` header is only set automatically to `x-www-form-urlencoded` when an instance of `URLSearchParams` is given as such:
  107. ```js
  108. const { URLSearchParams } = require('url');
  109. const params = new URLSearchParams();
  110. params.append('a', 1);
  111. fetch('https://httpbin.org/post', { method: 'POST', body: params })
  112. .then(res => res.json())
  113. .then(json => console.log(json));
  114. ```
  115. #### Handling exceptions
  116. NOTE: 3xx-5xx responses are *NOT* exceptions and should be handled in `then()`; see the next section for more information.
  117. Adding a catch to the fetch promise chain will catch *all* exceptions, such as errors originating from node core libraries, network errors and operational errors, which are instances of FetchError. See the [error handling document](ERROR-HANDLING.md) for more details.
  118. ```js
  119. fetch('https://domain.invalid/')
  120. .catch(err => console.error(err));
  121. ```
  122. #### Handling client and server errors
  123. It is common to create a helper function to check that the response contains no client (4xx) or server (5xx) error responses:
  124. ```js
  125. function checkStatus(res) {
  126. if (res.ok) { // res.status >= 200 && res.status < 300
  127. return res;
  128. } else {
  129. throw MyCustomError(res.statusText);
  130. }
  131. }
  132. fetch('https://httpbin.org/status/400')
  133. .then(checkStatus)
  134. .then(res => console.log('will not get here...'))
  135. ```
  136. ## Advanced Usage
  137. #### Streams
  138. The "Node.js way" is to use streams when possible:
  139. ```js
  140. fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png')
  141. .then(res => {
  142. const dest = fs.createWriteStream('./octocat.png');
  143. res.body.pipe(dest);
  144. });
  145. ```
  146. In Node.js 14 you can also use async iterators to read `body`; however, be careful to catch
  147. errors -- the longer a response runs, the more likely it is to encounter an error.
  148. ```js
  149. const fetch = require('node-fetch');
  150. const response = await fetch('https://httpbin.org/stream/3');
  151. try {
  152. for await (const chunk of response.body) {
  153. console.dir(JSON.parse(chunk.toString()));
  154. }
  155. } catch (err) {
  156. console.error(err.stack);
  157. }
  158. ```
  159. In Node.js 12 you can also use async iterators to read `body`; however, async iterators with streams
  160. did not mature until Node.js 14, so you need to do some extra work to ensure you handle errors
  161. directly from the stream and wait on it response to fully close.
  162. ```js
  163. const fetch = require('node-fetch');
  164. const read = async body => {
  165. let error;
  166. body.on('error', err => {
  167. error = err;
  168. });
  169. for await (const chunk of body) {
  170. console.dir(JSON.parse(chunk.toString()));
  171. }
  172. return new Promise((resolve, reject) => {
  173. body.on('close', () => {
  174. error ? reject(error) : resolve();
  175. });
  176. });
  177. };
  178. try {
  179. const response = await fetch('https://httpbin.org/stream/3');
  180. await read(response.body);
  181. } catch (err) {
  182. console.error(err.stack);
  183. }
  184. ```
  185. #### Buffer
  186. If you prefer to cache binary data in full, use buffer(). (NOTE: `buffer()` is a `node-fetch`-only API)
  187. ```js
  188. const fileType = require('file-type');
  189. fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png')
  190. .then(res => res.buffer())
  191. .then(buffer => fileType(buffer))
  192. .then(type => { /* ... */ });
  193. ```
  194. #### Accessing Headers and other Meta data
  195. ```js
  196. fetch('https://github.com/')
  197. .then(res => {
  198. console.log(res.ok);
  199. console.log(res.status);
  200. console.log(res.statusText);
  201. console.log(res.headers.raw());
  202. console.log(res.headers.get('content-type'));
  203. });
  204. ```
  205. #### Extract Set-Cookie Header
  206. Unlike browsers, you can access raw `Set-Cookie` headers manually using `Headers.raw()`. This is a `node-fetch` only API.
  207. ```js
  208. fetch(url).then(res => {
  209. // returns an array of values, instead of a string of comma-separated values
  210. console.log(res.headers.raw()['set-cookie']);
  211. });
  212. ```
  213. #### Post data using a file stream
  214. ```js
  215. const { createReadStream } = require('fs');
  216. const stream = createReadStream('input.txt');
  217. fetch('https://httpbin.org/post', { method: 'POST', body: stream })
  218. .then(res => res.json())
  219. .then(json => console.log(json));
  220. ```
  221. #### Post with form-data (detect multipart)
  222. ```js
  223. const FormData = require('form-data');
  224. const form = new FormData();
  225. form.append('a', 1);
  226. fetch('https://httpbin.org/post', { method: 'POST', body: form })
  227. .then(res => res.json())
  228. .then(json => console.log(json));
  229. // OR, using custom headers
  230. // NOTE: getHeaders() is non-standard API
  231. const form = new FormData();
  232. form.append('a', 1);
  233. const options = {
  234. method: 'POST',
  235. body: form,
  236. headers: form.getHeaders()
  237. }
  238. fetch('https://httpbin.org/post', options)
  239. .then(res => res.json())
  240. .then(json => console.log(json));
  241. ```
  242. #### Request cancellation with AbortSignal
  243. > NOTE: You may cancel streamed requests only on Node >= v8.0.0
  244. You may cancel requests with `AbortController`. A suggested implementation is [`abort-controller`](https://www.npmjs.com/package/abort-controller).
  245. An example of timing out a request after 150ms could be achieved as the following:
  246. ```js
  247. import AbortController from 'abort-controller';
  248. const controller = new AbortController();
  249. const timeout = setTimeout(
  250. () => { controller.abort(); },
  251. 150,
  252. );
  253. fetch(url, { signal: controller.signal })
  254. .then(res => res.json())
  255. .then(
  256. data => {
  257. useData(data)
  258. },
  259. err => {
  260. if (err.name === 'AbortError') {
  261. // request was aborted
  262. }
  263. },
  264. )
  265. .finally(() => {
  266. clearTimeout(timeout);
  267. });
  268. ```
  269. See [test cases](https://github.com/bitinn/node-fetch/blob/master/test/test.js) for more examples.
  270. ## API
  271. ### fetch(url[, options])
  272. - `url` A string representing the URL for fetching
  273. - `options` [Options](#fetch-options) for the HTTP(S) request
  274. - Returns: <code>Promise&lt;[Response](#class-response)&gt;</code>
  275. Perform an HTTP(S) fetch.
  276. `url` should be an absolute url, such as `https://example.com/`. A path-relative URL (`/file/under/root`) or protocol-relative URL (`//can-be-http-or-https.com/`) will result in a rejected `Promise`.
  277. <a id="fetch-options"></a>
  278. ### Options
  279. The default values are shown after each option key.
  280. ```js
  281. {
  282. // These properties are part of the Fetch Standard
  283. method: 'GET',
  284. headers: {}, // request headers. format is the identical to that accepted by the Headers constructor (see below)
  285. body: null, // request body. can be null, a string, a Buffer, a Blob, or a Node.js Readable stream
  286. redirect: 'follow', // set to `manual` to extract redirect headers, `error` to reject redirect
  287. signal: null, // pass an instance of AbortSignal to optionally abort requests
  288. // The following properties are node-fetch extensions
  289. follow: 20, // maximum redirect count. 0 to not follow redirect
  290. timeout: 0, // req/res timeout in ms, it resets on redirect. 0 to disable (OS limit applies). Signal is recommended instead.
  291. compress: true, // support gzip/deflate content encoding. false to disable
  292. size: 0, // maximum response body size in bytes. 0 to disable
  293. agent: null // http(s).Agent instance or function that returns an instance (see below)
  294. }
  295. ```
  296. ##### Default Headers
  297. If no values are set, the following request headers will be sent automatically:
  298. Header | Value
  299. ------------------- | --------------------------------------------------------
  300. `Accept-Encoding` | `gzip,deflate` _(when `options.compress === true`)_
  301. `Accept` | `*/*`
  302. `Content-Length` | _(automatically calculated, if possible)_
  303. `Transfer-Encoding` | `chunked` _(when `req.body` is a stream)_
  304. `User-Agent` | `node-fetch/1.0 (+https://github.com/bitinn/node-fetch)`
  305. Note: when `body` is a `Stream`, `Content-Length` is not set automatically.
  306. ##### Custom Agent
  307. The `agent` option allows you to specify networking related options which are out of the scope of Fetch, including and not limited to the following:
  308. - Support self-signed certificate
  309. - Use only IPv4 or IPv6
  310. - Custom DNS Lookup
  311. See [`http.Agent`](https://nodejs.org/api/http.html#http_new_agent_options) for more information.
  312. If no agent is specified, the default agent provided by Node.js is used. Note that [this changed in Node.js 19](https://github.com/nodejs/node/blob/4267b92604ad78584244488e7f7508a690cb80d0/lib/_http_agent.js#L564) to have `keepalive` true by default. If you wish to enable `keepalive` in an earlier version of Node.js, you can override the agent as per the following code sample.
  313. In addition, the `agent` option accepts a function that returns `http`(s)`.Agent` instance given current [URL](https://nodejs.org/api/url.html), this is useful during a redirection chain across HTTP and HTTPS protocol.
  314. ```js
  315. const httpAgent = new http.Agent({
  316. keepAlive: true
  317. });
  318. const httpsAgent = new https.Agent({
  319. keepAlive: true
  320. });
  321. const options = {
  322. agent: function (_parsedURL) {
  323. if (_parsedURL.protocol == 'http:') {
  324. return httpAgent;
  325. } else {
  326. return httpsAgent;
  327. }
  328. }
  329. }
  330. ```
  331. <a id="class-request"></a>
  332. ### Class: Request
  333. An HTTP(S) request containing information about URL, method, headers, and the body. This class implements the [Body](#iface-body) interface.
  334. Due to the nature of Node.js, the following properties are not implemented at this moment:
  335. - `type`
  336. - `destination`
  337. - `referrer`
  338. - `referrerPolicy`
  339. - `mode`
  340. - `credentials`
  341. - `cache`
  342. - `integrity`
  343. - `keepalive`
  344. The following node-fetch extension properties are provided:
  345. - `follow`
  346. - `compress`
  347. - `counter`
  348. - `agent`
  349. See [options](#fetch-options) for exact meaning of these extensions.
  350. #### new Request(input[, options])
  351. <small>*(spec-compliant)*</small>
  352. - `input` A string representing a URL, or another `Request` (which will be cloned)
  353. - `options` [Options][#fetch-options] for the HTTP(S) request
  354. Constructs a new `Request` object. The constructor is identical to that in the [browser](https://developer.mozilla.org/en-US/docs/Web/API/Request/Request).
  355. In most cases, directly `fetch(url, options)` is simpler than creating a `Request` object.
  356. <a id="class-response"></a>
  357. ### Class: Response
  358. An HTTP(S) response. This class implements the [Body](#iface-body) interface.
  359. The following properties are not implemented in node-fetch at this moment:
  360. - `Response.error()`
  361. - `Response.redirect()`
  362. - `type`
  363. - `trailer`
  364. #### new Response([body[, options]])
  365. <small>*(spec-compliant)*</small>
  366. - `body` A `String` or [`Readable` stream][node-readable]
  367. - `options` A [`ResponseInit`][response-init] options dictionary
  368. Constructs a new `Response` object. The constructor is identical to that in the [browser](https://developer.mozilla.org/en-US/docs/Web/API/Response/Response).
  369. Because Node.js does not implement service workers (for which this class was designed), one rarely has to construct a `Response` directly.
  370. #### response.ok
  371. <small>*(spec-compliant)*</small>
  372. Convenience property representing if the request ended normally. Will evaluate to true if the response status was greater than or equal to 200 but smaller than 300.
  373. #### response.redirected
  374. <small>*(spec-compliant)*</small>
  375. Convenience property representing if the request has been redirected at least once. Will evaluate to true if the internal redirect counter is greater than 0.
  376. <a id="class-headers"></a>
  377. ### Class: Headers
  378. This class allows manipulating and iterating over a set of HTTP headers. All methods specified in the [Fetch Standard][whatwg-fetch] are implemented.
  379. #### new Headers([init])
  380. <small>*(spec-compliant)*</small>
  381. - `init` Optional argument to pre-fill the `Headers` object
  382. Construct a new `Headers` object. `init` can be either `null`, a `Headers` object, an key-value map object or any iterable object.
  383. ```js
  384. // Example adapted from https://fetch.spec.whatwg.org/#example-headers-class
  385. const meta = {
  386. 'Content-Type': 'text/xml',
  387. 'Breaking-Bad': '<3'
  388. };
  389. const headers = new Headers(meta);
  390. // The above is equivalent to
  391. const meta = [
  392. [ 'Content-Type', 'text/xml' ],
  393. [ 'Breaking-Bad', '<3' ]
  394. ];
  395. const headers = new Headers(meta);
  396. // You can in fact use any iterable objects, like a Map or even another Headers
  397. const meta = new Map();
  398. meta.set('Content-Type', 'text/xml');
  399. meta.set('Breaking-Bad', '<3');
  400. const headers = new Headers(meta);
  401. const copyOfHeaders = new Headers(headers);
  402. ```
  403. <a id="iface-body"></a>
  404. ### Interface: Body
  405. `Body` is an abstract interface with methods that are applicable to both `Request` and `Response` classes.
  406. The following methods are not yet implemented in node-fetch at this moment:
  407. - `formData()`
  408. #### body.body
  409. <small>*(deviation from spec)*</small>
  410. * Node.js [`Readable` stream][node-readable]
  411. Data are encapsulated in the `Body` object. Note that while the [Fetch Standard][whatwg-fetch] requires the property to always be a WHATWG `ReadableStream`, in node-fetch it is a Node.js [`Readable` stream][node-readable].
  412. #### body.bodyUsed
  413. <small>*(spec-compliant)*</small>
  414. * `Boolean`
  415. A boolean property for if this body has been consumed. Per the specs, a consumed body cannot be used again.
  416. #### body.arrayBuffer()
  417. #### body.blob()
  418. #### body.json()
  419. #### body.text()
  420. <small>*(spec-compliant)*</small>
  421. * Returns: <code>Promise</code>
  422. Consume the body and return a promise that will resolve to one of these formats.
  423. #### body.buffer()
  424. <small>*(node-fetch extension)*</small>
  425. * Returns: <code>Promise&lt;Buffer&gt;</code>
  426. Consume the body and return a promise that will resolve to a Buffer.
  427. #### body.textConverted()
  428. <small>*(node-fetch extension)*</small>
  429. * Returns: <code>Promise&lt;String&gt;</code>
  430. Identical to `body.text()`, except instead of always converting to UTF-8, encoding sniffing will be performed and text converted to UTF-8 if possible.
  431. (This API requires an optional dependency of the npm package [encoding](https://www.npmjs.com/package/encoding), which you need to install manually. `webpack` users may see [a warning message](https://github.com/bitinn/node-fetch/issues/412#issuecomment-379007792) due to this optional dependency.)
  432. <a id="class-fetcherror"></a>
  433. ### Class: FetchError
  434. <small>*(node-fetch extension)*</small>
  435. An operational error in the fetching process. See [ERROR-HANDLING.md][] for more info.
  436. <a id="class-aborterror"></a>
  437. ### Class: AbortError
  438. <small>*(node-fetch extension)*</small>
  439. An Error thrown when the request is aborted in response to an `AbortSignal`'s `abort` event. It has a `name` property of `AbortError`. See [ERROR-HANDLING.MD][] for more info.
  440. ## Acknowledgement
  441. Thanks to [github/fetch](https://github.com/github/fetch) for providing a solid implementation reference.
  442. `node-fetch` v1 was maintained by [@bitinn](https://github.com/bitinn); v2 was maintained by [@TimothyGu](https://github.com/timothygu), [@bitinn](https://github.com/bitinn) and [@jimmywarting](https://github.com/jimmywarting); v2 readme is written by [@jkantr](https://github.com/jkantr).
  443. ## License
  444. MIT
  445. [npm-image]: https://flat.badgen.net/npm/v/node-fetch
  446. [npm-url]: https://www.npmjs.com/package/node-fetch
  447. [travis-image]: https://flat.badgen.net/travis/bitinn/node-fetch
  448. [travis-url]: https://travis-ci.org/bitinn/node-fetch
  449. [codecov-image]: https://flat.badgen.net/codecov/c/github/bitinn/node-fetch/master
  450. [codecov-url]: https://codecov.io/gh/bitinn/node-fetch
  451. [install-size-image]: https://flat.badgen.net/packagephobia/install/node-fetch
  452. [install-size-url]: https://packagephobia.now.sh/result?p=node-fetch
  453. [discord-image]: https://img.shields.io/discord/619915844268326952?color=%237289DA&label=Discord&style=flat-square
  454. [discord-url]: https://discord.gg/Zxbndcm
  455. [opencollective-image]: https://opencollective.com/node-fetch/backers.svg
  456. [opencollective-url]: https://opencollective.com/node-fetch
  457. [whatwg-fetch]: https://fetch.spec.whatwg.org/
  458. [response-init]: https://fetch.spec.whatwg.org/#responseinit
  459. [node-readable]: https://nodejs.org/api/stream.html#stream_readable_streams
  460. [mdn-headers]: https://developer.mozilla.org/en-US/docs/Web/API/Headers
  461. [LIMITS.md]: https://github.com/bitinn/node-fetch/blob/master/LIMITS.md
  462. [ERROR-HANDLING.md]: https://github.com/bitinn/node-fetch/blob/master/ERROR-HANDLING.md
  463. [UPGRADE-GUIDE.md]: https://github.com/bitinn/node-fetch/blob/master/UPGRADE-GUIDE.md