Skip to content

Commit a83ed28

Browse files
andris9andris9
committed
mongo-cursor-pagination-node6
1 parent 57f3137 commit a83ed28

File tree

2 files changed

+209
-145
lines changed

2 files changed

+209
-145
lines changed

‎README.md

Lines changed: 178 additions & 106 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,37 @@
1+
# NB!
2+
3+
This module is a fork of (mongo-cursor-pagination)[https://github.com/mixmaxhq/mongo-cursor-pagination] by MixMax. The fork was needed to keep supporting
4+
Node.js v6 as the upstream version started using async..await.
5+
6+
Use the following command to install this module:
7+
8+
npm install mongo-cursor-pagination-node6 --save
9+
10+
And then in your code:
11+
12+
const MongoPaging = require('mongo-cursor-pagination-node6');
13+
114
# mongo-cursor-pagination
215

3-
This module aids in implementing "cursor-based" pagination using Mongo range queries or relevancy-based search results. __This module is currently used in production for the [Mixmax API](https://developer.mixmax.com) to return millions of results a day__.
16+
This module aids in implementing "cursor-based" pagination using Mongo range queries or relevancy-based search results. **This module is currently used in
17+
production for the [Mixmax API](https://developer.mixmax.com) to return millions of results a day**.
418

519
## Background
620

721
See this [blog post](https://mixmax.com/blog/api-paging-built-the-right-way) for background on why this library was built.
822

923
API Pagination is typically implemented one of two different ways:
1024

11-
1. Offset-based paging. This is traditional paging where `skip` and `limit` parameters are passed on the url (or some variation such as `page_num` and `count`). The API would return the results and some indication of whether there is a next page, such as `has_more` on the response. An issue with this approach is that it assumes a static data set; if collection changes while querying, then results in pages will shift and the response will be wrong.
25+
1. Offset-based paging. This is traditional paging where `skip` and `limit` parameters are passed on the url (or some variation such as `page_num` and `count`).
26+
The API would return the results and some indication of whether there is a next page, such as `has_more` on the response. An issue with this approach is that
27+
it assumes a static data set; if collection changes while querying, then results in pages will shift and the response will be wrong.
1228

13-
2. Cursor-based paging. An improved way of paging where an API passes back a "cursor" (an opaque string) to tell the caller where to query the next or previous pages. The cursor is usually passed using query parameters `next` and `previous`. It's implementation is typically more performant that skip/limit because it can jump to any page without traversing all the records. It also handles records being added or removed because it doesn't use fixed offsets.
29+
2. Cursor-based paging. An improved way of paging where an API passes back a "cursor" (an opaque string) to tell the caller where to query the next or previous
30+
pages. The cursor is usually passed using query parameters `next` and `previous`. It's implementation is typically more performant that skip/limit because it
31+
can jump to any page without traversing all the records. It also handles records being added or removed because it doesn't use fixed offsets.
1432

15-
This module helps in implementing #2 - cursor based paging - by providing a method that make it easy to query within a Mongo collection. It also helps by returning a url-safe string that you can return with your HTTP response (see example below).
33+
This module helps in implementing #2 - cursor based paging - by providing a method that make it easy to query within a Mongo collection. It also helps by
34+
returning a url-safe string that you can return with your HTTP response (see example below).
1635

1736
Here are some examples of cursor-based APIs:
1837

@@ -64,32 +83,46 @@ var MongoClient = require('mongodb').MongoClient;
6483
var MongoPaging = require('mongo-cursor-pagination');
6584

6685
MongoClient.connect('mongodb://localhost:27017/mydb', (err, db) => {
67-
db.collection('myobjects').insertMany([{
68-
counter: 1
69-
}, {
70-
counter: 2
71-
}, {
72-
counter: 3
73-
}, {
74-
counter: 4
75-
}], (err) => {
76-
77-
// Query the first page.
78-
MongoPaging.find(db.collection('myobjects'), {
79-
limit: 2
80-
}, (err, result) => {
81-
console.log(result);
82-
83-
// Query next page.
84-
MongoPaging.find(db.collection('myobjects'), {
85-
limit: 2,
86-
next: result.next // This queries the next page
87-
}, (err, result) => {
88-
console.log(result);
89-
});
90-
91-
});
92-
});
86+
db.collection('myobjects').insertMany(
87+
[
88+
{
89+
counter: 1
90+
},
91+
{
92+
counter: 2
93+
},
94+
{
95+
counter: 3
96+
},
97+
{
98+
counter: 4
99+
}
100+
],
101+
err => {
102+
// Query the first page.
103+
MongoPaging.find(
104+
db.collection('myobjects'),
105+
{
106+
limit: 2
107+
},
108+
(err, result) => {
109+
console.log(result);
110+
111+
// Query next page.
112+
MongoPaging.find(
113+
db.collection('myobjects'),
114+
{
115+
limit: 2,
116+
next: result.next // This queries the next page
117+
},
118+
(err, result) => {
119+
console.log(result);
120+
}
121+
);
122+
}
123+
);
124+
}
125+
);
93126
});
94127
```
95128

@@ -111,7 +144,8 @@ page 2 { results:
111144
112145
### search()
113146
114-
Search uses Mongo's [text search](https://docs.mongodb.com/v3.2/text-search/) feature and will return paged results ordered by search relevancy. As such, and unlike `find()`, it does not take a `paginatedField` parameter.
147+
Search uses Mongo's [text search](https://docs.mongodb.com/v3.2/text-search/) feature and will return paged results ordered by search relevancy. As such, and
148+
unlike `find()`, it does not take a `paginatedField` parameter.
115149
116150
```
117151
Performs a search query on a Mongo collection and pages the results. This is different from
@@ -140,38 +174,55 @@ var MongoClient = require('mongodb').MongoClient;
140174
var MongoPaging = require('mongo-cursor-pagination');
141175
142176
MongoClient.connect('mongodb://localhost:27017/mydb', (err, db) => {
143-
db.collection('myobjects').ensureIndex({
144-
mytext: 'text'
145-
}, (err) => {
146-
147-
db.collection('myobjects').insertMany([{
148-
mytext: 'dogs'
149-
}, {
150-
mytext: 'dogs cats'
151-
}, {
152-
mytext: 'dogs cats pigs'
153-
}], (err) => {
154-
155-
// Query the first page.
156-
MongoPaging.search(db.collection('myobjects'), 'dogs', {
157-
fields: {
158-
mytext: 1
177+
db.collection('myobjects').ensureIndex(
178+
{
179+
mytext: 'text'
159180
},
160-
limit: 2
161-
}, (err, result) => {
162-
console.log(result);
163-
164-
// Query next page.
165-
MongoPaging.search(db.collection('myobjects'), 'dogs', {
166-
limit: 2,
167-
next: result.next // This queries the next page
168-
}, (err, result) => {
169-
console.log(result);
170-
});
171-
172-
});
173-
});
174-
});
181+
err => {
182+
db.collection('myobjects').insertMany(
183+
[
184+
{
185+
mytext: 'dogs'
186+
},
187+
{
188+
mytext: 'dogs cats'
189+
},
190+
{
191+
mytext: 'dogs cats pigs'
192+
}
193+
],
194+
err => {
195+
// Query the first page.
196+
MongoPaging.search(
197+
db.collection('myobjects'),
198+
'dogs',
199+
{
200+
fields: {
201+
mytext: 1
202+
},
203+
limit: 2
204+
},
205+
(err, result) => {
206+
console.log(result);
207+
208+
// Query next page.
209+
MongoPaging.search(
210+
db.collection('myobjects'),
211+
'dogs',
212+
{
213+
limit: 2,
214+
next: result.next // This queries the next page
215+
},
216+
(err, result) => {
217+
console.log(result);
218+
}
219+
);
220+
}
221+
);
222+
}
223+
);
224+
}
225+
);
175226
});
176227
```
177228
@@ -188,67 +239,82 @@ page 2 { results:
188239
189240
### Use with ExpressJS
190241
191-
A popular use of this module is with Express to implement a basic API. As a convenience for this use-case, this library exposes a `findWithReq` function that takes the request object from your Express middleware and returns results:
242+
A popular use of this module is with Express to implement a basic API. As a convenience for this use-case, this library exposes a `findWithReq` function that
243+
takes the request object from your Express middleware and returns results:
192244
193245
So this code using `find()`:
194246
195247
```js
196248
router.get('/myobjects', (req, res, next) => {
197-
MongoPaging.find(db.collection('myobjects'), {
198-
query: {
199-
userId: req.user._id
200-
},
201-
paginatedField: 'created',
202-
fields: { // Also need to read req.query.fields to use to filter these fields
203-
_id: 1,
204-
created: 1
205-
},
206-
limit: req.query.limit, // Also need to cap this to 25
207-
next: req.query.next,
208-
previous: req.query.previous,
209-
}, function(err, result) {
210-
if (err) {
211-
next(err);
212-
} else {
213-
res.json(result);
214-
}
215-
});
249+
MongoPaging.find(
250+
db.collection('myobjects'),
251+
{
252+
query: {
253+
userId: req.user._id
254+
},
255+
paginatedField: 'created',
256+
fields: {
257+
// Also need to read req.query.fields to use to filter these fields
258+
_id: 1,
259+
created: 1
260+
},
261+
limit: req.query.limit, // Also need to cap this to 25
262+
next: req.query.next,
263+
previous: req.query.previous
264+
},
265+
function(err, result) {
266+
if (err) {
267+
next(err);
268+
} else {
269+
res.json(result);
270+
}
271+
}
272+
);
216273
});
217274
```
218275
219276
Is more elegant with `findWithReq()`:
220277
221278
```js
222279
router.get('/myobjects', (req, res, next) => {
223-
MongoPaging.findWithReq(req, db.collection('myobjects'), {
224-
query: {
225-
userId: req.user._id
226-
},
227-
paginatedField: 'created',
228-
fields: {
229-
_id: 1,
230-
created: 1
231-
},
232-
limit: 25 // Upper limit
233-
}, function(err, result) {
234-
if (err) {
235-
next(err);
236-
} else {
237-
res.json(result);
238-
}
239-
});
280+
MongoPaging.findWithReq(
281+
req,
282+
db.collection('myobjects'),
283+
{
284+
query: {
285+
userId: req.user._id
286+
},
287+
paginatedField: 'created',
288+
fields: {
289+
_id: 1,
290+
created: 1
291+
},
292+
limit: 25 // Upper limit
293+
},
294+
function(err, result) {
295+
if (err) {
296+
next(err);
297+
} else {
298+
res.json(result);
299+
}
300+
}
301+
);
240302
});
241303
```
242304
243-
`findWithReq()` also handles basic security such as making sure the `limit` and `fields` requested on the URL are within the allowed values you specify in `params`.
305+
`findWithReq()` also handles basic security such as making sure the `limit` and `fields` requested on the URL are within the allowed values you specify in
306+
`params`.
244307
245308
### Number of results
246309
247-
If the `limit` parameter isn't passed, then this library will default to returning 50 results. This can be overridden by setting `mongoPaging.config.DEFAULT_LIMIT = <new default limit>;`. Regardless of the `limit` passed in, a maximum of 300 documents will be returned. This can be overridden by setting `mongoPaging.config.MAX_LIMIT = <new max limit>;`.
310+
If the `limit` parameter isn't passed, then this library will default to returning 50 results. This can be overridden by setting
311+
`mongoPaging.config.DEFAULT_LIMIT = <new default limit>;`. Regardless of the `limit` passed in, a maximum of 300 documents will be returned. This can be
312+
overridden by setting `mongoPaging.config.MAX_LIMIT = <new max limit>;`.
248313
249314
### Indexes for sorting
250315
251-
`mongo-cursor-pagination` uses `_id` as a secondary sorting field when providing a `paginatedField` property. It is recommended that you have an index for optimal performance. Example:
316+
`mongo-cursor-pagination` uses `_id` as a secondary sorting field when providing a `paginatedField` property. It is recommended that you have an index for
317+
optimal performance. Example:
252318
253319
```
254320
MongoPaging.find(db.people, {
@@ -276,23 +342,29 @@ To run tests, you first must [start a Mongo server on port 27017](https://mongod
276342
277343
## Changelog
278344
279-
* 5.0.0 Now `50` results are returned by default, and up to `300` results can be returned if the `limit` parameter is used. These can be overridden by setting `mongoPaging.config.DEFAULT_LIMIT` and `mongoPaging.config.MAX_LIMIT` respectively.
345+
* 5.0.0 Now `50` results are returned by default, and up to `300` results can be returned if the `limit` parameter is used. These can be overridden by setting
346+
`mongoPaging.config.DEFAULT_LIMIT` and `mongoPaging.config.MAX_LIMIT` respectively.
280347
281348
* 4.1.1 Fixed bug that would overwrite `$or` in queries passed in.
282349
283350
* 4.1.0 Adds `sortAscending` option to sort by the `paginatedField` ascending. Defaults to false (existing behavior).
284351
285-
* 4.0.0 Breaking API change: `next` and `previous` attributes are now always returned with every response (in case the client wants to poll for new changes). New attributes `hasPrevious` and `hasNext` should now be used know if there are more results in the previous or next page. Before the change, `next` and `previously` could not be replied upon to know if there were more pages.
352+
* 4.0.0 Breaking API change: `next` and `previous` attributes are now always returned with every response (in case the client wants to poll for new changes).
353+
New attributes `hasPrevious` and `hasNext` should now be used know if there are more results in the previous or next page. Before the change, `next` and
354+
`previously` could not be replied upon to know if there were more pages.
286355
287356
* 3.1.1 Don't use `let` for backwards compatibility.
288357
289-
* 3.1.0 `findInReq()` now accepts dot notation for fields. So you can pass `?fields=users.userId` to only turn the `userId` property for `users` in the response.
358+
* 3.1.0 `findInReq()` now accepts dot notation for fields. So you can pass `?fields=users.userId` to only turn the `userId` property for `users` in the
359+
response.
290360
291361
* 3.0.1 Fixed bug where the _id field was always returned when a paginatedField was used.
292362
293363
* 3.0.0 Breaking API change: `find()` no longer accepts a string for `limit`. Added `findWithReq`.
294364
295-
* 2.0.0 Changed API to so you now set global config on the config object instead of the root export itself (e.g. `require('mongo-cursor-pagination').config.MAX_LIMIT = 100`). The default `MAX_LIMIT` is now a more reasonable 25 instead of 100. Added `search()`. Fixed edge case where pages will be incorrect if paginatedField has duplicate values.
365+
* 2.0.0 Changed API to so you now set global config on the config object instead of the root export itself (e.g.
366+
`require('mongo-cursor-pagination').config.MAX_LIMIT = 100`). The default `MAX_LIMIT` is now a more reasonable 25 instead of 100. Added `search()`. Fixed edge
367+
case where pages will be incorrect if paginatedField has duplicate values.
296368
297369
* 1.1.0 Add `MAX_LIMIT` global setting to clamp
298370

0 commit comments

Comments
 (0)