Set up pagination on a REST dataset
Many REST APIs return results in pages. The Pagination Mode on a REST dataset tells AutomatedOn how to ask for the next page, so you get every record. Check the API's documentation to see which style it uses.
Pagination modes
| Mode | Settings to fill in | How it works |
|---|---|---|
| None | None | One request only. |
| Cursor | Next Value Path, such as pagination.nextCursor. Cursor Parameter, which defaults to cursor. | Reads the cursor from each response and sends it back in the cursor parameter, until there's no cursor or a page is empty. |
| Next link | Next Value Path, such as links.next. | Follows the next page address from each response, until there isn't one or a page is empty. The link must be on the same host as the data source's Base URL. |
| Offset/Limit | Page Parameter, which defaults to offset. Optional Total Path. | Starts at 0 and moves the offset on by the number of items in each page, until a page is empty or shorter than the page size, or the total is reached. |
| Page number | Page Parameter, which defaults to page. Optional Total Path. | Requests page 1, 2, 3 and so on, until a page is empty or shorter than the page size, or the total is reached. |
Page size
For every mode except None, set Page Size Parameter (the name the API uses, such as limit or pageSize) and Page Size (how many items per request, such as 100). Setting a page size also lets AutomatedOn spot the last page when it comes back short.
Total Path
If the API reports the total number of items, such as pagination.total, enter its path in Total Path. AutomatedOn stops once it has that many, and the Preview tab can show how many records match in total.
The 1,000-page limit
To protect against settings that never find the last page, a fetch stops with this error after 1,000 pages, rather than carrying on with a partial set:
REST dataset paging stopped after 1000 pages without reaching the end. Check the pagination settings, or add query parameters that narrow the result set.
If you see it, check that Next Value Path or Total Path match the real response, raise the Page Size if the API allows, or add query parameters that return fewer records. AutomatedOn also stops if an API keeps returning the same cursor or link.
Checking your settings
The dataset's Queries tab shows the first page's address and describes how further pages will be followed.