Pagination best practices

Mitch Dale

Hey @Product- generally speaking, what's the best practice to paginate through a list of items retrieved, say from an API call, in a Wrkflow? For example, if I were to retrieve a list of records from an API, and I set my limit to 50, and I get 3 pages of records, what's the best approach to go through them all? Set up a loop and have one page go through the entire Wrkflow and then go through the next page? Or build an array of all records initially, and then execute them all through the Wrkflow? Or something else?

Nick

Hey Mitch,

You can do either way that you've mentioned. However, typically we do a third way. I'll explain the way then explain why we prefer this method.

Process: Build the paginating API call loop as if that is the only thing you were doing in your Wrkflow. i.e. Hit the API, see if there is a next page, then hit the next page until done.

After you've built this loop connect the Success outcome of your API call to a new downstream Wrk Action that is the beginning of your path for processing that information. It ends up looking like the attached screenshot. This method is particularly effective if you don't need to know anything about the other pages to process the current page.

Reason we do it this way: The Wrk Platform has access to all upstream data in your Wrkflow Path. So, if you paginate on a big API and get a ton of data back from it each time then each Job is going to have to load a really large amount of data which may slow down your Wrkflow. When you split the paths like this each path ends up with the least amount of data possible. Also, all of your paths will run at around the same time instead of waiting for the first to finish before starting the second.