REST API & Paginated Responses

[HeadStand]

This isn't exactly a complaint, just more of a "why" question.

I'm wondering why paginated responses don't return header data only instead of the full object. As someone who has worked with a fairly large number of APIs from various sources (and has written quite a few as well), I can say from experience that most of the time when you make an API call for a list of items, you aren't going to use ALL the information for ALL of the items. Most of the time you only need some basic properties and the rest you would retrieve by making an API call for the individual item.

For example, if I'm calling /nexus/invoices, I get back the FULL INVOICE OBJECT for EACH RECORD. That includes the customer object, all items, all transactions, and all purchases... and each purchase also includes the customer object. That means if I have an invoice with 5 items (and therefore 5 purchases), I receive the same customer object 6 times in a single call. And that's just for one record in the list. It's a little excessive.

This approach also makes /nexus/invoices/{id} redundant. I don't get any additional data from that call that I don't already have from /nexus/invoices, so why would I ever need to make that call in the first place?

 

[Mark]

You'll probably be most familiar with relational databases (tables, rows) and thinking of data in those terms.

But REST is resource-oriented, when you do a GET on an index you're saying "show me the resources" - it is appropriate and correct to return either the list of the resources or HATEOS links to them - to return anything else, such as an ID, is not technically correct. Since there is no reason not to return the full resource, and doing so saves the client application making additional HTTP calls, that is exactly what we do.

As for why there is a separate endpoint for viewing an individual record - this is again because it is a proper REST API, and all resources must have their own URI. Also, if you do know the ID, you'll want just that record (and an appropriate 404 response if it no longer exists) rather than a paginated response.

[HeadStand]

3 hours ago, Mark said:

You'll probably be most familiar with relational databases (tables, rows) and thinking of data in those terms.

But REST is resource-oriented, when you do a GET on an index you're saying "show me the resources" - it is appropriate and correct to return either the list of the resources or HATEOS links to them - to return anything else, such as an ID, is not technically correct. Since there is no reason not to return the full resource, and doing so saves the client application making additional HTTP calls, that is exactly what we do.

As for why there is a separate endpoint for viewing an individual record - this is again because it is a proper REST API, and all resources must have their own URI. Also, if you do know the ID, you'll want just that record (and an appropriate 404 response if it no longer exists) rather than a paginated response.

I've written more than one REST API, I'm quite familiar with how it works, but whatever, I'm not going to argue.

From personal experience though, I can tell you that on every development team I've worked with, it is always preferable to return just the data that is necessary rather than everything all at once. (And BTW, if someone were using this to develop a mobile app, would you not want to be reducing the amount of data going back and forth or am I making that up?)

[Mark]

Imagine you're using the API to display a list of the latest 10 posts showing the post contents, who posted it and the name of the topic it was posted in and the forum that topic is in. Currently you need to make one API call. If we only returned the post data and just the ID numbers for other resources, you'd need to make:

• 1 to get the posts
• 10 to get the name of the author for each post
• 10 to get the topic titles for each post
• 10 to get the forum name for each post

Making 31 HTTP requests is definitely not "preferable" to making 1, even if it does mean that 1 returns a lot of data - we're talking about text - a few bytes, not large images where there would be a consideration about the amount of data being transferred.

 

Just 2 examples off the top of my head of other APIs that do it the same way:

Github API to list issues: https://developer.github.com/v3/issues/#list-issues - note how the assignee, milestone (and the creator of the milestone under that) are all present. 

Stripe API to list customers: https://stripe.com/docs/api#list_customers - note how the entire customer object, including any cards or subscriptions on their account, is present.

[HeadStand]

So let's rephrase. It really depends on what kind of data you're retrieving. For posts and comments, I would agree with you, because that's not considered a "top-level" object. Your "top level" would be a topic, and when you view the topic, you want the list of posts. For something like the invoices, it doesn't make sense at all - especially the way the object is constructed. I can't come up with a reason I would need the customer object multiple times for a single invoice. 

To be honest, I haven't looked through all the calls yet, but I'm sure I can find other examples of this.

Also, I never said to return just ID numbers, that's insane. You return basic data that would be logically used in a list form. So using your example above - IF I were advocating header data for posts (which I wasn't), it would have been

• 1 call to get the posts
• 1 call for each post, which would return the full object with everything in it

That's 11 calls, not 31. Give me a little credit, I'm not an idiot.

[HeadStand]

OK. I took a (very) quick look at all the available API calls. I think the invoice is the only one I have a real issue with. The rest I can live with (although the CMS and Downloads calls are throwing errors for me when I try to view them. very strange. I'll have to figure out why.)

To me, returning the invoices the way they are now is the same thing as returning a blog with all the entries included. Why are you including all the items and purchases and shipments? 

I also think there are calls missing. There don't seem to be any calls to return nodes. No call for a list of forums, list of calendars, etc. One would think that's useful for an external application - how would I know what forum ID to filter by if I don't have the forums?

And one last point (don't hate me please): edit calls should probably be PUT, not POST, if we're going with proper REST format.

[Mark]

No, REST does not have CRUD concepts in the same way that, say, a relational database does. Neither POST or PUT is equivalent to create or edit. PUT creates or replaces a resource with the body provided. POST processes the body of the request in some way related to the resource. Our API uses the appropriate one correctly (which, IIRC, is only PUT in two places).

I considered PATCH, which would be better for most of our edit calls, but after an internal struggle over whether it's okay to use something non-standard, decided against it.