You can use the /gallery/images endpoint for this, it accepts the category as parameter
* @apiparam string categories Comma-delimited list of category IDs (will also include images in albums in those categories)
And the gallery-image apiresponse contains already what you suggested.
* @apiresponse object images URLs to where the images are stored. Keys are 'original', 'large', and 'small', and values are URLs to the corresponding images