As you are utilizing the API in an abnormal manner here, you may encounter some "bumps in the road" as you have here. There is currently no whitelist. You can try restricting your API Key to your server's IP address if you haven't, however, likely this may be a new feature you need to suggest in our Feedback forum to allow for. Generally, it isn't a good idea in an API system like this.
Is the server sending no, bad, or corrupted key at any point or is the request not completing? This would be the only case to encounter this issue.