I have found the idea of creating robots.txt with Pages great. This way I would be able to edit the "file" without uploading it per FTP. I have used this guide 

This guide does not work for me. The URL of the page created with this guide is _https://mysite.com/robots.txt/ and not _https://mysite.com/robots.txt

 

We've done this several times on multiple sites - it should work fine. You may wish to submit a ticket if that trailing slash is being added for some reason.