If the requirement is for the availabiity of the player to depend on whether the user can download attachments (which is a forum level permission) then I added functionality for this for videos back when Invision 4.3 added the native player for videos (see my feedback of my concern on this)
Same functionality was extended to cover audio file after they were added in v4.6 (I think)
https://www.ynwa.tv/files/file/38-ne-html5-audiovideo-player/
Instead of a player being displayed, the file will be displayed as an attachment instead if the user doesn't have the "Download attachments" permission in the forum in which the post is located:
However, if you require the user to be able to play the audio, but not download it, then the above won't assist. The resource does have functionality to disable the 'Download' button in browsers that support the ability to disable it, and to disable right-click on the player. But ultimately, if the player is available then they can always get the URL of the file via the HTML source code if they really wanted to do so.
Any questions on the resource, https://www.ynwa.tv/forum/98-pre-sales-questions/