Not being familiar with Dovetail's integration with WordPress, it would be hard for us to say how to uninstall it.
It sounds like you're just using Invision Community now and everything has been brought over to Invision Community. If that is the case, you can contact the author of the Dovetail integration and find out how to uninstall that.
If there are no SSO's setup to your WordPress side or anything else that Invision Community is reaching out to WordPress for. You could ultimately remove WordPress files and database. However, as Randy mentioned, that assumes that your database is separate from your Invision Community database. If they are not, it is very manual that you'll need to determine what is a WordPress table or Invision Community.
Finally, if you do decide to go the route of delete files and databases, take a final backup before doing so (maybe of your whole server). That way, if you accidentally did something wrong, you have it.