My community is behind a membership paywall that blocks access, and users register and login via Wordpress. I'm then using the 3rd-party Invision Wordpress SSO plugin which logs them automatically into the community.

In this case, does have the reCAPTCHA on the community do anything since I don't use any Invision registration or login forms?

CAPTCHAs only will show to users who are not logged in so if there is not a way to access your community without logging in, there is really no need for it :) .