To get further here:
only hCaptcha is affected by problem
the "google" captchas are working fine
Looking into the IPS code make the probleme clear. The problem is how hCaptcha works and how it's integrated.
This IPS code part loads the js file from hcaptcha.com and this javascript code responsible for showing the hCaptcha.
The IPS hCaptcha template is simple and no data-controller is involved.
The external javascript code is doing some magic stuff, among other things, it is searches for an element with the class "h-captcha" and adds the iframe for the hCaptcha.
Summary:
As long as the registration form is hosted in the popup, no browser will load the external javascript file again and the captcha is not created.
For the browser, the js-file is already available and no need to load it again.
The registration form as a page (no popup) reloads everything, including the hcaptcha.com script. That's why it works there.