In the latest IPB release prior to 3.0 you have to manually remove each and every group that you've added to moderate a specific board.
Please make like two options 1 saying "Remove all moderation" or something to that effect and perhaps another which you can just click the boxes, and then click "Remove selected" Just like you can massively add 1 group to 50 forums. Hope this makes sense, thanks.

nice suggestion

Moderator management is improved in 3.0 already. This will NOT apply to moderators added prior to 3.0, for the record, but you can easily remove each existing moderator and then re-add them to make use of this new functionality.

Basically if you add a moderator to multiple forums in 3.0, only one record is saved in the database. If you want to add that moderator to another forum, you edit the existing saved record, and select the additional forum. If you want to remove the moderator from a forum/multiple forums, you can again edit the record to deselect the forums they shouldn't be able to moderate (or still click "Remove moderator" to remove them from an individual forum from the overview). If you want to remove the moderator from all forums, there's a new dropdown option "Remove from all forums" which does just that.

Basically, this feature is already in 3 - but it only works for newly added moderators.

You can optionally add individual moderator entries (e.g. add a "Moderators" group to one forum, and then again, separately, to another forum) if you need them to actually have different permissions in those different forums. You can not mass-manage those two separate records, however.