I would love it if there was the ability to add custom fields to the Calendar app.

Yes there is one MOD (or plugin) that makes exactly that:

http://community.invisionpower.com/files/file/5165-calendar-template-system/

Not for 4.x, looking at either a plugin or integrated into Calendar.

I repeatedly asked for calendar custom fields prior to and throughout the beta phase, but despite general support the feedback from IPS was that they didn't foresee the need. Shame really as they would be good for all sorts of uses.