Interesting topic.  I have had this problem too over the years but I believe if you and your boss have an agreement whereby you are allowed access to his inbox, even whilst he is in the office, then you should be able to monitor his emails during the day to pick up on anything he has missed for you to bring it to his attention.  Most manager love it when you are pro-active.

In my current role my boss will say, there's an email just in from X, would you confirm my attendance and book all necessary travel.  Today for example he has a meeting at 4pm and the location was changed from an off site meeting to an internal one.  I've just had to grab him to make sure he wasn't about to leave the building.

If you ask for some direction it will enable you to point out these things as and when they arise.  I agree it's difficult but if you are clear in your role, you will be able to assist.  Hopefully your boss will be a little more understanding on your part.

I would put his name at the top of the list for next year's party!