You can respond from the invitation that appears on your manager's calendar (which is shared with you).