Yet another in the note to self category.

When using App_offline.htm files to take down a site in IIS, if the file size isn't 512 bytes of content or greater then IE may choose to default to the "Show Friendly Http Errors". Comments count towards the byte size so client side comments will work too.

There is more about this at ScottGu's blog