This is a weird one...
A while back I changed our serializers from a reflection based model, to one that uses run-time code generation via the CodeDOM. Performance is two orders of magnitude better, so happy happy. However (there's always a however) a weird problem cropped up yesterday that I'm not quite sure what to do with.
One of our applications has a case where they need to touch web.config for their ASP.NET applications while under load. When they do that, they start getting InvalidCastExceptions from one or more of the cached, code generated serializers. Strange. Internally, after we generate the new serializer, we cache a delegate to the Serialize method for later retrieval. It looks like that cache is getting boogered, so that the type cast exceptions ensue. This is caused by serializers that are compiled against an assembly in one loader context, getting called by code that's running against the same assembly in a different loader context, hence the type case failure.
I had assumed (probably a bad idea) that when you touch web.config, that the whole AppDomain running that application goes away. It looks like that's not the case here. If the AppDomain really went away, then the cache (a static Hashtable) would be cleared, and there wouldn't be a problem.
To test the behavior, I wrapped the call to the cached serializer to catch InvalidCastExceptions, and in the catch block, clear the serializer cache so that they get regenerated. That works fine, the serializers get rebuilt, and all is well. However, I also put in a cap, so that if there was a problem, we wouldn't just keep recycling the cache over and over again. On the 4th time the cache gets cleared, we give up, and just start throwing the InvalidCastExceptions.
The behavior that was reported back by the application team was that said fix worked fine until the 4th time they touch web.config, then it starts throwing exceptions. Ah ha! That means the AppDomain couldn't be going away, because the counter is stored in a static, and so should be resetting to 0 every time the application restarts.
Unfortunately, just because I can characterize the problem doesn't mean I know what to do about it.