Well the thing is, I find it a bit more realistic when I forget which color corresponds to whom. It's just like in real life, where you can remember the face, but can't figure out the name.

Yes, I agree with Dean. The thing is, when the wagons are present in the camp (the ones that cause an illusion, I mean), it is giving the hint that the Mummers can create such stuff. Maybe, instead of making them able to alter every room's description, rename the skill to something like, I don't kno...