Toasting eggy breads

I've noticed that when toasting enriched/rich breads with eggs in them, they usually acquire an unpleasant sandy-floury mouth feel and a short,  crumbly texture.

I assume that the short texture can be a result of fats in the bread, but then crackers also contain plenty of fat yet stay pleasantly crisp, not sandy.

Can eggs in the bread be the fault? I notice this more in challah then in sweet breads (the former contains plenty of eggs, and very little extra fat, the later contains less eggs but plenty of fat). 

Do you toast challah bread/brioche successfully? (dry toast, not french toast obviously :P )