I do that sometimes. But often find the rest of the bread tastes fairly musty and unpleasant. I guess toasting it might have got round that :-)

While it's probably not too dangerous it doesn't seem the best idea with an already compromised immune system.