For years, I worked with various mixes of flour (bread, AP, spelt, whole wheat, rye) and I used one starter, and it worked on all mixes. It was slow - about the speed indicated in the books on baking.
More recently, I have moved to all whole wheat (with 2% rye & 2% bean flour), all the time, and I have seen my fermentation/final rise times go way down (considering the dough temperature.)
I wanted some white sourdough for a party, and it turns out that my current sourdough starter is much slower on white flour than my old general purpose starter, that was often fed flour mixes containing white flour.
The bottom line is that starters adapt to what they are fed, and they need time to adapt to a new diet.
Am I going to start keeping 2 starters? No! If guests want white bread, they can bring it. This will be a challange for my wife's concept of "hostess", but she knows some of our guests do not get enough fiber . . . (We are doing this for our guest's health, and not my convenience!! -- I will put forth that argument. My wife likes the whole wheat bread, she just does not think it is "fancy" enough for guests. I assert that my sourdough whole wheat is as good as any bread available, )