sourdough starter smell rye wheat white flour

Recently, I decided to start a new batch of sourdough cultures. Unfortunately, I've had problems with transitioning from their whole grain state to a white starter.

Over about 4 different starters (some using orange juice, some not, some using whole wheat, some rye) the initial activity was great and they seemed quite happy being maintained as wholegrain starters. Problems manifested as soon as I started transitioning to white flour.