starter and sour milk

normally, with a multi grain loaf bread I make, I uses a mix of AP and rye flour starters.

The liquid for the final dough is usually beer or water, sometimes whey.

Last couple times, I used a bit of sour (not gone and lumpy) milk in the final dough.

Might the sour milk have affected the rise from the natural yeast at all?
The results were heavy, barely risen loafs that I could have used for pavers, albeit, tasty pavers.