Here are some useful numbers for starters and for stepping up starters (these numbers are for a stir plate, based on my analysis of Kai's growth data):
You want about a Liter or Qt. (or more) for each 100 Billion, in order to avoid stunting growth. (You might get away with as little as 0.75 L and still get full growth, but you aren't leaving any overhead.) Below that, the yeast don't have enough food to both eat and reproduce at their full rate.
In that case, you'll add about 150 billion cells for each Liter. So, a vial goes into a liter, at 100 Billion per L, to yield 100 Billion + (1 L x 150 Billion/L) = 250 Billion. (0.75 L would yield 100 + (0.75*150) = 200 Billion, but below that growth is stunted.)
To repitch that 250 Billion, you would want 2.5 L+ (2 Liters, minimum, if you want to push it.)
I don't know that yeast are of poorer quality if their growth is stunted by over pitching a starter, but it is an indication that they are nutrient starved. That would be something worth trying to avoid when trying to optimize yeast health.