I have always milled the tiny rye malt separately, I just run it through my mill set normal, twice.
It was explained to me long ago that the high protein content of Rye, or wheat needs a very high lauter temp to keep it as viscous as possible so it doesn't gel.
That was probably the likely culprit right there. I think my mash temp was too low when I sparged. Because the mash tun was maxed out I had to mash thick too so I couldn't mash out (I usually don't) to raise the temp if I wanted to.