I think in 5 gallons I'd be more in the 2-3 oz range. I once used closer to 8 oz and damn it was a lot. too much of it kind of blew everything else out of the water. I do like it in moderation though - adds a nice complexity.
I can certainly understand that. I'm the same way with other malts that people seem to like by the pound.