I think there are a couple of other factors here too. Whether we like to admit it or not, BYU is not looked upon by other programs with any great envy. Yes, we have some respect, but they know this is an LDS school (with all those strange values, you know, no sunday practices, no alcohol, no pre-marital sex), and coaches are not keen on "infecting" their coaching staff or program with some of that "odd" stuff, so I think they don't look seriously at BYU.
Also, I think there is an unspoken (mostly) view of BYU football that BYU doesn't recruit many blacks, we don't have much speed, and we have a "system" that allows us to succeed, but really don't have very good athletes, and they play all those old guys, and they're married (how weird is that?). So the view is one of, "BYU is a curiosity, but don't take them seriously," kind of thing.
And if you are LDS, faithful LDS, and values the LDS values; then yes, BYU is a "sweet gig" and the best place for you to coach--if you aren't paid millions for coaching there. A good friend of mine just retired as a BYU professor. He often complained about how the pay at BYU was not very good, but he loved it there, and he never looked at teaching anywhere else.