I'll exaggerate a bit here and there for dramatic purposes, and i know some will disagree, but i've talked about this with enough people from enough places to know that, maybe not a majority, but i am at least part of a significant minorty in this view.
It's a mellange of things i suppose: talent, passion, a somewhat romantic political background*, and context. When someone told you "check this guys" its not as if you could google him, and watch last weeks youtube clips of his games. Most of the info at the time came from newspaper articles and mouth-to-mouth stories and anecdotes. It's not as if you could watch every game live when your national team played away. Watching a deferred game was a luxury (mostly because time zone differences and the fact that tv wasnt running 24/7). Getting copies of games in VHS too.
Imagine you just started watching rugby. Say 1980s. And you start reading and watching about the sport and everyone tells you about the all blacks, and how good they are. Then the first world cup comes along, and the expectations are huge. Everyone wants to be claimed as the first world champion. You are very excited and try to watch every game you, drooling. The all blacks dont dissapoint and win categorically. History is made.
So there you are, a week after that telling your non-rugby friends what happened: "you HAVE to see these guys, they are amazing, out of this world stuff. Another class. Unbeatable". Suddenly, from across the room a voice says "not quite". Then you are told this story about this not so rich country, who is not allowed to compete for political reasons, a pariah. The problem is, this outcast, this reject, happens to have a positive record against the unquestionable champions. Your first intinct is "not true", "impossible". But you do your best to see if that was true, a not easy task at the time (no internet), trust me, and find out it was actually true. So you ask yourself who are these guys?
Well, those are the ingredients for great storytelling. Talent, political problems, poor vs rich, general lack of reliable info, a hero and a false anti hero who is not allowed to participate. This is pretty much it.
It's as if you were told about dragons, but someone forgot to tell you about dragon slayers.
Basically, the springboks were seen as the team who inspired respect from a team that, results showed, respected no one.
And the more details you found, the more romantic the story became: all countries had sparring partners (aus and nz, 4 nations and france). RSA did not. It was them against the world.
And, to crown it all, when they were allowed to play, they won.
And i know rugby didnt start in the 1980s, but i chose it as 1987 was, whether we like it or not, a pivotal moment in rugby.
Again, I am exaggerating, course. And another thing. Some other teams appeal was there by design. Take the all blacks. The colour, the haka, etc. It was a choice (some say the shirt colour wasnt but that for another thread). The springboks didnt chose what happened to them. They didn't choose to be a top 1-2 team. They didn't choose to play for a country that happened to support apartheid. They just played.
Not particularly happy how this post turned out, but i hope it paints a decent picture and answers the question.
*: just to be absolutely clear, i am not saying apartheid was romantic. I am saying being left out for political reasons adds romanticism to the springboks brand at the time.