If a rugby player plays badly, they should be dropped. If a politican messes up the country, they get voted out. If a worker doesn't follow work safety rules, they get warned/fired. To an extreme, doctors/surgeons can face courts if they're negligent in their practise. So why should officiating matches be the only job in the world where no-one gives one about your performance? If you do badly in your job, what right do you have to keep it?

And I'm not saying that the result should be changed, that referees aren't allowed to make mistakes, etc. Just that if their overall performance isn't up to scratch, better referees should be appointed to officiate and people should feel hard done by if a referee makes numerous errors in officiating. It implies they haven't the ability to keep up with the game, which means they shouldn't have the job and shouldn't have been appointed to begin with.