How is it ridiculous? You don't have to be fluent. But make an effort, like Barnes. Even if you're not naturally linguistic does t mean you can't make an effort. I speak both French and Italian, and I can tell you Barnes' French is nowhere near fluent, but it's the effort that counts. Imagine you were French or Italian, playing against Ireland or England or whoever in a 6 nations match, and you don't understand what the ref is saying, while the other team understands perfectly? Just to have Barnes saying little things in French means the players aren't always translating.
I do understand that the players will obviously know most terms from the ref in english. But it's little things like when the scrum collapses for the 10th time and the referee wants to have a word. Or when he's sin binning someone and a player may not understand why. You saw how much it is appreciated whenever Rolland refereed France. The French were much more vocal in those matches.
In short, in no way should you be fluent, but I think an effort should be made at least.
Edit - listen to what Rolland says here.
http://youtu.be/hGXRjAGe5jA