Yeah, I know what you mean, I've got a friend who plays rugby (he's the same age as me); he's going to an academy once he finishes school this year, and he's massive! He told me that the guys at his club over here are way bigger than me so that's why I'm not ready to play just yet.
I never played at school, because to be honest, I hated getting wet and muddy. I've since grown a pair and now love the game!
So now I'm just watching the games, taking photos of the local teams and trying to bulk myself up a bit more so I can one day start playing. I need to get some more protein in and figure out some decent workouts to do that.