I had to write a 30 page paper on this a few years back, so I'll lend my opinion. I'm from the US, but I've got a father who's from Liverpool, and we're all massive Everton supporters. I've been to the UK and Australia quite a few times for good periods of time to see family as well, so I'm not writing from a totally American perspective.
First, to explain soccer's atmosphere, you need to look at the game itself. Compared to sports like rugby, it's a lot harder to see whats coming. Soccer can be a long stream of passes followed by a shot or breakaway out of nowhere. Rugby has this type of thing too, no doubt, but a sudden breakthrough in a gap is a lot more rare in rugby than it is in soccer. This kind of constant anticipation in soccer lends itself better to getting a crowd riled up. A good analogy is running -- rugby is constant cardio, whereas football is cardio broken up by sudden sprints. Obviously there are always exceptions to this rule, but for the most part, football has a lot more anticipation and unknown than sports like rugby (in rugby you can drive the ball down the field in phase after phase, in football it only takes seconds).
The second thing you need to look at is the origins of football, specifically when the modern game came into existence in the late 19th/early 20th century. This is a point in time where the class argument cannot be negated. With football growing in popularity and largely becoming a sport of the working class, football clubs became more than just football clubs. Each club took on a distinctive class, regional, or religious identity, and represented the community of that identity. For example, Everton was (in the early 20th century, not so much now) considered the working class Irish team of Liverpool, Celtic was the Catholic team of Glasgow, Tottenham was a Jewish North London team, and so on. However, as you moved into the 1900s, these identities still existed, but failed to result in mass football violence until the later half of the 20th century, as British culture was "distracted" during the first half -- World Wars, various political movements, and so on, all prevented Britain's culture from "needing" soccer violence.
At this point I'd just like to clarify that I'm focusing specifically on English speaking countries here. I haven't looked into places like South America enough to explain why they see violence as well, so I'm not going to try to.
But anyway, fast forward to post second world war and Britain was changing rapidly. Old social norms were being destroyed. The traditional class structure was crumbling. The Empire was collapsing, and with it Britain's sense of national pride and identity. Immigration to Britain rose, the country liberalized, and suddenly, the formerly clear definition of what was "British" was being called into question. Despite all these changes, one thing stayed the same -- football clubs. Even with the world changing around you, your football club was still your football club, and as such it became an outlet to express frustration. Social movements got mixed into football support, leading to the rise in hooliganism. A notable example is the skinhead movement, which became intertwined with football support (I'm talking about both the racist and non-racist skinheads here).
Going in to the 70s, 80s, and 90s, the collapse of the collective consensus in British politics, turn towards a neo-liberal economy, and further changes in society led to further frustration, and therefore more hooliganism. People saw football as a place to express their "Britishness" and working class identity, whether it was real or perceived. There's some irony to this as well -- although the majority of hooligans saw themselves as working class and "real" British, many of them held steady jobs, had disposable incomes, and some of the hooligan leaders were actually pretty well off and even non-white (Bill Bufford noted that some of the Man United firm leadership was black in his book among the thugs -- ironic because of the "Britishness" many of the hooligans wished to preserve). When all these factors came together, soccer became an outlet for violent frustration, only tapering off in the 90s and 2000s with a mass government response and negative response from society.
But as other people pointed out, why, in places like Australia, do we see violence without any sort of class component? Well, the answer to that lies in mass-media. If you took a poll and asked these "hooligans" if they had seen football movies like Green Street, Football Factory, the Firm, and so on, I bet most of them would say yes. Through movies and the news media, football support became associated with hooliganism, and people in places such as Australia began to imitate the stereotypical British model. And there is a class component, if you ask me. But it's not the fault of the working class. Many of these middle class hooligans want to express how tough and "working class" they are, and try to do it through hooliganism. So there is a class problem, but it's from the middle class down, not the working class up.
As to the question of why rugby support does/doesn't see the same type of violence, dependent on where you are, if you can answer that I know a few sociology professors who would love to talk to you. As others have pointed out, however, the idea of rugby being this "good ol' boys" sport with a code of honour is completely false in certain places -- somebody mentioned some New Zealand players had the mentality that they were going to rough up the "posh" British players. If you look at the incidents of where violence does pop up in rugby, I bet you can almost always find deeper reason for the violence than the sport itself. It can be racial problems, traditional regional rivalries extended into rugby, all sorts of things (or, in the example above, a perceived class divide between British and New Zealand rugby).
In conclusion, football has a long history in which outside factors caused it to become and outlet for violence. If you ask me, for the most part, sports support is not inherently violent. As with any situation where you assemble a crowd, there can always be violence, and things that happen on the pitch can cause violence off of it, but for any mass movement of violence to occur, you need factors exterior to the sport that are expressed within it. Using that model you can usually explain the majority of sports violence.
If anyone disagrees please do so. I'm interested to hear what other people think.