Its not just the charging points - its the entire grid around them.
Car batteries as they are, currently aren't removeable. It's just not feasible. There would need to be a very radical design change to make it feasible.
(Something like each cell, or a small group of cells is housed inside a small sphere - or more likely an ovoid, and those spheres are loaded into the car through one hole - like current fuel - and removed from the car through another hole - and inside the car has to line them up with connection points via shape & magnets. Pretty radical compared to current, it'd come with a fairly significant performance penalty - and it still doesn't really sort the infrastructure issue, it just sorts the charge time thing out.)