There are environmental costs to purifying tap water too. Not to forget the toxic materials necessary to pump and deliver the water to the tap.
Moreover the problems you raise are more with the bottles themselves and how we dispose of them than with the concept of bottled water per se. Biodegradable corn starch ‘polymers’ or a more responsible attitude to recycling would also solve the problem.

This doesn’t apply to most Western countries though and people there could easily rely on tap water for their needs.

Sometimes I am prepared to pay ‘up to 1,000 times the price of tap water’ for the convience of not having to carry water around with me. I can just buy a bottle when I need it and dispose of it when I’m finished. I’m prepared to pay a premuim for that.

The proposition forgets that most bottled waters marketed in this country are "mineral" or "spring" waters. For a water to possess the name of "mineral" or "spring" water, it must come from a spring in the ground and be bottled at source. Waters such as Dasani are unpopular – Dasani itself was itself completely withdrawn from sale in the UK. The bottled waters popular in this country, i.e. Evian, Volvic, San Pellegrino, Buxton, all come from natural springs and contain natural elements, such as iron, potassium etc, which are all necessary for the human body to function healthily. Often, some natural flourides occur in mineral water. Fluorides are only sometimes artificially added to tap waters, and in large quantities this has the undesired effect of staining teeth instead of improving their health. Tap water is often highly chlorinated. Bottled water does have a health benefit and is better than tap water.

In the UK the terms ‘natural mineral water’ and ‘spring water’ denote water from an aquifer or underground source, with rules on hygiene and mineral content. Similar laws on how you describe your product exist in the US. If you’re stupid enough to pay a premium for something which just says ‘water, product of the Coca-Cola company’ then you probably deserve getting the hefty dose of bromide that came free in bottles of Dasani.
More importantly, whether or not the bottled water comes from the tap, there is still the convenience of being able to purchase, albeit at a premium, water without having to carry a bottle around with you. Moreover if you were, say, travelling in India, or any similar area, the very process of filtering the tap water is rather useful given the reputation of the country’s ‘potable’ water supply.

Just because the quality of something is high does not mean that we should not have the right to drink it if we so wish. If we banned bottled water because "we don’t need it" where would this lead? We don’t need toasters – we can make fire. We don’t need washing machines because we can use the river. We don’t need cars because we’ve got legs. Banning bottled water would start an irreversible trend of banning that which it can be argued we don’t need.