Wing loading doesn't affect glide (unless the wing is distorted by the weight) you just go down the same glide angle at a higher speed if you are heavier.
Increasing wing loading increases speed by only the square root of the weight change - so on a particular glider if you can double your weight without distorting the wing then your speed will go up by a factor of 1.41...
Which means that adding the maximum ballast allowed under FAI/CIVL rules (about 10kg) can increase your speed by about 5 percent if you are an 80 kg pilot.
In a 4 hour competition that means that a given pilot on a given wing could finish (all other things being equal) 12 minutes earlier by carrying maximum ballast.
But all other things aren't usually equal. The ballasted pilot gets about a 5% increase in sink rate as well as flight speed. For any given turn radius a heavier loaded glider has to bank more steeply, and sink rate in turns increases rapidly with bank angle. So a ballasted pilot looses out in turns from both the direct increase in sink rate and the increase in sink rate that results from the steeper bank angles that are required for any given turn radius. The total effect is roughly proportional to the change in weight.
So if our competition pilot spends one third of the time during that 4 hour task climbing in thermals then by carrying ballast the higher speed on glides saves about 8 minutes, but the total time taken to climb is about 8 minutes longer.
If the conditions allow the task to be completed with only 1/3rd or less of the total time spent climbing then ballast could pay off. If more than about a third of total time is spent climbing then being light on the wing pays off.
But that isn't the whole story. Being heavy on the wing affects stability and agility as well.
A heavily loaded glider is (in general) more resistant to turbulence-induced collapses, but collapses more violently when it does go. This seems to be partly to do with internal pressure (dynamic pressure goes with speed squared, so it increases in proportion to any weight change), and partly to do with the angle of attack of the wing - at any given speed a heavily loaded wing is at a higher angle of attack than a lightly loaded one (to get the lift to balance weight the heavily loaded wing has to operate at a higher lift coefficient). Tucks are (usually) produced when turbulence causes the local angle of attack at the leading edge to go negative. The angle of attack effect seems to be stronger than the internal pressure effect (which is why flying fast (low angle of attack) makes the wing more likely to tuck - IMHO). The reason collapses are more violent on a heavily loaded wing is that the turbulence required to collapse them is more severe, and the speed prior to the collapse may be higher.
A heavily loaded wing is more agile because the pilot has greater control authority - shifting all that weight to one half of a tiny wing will make it bank up quickly....which is why comp wings feel like trucks if you are used to something with a shorter span, and why comp pilots end up wanging all over the sky on beginners wings when they borrow them. There also seems to be something going on with pitch stability and the speed of transmission of information about turbulence so that heavier loaded wings tell you more about what the air is doing - but I don't even have a theory about that.
All this goes to suggest that if you want to float around at the top of the stack all day then you should be light on your wing. If you want to fly cross country or in competitions then you have an optimization problem. If you expect to be racing spending 2/3rds or more of your time gliding then it might be good to be heavily loaded on your wing. This would be particularly true if you expect the thermals to be turbulent (when the extra agility and stability would help you core the thermal while lighter loaded wings get chucked about and thrown out of the core). This would be especially true if you expect to be flying in straight lines in lift a lot i.e. Alpine or desert flying. If however you fly in circumstances where you are likely to spend more than a third of the time climbing in thermals, where thermals are weak, or where climbing at all may be the key then you want to be lightly loaded. Every comp I have been in has had a task where staying in the air in weak lift was the key.
Finally, there is the question of big-glider performance. Big gliders glide better (other things being equal), at least so I am told by some of the best pilots (and designers) around. The only reason I have been given is that it is to do with their higher Reynolds number. Reynolds number depends on velocity, a characteristic length, air density (1.225kg/m3 at sea level) and air viscosity (17.9 x 10-6 kg/m/s at normal temperature and pressure). For a paraglider flying at 10m/s Re is about 106 times wing cord. So at a given speed a larger glider has a larger Reynolds number. As Reynolds number goes up drag goes down......all other things being equal. Going from the Argon 24 to Argon 26 changes wing cord from 2.71 to 2.82m. which changes the Re from 1.85 x 106 to 1.93 x 106 which is about a 4 percent increase. Von Mises reports NACA data for a NACA23012 airfoil which had a drag coefficient of 0.009 at 1.7 x 106 decreasing to 0.0075 at Re = 7.5 x 106...a 17% reduction in drag for a 4 fold increase in Re. so the effect being linear in this range our 4 percent increase in Re should give a 0.17% decrease in drag. Not insignificant, but not exactly huge either. At the same time though, the maximum lift coefficient of the wing increased from 1.4 to 1.6 which would mean that the larger wing can fly slower, or turn tighter before it stalls. Interesting.
SO maybe the reason so many comp pilots can be bothered to carry around huge bags of ballast is that it allows them to fly a bigger glider that glides a little bit better and can fly a little bit slower and turn a little bit tighter in thermals. As far as I can tell there are no disadvantages for big gliders except for carrying them up the hill.
The weight ranges manufacturers quote are almost always the certified weight ranges - the gliders have been tested at the bottom and top of the weight range. If a glider is particularly benign it passes the tests at a wider-than-expected range of weights (as was IIRC the case with the Argon). Beware gliders that have only a narrow weight range!
I spent this year (2000) flying a Nova Argon 24C at the top of its weight range. As expected the wing was stable, handling was fabulous and speed was high, but I had to work to climb with people if thermals were less than 4m/s or so. The advantages of high loading meant that I thoroughly enjoyed the rough air at the squad training camp in Castejon, the British open at Piedrahita (5th) and the Europeans, but I found it more difficult to get away XC in the UK than I did the year before when I was lightly loaded on a Nova X-Ray. Next year I will be changing to a wing where I am near the middle of the weight range, and if I expect to be racing or conditions are good (which is the same) then I will carry maximum ballast.
I've also put a lot of effort into reducing the weight of my flying gear this year, and by changing flying suits, and dumping excess gear I managed to loose 2Kg of useless weight. Weight is only useful as ballast if you can dump it when conditions get weak!