Getting faster...Lower weight or fitness - What's more important?

I'm trying to find the original info, but it's 52km with a hell of a lot of up and down

Since I started this thread I've lost 1st 4 lbs by getting out in all weathers, wearing less clothing than I need (to ensure I race along to keep warm), and adding some heavy but free rolling tyres (Crazy Bob's) to increase the workout 8)