Do I have to learn to ride all over again?

It takes time as road bikes handle very differently than mtbs.
If your stem is to short (under 100mm) that can affect handling but the best advise is to relax your shoulders and elbows then follow so experienced riders to see how quickly you can go.
Once you get used to it you will soon be going much quicker that you ever would on a knobbly beast

I had the same problem when I got my first road bike. You will get used to it, and as a bonus I found my fitne improved and my bike handling as well.

My God, I could have wrote your post back in April, almost word for word. Quite simply it was a matter of confidence (and set-up!).

After my first club run in April, I was left demoralised. I was terrified to take one hand off of the bars because I didn't feel stable at above 16mph. I couldn't drink on the bike, couldn't signal to other club members indicating potholes etc and couldn't indicate at high speeds.

I could catch the riders going up the hills, but on the descents I was doing about 23mph with my hands on the hoods and pressing the brakes. I eventually got up to 42mph on that bike with some confidence and the changes I mention towards the end of the post.

I sought help from my club and one rider kindly offered to help me descend. Outside pedal down, look 20-30 feet ahead of you where you want to be going, and hands in the drops - I had to get used to cycling with my hands in the drops on the flat first, usually in intervals of a few minutes to get my back and neck used to it. I would never descend now on the hoods - the leverage for breaking you get from the drops is fantastic and more importantly promotes confidence. I found out that the frame I was riding was too small for me. The alteration in set-up that I experienced are as follows:

Your bars may be too narrow or wide - they should be shoulder width. I had 44cm width bars and swapped for some 38cm ones. In hindsight I should have gone for 40cm as it's probably too twitchy. It didn't feel like I had my hands on a bus wheel though! I also found it easier to get out of the saddle too. The drops on the bars weren't ergo and also shallower meaning I could reach them easier which I found a lot better - the FSA Compact Omega's are what I got.

The stem length - as a guide, when in the saddle with hands on the hoods, take a look at the front hub. If your bars are obstructing it, then your stem is the correct length. If you can see the hub below the bars, a shorter stem is required and vice versa.

The shifters - I had Sora to begin with. The thumb shifting was shite though and I couldn't shift gears from the drops. I picked up some Ultegra shifters for a bargain £60 on Ebay and that encouraged me to go into the drops.

I'm not going to state that the changes to the set-up have transformed and propelled me into the zenith of bike handlers. It has however - combined with persisting - made me a lot safer and composed on the bike. I descend better, I can flag a pothole at 30mph, and I can drink about 20mph, albeit only freewheeling. I still have a lot to work on, but I'm getting there!

Get out, get used to the bike, get a feel for it and how it reacts. The confidence will come with hours in the saddle. It took me about 1000 miles to feel comfortable.