Help with Cervelo S3 2020 stem and handlebars

Hi, first time poster, long time reader here.

I need some help with my Cervelo S3 stem / handlebar adjustment. Its literally driving me insane.

I bought the bike this week and am trying to remove some spacers to bring the stem down a bit and get it dialled in. This is proving to be very difficult. I have loosened the screws inside the stem and taken the spacers off but now when I tighten the screws again the fork wont move and the stem is a bit loose.

Any tips on how to fix this?