The Ratchet inside could be worn and doesn't catch on the way down. I think it catches on the way up as you are 'easing' it onto the ratchet. On the way down, you are 'dropping it' with more force and slips off the ratchet.
I have the same problem on my Triban (microshift). A mechanic mate managed a temp fix on it, but did say it would go again, and it did.
I can get it to change down from big to middle, by not releasing/keeping the pressure on the lever as I change down. It was bit of a faff at first, but got used to it and works for me.
Hope that's of some help