You may find more benefit from setting them up than changing them. Toe angle Check, alignment with the rim Check, clearance from the rim Check etc. And cable quality/condition.
It can make a fair sized difference, yup. If there is less friction on the cable run then it'll feel easier to squeeze the brakes. If the outer is uncompressible, then less of your squeeze is wasted compressing the outer....
To be clear though, your 105 brake calipers, they are the current 5800 model and are being used with 11 speed 105 levers?
Mixing and matching can give poor results. One of my bikes I'm using 5800 levers with some old 105 calipers (9 speed vintage) and the pull ratios have changed so although the brakes come on quickly I have to squeeze really hard to get effective braking.
I have 105 5800 calipers with Swisstop BXP's and they are fantastic - mostly down to the BXP's being awesome, but they're also better than the 5700's I had on the old bike..