Issues (mostly) solved.
Got one of our engineering guys to come in and look at the issue. It was the sample rate setting.
He asked me if the Sample Rate section was the front End Sample Rate or the Back End Sample Rate. Seems from what made it work it is the Back End Sample Rate; with no Front End Sample Rate?

I had no idea.
I started this testing using the audio off the fibre here and that is at 48kHz so the 192kHz would have been appropriate but when I moved the setup I used a patch cable and took a 44.1 audio feed and still had sample rate in ST at 192kHz. He told me to change it to 176.4kHz and it worked without any buffer issues at all, even with much smaller buffer times.
When I selected Composite Clipper the buffer dropped all the way down and the sound started to slow down. I turned off Advanced Clipper and the buffer restored but the output level dropped. When I turned off the Advanced Clipper I did notice the Composite Clipper went off too.
So with no Composite Clipper it all worked perfectly. The CPU still only reached a maximum of 30% utilisation with everything on.
Thanks to those who offered suggestions. I tried everything suggested. Now to chase down the Composite Clipper buffer issue.
Clipper