Ok, yeah, Win+Space, I used that 99% of the time. Sometimes I'd just click on the desktop - because I had the hide setting enabled.
So you're suggesting that the issue only occurs (or occurs most often) when the Peek feature is used to bring forward Fences? For me, I'm pretty sure my click-on-desktop method also had the delay, but I can't be absolutely certain.
I did notice that I can't (with 5.04) get Win+Space to bring Fences "on-top". I thought that used to work.
Anyway, next time I'm scheduled for a reboot I'll try the out-of-band build just announced.