Purge/reinstalling changed nothing.
With a new account I was able to reproduce the issue only after adding A LOT of icons/folders and fences (actually I have 116 entries in 8 fences).
With my main account I tried msconfig disabling every services and startup apps except for fences, and this changed something, there was still some garbage here and there, but visibly less than before.
The last thing I tried was to uninstall the antivirus (Avast), and after a reboot (normal with full services ad all) I saw even less garbage, just 2 pieces of icons between fences.
But seems still needed a refresh