I did something similar, but instead of fans that aren't running until they're needed, I have fans that basically run slowly and have advanced bearings so that you can't hear them when they're running at their minimum speed. Those slow fans still keep the computer at under 55c under moderate load without increasing their speed. The weakest link is the GPU though, since it's a lot harder to replace the fans for GPUs than CPU and Case fans, so when the GPU is under load it still gets loud.
When I'm running the GPU at full load though, I have my headphones on and can't hear the noise as clearly.
All Noctua fans (80mmx2, 120mm, 200mm) and Noctua heatsink in a Mini-ITX case. I run an 8-core chip at stock speeds, rated at 65w.