Yes, it will negatively affect your performance as 64-bit binaries use 64-bit unit types by default rather than 32-bit ones so they'll occupy twice as much memory as the 32-bit versions. However, unless you're doing math-heavy work on a low-memory enviroment (a terrible idea by itself), the effect will likely be less than what you get by leaving your browser open while you play a game, ie: completely and utterly irrelevant.
As for the limitations of 32-bits, due to various things involving low-level OS design a 32-bit OS is limited to addressing 4 GBs of memory max, and from what I've heard Windows uses 512 MBs of those for its own personal uses so in the end its just 3.5 GBs you can effectively use in your Windows OS. To make it worse, that's not just the system's RAM but your graphic card's as well, so if you have a 1 GB video card (as many of us do), that leaves you with a max of 2.5 GBs of system RAM usable by the OS, which is... well, not little, but a bit constraining in the long term. I think it also had to include the swap file (AKA "virtual memory") which would further constrain your computer's memory, but I'm not sure on that one.
Now, as to why you should choose 64-bit versions of software it's simple: Windows (and Linux, and I believe MacOSX as well) has 32-bit versions of most system libraries to allow you to use your 32-bit programs if you so choose, but if you run a 64-bit app and a 32-bit one, the OS will have to load both the 64- and 32-bit versions of the libraries rather than making both apps share the same one, increasing your memory usage. Again, the end effect likely won't be noticeable on a modern system, performance-wise, but Windows pushes 32-bit apps in its own, unsightly-named folder away from pure 64-bit apps and some of us are suckers for cleanliness and order in our OS 
