You only see the karma on any one site that was given in the forum categories that appear on that site.
Addendum: for karma earned in a site-spanning category like Off-Topic, you won't see any karma on a specific site until someone gives you karma from that site. Then all your 'non-specific' karma will appear along with any karma from 'local' threads.
For example, one of Brad's nicknames (I forget which) showed 0 karma at Elemental when the site launched last November. But that number instantly jumped much higher after he got his first karma point from someone at the Elemental site. The difference was from his karma for site-spanning topics.
Go for it, you have 0 karma here. And it can't go negative, anyway
Annatar, you know better than to say "here" in a thread like this. There's no there here, or no here there, or something like that...