example: most times, you can mix pc-2700 and pc-3200 memory but they will run at pc-2700. of course, both (or all) sticks are ddr.
you can not mix ddr, ddr2, or ddr3. it's almost impossible to attempt since there are very few mobos that have two different types of memory slots and the few that do allow only one type of memory to be used at a time.
Asrock boards tend to do the side by side ram (DDR2 with DDR3), and their manuals specify that you can only use one or the other, but not both. And then some boards had different limits for different speeds at some point, like 8GB at 2700 but 4GB at 3200. The thing I hate about pre-builts is they rarely tell the customer upfront enough information about the motherboard inside.
I never claimed to be an expert on these things, but I've messed around with different builds (pre-built and custom) to have had this discussion a dozen times with different men and women who build or tinker with machines. There is always some funky ECS board or something that defies the general rule, but I stand by what I said, you can mix and match brands of RAM. Speed, suree, though only real reason to do it is price. Hell at some point even the different brands where just pimping Micron RAM in different skins.
And I still think the OP is running into an unusual situation with his board, not incompatible RAM. Am I 100% sure, no of course I am not. If i had the machine right next to me, I would be trying a few things, but for now I can only make suggestsions based on what I know.