Yep. Setting up a LAN isn't like setting up cable tv. Digital has to be replicated, not split. This is part of what a router does.
The main cable line coming into your house will be connected to your modem. The modem will be connected to a router. The router will then connect to all of your applicable components; some via direct wiring, some wirelessly. It is easier/cheaper to connect wirelessly (no cables to buy/run), but one or more components may have to be directly connected.
I would strongly suggest that you contact your local internet/cable provider, and speak to one of their tech support people. Tell them exactly what you want to do, and they should be able to walk you through the steps.
Best of luck.