To clarify, what we typically call a router is usually two devices in one, a router and a switch.
Regardless of whether you have dialup, dsl, cable, or whatever, you don't need a router; you can always use ICS or a similar program. But, ICS can be flakey and requires that one machine be an always-on server. If you reboot it, turn it off, or if it has other problems, all the other machines lose their network connection.
Routers that work with cable and dsl are ubiquitous and cheap now. Unfortunately, there are far fewer options for routers that will work with dialup. Netgear has the
RM356, but it is ridiculously expensive. I remember Jackmds posted links once to much cheaper options, but I don't remember what they were. If you can find something reasonably priced that worked well, I would recommend that over ICS. I used ICS for a while and it is flakey and it is a pain if something ever happens to the server machine.
As for buying a regular router now and using it only as a switch -- yes, you could do that. You need to do stuff like disable dhcp on the router, i.e. you need to kill the router part of the router. It'll be more hassle but is generally doable. Personally, I would just buy a switch or hub rather than go to the hassle, but you're not talking big bucks either way. Some routers have been going on sale lately at ridiculously low prices, so this could be your cheapest option.