You can read about dedicated servers at this wikipedia article:
http://en.wikipedia.org/wiki/Dedicated_hosting_service
From my understanding, a dedicated server is an individual computer that only you use to run your website(s). I've never heard of leasing one for $2/month, generally the lowest I've seen is around $100/month (this is for an average configured Microsoft software server).
However, I've had some experience with dedicated hosting as the company I work for during the day has some dedicated servers that I assist in 'administering'. We are an all Microsoft shop (which means basically no Linux, MySQL, or PHP), but instead (IIS, Sql Server, and ASP.NET). The host we use is 1and1.com and they've been pretty good. You can get a good dedicated server that will run your website(s) for a long time for around $100 - $150/month. We've been up 100% of the time since we got the server six months ago.
However, to be honest, a dedicated server is only necessary if you are having any of the following:
* insane amounts of traffic
* mission critical website pages that cannot go down
* the need to manage programs that run on your server
* the need to directly access the server and configure it
If you just have a regular site, you can get away with a $5 to $25 dollar a month host and be just fine.