Linux was designed mainly for operating as web host until they decided to make a competing OS. So Linux is likely to be your best bet.
When installing a forum its mostly automated and they take you through the procedure in steps you might need a text editor of some sort to edit the forum files that you chose to input your database information but once you have done that it will create and format the tables automatically.
Also once you have a valid license for Vbulletin you can post on there forums and they will provide support for you.
Have you looked into IPB also I find it much more user friendly than VBulletin when comparing the two but its completely down to personal preference both of them have plugins such as SEO plugins so you can optimize topics and posts automatically which is great.
Or theres a free option which is very capable (phpbb.com) you could test this out and get a feel for it before actually purchasing a forum software its really up to you.
Hope this helped and good luck!