Update 2006-04-11:
I actually got back the same error again after rebooting :( I searched for the error and turned off friendly error messages in IE and got the real error instead of HTTP error 500:
The parameter is incorrect.
I didn't find any solutions after a lot of searching, but I eventually got rid of the error. First I found out that you could access the Virtual Server Admin site by using http://127.0.0.1/* and only this worked (not localhost etc). But as you all realise, that is not too useful when you want to access it from another computer. What I did later, which I think is what fixed the problem was to add the vmrc/-entry in servicePrincipalName with the ports after the server. To be sure I added 4 entries, 2 netbios and 2 FQDN with the 2 ports that Virtual Server is using. I guess only one port is relevant, so probably one of them has no effect, but I don't see any real security issue by adding an extra entry. After restarting services, the error messages dissappeared from the Virtual Server event log, but the problem remained when trying to access the Admin Web site. After changing the Security Permissions on the servicePrincipalName, adding Network Service account (Read&Write), and rebooting the server, all problems were fixed. :)
2006-03-20:
I had Virtual Server 2005 R2 installed on a Windows 2003 Server SP1 running in Workgroup mode. I decided to upgrade the server to try out a domain and after that Virtual Server admin page or remote control application did not work. I only got HTTP error 500 when trying to access these applications.
There is a KB about how to fix this, but step 7 is not very clear.
http://support.microsoft.com/default.aspx?scid=kb;en-us;890893
6. In the Value to add box, type vssrvc/Your_Computer_Name, and then click Add.
7. In the Value to add box, type vssrvc/Your_Domain.Your_Domain, click Add, and then click OK.
In step 7, I typed vssrvc/domaincontrollercomputer.mydomain.local (where domaincontrollercomputer is my domain controller on which I have Virtual Server installed and mydomain.local is my FQDN), NOT DOMAIN TWICE as stated in the KB. If you unsure, then look at all the other attribute entries in servicePrincipalName there is probable a few other settings where you find your setting to add after the slash.
After I added these two vssrvc/ attributes, it works fine again.
At the end of the KB-article there is a sample of how to do it with SETSPN command prompt instead of adsiedit.msc. In there it shows how to add vmrc/* as well. I skipped this and my Virtual Server is running fine anyway.
Here is a lot of good information about Virtual PC and Virtual Server if you are in trouble:
http://blogs.msdn.com/virtual_pc_guy/
