BizTalk and DTC Cluster Buster

We have been installing Web Servers and BizTalk 2004 servers against a SQL 2000 cluster and it was not the easiest thing to do. Here are some tips which helped us solving our DTC problems.

- Make sure DTC is installed on the servers (Add/remove programs - Windows components)
- Go into Component services/Computers/My Computer -> Properties - MSDTC tab and enable Network transactions in the security configuration (Check the values in the settings)
- Disable RPC security on all machines involved:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSDTC
Add DWORD TurnOffRpcSecurity and change the value to 1
Restart SQL services and DTC services on the machines involved
- Enable NETBIOS Across All Machines. Alternatively, you can add entries in the HOSTS file (c:\windows\system32\drivers\etc) so that the machines can be pinged by server name.

If you change the logon userid for the DTC service, the change is not always complete. There are 3 places where the userid is stored! If you change it only in services you may end up with a broken DTC. I believe the best way to change it is in the options under Component services. If however you did this the wrong way there is also a registry entry with the userid which you should change as well. If you end up with a broken DTC (red cross in Component services and/or errors in the event viewer), you may have to uninstall the DTC service and reinstall it. Uncheck it in Add/remove programs and try these commands:
msdtc.exe -uninstall
msdtc.exe -install

You have to reboot when you uninstall the DTC. I have also read that there is a time-out of 15 minutes involved with the DTC so sometimes you have to reboot and wait for a while before continuing.

If you have firewalls involved you may have to use fixed ports for the DTC service as well. This setting is found in Component services/Computers/My Computer -> Properties - Default protocols tab and choose TCP/ip and properties. Add a range in there. Depending on your traffic you might want more ports. We used 5000-5200. There is also a port 1861 involved, which we had to enable in the firewall.

To check that the DTC is working properly the DTCTester tool is very useful. You can download it here:
http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q293799

There is also a DTC Ping tool available at MS downloads, but not as useful as the above DTCTester, since it was giving me "Everything OK", while the DTCTester found the connection errors. It might be helpful as well though. Here it is:
http://support.microsoft.com/default.aspx?scid=kb;en-us;306843

Here is another trouble-shooting link on how to re-install MSDTC:
http://support.microsoft.com/default.aspx?scid=kb;en-us;873160

Another problem that we encountered was that the cluster could not start the MSDTC service, because the default behaviour in a cluster was to start services in the wrong order. We had to change the order to start the DTC service before SQL and SQL agent. We also installed the COM+ 1.5 rollup package 3 (883955), which should fix this. This is probably part of Windows 2003 SP1, but it might be worth installing on non-SP1 systems if you have problems.

Some other useful links, which might help as well:
http://support.microsoft.com/default.aspx?scid=kb;en-us;243204
http://support.microsoft.com/default.aspx?scid=kb;en-us;279786

Happy clustering!
Niklas