I experienced many random connection problems trying to reach two Windows 2003 R2 64bit using Network Load Balancing Service.
I analyzed the connection timeout behavior using Network Monitor, when I was able to connect, I could see the TCP three-way handshake, when I got timeout, it returned TCP Reset.
I do a lot of testing but none of the following works:
- change cable
- upgrade NIC Firrmware
- reinstall drivers
- reset TCP stack
After investigating with Microsoft Support, I disabled RSS (Receive Side Scaling) and everything worked properly !
Receive Side Scaling enables parallelized processing of received packets on multiple processors.
I also found that other people experienced Receive Side Scaling problems on ISA Server.