Windows Server 2008 R2 SP1 install breaks RDP

After installing Service Pack 1 via windows update on a Windows Server 2008 R2 machine the other day, I discovered that I could no longer use remote desktop connection to access the server remotely. At first I wasn’t sure if the whole service pack installation was botched, but further testing revealed that all other functions of this server were operating normally. A quick google search showed that I was not alone in experiencing this problem, in fact there is enough information out there to solve this without another blog post from yours truly, but there is a bit of conflicting information so I thought I would document my experience to hopefully help others.

The problem manifests itself like this:

You try to RDP the affected server and it goes through the enitre logon process, and just when you think it is going to fire up the remote desktop it bombs out. An event was logged in the application log in my case event 4005 with a source of Winlogon, stating ‘The Windows logon process has terminated unexpectedly’ (shown below), although I have read of slightly different errors on other blog posts. Checking the Terminal Services logs indicate that the logon has completed successfully.

Event 4005 Source Winlogon after Service Pack 1 install on Windows Server 2008 R2

This situation it turns out, occurs when both KB2621440 and KB2667402 are applied to a system before Service Pack 1 is applied, as they effectively leave some of the RDP DLL files out of sync, specifically rdpcorekmts.dll. Now, alot of the posts I read online stated that you simply needed to uninstall the KB2777402 update, reboot, and then reapply it to solve this problem. I tried this first and it did not work. Various posts also stated that RDP started working after the removal of this patch, again this was not the case for me although I am sure it is correct in some cases.

So the fix I used in the end was this:

Firstly I uninstalled both KB2621440 and KB2667402 via Control Panel, Uninstall a Program, View Installed Updates, and then rebooted the server. I expected the DLLs to be set back to a working state at this point but this was not the case. Luckily in my case I had physical access to the server. Other posts state that the two updates can be uninstalled remotely using the following commands if you do not have that luxury:

wmic /node:<SERVER> /user:<USER> process call create “powershell wusa /uninstall /kb:2667402 /quiet /forcerestart”

wmic /node:<SERVER> /user:<USER> process call create “powershell wusa /uninstall /kb:2621440 /quiet /forcerestart”

Note, that the commands above will restart the server

As RDP still didn’t work for me at this point contrary to other information, I ran:

sfc /scannow

This picked up some issues and required a reboot. After rebooting the server I was able to use RDP again. That was great, but didn’t help with the fact that the two patches that were removed were to address the Critical RDP vulnerability MS12-020. I certainly didn’t fancy not applying these patches to this server so I reapplied KB2621440 and KB2667402 via Windows Update, and rebooted the server. Thankfully after this I had a fully patched server and working RDP.

If you need to do all of this remotely and find that you still can’t RDP the server after removing the two patches using the commands above I recommend running sfc /scannow using PSEXEC:

psexec \\SERVER sfc /scannow

Then when the scan is finished performing a remote reboot using:

psexec \\SERVER shutdown -r -t 01

This should get your RDP back to a working state, and then you can reapply the removed updates. I have not corfirmed this but expect this fix will also work for Windows 7.

Advertisements

Force reboot of a remote server that has hung shutting down

I had an issue last night when a remote server that I was applying windows updates to, hung while it was shutting down. I will still able to ping the server, and access its file shares, but was unable to get RDP access. I was cursing at this point, with the prospect of a long journey the following day to investigate and bring the server back online. I then considered what other steps I could take to try and force the server to reboot remotely. I used the PSTools command psexec to see if I could still get command line access to the remote server, and fortunately I could. Next I tried to force a reboot of the remote server using psexec and the shutdown command as follows, where REMOTE_SERVER_NAME is the name of the remote server that I was trying to reboot:

psexec \\REMOTE_SERVER_NAME shutdown /r /t 01

alternatively you could use:

shutdown /m \\REMOTE_SERVERNAME /r /t 01

This returned the following error:

1115 A system shutdown is in progress

This basically meant that a system shutdown was already in progress,  and therefore the command was unable to force a reboot. In the end I used the pskill command to stop the winlogon service on the remote server to try and release whichever process wass causing the server to hang on shutdown. I should stress that this was a last resort, and not something that I would recommend doing unless essential:

pskill \\REMOTE_SERVER_NAME  winlogon

Anyway, after another few minutes the remote server did finally restart, although there are a few other things that I should mention that happened in the process. The operating system on this machine was Windows Server 2008 R2. After the server came back up (verified by ping -t REMOTE_SERVER_NAME) I tried to RDP the box again. I was able to enter my credentials and the logon process appeared to start, but after a few seconds the following message appeared on the screen:

Please wait for the Windows Modules Installer

The machine sat like that for quite some time, and then started ‘Configuring Updates’. My RDP session then abruptly ended and the server restarted itself again. Again, when it was back up I tried to RDP the server again and received the ‘Please wait for the windows modules installer message’ for a second time. Thankfully, after a few minutes and another ‘configuring updates’ message, logon continued and ther server was back up and running. On checking the event log and windows update log I was able to verify that all the updates had installed OK, and there were no other errors worthy of note. So in summary, if you want to save yourself a long trip, to most likely press a power or reset switch, you may want to try the above first.