Windows::Server Performance::Troubleshooting::Citrix

Troubleshooting Server Performance
The discussion of a specific issue below is perhaps useful in a more general sense for troubleshooting and performance monitoring topics.

Problem: After upgrading to Citrix Presentation Server 4.5 a higher average cpu utilization is observed as well as a high rate of context switches. Previously we have often received warnings in Citrix Performance Monitor for %interrupt -- this issue continues and is perhaps seen more often in 4.5 servers as well.

Background: Running PS4.5 using published applications and desktops on a Microsoft Windows 2003 SP2 server on a physical machine. Running several "high maintenance" accounting applications on two PS4.5 as published applications on virtual machines on VMWare Virtual Infrastructure 3.0 cluster. These all exhibit the symtoms above just since the upgrade to 4.5. Also, we are still running 4.0 on several other servers in the same Citrix Farm and various versions of PNA are in use by client machines (predominantly 8.x)

Investigation regarding context switches
A lot of good resources turned up:
Intel: Using Windows Performance Monitor
Sysinternals
www.thomaskoetzing.de
MSDN-Context Switches
Analyzing Processor Activity
Since this issue occurs on both physical and virtual servers it is not a VM problem, but will investigate this avenue as well to ensure correct and optimal configuration.
VMware: improving scalability for Citrix PS
http://redmondmag.com/features/article.asp?editorialsid=718


- definition: CPU's share their time between all threads according to priority. When the CPU stops working on one thread and starts working on another that is a context switch.
- monitoring: A ballpark rule of thumb is "normally" there should be no more than 28000 context switches per CPU on a system.
- What to look for
- Page file - too small, or is allowed to dynamically grow - recommendation: set to larger fixed size.
- Consider write cache on RAID controller
- insufficient hardware
- poorly designed device drivers or applications

Tools
- PerfMon - system/context switches
- SysInternals - Process Explorer - View > select columns > Process Performance > context switches, context switch delta
- pstat.exe (windows resource kit or support tools

VMWare
Some asides that came up during this investigation explained some issues we have had with virtualizing citrix servers. We needed to keep 2 cpu's in the VM after we converted them. That is the opposite of the VMWare recommendations we have seen.
- The multiprocessor HAL had not been downgraded to single processor HAL.
- Hidden devices in device manager had not all been removed.

1. Click Start, click Run, type cmd.exe, and then press ENTER.
2. Type set devmgr_show_nonpresent_devices=1, and then press ENTER.
3. Type Start DEVMGMT.MSC, and then press ENTER.
4. Click View, and then click Show Hidden Devices.
5. Expand the Network Adapters tree.
6. Right-click the dimmed network adapter, and then click Uninstall
uninstall any other physical devices not needed

Investigation
- Interesting - on the VM servers when looking at Task Manager the %cpu listed individually for all the processes for all users did not appear to add up to what was showing up on the Performance tab (at least 50% discrepency.) This was not observed on the physical server
- For both VM's and physical servers: Citrix Performance Monitor was showing warnings and intermittent error conditions on %cpu, %interrupt, context switches/sec.
- The VM's cpu utilization on the host machine is extremely high. On the server with the greatest number of users it maxed out the host cpu for much of the time I watched it.
- Watching performance monitor a few minutes showed context switches/sec to be in the hundreds of thousands.
- Opened Process Explorer and set view to show context switches and context switch deltas. I observed that at times it reported up to 50% cpu was due to hardware interrupts (this was not as dramatic when I checked it on the physical machine so I wonder if this is a reporting issue related to vmware's magic behind the scenes.) Also, the highest context switch delta was for hardware interrupts so Process Explorer was no help to further isolate it.
- To isolate what driver or program might be causing this issue, I piped the output of pstat.exe to a file and looked for the highest count of context switches. I took the memory address of that thread and looked it up in the bottom section to find what address range it fell in. In this case it was CDM.SYS
- google search of CDM.SYS turned up multiple articles about Citrix servers. I think CDM stands for Client Data Mapper. Of greatest interest is an article about a hotfix for PS4.5:
http://support.citrix.com/article/CTX114121 (and I see a lot of other post FR1 hotfixes out there too.)
The issue resolved in this hotfix is:

"Winlogon.exe shows higher than average CPU consumption on the server. The issue occurs because the server refreshes the smart card reader state more frequently than necessary. This occurs even if smart cards are not being used. With this fix, the reader state is refreshed only once per noticeable event."

Citrix::ICA::Session Sharing

Troubleshooting and Explaining Session Sharing
CTX159159

Understanding and Troubleshooting Citrix's Seamless Windows Engine
BrianMadden.com

Citrix::Program Neighborhood Agent::Rename Application

Good to know...
If you rename an application in Citrix Management Console -
during the time that PNA has not refreshed on the clients, if they double click their existing icon they get an error that the program is not available.

Citrix::Troubleshooting::Utilities

Citrix Guide > Tools

Windows Terminal Server::Outlook 2002::Safe Mode

SYMPTOMS
========

When using Outlook 2002 on a Windows 2000 terminal server, you may receive the following error:

Outlook failed to start correctly last time. Starting Outlook in safe mode
will help you correct or isolate a startup problem in order to successfully
start the program. Some functionality may be disabled in this mode.

Do you want to start Outlook in safe mode?

You can click either Yes or No and Outlook opens. The next time you launch Outlook you are not given the prompt. However, at the next launch of Outlook you are given the prompt. On the following logon, you are not given the prompt. This loop of Outlook 2002 prompting for Safe Mode every other time you launch Outlook continues indefinitely.

CAUSE
=====

HKEY_LOCAL_MACHINE\Windows NT\Current Version\Terminal Server\Install\Software\Microsoft\Office\10.0\Outlook\Resiliency with the key TermSrvCopyKeyOnce with a value of 0x1 exists in the Registry on the Windows 2000 Terminal Server in one of the subkeys under Resiliency.

When Outlook 2002 starts, the following Registry key is created:

HKEY_CURRENT_USER\Software\Microsoft\Office\10.0\Outlook\Resiliency.

NOTE: The Resiliency registry key for Outlook is new for Outlook 2002.

If no problems are encountered, the Resiliency key is deleted. If one of the many tests that Outlook makes at startup fail, subkeys are written under the Resiliency key and the Resiliency key is not deleted. If the key remains, the next start of Outlook 2002 gives the prompt.

If a subkey is deleted by an application from HKEY_CURRENT_USER (HKCU) -- specifically a subkey from HKEY_CURRENT_USER\Software\Microsoft\Office\10.0\Outlook\Resiliency while the server is in Terminal Server Install mode, the data is copied to HKEY_LOCAL_MACHINE (HKLM):

HKLM\Software\Microsoft\Windows NT\Current Version\Terminal Server\Install\Software\\Microsoft\Office\10.0\Outlook\Resiliency

When the application attempts with the server in Terminal Server Application Mode to access the key again and finds it missing, the application looks in this location (HKLM) for the key. If the Resiliency key is there and has the key TermSrvCopyKeyOnce under one of the subkeys of the Resiliency key, the Resiliency key is written back to HKCU.

In this case, the Resiliency key was written with subkeys while in install mode. Once the terminal server goes back to application mode every user is affected since Outlook 2002 always looks for the Resiliency key, thus causing the loop.

RESOLUTION
==========

Deleting HKLM\Software\Microsoft\Windows NT\Current Version\Terminal Server\Install\Software\Microsoft\Office\10.0\Outlook\Resiliency resolves the problem.

How to disable the AutoArchive feature and the Personal Folders file feature in Outlook

!Do this on Citrix servers!

Inside Citrix Presentation Server's Application Isolation Environments

How To: Unattended Installation of the Citrix Web Interface: "How To: Unattended Installation of the Citrix Web Interface"

CITRIX::VMWARE

BriForum 2006 Video: Ron Oglesby on the feasibility of running Citrix in a VM: "Video: Ron Oglesby on the feasibility of running Citrix in a VM"

Physical Address Extension - PAE Memory and Windows: "Physical Address Extension - PAE Memory and Windows
Updated: February 9, 2005"
Fixing the 4GB memory limit issue

ThomasKoetzing.de - Understanding and Troubleshooting Citrix's Seamless Windows Engine: "This article will explain how these technical components work and how you can fix seamless windows problems "

ICA Client Printer Creation

Citrix White Paper
http://support.citrix.com/servlet/KbServlet/download/584-102-8107/icaprinting.pdf

CTX681954 - Troubleshooting Citrix ICA Printer Auto-creation: "Troubleshooting Citrix ICA Printer Auto-creation "

Wfshell.exe crashes - Brian Madden's Community Forum: "Grimace Not-So-Newbie


Posts: 25
Score: 0
Joined: 3/17/2005
Status: offline This issue has been addressed in a Citrix Hotfix Rollo-up:
PSE400W2KR01

http://support.citrix.com/article/CTX107504

_____________________________

La Grimace

(in reply to Joe McGaugh)
Post #: 8


RE: Wfshell.exe crashes - 1/24/2006 7:59:23 AM


Joe McGaugh
Master Yoda





Posts: 676
Score: 13
Joined: 4/15/2005
From: Florida
Status: offline Yes, this roll-up addresses the issue, but I have not seen it Resolve the issue in all instances. If you don't do the necessary maintenance as Jeff and I both have talked about, your putting a bandaid on the issue. Clean the servers and remove all the native drivers first, set printing policies for UDP and do not allow native drivers to install automatically, and if your still having an issue, then try the patch.

_____________________________

Joe "

wfshell.exe error

From one kind soul:

Duplicate Problem by terminating application by clicking on "X". Log back in and you should see the wfshell error.

Resolution -

In the "CCC" under advanced setting, change the configuration from "reset" on a broken connection to disconnect. After this configuration is made, log off and log back on and terminate application by clicking on the "X" again. After you log back on, the wfshell error should not be there nor in the eventlog anymore. You can modify the idle timeout settings for users to control the number of active idle session on your Citrix Servers. Hope this helps and it is a relief to finally find the cause of this error.

CDM Redirector Timeout Event ID 70 in MetaFrame 1.8 :: Tokeshi.com :: Citrix Support :: Thin Client Support :: Terminal Server Help

In the Events Viewer, one might find a varied number of Event ID 70 in Source: CDM Redirector. This occurs when the CDM Redirector timed out on a request to access a client device.

This can happen when an application is attempting to access a client device during a disconnect. Check Citrix Server Administration when you see this event appear. The best way to determine what client device is causing this issue is to turn off client device mapping (client drive, client COM ports, client printers). Enable them one at a time and monitor the client's logon process.

If you find that you are receiving these timeouts due to network latency, you can modify the timeout values in the registry key shown below:

WARNING: Using Registry Editor incorrectly can cause serious problems that may require you to reinstall your operating system. Microsoft cannot guarantee that problems resulting from the incorrect use of Registry Editor can be solved. Use Registry Editor at your own risk.

For information about how to edit the registry, view the "Changing Keys and Values" Help topic in Registry Editor (Regedit.exe) or the "Add and Delete Information in the Registry" and "Edit Registry Data" Help topics in Regedt32.exe. Make sure you back up the registry before you edit it. If you are running Windows NT, be sure to update your Emergency Repair Disk (ERD).

The registry key is:

HKLM\SYSTEM\CurrentControlSet\Services\Cdm\Parameters

Increase the settings for the following:
RequestTimeout: Default is 60 (in seconds)
DirCacheTimeout Default is 30 seconds
CacheTimeout is 60 seconds

Citrix::Performance Metrics

Percent Interrupt Time
The processor is spending a large amount of time responding to I/O rather than user processing. A large value for interrupt time indicates a hardware problem or a very busy server.

Context Switches per Second
A large # of threads and/or processes are competing for processor time.

Windows::Citrix::Event Log

Event 10016 caused by COM component permissions or failure of service account:

http://support.microsoft.com/default.aspx?scid=kb;en-us;899965&sd=ee