NetScaler Console 14.1 – Citrix ADM 13.1

Last Modified: Apr 30, 2024 @ 10:20 am

Navigation

In early 2024, NetScaler renamed Application Delivery Management (ADM) to NetScaler Console.

This post is for versions NetScaler Console 14.1 through Citrix ADM 13.1.

💡 = Recently Updated

Change Log

Planning

Why NetScaler Console?

The biggest change in ADM 13.1 and newer is the restructured menu.

NetScaler Console (formerly ADM) enables every NetScaler administrator to achieve the following:

  • Alert notifications – Receive email alerts whenever something goes down. For example, if a Load Balancing service goes down, you can receive an email alert.
    • NetScaler Console can email you for any SNMP trap produced by any NetScaler ADC appliance.
  • Automatically backup all NetScaler ADC instances.
    • NetScaler Console can even transfer the backups to an external system, which is then backed up by a normal backup tool.
  • SSL Certificate Expiration – Alert you when SSL certificates are about to expire.
    • Show you all SSL certificates across all NetScaler ADC appliances.
  • Configuration Record and Play – Use the Configuration Recorder to configure one NetScaler ADC appliance, and then push out the same configuration changes to additional appliances. This is the easiest method of managing NetScaler ADC appliances in multiple datacenters.
  • AppFlow Reporting – Receive ICA AppFlow traffic from NetScaler Gateway and show it in graphs.
    • Integrate NetScaler Console with Citrix Director so Help Desk can see the AppFlow data.

Everything listed above is completely free, so there’s no reason not to deploy NetScaler Console.

NetScaler Console Overview

For an overview of NetScaler Console, see Citrix’s YouTube video Citrix NetScaler MAS: Application visibility and control in the cloud.

Citrix Tech Zone Citrix Application Delivery Management (ADM) Overview Cheat Sheet

Cloud vs on-prem

NetScaler Console is available both on-premises and as a Cloud Service.

The Cloud version of NetScaler Console has new features that are not available in the on-premises version of NetScaler Console.

For the Cloud Service, you import a NetScaler Console Agent appliance to an on-prem hypervisor or deploy a NetScaler Console Agent to AWS or Azure. The NetScaler Console Agent is the proxy between the Cloud Service and the on-prem (or cloud hosted) NetScaler ADC appliances. For more info on the NetScaler Console Cloud Service, see the following:

The rest of this article focuses on the on-premises version, but much of it also applies to the Cloud Service.

On-premises NetScaler Console Licensing:

  • Instance management is free (unlimited). This includes Configuration Jobs, Instance Backups, Network Functions/Reporting. Basically everything in the Infrastructure node is free.
  • Analytics and Application monitoring are free in NetScaler Console build 21 and newer.

NetScaler Console version – The version/build of NetScaler Console must be the same or newer than the version/build of the NetScaler ADC appliances being monitored. NetScaler Console 14.1 can monitor many NetScaler ADC appliance versions including version 11.1, version 12.1, version 13.0, version 13.1, and version 14.1.

HDX Insight

See CTX239748 for a list of HDX Insight Quality Improvements in Citrix Gateway 12.1 and newer. These include:

  • NSAP protocol for reduced performance impact on NetScaler ADC
  • EDT support

HDX Insight Requirements (aka AppFlow Analytics for Citrix ICA traffic):

  • Your NetScaler ADC appliance must be running Advanced Edition or Premium Edition.
  • For EDT (UDP-based ICA), NetScaler ADC must be 12.1 build 49 or newer.
  • AppFlow statistics are only generated when ICA traffic flows through a Citrix Gateway. Internally, when a user clicks an icon from StoreFront, an ICA connection is established directly from Workspace app to the VDA, thus bypassing the internal NetScaler Gateway. Here are some methods of getting ICA traffic to flow through an internal NetScaler ADC:
    • Implement NetScaler Gateway ICA Proxy (SSL) internally.
    • Route ICA traffic (TCP/UDP 1494 and TCP/UDP 2598) through a NetScaler ADC SNIP, and NetScaler ADC routes it to the VDAs.
    • NetScaler ADC can proxy ICA traffic through a SOCKS protocol Cache Redirection Virtual Server.
    • NetScaler Docs Enabling HDX Insight Data Collection details additional ICA routing/proxy considerations – Transparent Mode, Citrix Gateway Single-Hop and Double-Hop, LAN User Mode (NetScaler ADC as SOCKS Proxy), Multi-Hop (NetScaler ADC connection chaining)
  • A new Workspace app Virtual Channel named NetScaler App Experience or NSAP can dramatically reduce the CPU needed on the NetScaler ADC to process AppFlow. Details at Citrix Blog Post HDX Insight 2.0.
  • For ICA round trip time calculations, in a Citrix Policy, enable the following settings:
    • ICA > End User Monitoring > ICA Round Trip Calculation
    • ICA > End User Monitoring > ICA Round Trip Calculation Interval
    • ICA > End User Monitoring > ICA Round Trip Calculation for Idle Connections
  • Citrix CTX215130 HDX Insight Diagnostics and Troubleshooting Guide contains the following contents:
    • Introduction
    • Prerequisites for Configuring HDX Insight
    • Troubleshooting
      • Issues Related to ICA parsing
      • Error Counter details
    • Checklist before Contacting Citrix Technical Support
    • Information to collect before Contacting Citrix Technical support
    • Known Issues

Citrix CTX204274 How ICA RTT is calculated on NetScaler Insight: ICA RTT constitutes the actual application delay. ICA_RTT = 1 + 2 + 3 + 4 +5 +6:

  1. Client OS introduced delay
  2. Client to NS introduced network delay (Wan Latency)
  3. NS introduced delay in processing client to NS traffic (Client Side Device Latency)
  4. NS introduced delay in processing NS to Server (XA/XD) traffic (Server Side Device Latency)
  5. NS to Server network delay (DC Latency)
  6. Server (XA/XD) OS introduced delay (Host Delay)

Multi-Datacenter Deployment Architecture

In a main datacenter, import two NetScaler Console appliances into the same subnet and configure them as an HA pair with a Floating IP address.

In a DR datacenter, import a DR node NetScaler Console appliance and configure it to replicate with the main datacenter.

  • Note: DR node requires a Floating IP, which requires NetScaler Console HA to be configured in the main datacenter.
  • Documentation at Configure disaster recovery for high availability at NetScaler Docs and will be detailed later in this article.

For NetScaler ADC appliances in additional datacenters, import two NetScaler Console Agent appliances into each datacenter. Remote NetScaler ADC instances are discovered and managed through remote NetScaler Console agents.

Import NetScaler Console Appliance

If you are upgrading an existing NetScaler Console or ADM, skip to the Upgrade section.

There are two different NetScaler Console appliances:

  • ADM appliance for the main datacenter, including High Availability, and for the DR node.
  • ADM Agent appliance for remote datacenters

To import a NetScaler Console Appliance into vSphere, do the following:

  1. Download Citrix ADM Image for ESX.

    • The download page for NetScaler Console has two different images: one called ADM Image, and one called ADM Agent Image. The first image should be the non-agent image.
  2. Extract the downloaded .zip file for the non-agent image (MAS-ESX-14.1).
  3. In vSphere Web Client, right-click a cluster, and click Deploy OVF Template.
  4. In the Select an OVF Template page, select Local file and browse to the NetScaler Console .ovf files. If .ova file is available, then only select the one .ova file. Otherwise, select all three files (.ovf, .mf, and .vmdk). Click Next.

  5. In the Select name and folder page, enter a name for the virtual machine, and select an inventory folder. Then click Next.
  6. In the Select a resource page, select a cluster or resource pool, and click Next.
  7. In the Review details page, click Next.
  8. In the Select storage page, select a datastore. Due to high IOPS requirement, SSD is recommended.
  9. Change the virtual disk format to Thin Provision. Click Next.
  10. In the Select networks page, choose a valid port group, and click Finish.
  11. In the Ready to Complete page, click Finish.

Appliance Hardware Configuration

  1. Before powering on the appliance, you can review its hardware specs. Right-click the NetScaler Console virtual machine and click Edit Settings.
  2. Review the specs. NetScaler Docs NetScaler Console on VMware ESXi recommends 8 vCPUs and 32 GB RAM.
  3. You can add a second hard disk at this time.
  4. NetScaler Docs Attach an additional disk to NetScaler Console says that an additional disk must be added before initial deployment.
    • Use the NetScaler Console storage calculator to determine the recommended size of the disk. Ask your Citrix Partner for the tool.
    • The new disk must be larger than 120 GB.
    • The new disk can be larger than 2 TB.
    • The new disk can be grown later, and /mps/DiskPartitionTool.py can resize the partition, but only up to 2 TB. If you need more than 2 TB, the initial disk should be larger than 2 TB.
  5. Power on the Virtual Machine.

Appliance IP Address Configuration

  1. Open the console of the virtual machine.
  2. Configure IP address information.
  3. Enter 7 when done.

Second Disk

  1. SSH to the appliance and login as nsrecover/nsroot.
  2. Enter /mps/DiskPartitionTool.py

  3. Enter info to see that there are no existing partitions on the second disk.
  4. Enter create to create partitions on the second disk. A reboot is required.
  5. During the reboot, the database is moved to the second disk.
  6. After the reboot, the Disk Partition Tool info command shows the partition on the second disk.
  7. If you need to increase the size of the disk, reboot the NetScaler Console appliance so it detects the larger size. Then use the Disk Partition Tool resize command.

Deployment Modes

HA Pair in the Main Datacenter

If NetScaler Console 14.1 build 17 or newer, HA is no longer configured from the CLI. Instead, use the GUI.

  1. Latency to the HA node must not exceed 10 ms.
  2. The HA nodes must be on the same subnet.
  3. Import a second NetScaler Console appliance.
  4. If you added a second disk to the first NetScaler Console appliance, then you must add the same size second disk to the second NetScaler Console appliance.
  5. Configure the new node’s IP address.
  6. SSH to the second appliance, login as nsrecover/nsroot, and run the Disk Partition tool.
  7. Point your browser using https to the first NetScaler Console IP address.
  8. Login using nsroot/nsroot credentials.
  9. Change the nsroot password when prompted.
  10. Click Get Started.
  11. In the Instances page, click Next.
  12. In the Connect to NetScaler Console Service page, click Skip.
  13. In the Notifications page, click Skip.
  14. Click Finish.
  15. In the left menu, expand Settings and click Administration.
  16. On the right, click IP Address, Second NIC, Host Name and Proxy Server.
  17. Configure the Alternate DNS and click Save. You can only configure this before you create the HA pair. Repeat on both nodes.
  18. On the right, click Configure NetScaler Console High Availability (HA).
  19. Enter the IP of the second NetScaler Console appliance.
  20. Enter a new IP that will float between the two nodes. Click Configure.
  21. Click Yes to reboot.
  22. It will take several minutes to configure HA.
  23. After the reboot, log into the floating IP.
  24. In the left menu, expand Settings and click HA Deployment.
  25. You can view the status of the HA pair and fix the database sync if it is broken. You can Break HA, Force Failover, or change the Floating IP (in HA Settings).

If this is older Citrix ADM, on the First Node, do the following:

  1. SSH to the first node and login as nsrecover/nsroot.
  2. Enter deployment_type.py.
  3. Enter 1 for NetScaler Console Server.
  4. Enter no when prompted for NetScaler Console Standalone deployment.
  5. For the First Server Node prompt, enter yes.
  6. Enter yes to Restart the system.

Older Citrix ADM Second Node:

  1. Import another ADM appliance to the same subnet and configure an IP address.
    • Latency to the HA node must not exceed 10 ms.
    • The HA nodes must be on the same subnet.
  2. If you added a second disk to the first ADM appliance, then you must add the same size second disk to the second ADM appliance.
  3. Configure the new node’s IP address.
  4. SSH to the second appliance, login as nsrecover/nsroot, and run the Disk Partition tool.
  5. SSH to the second appliance, login as nsrecover/nsroot, and run deployment_type.py.
  6. Enter 1 for Citrix ADM Server.
  7. Enter no when prompted for Citrix ADM Standalone deployment.
  8. Enter no when prompted is this is First Server Node.
  9. Enter the IP address of the first ADM node.
  10. Enter the nsroot password of the first node. The default password is nsroot.
  11. Enter a new Floating IP address.
  12. Enter yes to restart the system.

Older Citrix ADM Get Started:

  1. Use a browser to log into the first ADM appliance as nsroot/nsroot.
  2. Logging in to NetScaler ADM might show you the Get Started wizard. If you don’t see this wizard, then skip to the next section.
  3. In the Add NetScaler Instances page, you can Add Instance now, or just click Next and add instances later.
  4. In the Customer Identity page, you can login to Citrix Cloud, configure data sharing, or click Skip to do it later.

  5. In the System Notifications page, you can configure Email notifications now, or click Skip and do it later.
  6. In the Done page, click Finish.

Older Citrix ADM Deploy HA Configuration:

  1. After both appliances are fully booted, point your browser to the first appliance’s IP address and login as nsroot/nsroot. It will take several minutes after booting before the ADM appliance is ready.
  2. The top of the screen has some banners.
  3. If you want to make any network changes (e.g., DNS servers) to either node, then you must make those changes before you deploy the HA pair. Move your mouse over the left menu, expand Settings and click Administration.
  4. On the right, click IP Address, Second NIC, Host Name and Proxy Server.

    1. Enter an Alternate DNS and then click Save.
    2. Click the back arrow to go back.
    3. If you already created the HA pair, then the only way to add a second DNS server is through the command line on both nodes. See CTX281388 Error Message “Network configuration change is not allowed in Citrix ADM HA setup” When Changing Network Settings in ADM
      echo "echo \"nameserver DNS_IP\" >> /etc/resolv.conf" >> /mpsconfig/svm.conf
  5. Move your mouse to the left side of the screen, expand Settings, and then click Deployment.
  6. In the top right, click Deploy.
  7. Click Yes to reboot.
  8. It takes around 10 minutes to restart.
  9. After deployment, you can now use the Floating IP to manage the appliance pair.
  10. Logging in might show you the Get Started wizard. Proceed through the wizard as described in the previous section.
  11. Move your mouse to the left menu, expand Settings, and click Deployment.
  12. The Settings > Deployment page should show both nodes as UP and syncing.

Afterwards, you can manage High Availability.

  1. Settings> Deployment lets you see the HA nodes.
  2. You can Force Failover from here. Note: HA failover only occurs after three minutes of no heartbeats.
  3. On the top right is a HA Settings button that lets you change the Floating IP.

DR Node

Requirements for the DR node:

  • The main datacenter must have an HA pair of NetScaler Console appliances. Standalone in the main datacenter is not supported.
  • Latency from the main datacenter HA pair to the DR node must not exceed 200 ms.
  • Ports 5454 and 22 open between the NetScaler Console nodes.

To configure a DR node:

  1. Import another NetScaler Console appliance into a remote datacenter and configure an IP address.
  2. If you added a second disk to the main datacenter NetScaler Console appliances, then you must add the same size second disk to the DR NetScaler Console appliance.
  3. After configuring the new nodes’ IP address, SSH to the DR appliance and login as nsrecover/nsroot.
  4. Enter deployment_type.py.
  5. Enter 2 for Remote Disaster Recovery Node.
  6. Enter the Floating IP address of the HA pair in the main datacenter.
  7. Enter the nsroot password, which is nsroot by default.
  8. The DR node registers with the NetScaler Console HA Pair.
  9. You can change the password of the DR node by running the following command:
    ./mps/change_freebsd_password.sh <username> <password>
  10. Point your browser to the Floating IP Address and login.
  11. Go to Settings > Administration.
  12. On the right, in the right column, click Disaster Recovery Settings.
  13. The Registered Recovery Node should already be filled in. Click Deploy DR Node.
  14. Click Yes to enable DR.
  15. A System Backup is performed and replicated to the DR appliance. Click Close when done.
  16. The status of the DR node is displayed. You can click the Refresh icon on the top right to update the display.
  17. There’s a Sync DR Node button in case it gets out of sync.
  18. Disaster Recovery is not automatic. See the manual DR procedure at NetScaler Docs. Docs also shows how to fail back.
    • /mps/scripts/pgsql/pgsql_restore_remote_backup.sh

NetScaler Console Agents

NetScaler Console Agents help NetScaler Console discover and manage instances on the other side of a high latency WAN link.

The virtual appliance for NetScaler Console Agent is different than the normal NetScaler Console appliance.

  1. Download the NetScaler Console Agent from the main NetScaler Console download page. On the NetScaler Console download page for a particular build, scroll down the page to find the ADM Agent images.
  2. Extract the downloaded .zip file.
  3. Import the MASAGENT .ova to vSphere. You can import the single .ova file, or you can import the .ovf file plus the .mf file and the .vmdk file.


  4. Edit the settings of the virtual machine to see the allocated CPU and Memory.
  5. There’s no need to add a disk to the Agent.
  6. Power on the NetScaler Console Agent virtual machine.
  7. At the virtual machine’s console, configure an IP address.
  8. Login as nsrecover/nsroot.
  9. Run /mps/register_agent_onprem.py
  10. Enter the floating IP address of the main NetScaler Console HA Pair. Enter nsroot credentials for NetScaler Console. Enter a new password for NetScaler Console Agent.
  11. The Agent will be registered, and services restarted.
  12. To change the nsrecover password on NetScaler Console Agents, putty (SSH) to the NetScaler Console Agent appliance, login as nsrecover and then run the script at /mps/change_agent_system_password.py. Or you can change the password in the NetScaler Console interface at Infrastructure > Agents.
  13. Login to the NetScaler Console Floating IP.
  14. Go to Infrastructure > Instances > Agents.
  15. On the right, select the NetScaler Console Agent, and then click Attach Site.
  16. In the Site drop-down, if you don’t see your site, then you can click the Add button to create a new site.

    1. Enter a name and other location information.
    2. Make sure you enter the coordinates. Google can find coordinates for various locations. If Longitude is West, then the value is negative.
    3. Click Create when done.
  17. Click Save to attach the Site to the Agent. Any NetScaler instance discovered through this Agent will be attached to the configured Site.
  18. For Agent HA, import two NetScaler Console Agents into your hypervisor and attach both Agents to the same Site.
  19. You can change the Agent’s nsrecover password from the NetScaler Console GUI.
  20. ADM 13.1 build 24 and newer have a Settings button on the top-right of the Agents page where you can enable Notifications when a  NetScaler Console Agent is unreachable.

NetScaler Console Appliance Maintenance

Add NetScaler Instances

NetScaler Console must discover NetScaler instances before they can be managed. NetScaler Docs How NetScaler Console discovers instances.

  1. Point your browser to the NetScaler Console Floating IP address and login as nsroot/nsroot.

Before adding instances, NetScaler Console needs to know the nsroot password for the instances. You create Admin Profiles to specify the nsroot passwords.

  1. When adding instances during the initial Welcome wizard, next to Profile Name, click Add to create an Admin Profile.
  2. Or in the main interface, to edit or create new Admin Profiles, move your mouse to the left menu, then go to Infrastructure > Instances > NetScaler.
  3. On the right, open the menu named Select Action, and click Profiles.

  4. Click the Add button to create an Admin Profile.
  5. In the top half, give the Profile a name and enter the password for the instance’s nsroot account. Create a separate Admin Profile for each unique nsroot password.
  6. In the bottom, make up some SNMP settings. You can do SNMP v3. Change the Authentication Type and Privacy Type to stronger options.
  7. Click Create when done.

To add instances:

  1. Move your mouse to the left, expand Infrastructure, expand Instances, and click NetScaler.
  2. On the right, select a tab (e.g. MPX), and then click Add.
  3. The Add instance screen is the same as shown during the getting started wizard. To authenticate to the NetScaler ADC using nsroot, select an existing Profile or create a new one. If you have Sites or Agents, you can select one. Select a Site so it’s shown correctly on the world map. Click OK when done.

Tags:

  1. You can assign Tags to instances. See How to create tags and assign to instances at NetScaler Docs.

  2. You can then search instances based on the Tags.

Instance Authentication from NetScaler Console

By default, when you click the blue link for one of the instances, NetScaler Console will do single sign-on to the instance using nsroot credentials. This is probably a security risk, or certainly an auditing risk.

To prevent NetScaler Console from doing single sign-on to instances:

  1. In NetScaler Console, go to Settings > Administration.
  2. On the right, click System, Time zone, Allowed URLs and Agent Settings.
  3. In the Basic Settings page, check the box next to Prompt Credentials for Instance Login and click Save.

NetScaler SDX

  1. At Infrastructure > Instances > NetScaler, on the SDX tab, you can click Add to discover a SDX appliance plus all VPXs on that SDX appliance. You don’t have to discover the VPXs separately.
  2. In the Add NetScaler SDX page, click the Add button next to the Profile Name drop-down to create an SDX profile. Note: SDX profiles are different than VPX profiles.

    1. Enter the credentials for the SDX SVM Management Service.
    2. For NetScaler Profile, select an admin profile that has nsroot credentials for the VPX instances. After the SDX’s VPX instances are discovered, NetScaler Console uses this NetScaler Profile to login to each VPX. If you don’t have a VPX Admin Profile in your drop-down list, click the Add button. Note: You can only select one NetScaler Profile. If each VPX instance has different nsroot credentials, you can fix it after SDX discovery has been performed. The NetScaler Profile is different than the SDX Profile.
    3. Back in the Configure NetScaler Profile page, enter new SNMP settings that SDX will use to communicate with NetScaler Console.
    4. Click Create when done.
  3. Back in the Add NetScaler SDX page, select a Site, and optionally an Agent.
  4. Click OK to start discovery.
  5. After discovery is complete, switch to the VPX tab. You should automatically see the VPX instances.
  6. To specify the nsroot credentials for a VPX, right-click the VPX, and click Edit.

    • In the Modify NetScaler VPX page, either select an existing Profile Name, or click the Add button to create a new one. Click OK when done. It should start rediscovery automatically.
  7. After fixing the nsroot credentials, right-click the VPX instance and click Configure SNMP. NetScaler Console will configure the VPX to send SNMP Traps to NetScaler Console.

Instance management

  • REST API proxy – NetScaler Console can function as a REST API proxy server for its managed instances. Instead of sending API requests directly to the managed instances, REST API clients can send the API requests to NetScaler Console. See NetScaler Console as an API Proxy Server
  • NetScaler Flexed Licensing – Your Flexed license includes software instance licenses (VPX/CPX/BLX, SDX, MPX, and VPX FIPS) and bandwidth capacity licenses. You must apply the Flexed license on NetScaler Console. You must also apply the MPX Z-Cap and SDX Z-Cap license on NetScaler MPX and NetScaler SDX hardware respectively. A Flexed license also offers analytics for unlimited virtual servers. See Flexed capacity license at NetScaler Docs.

Enable AppFlow / Insight / Analytics

NetScaler Console build 21 and newer remove the VIP licensing requirement for Analytics.

  1. Go to Infrastructure > Instances > NetScaler.
  2. On the right, switch to one of the instance type tabs (e.g. VPX).
  3. Select an instance, open the Select Action menu, and click Configure Analytics.
  4. Select one or more Virtual Servers and then click the button labelled Enable Analytics.
  5. Different options are available for different types of Virtual Servers.
  6. For NetScaler Gateways, you want HDX Insight. Gateway Insight provides AAA and EPA info for the Gateway. Enable WAF Security Violations if you enabled WAF on your Citrix Gateway.

    • Expand Advanced Settings and select NetScaler Gateway.
  7. For HTTP Load Balancing Virtual Servers, you want Web Insight. If you are licensed for NetScaler ADC Premium Edition, then you can also enable WAF Security Violations for Web App Firewall and Bot Protection monitoring.

    • For analytics on HTTP Virtual Servers, expand Advanced and click Enable X-Forwarded-For.
  8. Click OK to enable AppFlow on the Virtual Servers.
  9. Click Close when configuration is complete.
  10. Enable Analytics on more Virtual Servers.
  11. Login to the NetScaler ADC (not NetScaler Console) and go to System > Settings.
  12. On the right, click Configure Modes.
  13. If you are using LogStream, then make sure ULFD is checked. Click OK.

    enable mode ulfd
  14. On the right, click Change Global System Settings.
  15. Scroll down to ICA port(s) and add 1494 and 2598 to the list. Click OK. (Source = Citrix Discussions)

    set ns param -icaPort 1494 2598
  16. On the right, click Change HTTP Parameters.
  17. At the top, add 80 and 443 to the Http Ports list. Click OK. (Source = Citrix Discussions)

    set ns param -httpPort 80 443
  18. By default, with AppFlow enabled, if a NetScaler ADC High Availability pair fails over, then all Citrix connections will drop, and users must reconnect manually. NetScaler ADC has a feature to replicate Session Reliability state between both HA nodes.
    1. From Session Reliability on NetScaler High Availability Pair at NetScaler Docs: Enabling this feature will result in increased bandwidth consumption, which is due to ICA compression being turned off by the feature, and the extra traffic between the primary and secondary nodes to keep them in sync.
    2. On NetScaler ADC, go to System > Settings.
    3. On the right, in the Settings section, click Change ICA Parameters.
    4. Check the box next to Session Reliability on HA Failover and click OK.
  19. On NetScaler ADC at System > AppFlow > Collectors, you can see if the AppFlow Collector (NetScaler Console) is up or not.

  20. Go to Traffic Management > Load Balancing > Services and find the adm_metric_collector_svc. If it’s not UP, then you can change it to use NSIP instead of SNIP.

    1. Go to System > AppFlow. On the right, click Change AppFlow Settings.
    2. Check the box next to Time Series Data Over NSIP and click OK.
  21. When AppFlow is enabled on a Gateway Virtual Server, an AppFlow policy is bound to twice to the Gateway: once for Request Policies (i.e., HTTP), and once for ICA Request Policies. You might want to verify that these bindings are actually configured.
  22. On the NetScaler Console appliance, AppFlow for ICA (HDX Insight) information can be viewed under the Gateway > HDX Insight node.
  23. Web Insight for HTTP Virtual Servers is under Applications > Web Insight. WAF Violations is under Security.

NetScaler Console nsroot Password

Changing NetScaler Console’s nsroot password also changes NetScaler Console’s nsrecover password.

  1. In NetScaler Console, go to Settings > Users & Roles.
  2. On the right, on the tab named Users, select the nsroot account, and click Edit.
  3. Check the box next to Change Password and enter a new password.
  4. You can also specify a session timeout by checking the box next to Configure Session Timeout.
  5. Click OK.

NetScaler Console Management Certificate

  1. The certificate to upload must already be in PEM format. If you have a .pfx, you must first convert it to PEM (Base64 certificate and key files). You can use a NetScaler ADC’s Import PKCS#12 feature to convert the .pfx to PEM and then download the converted certificate from the NetScaler ADC appliance.
    1. On any NetScaler ADC, go to Traffic Management > SSL.
    2. On the right, click Import PKCS#12.
    3. Enter a name for a new file that will contain the PEM certificate and PEM key.
    4. Browse to the .pfx file and enter the password.
    5. You can optionally encrypt the PEM key by selecting an Encoding Format and entering an encryption key.
    6. Click OK.
    7. To download the PEM file, go to Manage Certificates / Keys / CSRs.
    8. Scroll to the bottom of the list, right-click the new file, and click Download.
  2. Back in NetScaler Console, go to Settings > Administration.
  3. On the right, in the SSL Settings section, click Install SSL Certificate.
  4. Click Choose File to browse to the PEM format certificate and key files. If the PEM certificate and PEM key are in the same file, then browse to the same file for both fields.
  5. If the keyfile is encrypted, enter the password.
  6. Click OK.
  7. Click Yes to reboot the system.

  8. To force users to use https when accessing the NetScaler Console management page, go to Settings > Administration.

  9. On the right, click System, Time zone, Allowed URLs and Agent Settings.

  10. On the Basic Settings page, check the box next to Secure Access Only and click Save.

System Configuration

  1. Go to Settings > Administration.
  2. On the right, click System, Time zone, Allowed URLs and Agent Settings.

    1. Check the box next to Enable Session Timeout and specify a value.
    2. By default, at InfrastructureInstances > NetScaler , if you click a blue IP address link, NetScaler Console does single sign on to the instance using the nsroot credentials. If you want to force NetScaler Console users to login using non-nsroot credentials, then check the bottom box for Prompt Credentials for Instance Login.

    3. Click Save.
    4. On the left, click the Message of the day tab.
    5. On the right, check the box next to Enable Message.
    6. Enter a message, and then click Save.
    7. Click the back arrow when done.
    8. When you login to NetScaler Console, you’ll be shown the message.
  3. Settings > Administration > Configure SSL Settings lets you disable TLS 1 and TLS 1.1.

    1. On the right, click the Protocol Settings section in the Edit Settings section on the right side of the screen.
    2. On the left, uncheck TLSv1 and TLSv1.1. Then click OK.
    3. Click Yes when asked to confirm the restart.

Prune Settings

  1. To see the current database disk usage, go to Settings > Data Storage Management. You can manually initiate pruning from this page.
  2. Go to Settings > Data Storage Management > Data Retention Policy.
  3. System Pruning defaults to deleting System Events, Audit Logs, and Task Logs after 15 days. System events are generated by the NetScaler Console appliance, which is different than Instance events (SNMP traps) that are generated by NetScaler ADC appliances.
  4. If you change anything on these pages, click the Save button before switching to a different tab/node/page.
  5. NetScaler Console can initiate a purge automatically as the database starts to get full.
  6. Instance Events page controls when instance SNMP traps are pruned, which defaults to 40 days.

Backup Settings

  1. In Settings > Administration, in the middle column, under Backup, click Configure System and Instance backup.
  2. System Backup Settings defines how many NetScaler Console backups you want to keep. These are NetScaler Console backups, not NetScaler ADC backups.
    1. There’s an option for External Transfer.
    2. NetScaler Console System backups (not Instance Backups) are at Settings > Backup Files.
  3. The Instance page lets you configure how often the instances are backed up.
    1. You probably want to increase the number of instance backups or decrease the backup interval. The backups are quite small (e.g. 700 KB).
    2. There is an option to perform a backup whenever the NetScaler ADC configuration is saved.
    3. The Enable External Transfer checkbox lets you transfer the backups to an external system so it can be backed up by your backup tool.
    4. Instance backups can be found at Infrastructure > Instances > NetScaler. Right-click an instance and click Backup/Restore.
    5. You can Restore a backup, Download the backup, or Transfer it to an external system.

Analytics Settings

  1. There are more settings at Settings > Analytics Settings.
  2. ICA/Gateway Session Timeout can be configured by clicking the link.

    • If NetScaler Console doesn’t receive AppFlow records for a session, it will consider that session has got terminated in NetScaler ADC and stops monitoring that session further. The time for which NetScaler Console needs to wait before considering a session terminated is ICA session timeout. This is configurable in NetScaler Console, by default it is set to 15 minutes. (source = Citrix Discussions)
  3. Go to Applications > Dashboard.
  4. On the top right, click the gear icon.
  5. Configure App Score factors and thresholds.
  6. Settings > Analytics Settings > Data Persistence lets you configure how long Analytics data is retained. Adjusting these values could dramatically increase disk space consumption.

    • To see the current database disk usage, go to Settings > Data Storage Management.

NTP Servers

  1. On the left, click Settings > Administration.
  2. On the right, click NTP Servers.
  3. Click Add.
  4. Enter an NTP server and click Create.

  5. After adding NTP servers, click the NTP Synchronization button.
  6. Check the box next to Enable NTP Synchronization and click OK.
  7. Click Yes to restart.

Syslog

This is for syslog entries generated by NetScaler Console server, and not for syslog entries generated by the instances.

  1. Go to Settings > NetScaler Console Audit Log Messages > Syslog Servers.
  2. On the right, click Add.
  3. Enter the syslog server IP address and select Log Levels. Click Create.
  4. You can click Syslog Parameters to change the timezone and date format.

Email Notification Server

  1. Go to Settings > Notifications.
  2. On the right, on the Email tab, click the button named Email Servers.

    1. Click Add.
    2. Enter the SMTP Email server address and click Create.
  3. In the breadcrumb, click Notifications.
  4. On the right, on the Email tab, and click Add.

    • Enter information for a destination distribution list and click Create.
  5. You can highlight a Distribution List and click the Test button.


  6. On the left, click Settings > Administration.
  7. On the right, click Change Event Notification and Digest.

    1. Move notification categories (e.g. UserLogin) to the right.
    2. Check the box next to Send Email. Select a notification distribution list. Then click Save.

Authentication

  1. Go to Settings > Authentication.
  2. On the right, switch to the tab named LDAP.
  3. Click Add.
  4. This is configured identically to NetScaler ADC.
    1. Enter a Load Balancing VIP for LDAP.
    2. Change the Security Type to SSL, and Port to 636. Scroll down.
    3. Enter the Base DN in LDAP format.
    4. Enter the bind account credentials.
    5. Check the box for Enable Change Password.
    6. Click Retrieve Attributes and scroll down.
    7. For Server Logon Attribute, select sAMAccountName.
    8. For Group Attribute, select memberOf.
    9. For Sub Attribute Name, select cn.
    10. To prevent unauthorized users from logging in, configure a Search Filter. Scroll down.
    11. If desired, configure Nested Group Extraction.
  5. Click Create.
  6. On the left, go to Settings > Users & Roles.
  7. On the right, click the tab named Groups.
  8. On the right, click Add.

    1. Enter the case sensitive name of your NetScaler Console Admins AD group.
    2. Move the admin Role to the right.
    3. The Configure User Session Timeout checkbox lets you configure a session timeout.
    4. Click Next.
    5. On the Authorization Settings page, if you are delegating limited permissions, you can uncheck these boxes and delegate specific entities.
      • All DNS Domain Names (GSLB) is an option for Stylebooks in ADM 12.1 build 49 and newer.
    6. Click Next.
    7. In the Assign Users page, click Finish. Group membership comes from LDAP, so there’s no need to add local users.
  9. On the top right, click the button named Settings.

    • If desired, check the box next to Enable User Lockout, and configure the maximum logon attempts. Click OK.
  10. On the left, go to Settings > Authentication.
  11. On the top right, click the button named Settings.
  12. Change the Server Type to EXTERNAL and click Insert.
  13. Select the LDAP server you created and click OK.
  14. Make sure Enable fallback local authentication is checked and click OK.

Analytics Thresholds

  1. Go to Settings > Analytics Settings > Thresholds.
  2. On the right, click Add.
  3. Enter a name.
  4. Use the Traffic Type drop-down to select HDXWEBSECURITY, or APPANALYTICS.
  5. Use the Entity drop-down to select a category of alerts. What you choose here determines what’s available as Metrics when you click Add Rule.
  6. Click Add Rule to select a metric and threshold.

    • To add multiple rules for multiple Entity types, simply change the Entity drop-down before adding a new rule.
  7. If the Traffic Type is HDX, and the Entity drop-down is set to Users, on the bottom in the Configure Geo Details section, you can restrict the rule so it only fires for users for a specific geographical location.
  8. In the Notification Settings section, check the box to Enable Threshold.
  9. Check the box to Notify through Email, and select an existing Email Distribution List.
  10. Click Create.

Private IP Blocks

You can define Geo locations for internal subnets.

  1. Go to Settings > Analytics Settings > IP Blocks.
  2. On the right, click Add.
  3. In the Create IP Blocks page:
    1. Enter a name for the subnet.
    2. Enter the starting and ending IP address.
    3. Select a Geo Location (Country, Region, City). As you change the fields, the coordinates are automatically filled in.
  4. Click Create.

SSL Certificate Expiration Notification

SSL Dashboard can notify you when certificates will expire soon.

  1. In the NetScaler Console menu, expand Infrastructure and click SSL Dashboard.
  2. On the top right, click the button named Settings.
  3. In the Certificate is expiring in (days) field, enter the number of days before expiration that you want to receive a notification. The default is 30 days.
  4. Check one of the boxes (e.g. Email) below How would you like to be notified.
  5. Select a notification profile (e.g. Mail Profile) or Add one.
  6. Click Save and Exit or click Next to see more SSL Dashboard settings.

Instance Email Alerts (SNMP Traps)

You can receive email alerts whenever a NetScaler ADC appliance sends a critical SNMP trap.

  1. On the left, go to Infrastructure > Events > Rules.
  2. On the right, click Add.
  3. Give the rule a name.
  4. Move Severity filters (e.g. Major, Critical) to the right by clicking the plus icon next to each Severity.
  5. While scrolling down, you can configure additional alert filters. Leaving them blank will alert you for all categories, objects, and instances. If you want to exclude some Categories (e.g., entitydown), then move all Categories to the right and move the excluded categories back to the left.
  6. On the bottom of the page, in the Event Rule Actions section, click Add Action.
  7. In the Add Event Action page:
    1. Select an Action Type (e.g. Send e-mail Action).
    2. Select the recipients (or click the Add button to add recipients).
    3. Optionally, enter a Subject and/or Message.
    4. If you enter a Subject, you can check Prefix severity, category, and failure object information to the custom email subject.
    5. Emails can be repeated by selecting Repeat Email Notification until the event is cleared.
  8. Click OK.
  9. Then click Create to finish creating the event rule.
  10. See the Event Management section at All how to articles at NetScaler Docs.

Events Digest

NetScaler Console can email you a daily digest (PDF format) of system and instance events.

To enable the daily digest:

  1. Go to Settings > Administration.
  2. On the right, click Configure Event Notification and Digest.
  3. Switch to the Event Digest page.
  4. Uncheck the box next to Disable Event Digest.
  5. Configure the other settings as desired and click OK.

Director Integration

Integrating NetScaler Console with Director adds Network tabs to Director’s Trends and Session Details views. Citrix Blog Post Configure Director with Netscaler Management & Analytics System (MAS)

Requirements:

  • Citrix Virtual Apps and Desktops (CVAD) must be licensed for Premium Edition (formerly known as Platinum Edition). This is only required for the Director integration. Without Premium, you can still access the HDX Insight data by visiting the NetScaler Console web site instead of from Director.
  • Director must be 7.11 or newer for NetScaler Console support.

To link Citrix Director with NetScaler Console:

  1. On the Director server, run C:\inetpub\wwwroot\Director\tools\DirectorConfig.exe /confignetscaler.
  2. Enter credentials for a user that only has HDX Insight permissions. 
    User Role for ADM Director Integration
  3. If HTTPS Connection (recommended), the NetScaler Console certificate must be valid and trusted by both the Director Server and the Director user’s browser.
  4. Enter 1 for NetScaler Console (formerly Insight).
  5. Do this on both Director servers.

Use NetScaler Console

Infrastructure

Everything under the Infrastructure node is free.

Infrastructure Analytics – there’s an Infrastructure Analytics node under the Infrastructure node. For details, see Infrastructure Analytics at NetScaler Docs.

  • On the top right, the gear icon above the table shows the Settings Panel.
  • The tab named Score Indicator Settings lets you adjust how Infrastructure Analytics scores instance CPU, Memory, Disk, etc.
  • The Notifications tab lets you be notified when score thresholds are crossed.
  • You can click the Circle Pack button to change to the Circle Pack view.

At Infrastructure > Instances > NetScaler, select an instance and view its Dashboard.

  • The Instance Dashboard has tabs.

Backups are available by selecting an instance and clicking Backup/Restore.

Infrastructure > Network Reporting lets you create Dashboards where you can view Instance performance data.


Infrastructure > Network Reporting has a Thresholds button that lets you create thresholds when counters cross a threshold. For example, you might want a notification when Throughput gets close to the licensed limit.


At the bottom of the threshold are Notification Settings.

Configuration Record and Play

Use NetScaler Console to record a configuration change on one instance and push the change to other instances.

  1. Go to Infrastructure > Configuration > Configuration Jobs.
  2. On the right, click Create Job.
  3. Give the job a name.
  4. Change the Configuration Source drop-down to Record and Play.
  5. Change the Source Instance drop-down to the instance you want to record.
  6. Click Record.
  7. You might have to allow pop-ups in your browser.
  8. NetScaler Console opens the instance GUI. Make changes as desired.
  9. When done, go back to NetScaler Console and click Stop.
  10. NetScaler Console retrieves the changed config.
  11. On the left, you’ll see the changed commands. Drag them to the right.
  12. On the right, you can change instance-specific values to variables by simply highlighting the values. This allows you to change the values for each instance you push this config to.
  13. Proceed through the rest of the Configuration Job wizard like normal. You’ll select instances, specify variable values for each instance, and schedule the job.

Analytics and Applications

The AppFlow Analysis tools (e.g., HDX Insight) are located under the Applications, Security, and Gateway nodes. See Viewing HDX Insight Reports and Metrics at NetScaler Docs.

Applications > Dashboard automatically includes all Virtual Servers.

  • On the top right, click Manage Apps to add a custom group of Virtual Servers together into an application. The grouped Virtual Servers are removed from the Others list.
  • Click New Application.

  • Back in the App Dashboard, you can then click any Application’s box to view stats.
  • For Custom Applications, it combines stats about all of the vServers in that Custom Application.
  • There are buttons at the top the page to view more info about the application.

Applications > Configurations > StyleBooks lets you use StyleBooks to create new NetScaler ADC configurations.

There are built-in Enterprise StyleBooks for Exchange, SharePoint, Oracle, ADFS, etc. Or you can create your own StyleBook and use it to create NetScaler ADC configurations. For details, see StyleBooks at NetScaler Docs.

The Applications Node has quite a bit of functionality. See Application Analytics and Management at NetScaler Docs for details.

Link:

HDX Insight

HDX Insight Dashboard displays ICA session details including the following:

  • WAN Latency
  • DC Latency
  • RTT (round trip time)
  • Retransmits
  • Application Launch Duration
  • Client Type/Version
  • Bandwidth
  • Licenses in use

Citrix CTX215130 HDX Insight Diagnostics and Troubleshooting Guide contains the following contents:

  • Introduction
  • Prerequisites for Configuring HDX Insight
  • Troubleshooting
    • Issues Related to ICA parsing
    • Error Counter details
  • Checklist before Contacting Citrix Technical Support
  • Information to collect before Contacting Citrix Technical support
  • Known Issues

Gateway Insight

In the Gateway node is Gateway Insight.

This feature displays the following details:

  • Gateway connection failures due to failed EPA scans, failed authentication, failed SSON, or failed application launches.
  • Bandwidth and Bytes Consumed for ICA and other applications accessed through Gateway.
  • Number of users
  • Session Modes (clientless, VPN, ICA)
  • Client Operating Systems
  • Client Browsers

More details at Gateway Insight at NetScaler Docs.

Security Dashboard

The Security Dashboard uses data from Application Firewall to display Threat Index (criticality of attack), Safety Index (how securely NetScaler ADC is configured), and Actionable Information. More info at Application Security Dashboard at NetScaler Docs.

Troubleshooting

Citrix CTX215130 HDX Insight Diagnostics and Troubleshooting Guide: Syslog messages; Error counters; Troubleshooting checklist, Logs

Citrix CTX224502 Frequently Asked Questions During NetScaler MAS Troubleshooting

Upgrade NetScaler Console

Licensing – NetScaler Console build 21 and newer remove Analytics licenses thus enabling unlimited Analytics VIPs. 

  1. Upgrade paths from Before you upgrade at NetScaler Docs.
    1. If you upgrade from 12.0 build 57.24 and higher, first upgrade to 12.1, then to 13.1, and then to 14.1.
    2. If you upgrade from 12.1, you must first upgrade to 13.0 64.xx, and then directly to 14.1.
    3. If you upgrade from versions lower than 13.0 64.xx, for better user experience, first upgrade to 13.0 64.xx and then to 14.1.
  2. Download the latest Citrix Application Delivery Management (ADM) Upgrade Package. You want the ADM Upgrade Package, not the ADM image. It’s around halfway down the page.
  3. Login to NetScaler Console Floating IP or Active Node. Upgrading the Active Node automatically upgrades the Passive Node.
  4. Go to Settings > Deployment and make sure both nodes are online and replicating.

  5. Go to Settings > Administration.
  6. On the right, in the far-right column, click Upgrade NetScaler Console.
  7. Browse to the build-mas-14.1…tgz Upgrade Package and click OK. The file name starts with build-mas-14.1 or build-mas-13.1 (not masagent).


  8. Click Upgrade.
  9. Click Yes to continue with the upgrade.

  10. After it says that NetScaler Console upgrade completed, click Login again.
  11. The new firmware version can be seen by clicking your username in the top right corner.

Upgrade Disaster Recovery Node

After you upgrade the HA pair in the primary datacenter, you can upgrade the DR node.

  1. Use WinSCP or similar to connect to the DR node using the nsrecover credentials.
  2. On the NetScaler Console DR node, navigate to /var/mps/mps_images.
  3. Create a new Directory with the same name as the 13.1 build number. Then double-click the new directory to open it.

  4. Double-click the new directory to open it and then upload the file named build-mas-14.1-##.##.tgz or build-mas-13.1-##.##.tgz to the version-specific directory. This is the regular NetScaler Console upgrade file with a name starting with build-mas-14.1 or build-mas-13.1. It’s not the Agent upgrade file.
  5. SSH (Putty) to the DR node and login as nsrecover.
  6. Enter the following. Replace the # with the version number.
    cd /var/mps/mps_images/14.1-##.##
    tar xvzf build-mas-14.1-##.##.tgz

  7. Then enter the following. The appliance will reboot automatically.
    ./installmas

  8. After the reboot, the file /var/mps/log/install_state
  9. …shows you the installed version.

Upgrade NetScaler Console Agents

After you upgrade the NetScaler HA pair in the primary datacenter, and after you upgrade the DR node, you can then upgrade the NetScaler Console Agents.

  1. From the NetScaler Console download page, at the bottom of the page, download the ADM Agent Upgrade Package. This Agent Upgrade file is different than the regular NetScaler Console upgrade file. And it is different than the files to deploy a new Agent. Find it at the bottom of the downloads page.
  2. Use WinSCP or similar to connect to the NetScaler Console Agent using the nsrecover credentials.
  3. On the NetScaler Console Agent, navigate to /var/mps/mps_images.
  4. Create a new Directory with the same name as the agent build number. Then double-click the new directory to open it.

  5. Upload the file named build-masagent-14.1-##.##.tgz or build-masagent-13.1-##.##.tgz to the version-specific directory. This is the NetScaler Console Agent upgrade file, and not the regular NetScaler Console upgrade file.
  6. SSH (Putty) to the NetScaler Console Agent and login as nsrecover.
  7. Enter the following. Replace the # with the version number.
    cd /var/mps/mps_images/14.1-##.##
    tar xvzf build-masagent-14.1-##.##.tgz

  8. Then enter the following. The appliance will reboot automatically.
    ./installmasagent

  9. After the reboot, the file /var/mps/log/install_state
  10. …shows you the installed version.
  11. Repeat for any additional NetScaler Console Agents.
  12. If you login to NetScaler Console and go to Infrastructure > Instances > Agents
  13. …you should see the new Version. It will take several minutes for the version number to update.

Migrate NetScaler config to new appliances

Last Modified: Nov 21, 2024 @ 9:57 am

Navigation

Changelog

  • 2024 May 22 – added link to NetScaler Docs Migrate the configuration of a NetScaler instance across SDX appliances
  • 2022 Sep 11 – added a note about question marks ( ? ) in the config
  • 2022 Sep 9 – added /var/download
  • 2022 Sep 1 – added /var/netscaler/locdb
  • 2022 Aug 10 – check management services to make sure TLS12 is not disabled.
  • 2021 Sept 8 – added KEK key files

Overview

This article details one method of migrating a Citrix NetScaler ADC configuration from an old appliance pair to a new appliance pair while preserving the IP addresses. It requires a brief outage to move the IP addresses from the old pair to the new pair. Keeping the existing IP addresses avoids needing to make any changes to DNS, NAT, firewall, and routing.

Notes:

  • The new appliances must be connected to the same networks as the old appliances. If not, then the IP addresses need to change.
  • If you migrate the entire configuration at once, then the SNIPs can also be migrated.
    • The new appliances won’t have a SNIP until cutover time, so DNS will not work on the new appliances until cutover.
  • It’s much easier and quicker if you don’t migrate the NSIPs, so the new appliances will need new NSIPs.
  • You can use temporary SNIPs to validate connectivity on the new appliances.
  • Firmware versions can usually be different on the new appliances as compared to the old appliances.
    • If the old appliances are 10.5 or older, and if you have NetScaler Gateway with a customized logon page, then you’ll have to redo the customization as a Portal Theme on the new appliances.
    • GSLB Sites (MEP) can usually have different firmware versions.

Here’s a summary of the migration process. See the rest of this article for details.

  1. Prepare new appliances by connecting them to the network and pairing them together.
  2. Export files (e.g., SSL certificates) from old appliances and import to new appliances.
  3. Export configuration from old appliance and modify the configuration for later import to the new appliances.
  4. During a brief outage window, disable the old appliances (power off or disconnect), then run the config on the new appliances.
  5. Test the new appliances. If a problem is discovered, then you can roll back by disabling the new appliances and re-enable the old appliances.

Prepare New appliances

If VPX (not on SDX):

  1. Import new VPX appliances into your hypervisor.
    • If vSphere, change NIC to VMXNET3.
  2. Configure anti-affinity for VPX appliances in the same hypervisor cluster.
  3. Configure new management IP Addresses in the virtual machine console.
    • Default gateway should be on the NSIP subnet. The default gateway will be changed to a data VLAN during the cutover.
  4. In the Welcome Wizard, don’t configure any SNIPs. And DNS won’t work without SNIP.
  5. Install licenses on the new VPX appliances.
    • If pooled licensing, enter ADM IP address instead of DNS name to avoid needing DNS/SNIP configuration.
  6. Optional: configure VPX CPU Yield.

If VPX on SDX:

  1. Configure switch ports for LACP port channel and VLAN trunking.
  2. Connect the physical SDX hardware to port channels or individual interfaces.
  3. In SDX SVMs, create Channels and confirm that the channels are distributing.
  4. In SDX SVMs, upload desired VPX XVA firmware virtual appliance.
  5. In SDX SVMs, create new instance and connect the instance to channels.
    • New management IP address for each instance.
    • Default gateway should be on the NSIP subnet. The default gateway will be changed to a data VLAN during the cutover.
    • Enable NSVLAN if the VLAN trunk doesn’t have a native untagged VLAN.
    • For HA pair, create each instance on separate SDX appliances.

If MPX:

  1. Configure switch ports for LACP port channel and VLAN trunking.
  2. Connect the physical MPX hardware to port channels or individual interfaces.
  3. Use LCD, LOM, serial cable, or 0/1 Ethernet crossover to configure new NSIP for each MPX appliance.
    • Default gateway should be on the NSIP subnet. The default gateway will be changed to a data VLAN during the cutover.
    • Enable NSVLAN if the VLAN trunk doesn’t have a native untagged VLAN.
  4. In the Welcome Wizard, don’t configure any SNIPs. And DNS won’t work without SNIP.
  5. Upgrade firmware to desired version.
  6. At System > Network > Interfaces, configure LACP and confirm that the channels are distributing.
  7. At System > Network > Interfaces, disable all interfaces that are not plugged in.
  8. Install licenses on the new MPX appliances.
    • If pooled licensing, enter ADM (or ADM Agent) IP address instead of DNS name to avoid needing DNS/SNIP configuration.

For all ADC models:

  1. Create new DNS names for the new NSIP IP Addresses.
  2. Ensure you can SSH (e.g., Putty) and SCP (e.g., WinSCP) to the new NSIPs and old NSIPs.
  3. In the Welcome Wizard, don’t configure any SNIPs. And DNS won’t work without SNIP.
  4. Configure NTP and Time Zone on each node before you pair them together.
    • Enter NTP IP address instead of DNS name to avoid needing DNS/SNIP configuration.
    • From CLI, run shell, then run ntpdate command with NTP Server IP address to sync the local time with NTP server. NTP Sync needs to be disabled before you can run this command.
  5. Pair the two new appliances together.
  6. Configure a new password on the RPC Nodes at System > Network > RPC. Do it on both nodes.
  7. Set NSIP to Secure Access Only at System > Network > IPs, edit NSIP, checkbox at bottom of page. Do it on both nodes.
  8. Configure VLANs without SNIPs.
  9. For dedicated management subnet, configure PBRs.
  10. If you have channels that are not configured with a native, untagged VLAN, then configure tagged High Availability Heartbeats for non-NSIP interfaces by marking one VLAN as untagged and enabling the tagall option on the channel.
    • From CLI, run show ha node to confirm that there are no interfaces on which heartbeats are not seen.
  11. Use new temporary SNIP addresses to test VLAN connectivity
    • Bind the temporary SNIP to VLAN and channel and try to ping it.
    • Remove temporary SNIP when done testing.
  12. Install custom Management certificates that match new NSIP DNS names.
  13. If Default SSL Profile is enabled on old appliance, then enable it on new appliance pair.
  14. Add new pair to Citrix ADM.
  15. Add new pair to SNMP Manager.
  16. Add new pair to ControlUp Monitoring or any other Citrix NetScaler ADC monitoring tool.
  17. If any RADIUS authentication, and if RADIUS is not load balanced, then add the new NSIPs as RADIUS Client IP Addresses on the RADIUS servers.
    • If RADIUS is load balanced, then SNIP is the RADIUS Client IP, and the SNIP will be moved over during cutover.

Export/Import Files

Enact a change freeze to prevent any changes to the old appliances so that the files and Running Configuration don’t have to be downloaded again.

A full backup of the ADC will contain every file you need to upload to the new appliances.

  • On Windows, you’ll need a tool like 7-zip to extract the .tgz file.
  • Inside the backup folder are the files that you’ll need to upload to the new appliances.

Use WinSCP to transfer the following files from the old appliances to the same place on the new appliances.

  • Certificate files and DH files from /nsconfig/ssl
    • UNIX is case sensitive but Windows is not. While downloading, if you see a message about overwriting files, that’s probably because you have two files with the same name but different cases.
    • Don’t upload the files that start with ns- (e.g. ns-server.cert) since those are management certificate files.
  • LDAP certificate verification files from /nsconfig/truststore, if exists.
  • KEK files from /flash/nsconfig/keys. This is more likely to be needed when moving a 13.0 and newer configuration.
  • Custom monitors (e.g. nsldaps.pl) from /nsconfig/monitors, if exists. Grant execute permission to the uploaded .pl files.
  • Login Schemas from /nsconfig/loginschema.
  • /var/download has Bot signatures, WAF signatures, and Responder HTML Page imports. Signatures have their own method of export/import. Import these objects manually using the ADC GUI.
  • Location database from /var/netscaler/locdb.
  • Custom Portal Themes from /var/netscaler/logon/themes, if exists.
    • There’s no need to download the built-in themes. Only download the non-built-in themes.
    • On the new appliances, create Portal Themes with the same names and same template as the old appliances.
    • Then upload the Portal Theme files to the new appliances and overwrite the default files.
  • VPN bookmarks from /var/vpn/bookmark, if exists.
  • Gateway logon page customizations from /var/vpn/vpn, if any.
    • Try to redo these customizations as Portal Themes instead of uploading to the new appliances. The .js files in NetScaler 11.0 and newer are quite different from NetScaler 10.5 and older.
  • /nsconfig/rc.netscaler if it contains anything other than ntpd.
    • If this file contains customizations for the Gateway logon page, then try to redo them as a Portal Theme.
    • After uploading this file to the new appliances, reboot the new appliances so the commands are executed. Or you can manually run rc.netscaler after you chmod +x to the file.
  • /nsconfig/nsafter.sh if it exists.
    • If this file contains customizations for the Gateway logon page, then try to redo them as a Portal Theme.
    • After uploading this file to the new appliances, reboot the new appliances so the commands are executed. Or you can manually run nsafter.sh after you chmod +x to the file.
  • SSH Banners, like /nsconfig/issue.net/nsconfig/issue, and /nsconfig/motd.

To test the Portal Themes, you can create a temporary Gateway Virtual Server with a temporary VIP. Remove the temporary configuration when done testing.

Export Configuration

Export running configuration from the old appliances by going to System > Diagnostics > Running Configuration and then click the link to Save text to a File.

Edit the downloaded nsrunning.conf file to prepare it to be executed on the new appliances:

  • Remove all NSIP and hostname lines, including IPv6 and MAC addresses. These lines start with the following:
    • set ns config -IPAddress
    • set lacp -sysPriority 32768 -mac
    • set ns hostName
    • add ns ip -type NSIP – leave the other “add ns ip” commands, especially the SNIP (-novserver) commands
    • add ns ip6 -type NSIP
  • Remove the nsroot line that starts with:
    • set system user nsroot
  • Remove all interface commands that start with:
    • set interface
  • Remove all channel commands that start with:
    • add channel
    • set channel
  • Fix VLAN binding commands for new channels/interfaces.
  • Edit set snmp mib command, if it exists.
    • set snmp mib
  • Remove PBR commands for NSIP that start with. Or you can adjust the commands for the new NSIPs.
    • add ns pbr NSIP
  • Remove High Availability / Cluster commands that start with:
    • add HA node 1
  • Remove RPC Node commands that start with:
    • set ns rpcNode
  • Review the add route commands to make sure they are correct for the new appliances.
  • Search for set ssl service and look for the ones that start with ns (e.g., nshttps-127.0.0.1-443). If any of them have the parameter -tls12 DISABLED, then remove that parameter.
  • Search the entire config file for question marks ( ? ). Question marks do not import correctly from the CLI so you’ll have to fix them in the GUI after the import is complete.

Make sure the name of the modified configuration file is all lower case and doesn’t have any spaces. The batch command doesn’t work if there are upper case letters or spaces.

Upload the modified config file to /tmp on the new primary appliance.

Optional: review the SSL commands and look for any bindings of custom ciphers to SSL vServers. After the cutover, the new appliances will have your custom ciphers but unfortunately will also have the DEFAULT ciphers. To speed up the post-cutover process, you can pre-create the CLI commands that unbind the DEFAULT cipher groups. Here’s an example command:

unbind ssl vserver MySSLvServer -cipherName DEFAULT

Cutover

Decide how you will disable the old appliances. Here are some options:

  • Shutdown MPX Ethernet interfaces on network switch – if rollback, it’s easy to unshut the interfaces
  • Power off old VPX appliances, including on SDX – if rollback, it’s easy to power them on again
  • If old appliances have dedicated management network (no SNIP on management network), then you can to go to System > Network > Interfaces and disable them – if rollback, easy to re-enable
  • Power off old MPX appliances – if rollback, is somebody available in the data center to power them on again?

Before outage – Before the outage window, do the following on the old appliances:

  • Backup the old appliances and download the backup files.
  • Review the old appliances and record what’s up or down – After cutover, you’ll confirm up/down is the same on the new appliances.
    • Look in Traffic Management > Load Balancing > Virtual Servers for up/down status, including the number of UP services.
    • Go to Authentication > Dashboard to see which authentication servers are reachable

Outage – During outage window, do the following:

  • Disable the old appliances.
  • SSH to the new primary appliance and run the batch command to run the commands in your modified configuration file. For example:
    batch -fileName /tmp/nsrunningmodified.conf -outfile /tmp/outfile.txt
  • Review the outfile for errors. Some warnings can be ignored. You can use WinSCP to access the out file. Or cat it.

After running the batch command, do the following to fix the imported configuration.

  • Default Route – Remove extra Default Route, if any.
    • System > Network > Routes. If there are two 0.0.0.0 routes, then remove the one for the NSIP subnet. Check it on both nodes.
  • Question Marks – for any line that had a question mark ( ? ), fix the config in the GUI. The question marks are probably missing.
  • VIPs are UP? – Ensure VIPs (Load Balancing, Content Switching, Gateway) are up or down as noted on the old appliance.
    • Traffic Management > Load Balancing > Virtual Servers.
  • Go to Authentication > Dashboard and ensure the status matches the old appliance.
    • If needed, fix LDAP Password in load balancing monitor and in LDAP Policies/Servers.
    • If needed, fix RADIUS Secret in load balancing monitor and in RADIUS Policies/Servers.
  • Portal Themes – Point your browser to the Citrix Gateway logon pages and verify Portal Themes customizations are correct.
  • GSLB MEP – Set GSLB RPC nodes with Secure checked.
    • System > Network > RPCs. Check the Secure box. Change RPC password.
  • DEFAULT ciphers – If Default SSL Profile is enabled, then remove DEFAULT cipher group from SSL Profiles that have custom Ciphers.
    • System > Profiles > SSL tab. Edit each profile. In SSL Ciphers section, remove DEFAULT if it’s not the only cipher.
  • If Default SSL Profile is not enabled, then remove DEFAULT cipher group from all SSL Virtual Servers that have custom ciphers.
    • E.g. Traffic Management > Load Balancing > Virtual Servers. Edit an SSL Virtual Server. Scroll down to the SSL Ciphers section. If you see both DEFAULT and something else, then remove DEFAULT.
  • SSL Labs – Use SSL Labs to test externally reachable VIPs.
  • HA failover – Do a High Availability failover and verify that the same VIPs and authentication servers are UP on new primary appliance.

Citrix Health Check

Last Modified: Feb 28, 2024 @ 2:29 pm

Navigation

💡 = Recently Modified

Change Log

Health Check Overview

Health Checks review an environment for configurations that might cause future problems, not necessarily existing problems. Health Checks tend to focus on non-functional qualities like the following:

  • Availability
  • Security
  • Manageability
  • User Experience
  • Performance
  • Reliability

The rest of this article is an incomplete list of health check assertions for Citrix environments.

StoreFront Load Balancing

  • Citrix connectivity infrastructure design is documented: StoreFront, Gateways, ADCs, multiple datacenters, Delivery Controllers, SQL, etc.
    • Separate test Citrix environment has identical architecture as production: multiple data centers, high availability for all components, etc. – enables testing changes, including HA/DR changes, before performing those changes in production. Some upgrades are performed differently for HA/DR than for single components.
  • The FQDN that users use to access Citrix (e.g. https://citrix.company.com) resolves to a Load Balancing VIP, not a single server.
    • The FQDN automatically fails over (e.g. GSLB) to a VIP in a different data center if the primary data center is down.
  • The certificate for the SSL Load Balancing VIP is valid: trusted, not expired, matches FQDN, no errors in Chrome, etc.
    • Someone is responsible for ensuring the certificate is not expired and receives pending certificate expiration notifications.
  • The Load Balancing VIP sends SSL traffic to two or more StoreFront servers in the local data center – for redundancy.
    • The ADC-to-StoreFront server communication is SSL/TLS encrypted, not HTTP – this traffic contains user credentials.
  • The ADC monitor for the StoreFront servers is type STOREFRONT, or does a GET request to /Citrix/Store/discovery – other monitors might not detect stopped services.
  • X-Forwarded-For is configured in the Load Balancing Services (or Service Group) for Client IP header insertion.
  • Load balancing persistence is SOURCEIP with a timeout that is as long as the Receiver for Web timeout – COOKIEINSERT doesn’t work on all client devices.

StoreFront Servers

  • If the StoreFront servers are on the same hypervisor cluster, then anti-affinity is configured to keep them on separate hypervisor hosts.
  • StoreFront server VMs do no have any old snapshots – slows down performance, and consumes disk space.
  • StoreFront version is updated to resolve Security vulnerability as of Jan 16, 2024.
    • Upgrades are performed in a separate test environment that has identical architecture as production before the updates are performed in production.
  • StoreFront server group have latency of less than 40 ms (with subscriptions disabled) or less than 3 ms (with subscriptions enabled) between each member.
  • StoreFront configuration is propagated to other servers in the StoreFront Server Group.
  • OS, Patch level and VM Configuration of all StoreFront Server Group members are identical.
  • No recent unknown errors in Event Viewer at Applications and Services -> Citrix Delivery Services.
  • StoreFront Base URL is an https URL, not http. The FQDN resolves to the Load Balancing VIP, not a single server.
  • SSL certificates are installed on each StoreFront server and bound to IIS Default Web site. The SSL certificates are not expired.
  • C:\Users does not contain a bunch of user profiles. Delprof2.exe should be scheduled to delete these profiles – caused by users changing expired passwords.
  • If HTML5 Workspace app is enabled, then HTML5 Receiver is up to date – New versions are released at least monthly.
  • If Workspace app is stored on StoreFront servers, then the local Workspace apps in C:\Program Files\Citrix\Receiver StoreFront\Receiver Clients is current.
  • If Favorites are enabled, then Favorites (aka Subscriptions) are replicated to a StoreFront Server Group in a different data center.
  • If Federated Authentication Service (FAS), then multiple FAS servers configured through Group Policy.
    • FAS Servers are the same version as StoreFront.
    • If the FAS servers are on the same hypervisor cluster, then anti-affinity is configured to keep them on separate hypervisor hosts.
    • FAS Get-FasAuthorizationCertificate shows registration certificate is OK and not MaintenanceDue.
    • FAS group policy .admx template is up to date in SYSVOL.
    • FAS User Rules restricts usage to just some StoreFront servers, some VDAs, and some users – not all
    • Auto-enrollment is not enabled on the FAS certificate templates..
    • The Certificate Authority database is not excessively large.
    • For CA that is dedicated to only FAS, only Citrix templates. Other templates (e.g. Domain Controller) removed.
  • Task Manager shows sufficient CPU and Memory for each StoreFront server.
  • There’s sufficient free disk space – check C:\inetpub\logs
  • A monitoring tool alerts administrators of any StoreFront performance metric issue, availability issue (e.g. service stopped), and Event Log errors.
  • Logon Simulator runs periodically to verify that StoreFront is functional.
  • StoreFront Disaster Recovery procedure is documented and tested.

StoreFront Configuration

  • Only one store. Or every store but one is hidden – if multiple stores are advertised, then Workspace app will prompt the user to select a store.
  • Each Delivery Controller farm is configured with two or more Delivery Controllers – for redundancy.
    • Or Delivery Controller XML can be load balanced. If load balanced, then ADC monitor is of type CITRIX-XD-DDC – so ADC can detect Local Host Cache outages.
    • Prefer separate farms per data center instead of stretched single farms (with zones) across multiple data centers.
  • Transport Type for Delivery Controllers is https, not http – this traffic includes user credentials.
  • Receiver for Web Session Timeout is not too short for user experience or too long for security.
  • Citrix Gateway configuration in StoreFront console:
    • The STAs in StoreFront match the STAs configured on the Citrix Gateway Virtual Server on the ADC appliances.
    • Session Reliability is enabled.
    • Callback URL is only needed for SmartAccess and Citrix FAS – Callback URL should be removed if it’s not needed.
    • Internal Beacon is only reachable internally.
    • External Beacon does not include citrix.com – ping.citrix.com is OK
  • HDX Optimal Routing can send ICA traffic through the Citrix Gateway that is closest to the VDA (i.e. farm).

Delivery Controllers

  • In CVAD 1906+, Citrix Scout Health Check does not show any errors or warnings.
  • If the Delivery Controller servers are on the same hypervisor cluster, then ensure anti-affinity is configured to keep them on separate hypervisor hosts.
  • Delivery Controller VMs do not have any old snapshots.
  • Delivery Controller version is an LTSR Cumulative Update version (e.g., 1912 CU7), or the two latest Current Release versions (e.g., 2305). No other versions are supported – Citrix Product Matrix shows support dates.
    • Delivery Controller Upgrades are performed in a separate test environment before performed in production.
    • Citrix upgrades or updates are performed around twice per year.
  • Run Get-BrokerDBConnection to see the SQL connection string. No SQL Express. For AlwaysOn Availability Group (AAG):
    • SQL String points to AAG Listener, not single node.
    • All AAG SQL nodes in one data center. For multiple data centers, prefer separate farms in each data center with local SQL.
    • SQL String contains MultiSubnetFailover.
    • Each SQL server has SQL Logins for all Delivery Controllers – SQL Logins usually don’t replicate between SQL nodes.
    • Prefer Synchronous Commit with Automatic Failover over Asynchronous replication.
    • AAG Dashboard in SQL Studio does not show any issues.
  • SQL databases for Site, Monitoring, and Log are separate, not combined.
  • SQL databases for Citrix are not excessively large. Database Backup tool is truncating the database logs.
  • SQL Servers have sufficient CPU/Memory to handle the Citrix SQL traffic. Monitoring tool alerts SQL DBAs of any performance or availability issues.
  • SQL Server version is supported by Citrix. https://support.citrix.com/article/CTX114501
  • Local Host Cache is enabled on the Delivery Controllers. Run Get-BrokerSite to confirm.
    • Delivery Controller virtual CPU allocation is 1 CPU socket with multiple cores – SQL Express LocalDB for Local Host Cache only runs on a single socket (up to four cores).
    • How are non-persistent virtual desktops handled during SQL outage?
    • In CVAD 1912 and newer, LocalDB is upgraded to SQL Server Express LocalDB 2017
  • SQL Disaster Recovery plan is documented and tested.
  • SSL Certificates are installed on Delivery Controllers to encrypt XML traffic from StoreFront.
    • SSL certificates are bound to IIS Default Web Site, or netsh http sslcert to perform binding. IIS Binding does not include hostname.
    • SSL certificate not expired.
  • Trust XML Requests is enabled for pass-through authentication, SmartAccess, FAS, etc. Run Get-BrokerSite to confirm.
  • Task Manager shows sufficient CPU and Memory for each Delivery Controller server.
  • A monitoring tool alerts administrators of any Delivery Controller performance metric issue, availability issue (e.g. service stopped), and Event Log errors.

Citrix Studio

  • Citrix Studio consoles installed on administrator machines are the same version as the Delivery Controllers.
  • Customer Experience Improvement Program is disabled in Citrix Studio > Configuration node > Product Support tab.
  • Licensing Model/Edition matches what you actually own.
  • Citrix Studio Administrators are periodically audited to ensure only authorized users are granted Studio access.
    • Administrators are added as Active Directory Groups, not individual users.
  • Applications are published to Active Directory Groups, not individual users.
  • If App Groups, applications are published to only App Groups. Applications are not published to both App Groups and Delivery Groups.
  • Hypervisor connection uses a service account, not an admin account.
    • Hypervisor permissions for the service account are the minimum permissions required (custom role), not full hypervisor administrator.
  • Each Hosting Resource only has one datastore selected, not multiple datastores – Citrix MCS does not have a datastore “Rebalance” option. More datastores means more copies of master image snapshots, which means longer time to push out an updated Master image.
  • MCS Memory Caching Option is not enabled unless VDA 1903 or newer – older VDA, including 7.15 VDA, has poor performing MCSIO driver.
  • If MCS, VDA restarts are not performed in hypervisor since hypervisor does not cause MCS reset like Studio restart does.
  • StoreFront URLs are not assigned to Delivery Groups using Studio – instead use Workspace app group policy to assign StoreFront URL.

Citrix License Server

  • Citrix License Server is version 11.17.2.0 build 40000 or newer to resolve Apache vulnerabilities.
  • Citrix License Server is uploading telemetry every 90 days as required by Citrix. Check c:\Program Files (x86)\Citrix\Licensing\LS\resource\usage\last_compliance_upload
  • The licenses installed on Citrix License Server match the purchased licenses at https://citrix.com/account – some Citrix License Servers have too many licenses installed.
  • If multiple Citrix License Servers, installed license count across all License Servers does not exceed the purchased licenses shown at https://citrix.com/account
  • Administrators are not frequently clearing named user license assignments to simulate concurrent licensing – license assignments should only be cleared when the user permanently no longer uses Citrix.
  • Subscription Advantage dates are not expired – if expired, download new license files and install them.
  • Usage and Statistics tab is configured as intended in the Citrix Licensing Manager gear icon.
  • Citrix License Server Disaster Recovery procedure is documented and tested.

Remote Desktop Services (RDS) Licensing

  • If RDSH VDAs, two or more activated RDS Licensing servers.
  • RDS Licensing Server operating system version matches (or newer) the RDSH VDA operating system version – e.g. Windows 2019 RDS Licensing for Windows 2019 RDSH servers. Windows 2019 RDS Licensing also works with Windows 2016 RDSH servers.
  • In RD Licensing Manager, right-click server -> Review Configuration shows green checkmarks.
  • The combined licenses installed on all RDS license servers do not exceed the purchased licenses.
  • On RDSH VDAs, HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services\LicenseServers shows two servers.
    • LicensingMode = 4, which is Per User mode, which is not enforced.

Citrix Director

  • Director version matches the Delivery Controller version.
  • If multiple Director servers:
    • Hypervisor Anti-affinity is configured.
    • Director Saved Filters are relocated to a UNC path instead of local C: drive.
  • Director server VMs do not have old snapshots – slows down servers, and increases disk space.
  • SSL certificate is installed on Director servers.
    • Admins and Support teams always use https to access Director. IIS or load balancer redirects from http to https.
  • Director website is SSL load balanced.
    • SSL protocol, not http, between load balancer and Director servers – this traffic contains user credentials.
  • Director logon page auto-populates the domain name – for user convenience. Might have to reconfigure the domain name after every Director upgrade.
  • Citrix Policy Settings for Director:
    • Enable Process monitoring is enabled.
    • Enable monitoring of application failures is enabled.
  • If Citrix Virtual Apps and Desktops (CVAD) is Premium Edition:
    • Director Alerts are configured to email CVAD administrators.
    • Citrix ADM HDX Insight is integrated with Director. HTTPS protocol, not HTTP.
    • Probes are configured – Probe Agent version matches the Director version.
  • Help Desk knows how to use Citrix Director to support users.
  • Average logon durations are not excessive.
  • Repetitive issues (e.g. profile resets) are analyzed for root cause analysis and future prevention.

VDAs

  • Catalog design is documented – storage design, network design, multiple datacenters design, recovery design, etc.
  • VDA version matches the Delivery Controller version.
  • VDA Subnets are added to Active Directory Sites & Services.
    • Check LOGONSERVER variable after logon to confirm correct Domain Controller.
  • DHCP is highly available. VDA IP Subnet router forwards DHCP requests to more than one DHCP server. DHCP scope is replicated to more than one DHCP server.
    • DHCP Scope has sufficient address availability for VDAs.
  • DNS Reverse Lookup Zone with PTR records for the Virtual Apps and Desktops machines.
  • If KMS, slmgr.vbs /dlv shows a unique KMS CMID for each VDA machine – another option is Active Directory-based activation.
  • If persistent (dedicated) Catalogs:
    • The VDA version matches the Delivery Controller version – VDA updates should be automated (e.g. SCCM).
    • Dedicated Catalogs are created as Full Clones – Fast Clones cannot be moved to different storage or different hypervisor cluster.
    • Persistent desktops are backed up, replicated, etc. Recovery process is documented and tested.
    • Persistent desktop provisioning process is automated, preferably from a self-service portal.
  • No Personal vDisk – User Layers instead
  • No User Layers – slows down logons, and not all apps work – prefer Persistent Desktops instead.
    • User Layers are backed up, and restore process is documented and tested.
    • User Layers are stored on a clustered file server that can handle failover of always-open VHD files (e.g. Windows File Share with Continuous Availability) – Replication won’t help with file server outage and already open User Layers
  • Multiple department-specific master images instead of a single monolithic image – during user logon, monolithic images need to be dynamically customized for user requirements, which slows down logons.
    • No double-hop – slows down logons and increases complexity since double hop requires Workspace app and icon management on the first-hop VDA machine – prefer master images with every application installed locally instead of double-hop to published applications.
    • No Shortcut visibility management – slows down logons
    • No Elastic Layering – slows down logons
    • No App-V – slows down logons, and slows down machine performance
    • Master Image update process is automated – e.g. SCCM can push updates to master images
  • Catalogs are upgraded to latest Catalog version available.
  • VDA registrations are somewhat evenly distributed across the Delivery Controllers.
  • ListOfDDCs registry value on VDAs has two or more Delivery Controllers.
  • Daily Health Check report shows registration status and maintenance mode status of every VDA machine.
  • RDSH Load Index Policy has not been modified from the default. CPU Metric is too volatile, and can cause a Denial of Service and uneven distribution of sessions. Current Load Index values should be almost the same on every RDSH VDA and not be anywhere near 10000.
  • In-guest monitoring agent shows VDA memory usage. Allocated VM Memory matches or exceeds memory Committed Bytes – Hypervisor monitoring can’t show actual VM memory usage.
  • RDSH VDAs are periodically restarted – net statistics workstation or net server statistics shows uptime.
    • In CVAD 1909+, MaxDelayMins is configured in Get-BrokerRebootScheduleV2.
  • For EDT protocol, MtuDiscovery is enabled on the VDAs. MtuDiscovery requires VDAs version 1912 and newer.
  • If Cloud-hosting of VDAs, PowerScale controls VDA power management.

VDAs – Hypervisor Hardware Clusters

  • Desktop VDAs are in their own hypervisor cluster that does not contain any Server virtual machines – avoids Windows Server licensing.
    • Hypervisor clusters with Windows Servers have proper Windows Server licensing.
  • Hypervisor admins don’t perform any hypervisor updates without first reviewing Citrix’s Supported Hypervisors article.
  • VDA vCenter is separate from non-VDA vCenter – allows non-VDA vCenter to be upgraded without affecting Citrix.
  • Hypervisor performance is monitored and alerted: CPU contention (aka CPU Ready Percentage), disk latency, CPU Usage, etc.
  • Capacity planning tool warns admins when more hypervisor hardware is needed.
  • vSphere clusters have N+1 or N+2 extra capacity for redundancy.
  • HA and DRS are enabled on vSphere cluster according to design – not all designs use these features
  • CPU and Memory consumption are evenly distributed across the hypervisor cluster
  • If VMFS6 datastores, vSphere 6.7 Update 3 is installed – see release notes
  • NTP is configured and running on hypervisor hosts.
  • Hypervisor hosts have High performance BIOS settings.
  • In larger environments, dedicated VLAN(s) for VDAs – not shared with non-Citrix workloads
    • MCS and PVS require DHCP
  • Network Uplinks are redundant and have sufficient capacity
    • ESXi Management/Vmotion/Storage traffic are separate VLANs from the VDA VLANs
    • Storage multipathing is functioning
  • NVIDIA vGPU software is current on hypervisor host and virtual machines. – vGPU Manager 11.0+ supports guest driver version one major version back (e.g., 10.0) – February 2024 security update
    • The newest hypervisors can vMotion GPU-configured virtual machines – vgpu.hotmigrate configured in vCenter Advanced Settings. DRS set to Manual or Partially Automated.
    • NVIDIA in-guest vGPU Driver is installed before the VDA is installed – otherwise HDX 3D Pro will not work.
    • ESXi Host Graphics Settings set to Shared Direct and spread across GPUs – Host GPUs set to Shared Direct.
    • NVIDIA license servers are redundant (failover support), or in the cloud.

VDAs – Virtual Machine Hardware (vSphere)

  • Network Interface type is VMXNET3, not E1000.
  • devices.hotplug=false is configured in Virtual Machine Configuration Settings.
  • If disk space is a concern, virtual machine memory is reserved to reduce .vswp file size.
  • If Citrix App Layering:
    • Paravirtual controller is not added.
    • Boot firmware is BIOS, not EFI.
  • Windows 10 version is supported by Citrix VDA version, and supported by App Layering version.
    • Windows 11 is supported with VDA 2109 and newer. It is not supported by VDA 1912.
  • VMware Tools version is current.

VDAs – Master Image Build

  • Master Image build process is documented.
  • Master Image virtual machine was built from scratch – not converted from a physical machine.
  • Security scan of the VDA Master Images shows compliance with enterprise security requirements.
  • VDA version resolves vulnerability – 2305, 2203 CU3, or 1912 CU7
  • Master Image updates:
    • Master Image maintenance is automated – e.g., SCCM can push updates to Master Images. A script can push Master Images to Catalogs.
    • Software Deployment team notifies the Master Image maintainers when applications or Windows require an update.
    • Master Image is sealed before shutdown – e.g., antivirus is generalized, SCCM Client is generalized – sealing should be scripted – Base Image Script Framework (BIS-F) can automate this
    • Master Image updates are tested before deployed to production. QA testing. Canary testing.
    • Master Image snapshots are deleted after a period of time.
  • Profile Management is patched to resolve Local privilege escalation vulnerability – 2106 Hotfix 1, 1912 CU3 Hotfix 1, or 7.15 CU7 Hotfix 1. 1912 CU4 includes the fix.
  • Antivirus is installed. Antivirus is optimized for non-persistent machines (aka VDI).
  • Other IT agents (e.g., software auditing, SCCM Agent) are optimized for non-persistent machines.
  • Local Groups:
    • Administrators group does not contain any non-administrators.
    • Direct Access Users group only contains authorized RDP users.
  • Citrix Optimizer or similar has removed Windows 10 Store Apps.
  • Windows Default profile was not modified – instead use group policy to control Windows appearance.
  • Windows Updates are current (i.e., last install date is within the last 60 days).
  • C: drive permissions are changed so Users can’t create folders on root of C: drive.
  • Power management is set to High Performance with no sleep timers.
  • If Citrix Provisioning:
    • Pagefile is shrunk so it fits on PVS cache disk – there’s no need to move the pagefile since PVS will move it for you. Just make sure it’s small.
    • Event Logs are moved to PVS cache disk.
  • Customer Experience Improvement Program is disabled in VDA registry.
  • FSLogix is a recent version – FSLogix version 2.9.7979.62170 resolves a security vulnerability in Cloud Cache.
  • Office 365 Shared Computer Activation is enabled.
    • FSLogix is implemented for Outlook search roaming.
  • Microsoft Teams is installed using machine-wide installer.
    • Microsoft Teams machine-wide installation is periodically manually updated – there’s no auto-update.
    • Teams cache folders excluded from roaming profiles.
  • For OneDrive Files On-demand, is only installed on Windows Server 2019 and newer, or Windows 10 1709 and newer
    • OneDrive is installed using machine-wide installer – check C:\Program Files (x86)\OneDrive
    • FSLogix saves OneDrive cache.

Citrix App Layering

  • Prefer automated (e.g. SCCM) Master Image updates over manual App Layering layer updates – if SCCM is mature, then there’s no need for App Layering.
  • Prefer SCCM-managed dedicated desktops over User Layers – SCCM is a known technology. User Layers are proprietary to Citrix and might not support every application.
  • Enterprise Layer Manager (ELM) version is current – ELM updates are required to support newer Citrix Virtual Apps and Desktops (CVAD) and newer Windows 10. There’s no LTSR version of ELM.
  • Citrix Provisioning Agent version matches the ELM version.
  • Directory Junction Bind account is a service account, not a regular user whose password expires.
    • LDAP is Secure (Use SSL).
  • Administrator role membership is periodically audited to ensure only authorized users are granted access.
  • ELM is backed up. Or layers are periodically exported from ELM.
  • Group Policy controls membership of local groups in VDA machines – e.g. add Domain Admins to local Administrators group.
  • Antivirus is configured properly for Layering.
  • Hypervisor Connector uses a service account with limited permissions.
  • Connector cache is enabled to speed up layering operations.
  • Offload Compositing is enabled in the Connectors.
  • File servers hosting Elastic Layers and User Layers are monitored for performance issues and capacity planning.
  • User Layers are backed up, replicated, etc.

Citrix Provisioning

Provisioning Servers:

  • Provisioning Servers version matches the Delivery Controller version.
  • Multiple Provisioning Servers for High Availability.
    • Hypervisor Anti-affinity is configured.
  • Sufficient RAM for vDisk caching in memory – around 2-3 GB of memory per active vDisk.
  • Only one NIC per Provisioning Server – simplifies the configuration.
  • Server Bootstrap has multiple Provisioning Servers listed.
  • Threads times Ports are sufficient for the number of target devices.
  • vDisk Boot Menu is disabled in the registry – enables maintenance mode Target Devices to automatically boot from maintenance mode vDisks.
  • Antivirus has exclusions for Citrix Provisioning.
  • Provisioning Server performance metrics are monitored and alerted.
    • NIC throughput is not saturated.

Provisioning Farm Properties:

  • Offline database is enabled.
  • Auditing is enabled.
  • Administrators list only contains authorized administrators, preferably from an Active Directory Group.
  • Customer Experience Improvement Program is disabled.
  • For AlwaysOn Availability Group, MultiSubnetFailover is configured in the database connection string.

vDisks:

  • If local storage, vDisk files are identical on all Provisioning Servers.
  • vDisk files are VHDX, not VHD – faster version merging.
  • vDisks are sized dynamic, not fixed – Saves disk space. Standard Mode vDisks don’t grow so no performance impact.
  • vDisk files are defragmented.
  • vDisk files are backed up.
  • vDisk updates are automated.

Target Devices:

  • Target Device Boot Method is highly available – Target Devices on same subnet Provisioning Servers. Or DHCP Option 66 with TFTP Load Balancing. Or Boot ISO/Boot Partition has multiple Provisioning Server addresses.
    • DHCP is highly available. Subnet’s router forwards DHCP requests to multiple DHCP servers. Replicated DHCP scope.
    • Use PXEChecker to verify multiple TFTP responses.
  • vDisk Write cache is configured for Target Device RAM with overflow to disk – health check script should periodically verify this.
  • WriteCache folders on Provisioning Servers are empty – no server-side caching.
  • If KMS, slmgr.vbs /dlv shows a unique KMS CMID for each Target Device machine – another option is Active Directory-based activation.
  • Target Devices are evenly distributed across multiple Provisioning servers – ensures that High Availability is working correctly – stop Stream Service to confirm HA
  • System Reserved Partition is removed from inside vDisk.
  • VMware Tools in Target Devices (vDisks) is up to date.
  • Target Device Software version matches the Citrix Provisioning version.
  • Target Device status shows low number of retries.

Group Policies and Active Directory

  • VDAs are placed in VDA-only OUs, no users – group policies apply to VDAs without affecting physical endpoints.
    • Separate OUs per Delivery Group – different group policies apply to different Delivery Groups.
  • Master Images are located in VDA OUs – computer-level GPO settings apply to the Master Images to avoid GPO timing issues on linked clones.
  • Block Inheritance OUs and Enforced GPOs are minimized.
  • .admx templates in SYSVOL > PolicyDefinitions are current – Windows 10 templates, Office templates, Citrix templates, etc.
  • Group Policy Loopback Processing Mode is enabled.
  • Duplicate, conflicting GPO settings are minimized – e.g. Group Policy Loopback Processing Mode is sometimes enabled in several GPOs.
    • Run Group Policy Results to show the actual GPO settings that applied to a specific session – compare with design
  • Lockdown GPO applies to non-administrators that log into VDA machines. Lockdown GPO doesn’t apply to administrators.
  • Remote Desktop Session Host (RDSH) session timeouts (idle, disconnect) are configured in a Microsoft GPO.
  • AppLocker or similar prevents users from running unauthorized executables (e.g. ransomware).
  • Initial application configuration is automated using group policy – e.g. auto configure application database connections, remove first time usage prompts.
  • Group Policy changes are tested in separate Test GPOs and separate Test VDAs before applying to production.
  • Monitoring tool shows group policy processing duration during logon.

Citrix Policies

  • Citrix Policies are configured in a Group Policy Object, not in Citrix Studio – a GPO can apply to multiple Citrix Virtual Apps and Desktops (CVAD) farms in multiple datacenters. Citrix Studio is single farm only.
    • Citrix Policies are not configured in both Citrix Studio and Group Policy – avoids confusion over which setting wins
    • If configured in Citrix Studio, and if multiple farms/sites, then Citrix Policy settings are identical in all farms/sites.
  • Citrix Group Policy Management plug-in on GPMC machines is same version included with CVAD ISO.
  • Unfiltered policy is on the bottom of the list (lowest priority) – most specific filters on top, least specific filters on bottom.
  • Client drive mapping, client clipboard, client printing, drag and drop, and client USB are disabled when connecting from external (e.g. SmartAccess) – only enabled by exception.
  • Client printing is set to Use Universal Print Driver only – avoids installing print drivers on VDA machines.
  • Audio is set to Medium quality – High Quality uses more bandwidth than Medium Quality.
  • Time zone redirection is configured in both Citrix Policy and RDSH Microsoft Group Policy.
  • For HDX Insight, ICA Round-Trip Time policy is enabled.
  • Visual quality and video codec settings are not modified from the defaults.
    • Legacy Graphics Mode is disabled.
  • Adaptive Transport (EDT) is enabled – it’s default disabled in 7.15. MTU might need to be decreased.
    • MtuDiscovery is enabled on the VDAs. MtuDiscovery requires VDAs version 1912 and newer.
  • Session Reliability is not disabled.
  • RDSH Session Timers are configured in Microsoft GPO, not Citrix Policy – Citrix Policy setting description shows if setting applies to Server OS or not.

Citrix Workspace Environment Management (WEM)

  • Prefer Group Policies over WEM – WEM requires extra infrastructure, extra learning, extra administration, and extra support. Some WEM user settings are per-machine (per configuration set) only. WEM can’t replace group policies since there’s currently no .admx support.
    • Citrix Profile Management and Microsoft Folder Redirection are configured using Microsoft Group Policy, not WEM – Group Policies are well known. WEM is proprietary to Citrix and requires WEM skills to troubleshoot.
  • WEM is within two versions of the latest – there’s no LTSR version of WEM.
    • WEM Consoles and WEM Agents match WEM Server version.
  • Multiple load balanced WEM Servers for High Availability.
    • If multiple WEM servers are on the same hypervisor cluster, then Hypervisor anti-affinity is configured for the multiple WEM servers.
    • WEM Agents point to WEM Server load balanced FQDN, not individual server.
    • WEM Console points to single WEM Server, not load balanced FQDN.
  • WEM Brokers are close the VDAs – WEM configuration can be exported/imported into WEM implementations in multiple data centers.
  • WEM Database is hosted on an AlwaysOn Availability Group or other Highly Available SQL solution.
    • SQL database is backed up. SQL database recovery is documented and tested.
  • In WEM 1909+, Infrastructure Service Enable performance tuning for Windows Communication Framework is enabled and set to the number of concurrent WEM Agents that will be connected to this one WEM server. Maximum value is 3000.
  • Antivirus exclusions are configured for Citrix WEM.
  • WEM .admx group policy template in SYSVOL > PolicyDefinitions is updated whenever WEM Servers are updated.
  • Settings are in WEM, or Group Policy, but not both – helps troubleshooting. Reduces confusion.
  • Bypass ie4uinit Check is enabled (Advanced Settings > Service Options) – for faster logons.
  • Drive mappings and printer mappings are moved to WEM and processed asynchronously (Advanced Settings > Agent Options).
  • Check Application Existence is enabled (Advanced Settings > Agent Options) – doesn’t create shortcut unless application exists
  • CPU Optimization is enabled – Memory management trades memory for disk; which is cheaper? Process exclusions might be needed.
    • In WEM 1909 and newer, CPU Spike Protection = Auto instead of Customize.
  • Fast logoff is enabled.
  • Unused action types are disabled from processing (Advanced Settings > Main Configuration) – speeds up logons.
  • Run Once enabled for Actions and scenarios that support it – speeds up logons.
  • WEM Agent Offline mode is enabled.
  • Computer startup script refreshes WEM Agent cache on each VDA reboot.
    • Script has correct Agent installation path and correct service name since they changed in 1909 and newer.
  • WEM Logs are reviewed for problems – enable debug logging. Look for Active Directory timeouts.
  • WEM Server performance is monitored for metric thresholds and future capacity issues.
  • WEM Server recovery is documented and tested.

Citrix Profile Management and Folder Redirection

  • No mandatory profiles on Windows 10 – benchmarks show slower performance.
  • Profile Management is configured in Group Policy, not Citrix Policy or Citrix WEM – Group Policy is the most reliable and most well-known option.
  • Profile file share:
    • File server is close to the VDAs – users log into VDAs that are closest to the file server (aka home site).
    • File share is highly available.
    • Caching is disabled on the file share.
    • No DFS multi-master replication. Single target only – neither Citrix nor Microsoft support merge replication.
    • Profiles are backed up and/or replicated. Recovery process is documented and tested.
    • Different profile folders for different operating system versions and/or different Delivery Groups.
    • NTFS permissions of individual user folders in the file share only grant access to the one user – no Users, no Domain Users, and no Authenticated Users.
    • Use TreeSize or similar to see profile size – adjust profile exclusions if too big.
    • Antivirus is not slowing down profile file transfer performance – time how long it takes to copy a profile folder to the local machine.
    • File servers are monitored for performance issues, including disk latency and free disk space.
  • Profile Management .admx file in SYSVOL > PolicyDefinitions matches the VDA version (or date).
  • Profile Management logs are stored on UNC share instead of local C: drive, especially if the VDAs are non-persistent.
    • Only Domain Computers have Modify permission to the Logs share – Users don’t need any permission.
  • Profile Management logs contain at least a few days of logons – if only a few minutes, then too much information is being logged and Log Settings GPO setting should be modified.
  • Profile streaming is enabled – speeds up logons.
  • Active Write Back is disabled – places extra load on file servers for not much benefit.
  • Customer Experience Improvement Program is disabled.
  • Locally cached profiles are deleted at logoff from RDSH machines that don’t reboot often.
  • No Start Menu roaming issues – might need ResetCache registry value.
  • Microsoft FSLogix is implemented for Outlook Search roaming – better than UPM’s Outlook search roaming.

Folder Redirection:

  • Folder Redirection is configured in Microsoft GPO settings, not in Citrix Profile Management settings – Microsoft GPO configuration is most reliable, most known, and can migrate existing files.
  • No AppData redirection – slows down applications.
  • “Grant the user exclusive rights” option is unchecked – allows administrators to access redirected profile folders.
  • Folder Redirection file share:
    • File share is highly available.
    • No DFS multi-master replication. Single target only – neither Citrix nor Microsoft support merge replication.
    • Redirected Folders are backed up and/or replicated. Recovery process is documented and tested.
    • NTFS permissions of individual user folders in the file share only grant access to the one user – no Users, no Domain Users, and no Authenticated Users.
    • Antivirus is not slowing down folder redirection performance.
    • File servers are monitored for performance issues, including disk latency and free disk space.

Home Directories:

  • File server is close to the VDAs – users log into VDAs that are closest to the file server (aka home site).
  • File share is highly available.
  • No DFS multi-master replication. Single target only – neither Citrix nor Microsoft support merge replication.
  • Home Directories are backed up and/or replicated. Recovery process is documented and tested.
  • NTFS permissions of individual user folders in the file share only grant access to the one user – no Users, no Domain Users, and no Authenticated Users.
  • Antivirus is not slowing down file transfer performance – time how long it takes to copy a Home Directory folder to the local machine.
  • File servers are monitored for performance issues, including disk latency and free disk space.

Endpoint Devices

  • Prefer Windows 10 endpoints over thin clients – thin clients don’t support all Citrix functionality (e.g. local printing, browser content redirection). ThinKiosk can lock down Windows 10 endpoints.
  • Newest VDAs and newest Workspace apps have better WAN performance than LTSR 7.15.
  • Browser Content Redirection offloads video (e.g. YouTube) from VDAs to endpoint – reduces CPU consumption in the data center.
  • Workspace app is periodically (e.g., twice per year) updated by endpoint management team. 
  • Workspace app (aka Receiver) ADMX templates in SYSVOL > PolicyDefinitions are current.
  • Group Policy adds StoreFront URL to Local Intranet zone.
  • Group Policy pushes StoreFront URL to Workspace app – so users don’t have to enter the URL.
  • Pass-through authentication is enabled for internal PCs – SSON Configuration Checker can verify proper configuration.
  • HKCU\Software\Citrix\Dazzle\Sites\store\type shows DS, not PNA – store added as Delivery Services (StoreFront), not PNAgent (legacy).
  • Internal Beacon at HKEY_CURRENT_USER\SOFTWARE\Citrix\Receiver\SR\Store\#\Beacons\Internal\Addr0 is internally reachable only – not reachable externally.
  • External Beacon at HKEY_CURRENT_USER\SOFTWARE\Citrix\Receiver\SR\Store\#\Beacons\External does not include citrix.com or ping.citrix.com.
  • EDT protocol (aka Adaptive Transport) is enabled. Director shows HDX protocol as UDP – Remote Display Analyzer can analyze problems with the graphics/codec.
  • HDX Insight: Newest VDAs and newest Workspace app have less AppFlow CPU impact on ADC than LTSR 7.15 VDAs.
  • Google Chrome detects Workspace app properly, especially through Gateway – requires Gateway ADC to able to resolve StoreFront Base URL to StoreFront IP
    • Chrome 77+ has receiver://* added to URL whitelist so the user isn’t prompted to open Workspace app

Citrix NetScaler ADC

  • NetScaler ADC Admins have subscribed to Citrix Security Bulletins at https://support.citrix.com/user/alerts
  • NetScaler ADC firmware build is patched for vulnerabilities as of Jan 16, 2024.
  • NetScaler ADC firmware updates are tested on separate test ADC appliances before performed in production. Test ADC appliances have test VIPs – application owners can test their VIPs on test ADC before firmware is upgraded in production.
  • NetScaler ADC VPX on vSphere:
    • NetScaler VPX NICs are VMXNET3, not E1000.
    • NetScaler is version that supports vSphere version.
    • DRS Cluster Anti-affinity is configured for the VPX appliances in the same HA pair.
    • CPU/Memory are reserved at hypervisor. If not reserved at hypervisor, then Yield CPU is not enabled so that VPX can reserve CPU itself.
  • NetScaler ADC license does not expire any time soon – check date inside license files at /nsconfig/license
    • ADM Pooled Licensing has license alerts enabled for email notifications.
  • Physical NetScaler ADC:
    • LOM port is connected and configured.
    • LOM nsroot password is changed from the default.
    • No VLAN is connected to multiple active interfaces unless those interfaces are in a port channel.
  • ADC nsroot password is not nsroot. nsroot password is managed by Privileged Identity Management tool. Admins don’t use nsroot to login.
  • Policies are Advanced Expressions instead of Classic Expressions. (source = CTX296948)
  • Management authentication is configured for external authentication server, typically LDAP.
    • LDAP is load balanced instead of multiple LDAP Policies to individual LDAP servers – avoids premature account lockout.
    • LDAP is encrypted: LDAPS on port 636.
    • LDAP Bind account is a service account – not a regular user whose password expires.
    • LDAP Search Filter only allows ADC Admins Active Directory Group to authenticate.
  • If TACACS, firmware is 12.0 build 57 or newer to prevent TACACS Accounting from blocking AAA.
  • nsroot account has external authentication disabled.
  • No local NetScaler ADC accounts except nsroot.
  • NTP and Time Zone are configured.
  • Syslog is configured to send logs to external SIEM, especially if ADC is performing authentication.
  • SNMP Traps are sent to Citrix ADM appliance.
    • Thresholds are configured for CPU and Memory alarms.
  • Customer Experience Improvement Program (CUXIP) is disabled.
  • Recommended TCP Profile Settings are configured.
  • Drop Invalid HTTP requests is enabled in HTTP global settings.
  • Secure Access Only is enabled on all NSIPs and all management-enabled SNIPs – check both nodes of High Availability pair.
    • Management certificate has no certificate errors.
  • Networking:
    • NetScaler ADC VLANs only have one interface (or one channel) – Best Practices at Citrix Docs.
    • If Dedicated Management Network, Policy Based Routes (PBR) are configured for NSIP reply traffic and NSIP-initiated traffic.
    • Unused network interfaces are disabled.
    • ADC instance is connected to only one security zone – if connected to multiple security zones, then a firewall is bypassed.
    • Default route should be Internet facing, or a data VLAN – not NSIP VLAN.
    • Only one default route – extra default routes can come from HA pairing or hardware migration.
  • Root DNS server address “h.root-servers.net” is set to 198.97.190.53 – might be old address due to older firmware
  • Unused NetScaler ADC configurations are removed – unused server objects, unused policies, etc.
  • Citrix ADM monitors and backs up the ADC appliances.
  • ADC Dashboard shows that CPU, Memory, and Throughput have not exceeded appliance capacity or appliance licensing.
  • /var/core and /var/crash do not have recent crash dumps.

NetScaler ADC High Availability Pair

  • Firmware build is identical on both nodes.
  • Installed Licenses are identical on both nodes.
  • NTP and time zones are configured on both appliances – Configuration node shows System Time.
  • Unused interfaces are disabled.
  • HA is synchronizing without error.
  • Both HA nodes are set to ENABLED – not STAYPRIMARY and/or STAYSECONDARY.
  • Fail-safe mode is enabled.
  • “show ha node” shows heartbeats across all interfaces – no “interfaces on which heartbeats are not seen”.
  • High Availability failover has been tested, including RADIUS authentication, which might come from a different source IP.
  • Sync VLAN configured to enable ISSU on ADC 13.0+

NetScaler ADC SDX

  • LOM port is connected and configured.
    • LOM nsroot password is not nsroot.
  • No hardware problems shown on SDX SVM dashboard page.
  • SDX firmware is current – should be same or newer than the VPX firmware.
  • SDX SVM nsroot password is not nsroot. nsroot password is complex. Admins don’t use nsroot to login.
  • Management authentication is configured for external authentication server, typically LDAP.
    • LDAP is load balanced instead of multiple LDAP Policies to individual LDAP servers – avoids premature account lockout.
    • LDAP is encrypted: LDAPS on port 636.
    • LDAP Bind account is a service account – not a regular user whose password expires
      • LDAP Bind account should be a regular domain account, not a Domain Admin.
      • LDAP Bind account should be dedicated to LDAP Bind and not used for anything else.
    • LDAP Search Filter only allows ADC SDX Admins Active Directory Group to authenticate.
  • No local accounts except nsroot.
  • No certificate errors when accessing SVM management using htttps.
    • HTTPS is forced in System Settings – HTTP is not allowed.
  • Multiple DNS servers are configured in Networking Configuration – initial setup only asks for one DNS server.
  • Channels are created at SDX SVM instead of inside VPX instances.
  • NTP is configured and enabled.
  • Syslog is configured.
  • SNMP traps are sent to Citrix ADM.
  • The number of SDX instance licenses installed matches what’s owned at https://citrix.com/account
  • SDX SVM Backups are configured with External Transfer – or download periodically – or ADM.
  • VPX Instances:
    • Platinum Edition license is assigned to instances.
    • SSL Chips are assigned to VPX instances.
    • All SDX hardware is allocated to VPX instances – If not, why not?
    • Production instances typically have Dedicated CPU cores. Test/Dev instances typically have Shared CPU.
    • VLANs are specified inside VPX instances instead of at instance properties on SDX Management Service – avoids reboot if you need to change the VLAN configuration.
    • No VMACs in instance interface settings.

NetScaler ADC Load Balancing and SSL

  • Load Balancing configurations are documented.
  • Monitors do more than just telnet – e.g. LDAP monitor performs LDAP query.
    • LDAP monitor bind account uses service account, not domain admin.
    • LDAP monitor is filtered to cn=builtin – to reduce result size.
    • RADIUS monitor looks for response code 2 or 3.
  • If multiple Virtual Servers for multiple ports on the same VIP, configure Persistency Group – e.g. Horizon Load Balancing.
  • Rewrite policies remove web server header information (Server, X-Powered-By, etc.)
  • SSL Labs SSL Server Test shows A or A+ grade for all Internet-facing SSL vServers.
  • Redirect Virtual Servers are UP (Responder method) instead of DOWN (Backup URL method).
  • Custom (non-default) ciphers are bound to every SSL Virtual Server – see Citrix Networking SSL / TLS Best Practices.
  • SSL v3 and TLS v1.0 are disabled on every SSL Virtual Server.
  • SSL Renegotiation is set to NONSECURE: configured globally, or in SSL Profiles (including default profile).
  • Root certificate is not linked to intermediate certificate.
  • Certificates are not expired.
  • SSL Services do not have “-TLS11 disabled” or “-TLS12 disabled” – might be disabled from older firmware.
  • ADM alerts ADC administrators when certificates are soon to expire.
  • ADM Analytics is enabled for the HTTP Virtual Servers.
    • ADM Web Insight is viewed.
  • Bot Management (13.0 build 41+) and/or Web App Firewall are configured if ADC Premium Edition.
    • ADM Security Insight is enabled and viewed.

Citrix NetScaler ADM

  • NetScaler ADM exists and manages all ADC appliances.
    • Prompt credentials for instance login is enabled in ADM System Settings – if ADM does Single Sign-on to instances, then all instance changes are logged as nsroot instead of ADM user.
  • NetScaler ADM firmware version is current.
    • ADM Agents and DR nodes have same firmware version as ADM – check /var/mps/log/install_state
  • Two DNS servers are configured in ADM Network Configuration – initial setup only asks for one DNS server.
  • Two NetScaler ADM appliances in High Availability mode with Floating IP – provides redundancy.
  • Every High Availability node and DR node has same disk size.
  • NetScaler ADM nsroot password is not nsroot. nsroot password is complex. Admins don’t use nsroot to login.
  • NetScaler ADM Agent nsrecover password has been changed from the default.
  • Management authentication is configured for external authentication server, typically LDAP.
    • LDAP is load balanced instead of multiple LDAP Policies to individual LDAP servers – avoids premature account lockout.
    • LDAP is encrypted: LDAPS on port 636.
    • LDAP Bind account is a service account – not a regular user whose password expires.
    • LDAP Search Filter only allows ADM Admins Active Directory Group to authenticate.
  • No local accounts except nsroot.
  • No certificate errors when accessing ADM management using htttps.
    • HTTPS is forced in System Settings – HTTP is not allowed
  • Time zone is configured.
  • NTP is configured and enabled.
  • NetScaler ADM Database is not full. Sufficient disk space.
  • Sufficient ADM CPU/Memory – verify at System > Statistics or System > Deployment.
  • All features enabled – verify at System > Administration > Disable or enable features
  • SSL Dashboard alert notifications are enabled to warn of upcoming certificate expiration.
  • Tasks page notifications are enabled
  • Event Rules are configured to email ADC administrators of Critical or Major ADC alarms.
  • NetScaler ADC Instance Backup settings on NetScaler ADM:
    • Number of NetScaler ADC instance backups retained is sufficient for restoring from history.
    • NetScaler ADC Backups are transferred to external SFTP, SCP, or FTP server.
    • NetScaler ADC Restore process is documented and tested.
  • VIP Licensing:
    • Installed license count on NetScaler ADM matches the licenses owned at https://citrix.com/account.
    • Licenses are assigned to Virtual Servers that need Analytics (e.g. HDX Insight) or Applications tab.
    • AppFlow/Insight is enabled on NetScaler Citrix Gateway and HTTP Virtual Servers.
    • TCP 5563 opened from SNIP to ADM for Metrics Collector.
    • License expiration notifications are enabled.
  • Private IP Blocks are configured for geo mapping of ADC instances and Analytics sessions.
  • Analytics Thresholds are configured – e.g., ICA Latency threshold.
  • Session Reliability on HA Failover is enabled on ADC instances in ICA Parameters – if not enabled, then sessions drop on failover.
  • ADM HDX Insight is linked to Director Premium Edition using https protocol, not http protocol.

NetScaler Citrix Gateway ICA Proxy

NetScaler Citrix Gateway Virtual Server:

  • SSL Labs SSL Server Test shows A or A+ when it scans the Gateway external FQDN.
  • If ICA Only is unchecked on the Gateway Virtual Server, then System > Licenses shows sufficient Maximum Citrix Gateway Users Allowed.
  • NetScaler Citrix Gateway Virtual Server Maximum Users is 0, which means unlimited.
  • TCP Profile is configured with Recommended TCP Profile Settings.
  • DTLS is enabled on the Virtual Server for EDT protocol.
    • UDP ports are open on firewall from Internet and to VDAs.
    • Director Session Details shows HDX protocol as UDP.
  • ICA Connections shows port 2598 (Session Reliability enabled), not 1494.
  • NetScaler Citrix Gateway communication to StoreFront is https protocol, not http.
  • NetScaler Citrix Gateway communication to StoreFront is load balanced to multiple StoreFront servers – not a single StoreFront server.
  • STAs on NetScaler Citrix Gateway matches StoreFront configuration.
  • Policies are Advanced Expressions instead of Classic Expressions. (source = CTX296948)
  • If EPA is used for SmartAccess, then Endpoint Analysis Libraries are updated.

NetScaler Citrix Gateway Authentication:

  • Encrypted LDAP:
    • LDAP is load balanced instead of multiple LDAP Policies to individual LDAP servers – avoids premature account lockout.
    • LDAP is encrypted: LDAPS on port 636.
    • LDAP Bind account is a service account – not a regular user whose password expires.
    • LDAP Search Filter only allows authorized remote users in an Active Directory group to authenticate.
  • Two-factor authentication – RADIUS:
    • For Workspace app, password fields are swapped.
    • Both factors are required to login. Can’t bypass second factor.
    • RADIUS tested from both High Availability nodes (perform failover).
  • SAML Authentication:
    • Prefer RADIUS over SAML so that ADC will have access to the user’s password to facilitate Single Sign-on to the VDA machines.
    • If SAML response does not provide user’s password, then Federated Authentication Service (FAS) is deployed .
    • For Workspace app support of SAML, SAML is configured in nFactor (AAA), not Gateway – requires ADC 12.1 and newest Workspace app.
    • SAML iDP Signing certificate is not expired. ADC administrators know how to update the Signing certificate.
    • relaystateRule configured in SAML Action to prevent session hijack – see https://support.citrix.com/article/CTX316577
  • Native OTP:
    • OTP Active Directory attribute is encrypted.
  • nFactor login fields are encrypted.

NetScaler ADC GSLB

  • If a DNS name resolves to multiple IP addresses, then the DNS name should be GSLB-enabled for automatic failover.
  • DNS Records are delegated to two or more ADC ADNS services, usually in separate data centers.
    • NS records and SOA records are added to ADC for delegated domain names and/or delegated sub zones.
  • All NetScaler ADC nodes that have ADNS listeners for the same DNS name have identical GSLB configuration.
  • Public GSLB Services have monitors that verify remote Internet connectivity – don’t give out IP if users can’t reach it.
  • Separate NetScaler ADC appliances for public DNS and internal DNS – If both are on one appliance, then how are the DNS configurations separated?
  • RPC nodes for Metric Exchange Protocol (MEP) should have Secure enabled.
  • Firewall should only allow the MEP endpoints to communicate over 3009 – don’t open to whole Internet.
  • If Static Proximity:
    • Static Proximity database is current.
    • GSLB Services show correct geo location.
    • Custom Entries are added for internal subnets.
  • If DNS Views, DNS Views are configured on all GSLB Services – if GSLB Service doesn’t have a DNS View, then that GSLB Service might not function correctly.
  • If Active/Active GSLB load balancing, then site persistence is functioning correctly.
  • DNS security options are configured to prevent ADNS Denial of Service.

Self-Service Password Reset (SSPR) – Citrix ADC

Last Modified: Dec 28, 2020 @ 5:49 am

Navigation

SSPR Flow

The SSPR feature in Citrix ADC is completely unrelated to the SSPR in Citrix StoreFront. They are separate registrations.

  • ADC SSPR registration is stored in an Active Directory user attribute.

SSPR is available in Citrix ADC 12.1 build 51.16 and higher.

SSPR requires nFactor, which is only available in Citrix ADC Advanced Edition and Premium Edition. You can use nFactor with Citrix Gateway.

The first factor is a Login Schema that has an optional Forgot Password link. The Login Schema file named SingleAuthPasswordResetRem.xml has this link. There are other built-in Login Schemas for Username Only and Dual Auth that also have the link.

If the user logs in normally, then ADC checks if the user has been enrolled in SSPR or not. If not enrolled, then the user is prompted to select questions and provide answers. This is a custom Login Schema that you can create from the ADC GUI as described later.

If the user clicks the Forgot Password link, then the user is prompted to answer two of the previously enrolled questions.

  • Optionally, the user might be prompted to retrieve a One-time password (OTP) sent to an Alternate Email address. The user might not have access to corporate email, so the OTP should be sent to the user’s personal email address. The user enters the alternate email address during SSPR registration.

  • Finally, the user is prompted to enter a new password.

Here’s what a sample SSPR nFactor configuration looks like:

# ** nFactor Visualizer 
# ** ------------------ 
# ** AAA vserver: AAA-SSPR
# **    Login Schema Policy = lschema_password_reset
# **       Priority = 20
# **       Rule = true
# **       Login Schema XML = "/nsconfig/loginschema/LoginSchema/SingleAuthPasswordResetRem.xml"
# **    Adv Authn Policy = LDAP_Password_Reset
# **       Priority = 10
# **       Rule = "aaa.LOGIN.VALUE(\"passwdreset\").EQ(\"1\")"
# **       Action = ldapAction named LDAP-Corp_NoAuth
# **       Goto if failed = NEXT
# **       Next Factor if Success = kba_validation
# **          Login Schema Profile = LSCHEMA_INT
# **          Adv Authn Policy = kba_validation
# **             Priority = 10
# **             Rule = true
# **             Action = ldapAction named LDAP-Corp_NoAuth
# **             Goto if failed = NEXT
# **             Next Factor if Success = email_validation
# **                Login Schema Profile = LSCHEMA_INT
# **                Adv Authn Policy = email_validation
# **                   Priority = 10
# **                   Rule = true
# **                   Action = emailAction named EmailOTP
# **                   Goto if failed = NEXT
# **                   Next Factor if Success = password_reset
# **                      Login Schema Profile = LSCHEMA_INT
# **                      Adv Authn Policy = LDAP-Corp-Adv_NoAuth
# **                         Priority = 100
# **                         Rule = true
# **                         Action = ldapAction named LDAP-Corp_NoAuth
# **                         Goto if failed = NEXT
# **    Adv Authn Policy = LDAP-Corp-Adv
# **       Priority = 20
# **       Rule = true
# **       Action = ldapAction named LDAP-Corp
# **       Goto if failed = NEXT
# **       Next Factor if Success = check_for_sspr_registration
# **          Login Schema Profile = LSCHEMA_INT
# **          Adv Authn Policy = switch_to_kba_register
# **             Priority = 1
# **             Rule = "AAA.USER.ATTRIBUTE(\"kba_registered\").EQ(\"0\")"
# **             Action = NO_AUTHN
# **             Goto if failed = NEXT
# **             Next Factor if Success = kba_registration
# **                Login Schema Profile = lschema_KBARegistration
# **                Adv Authn Policy = first_time_login_forced_kba_registration
# **                   Priority = 1
# **                   Rule = true
# **                   Action = ldapAction named LDAP-Corp
# **                   Goto if failed = NEXT
# **          Adv Authn Policy = NoAuth
# **             Priority = 100
# **             Rule = true
# **             Action = NO_AUTHN
# **             Goto if failed = END

Active Directory User Attribute

Citrix ADC SSPR stores its configuration in an Active Directory User attribute.

The attribute must be dedicated to SSPR. You can’t use the same attribute that you might already be using for Citrix ADC One-time passwords (OTP).

  • In Active Directory Users and Computers, enable Advanced Features mode (from the View menu), double-click a user, switch to the Attribute Editor tab, and find a String attribute that is not being used. When you double-click an attribute, look for String in the window title.
  • Open the Active Directory Schema console. Under Attributes, find your candidate attribute and double-click it. Make sure the Maximum length is high (e.g. 32767). The extensionAttributes default to 1024 or 2048, which is not long enough.
  • You might want to uncheck the bottom boxes (e.g. Attribute is copied when duplicating a user) since each user has their own SSPR configuration.
  • ADC needs permissions to set this Attribute and reset user passwords. You can use the Delegate Control wizard to grant these permissions.

    1. Consider creating a group for SSPR Admins and delegating permissions to that group. You can add your LDAP Bind Account and your Service Desk accounts to this group.
    2. In the Tasks to Delegate page, change it to Create a custom task to delegate and click Next.
    3. In the Active Directory Object Type page, change the selection to Only the following objects in this folder and select User objects. Click Next.
    4. In the Permissions page, on the top of the page, check the box for Property-specific.
    5. In the Permissions list, check the box next to Reset password. Scroll down.
    6. Find the attribute you are using for SSPR and check the Read and Write boxes. Click Next.

Help Desk will need to be trained to reset a user’s SSPR registration by double-clicking the attribute and clicking the Clear button. The next time the user logs in, the user will be prompted to enroll for SSPR again.

ADC uses a certificate to encrypt the contents of the Attribute. It currently is not possible to configure the certificate from the GUI, so you’ll need to SSH to the ADC and run the following command. The certificate should already be installed on the ADC.

bind vpn global -userDataEncryptionKey MyCertificate

The ADC SSPR configuration needs two LDAP Policy/Servers: one with authentication enabled, and one without authentication. In both LDAP Servers, in the Other Settings section, enter the Attribute name into the Alternate Email field and in the KB Attribute field. The same attribute can be used for both. However, you cannot use the same attribute for the OTP Secret field.

SSPR KBA Registration Login Schema

When creating the KBA Registration Login Schema you will be prompted to choose from a list of questions. This list is configured in the file KBQuestions.json located in /nsconfig/loginschema/LoginSchema.

  1. In ADC GUI, on the left, expand Security, expand AAA – Application Traffic, and click Login Schema.
  2. On the right, switch to the Profiles tab.
  3. On the right, click the button labelled Add KBA Registration Login Schema.
  4. Give the new Schema a name. ADC will create a file with the same name.
  5. Configure at least two questions. Move questions from the left to the right. Note: the XML file that it creates has size limits so don’t choose too many questions.
  6. On the right, you can optionally change the labels for Question Field and Answer Field. If you don’t enter anything, then the labels are Question, and Answer, as shown earlier in this article. Scroll down.
  7. Near the bottom are two More links you can click.
  8. The first More link lets you allow users to enter their own questions.
  9. The second More link lets you enable Register Alternate Email.
  10. Click Create when done.
  11. ADC creates a file in /nsconfig/loginschema/LoginSchema that matches the Schema name you entered earlier.
  12. ADC also adds a Login Schema Profile with the same name you entered earlier.

LDAP Servers

You’ll need two LDAP Servers: one with Authentication enabled, and the other with Authentication unchecked. Other than the Authentication checkbox, they are identical. Configure them like normal LDAP Servers.

The NoAuth server has Authentication unchecked.

Configure both LDAP Servers with the SSPR attribute as detailed earlier.

Email Action and Policy

You can optionally configure an Email action to send the user a OTP before allowing the password reset.

  1. In the ADC GUI, the easy way to find Email Actions is to enter “email” in the top left search box.
  2. On the right, click Add.
  3. Give the Email Action a name.
  4. In the Username box, enter the From email address. The emails will look like they come from this address.
  5. Enter the SMTP password for this From address.
  6. In the Server URL box, enter a smtps URL with port number similar to smtps://mail01.corp.com:587. Gmail’s URL smtps://smtp-relay.gmail.com:587 also works.
  7. In the Content box, enter a string containing $code. ADC replaces $code with the OTP.
  8. In the Type box, enter SMTP.
  9. In the Email Address box, enter aaa.user.attribute("alternate_mail"). This expression pulls the destination email address from Alternate Email attribute you defined in your LDAP Server.
  10. Click Create.
  11. It does not appear to be possible to use the ADC GUI to create an Email Authentication Policy. SSH to your ADC and run something similar to the following command:
    add authentication Policy email_validation -rule true -action EmailOTP

SSPR nFactor Configuration

AAA vServer and Login Schema

The first factor will be a Login Schema that has the Reset Password link. This section creates the following portion of the nFactor configuration:

# ** AAA vserver: AAA-SSPR
# **    Login Schema Policy = lschema_password_reset
# **       Priority = 20
# **       Rule = true
# **       Login Schema XML = "/nsconfig/loginschema/LoginSchema/SingleAuthPasswordResetRem.xml"

Do the following:

  1. Create a AAA vServer.
  2. It can be Non Addressable.
  3. Optionally bind a certificate to the AAA vServer so its status will be Green instead of Red.
  4. On the right, in the Advanced Settings column, click Login Schemas to move it to the left.
  5. On the left, in the Login Schemas section, click where it says No Login Schema.
  6. Next to the Select Policy field, click Add to create a new Login Schema Policy.
  7. Give the Login Schema Policy a name to indicate that this Login Schema includes a link for Self-Service Password Reset.
  8. Next to Profile, click Add to create a new Login Schema Profile.
  9. Give the Login Schema Profile a name.
  10. In the Authentication Schema field, click the edit icon.
  11. In the Login Schema Files section, click the LoginSchema folder.
  12. Highlight one of the PasswordReset files (e.g. SingleAuthPasswordResetRem.xml) and then click the blue Select button on the top right. You can also optionally modify the built-in files.
  13. The Create Authentication Login Schema window collapses and the Authentication Schema field shows the selected file. Click Create when done.
  14. Click Create to finish creating the Authentication Login Schema Policy.
  15. Click Bind to bind the Login Schema Policy to the AAA vServer.

SSPR Registration

Let’s start with the SSPR Registration flow. This section creates the following snippet of the full nFactor configuration:

# ** AAA vserver: AAA-SSPR
# **    Login Schema Policy = lschema_password_reset
# **       Priority = 20
# **       Rule = true
# **       Login Schema XML = "/nsconfig/loginschema/LoginSchema/SingleAuthPasswordResetRem.xml"
# **    Adv Authn Policy = LDAP-Corp-Adv
# **       Priority = 20
# **       Rule = true
# **       Action = ldapAction named LDAP-Corp
# **       Goto if failed = NEXT
# **       Next Factor if Success = check_for_sspr_registration
# **          Login Schema Profile = LSCHEMA_INT
# **          Adv Authn Policy = switch_to_kba_register
# **             Priority = 1
# **             Rule = "AAA.USER.ATTRIBUTE(\"kba_registered\").EQ(\"0\")"
# **             Action = NO_AUTHN
# **             Goto if failed = NEXT
# **             Next Factor if Success = kba_registration
# **                Login Schema Profile = lschema_KBARegistration
# **                Adv Authn Policy = first_time_login_forced_kba_registration
# **                   Priority = 1
# **                   Rule = true
# **                   Action = ldapAction named LDAP-Corp
# **                   Goto if failed = NEXT
# **          Adv Authn Policy = NoAuth
# **             Priority = 100
# **             Rule = true
# **             Action = NO_AUTHN
# **             Goto if failed = END

Do the following:

  1. In the AAA vServer, scroll down to the Advanced Authentication Policies section and click where it says No Authentication Policy.
  2. In the Select Policy field, select your normal LDAP policy. The Login Schema collects username and password and verifies it with the LDAP Policy that you select here.
  3. Change the Priority to 20 or similar. This binding must be a higher number than the Password Reset flow that we configure later.
  4. Next to the Select Next Factor field, click Add to create a Policy Label which will check if the user is registered or not.
  5. Name the Policy Label to indicate that this Policy Label checks if the user is registered or not.
  6. Leave the Login Schema set to LSCHEMA_INT (i.e. no schema), and click Continue.
  7. Next to the Select Policy field, click Add to create an Authentication Policy.
  8. Name this Authentication Policy to indicate that it does nothing more than check if the user is registered or not.
  9. In the Expression field, enter the following expression:
    AAA.USER.ATTRIBUTE("kba_registered").EQ("0")
  10. Click Create.
  11. Back in the Policy Label Policy Binding window, next to the Select Next Factor field, click Add to create another Policy Label.
  12. Name the Policy Label to indicate that this factor performs the SSPR Registration (i.e. Knowledge-Based Authentication registration).
  13. In the Login Schema field, select the KBA Registration Login Schema that you created earlier.
  14. Click Continue.
  15. Next to the Select Policy field, click Add to create an Authentication Policy.
  16. Name the Authentication Policy to indicate that ADC uses this LDAP Server to store the KBA registration information in the Active Directory User attribute.
  17. Change Action Type to LDAP.
  18. Change Action to your normal authenticating LDAP Server that has the KB Attribute defined.
  19. For Expression, enter true.
  20. Click Create.
  21. Back in the Policy Label Policy Binding page, click Bind.
  22. Click Done to finish creating the Policy Label for KBA Registration.
  23. Back in the “Check for SSPR Registration” Policy Label Policy Binding page, click Bind.
  24. To handle users that are already registered, click Add Binding.
  25. Next to the Select Policy field, click Add to create an Authentication Policy.
  26. Name the Policy to indicate that no authentication is occurring. This policy always returns true and is necessary to avoid errors for users that are already registered.
  27. Change Action Type to NO_AUTHN.
  28. In the Expression box, enter true.
  29. Click Create.
  30. Back in the Policy Label Policy Binding screen, click Bind.
  31. Click Done to close the “Check for SSPR Registration” Policy Label.
  32. Click Bind to bind the LDAP Policy and “Check for SSPR Registration” factor to the AAA vServer.

Password Reset

This section creates the following snippet of the full nFactor configuration:

# ** AAA vserver: AAA-SSPR
# **    Login Schema Policy = lschema_password_reset
# **       Priority = 20
# **       Rule = true
# **       Login Schema XML = "/nsconfig/loginschema/LoginSchema/SingleAuthPasswordResetRem.xml"
# **    Adv Authn Policy = LDAP_Password_Reset
# **       Priority = 10
# **       Rule = "aaa.LOGIN.VALUE(\"passwdreset\").EQ(\"1\")"
# **       Action = ldapAction named LDAP-Corp_NoAuth
# **       Goto if failed = NEXT
# **       Next Factor if Success = kba_validation
# **          Login Schema Profile = LSCHEMA_INT
# **          Adv Authn Policy = kba_validation
# **             Priority = 10
# **             Rule = true
# **             Action = ldapAction named LDAP-Corp_NoAuth
# **             Goto if failed = NEXT
# **             Next Factor if Success = email_validation
# **                Login Schema Profile = LSCHEMA_INT
# **                Adv Authn Policy = email_validation
# **                   Priority = 10
# **                   Rule = true
# **                   Action = emailAction named EmailOTP
# **                   Goto if failed = NEXT
# **                   Next Factor if Success = password_reset
# **                      Login Schema Profile = LSCHEMA_INT
# **                      Adv Authn Policy = LDAP-Corp-Adv_NoAuth
# **                         Priority = 100
# **                         Rule = true
# **                         Action = ldapAction named LDAP-Corp_NoAuth
# **                         Goto if failed = NEXT

Do the following to create the Password Reset flow:

  1. In the AAA vServer, in the Advanced Authentication Policies section, click where it says 1 Authentication Policy
  2. Note the Priority number of the existing binding. Our new binding will need a lower priority number so it is evaluated first. Click Add Binding.
  3. Next to the Select Policy field, click Add to create an Authentication Policy.
  4. Name the Authentication Policy to indicate that this policy only applies if the user clicked the Forgot Password link in the Login Schema.
  5. Change Action Type to LDAP.
  6. Change Action to the your LDAP Server that has Authentication unchecked (i.e. not enabled).
  7. In the Expression box, enter the following expression that returns true if the user clicked the link:
    aaa.LOGIN.VALUE("passwdreset").EQ("1")
  8. Click Create.
  9. Back in the Policy Binding page, change the Priority to a number that is lower than the existing Policy Binding.
  10. Next to the Select Next Factor field, click Add to create a Policy Label (i.e. factor)
  11. Name the Policy Label to indicate that this factor will verify the user’s questions and answers.
  12. Leave the Login Schema set to the default LSCHEMA_INT (i.e. no schema) and click Continue.
  13. Next to the Select Policy field, click Add to create an Authentication Policy.
  14. Name the Authentication Policy to indicate that this policy verifies the user’s KBA questions and answers.
  15. Change Action Type to LDAP.
  16. Change Action to your LDAP Server that does not require authentication.
  17. In the Expression box, enter true.
  18. Click Create.
  19. Back in the Policy Label Policy Binding page, if you wish to add email validation, then next to Select Next Factor, click Add to create another Policy Label (factor).
  20. Name the Policy Label to indicate that this factor will perform Email OTP verification.
  21. Leave the Login Schema set to the default LSCHEMA_INT (i.e. no schema) and click Add.
  22. In the Policy Label Policy Binding section, click where it says Click to select.
  23. Click the small circle next to the email policy that you created earlier from the ADC’s command line. It’s not possible to use the GUI to create the email policy.
  24. At the top of the screen, click the blue Select button.
  25. Back in the Policy Label Policy Binding screen, next to the Select Next Factor field, click Add to create another Policy Label (i.e. factor).
  26. Name this Policy Label to indicate that this factor performs the password reset.
  27. Leave the Login Schema set to the default LSCHEMA_INT (i.e. no schema) and click Continue.
  28. In the Policy Label Policy Binding section, next to the Select Policy field, click Add to create an Authentication Policy.
  29. Name the Authentication Policy to indicate that this is your LDAP Policy that has Authentication unchecked (not enabled).
  30. Change Action Type to LDAP.
  31. Change Action to your LDAP Server that has Authentication unchecked.
  32. In the Expression field, enter true.
  33. Click Create.
  34. That’s it for the AAA configuration. Now click Bind and Done several times to return to the AAA vServer.






  35. Click Close to finish the AAA vServer’s Authentication Policy bindings.

Bind to Citrix Gateway

If you are using nFactor configuration with Citrix Gateway, then configure the Authentication Profile like normal.

  1. Go to Citrix Gateway > Virtual Servers and edit a Gateway.
  2. On the right, in the Advanced Settings column, click Authentication Profile to move it to the left.
  3. On the left, in the Authentication Profile section, click Add.
  4. Name the Authentication Profile to indicate that this nFactor Flow does Self-Service Password Reset.
  5. In the Authentication Virtual Server field, click where it says Click to select.
  6. Click the small circle next to the AAA vServer that is configured with your SSPR flow.
  7. At the top of the page, click the blue Select button.
  8. Click Create.
  9. Click OK to select the Authentication Profile.

CLI Commands

Here’s a full sample ADC CLI configuration:

# Certs
# -----
add ssl certKey WildcardCorpCom.cer_CERT_KE_ic1 -cert WildcardCorpCom.cer_ic1 -passcrypt b217e266 -encrypted -encryptmethod ENCMTHD_3

add ssl certKey WildcardCorpCom.cer_CERT_KEY -cert WildcardCorpCom.pfx -key WildcardCorpCom.pfx -inform PFX -passcrypt 52e54af -encrypted -encryptmethod ENCMTHD_3 -expiryMonitor DISABLED

add ssl certKey CorpIntermediate -cert CorpIntermediate.cer -inform DER -passcrypt cc9675 -encrypted -encryptmethod ENCMTHD_3 -expiryMonitor DISABLED

# Cert Links
# ----------
link ssl certKey WildcardCorpCom.cer_CERT_KEY CorpIntermediate

link ssl certKey CorpIntermediate WildcardCorpCom.cer_CERT_KE_ic1


# AAA Global Settings
# -------------------
enable ns feature AAA


# LDAP Actions
# ------------
add authentication ldapAction LDAP-Corp -serverIP 10.2.2.11 -serverPort 636 -ldapBase "dc=corp,dc=local" -ldapBindDn ctxsvc@corp.local -ldapBindDnPassword 868d22c -encrypted -encryptmethod ENCMTHD_3 -ldapLoginName sAMAccountName -groupAttrName memberOf -subAttributeName cn -secType SSL -passwdChange ENABLED -nestedGroupExtraction ON -groupNameIdentifier sAMAccountName -groupSearchAttribute memberOf -groupSearchSubAttribute CN -KBAttribute extensionAttribute1 -alternateEmailAttr extensionAttribute1

add authentication ldapAction LDAP-Corp_NoAuth -serverIP 10.2.2.11 -serverPort 636 -ldapBase "dc=corp,dc=local" -ldapBindDn ctxsvc@corp.local -ldapBindDnPassword a4ffb4 -encrypted -encryptmethod ENCMTHD_3 -ldapLoginName sAMAccountName -groupAttrName memberOf -subAttributeName cn -secType SSL -authentication DISABLED -passwdChange ENABLED -nestedGroupExtraction ON -groupNameIdentifier sAMAccountName -groupSearchAttribute memberOf -groupSearchSubAttribute CN -KBAttribute extensionAttribute1 -alternateEmailAttr extensionAttribute1


# Email (SSPR) Actions
# --------------------
add authentication emailAction EmailOTP -userName admin@corp.local -password 7868e -encrypted -encryptmethod ENCMTHD_3 -serverURL "smtps://mail01.corp.local:587" -content "Your OTP is $code" -emailAddress "aaa.user.attribute(\"alternate_mail\")"


# Advanced Authentication Policies
# --------------------------------
add authentication Policy LDAP-Corp-Adv -rule true -action LDAP-Corp

add authentication Policy kba_validation -rule true -action LDAP-Corp_NoAuth

add authentication Policy LDAP_Password_Reset -rule "aaa.LOGIN.VALUE(\"passwdreset\").EQ(\"1\")" -action LDAP-Corp_NoAuth

add authentication Policy NoAuth -rule true -action NO_AUTHN

add authentication Policy switch_to_kba_register -rule "AAA.USER.ATTRIBUTE(\"kba_registered\").EQ(\"0\")" -action NO_AUTHN

add authentication Policy email_validation -rule true -action EmailOTP

add authentication Policy first_time_login_forced_kba_registration -rule true -action LDAP-Corp

add authentication Policy EmailOTP -rule true -action EmailOTP

add authentication Policy LDAP-Corp-Adv_NoAuth -rule true -action LDAP-Corp_NoAuth


# Login Schemas
# -------------
add authentication loginSchema lschema_KBARegistration -authenticationSchema "/nsconfig/loginschema/LoginSchema/lschema_KBARegistration.xml"

add authentication loginSchema lschema_password_reset -authenticationSchema "/nsconfig/loginschema/LoginSchema/SingleAuthPasswordResetRem.xml"


# Login Schema Policies
# ---------------------
add authentication loginSchemaPolicy lschema_password_reset -rule true -action lschema_password_reset


# Authentication Policy Labels
# ----------------------------
add authentication policylabel kba_registration -loginSchema lschema_KBARegistration

bind authentication policylabel kba_registration -policyName first_time_login_forced_kba_registration -priority 1 -gotoPriorityExpression NEXT

add authentication policylabel check_for_sspr_registration -loginSchema LSCHEMA_INT

bind authentication policylabel check_for_sspr_registration -policyName switch_to_kba_register -priority 1 -gotoPriorityExpression NEXT -nextFactor kba_registration

bind authentication policylabel check_for_sspr_registration -policyName NoAuth -priority 100 -gotoPriorityExpression END

add authentication policylabel password_reset -loginSchema lschema_noschema

bind authentication policylabel password_reset -policyName LDAP-Corp-Adv_NoAuth -priority 100 -gotoPriorityExpression NEXT

add authentication policylabel email_validation -loginSchema lschema_noschema

bind authentication policylabel email_validation -policyName email_validation -priority 10 -gotoPriorityExpression NEXT -nextFactor password_reset

add authentication policylabel kba_validation -loginSchema lschema_noschema

bind authentication policylabel kba_validation -policyName kba_validation -priority 10 -gotoPriorityExpression NEXT -nextFactor email_validation


# Authentication Virtual Servers
# ------------------------------
add authentication vserver AAA-SSPR SSL 0.0.0.0

bind authentication vserver AAA-SSPR -portaltheme RfWebUI

bind authentication vserver AAA-SSPR -policy lschema_password_reset -priority 20 -gotoPriorityExpression END

bind authentication vserver AAA-SSPR -policy LDAP_Password_Reset -priority 10 -nextFactor kba_validation -gotoPriorityExpression NEXT

bind authentication vserver AAA-SSPR -policy LDAP-Corp-Adv -priority 20 -nextFactor check_for_sspr_registration -gotoPriorityExpression NEXT


# Authentication Profiles
# -----------------------
add authentication authnProfile SSPR -authnVsName AAA-SSPR


# NetScaler Gateway Global Bindings
# ---------------------------------
bind vpn global -userDataEncryptionKey WildcardCorpCom.cer_CERT_KEY


# NetScaler Gateway Virtual Servers
# ---------------------------------
add vpn vserver gateway2 SSL 10.2.4.220 443 -Listenpolicy NONE -tcpProfileName nstcp_default_XA_XD_profile -httpProfileName http2 -deploymentType ICA_STOREFRONT -authnProfile SSPR -vserverFqdn gateway3.corp.com

bind vpn vserver gateway2 -portaltheme RfWebUI

bind vpn vserver gateway2 -policy PL_OS_10.2.4.120 -priority 100

bind vpn vserver gateway2 -policy PL_WB_10.2.4.120 -priority 100


# SSL Virtual Servers
# -------------------
bind ssl vserver gateway2 -certkeyName WildcardCorpCom.cer_CERT_KEY

bind ssl vserver AAA-SSPR -certkeyName WildcardCorpCom.cer_CERT_KEY

SSL Virtual Servers – Citrix ADC 13

Last Modified: Apr 26, 2024 @ 7:34 am

Navigation

This page contains generic SSL instructions for all SSL-based Virtual Servers, including: Load Balancing, Citrix Gateway, Content Switching, and AAA.

💡 = Recently Updated

Change Log

Custom Cipher Group

References:

To get an A+ at SSL Labs, create a custom secure cipher group:

  1. Enable SSL Secure Renegotiation:
    1. In the left menu, expand Traffic Management, and then click SSL.
    2. On the right, in the right column, click Change advanced SSL settings.
    3. Find Deny SSL Renegotiation, and set the drop-down to NONSECURE.
    4. Scroll down and click OK.

      set ssl parameter -denySSLReneg NONSECURE
  2. The easiest way to create a cipher group is from the CLI. See Citrix Blogs Scoring an A+ at SSLlabs.com with Citrix NetScaler – Q2 2018 update for cipher group CLI commands. Putty (SSH) to the Citrix ADC and paste the following commands.
    Note: The TLS 1.3 ciphers are not in the Citrix Blog Post. You can add TLS 1.3 ciphers to the cipher group. Make sure the TLS 1.3 ciphers are listed first (top of the list).
    add ssl cipher ssllabs-smw-q2-2018
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.3-AES256-GCM-SHA384 -cipherPriority 1
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.3-CHACHA20-POLY1305-SHA256 -cipherPriority 2
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.3-AES128-GCM-SHA256 -cipherPriority 3
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.2-ECDHE-ECDSA-AES128-GCM-SHA256
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.2-ECDHE-ECDSA-AES256-GCM-SHA384
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.2-ECDHE-ECDSA-AES128-SHA256
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.2-ECDHE-ECDSA-AES256-SHA384
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1-ECDHE-ECDSA-AES128-SHA
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1-ECDHE-ECDSA-AES256-SHA
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.2-ECDHE-RSA-AES128-GCM-SHA256
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.2-ECDHE-RSA-AES256-GCM-SHA384
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.2-ECDHE-RSA-AES-128-SHA256
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.2-ECDHE-RSA-AES-256-SHA384
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1-ECDHE-RSA-AES128-SHA
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1-ECDHE-RSA-AES256-SHA
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.2-DHE-RSA-AES128-GCM-SHA256
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1.2-DHE-RSA-AES256-GCM-SHA384
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1-DHE-RSA-AES-128-CBC-SHA
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1-DHE-RSA-AES-256-CBC-SHA
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1-AES-128-CBC-SHA
    bind ssl cipher ssllabs-smw-q2-2018 -cipherName TLS1-AES-256-CBC-SHA
  3. Or you can create the cipher group using the GUI.
    1. Go to Traffic Management > SSL > Cipher Groups.
    2. On the right, click Add.
    3. Name it SSL Labs or similar.
    4. In the middle, click the button named Add.
    5. Use the search box to find a particular cipher.
    6. Check the box next to one of the results, and click the arrow to move it to the right.
    7. For TLS 1.3 support, add the TLS 1.3 ciphers first.
    8. Then add the ciphers from Citrix Blog Post Scoring an A+ at SSLlabs.com with Citrix NetScaler – Q2 2018 update.
    9. Use the up and down arrows to order the ciphers. Citrix ADC prefers the ciphers on top of the list, so the ciphers at the top of the list should be the most secure ciphers (TLS 1.3).
    10. Click Create when done.

Strict Transport Security – Rewrite Policy Method

To get an A+ at SSLLabs.com, you need to insert the Strict-Transport-Security HTTP header in the responses. Citrix ADC Rewrite Policy is one method of doing this. Another method is to enable HSTS in an SSL Profile, or enable it in SSL Parameters on a SSL vServer.

To create a Rewrite Policy that inserts the Strict-Transport-Security HTTP header:

  1. In the left menu, expand AppExpert, right-click Rewrite, and click Enable Feature.
  2. Create the Rewrite Action:
    1. Go to AppExpert > Rewrite > Actions.
    2. On the right, click Add.
    3. Name the action insert_STS_header or similar.
    4. The Type should be INSERT_HTTP_HEADER.
    5. In the Header Name field, enter Strict-Transport-Security.
    6. In the Expression field, enter the following:
      "max-age=157680000"

    7. Click Create.
  3. Create the Rewrite Policy:
    1. On the left, go to AppExpert > Rewrite > Policies.
    2. On the right, click Add.
    3. Name it insert_STS_header or similar.
    4. Select the previously created Action.
    5. In the Expression box, enter HTTP.REQ.IS_VALID.
    6. Click Create.
  4. Now you can bind this Rewrite Response policy to SSL-based SSL vServers.
    1. Edit an SSL vServer (Gateway vServer, Load Balancing vServer, etc.).
    2. if the Policies section doesn’t exist on the left, then add it from the Advanced Settings column on the right.
    3. On the left, in the Policies section, click the plus icon.
    4. Change the Choose Policy drop-down to Rewrite.
    5. Change the Choose Type drop-down to Response, and click Continue.
    6. In the Select Policy field, click where it says Click to select.
    7. Click the small circle next to insert_STS_header, and then click the blue Select button at the top of the page.
    8. Click Bind.
enable ns feature rewrite

add rewrite action insert_STS_header insert_http_header Strict-Transport-Security "\"max-age=157680000\""

add rewrite policy insert_STS_header true insert_STS_header

bind lb vserver MyvServer -policyName insert_STS_header -priority 100 -gotoPriorityExpression END -type RESPONSE

SSL Profiles – Default and Custom

You can use SSL Profiles to package several SSL settings together and apply the settings package (Profile) to SSL-based Virtual Servers and SSL-based Services. These SSL settings include:

  • Disable SSLv3, TLSv1, TLSv11
  • Bind secure ciphers
  • Bind ECC curves
  • Enable Extended Master Secret (EMS)
  • Enable HSTS (Strict Transport Security), etc.

There are default SSL Profiles, and there are custom SSL Profiles. The default SSL Profiles are disabled by default because they would impact every SSL-based Virtual Server and Service on the appliance. Once default SSL Profiles are enabled, you cannot disable the default SSL Profiles.

  • Some features of custom SSL Profiles require default SSL Profiles to be enabled. For example, you cannot configure ciphers in a custom SSL Profile unless the default SSL Profiles are enabled.

Default SSL Profiles are intended to provide a baseline SSL configuration for all newly created SSL Virtual Servers and SSL Services. You can still create Custom SSL Profiles to override the Default SSL Profiles.

Enabling the default SSL profile will wipe out any SSL configuration on SSL entities (e.g. SSL Virtual Servers) that do not have a custom SSL profile bound. Citrix offers a script that can read your existing SSL entity SSL configuration and convert them to custom SSL Profiles. See Enabling the Default Profiles at Citrix Docs. The default_profile_script can be downloaded from Github. The commands output by the script won’t work until the default SSL Profile is enabled.

To enable Default SSL profiles

Enabling Default SSL Profiles is irreversible.

  1. Make sure you are connected to the appliance NSIP using http, and not https.
  2. In the left menu, go to Traffic Management > SSL.
  3. On the right, in the right column, click Change advanced SSL settings.
  4. Near the bottom, check the box next to Enable Default Profile. Note: this will change SSL settings on all SSL Virtual Servers to match the default SSL profile. You might want to do this during a maintenance window.
  5. Click OK at the bottom of the page.
  6. Click Yes to confirm that you are enabling the Default profile.
  7. If you go back into Change Advanced SSL Settings, notice that the Default Profile is enabled, and there’s no way to disable it.

Edit Default SSL Profile with Recommended SSL Profile Settings

The Default SSL Profile applies to all newly created SSL vServers and all newly created SSL Services.

The Default SSL Profile also applies to the management GUI unless you change the SSL Profile assigned to the management Internal Services.

  1. In the left menu, expand System, and click Profiles.
  2. On the right, switch to the SSL Profile tab.
  3. Edit the default frontend or backend profile.
    • Frontend = client-side connections to SSL Virtual Servers.
    • Backend = server-side connections (SSL Services and Service Groups).
    • ns_default_ssl_profile_frontend is automatically applied to all newly created SSL vServers.
    • ns_default_ssl_profile_backend is automatically applied to all newly created SSL Services.
  4. This section focuses on FrontEnd profiles so edit ns_default_ssl_profile_frontend.
  5. Click the pencil icon in the Basic Settings section.

    1. Scroll all the way down to the Protocol section.
    2. Notice that SSLv3 is already unchecked.
    3. It’s recommended that you uncheck TLSv1 and TLSv11 in the Default FrontEnd SSL profile.
      • If you have a website that requires TLSv1, then you can create another Custom FrontEnd SSL Profile that has TLSv1 enabled and manually bind the new Custom SSL Profile to the SSL vServers that need it.
    4. TLSv13 is an option.

      • You can either enable TLSv13 in this Default SSL Profile, or you can create new Custom SSL Profiles that have TLSv13 enabled.
      • If you enable TLSv13, then make sure your cipher group includes TLS 1.3 ciphers.
      • To log SSL Protocol usage, see NetScaler SSL Protocol’s Used (SSLv3, TLS1.0, etc) at Citrix Discussions.
    5. In ADC 13.0 build 61 and newer, just below the protocols. there is an option to enable Allow Extended Master Secret. Windows machines enforce EMS for resumption. 💡
    6. Find Deny SSL Renegotiation and set it to NONSECURE. To find the setting, press Ctrl+F in your browser and search for it.
    7. To enable Strict Transport Security (HSTS), scroll up a little, and check the box next to HSTS.
    8. Enter 157680000 in the Max Age box.

      • You can either enable HSTS in an SSL Profile, or you can enable HSTS by binding a Rewrite policy to your SSL vServer. Do not use both methods at the same time or else you will end up with two Strict-Transport-Security headers in your responses.
    9. If you do any SSL Offload (SSL on the client side, HTTP on the server side), then you’ll need to enable SSL Redirect. It’s above HSTS. With this option enabled, any 301/302 redirects from the server with HTTP Location headers are rewritten to HTTPS Location headers. You might need this option for StoreFront load balancing if doing SSL Offload (port 80 to the StoreFront servers).

      • This setting might be more appropriate in a Custom SSL Profile instead of the Default SSL Profile.
    10. For SSL vServers that need multiple SSL certificates, you can enable the SNI Enable checkbox.

      • This setting might be more appropriate in a Custom SSL Profile instead of the Default SSL Profile.
    11. Click OK when done modifying the Basic Settings section.
  6. Scroll down to the SSL Ciphers section, and click the pencil icon.

    1. Scroll down, and select your custom cipher group (e.g. SSL Labs). Click the arrow to move it to the right.
      If you enabled TLS 1.3, then make sure your cipher group includes TLS 1.3 ciphers.

    2. In the Configured list on the right, click the minus sign next to the DEFAULT cipher group to remove it from the Configured list.
    3. Click OK to close the SSL Ciphers section.
    4. If you make changes to the Custom Cipher Group, then you might have to edit the SSL Profile, remove the Custom Cipher Group, and rebind it.

Create a custom SSL Profile

You can create Custom SSL Profiles to override the Default SSL Profile.

  1. In the left menu, expand System, and click Profiles.
  2. On the right, switch to the tab named SSL Profile.
  3. To copy the existing settings from the Default SSL Profile, check the box next to ns_default_ssl_profile_frontend and then click the button named Add.

    1. Enter a name.
    2. Change the SSL Profile Type to FrontEnd or BackEnd.
    3. Configure SSL Profile settings as desired (see above for some recommendations).
    4. For example, your Custom SSL Profile can enable TLSv13 protocol and include TLSv13 ciphers.
    5. For example, your Custom SSL Profile can enable Client Authentication (client certificates).
    6. Another example, your Custom SSL Profile can enable SNI.
    7. When done, click OK at the bottom of the page.
  4. When you clone an SSL Profile, the Ciphers are not cloned so you’ll have to rebind the ciphers to the Custom SSL Profile.
  5. After the SSL Profile is created, edit any SSL-based Virtual Server.

    1. If you don’t see the SSL Profile section on the left, then on the right, in the Advanced Settings column, click SSL Profile to add the section to the left.
    2. On the left, scroll down to the SSL Profile section and click the pencil icon.
    3. Select an SSL Profile.
    4. Click OK to close the SSL Profile section.

SSL vServers – Bind Certificate, Bind Cipher Group, Disable SSLv3, Enable STS

If you enabled the Default SSL Profiles feature, you can either leave the vServer configured with the Default SSL Profile; or you can change the vServer to use a Custom SSL Profile.

If you don’t use the Default SSL Profiles feature, then you’ll need to manually configure ciphers and SSL settings on every SSL vServer.

Do the following on every SSL vServer:

  1. When creating an SSL Virtual Server (e.g. SSL Load Balancing vServer), on the left, in the Certificates section, click where it says No Server Certificate.

    1. In the Select Server Certificate field, click where it says Click to select.
    2. Click the small circle next to a certificate, and then click the blue Select button at the top of the page.
    3. Click Bind.

      bind ssl vserver MyvServer -certkeyName MyCert
  2. You can bind a Custom SSL Profile:
    1. Find the SSL Profile section on the left, and click the pencil icon.

      • If you don’t see the SSL Profile section on the left, then add the SSL Profile section from the Advanced Settings column on the right.
    2. Select a Custom SSL Profile, and click OK.
  3. If default SSL Profiles are not enabled:
    1. On the left, in the SSL Parameters section, click the pencil icon. You won’t see this section if Default SSL Profiles are enabled.
    2. Uncheck the box next to SSLv3.
    3. Optionally uncheck the boxes next to TLSv1 and TLSv11.
    4. Make sure TLSv12 is checked.
    5. TLSv13 is an option. If you enable this, then make sure your cipher group includes TLS 1.3 ciphers.
    6. Click OK.

      set ssl vserver MyvServer -ssl3 DISABLED -tls12 ENABLED
    7. Scroll to the SSL Ciphers section, and click the pencil icon.
    8. In the middle, click Add.
    9. Select your custom cipher group. It’s probably at the bottom of the list. If you enabled TLS 1.3, then make sure your cipher group includes TLS 1.3 ciphers. Move the cipher group to the right.
    10. Remove the DEFAULT cipher group from the Configured list

      unbind ssl vserver MyvServer -cipherName ALL
      bind ssl vserver MyvServer -cipherName ssllabs-smw-q2-2018
    11. Then click OK to close the SSL Ciphers section.
  4. SSL Virtual Servers created on newer versions of Citrix ADC will automatically have ECC Curves bound to them. However, if this appliance was upgraded from an older version, then the ECC Curves might not be bound. If you are not using SSL Profile, then on the right, in the Advanced Settings column, click ECC Curve.

    1. On the left, in the ECC Curve section, click where it says No ECC Curve.
    2. In the ECC Curve Binding field, click where it says Click to select.
    3. Click the small circle next to ALL, and then click the blue Select button at the top of the page.
    4. Click Bind.

      bind ssl vserver MyvServer -eccCurveName ALL
  5. If HSTS is not enabled in a bound SSL Profile, you can enable it in SSL Parameters, or you can enable it by binding a Rewrite policy.
  6. To enable HSTS by configuring SSL Parameters:
    1. On the left, find the SSL Parameters section, and click the pencil icon. This section is only present if Default SSL Profiles are not enabled.
    2. In the right column, check the box next to HSTS.
    3. Enter 157680000 in the Max Age box.
    4. Click OK to close SSL Parameters.
  7. If enabling HSTS in an SSL Profile or SSL Parameters causes technical issues, then bind a Rewrite policy instead. Don’t do both methods.
    1. If the Policies section doesn’t exist on the left, then add it from the Advanced Settings column on the right.
    2. On the left, find the Policies section, and click the plus icon.
    3. Change the Choose Policy drop-down to Rewrite.
    4. Change the Choose Type drop-down to Response, and click Continue.
    5. In the Select Policy field, click where it says Click to select.
    6. Click the small circle next to the insert_STS_header policy, and then click the blue Select button at the top of the page.
    7. Click Bind.

      bind lb vserver MyvServer -policyName insert_STS_header -priority 100 -gotoPriorityExpression END -type RESPONSE

If you experience SSL performance problems on a Citrix ADC MPX, Citrix CTX207005 Performance Issues with NetScaler MPX SSL recommends creating and binding the following TCP Profile:

add ns tcpProfile tcp_test -WS ENABLED -SACK ENABLED -maxBurst 20 -initialCwnd 8 -bufferSize 4096000 -flavor BIC -dynamicReceiveBuffering DISABLED -sendBuffsize 4096000

SSL Tests

After you’ve created an SSL Virtual Server and configured SSL settings, run the following test:

SSL Redirect – Methods

There are typically three methods of performing SSL Redirect (http to https) in Citrix ADC:

  • Load Balancing Virtual Server Method – enable SSL Redirect directly on the Load Balancing Virtual Server. This is the easiest method.
    • This option is not available for Gateway Virtual Servers and Content Switching Virtual Servers.
    • There’s nothing in the GUI to indicate that the SSL Virtual Server is also listening on port 80.
  • Down vServer Method – create a new Load Balancing Virtual Server on Port 80, and configure the Redirect URL for when it is down.
    • The Virtual Server must be DOWN for the Redirect to occur. These Virtual Servers are shown as Red instead of Green.
  • Responder Method – create a new Load Balancing Virtual Server on Port 80, and bind a Responder policy that redirects to https.
    • The Responder policy only works if the Virtual Server is UP, which means it is shown as Green.
    • Some setup tasks are required – create the AlwaysUP service, and create the Responder Policy. But once setup is complete, it only requires slightly more steps than the Down vServer method.

SSL Redirect – SSL Load Balancing vServer Method

You can configure SSL Redirect directly in an SSL Load Balancing vServer (port 443) instead of creating a separate HTTP (port 80) Load Balancing vServer.

Limitations:

  • This is only an option for SSL Load Balancing vServers; it’s not configurable in Gateway vServers or Content Switching vServers.
  • Only one Redirect URL can be specified. Alternatively, the Responder method can handle multiple FQDNs to one VIP (e.g. wildcard certificate) and/or IP address URLs.

To configure an SSL Load Balancing vServer to redirect from HTTP to HTTPS:

  1. Edit the SSL Load Balancing vServer (port 443).
  2. In the Basic Settings section, click the pencil icon.
  3. Click More.
  4. In the Redirect from Port field, enter 80.
  5. In the HTTPS Redirect URL field, enter https://MyFQDN.
  6. Click Continue twice.
  7. When you view the list of Load Balancing Virtual Servers, there’s no indication that it’s listening on port 80.

SSL Redirect – Down vServer Method

If you created an SSL Virtual Server that only listens on SSL 443, then users must enter https:// when navigating to the website. To make it easier for the users, create another load balancing Virtual Server on the same VIP, but listens on HTTP 80, and then redirects the user’s browser to reconnect on SSL 443.

The Down Virtual Server Method is easy, but the Redirect Virtual Server must be down in order for the redirect to take effect. Another option is to use Responder policies to perform the redirect.

To create the down Redirect Virtual Server:

  1. On the left, under Traffic Management > Load Balancing, click Virtual Servers.
  2. On the right, right-click an SSL Virtual Server you’ve already created, and click Add. Doing it this way copies some of the data from the already created Virtual Server.
  3. Or, if you are redirecting Citrix Gateway, create a new Load Balancing vServer with the same VIP as the Gateway.
  4. Change the name of the Virtual Server to indicate that this new Virtual Server is an SSL Redirect.
  5. Change the Protocol to HTTP on Port 80.
  6. The IP Address should already be filled in. It must match the original SSL Virtual Server (or Gateway vServer). Click OK.
  7. Don’t bind any services. This vServer must intentionally be marked down so the redirect will take effect. Click Continue.
  8. On the right, in the Advanced Settings column, click Protection.
  9. On the left, in the Protection section, in the Redirect URL field, enter the full URL including https://. For example: https://storefront.corp.com/Citrix/StoreWeb.
  10. Click OK to close the Protection section.
  11. Click Done.
  12. When you view the SSL redirect Virtual Server in the list, it will have a state of DOWN. That’s expected. The Port 80 Virtual Server must be DOWN for this redirect method to work.

SSL Redirect – Responder Method

The Down Virtual Server Method is easy, but the Redirect Virtual Server must be down in order for the redirect to take effect. Another option is to use Responder policies to perform the redirect. The Responder method requires the Redirect Virtual Server to be UP.

Responder Method Setup Tasks

The following setup tasks are performed once no matter how many redirects you want to configure.

Create a dummy Load Balancing service. This dummy service can be bound to multiple Redirect Virtual Servers.

  1. Go to Traffic Management > Load Balancing > Services.
  2. On the right, click Add.
  3. Name the service AlwaysUp or similar.
  4. Enter a fake IP address. Try not to use any IP address owned by the Citrix ADC to avoid being able to access the ADC management web page.
  5. Click the More link.
  6. This dummy service must always be UP, so uncheck the box next to Health Monitoring.
  7. Click OK, and then click Done to close the Load Balancing Service.

    add server 1.1.1.1 1.1.1.1
    add service AlwaysUp 1.1.1.1 HTTP 80 -healthMonitor NO

Create the Responder Action:

  1. On the left, expand AppExpert, and click Responder.
  2. If Responder feature is not enabled, right-click Responder, and click Enable Feature.

    enable ns feature RESPONDER
  3. Under Responder, click Actions.
  4. On the right, click Add.
  5. Give the action a name.
  6. Change the Type to Redirect. If you leave this set to Respond With then it won’t work.
  7. Enter an expression. The following expression redirects to https on the same URL the user entered in the browser. Or you can create a Responder Action with a more specific Target.
    "https://" + HTTP.REQ.HOSTNAME.HTTP_URL_SAFE + HTTP.REQ.URL.PATH_AND_QUERY.HTTP_URL_SAFE

    add responder action http_to_ssl_redirect_responderact redirect "\"https://\" + HTTP.REQ.HOSTNAME.HTTP_URL_SAFE + HTTP.REQ.URL.PATH_AND_QUERY.HTTP_URL_SAFE" -responseStatusCode 302
  8. Click Create.

Create the Responder Policy:

  1. On the left, under Responder, click Policies.
  2. On the right, click Add.
  3. Give the policy a name.
  4. Select the previously created Responder action.
  5. For the expression, enter the following.
    HTTP.REQ.IS_VALID

    add responder policy http_to_ssl_redirect_responderpol HTTP.REQ.IS_VALID http_to_ssl_redirect_responderact
  6. Then click Create.

Enable Redirect using Responder Policy

Repeat this section for each HTTP to HTTPS redirect that you want to configure:

  1. Create a Load Balancing Virtual Server with Protocol HTTP, and Port 80.
  2. The vServer’s VIP should match an existing SSL Virtual Server or existing Citrix Gateway Virtual Server.

  3. Bind the AlwaysUp service.
    1. In the Services and Service Groups section, click where it says No Load Balancing Virtual Server Service Binding.
    2. In the Select Service field, click where it says Click to select.
    3. Check the box next to AlwaysUp, and then click the blue Select button at the top of the page.
    4. Click Bind.
    5. Click Continue to close Services and Service Groups.
  4. Bind the Responder Policy:
    1. On the right, in the Advanced Settings column, click Policies.
    2. On the left, scroll down to the Policies section, and click the plus icon in the top right of the Policies box.
    3. Change the Choose Policy drop-down to Responder. Click Continue.
    4. In the Select Policy field, click where it says Click to select.
    5. Click the small circle next to the redirect Responder policy, and then click the blue Select button at the top of the page.
    6. Click Bind.
    7. Then click Done to close the Load Balancing Virtual Server.

      add lb vserver MyvServer-HTTP-SSLRedirect HTTP 10.2.2.201 80
      
      bind lb vserver storefront.corp.com-HTTP-SSLRedirect AlwaysUp
      
      bind lb vserver storefront.corp.com-HTTP-SSLRedirect -policyName http_to_ssl_redirect_responderpol -priority 100 -gotoPriorityExpression END -type REQUEST
  5. The primary advantage of this method is that the Redirect Virtual Server is UP.

Related Pages

Certificates – Citrix ADC 13

Last Modified: Dec 6, 2020 @ 8:27 am

Navigation

💡 = Recently Updated

Change Log

  • 2020 Dec 6 – 13.0 build 71 and newer shows SANs (Subject Alternative Names) in Certificate Details
  • Updated screenshots and procedures for ADC 13

Convert .PFX Certificate to PEM Format

You can export a certificate (with private key) from Windows, and import it to Citrix ADC.

To export a Windows certificate in .pfx format

  1. If Windows Server 2012 or newer, on the Windows server that has the certificate, you can run certlm.msc to open the Certificates console pointing at Local Computer.
    1. Or, run mmc.exe, manually add the Certificates snap-in, and point it to Local Computer.
  2. Go to Personal > Certificates.
  3. Right-click the certificate, expand All Tasks, and click Export.
  4. On the Welcome to the Certificate Export Wizard page, click Next.
  5. On the Export Private Key page, select Yes, export the private key, and click Next.
  6. On the Export File Format page, click Next.
  7. On the Security page, check the box next to Password, and enter a new temporary password. Click Next.
  8. On the File to Export page, specify a save location and name the .pfx file. Don’t put any spaces in the filename. Click Next.
  9. In the Completing the Certificate Export Wizard page, click Finish.
  10. Click OK when prompted that the export was successful.

Import a .pfx file to Citrix ADC

Citrix ADC 13 imports .pfx files and uses them in their native encrypted format.

To import the .pfx file:

  1. On the Citrix ADC, expand Traffic Management, and click SSL.
  2. If the SSL feature is disabled, right-click the SSL node, and click Enable Feature.
  3. Go to Traffic Management > SSL > Certificates > Server Certificates.
  4. There are four different certificate nodes:
    1. Server Certificates have private keys. These certificates are intended to be bound to SSL vServers.
    2. Client Certificates also have private keys, but they are intended to be bound to Services so Citrix ADC can perform client-certificate authentication against back-end web servers.
    3. CA Certificates don’t have private keys. The CA certificates node contains intermediate certificates that are linked to Server Certificates. CA certificates can also be used for SAML authentication, and to verify client certificates.
    4. Unknown Certificates list the certificates that don’t fall under the other categories. Some SAML certificates (e.g. Azure) show up here.
  5. On the left, click the Server Certificates node.
  6. On the right, click Install.
  7. Give the certificate (Certificate-Key Pair) a name.
  8. Click the drop-down next to Choose File, select Local, and browse to the .pfx file that you exported earlier.
  9. After browsing to the .pfx file, Citrix ADC will prompt you to enter the Password for the .pfx file. Ignore the Key File Name field.
  10. Then click Install.
  11. If you click the information icon next to the new certificate…
  12. You’ll see that Citrix ADC uses the file in native .pfx format. No PEM conversion.
  13. In 13.0 build 71 and newer, if you right-click the certificate and click Details, you can see the SANs (Subject Alternative Names) in the certificate.

  14. You can now link an intermediate certificate to this SSL certificate, and then bind this SSL certificate to SSL and/or Citrix Gateway Virtual Servers.
  15. To automatically backup SSL certificates and receive notification when the certificates are about the expire, deploy Citrix Application Delivery Management (ADM). Also see Citrix CTX213342 How to handle certificate expiry on NetScaler.

To convert PFX to PEM (with Private Key encryption)

If you followed the previous section to import a .pfx in native format, then you can skip this section.

Sometimes you need to convert a .pfx file to PEM format so you can use the PEM certificate on other systems. To use Citrix ADC to convert PFX to PEM, do the following:

  1. In the Citrix ADC Configuration GUI, on the left, expand Traffic Management, and click SSL.
  2. In the right column of the right pane, in the Tools section, click Import PKCS#12.
  3. In the Import PKCS12 File dialog box:
    1. In the Output File Name field, enter a name for a new file where the converted PEM certificate and private key will be placed. This new file is created under /nsconfig/ssl on the Citrix ADC appliance.
    2. In the PKCS12 File field, click Choose File, and browse to the .pfx file.
    3. In the Import Password field, enter the password you specified when you previously exported the .pfx file.
    4. By default, the private key in the new PEM file is unencrypted. To encrypt the private key, change the Encoding Format selection to AES256 or DES3. This causes the new PEM file to be password protected, and encrypted.
    5. Enter a permanent password for the new PEM file, and click OK.
  4. You can use the Manage Certificates / Keys / CSRs link to view the new PEM file.

    1. The new file is usually at the bottom of the page.
    2. Right-click the new file, and click View.
    3. Notice that the Private Key is encrypted.
    4. If you scroll down, notice that the file contains both the certificate, and the RSA Private key.
  5. If you want to use this PEM certificate on a different system, then you can right-click the file and Download it.

Install PEM Certificate on Citrix ADC

If you want to bind the PEM certificate to ADC SSL Virtual Servers, then you must first install the PEM certificate. Or you can import a .pfx file in native format as described earlier.

  1. ADC probably won’t import the PEM certificate file if it contains CA certificates. Download the PEM file.

    1. Edit the downloaded file.
    2. Scroll down. Skip the first certificate. Then delete the rest of the certificates in the file. When done, you should have a Private Key and one Certificate (the first one in the file).
    3. Save the file with a new name. You can upload the file, or browse to it later when installing the certificate.
  2. On the left side of the Citrix ADC Configuration GUI, go to Traffic Management > SSLCertificates > Server Certificates.
  3. On the right, click Install.
  4. In the Certificate-Key Pair Name field, enter a friendly name for this certificate.
  5. In the Certificate File Name field, click the drop-down next to Choose File, and select Local.
  6. Browse to the PEM file that you downloaded and edited to remove the CA certificates.
  7. Citrix ADC will ask you to enter the Password for the encrypted private key.
  8. Ignore the Key File Name since the converted PEM file contains both the certificate and the key.
  9. Click Install.
  10. In 13.0 build 71 and newer, if you right-click the certificate and click Details, you can see the SANs (Subject Alternative Names) in the certificate.

  11. You can now link an intermediate certificate to this SSL certificate, and then bind this SSL certificate to SSL and/or Citrix Gateway Virtual Servers.
  12. To automatically backup SSL certificates and receive notification when the certificates are about the expire, deploy Citrix Application Delivery Management (ADM). Also see Citrix CTX213342 How to handle certificate expiry on NetScaler.
  13. You can also export the certificate files and use them on a different Citrix ADC.

Create Key and Certificate Request

If you want to create free Let’s Encrypt certificates, see John Billekens’ PowerShell script detailed at Let’s Encrypt Certificates on a NetScaler.

You can create a key pair and Certificate Signing Request (CSR) directly on the Citrix ADC appliance. The CSR can then be signed by an internal, or public, Certificate Authority.

Most Certificate Authorities let you add Subject Alternative Names when creating (or purchasing) a signed certificate, and thus there’s no reason to include Subject Alternative Names in the CSR created on Citrix ADC. You typically create a CSR with a single DNS name. Then when submitting the CSR to the Certificate Authority, you type in additional DNS names.

  • For a Microsoft Certificate Authority, you can enter Subject Alternative Names in the Attributes box of the Web Enrollment wizard.
  • For public Certificate Authorities, you purchase a UCC certificate or purchase a certificate option that lets you type in additional names.

To create a key pair on Citrix ADC

  1. On the left, expand Traffic Management, expand SSL, and click SSL Files.
  2. On the right, switch to the tab named Keys.
  3. Click the button named Create RSA Key.
  4. In the Key Filename box, enter a new filename (e.g. wildcard.key). Key pair files typically have a .key extension.
  5. In the Key Size field, enter 2048 bits.
  6. By default, the private key is unencrypted. To encrypt it, set the PEM Encoding Algorithm drop-down to AES256 or DES3.
  7. Enter a password to encrypt the private key.
  8. Click Create.
  9. The new file is probably at the bottom of the list. Select it and click the button named View.
  10. The Private Key should be encrypted with your chosen encoding algorithm.

To create CSR file

  1. Back in the SSL Files page, on the right, switch to the tab named CSRs.
  2. Click the button named Create Certificate Signing Request (CSR).
  3. In the Request File Name field, enter the name of a new CSR file. CSR files typically have .csr or .txt extension.
  4. In the Key Filename field, click Choose File (Appliance) and select the previously created .key file. It’s probably at the bottom of the list.

  5. If the key file is encrypted, enter the password.
  6. You can optionally change the CSR Digest Method to SHA256. This only applies to the CSR and does not affect the CA-signed certificate.
  7. Citrix ADC 13 lets you specify up to three Subject Alternative Names in the CSR. Some Certificate Authorities ignore this field and instead require you to specify the Subject Alternative Names when purchasing the signed certificate. See CTX232305 How to create a SAN CSR in NetScaler 12.0 57.19.
  8. In the Common Name field, enter the FQDN of the SSL enabled-website. If this is a wildcard certificate, enter * for the left part of the FQDN. This is the field that normally must match what users enter into their browser address bars.
  9. In the Organization Name field, enter your official Organization Name.
  10. Enter IT, or similar, as the Organization Unit.
  11. Enter the City name.
  12. In the State field, enter your state name without abbreviating.
  13. Scroll down and click Create.
  14. The new CSR file is at the bottom of the list. You can select the new .csr file, and click the buttons named View or Download.

Get CSR signed by CA, and then install certificate on Citrix ADC

  1. View the CSR file or open the downloaded .csr file with Notepad, and send the contents to your Certificate Authority.

    1. Chrome requires every certificate to have at least one Subject Alternative Name that matches the FQDN entered in Chrome’s address bar. Public CAs will handle this automatically. But for Internal CAs, you typically must specify the Subject Alternative Names manually when signing the certificate.

    2. If the CA asks you for the type of web server, select Apache, or save the CA response as a Base 64 file.
  2. After you get the signed certificate, on the left side of the Citrix ADC Configuration GUI, expand Traffic Management > SSL > Certificates, and click Server Certificates.
  3. On the right, click Install.
  4. In the Certificate-Key Pair Name field, enter a friendly name for this certificate.
  5. In the Certificate File Name field, click the drop-down next to Choose File, and select Local.
  6. Browse to the Base64 (Apache) .cer file you received from the Certificate Authority.
  7. In the Key File Name field, click the drop-down next to Choose File, and select Appliance.
  8. Select the key file you created earlier, and click Open. It’s probably at the bottom of the list.
  9. If the key file is encrypted, enter the password.
  10. Click Install.
  11. The certificate is now added to the list.
  12. In 13.0 build 71 and newer, if you right-click the certificate and click Details, you can see the SANs (Subject Alternative Names) in the certificate.

  13. You can now link an intermediate certificate to this SSL certificate, and then bind this SSL certificate to SSL and/or Citrix Gateway Virtual Servers.
  14. To automatically backup SSL certificates and receive notification when the certificates are about the expire, deploy Citrix Application Delivery Management (ADM). Also see Citrix CTX213342 How to handle certificate expiry on NetScaler.
  15. You can also export the certificate files and use them on a different Citrix ADC.

Intermediate Certificate

If your Server Certificate is signed by an intermediate Certificate Authority, then you must install the intermediate Certificate Authority’s certificate on the Citrix ADC. This Intermediate Certificate then must be linked to the Server Certificate.

Get the correct intermediate certificate

  1. Log into Windows, and double-click the signed certificate file.
  2. On the Certification Path tab, double-click the intermediate certificate (e.g. Go Daddy Secure Certificate Authority. It’s the one in the middle).
  3. On the Details tab, click Copy to File.
  4. In the Welcome to the Certificate Export Wizard page, click Next.
  5. In the Export File Format page, select Base-64 encoded, and click Next.
  6. Give it a file name, and click Next.
  7. In the Completing the Certificate Export Wizard page, click Finish.

To import the intermediate certificate

  1. In the Citrix ADC configuration GUI, expand Traffic Management, expand SSL, expand Certificates, and click CA Certificates.
  2. On the right, click Install.
  3. Name it Intermediate or similar.
  4. Click the arrow next to Choose File, select Local, and browse to the Intermediate certificate file and open it.
  5. Click Install.

Link Intermediate Certificate to Server Certificate

  1. Go back to Traffic Management > SSL > Certificates >Server Certificates.
  2. On the right, right-click the server certificate, and click Link.
  3. The previously imported Intermediate certificate should already be selected. Click OK.
  4. You might be tempted to link the Intermediate certificate to a Root certificate. Don’t do this. Root certificates are installed on client machines, not on Citrix ADC. Citrix ADC must never send the root certificate to the client device. If you run ssllabs.com against your website, SSL Labs might show Contains anchor. If so, then you linked your intermediate to your root when you shouldn’t have.

Export Certificate Files from Citrix ADC

You can easily export certificate files from the Citrix ADC, and import them to a different Citrix ADC.

  1. In the menu, expand Traffic Management, expand SSL, expand Certificates, and click one of the certificate types.
  2. Move your mouse over the certificate you want to export, and then click the information icon on the far left.
  3. Note the file names. There could be one file name or two file names.
  4. On the left, go to Traffic Management > SSL.
  5. On the right, in the right column, click Manage Certificates / Keys / CSRs.
  6. Find the file(s) in the list, right-click the file, and click Download.
    1. You can only download one file at a time.
    2. In the search area, you can enter “Name:myfilename” to filter the list.
    3. You might have to increase the number of files shown per page, or go to a different page.
  7. Also download the files for any linked intermediate certificate.
  8. You can also use WinSCP to download the SSL certificate files from /nsconfig/ssl.
  9. You can now use the downloaded files to install certificates on a different Citrix ADC.

Replace Management Certificate

You can replace the default management certificate with a new trusted management certificate.

High Availability – When a management certificate is installed on one node of a High Availability pair, the management certificate is synchronized to the other node and used for the other node’s NSIP too. So make sure the management certificate matches the DNS names of both nodes. This is easily doable using a Subject Alternative Name certificate. Here are some SAN names the management certificate should match (note: a wildcard certificate won’t match all of these names):

  • The FQDN for each node’s NSIP. Example: adc01.corp.local and adc02.corp.local
  • The shortnames (left label) for each node’s NSIP. Example: adc01 and adc02
  • The NSIP IP address of each node. Example: 192.168.123.14 and 192.168.123.29
  • If you enabled management access on your SNIPs, add names for the SNIPs:
    • FQDN for the SNIP. Example: adc.corp.local
    • Shortname for the SNIP. Example: adc
    • SNIP IP address. Example: 192.168.123.30

If you prefer to create a separate management certificate for each HA node, then see CTP George Spiers How to secure management access to NetScaler and create unique certificates in a highly available setup.

Request Management Certificate

If you are creating a Subject Alternative Name certificate, it’s probably easiest to request a SAN certificate from an internal CA using the MMC Certificates snap-in on a Windows box.:

  1. Open the MMC certificates snap-in by running certlm.msc on a Windows 2012 or newer machine.
  2. Go to Personal, right-click Certificate, expand All Tasks, and click Request New Certificate.
  3. A web server certificate template should let you specify subject information.
  4. In the top half, change the Subject name > Type drop-down to Common Name. Enter a DNS name, and click Add to move it to the right.
  5. In the bottom half, change the Alternative Name > Type drop-down to either DNS or IP address (v4).
  6. Type in different names or IPs as detailed earlier, and click Add to move them to the right.
  7. Switch to the tab named Private Key.
  8. Expand Key Options, and make sure Mark private key as exportable is checked.
  9. Click OK. Then finish Enrolling the certificate.
  10. Export the certificate and Private Key to a .pfx file.

  11. Then follow one of the procedures below to replace the ADC’s management certificate.

Methods of replacing the Management Certificate

There are two methods of replacing the management certificate:

  • In the Citrix ADC GUI, right-click ns-server-certificate, and click Update. This automatically updates all of the Internal Services bindings too. This method is intended for dedicated management certificates, not wildcard certificates. Notes:
    • You cannot rename the ns-server-certificate in the Citrix ADC GUI. It remains as ns-server-certificate.
    • ns-server-certificate cannot be bound to Virtual Servers, so make sure you are replacing it with a dedicated management certificate.
  • Or manually Bind a new management certificate to each of the Internal Services.

Update Certificate Method

The Update Certificate button method is detailed below:

  1. The Update method doesn’t work with .pfx files so you’ll first have to convert your .pfx to PEM.
    1. In the Citrix ADC Configuration GUI, on the left, expand Traffic Management, and click SSL.
    2. In the right column of the right pane, in the Tools section, click Import PKCS#12.
    3. In the Output File Name field, enter a name for a new file where the converted PEM certificate and private key will be placed. This new file is created under /nsconfig/ssl on the Citrix ADC appliance.
    4. In the PKCS12 File field, click Choose File, and browse to the .pfx file.
    5. In the Import Password field, enter the password you specified when you previously exported the .pfx file.
    6. By default, the private key in the new PEM file is unencrypted. To encrypt the private key, change the Encoding Format selection to AES256 or DES3. This causes the new PEM file to be password protected, and encrypted.
    7. Enter a permanent password for the new PEM file, and click OK.
  2. You can’t update the certificate while connected to the Citrix ADC using https, so make sure you connect using http.
  3. On the left, expand Traffic Management, expand SSL, expand Certificates, and click Server Certificates.
  4. On the right, right-click ns-server-certificate, and click Update.
  5. Check the box next to Click to update the certificate and key.
  6. Click Choose File, and browse to the new PEM (not PFX) management certificate. It could be on the appliance, or it could be on your local machine.
  7. Click Yes to update the certificate.
  8. For the Key File Name, browse to the same PEM certificate file.
  9. If the PEM private key is encrypted, enter the password.
  10. Check the box next to No Domain Check. Click OK.
  11. You can now connect to the Citrix ADC using https protocol. The certificate should be valid (no certificate errors).

Manual Binding Method

The manual Binding to Internal Services method is detailed below:

  1. You can’t update the certificate while connected to the Citrix ADC using https, so make sure you connect using http.
  2. On the left, expand Traffic Management, expand SSL, expand Certificates, and click Server Certificates.
  3. On the right, use the Install button to install the new management certificate, which can be .pfx format, or PEM format.

  4. In the menu, expand Traffic Management, expand Load Balancing, and click Services.
  5. On the right, switch to the tab named Internal Services.
  6. Right-click one of the services, and click Edit.
  7. Scroll down to the Certificate section and click where it says 1 Server Certificate.
  8. Click the button named Add Binding.
  9. In the Select Server Certificate field, click where it says Click to select.
  10. Click the small circle next to the new management certificate and then click the blue Select button at the top of the page.
  11. Click Bind.
  12. Click Close.
  13. If Default SSL Profile is not enabled, then you can modify the SSL Parameters and/or Ciphers on each of these Internal Services to disable SSLv3 and bind stronger ciphers.

  14. Repeat for each of the rest of the internal services. There should be at least 6 services. Additional Internal Services are created for SNIPs that have management access enabled.

Force Management SSL

By default, administrators can connect to the NSIP using HTTP or SSL. This section details how to disable HTTP.

  1. Connect to the NSIP using https.
  2. On the left, expand System, expand Network, and click IPs.
  3. On the right, right-click your NetScaler IP, and click Edit.
  4. Near the bottom, check the box next to Secure access only, and then click OK.

    set ns ip 10.2.2.126 -gui SECUREONLY
  5. Repeat this procedure on the secondary appliance.
  6. Repeat for any SNIPs that have management access enabled.

Also see:

SSL Certificate – Update

There are two options for updating a certificate:

  • Create or Import a new certificate to Citrix ADC > Traffic Management > SSL > Certificates > Server Certificates. Then find all of the places the original certificate is bound, and manually replace the original certificate binding with the new certificate. This method is obviously prone to errors.
    • You can right-click a certificate and click Show Bindings to see where the certificate is being used.
  • On Citrix ADC, simply right-click the existing certificate, and click Update. This automatically updates all of the bindings. Much faster and easier.

To update a certificate using the Update method:

  1. Create an updated certificate, and export it as .pfx file (with private key). Don’t install the certificate onto Citrix ADC yet, but instead, simply have access to the .pfx file.
  2. In Citrix ADC, navigate to Traffic Management > SSL > Certificates > Server Certificates.
  3. On the right, right-click the certificate you intend to update, and click Update.
  4. Check the box next to Update the certificate and key.
  5. Click Choose File > Local, and browse to the updated .pfx file.
  6. For Key File Name, browse to the same .pfx file.
  7. Enter the .pfx file password.
  8. Click OK. This will automatically update every Virtual Server on which this certificate is bound.
  9. Click OK when told that cert links were broken.
  10. Intermediate certificate – After replacing the certificate, you might have to update the cert link to a new Intermediate certificate.
    1. Right-click the updated certificate, and click Cert Links, to see if it is currently linked to an intermediate certificate.
    2. If not, right-click the updated certificate, and click Link, to link it to an intermediate certificate. If it doesn’t give you an option to link it to, then you’ll first have to install the new intermediate certificate on the Citrix ADC.

Certificates can also be updated in Citrix Application Delivery Management (ADM).

Certificates can be updated from the CLI by running update ssl certKey MyCert. However, the certificate files must be stored somewhere on the appliance, and already be in PEM format.

Next Steps

System Configuration – NetScaler 14.1 – Citrix ADC 13

Last Modified: Apr 25, 2024 @ 1:39 pm

Navigation

💡 = Recently Updated

Change Log

VPX Virtual Hardware

Magnus Andersson Deploy Citrix ADC VPX On Nutanix AHV

VMware Compatibility (source = Support matrix for Citrix ADC at Citrix Docs)

  • Citrix ADC 13.1 build 21.50 and newer support VMware ESXi 7.0 update 3c.
  • Citrix ADC 13.0 build 71 and newer supports ESXi 7.0.

Download and import VPX:

  1. Download Citrix ADC VPX Release 13.1 or NetScaler Release 14.1 Virtual Appliance.

  2. Download one of the VPX Packages for New Installation.

  3. Extract the downloaded .zip file.
  4. In vSphere Client, Deploy OVF template.

Before powering on the appliance, edit the virtual hardware.

  1. If you are licensed for VPX 1000 or higher, increase the CPU count. VPX 1000 is licensed for 4 vCPUs.
  2. Consider setting Memory to 4 GB for each packet engine. A VPX 200 has one packet engine. A VPX 1000 has three packet engines.
  3. Change the NIC Adapter type to VMXNET3 or SR-IOV.
    • Citrix CTX224576 NetScaler VPX Loses Network Connectivity Intermittently on VMware ESXi After Upgrading to Version 12.0 recommends VMXNET3 as a workaround to network connectivity issues in recent Citrix ADC VPX builds.
    • If you choose to use VMXNET3 instead of E1000, make sure all of the NICs are VMXNET3. You cannot mix NIC types.
    • If you already licensed your appliance, jot down the E1000 MAC address, and configure the new VMXNET3 NIC with the same MAC address as the E1000 NIC so you don’t have to redo your license file.
    • To change the NIC Adapter Type, remove the existing NIC, click OK to close Edit Settings, then go back into Edit Settings and Add New Device.
  4. NetScaler 14.1 build 21 and later let you expand size of Hard Disk 1 and after the reboot NetScaler will automatically expand the /var partition, which you can see by running df -h in the shell.

  5. ADC 13.1 build 21.50 and later let you add a second disk to the ADC for storage of logs and crash files. These log and crash files tend to consume disk space on the main drive, which prevents firmware updates, so moving them to a second disk should allow firmware updates to succeed. You can add the second disk at any time, but pre-existing files will not be moved, so it’s best to add the second disk when the VPX is first deployed.

    • The new disk is mounted under /var/crash.
    • The /var/crash directory has several folders that are symlinked from the first drive to the second drive.

  6. NetScaler 14.1 build 21 and later let you expand size of Hard Disk 1 and after the reboot NetScaler will automatically expand the /var partition, which you can see by running df -h in the shell.

  7. Or see NetScaler Article Manually adjust NetScaler VPX Disk Space to use gpart to increase the partition size.

Auto-Provision IP Address

When importing VPX into a hypervisor, you can use VM advanced configuration parameters to set the NSIP. See CTX128250 How to Auto-Provision NetScaler VPX Appliance on a VMware ESX or ESXi Host, and CTX128236 How To Auto-Provision NetScaler VPX on XenServer.

Power On VPX and configure NSIP

  1. After importing the VPX OVF file and changing the NICs to VMXNET3, power on the Citrix ADC VPX appliance.
  2. Configure the management IP from the VM’s console.
  3. Then point your browser to the management IP using either http or https and login as nsroot with password nsroot.
  4. In ADC 13.0 build 67 and newer, you’ll be prompted to change the default nsroot password.

Customer User Experience Improvement Program

  1. You might be prompted to enable the Customer User Experience Improvement Program. Either click Enable, or click Skip.
  2. You can also enable or disable the Customer Experience Improvement Program by going to System > Settings.
  3. On the right is Change CUXIP Settings.
  4. Make your selection and click OK.
  5. See https://www.carlstalhood.com/delivery-controller-cr-and-licensing/#ceip for additional places where CEIP is enabled.
set system parameter -doppler ENABLED

Welcome Wizard

Citrix ADC has a Welcome! Wizard that lets you set the NSIP, hostname, DNS, licensing, etc. It appears automatically the first time you login.

  1. Click the Subnet IP Address box.
  2. You can either enter a SNIP for one of your production interfaces, or you can click Do it later and add SNIPs later after you configure Port Channels and VLANs.
    Note: If you have a dedicated management network, to prevent the management network from being used for outgoing traffic, don’t put a SNIP on the management subnet.

    add ns ip 10.2.2.60 255.255.255.0 -type SNIP
  3. Click the Host Name, DNS IP Address, Time Zone and NTP Server box.
  4. Enter a hostname. In a High Availability pair each node can have a different hostname. You typically create a DNS record that resolves the hostname to the NSIP (management IP).
  5. Enter one or more DNS Server IP addresses. Use the plus icon on the right to add more servers.
  6. Change the time zone to GMT-05:00-CDT-America/Chicago or similar.
  7. Add one or more NTP Servers.
  8. Click Done.

    set ns hostname ns02
    
    add dns nameServer 10.2.2.11
    
    set ns param -timezone "GMT-06:00-CST-America/Chicago"
  9. Click Yes to save and reboot.
  10. Click the Licenses box.
  11. On the far right side of the screen you’ll see the Host ID. You’ll need this to allocate your licenses at mycitrix.com. See below for detailed instructions on how to allocate the license to this Host ID.
  12. On the left, select Upload license files, and click Browse.
  13. Browse to the license file, open it, and click Reboot when prompted.

    • License files are stored in /nsconfig/license.
  14. After the reboot and logging in, a box will pop up showing you the installed license, including Days to Expiration.
  15. Also look in the top left corner to make sure it doesn’t say ADC VPX (Freemium). The number in the parentheses should match the MPX or VPX model number.

  16. 30 days prior to license expiration will be a banner at the top of the GUI. It also sends an SNMP trap.
  17. In ADC 13.1 build 24 and newer, you can adjust the number of days before expiration by clicking Manage Licenses, and then edit the Notification Settings on the bottom.

Licensing – VPX Mac Address

To license a Citrix ADC VPX appliance, you will need its MAC address.

  1. Go to the Configuration tab.
  2. In the right pane, look down for the Host Id field. This is the MAC address you need for license allocation.
  3. Another option is to SSH to the appliance and run shell.
  4. Then run lmutil lmhostid. The MAC address is returned.

License Allocation at MyCitrix.com

Allocate a Citrix ADC VPX license:

  1. Login to http://mycitrix.com.
  2. On the left, click Manage Licenses.
  3. If you are activating an eval license, at the bottom of the page, click Don’t see a new license and enter the eval license key.

  4. In the blue Find a license by… box, change the drop-down to Product name, enter adc in the text box, and click Search.
  5. Select one of your ADC VPX licenses, open the Select an action menu, and click Allocate licenses.
  6. Change the quantity to 1.
  7. In the Host field, enter the Host ID (Mac Address) you got from your VPX appliance.
  8. Click Create license file.
  9. Click Yes, create license file.
  10. Click Download license file and save it somewhere.

In Citrix ADC Standard Edition or higher, some Citrix Gateway Universal Licenses are included in your Citrix ADC platform license. There is no need to allocate a license file for these built-in licenses.

  • Citrix ADC Standard Edition comes with 500 Gateway Universal licenses
  • Citrix ADC Advanced Edition comes with 1,000 Gateway Universal licenses
  • Citrix ADC Premium Edition comes with unlimited Gateway Universal licenses

Citrix Gateway VPX Enterprise Edition does not come with any Gateway Universal Licenses. Citrix Gateway VPX Enterprise Edition is a Gateway-only edition that has fewer features than Citrix ADC Standard Edition.

If you need more Gateway Universal licenses than your ADC Edition provides, then you can acquire Gateway Universal licenses by purchasing Citrix Virtual Apps and Desktops (CVAD) Premium Edition, Citrix Endpoint Management Enterprise Edition, or a la carte. Then allocate the additional Citrix Gateway Universal licenses at mycitrix.com.

  1. Search for gateway licenses, select Citrix Gateway Universal License, open the Select an action menu, and click Allocate licenses.
  2. Change the quantity, or leave it set to allocate all licenses.
  3. Enter your appliance hostname (not Mac address). If you have two appliances in a HA pair, allocate these licenses to the first appliance hostname, then reallocate them to the second appliance hostname.

    1. To get a Citrix ADC’s hostname, login to the ADC as nsroot, and then click the gear icon on the top right.
    2. In the third row, notice the case sensitive Host Name.
  4. Click Create license file.
  5. Click Yes, create license file.
  6. Click OK when prompted to download your license file. Save it somewhere you know where you are saving it.
  7. If you have two appliances in a High Availability pair with different hostnames then you will need to return the Citrix Gateway Universal licenses and reallocate them to the other hostname. The Select an action menu has a Return allocations option.

Install Licenses on Appliance

If you haven’t already installed licenses on your appliance, then do the following:

  1. If the Setup wizard is open, click the fourth row for Licenses.
  2. Otherwise, in the Citrix ADC Configuration GUI, on the left, expand System, and click Licenses.
  3. On the top right, click Manage Licenses.
  4. Click Add New License.
  5. If you have a license file, select Upload license files, and then click Browse. Select the license file(s), and click Open.

    • License files are stored in /nsconfig/license.
  6. Click Reboot when prompted.


  7. After reboot, log in. If you allocated and installed your license correctly, then the top left should no longer say Freemium.
  8. A window will appear showing the installed license.

    1. Notice that Maximum ICA Users Allowed is set to Unlimited.
    2. Maximum NetScaler Gateway Users Allowed will vary depending on your Citrix ADC Edition.
    3. Days to Expiration is shown.
  9. Note: the Citrix ADC SNMP counter allnic_tot_rx_mbits must remain less than the licensed bandwidth or packets will drop.

VPX 100% CPU

Citrix ADC VPX packet engine consumes 100% of the hypervisor CPU. VPX 200 and lower only have one packet engine, so it’s probably consuming around 50% CPU.

You can change this behavior by doing the following:

  1. On the left, go to System > Settings.
  2. On the right, in the bottom of the second column, click Change VPX Configuration Settings.
  3. Change the CPU Yield drop-down to YES, and click OK.

  4. After making this change, you can see an immediate drop-off in CPU consumption.

Upgrade Firmware

Citrix CTX241500 Citrix ADC Firmware Release Cycle:

  • Versions that end in x.1 (e.g. 11.1, 12.1, 13.1, 14.1 etc.) get three years of maintenance releases after one year of feature releases (new features).
  • Versions that end in x.0 (e.g. 12.0, 13.0, 14.0, etc.) get one year of maintenance releases after one year of feature releases (new features).

See the Health Check article for the latest firmware versions that resolve security vulnerabilities.

License – Newer firmware is enforcing license subscription dates in the license file. If you don’t have pooled licensing, then check the CSS dates in your license file:

  1. On NetScaler ADC, go to /nsconfig/license and edit your license file.
  2. Make sure the left date is newer than today. If not, then you’ll need to go to http://citrix.com/account and re-download the license file for this appliance. The newer license file should have a newer date.

Classic Policies – Gateway Classic Policies (classic authentication policies, classic Session Policies, classic Authorization Policies, etc.) still work in NetScaler 14.1, so it’s not necessary to convert your Gateway Classic Policies to Gateway Advanced Policies before you upgrade.

In ADC 13.1 and newer, some non-Gateway features have been removed. To verify that your config does not use the removed features, see Citrix’s Github for a validation script. NetScaler 14.1 build 17 and newer have an upgrade option in the GUI to Enable NSPEPI Tool to check the config before upgrading.

  1. Go to the root of the Github ADC-scripts repository, click the green Code button and then click Download ZIP. Unzip it after downloading.
  2. Use WinSCP or similar to upload the files to the ADC. In the nspepi folder, upload the files nspepi and check_invalid_config to /netscaler on the ADC. Also upload the \nspepi\nspepi2 folder to the /netscaler/nspepi2 folder on the ADC. Replace the existing files.
  3. SSH to ADC and run shell.
  4. Run the following commands and then review the output:
    cd /netscaler
    chmod +x check_invalid_config
    ./check_invalid_config /nsconfig/ns.conf

NetScaler Console (formerly known as Citrix ADM) can upgrade firmware. NetScaler Console can also schedule the firmware upgrade instead of doing it immediately. NetScaler Console does a precheck to make sure there are no upgrade issues. For more details, see Creating Maintenance Tasks at NetScaler Docs.

To upgrade firmware using the NetScaler GUI (source = Upgrade a high availability pair at NetScaler Docs):

  1. To do an ISSU upgrade (zero downtime, no TCP disconnects), NetScaler Docs says that HA Sync VLAN should be configured. In a typical deployment where NSIP is not on a tagged VLAN, set it to VLAN 1. Set it on each node at System > High Availability > Nodes > 0 > Edit.
  2. Download firmware. Ask your Citrix Partner or Citrix Support for recommended versions and builds. You want the Firmware Build, not the VPX (Virtual Appliance).
    Note: Firmware for NetScaler Gateway is identical to firmware for NetScaler. Either one will work on either appliance type.

  3. Watch the Security Bulletins to determine which versions and builds resolve security issues. You can subscribe to the Security Bulletins at http://support.citrix.com by clicking your avatar on the top right after logging in and then clicking My support alerts.

    • Some security updates require additional configuration. For example, CTX330728 requires the nsapimgr_wr.sh command.
      nsapimgr_wr.sh -ys maxclientForHttpdInternalService=30
  4. Save config – Make sure you Save the config before beginning the upgrade. If you do this on the Primary, then both nodes will save their configs.
  5. License – Transferring the firmware upgrade file to the appliance will be slow unless you license the appliance first. An unlicensed appliance will reduce the maximum upload speed to 20 Mbps.
  6. Backup – Before upgrading the appliance, consider using WinSCP or similar to back up the /flash/nsconfig directory.
  7. Start with the Secondary appliance.
  8. Disk Cleanup – VPXs usually don’t have enough free space to perform the upgrade.

    • NetScaler 14.1 build 21 and later let you expand size of Hard Disk 1 and after the reboot NetScaler will automatically expand the /var partition, which you can see by running df -h in the shell.

    • If you SSH to the appliance, run shell, run cd /var, then you can run the following command to see disk space consumption sorted by highest:
      du -d 1 | sort -n -r

    • /var/nsinstall has old firmware upgrades that can be deleted.
    • Check /var/netscaler/nsbackup for old backup files.
    • A common consumer of disk is the counter files located in /var/nslog.
    • Also look in /var/core for crash dumps.
  9. In the Citrix ADC GUI, with the top left node System selected, on the right, click System Upgrade.
  10. Click Choose File and browse to the build…tgz file.
  11. NetScaler 14.1 build 17 and newer have an upgrade option to Enable NSPEPI Tool to check the config before upgrading.
  12. Click Upgrade.
  13. The firmware will upload.
  14. You should eventually see a System Upgrade window with text in it. If you previously saved your config, then click Yes when asked to reboot. Otherwise, click No, save your config, and then click Reboot.
  15. Go back to the System node. On the right, click the Reboot button.
  16. Click OK to reboot.
  17. After the reboot, after you login, you can see the firmware version by clicking your name on the top right of the browser window.
  18. Once the Secondary is done, login, go to System, and click the Migration button to start the zero downtime upgrade as detailed at NetScaler Docs. This is a new feature in ADC 13.

    • Or go to System > High Availability > Nodes and do a Force Failover.
  19. Click Start Migration. It will take some time for client connections to drain off of the Primary and move to the upgraded appliance.
  20. You can go back into the Migration window and click the link labelled Click to show migration details to see if the failover is complete or not.

  21. Then upgrade the firmware on the former Primary.
  22. Go to System > HA > Nodes and verify the Synchronization State. If one of them is disabled, then edit the node, and check the box next to Secondary node will fetch the configuration from Primary.

To install firmware by using the command-line interface

  1. To upload the software to the Citrix ADC, use a secure FTP client (e.g. WinSCP) to connect to the appliance.
  2. Create a version directory under /var/nsinstall (e.g. /var/nsinstall/14.1.36.27).
  3. Copy the software from your computer to the /var/nsinstall/<version> (e.g. /var/nsinstall/14.1.26.27) directory on the appliance.
  4. Open a Secure Shell (SSH) client (e.g. Putty) to open an SSH connection to the appliance.
  5. At a command prompt, type shell.
  6. At a command prompt, type cd /var/nsinstall/<version> to change to the nsinstall directory.
  7. To view the contents of the directory, type ls.
  8. To unpack the software, type tar -xvzf build_X_XX.tgz, where build_X_XX.tgz is the name of the build to which you want to upgrade.
  9. To start the installation, at a command prompt, type ./installns.
  10. When the installation is complete, restart Citrix ADC.
  11. When the Citrix ADC restarts, at a command prompt type what or show version to verify successful installation.
  12. To failover the pair without downtime, enter migrate ns and wait for connections to drain off the former primary appliance.
  13. Then repeat these steps to upgrade the former primary appliance.

High Availability

Configure High Availability as soon as possible to ensure that almost all configurations are synchronized across the two appliances. The synchronization exceptions are mainly network interface configurations (e.g. LACP).

High Availability will also sync files between the two appliances. See CTX138748 File Synchronization in NetScaler High Availability Setup for more information.

  1. Prepare the secondary appliance:
    1. The secondary appliance must be the same hardware as the primary appliance.
    2. Login to the second appliance and configure a NSIP.
    3. Don’t configure a SNIP. In Step 2, Subnet IP Address, you can click Do It Later to skip the wizard. You’ll get the SNIP later when you pair it with the primary.
    4. Configure Hostname, Time Zone, and NTP Servers.
    5. Don’t configure DNS since you’ll get those addresses when you pair it. However, if NTP points to a DNS name, then NTP won’t work until you pair the appliance.
    6. License the secondary appliance. The new secondary appliance must be the same edition (License Type) as the former primary appliance.

    7. Upgrade firmware on the secondary appliance. The firmware of both nodes must be identical.
  2. On the secondary appliance, go to System > High Availability > Nodes.
  3. On the right, edit the local node.

    1. Change High Availability Status to STAY SECONDARY and click OK. If you don’t do this then you run the risk of losing your config when you pair the appliances.

      set ha node -hastatus STAYSECONDARY
  4. On the primary appliance, on the left, expand System, expand Network, and click Interfaces.

    1. On the right, look for any interface that is currently DOWN.
    2. You need to disable those disconnected interfaces before enabling High Availability. Right-click the disconnected interface, and click Disable. Repeat for the remaining disconnected interfaces.

      show interface
      disable interface 1/1
  5. On the primary appliance, on the left, expand System, expand High Availability, and click Nodes.
  6. On the right, edit node 0.

    1. Change the High Availability Status to STAY PRIMARY, and click OK.
  7. On the right, click Add.

    1. Enter the other Citrix ADC’s NSIP address.
    2. Enter the other Citrix ADC’s login credentials, and click Create.

      add ha node 1 192.168.123.14
      Note: this CLI command must be run separately on each appliance.
  8. If you click the refresh icon near the top right, Synchronization State will probably say IN PROGRESS. Keep refreshing until it says SUCCESS.
  9. Edit Node ID 0 (the local appliance).

    1. Change High Availability State back to ENABLED.
    2. Under Fail-safe Mode, check the box next to Maintain one primary node even when both nodes are unhealthy.
    3. Scroll down, and click OK.

      set ha node -failSafe ON
  10. If you login to the Secondary appliance, you might see a message warning you against making changes. Always apply changes to the Primary appliance.
  11. On the secondary appliance, go to System > High Availability > Nodes and edit the local node 0.
  12. Change it from STAY SECONDARY to ENABLED. Also enable Fail-safe Mode. Click OK.
  13. On the new secondary appliance, go to System > Network > Routes, and make sure you don’t have two 0.0.0.0/0.0.0.0 routes. Joining an appliance to an HA pair causes the default route on the primary appliance to sync to the secondary appliance. But, it doesn’t delete the default gateway that was formerly configured on the secondary appliance.
  14. From the Citrix ADC CLI (SSH), run “sh ha node” to see the status. You should see heartbeats on all interfaces. If not, configure VLANs as detailed later..
  15. You can also disable HA heartbeats on specific network interfaces (System > Network > Interfaces).

    1. Note: Make sure HA heartbeats are enabled on at least one interface/channel.
    2. Note: this is an interface configuration, which means this configuration change is not propagated to the other node.
  16. To do an ISSU upgrade (zero downtime, no TCP disconnects), Citrix Docs says that HA Sync VLAN should be configured. In a typical deployment where NSIP is not on a tagged VLAN, set it to VLAN 1. Set it on each node at System > High Availability > Nodes > 0 > Edit.

HA Failover

  1. HA Failover changes the MAC address associated with VIPs and SNIPs. If your firewall (e.g. Cisco ASA) doesn’t like Gratuitous ARP, then see CTX112701 – The Firewall Does not Update the Address Resolution Protocol Table
  2. ADC 13 adds a graceful node Migrate operation which drains sessions instead of drops sessions.
    1. Once migration is started, the two nodes no longer synchronize their configurations. The only way to clear the migration status is to reboot the Secondary. See zero downtime upgrade at Citrix Docs.
    2. Go to System, and click the Migrate button to start the Migrate operation.
    3. Click Start Migration. It will take some time for client connections to drain off of the Primary and move to the upgraded appliance.

      migrate ns
    4. To clear the migration and allow nodes to be failed over again, reboot the secondary.
    5. Go to System > HA > Nodes and verify the Synchronization State. If one of them is disabled, then edit the node, and check the box next to Secondary node will fetch the configuration from Primary.

  3. Instead of Migrate, you can do a Force Failover, which does not require a reboot, and the nodes continue synchronizing.
    1. Go to System > High Availability > Nodes, open the menu named Select Action, and do a Force Failover.

      force ha failover

Port Channels on Physical Citrix ADC MPX

If you are configuring a Citrix ADC MPX (physical appliance), and if you plugged in multiple cables, and if more than one of those cables is configured on the switch for the same VLAN(s), then you must bond the interfaces together by configuring a Port Channel.

  • On the switch, create a Port Channel, preferably with LACP enabled.
  • The Port Channel can be an Access Port (one VLAN), or a Trunk Port (multiple VLANs).
  • On the Citrix ADC, configure LACP on the network interfaces, or create a Channel manually. Both methods are detailed below.

Also see Webinar: Troubleshooting Common Network Related Issues with NetScaler.

LACP Port Channel

To configure Port Channels on a Citrix ADC, you can either enable LACP, or you can configure a Channel manually. If your switch is configured for LACP, do the following on Citrix ADC to enable LACP on the member interfaces.

  1. Go to System > Network > Interfaces.
  2. On the right, edit one of the Port Channel member interfaces.
  3. Scroll down.
  4. Check the box next to Enable LACP.
  5. In the LACP Key field, enter a number. The number you enter here becomes the channel number. For example, if you enter 1, Citrix ADC creates a Channel named LA/1. All member interfaces of the same Port Channel must have the same LACP Key. Click OK when done.
  6. Continue enabling LACP on member interfaces and specifying the key (channel number). If you are connected to two port channels, one set of member interfaces should have LACP Key 1, while the other set of member interfaces should have LACP Key 2.
  7. In an HA pair, you must perform this interface configuration separately on both nodes. The LACP commands are not propagated across the HA pair.
  8. If you go to System > Network > Channels.
  9. You’ll see the LACP Channels on the right. These were created automatically after you set the LACP Key on the interface.
  10. If you edit a Channel, there’s a LACP Details tab that shows you the member interfaces.

Manual Channel

If your switch ports are not configured for LACP, then you can instead create a Channel manually.

  1. Go to System > Network > Channels.
  2. On the right, click Add.
  3. At the top, choose an unused Channel ID (e.g. LA/1).
  4. On the bottom, in the Bind/Unbind section, click Add.
  5. Click the plus icon next to each member interface to move it to the right. Then click Create.

Redundant Interface Set

You can also configure the Citrix ADC for switch-independent teaming. Create a Channel manually, but select a Channel ID starts with LR instead of LA. This is called Link Redundancy or Redundant Interface Set.

Channel Minimum Throughput

Channels can be configured so that a High Availability failover occurs when the Channel throughput drops below a configured value. For example, if you have four members in a Channel, you might want a High Availability failover to occur when two of the member interfaces fail.

  1. Go to System > Network > Channels, and edit a Channel.
  2. Near the top, enter a minimum threshold value in the Throughput field. If the total bonded throughput drops below this level, a High Availability failover will occur.

Trunk Port and High Availability

If you are trunking multiple VLANs across the channel, and if every VLAN is tagged (no native VLAN), then a special configuration is needed to allow High Availability heartbeats across the channel.

  1. Go to System > Network > VLAN.
  2. Add a VLAN object.
  3. Bind the VLAN to a channel or interface. To bind multiple VLANs to a single interface/channel, the VLANs must be tagged.
  4. Configure one of the VLANs as untagged. Only untag one of the VLANs. Which one you untag doesn’t matter. If your switch doesn’t allow untagged packets, don’t worry, we’ll fix that soon.
  5. If your switch doesn’t allow untagged packets, go to System > Network > Channels, and edit the channel.
  6. Scroll down and switch to the tab named Settings.
  7. Set Tag all VLANs to ON to cause Citrix ADC to tag all packets, including the VLAN you formerly marked as untagged.
    • We essentially moved the VLAN tagging from the VLAN to the Channel/Interface, which means VLAN tagging happens lower in the network stack so High Availability heartbeat packets are also tagged.
  8. Note: in an HA pair, you must perform this Tagall configuration separately on both nodes. The Tagall command is not propagated across the HA pair.

Common physical interface configuration

Here is a common Citrix ADC networking configuration for a physical Citrix ADC MPX that is connected to both internal and DMZ.

Note: If the appliance is connected to both DMZ and internal, then be aware that this configuration essentially bypasses (straddles) the DMZ-to-internal firewall. That’s because if a user connects to a public/DMZ VIP, then Citrix ADC could use an internal SNIP to connect to the internal server: in other words, traffic comes into a DMZ VLAN, but goes out an internal VLAN. A more secure approach is to have different appliances for internal and DMZ. Or use Citrix ADC SDX, partitioning, or traffic domains.

  • 0/1 connected to a dedicated management network. NSIP is on this network.
    • No data on 0/1 – 0/1 is not optimized for high throughput so don’t put data traffic on this interface. If you don’t have a dedicated management network, then put your NSIP on one of the other interfaces (1/1, 10/1, LA/1, etc.) and don’t connect any cables to 0/1.
    • No SNIP on management network – To prevent Citrix ADC from using this dedicated management interface for outbound data traffic, don’t put a SNIP on this management network, and configure the default gateway (route 0.0.0.0) to use a router on a different data network (typically the DMZ VLAN). However, if there’s no SNIP on this VLAN, and if the default gateway is on a different network, then there will be asymmetric routing for management traffic, since inbound management traffic goes in 0/1, but reply traffic goes out LA/1 or LA/2. To work around this problem, enable Mac Based Forwarding, or configure Policy Based Routing. Both of these options are detailed in the next section.
    • Management VLAN tagging – It’s easiest if the switch port for this dedicated management interface is an Access Port (untagged). If VLAN tagging is required, then NSVLAN must be configured on the Citrix ADC.
  • 10/1 and 10/2 in a LACP port channel (LA/1) connected to internal VLAN(s). Static routes to internal networks through a router on one of these internal VLANs.
    • Access Port – If only one internal VLAN, configure the switch ports/channel as an Access Port.
    • Trunk Port – If multiple internal VLANs, configure the switch ports/channel as a Trunk Port. Set one of the VLANs as the channel’s Native VLAN so it doesn’t have to be tagged.
    • Tag HA heartbeat packets – If the networking team is unwilling to configure a Native VLAN on the Trunk Port, then Citrix ADC needs special configuration (tagall) to ensure HA heartbeat packets are tagged.
  • 1/1 and 1/2 in a LACP port channel (LA/2) connected to DMZ VLAN(s). The default gateway (route 0.0.0.0) points to a router on a DMZ VLAN so replies can be sent to Internet clients.
    • Access Port – If only one DMZ VLAN, configure the switch ports/channel as an Access Port.
    • Trunk Port – If multiple DMZ VLANs, configure the switch ports/channel as a Trunk Port. Set one of the VLANs as the channel’s Native VLAN so it doesn’t have to be tagged.
    • Tag HA heartbeat packets – If the networking team is unwilling to configure a Native VLAN on the Trunk Port, then Citrix ADC needs special configuration (tagall) to ensure HA heartbeat packets are tagged.

Dedicated Management Subnet

If your Citrix ADC is connected to multiple subnets, then one of those subnets could be a Dedicated Management Subnet. If you have a subnet that is for NSIP only, and if you don’t want to use the NSIP subnet for data traffic, then you’ll want to move the default route off of the NSIP subnet and onto a different data subnet. However, moving the default route breaks traffic from the NSIP. To work around this problem, create PBRs for the NSIP traffic, including both replies from NSIP, and traffic sourced by the NSIP (e.g. Syslog).

Citrix Blog Post Separating NetScaler Management and Data Traffic for DISA STIGs also uses PBRs.

  1. Go to System > Network > PBRs. You can also search the menu for PBRs.
  2. On the right, click Add.
  3. Give the PBR a name (e.g. NSIP)
  4. Set the Next Hop Type drop-down to New.
  5. In the Next Hop field, enter the router IP address that is on the same network as the NSIP.
  6. In the Configure IP section, set the first Operation drop-down to =.
  7. In the Source IP Low field, enter the NSIP. This causes the PBR to match all traffic with NSIP as the Source IP address.
    • In an HA pair, the PBR command is synced and applied to both nodes in the pair. To accommodate this, in the Source IP Low field, enter the lower NSIP address. Then in the Source IP High field, enter the higher NSIP address.
  8. You don’t need anything else.
  9. Scroll down, and click Create. This rule routes any traffic with NSIP as source IP address through a router on the NSIP subnet. The default route will be ignored, but only for NSIP traffic.
  10. DNS traffic is special. To handle DNS traffic sourced by the NSIP, create another PBR by right-clicking the existing one, and clicking Add.
  11. Change the name to NSIP-DNS or similar.
  12. Change the Action drop-down to DENY, which tells ADC that traffic matching this PBR should use normal routing instead of overriding to a different gateway.
  13. Change the Priority to a lower number than the original PBR so this rule is matched before the general NSIP rule. Scroll down.
  14. In the Configure IP section, remove all settings.
  15. In the Configure Protocol section, click the Protocol drop-down, and select UDP (17).
  16. Above the two Destination port fields, change the Operation to =.
  17. In the Destination port Low field, enter 53.
  18. Scroll down, and click Create.
  19. Make sure the DENY PBR is higher in the list (lower priority number) than the ALLOW PBR.
  20. Then open the Action menu, and click Apply.
  21. Click Yes to apply.

    add ns pbr NSIP-DNS DENY -destPort = 53 -nextHop 10.2.2.1 -protocol UDP -priority 5
    add ns pbr NSIP ALLOW -srcIP = 10.2.2.126-10.2.2.127 -nextHop 10.2.2.1
    apply ns pbrs

Floating Management IP

If you want a floating management IP that is always on the Primary appliance, here’s a method of granting management access without adding a SNIP to the management subnet:

  1. Create a Load Balancing Service on HTTP 80 on IP address 127.0.0.1. Note: Citrix ADC doesn’t allow creating a Load Balancing service on IP address 127.0.0.1 and port 443 (SSL).

    • The IP address you enter is 127.0.0.1. When you view the Load Balancing Service, it shows the local NSIP. After a HA failover, the IP Address will change to the other NSIP.
  2. Create a Load Balancing Virtual Server using a VIP on the management subnet. Protocol = SSL. Port number = 443. Bind a certificate.

    1. Bind the loopback:80 service to the Load Balancing Virtual Server. In summary: the front end is 443 SSL, while the LB Service is 80 HTTP.
  3. Add the new VIP to the PBRs so the replies go out the correct interface. Re-apply the PBRs after you modify them.


  4. You should then be able to point your browser to https://Step2VIP to manage the appliance.
  5. You can perform the same loopback trick for 22 SSH. Create a Load Balancing Service on TCP 22 on IP address 127.0.0.1.
  6. Create a Load Balancing Virtual Server using the same management VIP specified earlier. Protocol = TCP. Port number = 22.

    1. Bind the loopback:TCP:22 service to the Load Balancing Virtual Server.
  7. You should then be able to point your SSH Client to <Step2VIP> to manage the appliance.
  8. CLI Commands for the floating management VIP:
    add service mgmt 127.0.0.1 HTTP 80
    add service mgmt-SSH 127.0.0.1 TCP 22
    add lb vserver mgmt-SSL SSL 10.2.2.128 443
    add lb vserver mgmt-SSH TCP 10.2.2.128 22
    bind lb vserver mgmt-SSL mgmt
    bind lb vserver mgmt-SSH mgmt-SSH
    set ns pbr NSIP-DNS DENY -srcIP = 10.2.2.126-10.2.2.128 -destPort = 53 -protocol UDP -priority 5
    set ns pbr NSIP ALLOW -srcIP = 10.2.2.126-10.2.2.128 -nextHop 10.2.2.1
    apply ns pbrs

Multiple Subnets / Multiple VLANs

Best practices for network configurations at Citrix Docs and Citrix CTX214033 Networking and VLAN Best Practices for NetScaler discusses many of the same topics detailed in this section.

If this is a physical MPX appliance, see the previous Port Channel section first.

If you only connected Citrix ADC to one subnet (one VLAN) then skip ahead to DNS servers.

Configuration Overview

The general configuration process for multiple subnets is this:

  1. Create a SNIP for each subnet/VLAN.
  2. Create a VLAN object for each subnet/VLAN.
    1. Bind the VLAN object to the SNIP for the subnet.
    2. Bind the VLAN object to the Port Channel or single interface that is configured for the VLAN/subnet.

SNIPs for each VLAN

You will need one SNIP for each connected subnet/VLAN. VLAN objects (tagged or untagged) bind the SNIPs to particular interfaces. Citrix ADC uses the SNIP’s subnet mask to assign IP addresses to particular interfaces.

NSIP Subnet

The NSIP subnet is special, so you won’t be able to bind it to a VLAN. Use the following SNIP/VLAN method for any network that does not have the NSIP. The remaining interfaces will be in VLAN 1, which is the VLAN that the NSIP is in. VLAN 1 is only locally significant so it doesn’t matter if the switch is configured with it or not. Just make sure the switch has a native VLAN configured, or configure the interface as an access port. If you require trunking of every VLAN, including the NSIP VLAN, then additional configuration is required (NSVLAN or Tagall).

Configure Subnets/VLANs

To configure Citrix ADC with multiple connected subnets:

  1. Add a subnet IP for every network the Citrix ADC is connected to, except the dedicated management network. Expand System, expand Network, and click IPs.
  2. On the right, click Add.

    1. Enter the Subnet IP Address for this network/subnet. The SNIP will be the source IP address the Citrix ADC will use when communicating with any other service/server on this network. The Subnet IP is also known as the Interface IP for the network. You will need a separate SNIP for each connected network (VLAN).
    2. Enter the netmask for this network.
    3. Ensure the IP Type is set to Subnet IP. Scroll down.

      add ns ip 172.16.1.11 255.255.255.0 -type SNIP
    4. Under Application Access Controls, decide if you want to enable GUI management on this SNIP. This feature can be particularly useful for High Availability pairs, because when you point your browser to the SNIP, only the primary appliance will respond. However, enabling management access on the SNIP can be a security risk, especially if this is a SNIP for a DMZ network.
    5. Click Create when done. Continue adding SNIPs for each connected network (VLAN).

      set ns ip 172.16.1.11 -mgmtAccess ENABLED -telnet DISABLED -ftp DISABLED
  3. On the left, expand System, expand Network, and click VLANs.
  4. On the right, click Add.

    1. Enter a descriptive VLAN ID. The actual VLAN ID only matters if you intend to tag the traffic. If not tagged, then any ID (except 1) will work.
    2. In the Interface Bindings section, check the box next to one physical interface or channel (e.g. LA/1) that is connected to the network.
    3. If this is a trunk port, select Tagged if the switch port/channel is expecting the VLAN to be tagged.
    4. If your switches do not allow untagged packets, then you will need to use the tagall interface option to tag Citrix ADC High Availability heartbeat packets. See CTX122921 Citrix NetScaler Interface Tagging and Flow of High Availability Packets
    5. If you don’t tag the VLAN, then the Citrix ADC interface/channel is removed from VLAN 1, and instead put in this VLAN ID.
    6. Switch to the tab named IP Bindings.
    7. Check the box next to the Subnet IP for this network. This lets Citrix ADC know which interface is used for which IP subnet. Click Create when done.

      add vlan 50
      bind vlan 50 -ifnum LA/1 -IPAddress 172.16.1.11 255.255.255.0
  5. Add static routes to the internal networks through an internal router.
    1. On the left, expand System, expand Network, and click Routes.
    2. On the right, click Add.
    3. Make sure NULL Route is set to No.
    4. Set the Gateway (next hop) to an internal router.
    5. Then click Create.

      add route 10.2.0.0 255.255.0.0 10.2.2.1
  6. The default route should be changed to use a router on the DMZ network (towards the Internet). Before deleting the existing default route, either enable Mac Based Forwarding, or create a Policy Based Route, so that the replies from NSIP can reach your machine. You usually only need to do this for dedicated management networks.
    1. Note: PBR is recommended over MBF, because PBR can handle traffic sourced by NSIP (e.g Syslog traffic), while MBF cannot.
    2. Mac Based Forwarding sends replies out the same interface they came in on. However, MBF ignores the routing table, and doesn’t handle traffic sourced by the NSIP (e.g. LDAP traffic). To enable MBF:
      1. On the left, expand System, and click Settings.
      2. On the right, in the left column, click Configure modes.
      3. Check the box next to MAC Based Forwarding (MBF), and click OK. More info on MAC Based Forwarding can be found at Citrix CTX1329532 FAQ: Citrix NetScaler MAC Based Forwarding (MBF).

        enable mode mbf
  7. Go back to System > Network > Routes.
    1. On the right, delete the 0.0.0.0 route. Don’t do this unless the Citrix ADC has a route, PBR, or MBF to the IP address of the machine you are running the browser on.

      rm route 0.0.0.0 0.0.0.0 10.2.2.1
    2. Then click Add.
    3. Set the Network to 0.0.0.0, and the Netmask to 0.0.0.0.
    4. Make sure NULL Route is set to No.
    5. Enter the IP address of the DMZ (or data) router, and click Create.

      add route 0.0.0.0 0.0.0.0 172.16.1.1

DNS Servers

  1. To configure DNS servers, expand Traffic Management, expand DNS, and click Name Servers.
  2. On the right, click Add.

    1. Enter the IP address of a DNS server, and click Create.
    2. Note: The Citrix ADC must be able ping each of the DNS servers, or they will not be marked as UP. The ping originates from the SNIP.

      add dns nameServer 10.2.2.11
  3. Citrix ADC includes DNS Security Options, which are useful if you use this Citrix ADC to provide DNS services to clients (e.g. DNS Proxy/Load Balancing, GSLB, etc.). You can configure them at Security > DNS Security.

  4. Additional DNS Security Options are detailed at DNS security options at Citrix Docs.

NTP Servers

  1. On the left, expand System, and click NTP Servers.
  2. On the right, click Add.
  3. Enter the IP Address of your NTP Server (or pool.ntp.org), and click Create.

    add ntp server pool.ntp.org
  4. On the right, open the Action menu, and click NTP Synchronization.
  5. Select ENABLED, and click OK. This starts the NTP daemon in BSD to perform the NTP sync.

    enable ntp sync
  6. You can click the System node to view the System Time.
  7. If you need to manually set the time:
    1. SSH (Putty) to the Citrix ADC appliances.
    2. Run shell to access BSD.
    3. Run date to manually set the time. Run date –help to see the syntax.
    4. Ntpdate –u pool.ntp.org will cause an immediate NTP time update. You’ll need to disable NTP Sync before you can run this command.

SYSLOG Server

Citrix CTX120609 NetScaler Log Rotation and Configuration Using Newsyslog

The Citrix ADC will, by default, store a few syslogs on the local appliance. You can create a syslog policy to also send the syslog entries to an external server, like Citrix Application Delivery Management (ADM).

  1. On the left, expand System, expand Auditing, and click Syslog.
  2. On the right, switch to the Servers tab, and click Add.

    1. Enter a name for the Syslog server.
    2. You can change Server Type to Server Domain Name, and enter a FQDN.
    3. Enter the IP Address or FQDN of the SYSLOG server, and 514 as the port.
    4. Configure the Log Levels you’d like to send to it by clicking CUSTOM – typically select everything except DEBUG.
    5. Select your desired Time Zone and Date Format.
    6. You can optionally enable other logging features. User Configurable Log Messages lets you use Responder policies to generate log entries.
    7. Then click Create.

      add audit syslogAction MySyslogServer 10.2.2.12 -logLevel EMERGENCY ALERT CRITICAL ERROR WARNING -timeZone LOCAL_TIME -userDefinedAuditlog YES
      add audit syslogAction MySyslogServer syslog.corp.local -logLevel EMERGENCY ALERT CRITICAL ERROR WARNING -timeZone LOCAL_TIME -userDefinedAuditlog YES
  3. On the right, switch to the Policies tab, and then click Add.

    1. Give the policy a descriptive name,
    2. Change the Expression Type selection to Advanced Policy.
    3. Select the previously created Syslog server.
    4. And then click Create.

      add audit syslogPolicy MySyslogServer true MySyslogServer
  4. While still on the Policies tab, open the Actions menu, and click Classic Policy Global Bindings or Advanced Policy Global Bindings, depending on which one you chose when creating the Syslog policy.

    1. Click Add Binding.
    2. Click where it says Click to select.
    3. Click the radio button next to the Syslog policy you want to bind, and then click the blue Select button at the top of the page.
    4. Change the Priority to 100 or similar.
    5. If you don’t select anything in Global Bind Type, then it defaults to SYSTEM_GLOBAL.
    6. Click Bind.
    7. Click Done.

      bind audit syslogGlobal -policyName MySyslogServer -priority 100
      bind system global MySyslogServer -priority 100

SNMP – MIB, Traps, and Alarms

  1. On the left, expand System, and click SNMP.
  2. On the right, click Change SNMP MIB.

    1. Change the fields as desired. Your SNMP tool (e.g. Citrix Application Delivery Management (ADM)) will read this information. Click OK.
    2. This configuration needs to be repeated on the other node.

      set snmp mib -contact NSAdmins@corp.com -name ns02 -location Corp
  3. Expand System, expand SNMP, and click Community.

    1. On the right, click Add.
    2. Specify a community string, and the Permission, and click Create.

      add snmp community public GET
  4. On the left, under SNMP, click Traps.

    1. On the right, click Add.
    2. Specify a trap destination. The fields will vary for V2 vs V3. Click Create. You’ll have to add the Trap Destinations twice so you can select both Generic and Specific.

      add snmp trap generic 10.2.2.12 -communityName public
      add snmp trap specific 10.2.2.12 -communityName public
  5. On the left, under SNMP, click Managers.

    1. On the right, click Add.
    2. Change the selection to Management Network.
    3. Specify the IP of the Management Host, and click Create.

      add snmp manager 10.2.2.12
  6. The Alarms node allows you to enable SNMP Alarms and configure thresholds.

    1. You can Edit an alarm to set thresholds. For example, CPU-USAGE can be set to 90% alarm, and 50% normal, with a Critical severity.

      set snmp alarm CPU-USAGE -thresholdValue 90 -normalValue 50 -severity Critical
    2. You can also configure the MEMORY alarm.

      set snmp alarm MEMORY -thresholdValue 90 -normalValue 50 -severity Critical

From http://www.slideshare.net/masonke/net-scaler-tcpperformancetuningintheaolnetwork: In addition to the usual OIDs, we have found these very useful to warn of potential problems.

  • ifTotXoffSent – .1.3.6.1.4.1.5951.4.1.1.54.1.43
  • ifnicTxStalls – .1.3.6.1.4.1.5951.4.1.1.54.1.45
  • ifErrRxNoBuffs – .1.3.6.1.4.1.5951.4.1.1.54.1.30
  • ifErrTxNoNSB – .1.3.6.1.4.1.5951.4.1.1.54.1.31

Call Home

Citrix Blog Post – Protect Your NetScaler From Disaster With Call Home!: If you have a physical Citrix ADC (MPX or SDX) with an active support contract, you may optionally enable Call Home to automatically notify Citrix Technical Support of hardware and software failures.

Call Home at Citrix Docs has information on how it work.

From the Citrix ADC release notes: Call Home is now enhanced to send Citrix ADC usage metrics to Citrix Insight Services (CIS) periodically. Citrix collects the data to understand how the appliance works and how to improve the product. By default, Call Home sends the metrics once in every 7 days. For more information, see Call Home at Citrix Docs.

To enable Call Home:

  1. On the left, expand System, and click Diagnostics.
  2. On the right, in the left column, in the Technical Support Tools section, click Call Home.
  3. Check the box next to Enable Call Home.
  4. Optionally enter an email address to receive notifications from Citrix Technical Support. Click OK.
  5. If you go back into Call Home, it should indicate if registration succeeded or failed. Successful registration requires an active support contract.

Change nsroot Password

  1. In ADC 13.0 build 67 and newer, you’ll be forced to change the default nsroot password.
  2. If you want to force strong passwords for local accounts, go to System > Settings, and on the right, click Change Global System Settings

    1. Scroll down to the Password section.
    2. You can change Strong Password to Enable Local, and also specify a Min Password Length. Click OK.
  3. Expand System, expand User Administration, and click Users.
  4. On the right, right-click nsroot, and click Change Password.
  5. Specify a new password and click OK.

    set system user nsroot Passw0rd

TCP, HTTP, SSL, and Security Settings

Citrix Docs Introduction to best practices for Citrix ADC MPX, VPX, and SDX security

Best practice settings:

  1. On the left, expand System, and click Settings.
  2. On the right, in the right column, click Change TCP parameters.

    1. Check the box for Window scaling (near the top) and set the Factor to 8.
    2. Scroll down and check the box for Selective Acknowledgement.
    3. Nagle’s algorithm should not be checked.
    4. Click OK.

      set ns tcpParam -WS ENABLED -SACK ENABLED
  3. On the right, click Change HTTP parameters.

    1. Under Cookie, change the selection to Version1. This causes Citrix ADC to set Cookie expiration to a relative time instead of an absolute time.

      set ns param -cookieversion 1
    2. Check the box next to Drop invalid HTTP requests. Note: this might break some web sites.
    3. Scroll down, and click OK.

      set ns httpParam -dropInvalReqs ON
  4. From Citrix CTX232321 Recommended TCP Profile Settings for Full Tunnel VPN/ICAProxy from NetScaler Gateway 11.1 Onwards:
    1. Expand System, and click Profiles.
    2. On the right, on the tab named TCP Profiles, edit the nstcp_default_profile.
    3. Enable Window Scaling with a factor of 8.
    4. Set Minimum RTO (in millisec)600.
    5. Set TCP Buffer Size (bytes)600000
    6. Set TCP Send Buffer Size (bytes)600000
    7. Change TCP Flavor = BIC.
    8. Enable Selective Acknowledgement. Don’t enable Nagle’s algorithm.
    9. Click OK when done.
  5. You can run the following command to see statistics on the dropped packets:
    nsconmsg -g http_err_noreuse_ -d stats
  6. See CTX209398 Addressing false positives from CBC and MAC vulnerability scans of SSHD to harden SSHD by editing /nsconfig/sshd_config with the following. Then run kill -HUP `cat /var/run/sshd.pid` to restart SSHD.
    Ciphers aes128-ctr,aes192-ctr,aes256-ctr
    MACs hmac-sha1,hmac-ripemd160

Citrix Knowledgebase articles:

The following security configurations are detailed by Jason Samuel at Mitigating DDoS and brute force attacks against a Citrix Netscaler Access Gateway:

  • Maximum logon attempts on Citrix Gateway Virtual Server
  • Rate Limiting for IP.SRC and HTTP.REQ.URL.
  • nstcp_default_XA_XD_profile TCP profile on the Citrix Gateway Virtual Server.
  • Syslog logging
  • External website monitoring
  • Obfuscate the Server header in the HTTP response
  • Disable management access on SNIPs
  • Change nsroot strong password, use LDAP authentication, audit local accounts
  • Don’t enable Enhanced Authentication Feedback
  • SSL – disable SSLv3, deny SSL renegotiation, enable ECDHE ciphers, disable RC4 ciphers.
  • 2-factor authentication
  • Citrix Application Delivery Management (ADM)
  • Review IPS/IDS & Firewall logs

Management Authentication – LDAP

Load balancing of LDAP servers is strongly recommended. If you bind multiple LDAP servers instead of load balancing them, Citrix ADC will try each of the LDAP servers, and for incorrect passwords, will lock out the user sooner than expected. But if you instead load balance your LDAP servers, the authentication attempt will only be sent to one of them.

  1. Expand System, expand Authentication, expand Basic Policies, and then click LDAP.
  2. On the right, switch to the Servers tab. Then click Add.

    1. Enter LDAPS-Corp-Mgmt or similar as the name. If you have multiple domains, you’ll need a separate LDAP Server per domain so make sure you include the domain name. Also, the LDAP policy used for management authentication will be different than the LDAP policy used for Citrix Gateway.
    2. Change the selection to Server IP. Enter the VIP of the Citrix ADC load balancing vServer for LDAP.
    3. Change the Security Type to SSL.
    4. Enter 636 as the Port. Scroll down.
    5. In the Connection Settings section, in the Base DN field, enter your Active Directory DNS domain name in LDAP format.
    6. In the Administrator Bind DN field, enter the credentials of the LDAP bind account in userPrincipalName format.
    7. Enter the Administrator Password (bind account password). Click Test LDAP Reachability. Scroll down.
    8. In the Other Settings section, use the drop-down next to Server Logon Name Attribute, Group Attribute, and Sub Attribute Name to select the default fields for Active Directory.
    9. On the right, check the box next to Allow Password Change.
    10. It is best to restrict access to only members of a specific group. In the Search Filter field, enter memberOf=<GroupDN>. See the example below:
      memberOf=CN=Citrix ADC Administrators,OU=Citrix,DC=corp,DC=local
      You can add :1.2.840.113556.1.4.1941: to the query so it searches through nested groups. Without this, users will need to be direct members of the filtered group.
      memberOf:1.2.840.113556.1.4.1941:=CN=Citrix ADC Administrators,OU=Citrix,DC=corp,DC=local
      An easy way to get the full distinguished name of the group is through Active Directory Users & Computers. Make sure Advanced Features is enabled in the View menu. Double-click the group object. Switch to the Attribute Editor tab. Find Distinguished Name and copy it.
      Scroll down to distinguishedName, double-click it, and then copy it to the clipboard.
      Back on the Citrix ADC, in the Search Filter field, type in memberOf=, and then paste the Distinguished Name right after the equals sign. Don’t worry about spaces.
    11. Scroll down and click More to expand it.
    12. For Nested Group Extraction, if desired, change the selection to Enabled.
    13. Set the Group Name Identifier to samAccountName.
    14. Set Group Search Attribute to –<< New >>–, and enter memberOf.
    15. Set Group Search Sub-Attribute to –<< New >>–, and enter CN.
    16. Example of LDAP Nested Group Search Filter Syntax
    17. Scroll down, and click Create.

      add authentication ldapAction Corp-Mgmt -serverIP 10.2.2.210 -serverPort 636 -ldapBase "dc=corp,dc=local" -ldapBindDn "corp\\ctxsvc" -ldapBindDnPassword Passw0rd -ldapLoginName samaccountname -searchFilter "memberOf=CN=Citrix ADC Admins,CN=Users,DC=corp,DC=local" -groupAttrName memberOf -subAttributeName CN -secType SSL -passwdChange ENABLED
  3. On the left, go to System > Authentication > Advanced Policies > Policy.
  4. On the right, click Add.

    1. Enter the name LDAPS-Corp-Mgmt or similar.
    2. Change the Action Type drop-down to LDAP.
    3. Select the previously created LDAPS-Corp-Mgmt server.
    4. On the bottom, in the Expression area, type in true.
    5. Click Create.

      add authentication Policy LDAPS-Corp-Mgmt -rule true -action LDAPS-Corp-Mgmt
  5. Click Global Bindings in the right pane.

    1. Click where it says Click to select.
    2. Click the radio button next to the newly created LDAP policy, and then click the blue Select button at the top of the page.
    3. Click Bind.
    4. Click Done.

      bind system global LDAPS-Corp-Mgmt -priority 100 -gotoPriorityExpression NEXT
  6. Under System, expand User Administration, and click Groups.

    1. On the right, click Add.
    2. In the Group Name field, enter the case sensitive name of the Active Directory group containing the Citrix ADC administrators.

      • ADC 13.0 build 41 and newer lets you restrict a group to the CLI only. Note that GUI access requires API permissions.
    3. In the Command Policies section, click Bind.
    4. Select the superuser policy, and click Insert.
    5. Scroll down, and click Create.

      add system group "Citrix ADC Admins" -timeout 900
      bind system group "Citrix ADC Admins" -policyName superuser 100
  7. To prevent somebody from creating an nsroot account in LDAP (Active Directory) and then using that external nsroot account to login to ADC, disable external authentication on the local nsroot account.
    1. On the left, go to System > User Administration > Users.
    2. On the right, edit the nsroot user.
    3. At the top of the page, in the System User section, click the pencil icon.
    4. Uncheck the box next to Enable External Authentication and then click Continue.
    5. Click Save and then click Done.
  8. If you logout:
  9. You should be able to login to Citrix ADC using an Active Directory account.

Management Authentication – Two Factor

Citrix ADC supports two factor authentication for management access. The technology is based on nFactor but works in all editions of ADC (no licensing restrictions). Here’s a summary of the configuration steps with more detail coming later:

  1. The first authentication factor must be an Advanced Authentication Policy that is bound globally. Classic Authentication Policies will not work.
  2. Create a Login Schema to ask for the second factor password (i.e. passcode).
    • This Login Schema is for second factor only and has no effect on the first factor. The second factor Login Schema should only ask for a single password prompt. It doesn’t appear to be possible to ask for both factors using the same Login Schema.
    • Login Schema for the second factor does not use the normal nFactor language files and you instead must hard code the password prompt label for the second factor logon field directly in the Login Schema .xml file.

  3. Create an Advanced Authentication Server and Policy for the second factor (e.g. RADIUS).
  4. Create an Authentication Policy Label with Feature Type set to RBA_REQ. This is not the default so make sure you change the Feature Type drop-down field.

    1. When creating the Policy Label, select the Login Schema for the second factor.
    2. Bind the second factor Advanced Authentication Policy to the Policy Label.
  5. Go to Global Bindings for Authentication, edit the existing authentication binding, click Next Factor, and select your new Policy Label. That’s it.

Here are detailed configuration instructions for adding a second authentication factor to the management logon page.

  1. Login Schema XML File:
    1. Point WinSCP to your ADC appliance.
    2. Navigate to /nsconfig/loginschema/LoginSchema and download the SingleAuth.xml file.
    3. Rename the file to MgmtNextFactor.xml or something like that.
    4. Edit the file.
    5. Look for the <Requirement> element with ID of passwd. Then look for the Label and set the Text field to whatever you want displayed on the second password page. Save the file when done.
    6. The Label Text you enter will be shown on the second factor logon page.
    7. In WinSCP, change the directory to /nsconfig/loginschema, which is one directory up from where you downloaded the file.
    8. Upload your modified file.
  2. RADIUS Authentication Server:
    1. Follow the link for instructions to create a RADIUS Server. Only create the Server object. The Policy object will be created later when creating the Authentication Policy Label.
      add authentication radiusAction RADIUSMgmt -serverName 10.2.2.42 -serverPort 1812 -radKey b746744 -encrypted -encryptmethod ENCMTHD_3
  3. On the left, go to System > Authentication > Advanced Policies > Policy Label.
  4. On the right, click Add.
  5. Name the Policy Label MgmtNextFactor or similar.
  6. In the Login Schema field, click Add.

    1. Name the Login Schema MgmtNextFactor or similar.
    2. In the Authentication Schema field, click the pencil icon.
    3. On the left, select the Login Schema .xml file you uploaded earlier.
    4. On the top right, click the blue Select button. Do NOT click Create on the bottom left until you’ve clicked this blue Select button.
    5. The window collapses showing you the Login Schema file that you selected. Now you can click Create.

      add authentication loginSchema MgmtNextFactor -authenticationSchema "/nsconfig/loginschema/MgmtNextFactor.xml"
  7. Back in the Authentication Policy Label screen, notice that you can edit the Login Schema object from here.
  8. Change the Feature Type drop-down to RBA_REQ. If you don’t do this, then you won’t be able to bind this later.
  9. Click Continue.

    add authentication policylabel MgmtNextFactor -type RBA_REQ -loginSchema MgmtNextFactor
  10. In the Policy Label’s Policy Binding field, click Add.

    1. Name the Authentication Policy RADIUSMgmt or similar.
    2. Change the Action Type drop-down to RADIUS.
    3. Select the RADIUS server that you created earlier. Or you can Add one from here.
    4. In the Expression box, enter the word true and then click Create.

      add authentication Policy RADIUSMgmt -rule true -action RADIUSMgmt
  11. Back in the Policy Label Policy Binding screen, click Bind.

    bind authentication policylabel MgmtNextFactor -policyName RADIUSMgmt -priority 100 -gotoPriorityExpression NEXT
  12. The Authentication Policy Label configuration is complete so click Done.
  13. On the left, go to System > Authentication > Advanced Policies > Policy.
  14. On the right, click the Global Bindings button.
  15. You should already have an Advanced Authentication Policy bound globally.
    add authentication ldapAction LDAPS-Corp-Mgmt -serverIP 10.2.2.11 -serverPort 636 -ldapBase "dc=corp,dc=local" -ldapBindDn ctxsvc@corp.local -ldapBindDnPassword 5054fc33f673bf4c5c6 -encrypted -encryptmethod ENCMTHD_3 -ldapLoginName sAMAccountName -groupAttrName memberOf -subAttributeName cn -secType SSL -passwdChange ENABLED
    
    add authentication Policy LDAPS-Corp-Mgmt -rule true -action LDAPS-Corp-Mgmt
    
    bind system global LDAPS-Corp-Mgmt -priority 100 -gotoPriorityExpression END
  16. Right-click your existing global binding and click Edit Binding.
  17. In the Next Factor field, click where it says Click to select.
  18. Click the small circle next to your Management Next Factor Policy Label and then click the blue Select button at the top of the page.
  19. Back in the Policy Binding screen, click Bind.

    bind system global LDAPS-Corp-Mgmt -priority 100 -nextFactor RADIUSMgmt -gotoPriorityExpression END
  20. Click Done to close the Global Authentication Policy Binding screen.

CLI Prompt

  1. When you connect to the Citrix ADC CLI prompt, by default, the prompt is just a >.
  2. You can run set cli prompt %u@%h to make it the same as a UNIX prompt. See Citrix Docs for the cli prompt syntax.

Backup and Restore

Citrix Application Delivery Management (ADM) can automatically backup your instances. Or you can do it manually:

  1. Save the configuration.
  2. On the left, expand System, and click Backup and Restore.
  3. On the right, click Backup/Import.
  4. Give the backup file a name.
  5. For Level, select Full, and click Backup.
  6. Once the backup is complete, you can download the file.

For a PowerShell script, see John Billekens Create offline backups of the NetScaler config

To restore:

  1. If you want to restore the system, and if the backup file is not currently on the appliance, click the Backup/Import button.
  2. Change the selection to Import.
  3. Browse Local to the previously downloaded backup file.
  4. Then click Backup. This uploads the file to the appliance and adds it to the list of backup files.
  5. Now you can select the backup, and click Restore.

Next Steps

Citrix ADC SDX 14.1 and 13

Last Modified: Mar 6, 2024 @ 6:11 am

Navigation

Change Log

Overview

Citrix CTX226732 Introduction to Citrix NetScaler SDX.

Citrix ADC SDX is normal Citrix ADC hardware, but runs XenServer hypervisor, and several virtual machines that are listed below:

  • Service VM (aka Management Service, aka SVM) – every SDX comes with this Virtual Machine. The SVM enables the SDX Administrator to create additional VMs on XenServer. It’s analogous to vCenter, except each SDX has its own SVM.
    • It’s not possible to build this VM yourself. If it something happens to it, your only choice is to do a factory reset on the physical appliance, which deletes all local virtual machines, and recreates the Service VM.
    • Each Service VM only manages the VMs on the local SDX. Each SDX has its own Service VM. To manage multiple SDXs, use Citrix Application Delivery Management (ADM).
    • XenServer on SDX is a special build. Do not attempt to directly upgrade XenServer, patch XenServer, configure XenServer, etc. Instead, all upgrades and configurations should be performed by the Service VM.
  • Citrix ADC VPX Instances – you create one or more Citrix ADC instances on top of XenServer.
    • The number of Citrix ADC instances you can create is limited by your SDX license. Most models let you buy more instances.
    • The physical resources (CPU, Memory, NICs, SSL Chips, FIPS HSM) of the SDX are partitioned to the different instances.
    • The amount of bandwidth (throughput) available to the VPX instances depends on your license. For example, the 14040 SDX license gives you 40 Gbps of throughput, which is partitioned across the instances.
    • The Citrix ADC instances are created from a normal XenServer .xva template.
    • Each VPX has its own NSIP. Once the VPX is provisioned, you connect to the NSIP, and configure it like a normal Citrix ADC .

If the top left of the window says SDX, then you are logged into the Management Service (aka Service VM, aka SVM). If it says VPX, then you are logged into an instance.

High Availability – Citrix ADC SDX does not have any High Availability capability at the XenServer or SVM layer. In other words, every SDX is completely standalone. To achieve HA, you create Citrix ADC VPX instances on two separate SDXs, and pair the VPX instances in the normal fashion. See Citrix ADC High Availability.

Why Citrix ADC VPX on top of SDX instead of normal hypervisors?

  • VPX on SDX gets physical access to SSL chips. These SSL ASICs are not available on normal hypervisors. SSL Chips provide significantly higher SSL throughput than normal hypervisors.
  • VPX on SDX gets SR-IOV access to the Network interfaces. This enables full 50 Gbps throughput to a single VM.
  • The SDX NICs can filter VLANs to different instances, thus ensuring that VPX instances cannot cross security boundaries by adding the wrong VLANs.
  • Some SDXs have Hardware Security Modules (HSM) for FIPS compliance. The VPXs on SDX can utilize this hardware security resource.

SDX Networking

  • Management port – Every SDX has a 0/1 port.
    • The SVM and XenServer management IP are on this NIC.
    • You need a minimum of two IPs on a management network connected to the 0/1 port.
    • SVM and XenServer cannot use any of the data ports for management.
  • LOM port – Every SDX has a Lights Out Management (LOM) port.
    • The LOM port gives you out-of-band console access to XenServer. Once you’re on XenServer, you can use Xen commands to see the SVM console, and/or VPX consoles.
  • Data ports – The remaining interfaces can be aggregated into port channels. Port channels are configured at XenServer, not from inside the VPXs. Use the Service VM to create channels, and then connect the VPXs to the channels.
  • VPX networking – When VPXs are created, you specify which physical ports to connect the virtual machine to.
    • If you want the VPX NSIP to be on the same subnet as SVM and XenServer, then connect the VPX to 0/1.
    • Connect the VPX to one or more LA/x interfaces (port channels).
    • Once the VPX is created, log into it, and create VLAN objects in the normal fashion. VLAN tagging is handled by the VPX, not XenServer.
    • On SVM, when creating the VPX instance, you can specify a list of allowed VLANs. The VPX administrator is only allowed to add VLANs that are in this list.
  • SVM to NSIP – SVM must be able to communicate with every VPX NSIP. If VPX NSIP is on a different subnet than SVM, then ensure that routing/firewall allows this connection.
    • SDX 13 has a new internal network for SVM-to-VPX communication, thus eliminating the need for VPX and SVM to be on the same subnet.

LOM IP Configuration

For new SDX models 16000 / 9100, LOM is locked for external out of band access by default on power reset. Please run the following command in XS/Dom0 shell. More info at CTX477557 On new SDX, LOM access via the GUI works but configuration settings are disabled / read-only.
/usr/sbin/sdx_bmc_unlock.sh

There are two ways to set the IP address of the Lights Out Module (LOM):

  • ipmitool from the NetScaler SDX XenServer command line
    • For MPX, you can run ipmitool from the BSD shell.
  • Crossover Ethernet cable from a laptop with an IP address in the 192.168.1.0 network.

Ipmitool Method:

  1. For SDX, SSH to the XenServer IP address (not the Service VM IP).
    • For MPX, SSH to the Citrix ADC NSIP.
  2. Default XenServer credentials are root/nsroot.
    1. Default MPX credentials are nsroot/nsroot.
  3. If MPX, run shell. XenServer is already in the shell.
  4. Run the following:
    ipmitool lan set 1 ipaddr x.x.x.x
    ipmitool lan set 1 netmask 255.255.255.0
    ipmitool lan set 1 defgw ipaddr x.x.x.x

  5. You should now be able to connect to the LOM using a browser.

Laptop method:

  1. Configure a laptop with static IP address 192.168.1.10 and connect it to the Lights Out Module port.
  2. In a Web browser, type the IP address of the LOM port. For initial configuration, type the LOM port’s default address: http://192.168.1.3
  3. In the User Name and Password boxes, type the administrator credentials. The default username and password are nsroot/nsroot.
  4. In the Menu bar, click Configuration, and then click Network.
  5. Under Options, click Network, and type values for the following parameters:
    1. IP Address—The IP address of the LOM port.
    2. Subnet Mask—The mask used to define the subnet of the LOM port.
    3. Default Gateway—The IP address of the router that connects the appliance to the network.
  6. Click Save.
  7. Disconnect the laptop, and instead connect a cable from a switch to the Lights Out Module.

LOM Firmware Upgrade

The LOM firmware at https://www.citrix.com/downloads/citrix-adc/components/lom-firmware-upgrade.html differs depending on the hardware platform. The LOM firmware for the 8000 series is different than the 11000 series and the 14000 series. Do not mix them up.

SDX automatically upgrades the LOM firmware when you upgrade the SDX firmware. For SDX firmware older than 12.0 build 57, update the LOM firmware manually.

Citrix ADC MPX has a new method for updating LOM as detailed at CTX218264 How to Upgrade the LOM Firmware on Any NetScaler MPX Platform

To manually update the LOM firmware:

  1. Determine which firmware level you are currently running. You can point your browser to the LOM and login to the see the firmware level. Or you can run ipmitool mc info from the XenServer shell.
  2. If your LOM firmware is older than 3.0.2, follow the instructions at CTX137970 How to Upgrade LOM Firmware on NetScaler 115xx and CloudBridge 4xxx/5xxx Model Families From AMI 2.52 Using CLI to upgrade the firmware.
  3. If your LOM firmware is version 3.02 or later, follow the instructions at CTX218514 How to Upgrade LOM Firmware to the latest 14k_xxx revision on NetScaler 14xxx, NetScaler 25xxx Model Families, CloudBridge 5100, T1120 and T1300 to upgrade the firmware.
  4. If your firmware is version 3.0.2 or later, you can upgrade to 3.39. Download LOM firmware.
  5. In the LOM, click the Maintenance menu and then click Firmware Update.
  6. On the right, click Enter Update Mode.
  7. Click OK when prompted to enter update mode.
  8. Click Choose File, and browse to the extracted bin file.
  9. After the file is uploaded, click Upload Firmware.
  10. Click Start Upgrade.
  11. The Upgrade progress will be displayed.
  12. After upgrade is complete, click OK to acknowledge the 1 minute message.
  13. The LOM will reboot.
  14. After the reboot, login and notice that the LOM firmware is now 3.39.

SDX IP Configuration

The default IP address for the SVM Management Service is 192.168.100.1/16 bound to interface 0/1. Use a laptop with crossover cable to reconfigure the IP. Point your browser to http://192.168.100.1. Default login is nsroot/nsroot.

The default IP address for XenServer is 192.168.100.2/16. Default login is root/nsroot.

  • There should be no need to connect to XenServer directly. Instead, all XenServer configuration (e.g. create new virtual machine) is performed through the Management Service (SVM).
  • When you set the SVM’s IP Address, there is also a field to also set the XenServer IP address (aka Application supportability IP). XenServer IP and SVM Management Service IP must be on the same subnet.

To change the XenServer IP, make the change through the SVM as detailed below:

  1. Point a browser to http://192.168.100.1, and login as nsroot/nsroot.
  2. When you first login to the SDX SVM Management Service, the Welcome! Wizard appears. Click the first row for Management Network.
  3. Configure the IP addresses.
    1. Application supportability IP = XenServer. You’ll almost never connect to this IP.
    2. Appliance Management IP = SVM (Management Service). This is the IP you’ll normally use to manage SDX.
    3. The bottom has an Additional DNS checkbox that lets you enter more DNS servers.
    4. You can change the nsroot password at this time, or change it later after LDAP is configured.
  4. Click Done.
  5. Click the System Settings box.
  6. Enter a Host Name.
  7. You can optionally check Secure Access only to ensure that administrators must use https when connecting to the SVM.
  8. Select the time zone, and click Continue.
  9. Click the Licenses box.
  10. Click Add License File.
  11. Allocate Citrix ADC SDX licenses at mycitrix.com using the normal license allocation process.
    1. The SDX license defines the number of instances you can create.
    2. The SDX license also defines the amount of throughput available to the instances.
    3. The SDX license is allocated to ANY, which means you can use the same license on all SDX hardware, assuming all of them are purchased with the same license model.
  12. Click Browse to upload the license file. After uploading, click Finish and it should apply automatically.
  13. Or you can click Apply Licenses.
  14. Then click Continue to close the Welcome! wizard.

Another way to change the SVM Management Service IP address is through the serial port. This is actually the XenServer Dom0 console. Once logged in to XenServer, run ssh 169.254.0.10 to access the SVM Management Service virtual machine. Then follow instructions at CTX130496 How to Configure the Service Virtual Machine on NetScaler SDX to change the IP.

The console of the SVM Management Service virtual machine can be reached by running the following command in the XenServer Dom0 shell (SSH or console):

xe vm-list params=name-label,dom-id name-label="Management Service VM"

Then run /usr/lib64/xen/bin/xenconsole <dom-id>

SDX Platform Software Bundle

If your Citrix ADC SDX is not version 11 or newer, and if your Citrix ADC SDX is running 10.5 build 57 or later, then do the following:

  1. Go to Management Service > Software Images, and upload a Single Bundle older than 13. The single bundle is around 1.5 GB.
    Note: Direct upgrade from version 10.5 to 13.0 is not supported. You must first upgrade from 10.5 to 11.0 or 11.1 or 12.0 or 12.1, and then upgrade to SDX 13.0.
  2. On the left, click System.
  3. On the right, click Upgrade Management Service. Select the Single Bundle upgrade file you already uploaded.
  4. Management Service will upgrade and reboot. A few minutes after that, XenServer will be upgraded. Be patient as there’s no notification that the box will reboot again.

Starting with SDX 11.0, all updates are bundled together and installed at once.

  1. Make sure your Management Service (SVM) is running SDX 10.5 build 57 or newer.
    Note: Direct upgrade from version 10.5 to 13.0 is not supported. You must first upgrade from 10.5 to 11.0 or 11.1 or 12.0 or 12.1, and then upgrade to SDX 13.0.
  2. Download the latest SDX Platform Software bundle from Downloads > Citrix ADC > Release 13 > Service Delivery Appliances.

  3. Login to the SDX Management Service and go to Configuration > System.
  4. On the right, in the right column, click Upgrade Appliance.
  5. Browse to the build-sdx-13.0.tgz software bundle, and click OK.
  6. It will take some time to upload the SDX image.
  7. It should show you the estimated installation time.
  8. Check boxes next to the instances that need configs saved.
  9. Click Upgrade.
  10. Click Yes to continue with the upgrade.
  11. The SVM Management Service displays installation progress. It will take a while.
  12. Once the upgrade is complete, click Login.

  13. If you click the Configuration tab, the Information page will be displayed showing the version of XenServer, Management Service (Build), etc.

FIPS

If your SDX is a FIPS appliance, see Citrix Blog Post Meet Security Compliance and Be Scalable with NetScaler FIPS SDX for detailed HSM setup instructions:

  1. Zeroize the HSM
  2. Upgrade HSM firmware
  3. Create HSM partitions
  4. Create Citrix ADC instance and attach HSM partition:
    • Only one CPU core
  5. From inside Citrix ADC instance:
    1. Reset FIPS
    2. Initialize FIPS
    3. Create FIPS Key
    4. Create HA Pair and synchronize FIPS

DNS Servers

To add more than one DNS server, do the following:

  1. In the SVM Management Service, on the left, click System.
  2. On the right, click Network Configuration.
  3. On the bottom, there’s a checkbox for Additional DNS that lets you put in more DNS servers.
  4. Click OK when done.

SVM Management Service NTP

  1. On the Configuration tab, in the navigation pane, expand System, and then click NTP Servers.
  2. To add a new NTP server, in the right pane, click Add.
  3. In the Create NTP Server dialog box, enter the NTP server name (e.g. pool.ntp.org), and click Create.
  4. Click Yes when prompted to restart NTP Synchronization.
  5. In the right pane, click NTP Synchronization.
  6. In the NTP Synchronization dialog box, select Enable NTP Sync. Click OK.
  7. Click Yes when asked to restart the SVM Management Service. This only restarts the SVM. Other instances/VMs on the same box won’t be affected.

Licensing

If you haven’t already licensed your SDX, you can upload a license file to the SDX appliance.

  1. Login to http://mycitrix.com and go to Manage Licenses.
  2. In the New Licenses section, find a Citrix ADC SDX license, and allocate it. There is no need to specify a hostname. You can use the same license file on multiple SDX appliances.

  3. On the SDX Configuration tab, in the navigation pane, expand System, and then click Licenses.
  4. In the right pane, click Add License File.
  5. Click Browse and upload the allocated license file.
  6. Click Finish.
  7. If you refresh your browser, the number shown on the top left of the window will indicate your licensed model number.

SVM Management Service Alerting

Syslog

  1. On the Configuration tab, expand System > Auditing, and click Syslog Servers.
  2. In the right pane, click the Add button.

    1. Enter a name for the Syslog server.
    2. Enter the IP address of the Syslog server.
    3. Change the Choose Log Level section to Custom, and select log levels.
  3. Click Create.
  4. On the right is Syslog Parameters.
  5. You can configure the Date Format and Time Zone. Click OK.

Mail Notification

  1. On the Configuration tab, expand System > Notifications, and click Email.
  2. In the right pane, on the Email Servers tab, click Add.
  3. Enter the DNS name of the mail server, and click Create.
  4. In the right pane, switch to the tab named Email Distribution List, and click Add.
  5. In the Create Email Distribution List page:
    1. Enter a name for the mail profile.
    2. Select the Email Server to use.
    3. Enter the destination email address (distribution list).
  6. Click Create.
  7. SDX 13 has a Test button for the Distribution List.

System SNMP

  1. Go to System > SNMP.
  2. On the right, click Configure SNMP MIB.
  3. Enter asset information, and click OK. Your SNMP management software will read this information.
  4. Under the SNMP node, configure normal SNMP including: Trap Destinations, Managers, Alarms, etc.

  5. MIBs can be downloaded from the Downloads tab.

Instance SNMP

  1. The instances will send SNMP traps to the Service VM. To get alerted for these traps, in the Configuration page, in the navigation pane, expand Citrix ADC, expand Events, and click Event Rules.
  2. On the right, click Add.

    1. Give the rule a name.
    2. Move the Major and Critical severities to the right.
    3. Scroll down.
    4. For the other sections, if you don’t configure anything then you will receive alerts for all of the devices, categories, and failure objects. If you configure any of them, then only the configured entities will be alerted.
    5. Scroll down.
    6. Click Save.
  3. Select an Email Distribution List, and click Done.

SVM Management Service nsroot Password and AAA

Change nsroot password

  1. On the Configuration tab, in the navigation pane, expand System, expand User Administration, and then click Users.
  2. On the right, in the Users pane, right-click the nsroot user account, and then click Edit.
  3. In the Configure System User dialog box, check the box next to Change Password.
  4. In Password and Confirm Password, enter the password of your choice.
  5. You can optionally Configure User Session Timeout.
  6. Scroll down and click OK.

AAA Authentication

To enable LDAP authentication for the Service VM:

  1. Go to Configuration > System > Authentication > LDAP.
  2. In the right pane, click Add.
  3. This is configured identically to Citrix ADC.
    1. Enter a Load Balancing VIP for LDAP servers.
    2. Change the Security Type to SSL, and Port to 636.
    3. Note: if you want to Validate LDAP Certificate, then there are special instructions for installing the root certificate on the SVM. See Installing CA certificates to the SDX/SVM for LDAPS user authentication at Citrix Discussions for details.
    4. Scroll down.
    5. Enter the Base DN in LDAP format.
    6. Enter the bind account in UPN format, or Domain\Username format, or DN format.
    7. Check the box for Enable Change Password.
    8. Click Retrieve Attributes, and scroll down.
    9. For Server Logon Attribute, select sAMAccountName.
    10. For Group Attribute, select memberOf.
    11. For Sub Attribute Name, select CN.
    12. To prevent unauthorized users from logging in, configure a Search Filter as detailed in the LDAP post. Scroll down.
  4. Click Create.
  5. Expand System, expand User Administration, and click Groups.
  6. On the right, click Add.
  7. In the Create System Group page:
    1. Enter the case sensitive name of the Active Directory group.
    2. Check the box next to System Access.
    3. Configure the Session Timeout.
    4. Optionally Configure User Session Timeout.
  8. Click Create.
  9. On the left, under System, click User Administration.
  10. On the right, click User Lockout Configuration.

    1. If desired, check the box next to Enable User Lockout, and configure the maximum logon attempts. Click OK.
  11. On the left, under System, click Authentication.
  12. On the right, click Authentication Configuration.

    1. Change the Server Type drop-down to EXTERNAL, and click Insert.
    2. Select the LDAP server you created earlier, and click OK at the top of the page.
    3. Make sure Enable fallback is enabled, and click OK.

SSL Certificate and Encryption

Replace SDX SVM Management Service Certificate

To replace the SVM Management Service certificate:

  1. PEM format: The certificate must be in PEM format. The SVM Management Service does not provide any mechanism for converting a PFX file to PEM. You can convert from PFX to PEM by using the Import PKCS#12 task in a Citrix ADC instance.
  2. On the left, click System.
  3. On the right, in the left column, in the Set Up Appliance section, click Install SSL Certificate.
  4. Select the certificate and key files in PEM format. If the key file is encrypted, enter the password. Then click OK.
  5. The SVM Management Service will restart. Only the SVM restarts; the Citrix ADC instances do not restart.

Force HTTPS to the SVM Management Service

  1. Connect to the SVM using HTTPS. You can’t make this upcoming change if you are connected using HTTP.
  2. On the Configuration tab, click System.
  3. On the right, click Change System Settings.
  4. Check the box next to Secure Access Only, and click OK. This forces you to use HTTPS when connecting to the SVM Management Service.

SSL Encrypt SVM Management Service to Citrix ADC Communication

From CTX134973 How to Secure Network Traffic from Service Virtual Machine to NetScaler VPX Instances: Communication from the SVM Management Service to the Citrix ADC VPX instances is HTTP by default. If you want to configure HTTPS access for the Citrix ADC VPX instances, then you have to secure the network traffic between the SVM Management Service and Citrix ADC VPX instances. If you do not secure the network traffic from the SVM Management Service configuration, then the Citrix ADC VPX Instance State appears as Out of Service and the Status shows Inventory from instance failed.

  1. Log on to the SVM Management Service .
  2. On the Configuration tab, click System.
  3. On the right, click Change System Settings.
  4. Change the Communication with Citrix ADC Instance drop-down to https, as shown in the following screen shot:
  5. Run the following command on the Citrix ADC VPX instance, to change the Management Access (-gui) to SECUREONLY:
    set ns ip ipaddress -gui SECUREONLY
  6. Or in the Citrix ADC VPX instance GUI, go to Network > IPs, edit the NSIP, and then check the box next to Secure access only.

SDX/XenServer LACP Channels

For an overview of Citrix ADC SDX networking, see Citrix CTX226732 Introduction to Citrix NetScaler SDX

To use LACP, configure Channels in the SVM Management Service, which creates them in XenServer. Then when provisioning a VPX instance, connect it to the Channel.

  1. In the SVM Management Service, on the Configuration tab, expand System, and click Channels.
  2. On the right, click Add.
  3. In the Create Channel page:
    1. Select a Channel ID.
    2. For Type, select LACP or STATIC. The other two options are for switch independent load balancing and are only supported for the management ports.
    3. In the Interfaces section, move the Channel Member interfaces to the right by clicking the right arrow.
    4. In the Settings section, for LACP you can select Long or Short, depending on switch configuration. Long is the default.
  4. Click Create when done.
  5. Click Yes when asked to proceed.
  6. The channel will then be created on XenServer.

VPX Instances – Provision

Admin profile

Admin profiles specify the nsroot user credentials for the instances. SVM Management Service uses these nsroot credentials later when communicating with the VPX instances to retrieve configuration data.

The default admin profile for an instance specifies a user name of nsroot, and the password is also nsroot. To specify a different nsroot password, create a new admin profile.

  • You can create a single admin profile that is used by all instances. To delegate administration, don’t give out the nsroot password to the instance administrators. One option is to enable LDAP inside the instance before granting access to a different department.
  • When creating an instance, there’s an option to create a non-nsroot account, which has almost the same permissions as nsroot, but leaves out some SDX specific features (e.g interfaces). This is another option for delegating administration to a different team.
  • Or you can create different admin profiles for different instances, which allows you to inform the different departments the nsroot password for their VPX instances.

Important: Do not change the password directly on the Citrix ADC VPX instance. If you do so, the instance becomes unreachable from the SVM Management Service. To change a password, first create a new admin profile, and then modify the Citrix ADC instance, selecting this new profile from the Admin Profile list.

  1. On the Configuration tab, in the navigation pane, expand Citrix ADC, and then click Admin Profiles.
  2. In the Admin Profiles pane, click Add.
  3. In the Create Admin Profile dialog box, set the following parameters:
    • Profile Name*—Name of the admin profile.
    • User Name—User name used to log on to the Citrix ADC instances. The user name of the default profile is nsroot and cannot be changed.
    • Password*—The password used to log on to the Citrix ADC instance. Maximum length: 31 characters.
    • Confirm Password*—The password used to log on to the Citrix ADC instance.
    • Use global settings for Citrix ADC communication – you can uncheck this box and change the protocol to https.
    • SNMP v2 or v3 – for SNMP communication between the SVM and the VPX
    • Community
  4. Click Create. The admin profile you created appears in the Admin Profiles pane.

Upload a Citrix ADC VPX .xva file for XenServer

You must upload a Citrix ADC VPX .xva file to the SDX appliance before provisioning the Citrix ADC VPX instances. XVA files are only used when creating a new instance. Once the instance is created, use normal firmware upgrade procedures.

  1. Go to the Citrix ADC VPX download page and select a VPX Release.
  2. Download the Citrix ADC VPX for XenServer.
  3. After downloading, use 7-zip to extract the .gz file. You can’t upload the .gz file to SVM. You must extract it first.

  4. On the SVM’s Configuration tab, in the navigation pane, expand Citrix ADC, and then click Software Images.
  5. On the right, switch to the tab named XVA Files, and then click Upload.
  6. In the Upload ADC Instance XVA dialog box, click Browse and select the XVA image file that you want to upload. Click Upload.
  7. The XVA image file appears in the XVA Files pane after it is uploaded.

Provision a Citrix ADC instance

  1. In the SVM Management Service, go to the Dashboard page.

    1. On the bottom right, the System Resource Utilization pane shows you the amount of physical resources that are available for allocation.
    2. Click Core Allocation to see the number of cores available for assignment.
    3. Click Crypto Capacity to see the SSL capacity.
  2. On the Configuration tab, in the navigation pane, expand Citrix ADC, and then click Instances.
  3. In the Citrix ADC Instances pane, click Add.
  4. In the Provision Citrix ADC section, enter a name for the instance.
  5. SDX 13 has an option for Manage through internal network which means that the VPX no longer needs NSIP on the same subnet as the SDX SVM.
  6. Enter the NSIP, mask, and Gateway.
  7. Nexthop to Management Service – If the instance’s NSIP is on a different subnet than the SVM IP, and if Manage through internal network is not checked, and if the instance’s default gateway is on a different network than the NSIP, then enter a next hop router address on the instance’s NSIP network, so the instance can respond to the SDX SVM Management Service.
  8. In the XVA File field, you can Browse > Local to select an XVA file on your local machine that hasn’t been uploaded to SDX yet. Or you can Browse > Appliance, and select an XVA file that has already been uploaded to SDX.

  9. Select an Admin Profile created earlier. Or you can click the Add button or plus icon to create a new Admin Profile.
  10. Enter a Description. Scroll down.
  11. In the License Allocation section, change the Feature License to Platinum.
  12. For Throughput, partition your licensed bandwidth. If you are licensed for 40 Gbps, make sure the total of all VPX instances does not exceed that number.
  13. For Allocation Mode, Burstable is also an option. Fixed bandwidth can’t be shared with other instances. Burstable can be shared. See Bandwidth Metering in SDX at Citrix Docs.
  14. In the Crypto Allocation field, allocate at least one multiple of Asymmetric Crypto Units. Clicking the up arrow should increment in the correct multiple. See Managing Crypto Capacity at Citrix Docs. You can find the minimum by dividing the total Asymmetric Crypto Units by the Crypto Virtual Interfaces. Enter in a multiple of this result.
  15. In the Resource Allocation section, consider changing the Total Memory to 4096.
  16. For CPU, for production instances, select one of the Dedicated options. Dev/Test instances can use Shared CPU. Then scroll down.
  17. In the Instance Administration section, you can optionally add an instance administrator has has fewer permissions than the nsroot account.. Enter a new local account that will be created on the VPX. Scroll down.
  18. In the Network Settings section, if the VPX NSIP is on the same network as the SDX SVM, then leave 0/1 selected.
  19. Click Add to connect the VPX to more interfaces.
  20. If you have Port Channels, select one of the LA interfaces.
  21. If you configure any VLAN settings here, then XenServer filters the VLANs available to the VPX instance. Changing the VLAN filtering settings later probably requires a reboot. Note: VLAN tagging is configured inside the instance, and not here.
  22. Click Add. Continue adding the interfaces needed by your new VPX.
  23. In the Management VLAN Settings section, do not configure anything in this section unless you need to tag the NSIP VLAN.
  24. Click Done.
  25. After a couple minutes the instance will be created. Look in the bottom right of Chrome to see the status.
  26. Click Close when it’s done booting.
  27. If you go to the Dashboard page…
  28. If you click an instance name, you can see how the instance is connected to the physical NICs.
  29. Back in Configuration > Citrix ADC > Instances, in your Instances list, click the blue IP address link to launch the VPX management console. Or, simply point your browser to the NSIP and login.
  30. Do the following at a minimum (instructions are in the Citrix ADC System Configuration article):
    1. Create Policy Based Route for the NSIP – System > Settings > Network > PBRs
    2. Add SNIPs for each VLAN – System > Network > IPs
    3. Add VLANs and bind to SNIPs – System > Network > VLANs
    4. Create Static Routes for internal networks – System > Network > Routes
    5. Change default gateway – System > Network > Routes > 0.0.0.0
    6. Create another instance on a different SDX, and High Availability pair them together – System > High Availability

VPX Instances – Manage

You may login to the VPX instance and configure everything normally. SDX also offers the ability to manage IP addresses and SSL certificates, from SDX, rather than from inside the VPX instance. The SDX SVM Management Service does not have the ability to create certificates, so it’s probably best to do that from within the VPX instance.

View the console of a Citrix ADC instance

  1. Connect to the SDX SVM Management Service using https.
    • Viewing the virtual machine console might not work unless you install a valid certificate for the SDX Management Service.
  2. In the SVM Management Service, go to Configuration > Citrix ADC > Instances.
  3. On the right, right-click an instance, and click Console.
  4. The instance console then appears.
  5. Another option is to use the Lights Out Module, and the xl console command, as detailed at Citrix Blog Post SDX Remote Console Access of VIs.

Start, stop, delete, or restart a Citrix ADC instance

  1. On the Configuration tab, in the navigation pane, expand Citrix ADC, and click Instances.
  2. On the right, in the Instances pane, right-click the Citrix ADC instance on which you want to perform the operation, and then click Start or Shut Down or Delete or Reboot.
  3. In the Confirm message box, click Yes.

Create a Subnet IP Address on a Citrix ADC Instance

  1. On the Configuration tab, in the navigation pane, click Citrix ADC.
  2. On the right, in the Citrix ADC Configuration pane, click Create IP.
  3. In the Create Citrix ADC IP dialog box, specify values for the following parameters.
    • IP Address* – Specify the IP address assigned as the SNIP address.
    • Netmask* – Specify the subnet mask associated with the SNIP address.
    • Type* – Specify the type of IP address. Possible values: SNIP.
    • Save Configuration* – Specify whether the configuration should be saved on the Citrix ADC . Default value is false.
    • Instance IP Address* – Specify the IP address of the Citrix ADC instance on which this SNIP will be created.
  4. Click Create.

Create a VLAN on a Citrix ADC instance

  1. Go to Citrix ADC > Instances.
  2. On the right, right-click an instance, and click VLAN Bindings.
  3. In the VLAN Bindings page, click Add.
  4. Enter a VLAN ID, and select an interface.
  5. Check the box for Tagged if needed.
  6. Notice there’s no way to bind a SNIP to the VLAN. You do that inside the instance. Click Create.

Save the configuration of a Citrix ADC instance

  1. On the Configuration tab, in the navigation pane, click Citrix ADC.
  2. On the right, in the Citrix ADC pane, click Save Configuration.
  3. In the Save Configuration dialog box, in Instance IP Address, select the IP addresses of the Citrix ADC instances whose configuration you want to save.
  4. Click OK.

Change NSIP of VPX Instance

The best way to change the NSIP is to edit the instance. Go to Configuration > Citrix ADC > Instances, right-click an instance, and click Edit.

Then change the IPv4 Address at the top of the page. Click Done. SVM will push the configuration change to the instance.

If you change NSIP inside of VPX instead of Editing the Instance in the SVM Management Service, see article CTX139206 How to Change NSIP of VPX Instance in SDX to adjust the XenServer settings.

Enable Call Home

  1. On the Configuration tab, in the navigation pane, click the Citrix ADC node.
  2. On the right, click Call Home.
  3. Enter an email address to receive communications regarding Citrix ADC Call Home.
  4. Check the box next to Enable Call Home.
  5. Click Add to select instances.
  6. Select the instances to enable Call Home by moving them to the right, and then click OK.
  7. You can view the status of Call Home by expanding Citrix ADC, and clicking Call Home.
  8. The right pane indicates if it’s enabled or not. You can also configure Call Home from here.

VPX Instance – Firmware Upgrade

Upload Citrix ADC Firmware Build Files

To upgrade a VPX instance from the SVM Management Service, first upload the firmware build file.

  1. Download the Citrix ADC firmware using the normal method. It’s in the Build section.
  2. On the SDX, in the Configuration tab, on the left, expand Citrix ADC, and click Software Images.
  3. On the right, in the Software Images tab, click Upload.
  4. Browse to the build-##.#…tgz file, and click Open.
  5. The uploaded image is shown in the list.

Upgrade Multiple Citrix ADC VPX Instances

You can upgrade multiple instances at the same time:

  1. To prevent any loss of the configuration running on the instance that you want to upgrade, save the configuration on the instance before you upgrade the instance.
  2. On the Configuration tab, in the navigation pane, click Citrix ADC.
  3. On the right, click Upgrade.
  4. Select a Software Image that you already uploaded.
  5. Select the instances that you want to upgrade.
  6. Click OK.

  7. Click Close when done.
  8. You can view the Task Log at Diagnostics > Task Log.

SVM Management Service Monitoring

  1. To view syslog, in the navigation pane, expand System, click Auditing, and then in the right pane click Syslog Message.
  2. To view the task log, in the navigation pane, expand Diagnostics, and then click Task Log.
  3. To view SVM Management Service events, on the Configuration tab, expand System and click Events.
  4. Citrix ADC > Entities lets you see the various Load Balancing entities configured on the instances. You might have to click Poll Now to get them to show up.
  5. To view instance alerts, go to Citrix ADC > Events > All Events.
  6. There is also event reporting.

SVM Management Service Backups

The SDX appliance automatically keeps three backups of the SVM Management Service configuration that are taken daily at 12:30 am.

Backups in Citrix ADC SDX contain the following:

  • Single bundle image
  • Citrix ADC XVA image
  • Citrix ADC upgrade image
  • SVM Management Service image
  • SVM Management Service configuration
  • Citrix ADC SDX configuration
  • Citrix ADC configuration

You can go to Management Service > Backup Files to backup or restore the SDX appliance’s configuration. And you can download the backup files.

You can configure the number of retained backups by clicking System on the left, and then clicking Backup Policy in the right pane.

You can even transfer the backup files to an external system.

Citrix Application Delivery Management (ADM) 13

Last Modified: Mar 13, 2023 @ 10:31 am

Navigation

The older 12.1 version of Citrix ADM is detailed in a different post.

💡 = Recently Updated

Change Log

Planning

Why ADM?

Citrix Application Delivery Management (ADM), formerly known as NetScaler Management and Analytics System (MAS), enables every Citrix ADC (formerly known as NetScaler) administrator to achieve the following:

  • Alert notifications – Receive email alerts whenever something goes down. For example, if a Load Balancing service goes down, you can receive an email alert.
    • ADM can email you for any SNMP trap produced by any ADC appliance.
  • Automatically backup all Citrix ADC instances.
    • ADM can even transfer the backups to an external system, which is then backed up by a normal backup tool.
  • SSL Certificate Expiration – Alert you when SSL certificates are about to expire.
    • Show you all SSL certificates across all ADC appliances.
  • Configuration Record and Play – Use the Configuration Recorder to configure one ADC appliance, and then push out the same configuration changes to additional appliances. This is the easiest method of managing ADC appliances in multiple datacenters.
  • AppFlow Reporting – Receive ICA AppFlow traffic from ADC and show it in graphs.
    • Integrate ADM with Citrix Director so Help Desk can see the AppFlow data.

Everything listed above is completely free, so there’s no reason not to deploy ADM.

ADM Overview

For an overview of ADM, see Citrix’s YouTube video Citrix NetScaler MAS: Application visibility and control in the cloud.

Citrix Tech Zone Citrix Application Delivery Management (ADM) Overview Cheat Sheet

Cloud vs on-prem

ADM is available both on-premises, and as a Cloud Service. For the Cloud Service, you import a ADM Agent appliance to an on-prem hypervisor, or deploy a ADM Agent to AWS or Azure. The ADM Agent is the broker between the Cloud Service and the on-prem (or cloud hosted) Citrix ADC appliances. For more info on the ADM Cloud Service, see the following:

The rest of this article focuses on the on-premises version, but much of it also applies to the Cloud Service.

On-premises ADM Licensing:

  • Instance management is free (unlimited). This includes Configuration Jobs, Instance Backups, Network Functions/Reporting. Basically everything in the Networks node is free.
  • Analytics and Application monitoring are free for up to 30 Virtual Servers (Load Balancing, Citrix Gateway, Content Switching, etc.).

ADM version – The version/build of Citrix ADM must be the same or newer than the version/build of the Citrix ADC appliances being monitored. ADM 13 can monitor many ADC appliance versions including version 11.1, version 12.1, and version 13.0.

HDX Insight

See CTX239748 for a list of HDX Insight Quality Improvements in Citrix Gateway 12.1 and newer. These include:

  • NSAP protocol for reduced performance impact on ADC
  • EDT support

HDX Insight Requirements (aka AppFlow Analytics for Citrix ICA traffic):

  • Your ADC appliance must be running Enterprise Edition or Platinum Edition.
  • ADC must be 10.1 or newer.
  • HDX Insight works with the following Receivers:
    • Receiver for Windows must be 3.4 or newer. Or upgrade to Workspace app.
    • Receiver for Mac must be 11.8 or newer. Or upgrade to Workspace app.
    • Receiver for Linux must be 13 or newer. Or upgrade to Workspace app.
    • No mobile Receivers. See the Citrix Receiver Feature Matrix for the latest details.
  • For ICA Session Reliability with AppFlow: NetScaler 10.5 build 54 and newer.
    • For ICA Session Reliability, AppFlow, and ADC High Availability: NetScaler 11.1 build 49 and newer.
  • For EDT (UDP-based ICA), Citrix ADC must be 12.1 build 49 or newer.
  • AppFlow statistics are only generated when ICA traffic flows through a Citrix Gateway. Internally, when a user clicks an icon from StoreFront, an ICA connection is established directly from Receiver to the VDA, thus bypassing the internal ADC. Here are some methods of getting ICA traffic to flow through an internal ADC:
  • A new Receiver / Workspace app Virtual Channel named NetScaler App Experience or NSAP can dramatically reduce the CPU needed on the ADC to process AppFlow. Details at Citrix Blog Post HDX Insight 2.0. NSAP requires the following:
    • VDA 7.17 or newer, including VDA 1903. VDA 7.15 (LTSR) does not include the NSAP functionality.
    • Workspace app or Receiver 4.10 and newer.
    • ADC 12.0 build 57.24 or newer, including ADC 12.1 and ADC 13.
  • For ICA round trip time calculations, in a Citrix Policy, enable the following settings:
    • ICA > End User Monitoring > ICA Round Trip Calculation
    • ICA > End User Monitoring > ICA Round Trip Calculation Interval
    • ICA > End User Monitoring > ICA Round Trip Calculation for Idle Connections
  • Citrix CTX215130 HDX Insight Diagnostics and Troubleshooting Guide contains the following contents:
    • Introduction
    • Prerequisites for Configuring HDX Insight
    • Troubleshooting
      • Issues Related to ICA parsing
      • Error Counter details
    • Checklist before Contacting Citrix Technical Support
    • Information to collect before Contacting Citrix Technical support
    • Known Issues

Citrix CTX204274 How ICA RTT is calculated on NetScaler Insight: ICA RTT constitutes the actual application delay. ICA_RTT = 1 + 2 + 3 + 4 +5 +6:

  1. Client OS introduced delay
  2. Client to NS introduced network delay (Wan Latency)
  3. NS introduced delay in processing client to NS traffic (Client Side Device Latency)
  4. NS introduced delay in processing NS to Server (XA/XD) traffic (Server Side Device Latency)
  5. NS to Server network delay (DC Latency)
  6. Server (XA/XD) OS introduced delay (Host Delay)

Multi-Datacenter Deployment Architecture

In a main datacenter, import two Citrix ADM appliances into the same subnet and configure them as an HA pair with a Floating IP address.

In a DR datacenter, import a DR node Citrix ADM appliance, and configure it to replicate with the main datacenter.

For Citrix ADC appliances in additional datacenters, import two ADM Agent appliances into each datacenter. Remote ADC instances are discovered and managed through remote ADM agents.

  • The virtual appliance for ADM Agent is different than the normal ADM appliance.
  • Documentation at Configure multisite deployment at Citrix Docs, and will be detailed later in this post.

Import ADM Appliance

If you are upgrading an existing ADM or MAS, skip to the Upgrade section.

There are two different ADM appliances:

  • ADM appliance for the main datacenter, including High Availability, and for the DR node.
  • ADM Agent appliance for remote datacenters

To import a ADM Appliance into vSphere, do the following:

  1. Download Citrix ADM Image for ESX.

    • The download page for ADM has two different images: one called ADM Image, and one called ADM Agent Image. The first image should be the non-agent image.
  2. Extract the downloaded .zip file for the non-agent image.
  3. In vSphere Web Client, right-click a cluster, and click Deploy OVF Template.
  4. In the Select an OVF Template page, select Local file, and browse to the Citrix ADM .ovf files. If vCenter 6.5 or newer, select all three files. Click Next.

  5. In the Select name and folder page, enter a name for the virtual machine, and select an inventory folder. Then click Next.
  6. In the Select a resource page, select a cluster or resource pool, and click Next.
  7. In the Review details page, click Next.
  8. In the Select storage page, select a datastore. Due to high IOPS requirement, SSD is recommended.
  9. Change the virtual disk format to Thin Provision. Click Next.
  10. In the Select networks page, choose a valid port group, and click Finish.
  11. In the Ready to Complete page, click Finish.
  12. Before powering on the appliance, you can review its specs. Right-click the virtual machine, and click Edit Settings.
  13. Review the specs. Citrix Docs VMware ESXi Hardware Requirements has recommended specs.
  14. The OVF defaults to 8 vCPU and 32 GB of RAM.
  15. You can add a second hard disk at this time.
  16. Citrix Docs Attach an additional disk to Citrix ADM says that an additional disk must be added before initial deployment.
    • Use the ADM storage calculator to determine the recommended size of the disk. Ask your Citrix Partner for the tool.
    • The new disk must be larger than 120 GB.
    • In ADM 13, the new disk can be larger than 2 TB.
    • In ADM 13, the new disk can be grown later, and /mps/DiskPartitionTool.py can resize the partition, but only up to 2 TB. If you need more than 2 TB, the initial disk should be larger than 2 TB.
  17. Power on the Virtual Machine.

Appliance IP Address Configuration

  1. Open the console of the virtual machine.
  2. Configure IP address information.
  3. Enter 7 when done.

Second Disk

  1. SSH to the appliance and login as nsrecover/nsroot.
  2. Enter /mps/DiskPartitionTool.py

  3. Enter info to see that there are no existing partitions on the second disk.
  4. Enter create to create partitions on the second disk. A reboot is required.
  5. During the reboot, the database is moved to the second disk.
  6. After the reboot, the Disk Partition Tool info command shows the partition on the second disk.
  7. If you need to increase the size of the disk, reboot the ADM appliance so it detects the larger size. Then use the Disk Partition Tool resize command.

Deployment Modes

HA Pair in the Main Datacenter

First Node:

  1. SSH to the first node and login as nsrecover/nsroot.
  2. Enter deployment_type.py.
  3. Enter 1 for Citrix ADM Server.
  4. Enter no when prompted for Citrix ADM Standalone deployment.
  5. For the First Server Node prompt, enter yes.
  6. Enter yes to Restart the system.

Second Node:

  1. Import another ADM appliance to the same subnet, and configure an IP address.
    • Latency to the HA node must not exceed 10 ms.
    • The HA nodes must be on the same subnet.
  2. If you added a second disk to the first ADM appliance, then you must add the same size second disk to the second ADM appliance.
  3. Configure the new nodes’ IP address.
  4. SSH to the second appliance, login as nsrecover/nsroot, and run the Disk Partition tool.
  5. SSH to the second appliance, login as nsrecover/nsroot, and run deployment_type.py.
  6. Enter 1 for Citrix ADM Server.
  7. Enter no when prompted for Citrix ADM Standalone deployment.
  8. Enter no when prompted is this is First Server Node.
  9. Enter the IP address of the first ADM node.
  10. Enter the nsroot password of the first node. The default password is nsroot.
  11. Enter a new Floating IP address.
  12. Enter yes to restart the system.

Deploy HA Configuration:

  1. After both appliances are fully booted, point your browser to the first appliance’s IP address, and login as nsroot/nsroot. It will take several minutes after booting before the ADM appliance is ready.
  2. If you see Customer User Experience Improvement Program, click Enable, or click Skip.
  3. In the What is Application Delivery Management page, click the blue Get Started button.
  4. On the Instances page, click Add Instance. This wizard will keep appearing at logon until you add an instance.

    1. Enter IP Address or FQDN of one of your ADC instances.
    2. The Profile Name contains the password for your ADC nsroot account. To specify the password, you can Edit the built-in ns_nsroot_profile, or you can create a new Profile.
    3. Check the box next to Change Password and enter the ADC’s nsroot password.
    4. Scroll down and click OK to close the ADC Profile window.
    5. Click OK to close the Add Instances window and begin the discovery process.
    6. Click Close when Operation completed successfully. If there’s an error, switch to the Error tab to see more details.
    7. You can add more instances, or just click Next.
  5. In the Customer Identity page, make your choices, and then scroll down and click Next.

  6. In the System Notifications page, check the box next to Send Email and then click Add.

    1. In the Create Email Distribution List page, next to Email Servers, click Add.
    2. Enter your email server details and then click Create to close the Create Email Server window.
    3. Back in the Create Email Distribution List window, enter a From address, enter a To address, and then click Create.
    4. Back in the Getting Started wizard, click Next.
  7. On the Done page, click Finish.
  8. If you want to make any network changes (e.g. DNS servers) to either node, then you must make those changes before you deploy the HA pair. Click the menu icon on the top left. Then go to System > Administration > IP Address, Host Name and Proxy Server.

    1. Enter an Alternate DNS and then click Save.
  9. On the top left, click the menu (hamburger) icon, expand System, and then click Deployment.
  10. In the top right, click Deploy.
  11. Click Yes to reboot.
  12. It takes around 10 minutes to restart.
  13. After deployment, you can now use the Floating IP to manage the appliance.
  14. The System > Deployment page should show both nodes as UP.

Afterwards, you can manage High Availability.

  1. System > Deployment lets you see the HA nodes.
  2. You can Force Failover from here. Note: HA failover only occurs after three minutes of no heartbeats.
  3. On the top right is a HA Settings button that lets you change the Floating IP.

DR Node

Requirements for the DR node:

  • The main datacenter must have an HA pair of ADM appliances. Standalone in the main datacenter is not supported.
  • Latency from the main datacenter HA pair to the DR node must not exceed 200 ms.
  • Ports 5454 and 22 open between the ADM nodes.

To configure a DR node:

  1. Import another ADM appliance into a remote datacenter, and configure an IP address.
  2. If you added a second disk to the main datacenter ADM appliances, then you must add the same size second disk to the DR ADM  appliance.
  3. After configuring the new nodes’ IP address, SSH to the DR appliance and login as nsrecover/nsroot.
  4. Enter deployment_type.py.
  5. Enter 2 for Remote Disaster Recovery Node.
  6. Enter the Floating IP address of the HA pair in the main datacenter.
  7. Enter the nsroot password, which is nsroot by default.
  8. The DR node registers with the ADM HA Pair.
  9. You can change the password of the DR node by running the following command:
    ./mps/change_freebsd_password.sh <username> <password>
  10. Point your browser to the Floating IP Address and login.
  11. Go to System > Administration.
  12. On the right, in the right column, click Disaster Recovery Settings.