Migrate Citrix ADC config to new ADC appliances

Last Modified: Nov 25, 2019 @ 11:23 am

Navigation

Overview

This article details one method of migrating a Citrix NetScaler ADC configuration from an old appliance pair to a new appliance pair while preserving the IP addresses. It requires a brief outage to move the IP addresses from the old pair to the new pair. Keeping the existing IP addresses avoids needing to make any changes to DNS, NAT, firewall, and routing.

  • An alternative migration method is to migrate one VIP at a time. However, you won’t be able to move the SNIP so you might need new firewall rules for the new appliance pair.

Notes:

  • The new appliances must be connected to the same networks as the old appliances. If not, then the IP addresses need to change.
  • If you migrate the entire configuration at once, then the SNIPs can also be migrated.
    • The new appliances won’t have a SNIP until cutover time, so DNS will not work on the new appliances until cutover.
  • It’s much easier and quicker if you don’t migrate the NSIPs, so the new appliances will need new NSIPs.
  • You can use temporary SNIPs to validate connectivity on the new appliances.
  • Firmware versions can usually be different on the new appliances as compared to the old appliances.
    • If the old appliances are 10.5 or older, and if you have NetScaler Gateway with a customized logon page, then you’ll have to redo the customization as a Portal Theme on the new appliances.
    • GSLB Sites (MEP) can usually have different firmware versions.

Here’s a summary of the migration process. See the rest of this article for details.

  1. Prepare new appliances by connecting them to the network and pairing them together.
  2. Export files (e.g. SSL certificates) from old appliances and import to new appliances.
  3. Export configuration from old appliance and modify the configuration for later import to the new appliances.
  4. During a brief outage window, disable the old appliances (power off or disconnect), then run the config on the new appliances.
  5. Test the new appliances. If a problem is discovered, then you can roll back by disabling the new appliances and re-enable the old appliances.

Prepare New appliances

If VPX (not on SDX):

  1. Import new VPX appliances into your hypervisor.
    • If vSphere, change NIC to VMXNET3.
  2. Configure new management IP Addresses in the virtual machine console.
    • Default gateway should be on the NSIP subnet. The default gateway will be changed to a data VLAN during the cutover.
  3. In the Welcome Wizard, don’t configure any SNIPs. And DNS won’t work without SNIP.
  4. Install licenses on the new VPX appliances.
    • If pooled licensing, enter ADM IP address instead of DNS name to avoid needing DNS/SNIP configuration.
  5. Optional: configure VPX CPU Yield.

If VPX on SDX:

  1. Configure switch ports for LACP port channel and VLAN trunking.
  2. Connect the physical SDX hardware to port channels or individual interfaces.
  3. In SDX SVMs, create Channels and confirm that the channels are distributing.
  4. In SDX SVMs, upload desired VPX XVA firmware virtual appliance.
  5. In SDX SVMs, create new instance and connect the instance to channels.
    • New management IP address for each instance.
    • Default gateway should be on the NSIP subnet. The default gateway will be changed to a data VLAN during the cutover.
    • Enable NSVLAN if the VLAN trunk doesn’t have a native untagged VLAN.
    • For HA pair, create each instance on separate SDX appliances.

If MPX:

  1. Configure switch ports for LACP port channel and VLAN trunking.
  2. Connect the physical MPX hardware to port channels or individual interfaces.
  3. Use LCD, LOM, serial cable, or 0/1 Ethernet crossover to configure new NSIP for each MPX appliance.
    • Default gateway should be on the NSIP subnet. The default gateway will be changed to a data VLAN during the cutover.
    • Enable NSVLAN if the VLAN trunk doesn’t have a native untagged VLAN.
  4. In the Welcome Wizard, don’t configure any SNIPs. And DNS won’t work without SNIP.
  5. Upgrade firmware to desired version.
  6. At System > Network > Interfaces, configure LACP and confirm that the channels are distributing.
  7. Install licenses on the new MPX appliances.
    • If pooled licensing, enter ADM IP address instead of DNS name to avoid needing DNS/SNIP configuration.

For all ADC models:

  1. Create new DNS names for the new NSIP IP Addresses.
  2. Ensure you can SSH (e.g. Putty) and SCP (e.g. WinSCP) to the new NSIPs and old NSIPs.
  3. In the Welcome Wizard, don’t configure any SNIPs. And DNS won’t work without SNIP.
  4. Configure NTP and Time Zone on each node before you pair them together.
    • Enter NTP IP address instead of DNS name to avoid needing DNS/SNIP configuration.
    • From CLI, run shell, then run ntpdate command with NTP Server IP address to sync the local time with NTP server. NTP Sync needs to be disabled before you can run this command.
  5. Pair the two new appliances together.
  6. Change the password on the RPC Nodes at System > Network > RPC.
  7. Configure VLANs without SNIPs.
  8. For dedicated management subnet, configure PBRs.
  9. If you have channels that are not configured with a native, untagged VLAN, then configure tagged High Availability Heartbeats for non-NSIP interfaces by marking one VLAN as untagged and enabling the tagall option on the channel.
    • From CLI, run show ha node to confirm that there are no interfaces on which heartbeats are not seen.
  10. Use new temporary SNIP addresses to test VLAN connectivity
    • Bind the temporary SNIP to VLAN and channel and try to ping it.
    • Remove temporary SNIP when done testing.
  11. Install custom Management certificates that match new NSIP DNS names.
  12. If Default SSL Profile is enabled on old appliance, then enable it on new appliance pair.
  13. Add new pair to Citrix ADM.
  14. Add new pair to SNMP Manager.
  15. Add new pair to ControlUp Monitoring or any other Citrix NetScaler ADC monitoring tool.
  16. If any RADIUS authentication, and if RADIUS is not load balanced, then add the new NSIPs as RADIUS Client IP Addresses on the RADIUS servers.
    • If RADIUS is load balanced, then SNIP is the RADIUS Client IP, and the SNIP will be moved over during cutover.

Export/Import Files

Enact a change freeze to prevent any changes to the old appliances so that the files and Running Configuration don’t have to be downloaded again.

Use WinSCP to download the following files from the old appliances and upload to the same place on the new appliances.

  • Certificate files from /nsconfig/ssl
    • UNIX is case sensitive but Windows is not. While downloading, if you see a message about overwriting files, that’s probably because you have two files with the same name but different cases.
    • Don’t upload the files that start with ns- (e.g. ns-server.cert) since those are management certificate files.
  • LDAP certificate verification files from /nsconfig/truststore, if exists.
  • Custom monitors (e.g. nsldaps.pl) from /nsconfig/monitors, if exists.
  • Custom Portal Themes from /var/netscaler/logon/themes, if exists.
    • There’s no need to download the built-in themes. Only download the non-built-in themes.
    • On the new appliances, create Portal Themes with the same names and same template as the old appliances.
    • Then upload the Portal Theme files to the new appliances and overwrite the default files.
  • VPN bookmarks from /var/vpn/bookmark, if exists.
  • Gateway logon page customizations from /var/vpn/vpn, if any.
    • Try to redo these customizations as Portal Themes instead of uploading to the new appliances. The .js files in NetScaler 11.0 and newer are quite different from NetScaler 10.5 and older.
  • /nsconfig/rc.netscaler if it contains anything other than ntpd.
    • If this file contains customizations for the Gateway logon page, then try to redo them as a Portal Theme.
    • After uploading this file to the new appliances, reboot the new appliances so the commands are executed. Or you can manually run rc.netscaler after you chmod +x to the file.
  • /nsconfig/nsafter.sh if it exists.
    • If this file contains customizations for the Gateway logon page, then try to redo them as a Portal Theme.
    • After uploading this file to the new appliances, reboot the new appliances so the commands are executed. Or you can manually run nsafter.sh after you chmod +x to the file.

To test the Portal Themes, you can create a temporary Gateway Virtual Server with a temporary VIP. Remove the temporary configuration when done testing.

Export Configuration

Export running configuration from the old appliances by going to System > Diagnostics > Running Configuration and then click the link to Save text to a File.

Edit the downloaded nsrunning.conf file to prepare it to be executed on the new appliances:

  • Remove all NSIP and hostname lines, including IPv6 and MAC addresses. These lines start with the following:
    • set ns config -IPAddress
    • set lacp -sysPriority 32768 -mac
    • set ns hostName
    • add ns ip -type NSIP – leave the other “add ns ip” commands, especially the SNIP (-novserver) commands
    • add ns ip6 -type NSIP
  • Remove the nsroot line that starts with:
    • set system user nsroot
  • Remove all interface commands that start with:
    • set interface
  • Remove all channel commands that start with:
    • add channel
    • set channel
  • Fix VLAN binding commands for new channels/interfaces.
  • Remove PBR commands for NSIP that start with. Or you can adjust the commands for the new NSIPs.
    • add ns pbr NSIP
  • Remove High Availability / Cluster commands that start with:
    • add HA node 1
  • Remove RPC Node commands that start with:
    • set ns rpcNode
  • Review the add route commands to make sure they are correct for the new appliances.

Make sure the name of the modified configuration file is all lower case and doesn’t have any spaces. The batch command doesn’t work if there are upper case letters or spaces.

Upload the modified config file to /tmp on the new primary appliance.

Optional: review the SSL commands and look for any bindings of custom ciphers to SSL vServers. After the cutover, the new appliances will have your custom ciphers but unfortunately will also have the DEFAULT ciphers. To speed up the post-cutover process, you can pre-create the CLI commands that unbind the DEFAULT cipher groups. Here’s an example command:

unbind ssl vserver MySSLvServer -cipherName DEFAULT

Cutover

Decide how you will disable the old appliances. Here are some options:

  • Shutdown MPX Ethernet interfaces on network switch – if rollback, it’s easy to unshut the interfaces
  • Power off old VPX appliances, including on SDX – if rollback, it’s easy to power them on again
  • If old appliances have dedicated management network (no SNIP on management network), then you can to go to System > Network > Interfaces and disable them – if rollback, easy to re-enable
  • Power off old MPX appliances – if rollback, is somebody available in the data center to power them on again?

Before outage – Before the outage window, do the following on the old appliances:

  • Backup the old appliances and download the backup files.
  • Review the old appliances and record what’s up or down – After cutover, you’ll confirm up/down is the same on the new appliances.
    • Look in Traffic Management > Load Balancing > Virtual Servers for up/down status, including the number of UP services.
    • Go to Authentication > Dashboard to see which authentication servers are reachable

Outage – During outage window, do the following:

  • Disable the old appliances.
  • SSH to the new primary appliance and run the batch command to run the commands in your modified configuration file. For example:
    batch -fileName /tmp/nsrunningmodified.conf -outfile /tmp/outfile.txt
  • Review the outfile for errors. Some warnings can be ignored. You can use WinSCP to access the out file. Or cat it.

After running the batch command, do the following to fix the imported configuration.

  • Default Route – Remove extra Default Route, if any.
    • System > Network > Routes. If there are two 0.0.0.0 routes, then remove the one for the NSIP subnet.
  • VIPs are UP? – Ensure VIPs (Load Balancing, Content Switching, Gateway) are up or down as noted on the old appliance.
    • Traffic Management > Load Balancing > Virtual Servers.
  • Go to Authentication > Dashboard and ensure the status matches the old appliance.
    • If needed, fix LDAP Password in load balancing monitor and in LDAP Policies/Servers.
    • If needed, fix RADIUS Secret in load balancing monitor and in RADIUS Policies/Servers.
  • Portal Themes – Point your browser to the Citrix Gateway logon pages and verify Portal Themes customizations are correct.
  • GSLB MEP – Set GSLB RPC nodes with Secure checked.
    • System > Network > RPCs. Check the Secure box. Change RPC password.
  • DEFAULT ciphers – If Default SSL Profile is enabled, then remove DEFAULT cipher group from SSL Profiles that have custom Ciphers.
    • System > Profiles > SSL tab. Edit each profile. In SSL Ciphers section, remove DEFAULT if it’s not the only cipher.
  • If Default SSL Profile is not enabled, then remove DEFAULT cipher group from all SSL Virtual Servers that have custom ciphers.
    • E.g. Traffic Management > Load Balancing > Virtual Servers. Edit an SSL Virtual Server. Scroll down to the SSL Ciphers section. If you see both DEFAULT and something else, then remove DEFAULT.
  • SSL Labs – Use SSL Labs to test externally reachable VIPs.
  • HA failover – Do a High Availability failover and verify that the same VIPs and authentication servers are UP on new primary appliance.

26 thoughts on “Migrate Citrix ADC config to new ADC appliances”

  1. Hi Carl,

    Great stuff. One question, We have 11500 SDX’s EOL and would want to migrate the VPX instances to existing 1400 series SDX. Since the EOL VPX’s are in HA, can I build a new VPX instance on 1400 SDX then introduce this 3rd instance in the cluster. Sync the configuration and remove the existing EOL VPX.

    Is clustering possible with different SDX Hardware models ?

    Thanks
    Abhijith.

    1. HA is two nodes max. It’s not supported to mix different hardware models in the same HA pair. Your 14000 might have different interfaces than your 11500, which causes issues when syncing the config. However, some customers have had success doing this.

      1. Cluster and HA are vastly different (apples to tractors type different), original comments indicate HA not cluster. as Carl has stated be careful that any new instance on the 1400 has the same kind of interfaces presented, although there are ways around this, a HA pair on the same site typically wont be configured in a way that can support this without downtime to reconfigure it from a more traditional build to one that can. This can also be a big issue migrating hypervisors. For a migration breaking replication, updating the passive and doing a forced fail over works, but you don’t want to miss anything when doing it, and it should not be a long term strategy for managing fail overs between dissimilar nodes.

        There is hardware limitation on clustering on MPX’s but VPX wise they must be the same type of virtual (ie only vmware, only SDX,etc). All nodes must have same licences, which you’d need to check the SDX’s to confirm, unless you are not using the SDX licences, and a cluster licence.This also means if one day you decide to leave the 1400’s and abandon SDX’s all together and go to another hypervisor (vmware/KVM/etc) you can’t just migrate a node at a time, like you can with HA.

        Personally I would not bother with clustering except maybe under certain situations ie a single netscaler can’t handle the load, clustering has its own limitations and caveats. Outside of doing any qualification exam that needs it most people will likely never use it, for good reasons. I know one of our clients didn’t like the idea of active-passive and thought clustering would address that, but after some detailed discussions on how clustering works, and work required to migrate, they decided to stay HA instead.

        You could remove one device from HA and simply replicate to the new one, if that’s what you mean instead, but that’s what these steps are all about anyway.

  2. Good Afternoon Sir, I am trying to migrate from an ADC to a VPX. What would be the best way to get this done. I have setup the VPX with a NISP, Management IP, Hostname and License. I have also setup the Network Adapters for all of my interfaces that are setup on the ADC. I have made a copy of the ns.conf and have it pulled up in in notepad, just not sure what to change. Once I get this one done, which is my Secondary I will setup the Primary. Any info you can give would be greatly appreciated. Thanks,

  3. hi, i have a problems with the XML files in “var/vpn/bookmark/”, this are not associate with the LDAP user id.

    I can see all files in this format, “bm_prefix_a2d1YWphcjE=.xml”, and cannot be copy and backup.

    can you help me please? My ADC is NS12.1 55.18.nc and my others vm_ADC are not have the same problems. this other machine did have
    NS12.1 54.13.nc

    regards, from Chile
    Guilliano Palavecino Rojas
    technical support

  4. before this article was posted we did something similar in a POC on a HA pair of vpx’s (on SDX) to vpx’s on vmware migration, the only thing to remember there is the management vlan might not be on the vpx on the sdx and will need to be added when migrated to vmware, depending on how the sdx/vmware networking is configured. (ie what is and isn’t trunked)

    NOTE: you might want to check if anyone did something ‘funky’ at the shell level that isn’t matched by a nsconfig item (or in the exported nsconfig folder), as these can be ‘missed’ in the migration process, depending on where the change is made.

    @mark hodges

    as per Carl’s comments, and automating this isn’t too hard depending on how far you want to take it.

    script config extraction (ssh/scp/nitro), add/replace items based on some lookup.
    nitro will let you focus on only the items you want to extract, so once you know those it can be a doddle to extract and rewrite a config, same in powershell (or favorite scripting language) via ssh (and probably easier for most)
    OR via ADM, backup nightly, extract from backup – replace/remove key lines, and you can have a DR config ready for applying.

    if you look a the whole config, rather than the load balancing you can replace the known items for your current DR site, and with a few variable changes switch the DR config to a new site as needed. (interface,snip,vlan etc changes). it really is worth it if there is a good chance your DR site will change over time, script it well and you can do a dozen or more pairs with the one script if you’re supporting a decent number of these and have to do so.

  5. Hi, Carl
    do you know if this process works with an azure ADC ?
    I have to migrate from an on-premise ADC to an azure ADC

  6. Great stuff Carl,

    Quick question..Want to replace exising 11500 to 15000z.. want to do it in one shot.. would it make sense to take the standby adc out.. add the new 15000z sync the configs than fail it over?

    1. Citrix doesn’t support different hardware types in an HA pair. But I’ve seen reports of successfully doing this. If the interfaces are different, then this won’t work without downtime after failover.

      1. What if its 11500 SDX… to a new 15100 SDX..

        Moving VPX from 11500 to 15100 VPX… can I sync .. VPX in 11500 to VPX in 15100

  7. Hi Carl, I sincerely appreciate your response. I have checked via CLI and its showing down, I have logged a case with Citrix however for security reason I can neither shadow the sessions nor share the log so I am not hoping to get much from them. I have ruled out its STA server issue as if I use earlier Citrix ADC with same IP address as the problematic one STA is working fine on earlier ADC. So it look like there is a issue with ADC which is not able to communicate to STA server. I have tried telnet to STA on port 80 and it was successful however I assume it must be using NSIP as source rather than using SNIP.
    As I could not figure out why STA is down I am setting up two VPC ADC in HA and restoring the config and I am hoping it will work. However I am bit concerned if on two ADC STA is down then how do I troubleshoot further.
    I have tried re configuring the STA on virtual server using below article and its still showing down
    https://docs.citrix.com/en-us/citrix-gateway/12-1/integrate-web-interface-apps/ng-wi-comm-wrapper-con/ng-wi-comm-sta-config-tsk.html

    Any help or suggestion will be highly appreciated

  8. Hi Carl,
    First of all thanks a lot for sharing this article, using this I have restored the setting from 11.1 to 13.0 ADC and its restored all the settings and its worked fine.Yesterday there was issue reported and both (primary and secondary) appliance is factory restored.
    I have deleted all the configs and paired them in HA and tried restoring the earlier config. Everything is working fine except the STA on virtual Gateway is down. I can ping both sta ip address from ADC CLI and telent to port 80 however its still showing down. There are many articles talks about adding STA server in virtual gateway which seems to be easy one. Because its down for me I have unbinded and bind again however still its showing down.
    Can you please recommend some solution.
    ADC version is 13.0, STA in virtual gateway I am using IP address which is same as storefront. Storefront and Delivery controller on same server. XML default port is 80

    1. There might be a GUI issue in ADC 13. In CLI, if you run “show vpn vserver myGateway”, does it show the STAs as up?

  9. Hi Carl

    Am planning to migrate our VPX version 12 from VMware to Hyper-V in HA and am not sure if the steps mentioned above will work. I am thinking of migrating the secondary node first from VMware to Hyper-V and reusing the existing license, and copy the config including SSL. Am trying to avoid downtime and am not sure if this approach will work .

    Cheers

    Jesse

    1. Some people have tried that approach even though it’s probably not supported since the two nodes could be considered different hardware.

    2. Thanks Carl. Since it’s a virtual appliance and the licenses are mapped to Mac addresses, am thinking of changing the MAC address of the new VMs in Hyper-V to be same as current appliances on VMware. Make secondary appliance STAY SECONDARY. Shut it down and apply the license to the new appliance in Hyper-V. Keeping my fingers crossed. Any suggestions on this?

      1. I finally migrated Citrix ADC 12 from VMware to Hyper-V without downtime. Most important thing was to reuse the static MAC addresses on VMware appliance in Hyper-V, reason being the license is associated to the MAC
        -i copied the configuration, certificates and license files to appliance in Hyper V
        -shutdown secondary appliance in VMware
        -Changed the NSIP and Mac address on the secondary appliance in Hyper V to match secondary appliance in VMware.
        -Failed over traffic to new appliance in Hyper V.
        – Repeated same steps on Primary appliance

        Was very smooth without any downtime

  10. is there a way to export/import the Dashboard that are created when going through the wizard for “XenApp and XenDesktop” under “Integrate with CitrixProducts” in Netscaler?

    i have tried to do export from such setup and imported to new netscaler and it is working as the old one but the Dashboard is missing for the Gateway.

    BR

    Ingvar

  11. Hi Carl, do you think this method will works also on AWS instances?
    I have to migrate a single instance to an HA one (both on AWS), and it looks not so easy to do as it would be on an on prem ones.
    The geneal idea is to create a new cluster by the AWS store and then migrate the single instance configuration.

      1. I will give a look.
        The real challeng is related to the number of vips and policies configured on that particolar instance: we are talking about 50 vservers and with a very complex AAA configurator.
        My original idea was to add a second VPX instance and configure HA with the old one, but the existing VPX is a flat one (2 nic, but on the same network) and all the documentation related to AWS HA talks about 3 nic with logic separation of FE/BE/MGMT: I strongly suspect that networking does not work as flat as on prem installations.

  12. This is handy, but what would be really nice is a way to easily create a DR configuration from an existing environment, especially when it comes to the VIP’s..being able to quickly take an existing Config, ignore all the SNIP settings (since those would be pre-set) and just find and replace the VIP ip’s with new ones and import would be nice…

    1. You can search for “add lb vserver”, “add cs vserver”, “add vpn vserver” and change their IP addresses.

      Then remove any lines that reference the SNIP.

Leave a Reply