- Part 1 – Request-Response, HTTP Basics, and Networking (separate page)
- HTTP Encryption
- NetScaler VPN Networking
- NetScaler GSLB
SSL Protocol, and Keys
SSL/TLS Protocol – SSL (Secure Sockets Layer) and TLS (Transport Layer Security) are two names for encrypted HTTP. SSL is the older, more well-known name, and TLS is the newer, less well-known name. The names can usually be used interchangeably, although pedantic people will insist on TLS instead of SSL. In NetScaler, you’ll mostly see the term SSL instead of TLS.
- Port 443 – Web Servers that support SSL/TLS protocol listen for encrypted HTTP on TCP 443. Enabling SSL/TLS on a web server requires creation of keys/certificate, and binding them to a TCP 443 listener.
- https protocol – Users enter https://FQDN into a web browser to connect to a web server using SSL/TLS protocol on TCP 443. Technically, https = HTTP on top of SSL/TLS.
- Disable http? – You can optionally disable clear-text HTTP over TCP 80. Or you can leave the TCP 80 listener and configure it to redirect TCP 80 connections to TCP (https) 443.
SSL/TLS version – There are several versions of the SSL/TLS protocol. TLS v1.2 is the newest, but TLS v1.3 is coming soon. PCI compliance dictates that TLS v1.0 and v1.1 should be disabled, but not all software is compatible with TLS v1.2 yet. SSL v3 is the older protocol, and must be disabled on all SSL vServers. All versions of TLS are newer than SSL v3.
- SSLLabs.com can check your SSL Listener to make sure it adheres to the latest SSL security standards.
SSL Session – HTTP is transmitted on top of SSL/TLS. First, the SSL Client and SSL Server create an encrypted SSL Session. Then HTTP is transmitted across this SSL Session.
- SSL Handshake is expensive – Establishing the SSL session (SSL Handshake) is an expensive (CPU) operation, and modern web servers and web browsers try to minimize how often it occurs, preferably without compromising security.
- Bulk encryption – The traffic on top of the established SSL Session is bulk encrypted, which is far less impactful than initial SSL Session establishment.
- SSL Specs – The NetScaler data sheets provide different numbers for SSL Transactions/sec (initial session establishment) and SSL Throughput (bulk encryption).
Public/Private key pair – The SSL vServer needs a public/privatekey pair. The Public Key and Private Key are linked together. Data encrypted by Public Key can only be decrypted by one Private Key. Data encrypted by the Private Key can only be decrypted by one Public Key.
- Private key – The Private Key is called private because it needs to remain private. You must make sure that the Private Key is never revealed to any unauthorized individual. If that were to occur, the unauthorized person could use the Private Key to emulate the web server, and unsuspecting users might submit private data, including passwords, to the hacker.
- Public key – The Public Key is called public because it doesn’t matter who has it. The public key is worthless without also having access to the private key.
Key size – When you create a public/private key pair, you specify a key size in bits (e.g. 2048 bits). The higher the bit size, the harder it is to crack. However, larger key sizes mean exponentially more processing power required. 2048 is the current recommended key size, even for Certificate Authorities. 2048 balances security with performance.
Private/Public Key vs Session Key – The Public/Private key pair is only used during initial SSL session establishment. The bulk of the encryption is performed using a new Session Key instead of the Public/Private key pair. The Session Key is generated by the SSL Client and sent to the SSL Server so both parties know the Session Key. Traffic encrypted by the Session Key can only be decrypted by the Session Key.
- Session Keys are Symmetric Encryption – Since only one Session Key is used to both encrypt and decrypt, this is called Symmetric Encryption.
- Public/Private Keys are Asymmetric Encryption – In a public/private key pair, different keys are used for encryption and decryption. This is called Asymmetric Encryption.
How the SSL Session Key is generated
- The SSL Client initiates the SSL Connection with the SSL Server.
- The SSL Server sends its Public Key to the SSL Client, usually in a certificate.
- The SSL Client generates a random SSL Session Key.
- The SSL Client, encrypts the Session Key using the SSL Server’s Public Key, and sends the encrypted Session Key to the SSL Server.
- The SSL Sever uses its Private Key to decrypt the SSL Session key.
- Now both sides know the Session Key and can use that Session Key to perform bulk encryption.
Session Key size – the Session Key is much shorter than the private/public keys. For example, the Session Key can be 256 or 384 bits, while the public/private keys are 2048 bits. Because of their shorter size, Session Keys are thus much faster than public/private keys. The length of the Session Key depends on the negotiated cipher, as detailed later.
Renegotiation – SSL Clients and SSL Servers will sometimes want to redo the SSL Handshake while in the middle of an SSL Session. This is called Renegotation.
- Without Forward Secrecy, if you take a packet trace, you can use the SSL Server’s Private Key to decrypt Session Keys in the packet trace, and use that decrypted Session Keys to decrypt the rest of the packet trace.
- With Forward Secrecy, the public/private key pair are generated every time a new SSL Session is established. Thus every session has a different private key. It’s not possible to find the per-session private key, and use it to decrypt the Session Key.
- DH vs RSA – Diffie-Hellman (DH) Key Exchange algorithm enables Forward Secrecy. RSA Key Exchange does not. ECDHE is the modern version of DH Key Exchange that is preferred by security professionals.
Public Key in Server Certificate – The SSL Server’s Public Key is included in the SSL Sever’s Certificate, which is transmitted to the SSL Client when the SSL Client initially connects. The certificate contains the public key, plus additional fields like: Subject, Issuer, Validity Dates, CA Signature, Certificate Revocation Location, etc.
- Keys and certificates are different things – The public/private key pair, and the certificate, are two different things. First, you create a public/private key pair. Then you create a certificate that contains the public key. Windows hides this operation from you so many Windows admins are not aware that keys are separate from the certificate. However, NetScaler is UNIX-based, which stores the keys and certificate in separate files.
Certificates provide a form of authentication – The SSL Client needs to authenticate the SSL-enabled web server so that the SSL Client only sends confidential data to trusted web servers. There are several fields in the SSL Server’s certificate that clients use to verify web server authenticity. Each of these is detailed later in this section.
- FQDN – the hostname entered in the browser’s address bar must match the FQDN in the server’s certificate.
- CA Signature – the server’s certificate must be signed a trusted Certificate Authority that has verified the owner of the website.
- Validity Dates – the certificate must not be expired.
- Revocation – the certificate must not be revoked.
Types of certificates – there are different types of certificates for different use cases. All certificates are essentially the same, but some of the certificate fields control how a certificate can be used:
- Server Certificates – when linked to a private key, these certificates enable encrypted HTTP.
- CA Certificates – used by a SSL client to verify the CA signatures contained within a Server Certificate.
- Client Certificates – used by clients to authenticate the client machine or user to the web server. Requires a private key on the client side.
- SAML Certificate – self-signed certificate exchanged to a different organization to authenticate SAML Messages (federation).
- Code Signing Certificate – developers use this certificate to sign the applications they developed.
Digital Signatures – Signatures are used to verify that a file has not been modified in any way. A Hashing Algorithm (e.g. SHA256) produces a hash of the file. A Private Key encrypts the hash. When a machine receives the signed file, the receiving machine generates its own hash of the file. Then it decrypts the file’s signature using the Public Key that is linked with the Private Key that encrypted it, and compares the hash in the file with the hash that the receiving machine generated. If a third party used it’s Private Key to encrypt the hash, then the receiving machine needs a copy of the third-party’s Public Key to decrypt the hash.
- Third party Signatures – A file can be hashed and signed by the same organization that created the file. Or the file can be hashed and signed by a different organization (third party). When a third party signs a file that it did not generate, the assumption is that the 3rd party verified the authenticity of the file. Only the third party’s Public Key can be used to verify the signature.
Certificate Signature – Certificates are usually digitally signed by a trusted third party, formally known as the Certificate Authority. This third party verifies that the organization that produced the certificate actually owns the server that is hosting the certificate. The name of the Certificate Authority is contained in the Issuer field of the certificate.
- CAs use the CA’s Private Key to sign a certificate – The CA generates a hash of the certificate and signs the hash using the CA’s Private Key. Later, the client machine will use the CA’s Public Key to verify the hash.
- Self-signed certificate – Instead of using a third party’s private key to encrypt the hash, it’s also possible for a certificate to use its own key pair to sign its own certificate. In this case, the Issuer of the certificate and the Subject of the certificate are the same. Most browsers will not accept self-signed certificates.
- CA Chain – The CA Certificate (with private key) that signed the Server Certificate can itself be signed by a different CA Issuer with its own private key. In this case, the Issuer of the CA’s certificate is a different CA certificate. For example: Intermediate CA 2 signs your Server Certificate. Intermediate CA 1 signs Intermediate CA 2’s certificate. Root CA signs Intermediate CA 1’s certificate.
- Root CA certificates are self-signed– As you go up the CA certificate chain, you’ll eventually reach a Root CA certificate that is signed by itself, and not by a different CA certificate. Thus all CA Root Certificates are self-signed.
- Root CA certs are installed on SSL clients. CA’s pay Microsoft, Chrome, Mozilla, Apple, etc. to install their Root CA Certificates on client machines. The Root CA certificates contain the Public Keys that clients use to verify CA signatures on certificates.
- Intermediate CA certificates – In the CA Signature Chain, between the Server Certificate and the Root CA Certificate are Intermediate CA Certificates. These Intermediate CA certificates are not installed on client machines. Instead, they must be transmitted to the SSL Client by the SSL Server. On a NetScaler, you link Intermediate CA Certificates to the Server Certificate, which causes the NetScaler to transmit multiple certificates during the SSL Handshake. On IIS, you simply install the Intermediate Certificate in the web server computer’s Intermediate CA certificate store.
- The Root CA certificate must never be transmitted by the SSL Server; only the Intermediate CA certificates must be transmitted. The Root CA certificate is already installed on the client machine through an out-of-band operation (e.g. included with the operating system). If the Root CA certificate could be transmitted by the SSL Server, then the entire third party trust model is broken. (SSL Labs will report this as “Chain issues: Contains anchor”.)
- Internal Certificate Authority – you could build your own internal Certificate Authority and use it to sign your certificates. This can be a Microsoft CA server. Or you can use your NetScaler as a CA.
- Internal CA Root Cert certificate distribution – Since your CA’s Root Certificate is not installed on the client devices, your internally-signed certificates won’t be trusted. You must find some out-of-band mechanism for distributing the CA Root Certificate. Typically this distribution is performed using Group Policy.
- Self-signed certificates, and internally-signed certificates, are two different things. If you build your own Certificate Authority, and use it to sign your certificates, that is not self-signed, because the Issuer and Subject are different. Self-signed only occurs when the Issuer and the Subject are the same.
To create a certificate – The process to create a certificate is as follows:
- Create keyfile – Use OpenSSL or similar to create a public/private key pair file.
- Create CSR – Use OpenSSL or similar to create a Certificate Signing Request (CSR). The CSR contains the public key, and several other fields, including: server’s DNS name, and name of the Organization that owns the server.
- Send CSR to CA – The CSR is sent to a Certificate Authority (CA) to get a signature. Public CAs usually charge a fee for this service.
- CA verifies ownership – The CA verifies that the Organization Name specified in the CSR actually owns the server’s DNS name. One method is to email somebody at the same organization. More stringent verifications include background checks for the DUNS name. Higher verification usually requires a higher fee.
- CA signs the certificate – If owner verification is successful, then the CA signs the certificate and sends it back to the administrator. The CA can use multiple chained CA Certificates to sign your Server certificate.
- Complete certificate request – The administrator links the signed certificate with the keyfile. In IIS, this is called “Complete Certificate Request”. In NetScaler, when installing a cert-key pair, you browse to both the certificate file, and the key file.
- Create a TCP/SSL 443 listener and bind the certificate to it – Configure the web server to use the certificate. In IIS, add a https binding to the Default Web Site and select the certificate. In NetScaler, create a SSL vServer, and bind the certificate to it.
Keys and Certificates are stored in separate files – On UNIX/Linux, certificates/keys are stored in two files – a key file (.key extension), and a certificate file (.pem, .cer, .cert extension). Both are Base64 encoded, which is also known as PEM. PEM is the native format that NetScaler uses.
- Private Keys on Windows – On Windows, the keys are stored in a hidden portion of the file system that is not intended to be accessed by users. Instead, you can export a certificate and keys to a password-encrypted PFX file, which is also known as PKCS#12. Newer versions of NetScaler can directly import PFX files, while older versions of NetScaler require the PFX files to be converted to PEM format first.
- Private key linked to certificate? – On Windows, when you double-click a certificate in the IIS Console, or the Certificate MMC snap-in, the bottom of the first General page indicates if there’s a private key linked to this certificate. Only certificates with private keys can be used with IIS website hosting. Certificates without private keys are usually CA certificates (root or intermediate).
- DER Format – On Windows, if your certificate doesn’t have a private key associated with it, then you can save the certificate to a file in DER format, or in Base64 (PEM) format. Newer versions of NetScaler can directly import DER files automatically, while older versions of NetScaler require you to specify DER format manually while importing the certificate file.
- Certificate/key file storage on NetScaler – On NetScaler, certificate files and key files are stored in /nsconfig/ssl. The NetScaler configuration for SSL certificates points to files in this directory.
Private Keys should be encrypted – the key file that contains the Private Key should be encrypted, usually with a password. On NetScaler, when creating a key pair, you specify PEM Encoding, and set it to 3DES (Triple DES) encryption. You enter a permanent password, which is used to encrypt the private key.
- When converting a PFX file to PEM, the new private key in the PEM file should be encrypted. Specify a 3DES password when performing this conversion.
- Hardware Security Module (HSM) – Another storage option for private keys is a Hardware Security Module (HSM). HSMs are physical devices that can’t be physically compromised. The NetScaler FIPS appliances include a HSM module inside the appliance. Or, you can connect a NetScaler to a network-addressable HSM appliance. The HSM performs all private key operations so that the private key never leaves the HSM device.
- Smart Card – Another storage option for private keys is a Smart Card. Smart Cards require the user to enter a PIN to unlock the smart card to use the private key, similar to an HSM. Smart Cards are typically used with Client Certificates, which are detailed later.
Subject field – One of the fields in the Server Certificate is called Subject. On the far right of the subject is CN=, which means Common Name, which might be familiar to LDAP (e.g. Active Directory) administrators. The Common Name is the DNS name of the web server.
- User enters FQDN in browser address bar – User opens a browser and enters https://FQDN in the browser’s address bar to connect to the web server.
- FQDN in browser’s address bar is matched to Certificate Subject Name – The Server Certificate is downloaded from the web server. The Common Name of the certificate is then checked to make sure it matches what the user entered in the browser’s address bar. If they don’t match, then a certificate error is displayed.
Subject Alternative Names – Another related Certificate field is called Subject Alternative Names (SAN). The Subject field (Common Name) only supports a single DNS name. SAN can support as many DNS names as desired.
- Public CAs charge extra for each additional SAN name. And there are restrictions on the names can be added to the SAN field.
- Common Name must be added to SAN list – When you create a CSR and specify a Common Name, Public CAs always add this Common Name to the SAN list. Chrome recently started requiring that the FQDN in the browser’s address bar to be in the Server Certificate’s SAN list. For Public CA-issued certificates, this is not a problem, since they automatically add the name to the SAN list. However, if you issued certificates from an internal CA, then the FQDN might be missing from the SAN List.
Wildcard certificate – The Certificate Common Name can be a wildcard (e.g. *.company.com) instead of a single DNS name. This wildcard matches all FQDNs that end in .company.com. The wildcard only matches one word and no periods to the left of .company.com. It will match www.company.com, but it will not match www.gslb.company.com, because there’s two words instead of one. It will also not match company.com because it requires one word where the * is located.
- Wildcard certificates cost more from Public CAs than single name certificates and SAN certificates.
- Wildcard certificates are less secure than single name certificates, because you typically use the same wildcard certificate on multiple web servers, and if one of the servers is compromised, then all are compromised.
CA Signature verification – In addition to verifying the certificate Common Name, the browser also verifies that it trusts the CA signatures. On every client machine is a pre-installed list of CA root certificates. These root certificates are used to verify the CA signatures on the certificate. The signature is created with the CA’s private key, and the signature is verified using the CA’s public key, which is in the CA’s root certificate.
- Certificate chain – The SSL server’s certificate might be signed by more than one CA certificate, which are linked together in a chain. The SSL cert is signed by one of more intermediate certificates, which are signed by a root certificate.
- Intermediate certificates – The NetScaler must be configured to send both the SSL cert and the intermediate certificates (usually only one intermediate certificate). The root certificate is already on the client machine so the NetScaler must not send the root. To send both SSL cert and intermediate, you install the intermediate certificate on the NetScaler, and Link them together.
Validity Dates – Another field in the certificate is expiration date. If the date is expired, then the client’s browser will show a certificate error. When you purchase a certificate, it’s only valid between 1 year and 5 years, with CAs charging more for longer terms.
- Expiration Warning – NetScaler MAS can alert you when a certificate is about to expire. The CA will also remind you to renew it.
- Renew with existing keys? – When renewing the certificate, you can create a new SSL key pair, or you can use the existing key pair. If you create a new key pair, then you need to create a new CSR and submit it to the CSR. If you intend to use the existing keys, then simply download the updated certificate from the CA after paying for the later expiration date.
Certificate Revocation – certificates can be revoked by a CA. The list of revoked certificates is stored at a CA-maintained URL. Inside each SSL certificate is a field called CRL Distribution Points, which contains the URL to the Certificate Revocation List (CRL). Client browsers will connect to the CRL to verify that the SSL server’s certificate has not been revoked. Revoking is usually necessary when the web server’s private key has been compromised.
- CAs might revoke a certificate when Rekey – if you rekey a certificate, the certificate with the former keys might be revoked. This can be problematic for wildcard certificates that are installed on multiple machines, since you only have a limited time to replace all of them. Pay attention to the CA’s order form to determine how long before the prior certificate is revoked.
- Online Certificate Status Protocol – An alternative form of revocation checking is Online Certificate Status Protocol. The address for the OCSP server can be found in the certificate’s Authority Information Access field.
Client Certificate – Another type of certificate is Client Certificate. This is a certificate with private key, just like a Server Certificate. Client Certificates are installed on client machines, and usually are not portable (the private key can’t be exported). Client machines use the Client Certificate to authenticate with web servers. This client certificate authentication can simply verify the presence of the certificate on a corporate managed device. Or the user’s username can be extracted from the client certificate and used to authenticate the user.
- Smart cards have client certificates with private key installed on them. The client certificate on the smart card authenticates the user to the web server (or NetScaler). Users enter a PIN number to unlock the smart card to use the client certificate’s private key. Smart Cards eliminate needing to enter a password to authenticate with a web server.
- Virtual Smart Cards – There are also virtual smart cards, which use hardware features of the client device to protect the client certificate. The client’s TPM (Trusted Platform Module) encrypts the private key. If the client certificate were moved to a different device, then the new device’s TPM wouldn’t be able to decrypt the private key. Windows Hello for Business and Passport for Work is an example of this technology.
Cipher Suites are negotiated – During the SSL Handshake, the SSL Client and the SSL Server negotiate which cipher algorithms to use. A detailed explanation of ciphers would require advanced mathematics, but here are some talking points:
- Recommended list of ciphers – Security professionals (e.g. OWASP, NIST) publish a list of the recommended, adequately secure, cipher suites. These ciphers are chosen because of their very low likelihood of being brute force decrypted within a reasonable amount of time.
- Higher security means higher cost. For SSL, this means more CPU on both the web server (NetScaler), and SSL Client. You could go with high bit-size ciphers, but they require exponentially more hardware.
- Each cipher suite is a combination of cipher technologies. There’s a cipher for key exchange. There’s a cipher for bulk encryption. And there’s a cipher for message authentication. So when somebody says “cipher”, they really mean a suite of ciphers.
Ephemeral keys and Forward Secrecy – ECDHE ciphers are ephemeral, meaning that if you took a network trace, and if you had a the web server’s private key, it still isn’t possible to decrypt the traffic. This is also called Forward Secrecy.
- DHE = Ephemeral Diffie Hellman, which provides Forward Secrecy.
- EC = Elliptical Curve, which is a formula that allows smaller key sizes, and thus faster computation.
- How to trace when Forward Secrecy is enabled? – Since Forward Secrecy packet traces can’t be decrypted, when you want to trace SSL traffic for troubleshooting, you must first disable all ECDHE cipher suites.
Ciphers in priority order – The SSL server is configured with a list of supported cipher suites in priority order. The top cipher suite is preferred over lower cipher suites.
- The highest common cipher between SSL Server and SSL Client is chosen – When the SSL Client starts an SSL connection to an SSL Server, the SSL Client transmits the list of cipher suites that it supports. The SSL Server then chooses the highest matching cipher suite. If neither side supports any of the same cipher suites, then the SSL connection is rejected.
Citrix-recommended cipher suites – See https://www.citrix.com/blogs/2016/06/09/scoring-an-a-at-ssllabs-com-with-citrix-netscaler-2016-update/ for the list of cipher suites that Citrix currently recommends. Every SSL vServer created on the NetScaler should be configured with this list of cipher suites in the order listed.
- GCM ciphers seem to be preferred over CBC ciphers.
- EC (Elliptical Curve) ciphers seem to be preferred over non-EC ciphers.
- Ephemeral ciphers seem to be preferred over non-Ephemeral ciphers.
NetScaler Authentication Overview
NetScaler supports several different authentication mechanisms.
Single Sign-on – Once a user has been authenticated by NetScaler, NetScaler can usually perform Single Sign-on to the back-end resource (webpage, Citrix StoreFront, etc.).
- Different authentication methods for client and server – The authentication method used to authenticate the user doesn’t have to match the authentication method used for Single Sign-on. For example, NetScaler can use LDAP+RADIUS on the client side, and convert it to Kerberos on the server side.
LDAP for Active Directory authentication – LDAP is just one of the authentication mechanisms supported by Active Directory. Another common authentication mechanism is Kerberos. Kerberos and LDAP are completely different technologies.
- HTML logon page – LDAP credentials are typically entered in a HTML Form logon page.
RADIUS is typically used for two-factor authentication (e.g. passcodes).
- Enable RADIUS in the two-factor product – To integrate a two-factor authentication product with NetScaler, you typically enable RADIUS protocol in the two-factor authentication product.
RADIUS is typically combined with LDAP.
- A single HTML logon page can request the AD password and two-factor passcode at the same time.
- NetScaler logs the user into Active Directory using LDAP.
- Two-factor passcodes are verified using RADIUS.
SAML is primarily used to federate authentication between two or more organizations. The resources the user is trying to access (web sites) is in one organization, but authentication is handled in a different organization.
- With SAML, the user’s password stays at the authenticating organization. The resource organization never sees the user’s password.
- Trust between the two organizations is provided by certificates. The two organizations share certificates with each other to verify each other’s identity (signatures).
Kerberos is another method of authenticating with Active Directory.
- Kerberos and LDAP authentication are completely different authentication methods.
- To perform Kerberos authentication, the client machine must be able to communicate with a Domain Controller. This means Kerberos usually doesn’t work on the Internet, at least not without a VPN connection.
- If NetScaler uses Kerberos to Single Sign-on to the back-end web servers, then NetScaler needs connectivity to internal DNS and internal Domain Controllers.
OAUTH allows a user to delegate credentials to a program. The program can log into a different Service (HTTP API) using the user’s authorization. For example, Outlook with ShareFile Plug-in can automatically log into ShareFile without prompting the user for ShareFile credentials every time Outlook communicates with ShareFile.
- Access Token – OAuth Authorization Server returns an Access Token to the Client, which can then be used by the client to authenticate to HTTP-based APIs. The Access Token has specific API permissions attached to it.
- OpenID Connect is an authentication mechanism for OAUTH. It adds an ID Token to the Access Token. The response from the OAuth Server is two tokens: Access Token, and ID Token.
- Subject Identifier– ID Token contains the user’s email address and a subject ID. Web servers can get more info about the user by sending an HTTP Request with Access Token to a UserInfo endpoint (API) on the OAuth server. Since UserInfo is an API, the Access Token grants access.
- Book – For an excellent book on OAuth and OpenID Connect, see OAuth 2 in Action.
NSIP is Source IP – By default, NetScaler uses its NSIP (management IP) as the Source IP when communicating with authentication servers.
- To use SNIP, load balance – You can force authentication to use SNIP by load balancing the authentication servers, even if there’s only one authentication server.
LDAP (Lightweight Directory Access Protocol)
LDAP authentication process:
- HTML Form to gather credentials – NetScaler prompts user to enter a username and password, typically from an HTML Form.
- Connect to LDAP Server – NetScaler connects to the LDAP Server on TCP 389 or TCP 636, depending on if encryption is enabled or not.
- NetScaler logs into LDAP using a Bind account. This LDAP Bind account is an Active Directory service account whose password never expires. The only permissions it needs is to be able to search the LDAP directory, which is provided by being a member of the Domain Users group.
- NetScaler sends an LDAP Query to the LDAP Server. The LDAP Query asks the LDAP Server to find the username that was entered in step 1. The LDAP Server returns the user’s full Distinguished Name (DN), which is the full path to the user’s account in the directory.
- NetScaler’s LDAP Search Filter controls the LDAP Query that is sent in step 4. A common usage is to only return users that are members of a specific AD Group.
- Login as user’s DN – NetScaler reconnects to the LDAP Server but this time logs in as the user’s DN (from step 4), and the user’s password (from step 1).
- Extract attributes – After authentication, NetScaler can be configured to extract attributes from the user’s LDAP account. A common configuration is to extract the user’s group membership. Another is to get the user’s userPrincipalName so it can be used during Single Sign-on to back-end web servers.
Password expiration requires LDAP encryption – If the user’s password has expired, then NetScaler can prompt the user to change the password. However, the user’s password can only be changed if the LDAP connection is encrypted, which means certificates must be installed on the LDAP servers (Active Directory Domain Controllers).
- NetScaler does not inform the user how long before the user’s password expires.
LDAP Communication Protocols:
- Clear text LDAP connects to the LDAP Server on TCP 389.
- Two encrypted LDAP protocols – If the LDAP server has a certificate, then you can use one of two different protocols to connect to the LDAP Server. NetScaler supports both encrypted connection methods.
- LDAPS – this is encrypted LDAP. It’s a different port than LDAP, just like HTTPS is a different port than HTTP. LDAPS is typically TCP 636 on the LDAP Server.
- LDAP TLS – it starts as a clear text connection to the LDAP Server on TCP 389. Then both sides of the connection negotiate encryption parameters, and switch to encrypted communication on TCP 389.
RADIUS (Remote Authentication Dial-In User Service)
RADIUS authentication process:
- HTML Form to gather credentials – NetScaler prompts user to enter username and passcode, typically from an HTML Form.
- Send login request to RADIUS server – NetScaler sends a login request (Access-Request) to the RADIUS Server on UDP 1812. Since it’s UDP, there’s no acknowledgment from the Server.
- Passcode encryption using shared secret – The user’s passcode in the RADIUS packet is encrypted using a shared secret key that is configured on the NetScaler and RADIUS Server. The secret key entered on NetScaler must match the secret key configured on the RADIUS Server. Each RADIUS Client usually has a different secret key.
- RADIUS Attributes – The RADIUS Client (NetScaler) adds RADIUS attributes to the packet to help the RADIUS Server identify how the user is connecting.
- RADIUS Server:
- RADIUS Clients configured on RADIUS Server – The RADIUS Server first verifies that the RADIUS Client (NetScaler) is authorized to perform authentication. The NAS IP (NetScaler Source IP) of the RADIUS Access-Request packet is compared to the list of RADIUS Clients configured on the RADIUS Server. If there’s no match, RADIUS does not respond.
- Shared secret – The RADIUS server finds the RADIUS Client and looks up the shared secret key. The secret key decrypts the passcode in the RADIUS Access-Request packet.
- Verify RADIUS Attributes – RADIUS Server uses the RADIUS Attributes in the Access-Request packet to determine if authentication should be allowed or not.
- Authenticate the user – RADIUS authenticates the user. Most RADIUS Server products have a local database of usernames and passwords. Some can authenticate with other authentication providers, like Active Directory.
- Access-Accept and Attributes – RADIUS sends back a Access-Accept message. This response message can include RADIUS Attributes, like a user’s group membership.
- RADIUS Challenge – RADIUS Servers can send back an Access-Challenge, which asks the user for more information. NetScaler displays the RADIUS-provided Challenge message to the user, and sends back to the RADIUS Server whatever the user entered.
- SMS authentication uses RADIUS Challenge. The RADIUS server might send a SMS passcode to a user’s phone using SMS. Then RADIUS Challenge prompts the user to enter the SMS passcode.
- Extract RADIUS Attributes – NetScaler can be configured to extract the returned RADIUS Attributes and use them for authorization (e.g. AAA Groups).
RADIUS Client – RADIUS will not work unless you ask the RADIUS administrator to add NetScaler NSIP (or SNIP if load balancing) as a RADIUS Client.
- Secret key – The RADIUS administrator then gives you the secret key that was configured for the RADIUS Client. You enter this secret key in NetScaler when configuring RADIUS authentication.
SAML (Security Assertion Markup Language)
SAML uses HTTP Redirects to perform its authentication process. This means that Web Clients that don’t support Redirects (e.g. Citrix Receiver) won’t work with SAML.
SAML SP – The resource (webpage) the user is trying to access is called the SAML SP (Service Provider). No passwords are stored here.
SAML iDP – The authentication provider is called the SAML iDP (Identity Provider). This is where the usernames and passwords are stored.
SAML SP Authentication Process:
- User tries to access a NetScaler VIP (NetScaler Gateway, or Load Balancing/AAA) that is configured for SAML SP Authentication.
- NetScaler creates a SAML Authentication Request and signs it using its certificate (with private key).
- NetScaler sends to the user’s browser the SAML Authentication Request, and a HTTP Redirect (301) that tells the user’s browser to go to the SAML iDP’s authentication Sign-on URL.
- The user’s browser redirects to the iDP’s Sign-on URL and gives it the SAML Authentication Request that was provided by the NetScaler.
- The SAML iDP verifies that the SAML Authentication Request was signed by the NetScaler’s certificate.
- The SAML iDP authenticates the user. This can be a webpage that asks for two-factor authentication. Or it can be Kerberos Single Sign-on. It can be pretty much anything.
- The SAML iDP creates a SAML Token (aka SAML Assertion) containing SAML Attributes. At least one of the attributes is Name ID, which usually matches the user’s email address. The SAML iDP can be configured to send additional attributes (e.g. group membership).
- The SAML iDP signs the SAML Token using its certificate (with private key).
- The SAML iDP sends the SAML Token to the user’s browser and asks the browser to Redirect (301) back to the SAML SP’s Assertion Consumer Service (ACS) URL, which is different from the original URL that the user requested.
- The user’s browser redirects to the ACS URL and submits the SAML Token.
- The SAML SP verifies that the SAML Token was signed by the SAML iDP’s certificate.
- The SAML SP extracts the Name ID (email address) from the SAML Token. Note that the SAML SP does not have the user’s password; it only has the user’s email address.
- The SAML SP sends back to the user’s browser a cookie that indicates that the user has now been authenticated.
- The SAML SP sends back to the user’s browser a 301 Redirect, which redirects the browser to the original webpage that the user was trying to access in step 1.
- The user’s browser submits the cookie to the website. The website uses the cookie to recognize that the user has already been authenticated, and lets the user in.
Multiple SAML SPs to one SAML iDP – The SAML iDP could support connections from many different SAML SPs, so the SAML iDP needs some method of determining which SAML SP sent the SAML Authentication Request. One method is to have a unique Sign On URL for each SAML SP. Another method is to require the SAML SP to include an Issuer Name in the SAML Authentication Request. In either case, the SAML iDP looks up the SAML SP’s information to find the SAML SP’s certificate (without private key), and other SP-specific information.
- SP Certificate – NetScaler uses a certificate to sign the SAML Authentication Request. This NetScaler certificate must be copied to the iDP.
- iDP Certificate – The iDP uses a certificate to sign the SAML Token. The iDP certificate must be installed on the NetScaler.
Shadow Accounts – The SAML iDP sends the user’s email address to the SAML SP. The SAML SP has its own authentication directory. The email address provided by the SAML iDP is matched with a user account at the SAML SP. Even though the user’s password is never seen by the SAML SP, you still need to create user accounts for each user at the SAML SP. These SAML SP user accounts can have fake passwords. The SAML SP user accounts are sometimes called shadow accounts.
SAML Token can be converted to other authentication methods – It is often desirable to convert a SAML Token to another form of authentication. For example, the SAML SP can use the Name ID from the SAML Token to perform a database lookup to retrieve the user’s authorizations..
- No passwords – Not having access to passwords limits the authentication options at the SAML SP. Without a password, you can’t authenticate to Active Directory using LDAP or NTLM. It’s also not possible to request a Kerberos ticket for the user since NetScaler doesn’t have the user’s password.
- SAML Token and SSON to Citrix XenDesktop – Kerberos can use User Certificates to authenticate to Active Directory without a password. Citrix Federated Authentication Service takes advantage of this by requesting a User Certificate for each SAML Name ID, and then using the User Certificate to login to XenDesktop using Kerberos. Each SAML Name ID needs a matching shadow account.
Kerberos authentication process (simplified):
- User tries to access a web page that is configured with Negotiate authentication.
- To authenticate to the web page, the user must provide a Kerberos Service ticket. The Kerberos Service ticket is requested from a Domain Controller.
- The Kerberos Service ticket is limited to the specific Service that the user is trying to access. In Kerberos parlance, the resource the user is trying to access is called the Service Principal Name (SPN). User asks a Domain Controller to give it a ticket for the SPN.
- Web Site SPNs are usually named something like HTTP/www.company.com. It looks like a URL, but actually it’s not. There’s only one slash, and there’s no colon. The text before the slash is the service type. The text after the slash is the DNS name of the server running the service that the user is trying to access.
- If the user has not already been authenticated with a Domain Controller, then the Domain Controller will prompt the user for username and password. The Domain Controller returns a Ticket Granting Ticket (TGT).
- The user presents the TGT to a Domain Controller and asks for a Service Ticket for the Target SPN. The Domain Controller returns a Service Ticket.
- The user presents the Service Ticket to the web page the user originally tried to access in step 1. The web page verifies the ticket and lets the user in.
The Service and the Domain Controller do not communicate directly with each other. Instead, the Kerberos Client talks to both of them to get and exchange Tickets.
Kerberos Delegation – The Kerberos Service Ticket only works with the Service listed in the Ticket. If that Service needs to talk to another Service on the user’s behalf, this is called Delegation. By default, Kerberos will not allow this Delegation. You can selectively enable Delegation by configuring Kerberos Constrained Delegation in Active Directory.
- In Active Directory Users & Computers, edit the AD Computer Account for the First Service. On the Delegation tab, specify the Second Service. This allows the first Service to delegate user credentials to the Second Service. Delegation will not be allowed from the First Service to any other Service.
Kerberos Impersonation – If NetScaler has the user’s password (maybe from LDAP authentication), then NetScaler can simply use those credentials to request a Kerberos Service Ticket for the user from a Domain Controller. This is called Kerberos Impersonation.
Kerberos Constrained Delegation – If the NetScaler does not have the user’s password, then NetScaler uses its own AD service account to request a Kerberos Service Ticket for the back-end service. The service account then delegates the user’s account to the back-end service. In other words, this is Kerberos Constrained Delegation. On NetScaler, the service account is called a KCD Account.
- The KCD Account is just a regular user account in AD.
- Use setspn.exe to assign a Kerberos SPN to the user account. This action unlocks the Delegation tab in Active Directory Users & Computers.
- Then use Active Directory Users & Computers to authorize Kerberos Delegation to back-end Services.
Negotiate – Kerberos and NTLM – Web Servers are configured with an authentication protocol called Negotiate (SPNEGO). This means Web Servers will prefer that users login using Kerberos Tickets. If the client machine is not able to provide a Kerberos ticket (usually because the client machine can’t communicate with a Domain Controller), then the Web Sever will instead try to do NTLM authentication.
- NTLM is a challenge-based authentication method. NTLM sends a challenge to the client, and the client uses the user’s Active Directory password to encrypt the challenge. The web server then verifies the encrypted challenge with a Domain Controller.
- Negotiate on client-side with NTLM Web Server fallback – NetScalers can use Negotiate authentication protocol on the client side (AAA or NetScaler Gateway). Negotiate will prefer Kerberos tickets. If Kerberos tickets are not available, then Negotiate can use NTLM as a fallback mechanism. In the NTLM scenario, NetScaler can be configured to connect to a domain-joined web server for the NTLM challenge process. By using a separate web server for the NTLM challenge, there’s no need to join the NetScaler to the domain.
HTTP URL format: e.g. https://www.corp.com:444/path/page.html?a=1&key=value
- https:// = the scheme. Essentially, it’s the protocol the browser will use to access the web server. Either http (clear-text) or https (SSL/TLS).
- www.corp.com = the hostname. It’s the DNS name that the browser will resolve to an IP address. The browser then connects to the IP address using the specified protocol.
- :444 = port number. If not specified, then it defaults to port 80 or port 443, depending on the protocol. Specifying the port number lets you connect to a non-standard port number, but firewalls might not allow it.
- /path/page.html = the path to the file that the Browser is requesting.
- ?a=1&key=value = query parameters to the file. The query clause beings with a ? immediately following the file name. Multiple parameters (key=value pairs) are separated by &. These parameters are typically generated using HTML forms. Query parameters are method for the HTTP Client to upload a small amount of data to the Web Server. There can be many parameters, or just one parameter.
URLs must be safe encoded (Percent encoded), meaning special characters are replaced by numeric codes (e.g. # is replaced by %23). See https://en.m.wikipedia.org/wiki/Percent-encoding.
HOST Header – one of the HTTP headers inserted into the HTTP Request Packet is named Host. The value of this header is based on whatever hostname the user typed into the browser’s address bar. It’s the part of the URL after the scheme (http://), and before the port number (:81) or path (/).
- Web Servers use the Host Header to determine which website content should be served. This allows a Web Server to host multiple websites on one IP address. In this configuration, if the Host Header is not set correctly, then you might not see the website content you were hoping for. If the Web Server requires the Host Header to be set to a particular DNS Name, and if you enter the IP address into your browser’s address bar, then you won’t see the website.
- NetScaler Load Balancing Monitors do not include the Host Header by default. If the Web Server requires the Host Header, then you must modify the NetScaler Monitor configuration to specify the Host header.
HTTP Body – The HTTP Body is the content you are transporting across the network. The HTTP body of an HTTP Request or HTTP Response can contain data, or a file.
- Data can be HTML Form Parameters, JSON object, XML document, etc. These are detailed below.
- File can be HTML file, .iso file, .png file, etc.
- HTTP Body vs HTML Body – HTTP Body and HTML Body are completely different. An HTTP Body can contain an entire HTML file, which has a HTML Header and HTML Body. Or an HTTP Body can contain non-HTML file or data. HTML is just one of the file types that an HTTP Body can transport.
Client-side Data Storage – Web Servers sometimes need to store small pieces of data in a user’s web browser. The user’s browser is then required to send the data back to the web server with every HTTP Request.
Set-Cookie – Web Servers add a Set-Cookie header to the HTTP Response. This Response Header contains a list of Cookie Names and Cookie Values.
Cookies are linked to domains – The Web Browser stores the Cookies in a place that is associated with the DNS name (host name) that was used to access the web site. The next time the user submits an HTTP Request to that DNS name, all Cookies associated with that host name are sent in the HTTP Request using the Cookie Request header.
- Notice that the two headers have different names. HTTP Response has a Set-Cookie header, while HTTP Request has a Cookie header.
Cookie security – Cookies from other domains (other DNS names, other web servers) are not sent. Cookies usually contain sensitive data (e.g. session IDs) and must not be sent to the wrong web server. Hackers will try to steal Cookies so they can impersonate the user.
Cookie lifetimes are either Session Cookies, or Persistent Cookies. Session Cookies are stored in the browser’s memory and are deleted when the browser is closed. Persistent Cookies are stored on disk and available the next time the browser is launched.
- Expiration date/time – Persistent Cookies are sent from the Web Server with an expiration date/time. This can be an absolute time, or a relative time.
NetScaler Cookie for Load Balancing persistence – NetScaler can use a Cookie to maintain load balancing persistence. The name of the Cookie is configurable. The Cookie lifetime can be Session or Persistent.
Web Server Sessions
Web Server Sessions preserve user data for a period of time – When users log into a web site, or if the data entered by a user (e.g. shopping cart) needs to be preserved for a period of time, then a Web Server Session needs to be created for each user.
Web Server Sessions are longer than TCP Connections – Web Server Sessions live much longer than a single TCP Connection, so TCP Connections cannot delineate a session boundary.
Each HTTP Request is singular – There’s nothing built into HTTP to link one HTTP Request to another HTTP Request. Various fields in the HTTP Request can be used to simulate a Web Server Session, but technically, each HTTP Request is completely separate from other HTTP Requests, even if they are from the same user/client.
Server-side Session data, and Client-side Session ID – Web Server Sessions have two components – server-side session data, and some kind of client-side indicator so the web server can link multiple HTTP Requests to the same server-side session.
A Cookie stores the Session ID – On the client-side, a session identifier is usually stored in a Cookie. Every HTTP Request performed by the client includes the Cookie, so the web server can easily associate all of these HTTP Requests with a Server-side session.
Server-side data storage – On the server-side, session data can be stored in several places:
- Memory of one web server – this method is the easiest, but requires load balancing persistence
- Multiple web servers accessing a shared memory cache (e.g. memcached)
- Shared Database – each load balanced web server can pull session data from the database. This is typically slower than memory caches
Load Balancing Persistence and Web Server Sessions – some web servers store session data on the same web server the user initially connected to. If the user connects to a different web server, then the old session data can’t be retrieved, thus causing a new session. When load balancing multiple identical web servers, to ensure the user always connects to the same web server that was initially chosen by the user, configure Persistence on the load balancer.
Persistence Methods – When the user first connects to a VIP, NetScaler uses its load balancing algorithm to select a web server. NetScaler then needs to store the chosen server’s identifier somewhere. Here are common storage methods:
- Cookie – the chosen server’s identifier is saved on the client in a Cookie. The client includes the persistence Cookie in the next HTTP request, which lets NetScaler send the next HTTP Request to the same web server as before. Pros/Cons:
- No memory consumption on NetScaler
- Cookie can expire when the user’s browser is closed
- Each client gets a different Cookie, even if multiple clients are behind a common proxy.
- However, not all web clients support Cookies.
- Source IP – NetScaler records the client’s Source IP into its memory along with the web server it chose using its load balancing algorithm. Pros/Cons:
- Uses NetScaler Memory
- If multiple clients are behind a proxy (or common outgoing NAT), then all of these clients go to the same web server. That’s because all of them appear to be coming from one IP address.
- Works with all web clients.
- Rule-based persistence – use a NetScaler Policy Expression to extract a portion of the HTTP Request and use that for persistence. Ultimately, it works the same as Source IP, but it helps for proxy scenarios if the proxy includes the Real Client IP in one of the HTTP Request Headers (e.g. X-Forwarded-For).
- Server Identifier – the HTTP Response from a web server instructs the web client to append a Server ID to every URL request. The NetScaler can match the Server ID in the URL with the web server. XenMobile uses this method.
Authentication and Cookie Security – If a web site requires authentication, it would be annoying if the user had to login again with every HTTP Request. Instead, most authenticated web sites return a Cookie, and that Cookie is used to authorize subsequent HTTP Requests from the same user/client/browser.
- WAF Cookie Protection – Since the Web Session Cookie essentially grants permission, security of this Cookie is paramount. NetScaler Web App Firewall has several protections for Cookies.
Get Data from user – if the web site developer wants to collect any data from a user (e.g. Search term, account login, shopping cart item quantity, etc.), then the web developer creates HTML code that contains a <form> element.
Form fields – Inside the <form> element are one or more form fields that let users enter data (e.g. Name, Quantity), or let users select an option (drop-down box).
Submit button – The last field is usually a Submit button.
Field names – Each of the fields in the form has its own name.
GET and POST – The data is then submitted to the web server using one of two methods: GET or POST.
- With GET, each of the field names and field values is put in the Query Parameters portion of the URL (e.g. ?field1=value1&field2=value2), which is after the path and file name.
- With POST, the HTTP Request Method (HTTP Command) is set to POST, and the field names and field values are placed in the Body of the HTTP Request.
- The POST method is typically more secure. Web Servers can log the entire GET Method, including query parameters. But POST parameters in the body are never logged.
- NetScaler Web App Firewall (WAF) can do this inspection before the form data reaches the web server.
- WAF can also validate the form fields. For example, NetScaler WAF can ensure that only numeric characters can be entered in a zip code field.
Web App Firewalls for HTML Forms – HTML Forms are the most sensitive feature in any web application. Web Developers must write their web server code in a secure manner. Use features like NetScaler Web App Firewall to provide additional protection.
- WAF for JSON, XML – Other forms of submitting data to web servers, like JSON objects, XML documents, etc. should also be inspected. NetScaler Web App Firewall can do this too.
JSON vs XML –
- JSON is smaller than XML. XML is marked up with human-readable tags, bloating the size. JSON contains data, curly braces, colons, quotes, and square brackets. That’s it. Very little of it is dedicated to markup so most of it is pure data.
Commercial systems have a programmatically-accessible API (Application Programming Interface) that allows programs to control a remote system. Some API commands retrieve information from the system. Other API commands invoke actions (e.g. create object) in the system.
Use HTTP to call API functions – Modern APIs can be activated using the HTTP Protocol. Create a specially-crafted HTTP Request and send it to an HTTP endpoint that is listening for API requests.
SOAP Protocol – Older HTTP-based APIs operate by exchanging XML documents. This is called SOAP (Simple Object Access Protocol). However, XML documents are difficult to program, and the XML tags consume bandwidth.
REST API – Another newer HTTP-based architecture is to use all of the HTTP Methods (GET, POST, PUT, DELETE), and exchange JSON documents. JSON is leaner than XML.
- NetScaler Nitro API is a REST API.
REST is stateless. All information needed to invoke the API must be included in one HTTP Request.
Web Browsers typically only use GET and POST in their HTTP Requests. But there are other HTTP Methods like PUT and DELETE.
Programs other than browsers can use the HTTP protocol – Web Browsers are not the only programs that can make HTTP Requests. Every computer language has some mechanism for creating a HTTP Request and processing the HTTP Response. Non-browser Programs can use the HTTP Methods PUT and DELETE.
REST is HTTP.
- A REST-capable client is any client that can send HTTP Requests and process HTTP Responses. Some languages/clients have REST-specific functions. Others have only lower level functions for creating raw HTTP Requests.
- On Linux, use curl to send HTTP Requests to an HTTP-based API.
- In PowerShell, use Invoke-RestMethod to send an HTTP Request to an HTTP-based API.
- Inside a browser, use Postman or other REST plug-in to craft HTTP Requests and send them to an HTTP-based API.
To invoke an HTTP REST-based API:
- HTTP Request to login – Send an HTTP Request with user credentials. The exact login URL or session creation URL is in the API documentation.
- Session Cookie – The REST API server sends back a Session Cookie that can be used for authorization of subsequent REST/HTTP Requests. The REST Client saves the cookie, and adds it to every subsequent REST/HTTP Request.
- Read API Documentation – Use the API’s documentation to find the URLs and HTTP Methods (Commands) to invoke the API.
- Content-Type – Some REST API Requests require a specific Content-Type to be specified in the HTTP Request Header. Add it to the HTTP Request that you’re creating.
- JSON Object in Request – Most REST API Requests require a JSON object to be submitted in the HTTP Body. Use the language’s functions to craft a JSON object that contains the parameters that need to be sent to the API Call.
- URL Query portion – Some REST API Requests require parameters to be specified in the query portion of the URL.
- Send HTTP Request – Create the full HTTP Request with HTTP Method, URL, URL Parameters, Content-Type Header, Cookie Header, and JSON Body. Send it to the HTTP REST server endpoint.
- Process Response, including JSON – The REST API sends back a HTTP 200 success message with a JSON document. Or it sends back an error message, typically with error details in an attached JSON document.
NetScaler VPN Networking
SNIP vs IP Pool (Intranet IPs) – By default, when a NetScaler VPN Tunnel is established, a NetScaler SNIP is used as the Source IP for all traffic that leaves the NetScaler to internal Servers. The internal Servers then simply reply to the SNIP. Instead of all VPN Clients sharing a single SNIP, you can configure IP Pools (aka Intranet IPs), where each VPN Client gets its own IP address.
Use IP Pools if Servers initiate communication to clients – if servers initiate communication to VPN Clients (IP Phones, SCCM, etc.), then each VPN Client needs its own IP address. This won’t work if all VPN Clients are sharing a single SNIP.
Intranet IPs assignment – Intranet IPs can be assigned to the Gateway vServer, which applies to all users connected to that Gateway vServer. Or you can apply a pool of IPs to a AAA Group, which allows you to assign different IP Pools (IP subnets) to different AAA Groups.
IP Pools and Network Firewall – If different Pools for different AAA Groups, then a network firewall can control which destinations can be reached from each of those IP Pools.
Intranet IP Subnet can be brand new – the IP subnets chosen for VPN Clients can be brand new IP Subnets that the NetScaler is not connected to. NetScaler is a router, so there’s no requirement that the IP addresses assigned to the VPN Clients be on one of the NetScaler’s data (VIP/SNIP) interfaces.
Reply traffic from Servers to Intranet IPs – If the Intranet IP Pool is a new IP Subnet, then on the internal network (core router), create a static route with the IP Pool as destination, and a NetScaler SNIP as Next Hop. Any SNIP on the NetScaler can reach any VPN Client IP address.
IP Spillover – if there are no Intranet IPs (IP Pool) available, then a VPN Client can be configured to do one of the following: use the SNIP, or transfer an existing session’s IP. This means that a single user can only have a single Intranet IP from a single client machine.
Split Tunnel – by default, all traffic from the VPN Client is sent across the VPN Tunnel. For high security environments, this is usually what you want, so the datacenter security devices can inspect the client traffic. Alternatively, Split Tunnel lets you choose which traffic goes across the tunnel, while all other client traffic goes out the client’s normal network connection (directly to the Internet).
Split Tunnel is enabled in a Session Policy/Profile – the Session Policy/Profile can be bound to the Gateway vServer, which affects all VPN users, or it can be bound to a AAA Group.
Intranet Applications define traffic that goes across the Tunnel – If Split Tunnel is enabled, then you must inform the VPN Client which traffic goes across the Tunnel, and which traffic stays local. Intranet Applications define the subnets and port numbers that go across the Tunnel. The Intranet Applications configuration is downloaded to the VPN Client when the Tunnel is established.
Intranet Applications – Route Summarization – If Split Tunnel is enabled, a typical configuration is to use a summarized address for the Intranet Applications. Ask your network team for the smallest number of network prefixes that matches all internal IP addresses. For example, every private IP address (RFC 1918) can be summarized by three route prefixes. The summarized Intranet Applications can then be assigned to all VPN Clients. Most networking training guides explain route summarization in detail.
Intranet Applications – Specific Destinations – Alternatively, you can define an Intranet Application for every single destination Server IP and Port. Then bind different “specific” Intranet Applications to different users (AAA Groups). Note: this option obviously requires more administrative effort.
Split DNS – If Split Tunnel is enabled, then Split DNS can be set to Remote, Local, or Both. Local means use the DNS servers configured on the Client. Remote means use the DNS Servers defined on the NetScaler. Both will check both sets of DNS Servers.
There are three methods of controlling access to internal Servers across the VPN Tunnel – Authorization Policies, Network Firewall (usually with Intranet IPs), and Intranet Applications (Split Tunnel).
Authorization Policies control access no matter how the VPN Tunnel is established. These Policies use NetScaler Policy Expressions to select specific destinations and either Allow or Deny. In NetScaler 11.1 and older, Authorization Policies use Classic Policy Expressions only, which has a limited syntax. In NetScaler 12 and newer, Authorization Policies can use Default Syntax Policy Expressions, allowing matching of traffic based on a much broader range of conditions.
Intranet Applications – If Split Tunnel is enabled, then Intranet Applications can be used to limit which traffic goes across the Tunnel. If the Intranet Applications are “specific”, then they essentially perform the same role as Authorization Policies. If the Intranet Applications are “summarized”, then you typically combine them with Authorization Policies.
Network firewall (IP Pools) – If Intranet IPs (IP Pools) are defined, then a network firewall can control access to destinations from the VPN Client IPs. If Intranet IPs are not defined, then the firewall rules apply to the SNIP, which means every VPN Client has the same firewall rules.
VPN Tunnel Summary
In Summary, to send traffic across a VPN Tunnel to internal servers, the following must happen:
- If Split Tunnel is enabled, then Intranet Applications identify traffic that goes over the VPN tunnel. Based on Destination IP/Port.
- Authorization Policies define what traffic is allowed to exit the VPN Tunnel to the internal network.
- Static Routes for internal subnets – to send traffic to a server, the NetScaler needs a route to the destination IP. For VPN, NetScaler is usually connected to both DMZ and Internal, with the default route (default gateway) on the DMZ. To reach remote internal subnets, you need static routes.
- Network Firewall must allow the traffic from the VPN Client IP – either SNIP, or Intranet IP (IP Pool).
- Reply traffic – If the VPN Client is assigned an IP address (Intranet IPs aka IP Pool), then server reply traffic needs to route to a NetScaler SNIP. On the internal network, create a default route with IP Pool as destination, and a NetScaler SNIP as Next Hop.
Network Boot and PXE
Network Boot – Network Boot allows machines to download a bootstrap from a TFTP server while the machine is still in BIOS boot mode. No Operating System needed on the local hard drives.
PXE (Pre-boot Execution Environment) – PXE is a mechanism for Network Boot machines to discover the location of the bootstrap file. PXE is based on the DHCP protocol.
- Get IP from DHCP – Before a machine can perform a Network Boot, it needs an IP address, which is provided by DHCP.
- Discover TFTP Server address – Then the machine needs to discover the IP address of the TFTP Server, and the name of the bootstrap file that should be downloaded from the TFTP Server. PXE uses DHCP Protocol to perform this discovery.
Network Boot without PXE – You can also Network Boot without PXE by booting from an ISO file or local hard drive that has just enough code on it to get the rest of the bootstrap from the TFTP server. DHCP is still usually used to get an IP address, but the IP address of the TFTP server is usually burned into this locally accessible code.
NICs and PXE – Network cards (NICs) need Network Boot capability built into their BIOS. Almost every NIC, including virtual machine NICs, has this capability. A notable exception is Hyper-V Synthetic NICs; Hyper-V Legacy NIC can Network Boot, but Hyper-V Synthetic NICs cannot.
PXE works as follows:
- Boot from network – A machine’s BIOS is configured to boot from the Network.
- DHCP Request to get IP – The NIC performs a DHCP Request to get an IP address.
- PXE Request to get TFTP info – The NIC performs a PXE Request to get the TFTP IP address and file name.
- Download from TFTP – The NIC downloads the bootstrap file from the TFTP server and runs it.
- Run the bootstrap – The bootstrap file usually downloads additional files from a server machine (e.g. Citrix Provisioning Server) and runs them.
PXE vs DHCP
DHCP Options 66 and 67 – During step 2, if the NIC receives DHCP Options 66 and 67, which contain the TFTP Server’s IP address and file name, then there’s no need to perform step 3.
PXE Request to UDP 4011 – During step 3, the NIC performs another DHCP Request, but this time on port UDP 4011. A PXE Service listening on UDP 4011 responds with DHCP Options 66/67.
DHCP and PXE are different port numbers – A PXE Service can run on the same server that is running a DHCP Service because they listen on different port numbers. But typically, PXE Service and DHCP Service are on different servers.
PXE = DHCP Protocol – PXE Service uses the DHCP Protocol, which means the NIC performs a Layer 2 Broadcast, which does not cross a router. Many routers do not forward UDP 4011 to a remote PXE Service.
PXE Request does not cross routers – Thus Step 3 usuallly only works in the Network Boot client and the PXE Service are on the same subnet. If they are on different subnets, then use DHCP Options 66/67 instead.
DHCP Option 66 can only point to a single TFTP Server IP Address. You usually want redundancy.
NetScaler Load Balancing of TFTP – Use NetScaler to load balance two or more TFTP Servers and configure DHCP Option 66 to point to the NetScaler Load Balancing VIP.
DNS Round-Robin – Option 66 can point to a DNS Round Robin-enabled DNS name, where the single DNS name resolves to both TFTP Servers’ IP addresses. This assumes the DHCP Client receives DNS Server information from the DHCP Server.
Separate Option 66 configured on each DHCP Server – If you have multiple DHCP Servers, each DHCP Server can send back a different Option 66 TFTP Server IP address.
Network Boot Clients and PXE on same subnet – If Network Boot clients and PXE Services are on the same subnet, then each PXE Service can send back a different TFTP Server IP address. Either PXE Service can respond to the PXE Request, so if one is down, the other will respond.
Citrix Provisioning Services and PXE
How Citrix Provisioning Services (PvS) uses PXE:
- TFTP Server is installed on each PvS server.
- PXE Service can be enabled on each PvS server.
- PvS can create a Boot ISO that has the TFTP Server IP addresses built into them.
- Note: The Boot ISO uses a different TFTP Service than normal PXE. Normal TFTP is UDP 69, while the Boot ISO connects a TFTP service called Two-stage Boot TFTP Service that is listening on UDP 6969.
- PvS can burn boot code into a hard disk attached to a Network Boot machine. This works the same as the Boot ISO method.
- A DHCP Service (typically Microsoft) can be installed on the PvS servers. DHCP Service and PXE Service are two different things.
GSLB is only useful if a single DNS name can resolve to multiple IP addresses. If this is not the case, then it would be easier to just leave the single DNS name / single IP on regular DNS.
Limitations of DNS Servers when a single DNS name resolves to multiple IP addresses:
- The DNS Server doesn’t care if the IP address is reachable or not. There’s no monitoring.
- The DNS Server doesn’t know which IP address is closest to the user. There’s no proximity load balancing.
- The DNS Server can’t do site persistence, so you could get a different IP address every time you perform the DNS Query.
NetScaler GSLB is a DNS Technology that addresses these limitations. When you enable NetScaler GSLB, you enable your NetScaler appliances to resolve DNS Names. In other words, NetScaler appliances are essentially DNS Servers.
DNS names must be delegated to NetScaler DNS listeners. There are a few methods of doing this delegation:
- In the existing DNS zone, delegate specific DNS names to NetScaler DNS. Each DNS name needs a separate delegation.
- In the existing DNS zone, delegate an entire sub-zone (e.g. gslb.company.com) to NetScaler DNS. Then create CNAMEs for each DNS name that are alias’d to an equivalent DNS name in the sub-zone. For example: www.company.com is CNAME’d to www.gslb.company.com. Since the gslb.company.com sub-zone is delegated to NetScaler DNS, NetScaler will resolve this DNS name.
- Move the entire existing DNS zone to NetScaler. Note: NetScaler was never designed as a full-fledged DNS Service, so you might find limitations when choosing this option.
For redundancy, configure multiple NetScaler appliances/pairs for DNS delegation – At least two NetScaler appliances/pairs should be configured for DNS services. Delegate DNS names or subzones to two different NetScaler appliances/pairs.
- GSLB Configuration must be identical – Since it’s not possible to control which NetScaler appliance/pair resolves the DNS name, all NetScaler appliance/pairs must have identical GSLB configuration, so the DNS responses are always the same no matter which NetScaler appliance/pair resolves the DNS name.
- GSLB Sync – One NetScaler appliance/pair can replicate its GSLB configuration to another NetScaler appliance/pair. The sync communication protocol is SSH.
- GSLB Metric Exchange Protocol (MEP) – multiple GSLB-enabled appliances/pairs are configured to communicate with each other using a proprietary Metric Exchange Protocol. This communication is used for several purposes: Dynamic Proximity load balancing, Monitoring, and Persistence. These are detailed below.
Active/Passive – NetScaler GSLB can do active/passive, where the Active IP is given out if it is UP, and the Passive IP is given out if the Active IP is down. NetScaler needs to monitor the Active IP to verify it is reachable.
Active/Active – NetScaler GSLB can do active/active, where it load balances users across multiple Active IPs. A common load balancing algorithm is based on the client’s proximity to the Active IP; give out the Active IP that is closest to the user.
- Active/Active can be combined with Passive so that if all Active IPs are down, then give out the Passive IP.
Proximity Load Balancing Methods – there are two Proximity Methods:
- Static Proximity uses a location database stored on the appliance. The Source IP of the DNS Query is looked up in the location database to determine coordinates. The Active IPs are also looked up in the location database to determine coordinates. The coordinates are compared to determine which Active IP is closer to the Source IP. NetScaler has a built-in static proximity location database that you can use.
- Dynamic Proximity asks each NetScaler (through MEP) to ping the Source IP of the DNS Query to determine which ping is fastest.
Recursive DNS and Client IP – DNS Clients use Recursive DNS Servers to resolve DNS names. The Source IP of a DNS Query sent to NetScaler is the Recursive DNS Server’s IP Address, and not the actual DNS Client’s IP Address. Thus all proximity calculations are based on the IP address of the Recursive DNS Server and not the actual client. The Recursive DNS Servers are sometimes called Local DNS Servers (LDNS).
- ECS (EDNS-Client-Subnet) – Recursive DNS Servers can use the EDNS0 field on a DNS Request to include the client’s subnet. GSLB can then perform proximity calculations against ECS instead of against the Recursive DNS Server’s IP Address, thus improving accuracy. ECS functionality for GSLB is available in NetScaler 11.1 and newer.
GSLB Monitoring – GSLB has two methods of determining if an Active IP is up or not:
- MEP Monitoring – If the Active IP is a VIP on one of the NetScalers connected by MEP, then use MEP to ask the NetScaler for the UP/DOWN status of the VIP.
- Use regular Load Balancing Monitors – bind Load Balancing Monitors to the GSLB Service (Active IP). Monitors bound to a GSLB Service override MEP monitoring.
Monitoring of Internet circuits – If your Active IPs are Public IPs, then you don’t want to give out a Public IP if the IP is not reachable because the Internet circuit is down. There are a couple methods of monitoring the accessibility of an Internet circuit:
- Route MEP through the Internet – If the remote site’s Internet goes down, then MEP goes down, and the Active IPs in that remote site go down.
- Monitor a site-specific public IP address – Attach a monitor to the GSLB Service that monitors a public IP in the remote site. You probably have to specify the public IP inside the monitor configuration.
- Monitor local Internet? – If the appliance’s local Internet goes down, then it’s probably not a problem, because DNS Requests/Queries aren’t making it to this appliance anyways. It’s more important for the other NetScalers to determine Internet accessibility in this datacenter.
GSLB DNS TTL – TTLs (time-to-live) for GSLB DNS Responses is usually 5 seconds. Thus every 5 seconds, a DNS Client needs to resolve the GSLB DNS Name again. This allows for quick changes if an Active IP address is no longer reachable.
- Browsers have their own DNS cache that ignores the DNS TTL. For example, Internet Explorer tends to cache DNS responses for 30 minutes. In a GSLB failover scenario (Active IP is down), then the users will have to close all IE windows to clear the browser’s DNS cache. See Microsoft 263558 How Internet Explorer uses the cache for DNS host entries.
Site Persistence – because of the low TTL, it’s quite possible that a new DNS Query can return a different IP than the initial DNS Query. Some applications require you to always get the same IP address for a period of time. This is similar to Load Balancing Persistence needed for Web Sessions.
GSLB Site Persistence – GSLB has three methods of Site Persistence:
- Source IP – records DNS Query’s Source IP to GSLB Active IP in the NetScaler’s memory. If Source IP is the Recursive DNS Server, then many clients might end up with the same Active IP response. Accordingly, Source IP persistence is usually not recommended unless Cookie Persistence is not an option. Note: ECS in NetScaler 11.1 can be used for Source IP persistence.
- Cookie Persistence – For Active IPs that are HTTP VIPs on one of the NetScaler appliance/pairs connected by MEP, the first HTTP Response from the NetScaler VIP will include a cookie indicating which GSLB Site the Response came from. The HTTP Client will include this Site Cookie in the next HTTP Request to the VIP. If the DNS TTL expires (5 seconds), then DNS Query must be performed again. If the client gets a different IP address in a different GSLB Site, then the Site Cookie sent to that NetScaler will be for a different GSLB Site than the VIP it’s now trying to access. NetScaler has two options for getting the HTTP Request from the wrong VIP to the correct VIP:
- Redirect – redirect the user to a different DNS name that is site-specific (a Site Prefix is added to the original DNS name). This requires the certificate to match the original GSLB-enabled DNS name, plus the new site-specific DNS name.
- Proxy – proxy the HTTP Request to the correct GSLB Site. This means that NetScaler in the wrong GSLB Site must be able to forward the HTTP Request to the VIP in the correct GSLB Site.
GSLB is not in the data path – GSLB is nothing more than DNS: it resolves DNS names to IP addresses. Once the Client has the IP address, it creates a TCP connection directly to that IP address, and GSLB is no longer involved.
- GSLB Site Persistence cookies – since GSLB = DNS, and not HTTP, it’s not possible to include an HTTP Cookie in the DNS Response. The HTTP Cookie isn’t added until the the HTTP Client makes its first request to an HTTP/SSL VIP on one of the NetScaler appliances/pairs that is participating in GSLB.
Other GLSB Use Cases
GSLB and Multiple Internet Circuits – a common use case for GSLB is if you have multiple Internet circuits connected to a single datacenter and each Internet circuit has a different public IP subnet. In this scenario, you have one DNS name, and multiple public IP addresses, which is exactly the scenario that GSLB is designed for.
- Local Internet Circuit Monitoring – GSLB Services need a monitor that can determine if the local Internet is up or not. You don’t want to give out a Public IP on a particular Internet circuit if the local Internet circuit is down. You typically configure the GSLB Monitor to ping a circuit-specific IP address (router).
Internal GSLB – GSLB can also be used internally to give out internal IPs to DNS queries. However, there are a couple differences when compared to public IPs:
- Private IPs are not in the location database. If doing static proximity load balancing, you must manually add each internal subnet to the location database.
- Internal IPs are not affected by Internet outage. Thus GSLB monitoring for internal Active IPs is usually configured differently than GSLB monitoring for public Active IPs.
Mix internal and public GSLB on the same appliance? – Since internal and public have different GSLB monitoring configurations, you need separate GSLB configurations for internal and public. However, you can’t assign the same DNS name to two different GSLB configurations. Here are three options:
- Use different DNS names for public and internal.
- Don’t use the same NetScaler for both public and internal.
- Configure internal DNS to CNAME to a different internal DNS name.