Clarification of Step Order Netscaler HA Setup with FIPS

I found an old response, believe that this is a repeat conversation.  I will try again using this advice as this conversation came around again as it was not working for us.  -  The advice "You really need to configure the FIPS and HSM before you do the HA or anything else. Do this first.  Do FIPS, do the HA config, then configure storefront/gateway"

The secondary is that we need to use a WildCard certificate.  If there is anything additional to add here, please let me know.

Thanks

I've done several of these configurations, 12 Netscalers at one contract alone.  8 x 12000 series and 4 x 9700's.  Perhaps I can assist.

Configuring a FIPS appliance involves configuring the HSM immediately after completing the generic configuration process. You then create or import a FIPS key. After creating a FIPS key, you should export it for backup AND you need this FIPS key to configure HA.  Hence, first must complete steps below to initialize the HSM, which first requires basic configuration, then initialize the HSM, then generate FIPS key, then export key, then configure HA.

1. Complete the initial hardware configuration first, using:
https://docs.citrix.com/en-us/netscaler-hardware-platforms/mpx/netscaler-initial-configuration.html

2. Before initializing the HSM, you can upgrade to the latest build of the software on the Netscaler. (Optional)
Reference: https://docs.citrix.com/en-us/netscaler/12-1/upgrade-downgrade-netscaler-appliance.html

3. Change the SO password (see below)
Configuring HSM erases anything beyond the basic configuration.  I would change the SO password during this time as well, 14 character max, no symbols.  Why?  Because if you don't it will retain the password set at time of HSM initialize.  In other words, I would not want the initial password bound to the HSM.  Changing it after HSM initialization won't matter if you reinitialize later on.

4. Initialize HSM module using CLI (I prefer CLI but can use GUI)
A. Logon to appliance as super user (assumes initial configuration completed)

Type the following commands to configure the HSM and verify the configuration:

show ssl fips
reset ssl fips
reboot

** Post Reboot
set ssl fips -initHSM Level-2 \<newSOpassword\> \<oldSOpassword\> \<userPassword\> \[-hsmLabel \<string\>\]
save ns config

reboot

** Post 2nd Reboot
show ssl fips

5. Create FIPS Keys

Option 1 - Using GUI
You must specify the key type (RSA or ECDSA) and specify the curve for ECDSA keys.

Create a FIPS key by using the GUI
Navigate to Traffic Management > SSL > FIPS.
In the details pane, on the FIPS Keys tab, click Add.
In the Create FIPS Key dialog box, specify values for the following parameters:

FIPS Key Name*—fipsKeyName
Modulus*—modulus
Exponent*—exponent
*A required parameter

Click Create, and then click Close.
On the FIPS Keys tab, verify that the settings displayed for the FIPS key that you just created are correct.

Create a FIPS key by using the CLI (Option 2)
create ssl fipsKey \<fipsKeyName\> -modulus \<positive\_integer\> \[-exponent ( 3 | F4 )\]
show ssl fipsKey \[\<fipsKeyName\>\]

Example:
1. create fipskey Key-FIPS-1 -keytype RSA -modulus 2048 -exponent 3
2. show ssl fipsKey Key-FIPS-1
3. FIPS Key Name: Key-FIPS-1 Key Type: RSA Modulus: 2048   Public Exponent: F4 (Hex: 0x10001)

6. Export FIPS Key

Option 1: CLI
export ssl fipsKey \<fipsKeyName\> -key \<string\>

Example: export fipskey Key-FIPS-1 -key Key-FIPS-1.key

Option 2: GUI
Navigate to Traffic Management > SSL > FIPS
In the details pane, on the FIPS Keys tab, click Export.
In the Export FIPS key to a file dialog box, specify values for the following parameters:
FIPS Key Name*—fipsKeyName
File Name*—key (To put the file in a location other than the default, you can either specify the complete path or click the Browse button and navigate to a location.)
*A required parameter
Click Export, and then click Close.

7. Repeat steps 1-4 on HA partner
8. Import SSL FIPS Key

Option 1: CLI
import ssl fipsKey \<fipsKeyName\> -key \<string\> -inform SIM -exponent (F4 | 3)
show ssl fipskey \<fipsKeyName\>

Example:
import fipskey Key-FIPS-2 -key Key-FIPS-2.key -inform SIM -exponent F4
show ssl fipskey key-FIPS-2
FIPS Key Name: Key-FIPS-2 Modulus: 2048   Public Exponent: F4 (Hex value 0x10001)

Option 2: GUI
Import a FIPS key by using the GUI
Navigate to Traffic Management > SSL > FIPS
In the details pane, on the FIPS Keys tab, click Import.
In the Import as a FIPS Key dialog box, select FIPS key file and set values for the following parameters:
FIPS Key Name*
Key File Name*—To put the file in a location other than the default, you can either specify the complete path or click Browse and navigate to a location.
Exponent*
*A required parameter
Click Import, and then click Close.
On the FIPS Keys tab, verify that the settings displayed for the FIPS key that you just imported are correct.

8.  Now you can configure HA
https://docs.citrix.com/en-us/netscaler/12/ssl/fips/configure-fips-ha.html

Hope this helps.

Brian,

Thank you so much for the detailed steps, it is truly appreciated.  One more question, as far as the FIPS key itself, when you generate this you are actually binding a certificate to this module?  That's where I'm a little unsure if I understand what this is setting exactly.  We need to use a WildCard cert and with that, confirming this is where I'd add that as well as where I'd update when the certificate is renewed.

Thanks in advance.

Best use case for your wildcard certificate would be for "Content Switch" or "SSL VIP" on the Netscaler HA Pair post configuration of HSM.  The private key we generate on the HSM when creating the key is for that Netscaler device only but being you are implementing HA pair you can export that key and use on the other device.

Once you HA pair is configured you can then import the "wildcard" certificate to both appliances.  This is separate from the HSM configuration.

Keeping in mind that the HSM configuration and that certificate is specific to that appliance or the HA pair only and where only the Super User account would access this component.

Your wildcard certificate would be more specific to a valid registered domain name that you want others to use to connect over TLS\HTTPS.  

You get the most value out of the wildcard certificate using Content Switch where you can essentially write infinite rules to redirect traffic to virtual services defined down-level on the Netscaler.  

Or, you can bind it to SSL VIP that would also use services.

For the internal only services where you want to maintain encryption you can use private certificates from internal CA as long as it is Netscaler to Netscaler communication OR Netscaler to internal defined services like StoreFront, XML, etc... and where the public/private key reside on both appliances using the same import process stated prior.  Depending on how those certificates are generated you might need to convert but we can cross that bridge if we come to it.  

Best to use public certificates for your Content Switch or SSL VIP otherwise you have to provide all down-level clients with both the public/private key and import that to each CSR.  This is to use Load Balancing between services by placing an SSL/TLS VIP or content switch on the front end.  The content switch, for example, can front end thousands of websites using a single content switch with a single wildcard certificate.

Regarding the wildcard certificate and updating.... Your public/private key from a public provider or internal CA is imported to both Netscalers using CLI or GUI and that key pair is held in the SSL directory which you can see using WinSCP.  Using WinSCP, you copy the key pair to both Netscalers.  You then import the key pair.  Then you bind it to Content Switch, or SSL VIP.  
Additional reference on SSL: https://docs.citrix.com/en-us/netscaler/12/ssl/how-to-articles.html

Content Switch configuration: https://docs.citrix.com/en-us/netscaler/12/ssl/how-to-articles/config-secure-cs-vserver.html

The content switch does require advanced level configuration which we can work through if you wish.

The simpler configuration is SSL VIP where you simply want to manage traffic back to your Citrix StoreFront servers, for example, and does not require a wildcard but simply a public/private key pair bound to both Netscalers, copy the key pair over - might require conversion - followed by importing the key pair AND intermediate chain on each appliance, followed by binding that to your CS or SSL VIP.app
Reference this concept but traffic is encrypted: https://docs.citrix.com/en-us/netscaler/12/getting-started-with-netscaler/load-balancing.html

This process for the key pair, intermediate chain and linking appears to be adequately documented here:
https://docs.citrix.com/en-us/netscaler/12/ssl/ssl-certificates/add-group-certs.html

In addition, I recommend one of my articles that goes in more depth regarding SSL:
https://www.experts-exchange.com/articles/25021/Citrix-SSL-TLS-Vulnerabilities-and-Operating-System-Hardening.html

Let me know if this helps or you have any other questions.

Additional links of interest:(for later configs - not much we cannot achieve with Netscaler) Here are a few examples:
https://www.experts-exchange.com/articles/25020/Monitor-Internal-MSSQL-Database-with-Citrix-Netscaler-Advanced-Monitor.html
https://www.experts-exchange.com/articles/25019/Citrix-NetScaler-HTTP-Compression-Feature.html

Ok, thank you for the explanation.  I'm going to give your steps a try and see if I can get a successful FIPS/HA going.  We've been struggling with one set of devices in particular.  Looks from your steps, I was doing similar, but thinking I missed something.

Will comment here if I need further help with the WildCard.  We do have one working HA/FIPS (though the individual that got this working couldn't confirm what steps he did).  I tried to update the Renewed WildCard cert, and it gave me errors indicating it was not a FIPS compliant cert.  Therefore why all my questions around that.

Thanks again for your help.

Brian,

One more question.  I see here after the primary is done, you move to the secondary for these steps:

7. Repeat steps 1-4 on HA partner
8. Import SSL FIPS Key

Option 1: CLI
import ssl fipsKey \<fipsKeyName\> -key \<string\> -inform SIM -exponent (F4 | 3)
show ssl fipskey \<fipsKeyName\>

Example:
import fipskey Key-FIPS-2 -key Key-FIPS-2.key -inform SIM -exponent F4
show ssl fipskey key-FIPS-2
FIPS Key Name: Key-FIPS-2 Modulus: 2048   Public Exponent: F4 (Hex value 0x10001)


My Question is are you renaming the FIPS key to include the key-FIPS-2, so that each HA has an individual identity or is this supposed to be the same name as you are importing the one from the Primary Device?

Thanks

The name is merely for you to distinguish this as the FIPS HSM certificate and you do not need to modify anything to import on other appliance.  Actually, you want to keep them identical due to the HA synchronization process.  If you rename it, you would then have two names in the /SSL directory where you only want the one.

Regarding the other ciphers question and compatibility you can also reference:
https://docs.citrix.com/en-us/netscaler/12-1/media/cipher-support-on-a-citrix-mpx-9700-fips-appliance-with-firmware-2-2.pdf

Hello.  I've tried following your detailed steps a few times now and end up with the same result.  I am able to generate the key and follow the steps, where it errors is the import on the secondary device.  I get the same error each time.

So, not sure what I'm missing here.  It also takes several minutes to do the reboot on the device after the set ssl fips -initHSM step.  

So for clarity, I've provided my steps (removed passwords)

show ssl fips
reset ssl fips
reboot


set ssl fips -initHSM Level-2 <New Password> so12345 <New Password>  -hsmLabel wild_fips_cw
save ns config
reboot


show ssl fips

From GUI - Traffic Management/SSL/FIPS
Keys - ADD

create fipskey wild_fips_cw -keytype RSA -modulus 2048 -exponent F4
show ssl fipsKey wild_fips_cw

Should display
FIPS Key Name: wild_fips_cw Key Type: RSA Modulus: 2048   Public Exponent: F4 (Hex: 0x10001)



CLI -

export fipskey wild_fips_cw -key wild_fips_cw.key

*Also do from GUI to ensure a good backup
Traffic Management/SSL/FIPS, select key export - *Ensure you use a different file path so you don't overwrite the keys

*Copy local to my device using WinSCP and then use WinSCP to copy to secondary device*

Go to secondary Node
show ssl fips
reset ssl fips
reboot


set ssl fips -initHSM Level-2 <New Password> so12345 <New Password> -hsmLabel wild_fips_cw
save ns config
reboot


CLI
import fipskey wild_fips_cw -key wild_fips_cw.key -inform SIM -exponent F4
(Says can’t find the key and then I ended up trying to import via the GUI) - That’s when I get the “internal error”

show ssl fipskey wild_fips_cw
FIPS Key Name: wild_fips_cw Modulus: 2048   Public Exponent: F4 (Hex value 0x10001)

Any additional thoughts are appreciated.

Thanks
FIPS_Internal_Import_error.png

I don't see a step for copying the FIPS Key to Netscaler using WINSCP? To the /SSL Directory.

I tried to copy using WinSCP post reboot and pre reboot.  So this would go here:

Go to secondary Node
show ssl fips
reset ssl fips
Tried here (Copy FIPS via WinSCP to secondary node)
reboot
Copy FIPS Key via WinSCP to secondary node (Tried here)

set ssl fips -initHSM Level-2 <New Password> so12345 <New Password> -hsmLabel wild_fips_cw
save ns config
reboot

Okay, my apologies for not being more specific when configuring both nodes.  I think the issue might be a time constraint that is part of FIPS requirement.  Let me rehash the steps here for you although you might need to begin from step 1, which is okay... must be done correct.

This is going to take me some time but I will add notes as well and post all steps momentarily. Plus, these steps will be useful for future questions regarding FIPS HA configuration.

Next post will be more thorough and streamlined as I looked back over my post and they can be optimized.  Apologies for confusion.  Will post momentarily.

I'm using CLI so I will add steps to compensate for not using GUI.  GUI automatically syncs the SSL files whereas CLI does not.

Awesome thank you.  Will give this a try today and cross my fingers :)  I appreciate your extra effort to document this further for me.

My pleasure and this will be a good future reference for others.

Is there to be a certificate created prior to this step?

init fipsSIMsource /nsconfig/ssl/source.cert

I'm not sure where to find the source.cert or if this generates it for me.

I've attached what is currently on the Primary Netscaler minus the wild_fips_cw that I generated or is that what you're referring to?  I only ask as there is no extension my files.  


Thanks
NetscalerSourceCertquestion.png

Nadda.  Make sure you don't use that certificate for now.  We are initializing the SIM to setup HA and the FIPS key will be generated further down.  This would make more sense with GUI but I prefer CLI.  Everything will be stored, when done, in the /nsconfig/ssl/ which you can see using WINSCP or using the appropriate OpenBSD commands logged in as nsroot.

You probably have not run the previous commands but just to be sure see Step 13.  Use WINSCP to check /nsconfig/ssl/ directory for any existing certificates and remove them prior to continuing to Step 14.

Once you complete all steps HA will be configured and your good to go.  I can post the GUI steps if you wish but CLI is better learning experience and keep in mind I don't have a FIPS Netscaler in front of me right now so I'm going off my notes in OneNote. Plus, as we are going through this I can update my initial post with the steps to add notes as needed to help alleviate future questions.

Reading the article, looks like the reference to source.cert is really just using this syntax.  This is creating the cert specific to the Netscaler.  So I should just leave as is:

init fipsSIMsource /nsconfig/ssl/source.cert


I guess I'm still not understanding completely though where the 'target' key and secret get generated from or where I define them.  Or is this really using an existing certificate?

Can you please clarify this area.

Thanks
Jennifer

Yes, I did it this way to keep things simple and standardized.  I recommend leaving those lines, where I provided full syntax, identical.

Where you do need to change some of the syntax would be where I have <>.  Example, <Primary_IP>, where you would take out the greater than and less than symbols and replace that section with IP Address of your NSIP relative to Primary and Secondary nodes.

Perfect, thanks.

Which step does the timer actually start?  Is it with the CLI: set ssl fips -initHSM Level-2 \<newSOpass\> \<oldSOpass\> \<userPass\> \[-hsmLabel \<string\>\]  step or after that?

The reboot time alone is taking almost 5 minutes.

Thanks
Jennifer

Ok, I have followed the steps twice, and get this when I try to import

ERROR: Input file(s) not present or not accessible in current partition


If I try via the GUI, I get an internal Error

If I look via WinSCP to verify the file exists, it is in nsconfig/ssl

So, guessing I'm timing out... but not sure as the command is like it doesn't see it.  Is the import looking in a different directory than the nsconfig/ssl?


Thanks
Jennifer

This happens when SIM is initialized on primary and not secondary.

For now, disable HA on the Netscaler and repeat these steps: (We need to initialize SIM on both primary and secondary)

Disable HA sync and propagation on both the nodes.
Run the following command on the source appliance:
init fipsSIMsource /nsconfig/ssl/source.cert
Run the following command on the target appliance:
init fipsSIMtarget /nsconfig/ssl/source.cert /nsconfig/ssl/target.key /nsconfig/ssl/target.secret
Run the following command on the source appliance:
enable fipsSIMsource /nsconfig/ssl/target.secret /nsconfig/ssl/source.secret
Run the following command on the target appliance:
enable fipsSIMtarget /nsconfig/ssl/target.key /nsconfig/ssl/source.secret

Ok, I'll go through and just follow these steps when I get to this section as if I try now, says the file already exists.  Thanks for sticking with me, feel like we're close, but I'm definitely not understanding or missing something.

Give me some time to research..... Been stuck in meeting.

Nevermind my last comment, I deleted it.  I am doing all and getting the right result except the SSL Key Step on the Secondary.  I'll try one more time in the event I'm doing this too slow.

Thanks for your help - no worries on the meeting, I appreciate your help.

Oh, okay.  Good deal.  Let me know.

Just a bit of info - Did a true timer on the reboot itself, that alone is taking 6 minutes 30 seconds for a response.  I have been using Putty, but maybe I'll work over a console connection to see if this reduces the time.

Same issue :(  All works until the import.

Hm.  Well, the 6 minute timer has to do with the SIM key transfer.  Are you still getting the error posted previously?  

We can translate these steps to the GUI.  You would simply type in the same certificate names I provided earlier but in the GUI.  This replicates the files automatically.

It will take me some time to put together those new steps, however.  

After the last reboot, we are only doing the initialize SIM steps?  

Otherwise, it should force you at 6 minutes... has this happened?  Those last few steps?  Or did you still get that "not found" error?

Okay.  Can you just double check on the Primary first (using WINSCP for example) and look in the /nsconfig/ssl directory and check for any "sub-directories".  It might read like "sl" or "ls"... don't remember.

Just make sure there are no subs and if there are, check for any files mentioned above...particularly the .secret.

And, if not on Primary, check the secondary and see if somehow the process moved it to a sub-directory.

We might need to just move on to GUI being it is more automated and less steps.

Thanks -

From WinSCP - I have been clearing out the source.key, target.key, etc.... after this fails.

After last reboot, Steps performed are.

Disable HA sync and propagation on both the nodes.
Run the following command on the source appliance:
init fipsSIMsource /nsconfig/ssl/source.cert
Run the following command on the target appliance:
init fipsSIMtarget /nsconfig/ssl/source.cert /nsconfig/ssl/target.key /nsconfig/ssl/target.secret
Run the following command on the source appliance:
enable fipsSIMsource /nsconfig/ssl/target.secret /nsconfig/ssl/source.secret
Run the following command on the target appliance:
enable fipsSIMtarget /nsconfig/ssl/target.key /nsconfig/ssl/source.secret


This is successful

Then I was moving onto:

Step 21: Create FIPS Key Primary Node
Primary> create ssl fipskey vue01CWHSM -modulus 2048 -exponent F4
Primary> show ssl fipsKey vue01CWHSM

Step 22: Export the FIPS Key
Primary> export fipskey vue01CWHSM -key vue01CWHSM.key

*Copy the vue01CWHSM.key via WinSCP to Target (Secondary)

Step 24: Import SSL Key on Secondary
Secondary> import ssl fipskey vue01CWHSM -key vue01CWHSM2.key -inform SIM -exponent F4
Secondary> show ssl fipsKey vue01CWHSM

Step #24 is what is failing.  So perhaps, only Steps #21 - 24 need to happen via the GUI.

Sounds good, appreciate it... We'll give this a shot one more time today then taking a break from it for the night ;)  Hopefully that will be the magic trick.  I'll reply either way.

Okay.  We will get there it just takes a little longer being I cannot be sitting right there in front of the appliances with you.  And, we are close to having this figured out.

Oh my Gosh! So Excited!!!!!  Just doing that last step via the GUI worked.  I was able to import successfully and then added my second node under the HA so they are actually both Showing online.

So, will table for today as we finally had success - a HUGE Thankyou!!  Wish I could give you a gift certificate or something :)

So, next question is back to the WildCard certificate and how that works with the FIPS - to apply, renew, etc.  But am willing to wait until tomorrow and open a new thread on that so this one can help someone else.
FIPS.png

I have to say Brian was extremely helpful.  I was doing portions of this correct, but was missing some and he persisted with me until we got this working.  I am extremely grateful, been hitting my head against this wall for awhile, so nice to advance so I can continue with the other portions of the configuration.

Awesome!  Again, my pleasure.  Here to help.

One the wildcard certificate, IMO, it would be better to post another question so that we can keep that resolution separate from this thread.

I figured we were close on the SIM/HSM/HA configuration.  

Good to know we are one big step forward toward your final config.