How to fix Netstorage after recreating expired certificates (Netware 6.5.1 Apache/Tomcat)

Thank you. Before you posted your answer, I ran tckeygen, and it made a slight improvement--iManager is now working on port 2200. Netstorage is still not working.

Please clarify:

1) You say "Make sure the LDAP objects are pointing to the right certificates." What does that mean? If you're referring to the two LDAP objects in Console One, "LDAP group" points to "LDAP server" and "LDAP server" points to the new "SSL CertificateDNS." I would think that's correct, yes? When I look at Apache from iManager, it shows the new certs under its listening ports. (I can't log in to Tomcat admin or manager; either as "admin" or ".admin.context.")

2. How does one export the root cert.der public key and import it into Java, IIRC? When I ran tckeygen, I ran it at the console with no arguments, and I hadn't done anything previously--do I need to do an export first? If so, how?

BTW, when I ran tckeygen, it said it worked, and iManager came up. Since NetStorage still wasn't working, I deleted sys/adminsrv/conf/.keystore, and ran tckeygen again. Same result: everything working except NetStorage.

And just to be really clear, after running tckeygen, I restarted apache, tomcat, java, and nldap thus: adrmsrvdn, ap2webdn, tc4stop, tcadmdn, java -exit, unload nldap, nldap, admsrvup, ap2webup, tc4admup, tomcat4. I'm hoping that's enough and we don't need to reboot the server.

(BTW, they're on SP1 because they're cheap and they don't like downtime. Yes, I think being so far behind contributed to this problem because it seems that new certs are created with the application of a service pack. Some other clients have recent certs and we didn't create them. )

Thanks.

Sorry, I was thinking of importing a trusted cert (like fom Verisign).

Sounds like it's not an LDAP thing... and you're a step closer.

Are you getting any error messages from NetStorage?

I'm not sure what you mean by error message from NetStorage; is there a NetStorage log? We're getting the same "500" messages in the browser when we try to open NetStorage.

I’ve looked through the Apache and Tomcat logs. It looks to me like there are no errors in:
- tomcat’s localhost_log
- tomcat’s admin_localhost_log
- apache’s adminsrv logs

Tomcat’s catalina_log reports the following, however it's a few days old:

JNDIRealm[Standalone]: Exception performing authentication
javax.naming.CommunicationException: localhost:636 [Root exception is java.net.ConnectException: Connection refused]

Also, the apache2 logs report the following (I’ve removed the date/time for readability, and obscured the external IP addresses and domain name for privacy). My guess is that these errors are a symptom of the above error and aren't a problem in and of themselves.

[date/time] [notice] LDAP: Built with Novell LDAP SDK
[date/time] [notice] LDAP: SSL support unavailable
[date/time] [notice] LDAP: Built with Novell LDAP SDK
[date/time] [notice] LDAP: SSL support unavailable
[date/time] [notice] Apache/2.0.48 (NETWARE) mod_jk/1.2.5 configured -- resuming normal operations
[date/time] [error] [client 74.1.2.3] File does not exist: SYS:/apache2/htdocs/favicon.ico
[date/time] [error] [client 74.1.2.3] File does not exist: SYS:/apache2/htdocs/images/head_bg.gif, referer: http://www.company.com/brand.html
[date/time] [crit] [client 74.1.2.3] configuration error:  couldn't check user.  No user file?: /oneNet/NetStorage, referer: http://www.company.com/NetStorage/contentframe.html

That looks really familiar.

You said you were getting errors from NoRM saying the cert didn't match the URL.  This could be part of the same issue.

There are products that have a problem with mismatched certs, and for all I know, the LDAP auth to NetStorage is one.

Check the cert that the LDAP server object refers to, to see if the server cert is called "certificateDNS" but the "subject name" entry in the Public Key Certifiecate dialog on the Certificates tab in C1 doesn't match the DNS name of the server, or has the IP address instead, or vice-versa - the certificateIP has something other than the IP address in the "subject name."

You may need to, once again, recreate one or both certs, to straighten that out.  If that's not the issue, we need to look further at secure LDAP and how NetStorage leverages it - there may be reconfiguration things you'll need to address, within NetStorage's authentication processes.

I don't think the certs not matching is the problem, because they've never matched and everything worked before. However, I would very much like to know how to fix it.

The server was installed a long time ago--originally as NSBS 6.0--before Novell had instructions about how to put servers behind a firewall with NAT. Thus the certs say 192.168.0.1, which is required for things to work internally, and the complaint message is "192.168.0.1 doesn't match www.company.com" which everyone is used to seeing. Is there some way of creating multiple certificates, or multiple names in a certificate, so that it has both 192.etc and www.etc.?

Do you know the differnce between tckeygen and tcedirint? Has tcedirint been replaced by tckeygen, or do I need both? (The TIDs are very confusing on these tools.)

I did a "dstrace -all", took down apache and tomcat, did a "dstrace +ldap +pki +ldps +pkia", brought up apache and tomcat, and tried to load NetStorage in a browser window, and here are some excerpts from the dstrace.log file. I've added hyphens at the start of the lines for clarity. I can send you the complete file if you want.
- PKI_GetCertificates: Reading certs from object CN=SSL CertificateIP - <server_name>.O=<firm_name>
then, some lines later:
- PKI_GetCertificates: Success!
then, many lines later:
- DoTLSHandshake on connection 0x<etc>
- Completed TLS handshake on connection 0x<etc>
- DoBind on connection 0x<etc>
- Failed to resolve full context on connection 0x<etc>, err= no such entry (-601)
- Failed to authenticate full context on connection 0x<etc>, err=no such entry (-601)

There are a long series of -601 errors, some of which say "failed to resolve full context."

Before I had recreated the certs and run tckeygen, I tried tcedirint, and found the TIDs and readme.txt extremely confusing, in other words, I tried many different sets of command line arguments, most if not all were certainly wrong. This is a NSBS server and it's very simple (there are only six users in the firm). There's one box, and it's running everything. The hierarchy is "My World [root] | NDS [tree] | <tree_name> | <firm_name>". *Everything* is inside the <firm_name>: all users including admin, the server object, the LDAP server and group objects, and the certs. The only containers below this level are those that were installed by default. If I need to re-run tcedirint, would you be able to tell me the exact syntax, for example:
- tcedirint ou=<firm_name>,o=<tree_name> cn=admin,ou=<firm_name>,o=<tree_name> ou=<firm_name>,o=<tree_name>
Sorry, but I find the "CN"s and "O"s and "OUs" every confusing, especially when some docs say that CN means country, and others say to include "CN=admin" as one of the arguments.

The "server.xml" file says "cn={0},ou=<firm_name>,o=<tree_name>" if that helps.

I hope this helps.

The error usually is one you see when you have problems with Tomcat/iManager login, which I thought was resolved, but maybe it's all connected.

I think it may not be a bad idea to run a PKIDIAG with option 4 to verify your new certs anyway.

Regarding typeful notation, LDAP vs eDirectory:

eDirectory                                                                   LDAP
CN = Common Name (the name of the leaf object.)        cn
C = Country                                                                c
T = tree                                                                     N/A
O = Organization                                                         o
OU = Organizational Unit                                              ou

The ones you'll most likely use with LDAP-based notation will be O, OU and CN, but they'd be in lower-case.  LDAP doesn't use tree name, AFAIK.

The tcedirint command should not use CN, just the context of the admin user.  In your case, it would be thus:

tcediring o=<firm name> o=<firm name>

Here's a brief TID on tcedirint.
http://www.novell.com/support/search.do?cmd=displayKC&docType=kc&externalId=InfoDocument-2967014&sliceId=&dialogID=9204431&stateId=0%200%209208410

Here's a TID on general iManager troubleshooting:
http://www.novell.com/support/search.do?cmd=displayKC&docType=kc&externalId=10090732&sliceId=&dialogID=9204431&stateId=0%200%209208410

TCEDIRINT.NCF apparently is for prior to 6.5SP1. Since your client is on SP1, you should be using TCKEYGEN.NCF

Here's a link to the TCKEYGEN.NCF TID:
http://www.novell.com/support/search.do?cmd=displayKC&docType=kc&externalId=InfoDocument-2969539&sliceId=&dialogID=9204499&stateId=0%200%209208673

Note that it specifically mentions NetStorage.

Re: the context line you posted from your server.xml - what section was that from?  If it's the JNDIRealm userPattern parameter, I believe it should read, for this client:
"cn={0},o=<firm_name>"

Unless you misstated the tree structure, which I understand to be:

tree = <tree name>
o = <firm name>

with all leaf objects being in the O container, with no OU containers.

Question - does the tree name match the name of the O container?  

1. Yes, the iManager login (port 2200) is working, and everything inside it works, except, not surprisingly, the Tomcat web-based admin.

2. PKIDIAG reports no errors. I've pasted the repair.log below, changing the actual names to <tree_name>, <firm_name>, and <server_name>.

3. The file I'm referring to is sys\tomcat\4\conf\server.xml. It was modified within the last few days, noy by me, but probably by something I was doing.
The exact contents are:
            <Realm className="org.apache.catalina.realm.JNDIRealm" connectionURL="ldap://localhost:636" debug="0" userPattern="cn={0},ou=<firm_name>,o=<tree_name>" protocol="ssl" authentication="simple" roleBase="ou=Tomcat-Roles,ou=<firm_name>,o=<tree_name>" roleName="cn" roleSearch="member={0}" />

4. You're correct. Under "My World" is "NDS" and then there's only one tree, <tree_name>, and under that only one organization <firm_name>. All objects including <server_name>, admin, the certs etc., are inside <firm_name>. The only OU containers are those that were created during the server build: Extend, NSBSAPPS, Printers, Tomcat-Roles, and WebAccess

5. The <tree_name> and <firm_name> and <server_name> are all different strings. (If the firm were Smith and Jones, they'd be "sj_tree", "smithjones",  and "sj_server".

6. Yes, I've read that TID many times. I hate it when they write things like "change 'mykey' to some other value".  By "some other value" do they mean a value that means something, like "SSL CertificateIP" or can it be any old character string like "frodo"?

7. From sys\etc\certserv\repair.log:

Current Time: Mon Aug  7 05:00:13 2006
User logged-in as: admin.<firm_name>.
Diagnostics only mode

--> Server Name = '<server_name>'
---------------------------------------------------------------------------

Step 1  Verifying the Server's link to the SAS Service Object.
   Server '<server_name>.<firm_name>' points to SAS Service object 'SAS Service - <server_name>.<firm_name>'
Step 1 succeeded.

Step 2  Verifying the SAS Service Object
   SAS Service object 'SAS Service - <server_name>.<firm_name>' is backlinked to server '<server_name>.<firm_name>'.
Step 2 succeeded.

Step 3  Verifying the links to the KMOs
   Reading the links for SAS Service object 'SAS Service - <server_name>.<firm_name>'.
--->KMO Old SSL CertificateIP - <server_name>.<firm_name> is linked.
--->KMO Old SSL CertificateDNS - <server_name>.<firm_name> is linked.
--->KMO NAASKMO - <server_name>.<firm_name> is linked.
--->KMO NetIdentity - <server_name>.<firm_name> is linked.
--->KMO IP AG 192\.168\.0\.1 - <server_name>.<firm_name> is linked.
--->KMO DNS AG 192\.168\.0\.1 - <server_name>.<firm_name> is linked.
--->KMO SSL CertificateIP - <server_name>.<firm_name> is linked.
--->KMO SSL CertificateDNS - <server_name>.<firm_name> is linked.
Step 3 succeeded.

Step 4  Verifying the KMOs
---> Testing KMO 'SSL CertificateDNS - <server_name>.<firm_name>'.
     Rights check -- OK.
     Back link -- OK.
     Private Key -- OK.

---> Testing KMO 'SSL CertificateIP - <server_name>.<firm_name>'.
     Rights check -- OK.
     Back link -- OK.
     Private Key -- OK.

---> Testing KMO 'DNS AG 192\.168\.0\.1 - <server_name>.<firm_name>'.
     Rights check -- OK.
     Back link -- OK.
     Private Key -- OK.

---> Testing KMO 'IP AG 192\.168\.0\.1 - <server_name>.<firm_name>'.
     Rights check -- OK.
     Back link -- OK.
     Private Key -- OK.

---> Testing KMO 'NetIdentity - <server_name>.<firm_name>'.
     Rights check -- OK.
     Back link -- OK.
     Private Key -- OK.

---> Testing KMO 'NAASKMO - <server_name>.<firm_name>'.
     Rights check -- OK.
     Back link -- OK.
     Private Key -- OK.

---> Testing KMO 'Old SSL CertificateDNS - <server_name>.<firm_name>'.
     Rights check -- OK.
     Back link -- OK.
     Private Key -- OK.

---> Testing KMO 'Old SSL CertificateIP - <server_name>.<firm_name>'.
     Rights check -- OK.
     Back link -- OK.
     Private Key -- OK.
Step 4 succeeded.

Step 5  Re-verifying the links to the KMOs
   Reading the links for SAS Service object 'SAS Service - <server_name>.<firm_name>'.
KMO 'Old SSL CertificateIP - <server_name>.<firm_name>' is linked.
KMO 'Old SSL CertificateDNS - <server_name>.<firm_name>' is linked.
KMO 'NAASKMO - <server_name>.<firm_name>' is linked.
KMO 'NetIdentity - <server_name>.<firm_name>' is linked.
KMO 'IP AG 192\.168\.0\.1 - <server_name>.<firm_name>' is linked.
KMO 'DNS AG 192\.168\.0\.1 - <server_name>.<firm_name>' is linked.
KMO 'SSL CertificateIP - <server_name>.<firm_name>' is linked.
KMO 'SSL CertificateDNS - <server_name>.<firm_name>' is linked.
Step 5 succeeded.

Step 6  Creating IP and DNS Certificates if necessary.
--> Number of Server IP addresses = 1
--> The default IP address is: 192.168.0.1
--> The KMO SSL CertificateIP's IP Address is: 192.168.0.1
----> The IP addresses match.
--> Number of Server DNS names for the IP address 192.168.0.1 = 1
--> The server's default DNS name is:
      192.168.0.1
--> The KMO SSL CertificateDNS's DNS name is: 192.168.0.1
----> The DNS names match.
Step 6 succeeded.


Note: Occasionally multiple problems will be solved with a single fix.

Fixable problems found:     0
Problems fixed:             0
Un-fixable problems found:  0

P.S. re #6, I found another TID, 10091972, that says "-keystorealias=<Servername>   Can be any name; recommended to use the LDAP server name", so I did the following:
1. moved sys/adminsrv/conf/.keystore into a "trash" folder.
2. edited sys/system/tckeygen.ncf to change "-keystorealias=mykey" to "-keystorealias=<server_name>"
3. ran "tckeygen" at the console
4. admsrvdn, ap2webdn, tcadmdn, tc4stop, java -exit, unload nldap, nldap, admsrvup, ap2webup, tcadmup, tomcat4.
No change. apache, gwwebaccess, and iManager are all up; NetStorage is down.

The setup in server.xml is wrong.  The tree name should not be present, and the organization name should not be designated an OU.  

That setup is one of the things TCEDIRINT.NCF does.  It creates the JNDIRealm section with the settings you give it.  

Since you had problems understanding the syntax for TCEDIRINT, you created that stuff with the wrong context, so you need to either edit the SERVER.XML and change the line as I indicated earlier, or rerun TCEDIRINT.

TCKEYGEN doesn't do that JNDIRealm thing, I don't think, so you'll have to rerun TCEDIRINT:

tcedirint o=<firm name> cn=<admin>,o=<firm name> o=<firm name>

That should fix the server.xml file so tomcat can actually log in via LDAP.

This looks extremely promising. I'll be able to try what you've suggested in about an hour; I'll let you know. Thanks.

I am so sorry to report that it didn't work. At the console, I typed "tcedirint <space> o=<firm_name> <space> cn=admin,o=<firm_name> <space> o=<firm_name>". I then took down and restarted apache, tomcat, and nldap. Same result. I then ran "tckeygen" with no arguments, and again restarted apache, tomcat, and nldap. Again, same result: everything is up except NetStorage.

Tcedirint *did* make the changes you wanted it to; sys\tomcat\4\conf\server.xml now contains the following:

<Realm className="org.apache.catalina.realm.JNDIRealm" connectionURL="ldap://localhost:636" debug="0" userPattern="cn={0},o=<firm_name>" protocol="ssl" authentication="simple" roleBase="ou=Tomcat-Roles,o=<firm_name>" roleName="cn" roleSearch="member={0}" />

I cleared the dstrace.log before starting nldap/apache/tomcat the last time, and this is what it's reporting:

DoBind on connection 0x8c5549a0
Bind name:cn=iFolder_ServerAgent,O=<firm_name>, version:3, authentication:simple
Failed to resolve full context on connection 0x8c5549a0, err = no such entry (-601)
Failed to authenticate full context on connection 0x8c5549a0, err = no such entry (-601)
Sending operation result 32:"":"NDS error: no such entry (-601)" to connection 0x8c5549a0
Implied anonymous bind by operation 0x2:0x63 on connection 0x8c5549a0
DoSearch on connection 0x8c5549a0
Search request:
      base: "cn=iFolder_Settings,O=<firm_name>"
      scope:0  dereference:0  sizelimit:0  timelimit:20  attrsonly:0
      filter: "(objectClass=iFolderSettings)"
      no attributes
Cannot resolve NDS name 'CN=iFolder_Settings.O=<firm_name>' in ResolveAndAuthNDSName, err = no such entry (-601)
Base "cn=iFolder_Settings,O=<firm_name>" not found, err = no such entry (-601)
Sending operation result 32:"o=<firm_name>":"NDS error: no such entry (-601)" to connection 0x8c5549a0
Monitor 0x341 found connection 0x8c5549a0 ending TLS session
DoUnbind on connection 0x8c5549a0
Preempting operation 0x0:0x0 on connection 0x8c5549a0 before processing because connection is closing
Connection 0x8c5549a0 closed

I find it strange that it's looking for an object called iFolder_Settings; there isn't one, but there isn't one at any of our other firms running NSBS and their NetStorage is working. I wonder if this error message is a bit of a red herring.

BTW, tcedirint didn't prompt me for the admin user/password; is it supposed to, or am I confusing it with pkidiag? Any more ideas? Thanks.

I thought it was supposed to.  Password is a 4th, optional parameter to tcdeirint.

Can you launch nsadmin?  Have you verified all the relevant stuff is correct - the search context, the DNS name, etc.?  Have you tried resetting the proxy user password? (note: you have to be in registry edit mode.)  How about the certificate - now that I think about it, NetStorage certs and password hashes and stuff are stored in the NetWare registry, so it could just be a matter of redoing the certificate assignment via nsadmin.

It turns out that nsadmin generates the same 500 error. (I might have said iManager was running: I meant that the admin functions at port 2200 are running--I thought that that was iManager. I just tried iManager itself--both it and nsadmin give me the same 500 message.)

I thought I'd check the netware registry to see if NetStorage is looking for the right certificates, and there is no "Xtier" key in My Server | Software | Novell. The only keys there are Syslog, Winsock 2, Polimgr, NMA, NLSLRup, and Polimgr650. Am I looking in the right place? I can't imagine that they were all deleted. If this is, indeed, the problem, is there a way of importing the necessary keys (like .reg files in Windows), or do we need to reinstall NetStorage?

I really appreciate all your help.

That did it. Thank you!