Self-signed TLS certificate errors, impact on Postfix server and POP3 clients

Posted on 2012-04-12
Last Modified: 2012-04-20
I have a Linux mail server set up IAW I recently updated the TLS certificates on the server in an effort to renew my understanding of how they work, with the eventual goal of enabling SSL in my Evolution client SMTP server connections (I have SSL enabled to receive pop3 mail but not to send mail).

I generated my self-signed certificates using openssl directly since I'm having trouble getting to work properly on my server. These are the steps I used:

1. openssl genrsa -des3 -rand file1:file2.etc -out ../ca/server_ca1.crt 1024 #create CA
2. openssl rsa -in ca/server_ca1.crt -out ca/server_ca.crt #clear passphrase from CA
3. openssl rsa -in ca/server_ca.crt -out private/server_key.key #create private key from CA
4. openssl req -new -key /etc/ssl/private/server_key.key -out /etc/ssl/ca/ca_csr.csr #create CSR
5. openssl x509 -req -days 365 -in ca/ca_csr.csr -signkey private/server_key.key -out newcerts/server_crt.crt #create server certificate by signing private key
6. Change files to .pem format (cat filename.crt, filename.key > filename.pem), chmod 600 private/server_key.pem, copy files to /etc/postfix/ssl and update /etc/postfix/ settings:
  smtpd_tls_cert_file = /path/to/cert
  smtpd_tls_key_file = /path/to/key
  smtpd_tls_CAfile = /path/to/CA ...failed
So I disabled that entry and restarted postfix. Mail works as before, so my question is, what if anything is my server not doing that it was doing before the smtpd_tls_CAfile was disabled?

I may have a problem with the CA because:
$ openssl x509 -noout -text -in server_ca.crt
>unable to load certificate
4395:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:650:Expecting: TRUSTED CERTIFICATE
I'm confused about step 1 because if there's an error with the CA, the rest of the process (generating the private key and the self-signed certificate) should fail but they haven't: the key and cert files have a matching modulus and exponent. I've also read that for self-signed certificates, the CA file is not required, so I can disable this entry in Postfix. What do you recommend?

I've been doing a lot of reading about X509 certificate attributes etc and found that the server's common name or FQDN is the most important information that should appear in the certificate. I carefully checked /etc/ssl/openssl.cnf, /etc/apache2/mydomain.cnf, /etc/courier/pop3d.cnf and /etc/postfix/ to ensure that '' appears correctly and my configuration looks correct, but the CN and the Location (L) entries are missing from the certificate output:
$ openssl x509 -noout -text -in server_crt.pem
        Version: 1 (0x0)
        Serial Number:
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=AK, O=Mydomain
            Not Before: Apr 11 06:24:42 2012 GMT
            Not After : Apr 11 06:24:42 2013 GMT
        Subject: C=US, ST=AK, O=MyDomain
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (1024 bit) etc...

The same happens with a .crt extension. Here's my config:
[ req ]
prompt                  = no
distinguished_name      = server_distinguished_name

[ server_distinguished_name ]
commonName              =
stateOrProvinceName     = AK
countryName             = US
emailAddress            =
organizationName        = Mydomain
organizationalUnitName  = IT  

How to fix this?
Question by:sara_bellum
  • 17
  • 10
LVL 17

Expert Comment

by:Kent Dyer
ID: 37840888
Do you have to use openssl?  You can use keytool from the JDK if you want to attach self-signed cert to say Apache..


LVL 77

Expert Comment

ID: 37840896
Not sure you properly created the CA self-signed cert.

Have not looked at it a while, but there should be a which gets the data and the calls the OpenSSL commands, look in the to see the sequence of commands. That it runs after collecting the info.

Author Comment

ID: 37845320
Thanks for the links, is certainly well-researched and presented. There appear to be about 6000 documented ways of doing this, which is unsettling.

I created a demo subdirectory to /etc/ssl and checked openssl.cnf against the advice on the page. There are two directories, certs/ and newcerts/, and the reason for having both of them is not clear: I found that openssl defaults to the certs/ path when it tries to sign a certificate, so I'm not using the newcerts directory. That's a small matter though. The more serious problem is that the generation of the initial key and cert files fails to produce matching moduli.
# openssl req -config openssl.cnf -new -x509 -extensions v3_ca -keyout path/to/private/cakey.key -out path/to/certs/cacert.crt -days 365

# openssl rsa -noout -text -in path/to/private/cakey.key and
# openssl x509 -in path/to/certs/cacert.crt -noout -text
show a modulus mismatch, which appears when I try signing the server certificate:

# openssl ca -config openssl.cnf -policy policy_anything -out path/to/certs/cacert.crt -infiles  path/to/server.csr
...Using configuration from openssl.cnf
CA certificate and CA private key do not match
6404:error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch:x509_cmp.c:406:

I reviewed openssl.cnf and can find nothing strange that I myself modified, since most of the file content is out-of-the-box standard distribution. I tried using v3_req instead of v3_ca to see what would happen and the result was worse than the mismatch (I believe openssl was unable to load the private key).

I attach my openssl config, in case you see something. Let me know thanks.
I had tried using earlier and also found errors, and it was difficult to see whether those errors stemmed from my configuration of or from openssl itself.  So I'm looking for the most basic possible solution to this problem to avoid opening up the possibility for more errors to occur.
Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.


Author Comment

ID: 37845323
For some reason my file attachment didn't make it. Trying again...
LVL 77

Expert Comment

ID: 37845349
I think you first need to create a private key to be used for signing.
You conf file does not point to the /etc/ssl or where
I'll have to test setup one to double check what migh be missing.

-key /etc/ssl/keys/ca.key

Note your conf defines the default ca dir as in ./demoCA rather than where you stored the created cert privatekey.

Author Comment

ID: 37847803
I found some more guidance at which I also tried to implement, although not in my home directory.

Like others before this, the guidance just doesn't work.  When I try to sign the certificate, I get this error:

Enter pass phrase for ./demoCA/private/cakey.pem:
unable to load CA private key
8835:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:330:
8835:error:0906A065:PEM routines:PEM_do_header:bad decrypt:pem_lib.c:428:

Although this time the moduli and public exponent on the cert and the key do match, which is progress. I don't understand why the directions call for switching back and forth between the caconfig.cnf copy of openssl.cnf and mydomain.cnf, but I do have both config files in /etc/ssl/demoCA now.  I don't see the point of changing my CA directory path, it should work here...

Then again, there's a lot about this process that I don't understand. Why radically different openssl procedures are recommended for self-signed certificates on different Linux platforms would be a good place to start.
LVL 77

Expert Comment

ID: 37848377
The path for dir within he conf file must be explicit and not relative to the location from which the command is being run.

Have not had a chance to setup ca with OpenSSL. Will post probably tomorrow with the complete step by step.

Author Comment

ID: 37849373
This time I ran the Ubuntu instructions and they work - if you try following a proven procedure often enough, it's possible that it will actually work.

But trying often enough and in enough ways is not why I wrote EE - the reason I'm writing is to understand the process. The steps by themselves - on any given web site - explain only partially how this works.

The Ubuntu steps and the steps you referred me to accomplish the same purpose - if I understood how this process is supposed to work I'd save myself a lot of useless trial and error.

Author Comment

ID: 37849387
My initial goal was to enable SSL to send mail from my Evolution client. I imported the  PKCS#12 certificate without error, but still cannot send mail when SSL is enabled. Evolution tells me that it cannot connect to connection refused.

So now I have to slug my way into finding out why the connection is refused - there are seven types of encryption available to choose from, all of which appear to be supported. I tried a couple that fail, and expect the rest to fail as well. This promises to be not fun, because the mail log and syslog on the mail server show nothing.
LVL 77

Accepted Solution

arnold earned 500 total points
ID: 37849398
Does your mail server configured to allow port 25 connectins or is configured with ssl on other ports 465, 587 or for tls and modified master.conf to enable secure smtp?  for pop are you using dovcot for pop and added The sever certificate?  You would need to add the self signed ca certificate a trusted to the system on which you email client is on.

The data for the mail log should be in /var/log/maillog.

You did a Generate a new request CSR for and then used your CA private key to sign it to get the cert
Then did you then use the signed cert and the private key used in the creation of the CSR into the configuration of the postfix server and likely including a reference to the self sign CA cert?

Author Comment

ID: 37853371
First, I noticed that I had forgotten to check the file path on the smtpd_tls_CAfile in the postfix configuration. I fixed that, restarted postfix and my Evolution client prompted me to accept the certificate on sending mail with TLS enabled. So now this feature works.

You ask good questions: I ran telnet localhost tests which show that 25 and 110 are open as expected. I get an OK prompt on port 995 connections but there's no output, I can only exit to quit (is that expected behavior?).  

My Evolution client now accepts TLS encryption with Password authentication on Receive and PLAIN and Login authentication on Send. Is this a recommended configuration?

I can find nothing in the postfix config that tells me which ports it connects to or even which pop3 applications it uses. I had both courier and dovecot running on my server which is redundant: I stopped dovecot and mail works fine, so I'll keep things that way.

That's all for now, will check in later.
LVL 77

Assisted Solution

arnold earned 500 total points
ID: 37853566
To check ssl Port connectivity you can use openssl To test ssl connections as well as tls functionality
openssl s_client -connect And with -starttls

lsof -p processID you can see which ports are used by the process referencd by processID.

995 should be the ssl pop port and should not have an ok prompt unless you se the openssl s_client example above to establish the sl connection.
Pop interaction is
user username
pass password
If hey are good you will get a response on how many messages there are in the mailbox
retr 1
Will retrieve the first message, list will list the messages and their sizes.

Author Comment

ID: 37858346
Thanks for the qmail link, I can authenticate with
openssl s_client -starttls smtp -crlf -connect and send mail to myself at the command line, so life looks good at the moment : )

I'm a little confused by my lsof output:
% lsof -Pnl +M -i4
master  15419        0   12u  IPv4  72717      0t0  TCP *:25 (LISTEN)

lsof lists only the above entry for port 25 and none for ports 465, 993 or 995, presumably because I haven't made any changes to the default /etc/postfix/ configuration as you recommend above (this is in contrast to apache2 which lists 15 alternating entries for ports 80 and 443!)

As I understand it, to finish this project I'll need to research how to make the changes needed to (I see no port numbers in this file so there's no sample format to use). Again, your advice is welcome!
LVL 77

Assisted Solution

arnold earned 500 total points
ID: 37859044
To be clear startssl is different from smtps in the following manner. starttsl secures the data after the connection is made. the initial exchange of information is in plain text.
i.e. a mail client connects to the server and says "ehlo <name>"
The reserver responds accordingly and lists the available/supported extentions.  If STARTTSL is one of them, and if the mail client is configured to try, it will issue a starttls
at which point the server and the client will start the negotiation on the means by which the data in this connection will be encrypted......
SMTPS is a secure connection. Where the exchange of public keys/certificate is initiated right away to establish the connection.

When you enable smtps by uncommenting the line, smtps is the TCP/465 grep smtps /etc/services. i.e. /etc/services is what translates/converts the name to the port and port type
smtp is 25/tcp
pop3 is 110/tcp
while pop3s will be converted to 995/tcp
Both pop reference UDP, but have not seen it.

Author Comment

ID: 37863214
I'm almost there, thanks again!!  I enabled smtps in and after reloading postfix, port 465 shows up at % lsof -Pnl +M -i4 :
master  15419        0   12u  IPv4  72717      0t0  TCP *:25 (LISTEN)
master  15419        0  108u  IPv4  95149      0t0  TCP *:465 (LISTEN)
So the smtp/postfix pids are fine. But the pop3/courier pids (ports 110 and 995) do not appear in the above list, although there are active pids on those ports:
% ps -ef |grep courier
root     20026     1  0 12:07 ?   00:00:00 courierlogger -start -name=pop3d couriertcpd -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup -address=0 110 courierpop3login courierpop3d Maildir
root     20061     1  0 12:08 ?   00:00:00 courierlogger -start -name=pop3d-ssl couriertcpd -address=0 -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup 995 couriertls -server -tcpd courierpop3login courierpop3d Maildir
(paths truncated for brevity)

I did find pop3 by changing the arguments on lsof:
% lsof -Pnl |grep pop3
courierlo  1353   0 1023uW     REG       0,17     0    5688    /var/run/courier/
courierlo  3981   0 1023uW     REG       0,17     0    5704 /var/run/courier/

The timestamps at % ls -l /var/run/courier :
-rw-r--r-- 1 root   root     6 2012-04-18 12:07
-rw------- 1 root   root     0 2012-04-11 14:54
-rw-r--r-- 1 root   root     6 2012-04-18 12:08
-rw------- 1 root   root     0 2012-04-11 14:54

show the lock files are a week older than the pid files, which is good. I tried removing the lock files with lock-remove after closing my mail client but they remain. These may be remnants of my earlier attempts to configure SSL for my mail client - although which particular attempt would be very difficult to find, given that I had taken to removing crt and pem files after aborted attempts at signing the certs.

If the lock files were blocking those port numbers, I should not be able to receive mail at all, but this is not the case: I have received at least one email since I started writing this.
Are these lock files a problem and if so, what to do?

Author Comment

ID: 37863223
I meant to say ports, not pids, are missing from lsof output (sorry)!
LVL 77

Expert Comment

ID: 37863482
Presumably you only have courier imap, you need to look at its configuration.
ps -ef |grep courier
Use lsof -p <PID of courier> to see what resources it is bound to,
lsof -i:110

Author Comment

ID: 37863569
courier pop3d and pop3d-ssl are installed, and now for reasons unknown (but I'm not complaining!) the ports show up:

% lsof -i:110
couriertc 20308 root    3u  IPv6  97325      0t0  TCP *:pop3 (LISTEN)
% lsof -i:995
couriertc 20328 root    3u  IPv6  97341      0t0  TCP *:pop3s (LISTEN)

The pop3 pids I got on % ps -ef |grep courier (posted above) point to lock files (see Older courier pop3 pids, below). As I write this, the new pids have appeared and their nomenclature/args are quite different from the earlier ones (see New courier pop3 pids, below).

Just let me know whether I should do something about the lock files and I'll close this out, thanks.

New courier pop3 pids:
root     20308 20307  0 13:44 ?        00:00:00 /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup -address=0 110 /usr/lib/courier/courier/courierpop3login /usr/lib/courier/courier/courierpop3d Maildir
root     20328 20327  0 13:44 ?        00:00:00 /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup 995 /usr/bin/couriertls -server -tcpd /usr/lib/courier/courier/courierpop3login /usr/lib/courier/courier/courierpop3d Maildir

Older courier pop3 pids:
lsof -p 20307
courierlo 20307 root  cwd    DIR      251,0     4096       2 /
courierlo 20307 root  rtd    DIR      251,0     4096       2 /
courierlo 20307 root  txt    REG      251,0    14076  117792 /usr/sbin/courierlogger
courierlo 20307 root  mem    REG      251,0   113964 2080925 /lib/
courierlo 20307 root  mem    REG      251,0    38360 2080798 /lib/tls/i686/cmov/
courierlo 20307 root  mem    REG      251,0  1405508 2080817 /lib/tls/i686/cmov/
courierlo 20307 root    0r  FIFO        0,8      0t0   97320 pipe
courierlo 20307 root    1u   CHR        1,3      0t0     797 /dev/null
courierlo 20307 root    2u   CHR        1,3      0t0     797 /dev/null
courierlo 20307 root    3u  unix 0xdf935a00      0t0   97322 socket
courierlo 20307 root 1023uW  REG       0,17        0    5688 /var/run/courier/

root@orion:~# lsof -p 20327
courierlo 20327 root  cwd    DIR      251,0     4096       2 /
courierlo 20327 root  rtd    DIR      251,0     4096       2 /
courierlo 20327 root  txt    REG      251,0    14076  117792 /usr/sbin/courierlogger
courierlo 20327 root  mem    REG      251,0   113964 2080925 /lib/
courierlo 20327 root  mem    REG      251,0    38360 2080798 /lib/tls/i686/cmov/
courierlo 20327 root  mem    REG      251,0  1405508 2080817 /lib/tls/i686/cmov/
courierlo 20327 root    0r  FIFO        0,8      0t0   97336 pipe
courierlo 20327 root    1u   CHR        1,3      0t0     797 /dev/null
courierlo 20327 root    2u   CHR        1,3      0t0     797 /dev/null
courierlo 20327 root    3u  unix 0xd62cfc00      0t0   97338 socket
courierlo 20327 root 1023uW  REG       0,17        0    5704 /var/run/courier/

Author Comment

ID: 37863582
The pid numbers keep changing - in my earlier post they were 20026 and 20061. Those are gone now.
LVL 77

Expert Comment

ID: 37863708
Please lookup the parent process in the example above 20307 and 20327. If it is not one, lookup the parent again until it is init (1)  trying to see whether It is being run out of inetd/xinetd.conf
This means when a new connection on port 110 or 995 is seen by inetd, xinetd it will start/spawnthe courie process.

Author Comment

ID: 37864302
I found a link that helped at :
I changed the ubuntu default setting for /etc/postfix/sasl/smtpd.conf from:
pwcheck_method: authdaemond
authdaemond_path: /var/run/courier/authdaemon/socket
and restarted postfix.

Now there are two new pids 20307 and 20327, each with a parent pid of 1 (copied below).
Not copied are the remaining two pids with a parent process of  20307 and 20327 (pointing to the lock files). But this is progress!

I sent myself a test email and found no errors in mail.log, also good...will check the courier  pids in the morning and see what remains.

root     20307     1  0 13:44 ?        00:00:00 /usr/sbin/courierlogger -pid=/var/run/courier/ -start -name=pop3d /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup -address=0 110 /usr/lib/courier/courier/courierpop3login /usr/lib/courier/courier/courierpop3d Maildir

root     20327     1  0 13:44 ?        00:00:00 /usr/sbin/courierlogger -pid=/var/run/courier/ -start -name=pop3d-ssl /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=4 -nodnslookup -noidentlookup 995 /usr/bin/couriertls -server -tcpd /usr/lib/courier/courier/courierpop3login /usr/lib/courier/courier/courierpop3d Maildir

Author Comment

ID: 37868616
Sorry about that, I know next to nothing about pids as you can see. Since authorization was not the problem I reverted to the postfix default smtpd config for sasl.

I then installed the openbsd-inetd package on ubuntu and took a stab at writing an inetd.conf file - the man pages don't provide sample formats, so I tried following instructions on how to fill one out.

So far this has not worked: when I start inetd at the command line, I launch no courier pids,  even when there are no syslog errors. To keep mail running I started courier pop3 services at /etc/init.d manually, this time without imap which I'm not using. Of course the courier pid situation hasn't changed - the logger pid has a parent pid of 1 per my earlier posting, but the tcpd parent pids on both pop3 ports still point to lockfiles.

I'm scratching my head on this one. I was hoping there might be an easy fix here but it could take me a while to figure this out...

Author Comment

ID: 37868646
To make sure that I understand the problem, what the pids are telling us is that pop3d won't start at boot time? Or is something else also broken?

Since I don't have an inetd package installed by default, I looked for the init script:
% cat /etc/courier/pop3d |grep 'POP3DSTART=YES'

So maybe it will start on boot: I'll reboot the server without an inetd.conf file and see what happens.
LVL 77

Expert Comment

ID: 37868662
I'm not sure what we are working on now.
Look at you inetd.conf, xinetd.conf or xinetd.conf.d/

Inetd/xinetd is what binds to the 110, 995 ports with the courier instance with the appropriate options.

Author Comment

ID: 37868697
Ubuntu doesn't use xinetd. On reboot, all courier pids start as before.

I'm beginning to think that even though the pids look strange (only the courierlogger pids have a parent pid of 1 and the rest of the parent pids point to lock files) maybe this is normal for Debian Ubuntu. If all courier processes had a parent pid of 1, my OS could regard this as a duplication, given that the courierlogger is apparently controlling all courier pids.

Let me know if you think that something with this postfix/courier/evolution configuration is about to blow up. Otherwise we could be ready to close this out.
LVL 77

Assisted Solution

arnold earned 500 total points
ID: 37869009
Oh, ok.

I see instead of using inet/xinetd. It is using a logger that than starts Couriertcpd and that in turn starts the pop3d respective daemon.

The chain in the way it appears gets the errors, outputs to the logger.
Are you not including the IMAP option of courier?

It is good to go.

Author Comment

ID: 37869211
Thanks! I'll take a look at these and close this out tomorrow.

Author Closing Comment

ID: 37874303
It was a circuitous route to the solution but I learned a lot, and it works!

Featured Post

Ransomware: The New Cyber Threat & How to Stop It

This infographic explains ransomware, type of malware that blocks access to your files or your systems and holds them hostage until a ransom is paid. It also examines the different types of ransomware and explains what you can do to thwart this sinister online threat.  

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Wired Network vs Wireless 12 66
Windows 10 - Devices and Printers slow loading 29 170
Setting up static routes to  sonicwll 4 75
Samba Question 11 78
Even if you have implemented a Mobile Device Management solution company wide, it is a good idea to make sure you are taking into account all of the major risks to your electronic protected health information (ePHI).
Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
Viewers will learn how to connect to a wireless network using the network security key. They will also learn how to access the IP address and DNS server for connections that must be done manually. After setting up a router, find the network security…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor ( If you're interested in additional methods for monitoring bandwidt…

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question