At first glance this seems to be a client side problem. Please elaborate how you connect to the server?

If so, you need to enable ssl after making the connection;

$ lftp -e 'set ftp:ssl-force true' yourserver

Select allOpen in new window

This might also help:
https://forum.filezilla-project.org/viewtopic.php?f=6&t=16463

For the config:
http://www.cyberciti.biz/tips/configure-vsfptd-secure-connections-via-ssl-tls.html

helge000: I know the articles from your URL. I have read them before and did not change my problem.
Yes i connect with FileZilla to server over internet. I only alowe SSL connections.

Does the lftp command show the same error?
To debug this, please try from your client computer:

telnet yourserver 21
openssl s_client -connect yourserver:21
gnutls yourserver 21

Select allOpen in new window

Is one or all of those working? Eg, you getting some useful output)

Also, it might help if you post part of your vsftpd config

# openssl s_client -connect localhost:21
CONNECTED(00000003)
140051988035400:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:699:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 112 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

Seems like I have problem with certificate? :S

No, all seems to be fine.
Did you try lftp?

lftp -e 'set ftp:ssl-force true' secureftp-test.com
lftp secureftp-test.com:~>

Select allOpen in new window

Do this with your server.
And: Are you sure you use FTPS and not SFTP?

helge000: I tried your command. There was no error.

Hmm whats the difference between FTPS and SFTP. I use implicit FTP over SSL option in Filezilla.

This is part of conf file for VSFTPD:
tcp_wrappers=YES
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
# Select which SSL ciphers vsftpd will allow for encrypted SSL connections (required by FileZilla)
ssl_ciphers=HIGH
rsa_cert_file=/etc/vsftpd/ssl/vsftpd.pem
pasv_min_port=50000
pasv_max_port=50064

To be clear, you were able to connect to your server using lftp?
If so, there is a misconfig in FileZilla (?)

You are using FTPS; not sftp. SFTP is completely different: it is FTP via ssh using port 22 while your config is FTPS.

There was a problem with debuging in VSFTPD. It seems that VSFTPD does not show everything. So I had permision problems. You can read more on this URL:
http://ramblings.linkerror.com/?p=45