Server 2003. When I telnet to port 5900, Pop3 server replies

I have been unable to run VNC in server mode on our Windows 2003 Standard Edition Server. When I telnet to port 5900, I get the response : "220 Internet Acces Protocol 3". (Access is mis-spelt in the response).

Typing Help provides a list of 40 three and four-letter commands, none of which I can run because I can't get authenticated by the program.

I've uninstalled VNC and am pretty sure it is not a left-over service from that program. We run VNC on a couple of servers (without problem) and when I telnet to 5900 on these servers, I get "RFB 003.008".

My guess is that this is a trojan, but nothing that my AV software has found.

Checking through the system services provides no answer and, obviously if it's malicious, I want it off of our server.

Any ideas what this program might be, and any way to get rid of it.
Who is Participating?
legalsrlConnect With a Mentor Commented:
Hi Paul,

Glad the problem is fixed now......VNC is a security nightmare for most companies....

I would suggest implementing a VPN, just a Microsoft one will do before allowing anyone to connect to a Terminal Server or their own PC.

At least you know then that they have to have some access level to get into the network

Hi Paul,

The way VNC works is the following

Technical details:

1) Server sends its version, "RFB 003.008\n"
2) Client replies with its version, "RFB 003.008\n"
3) Server sends 1 byte which is equal to the number of security types offered
3a) Server sends an array of bytes which indicate security types offered
4) Client replies with 1 byte, chosen from the array in 3a, to select
the security type
5) The handshake, if requested, is performed, followed by "0000" from the server

Hence you are getting the above output on the servers that VNC works correctly on.

What AV software are you running and what else is running on that server ?

Can you download hijackthis from and run it, then paste the log file through the analyser on the site and post the link to results here please

I can then have a look to see what's lurking

PAULWINNETTAuthor Commented:
Hi Si

Sophos for our AV.  After sending a suspicious file I found on the server and sending it to Sophos, they confirmed it as a Trojan, updated their IDE files and Sophos removed the Trojan with its associated files when I did a sweep last night.

The VNC services running on a nunmber of workstations and servers all respond to telnet in a similar way except the this particular server, giving the telnet response "220 Internet Acces Protocol 3".

Hijack this logfile link:

Improved Protection from Phishing Attacks

WatchGuard DNSWatch reduces malware infections by detecting and blocking malicious DNS requests, improving your ability to protect employees from phishing attacks. Learn more about our newest service included in Total Security Suite today!

Hi Paul,

You've got a few nasties in there

These need to be fixed

Download SpySweeper from and install it, (14 day free trial) update it and scan the machine, removing anything it finds.

O23 - Service: system - Unknown owner - C:\WINDOW\ (file missing)

That worries me !

I'd also reinstall Sophos on the machine as some of the key parts are missing, most likely removed by the malware

O23 - Service: Sophos Agent - Unknown owner - C:\Program Files\Sophos\Remote Management System\ManagementAgentNT.exe" -service -name Agent (file missing)    <--- Not good

Unknown   These entries shows all services which are not from Microsoft. Often malware is starting as a systemservice and it's not easy to detect it.

   Unknown service. (ManagementAgentNT.exe)
  O23 - Service: Sophos AutoUpdate Agent - Unknown owner - C:\Program Files\Sophos\Remote Management System\AutoUpdateAgentNT.exe" -service -name ALC (file missing)  <--- Not good either

You also need to look at this

O23 - Service: Resultant Set of Policy Provider (RSoPProv) - Unknown owner - C:\Documents and Settings\Administrator.GTC\WINDOWS\system32\RSoPProv.exe (file missing)

O20 - Winlogon Notify: dimsntfy - dimsntfy.dll (file missing)

I presume this is you guys - O17 - HKLM\System\CS1\Services\Tcpip\Parameters: Domain =

This DOES need to be fixed

O10 - Broken Internet access because of LSP provider 'c:\documents and settings\administrator.gtc\windows\system32\mswsock.dll' missing

You can fix this by

at a command prompt...type
netsh winsock reset catalog

then type from a command prompt
netstat -a

Post the results here

Also, this entry here
R0 - HKCU\Software\Microsoft\Internet Explorer\Main,Local Page = C:\WINDOWS\system32\blank.htm

Just check that file is actually what it says it is i.e. blank !

That should keep you busy for a Friday afternoon.

Let me know what SpySweeper finds


PAULWINNETTAuthor Commented:
Hi Si

Thanks for the time and trouble, Si.

I've downloaded and installed Spy Sweeeper, as you suggested. Spy Sweeper won't be able to run it until I'm able to restart the server on Monday morning.
Hi paul,

No problem at all matey.....

You can run Spy Sweeper before you restart the server, it's just the Protection Shields won't function properly until the reboot.

It will still stop and detect spyware

PAULWINNETTAuthor Commented:
Hi Si

SpySweeper didn't sit on our system at all well. We run MS SQL 2000 on a 2003 Server and it seemed to trip up the SQL server and I've had to remove it this morning.  Having said that, I took your suggestion and reinstalled Sophos which then found a couple of Trojans and removed the associated files and services during a new sweep on Sunday. One of those programs was the file that was, in fact, the source of the problem that resulted in me starting this thread.

VNC is probably insecure. Our firewall has always allowed port 6900 to connect to our server from the Internet and I would think the intrusion has come via and taken over this port. I've seen many references to VNC users having access denied them, getting the message 'Invalid Protocol' in response to a failed connection and I would think that a trojan of this nature may wll be a cause.  

Now this program and associated service has been removed, I can now connect to 6900 internally and run VNC without problem.

My thanks to you for recommending Hijackthis and assisting cracking this problem.

Best regards

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.