FTP permission problem when direct access typing IP in IE

We have a strange problem about FTP permission.

 We have several different users, one IP address, each user accessing to its directory via virtual directories set on the FTP server. No problem until there. Allow anonymous login is unchecked. If we try to connect anonymously with FTP client, we are rejected.

But we realize that if we just type ftp://[IP address] in IE we can access, browse, read/write all the server !! If we uncheck read/write in the home directory of the main FTP account, then none of our account can access. So how can we avoid this access (we do not want any anonymous access anyway)

Thanks for help