FTP from LINUX. Too many files?

Experts,
I have an application on a linux server that writes some output files into a single directory. With recent growth in business that directory has about 147,000 files in it at any time.  These are mostly temp print files that are continually purged

I have a handful of users that use FTP clients (WS_FTP and Filezilla) to copy specific files from linux to their PC's. This is not working anymore, the FTP clients are only displaying a fraction of the total files.

I am not sure what is going on. Is there a limitation on either the FTP client of Linux server? Is some type of network or firewall issue preventing listing of all files? Is the issue at the desktop level?

As a temp solution, I am forced to access the Linux server and copy the file into a new (near empty) directory. The user can then grab the file from there no problem.
LVL 1
JDCamAsked:
Who is Participating?
 
ecarboneCommented:
I agree with David.  find ... delete is far more efficient (and possibly your only option with that many files). I thought Linux had a limit to the total number of files (regardless of directory). I am surprised you didn't hit that limit.

The files that are getting dumped onto your server - you mentioned you are moving them. But is your final intention to delete them? You can automate the deletion of the temp files.

I have an ecommerce site that runs on a linux server. The site dumps thousands of files into a directory. If I let it get too full, my website goes down. So I have a cron task that finds the files and deletes them and also returns to me (in an email) the number of files and the actual time it took to delete them all. It's incredible how fast it works vs deleting files one at a time:

     cd ./public_html/var/cache/smarty_cache && ls -1 | wc -l && time find . -type f -delete

(this deletes ALL of the files in the specified directory)
0
 
masnrockCommented:
Check the configuration of the server. Which Linux distro and version are you using?
0
 
JDCamAuthor Commented:
Linux version 4.1.12-61.1.28.el6uek.x86_64 (mockbuild@x86-ol6-builder-06) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #2 SMP Thu Feb 23 20:03:53 PST 2017

LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: OracleServer
Description:    Oracle Linux Server release 6.9
Release:        6.9
Codename:       n/a
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
masnrockCommented:
Also forgot to ask which FTP server you're using on that server.
0
 
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
If I understand what you're saying, trying to have 100,000s of files displaying in an FTP directory listing will likely fail in all sorts of ways.

FTP just wasn't meant to handle this number of files efficiently.

Simple solution, no FTP user will be downloading print files, so move these to a different directory.

Another tip, when purging these files, using find . {args} -delete is far more efficient than any other method. I ended up with some client with... less than intelligent developers, who were dumping small log files in one directory, with no pruning. This resulted in nearly 5,000,000 files which had to be removed. The find -delete trick was the only method that worked, without taking the machine down.

The solution you mention using, is similar to what I suggest + better to move the temp print files elsewhere, because your solution requires walking the entire directory to match + move files. This can take a huge amount of machine resource + be very slow, for large numbers of files.
1
 
JDCamAuthor Commented:
Let me clarify... Every file in the directory is a temp print file. Depending on the type, some are purged at 24 hrs, some at 7 days, some at 60 days and others 1 yr. The number of files on-hand is directly tied to how busy we are (# of print jobs).

Users might access at most 10-12 files per week. Unknown which file will be needed until they ask for it, so trying to redirect in advance would not work.

The FTP client method has been in place for 10 to 12 years with no issues until a few weeks ago. Seems we hit a tipping point.
0
 
ecarboneCommented:
If there's no way to control the number of files (like you said, it's tied to how busy you are), then what if your ftp home directory has 12 folders - one for each month of the year... 01, 02, 03 ... 12

Then as each file gets uploaded to FTP, have a script that automatically moves the incoming file into the appropriate folder.

So at any given moment your home folder has nothing in it except 12 folders.
Basically you're dividing your files into folders so you don't overload the FTP server and the linux os.

I'm sure a linux guru could help with the script.
Or look to the FTP server software to do it. For example, check out crushftp.com  It has a lot of automation options.
2
 
huacatCommented:
Agree with Ecarbone.

And seems it isn't a server problem, just a FTP client software problem. The FTP client software might can't handle too much files in a single directory. Most of the FTP client software didn't use the "virtual data item" tech to show the file lists, that's means it had to create too many list items to show them on the GUI, it will cause heavy efficient problem.

Keep a directory tree structure  to store files should solve the problem.
0
 
JDCamAuthor Commented:
Thanks,
I am unable to modify the directories without involving the Software company. And I will likely reccomend such changes for a future release. In the interim, I am stuck with what I have.

I agree that the issue is on the client side FTP program unable to handle so many files.
Playing around with Filezilla, I found that although the file may not be listed, the search function finds it no problem. I will simply show the users to use search instead of of scrolling through the files (which is more efficient anyways). \
0
 
huacatCommented:
congratulation! The work around solution is cool.

Anyway, Keep a directories structure needn't involving the software company.
You just need a periodic run script which scan the files and move then to sub direcotries by some rules.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.