Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Number of files in directory

Posted on 2002-07-21
5
Medium Priority
?
235 Views
Last Modified: 2013-12-27
Hi

I am currently working on a product where I am importing files into a database. The files are being placed in that directory faster than I can import it. If the files go past a couple of thousands say 2000, then the system slows down, thus I need to know if there are a max number of files per directory is before the performance will be affected. I need to know what the max value of files is.

I am currently running on Solaris 8. System hardware: 2gigs of Ram, dual P3 800Mhz Intel system
0
Comment
Question by:RonaldMundell
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
5 Comments
 
LVL 51

Expert Comment

by:ahoffmann
ID: 7167560
AFAIK there is no way to compute the **exact** amount of files allowed.
This depends on the filesystem, and the parameters used when it was created. The most important parameter is the number of inodes.
Use something like following to get the used parameters:

     mkfs -m /dev/dsk/c0t3d0s7

Then have a look at the nbpi= parameter, and check in the mkfs or newfs man-page how to interprete this value.
0
 
LVL 14

Accepted Solution

by:
chris_calabrese earned 80 total points
ID: 7169776
If you're using the default Sun UFS, the directory is kept in a linear list, which means that access time slows proportionately to the number of entries. How slow is too slow is of course, up to you.

But you may consider a strategy that places less files in each directory.

Here's a script that does such splitting...

#!/usr/bin/ksh
#
# Generate multi-level directory names so that we ensure
# no more than FACTOR files/directory over LEVELS levels
# of directories.
#
# With LEVELS=3 and FACTOR=100, we can deal with
# 10,000,000 files (FACTOR ^ (LEVELS+1)).
#
# Copyright (c) 2001 by Chris Calabrese
# (chris_calabrese@yahoo.com)
# The author grants permission to use/modify this program in
# any way as long as:
#   1. It is not resold for profit.
#   2. This copyright notice accompanies any distribution.
# All other rights reserved.
# Finally, the author makes no warantee of
# correctness or fitness for any purpose.
#
#
typeset -i LEVELS=3
typeset -i FACTOR=100

function path_of {
    print -r -- "$*" | cksum | read hash junk
    typeset -i i=0
    #
    # Hash is unsigned and ksh arithmatic is signed
    # so we need to deal with this before doing anything else
    #
    hash="$((hash / 2))"
    if [ "$hash" -lt 0 ]
    then    hash="$((0 - hash))"
    fi
    #
    # Generate a path that gives us
    # FACTOR subdirectories at each LEVEL
    #
    while [ "$i" -lt "$LEVELS" ]
    do  print -r -n -- "$((hash % FACTOR))/"
        hash="$((hash / FACTOR))"
        i="$((i+1))"
    done
    print -r -- "$*"
}

while [ "$#" -gt 0 ]
do      path_of "$1"
        shift
done
0
 

Author Comment

by:RonaldMundell
ID: 7172520
Hi All

I have not forgotten you. I am out of the office until friday. Then I will look at this.

Thank you
0
 
LVL 21

Expert Comment

by:tfewster
ID: 8094190
No comment has been added lately, so it's time to clean up this Topic Area.
I will leave a recommendation for this question in the Cleanup topic area as follows:

- Answered by chris_calabrese

Please leave any comments here within the next 7 days

PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER !

tfewster
Cleanup Volunteer
0
 
LVL 5

Expert Comment

by:Netminder
ID: 8144376
Per recommendation, force-accepted.

Netminder
EE Admin
0

Featured Post

Tech or Treat!

Submit an article about your scariest tech experience—and the solution—and you’ll be automatically entered to win one of 4 fantastic tech gadgets.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
I promised to write further about my project, and here I am.  First, I needed to setup the Primary Server.  You can read how in this article: Setup FreeBSD Server with full HDD encryption (http://www.experts-exchange.com/OS/Unix/BSD/FreeBSD/A_3660-S…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.
Suggested Courses

598 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question