Search in a large Array

Posted on 2012-03-09
Last Modified: 2012-05-10
I have sorted array which contains over 50,000 file names let sat this is Array F1. And there is another sorted array which contain the path and the file name let say this is Array P1. I will need to pass the file name from Array F1 to the array P1 to find the path and then store the path in a third array at the same index as of the index of the filename in array F1.

Should I use Binary search or is there any other better search algorithm for search from 50k array elements. I will need to use C++
Question by:javaCaravan0

Accepted Solution

ziceva earned 64 total points
Comment Utility

For sorted arrays the best way of searching an element is definitely binary search ... you cannot hope for something better than the log2 complexity of the binary search. For a 50k array (relatively small) the search will end in a maximum of 16 steps.

I once was confronted with a harder problem were I had to perform an array_diff between two arrays of 4 to 6 million elements in php and the php provided function was taking literary hours (because it was not taking into account the fact that the arrays were already sorted).
When taking EACH element from the first array and making a binary search in the second one (so a few million binary searches in an array of a couple of million elemens), the job was done in seconds ... so binary search will perform well in solving problems way harder than yours

As stated before, 16 steps it a very fast search for your 50k array ...

Author Comment

Comment Utility
thank you ... please note, F1 hs 50k filenames, so search will need to be conducted 50k times. Array P1 also has 50k paths, i.e. one for each element of F1. Should I delete the array element from P1 once it has been found, this way, the size of P1 will keep shrinking so there will be less element to search from. What do you think.

Expert Comment

Comment Utility
So there are 50k searches that will need a maximum of 16 steps ... no biggie ... As stated before I've tested something similar but with millions of elements instead of thousands as you have (and I was doing it in php, c++ will be much quicker)...
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.


Author Comment

Comment Utility
thank you
LVL 85

Assisted Solution

by:Mike Tomlinson
Mike Tomlinson earned 62 total points
Comment Utility
Where did this data come from?...wouldn't it be better to store those pieces of information together in a CLASS (since you're using C++)?  Why the separate arrays?
LVL 32

Assisted Solution

phoffric earned 62 total points
Comment Utility
I agree - why use separate arrays. Why search at all. You can store the P1 array as a map (or multimap) with filename being the key, and the path (excluding the filename) being the folder path.
C++ References:

Let the map do the searching for you using find:
BTW - as you insert the filename and path into the map, it is sorted automatically for you.

But you may want to consider hash algorithm for faster access time. (Haven't used it, but I will if the need arises.)
C++ References:
Idea is to hash the filename to come up with a list of paths (only 1 if all filenames are known to be unique). Brings search time closer to linear. (Still have to deal with collisions.)
LVL 22

Assisted Solution

ambience earned 62 total points
Comment Utility
>> Should I delete the array element from P1 once it has been found, this way, the size of P1 will keep shrinking so there will be less element to search from. What do you think.

Two things:

- Shrinking is not a good idea because either you can swap with last item and resort the array or you'd have to reallocate the array and copy left over item. In both cases, performance will suffer.

- Since you are willing to modify the array it appears as though these arrays are transient in nature, filled up for just this task and wont be reused. This make it important to consider optimizations at the time of population arrays.

For example,

Array F1 need not be sorted.


Cant you just
- load the second array into a Tree (like the Filesystem) with each directory as a node in the tree and filenames as the leaves.
- Then do an preorder traversal of the tree (left, root, right) and put the path in PATH array and FILENAME in Filenames array.

At the end you get sorted arrays of fienames and corresponding paths. Unless ofcourse if you expecting a case where filenames in F1 array may not exist in P1 array.

Again, depending upon how deep the paths can be, A tree could achieve lesser number of comparisons compared to binary search on a 50K array. The worst case would be the maximum number path segments.

Note that looking up a path for a filename is a classic dictionary lookup and there are algorithms and data-structures that can achieve performance much greater than a binary search. For example, using a Trie variant like PARTICIA. The cost of setting up such structures isnt trivial and in your case a binary tree/search ought to perform well enough.

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Suggested Solutions

There is an easy way, in .NET, to centralize the treatment of all unexpected errors. First of all, instead of launching the application directly in a Form, you need first to write a Sub called Main, in a module. Then, set the Startup Object to th…
Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
The goal of this video is to provide viewers with basic examples to understand and use switch statements in the C programming language.
This video teaches viewers about errors in exception handling.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now