Solved

Best Format To Use

Posted on 2012-12-28
6
135 Views
Last Modified: 2013-01-10
Hi Experts,

     I have a spreadsheet with over 100k rows of data.  I typically have to search on 100 or more keywords within this spreadsheet.  What would be the best or fastest language to use for this if developing an app to perform this function?    Right now, the Visual Basic App that I have is taking over 20 minutes to search on just 60 words through 100k rows of data on my spreadsheet.  There's got to be a quicker way.  Any suggestions are very much appreciated.
0
Comment
Question by:itsmevic
6 Comments
 
LVL 83

Assisted Solution

by:CodeCruiser
CodeCruiser earned 250 total points
ID: 38728185
Would data remain in Excel? If yes then fastest option would be VBA.
0
 
LVL 10

Accepted Solution

by:
broro183 earned 250 total points
ID: 38730305
hi,

Twenty minutes seems a long time. If you post your code we may be able to make some suggestions that will speed it up (eg remove any ".select", turn off calculation etc).


I've attached an example file with code that I wrote about 3 years ago (presented in: http://www.thecodecage.com/forumz/1054993832-post15.html ). The code is modified from Tushar's FindAll function. It is not polished, but it may give some ideas that can be included in your project.

On review of this old file, there are some things that I would write quite differently now, such as the "Public ranges" and the hardcoded "7" (bad Rob!) which resizes the search range to seven columns. The file uses ".find" which was fine for the 10-20k rows of data in the file's original usage. I'm not sure how it will perform with 100k rows of data, so it may not be as good an approach as the use of an "in memory (VBA) array".

Rob
1878d1329220888-spreadsheet-cont.xls
0
 
LVL 20

Expert Comment

by:clarkscott
ID: 38734328
Are you searching through every column in the rows... or just certain columns?  How many columns are you searching?  How much data is in these columns?

These are all important things to know to determine best methods.

Scott C
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 

Author Comment

by:itsmevic
ID: 38741334
would be doing a complete sweep of the spreadsheet.  The column range would be from A to M.  Report size varies from 80k to 120k rows.  Data in these columns consists of single words, multiple words, file paths, domain\username, etc, etc....
0
 

Author Closing Comment

by:itsmevic
ID: 38761254
Thank you!
0
 
LVL 10

Expert Comment

by:broro183
ID: 38762427
hi itsmevic,

Thank you for the points. Would you mind showing the final code you are now using?

Rob
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Deploying a Microsoft Access application in a Citrix environment is not difficult but takes a few steps. However, Citrix system people are often of little help, as they typically know next to nothing about Access. The script provided here will take …
This article will guide you to convert a grid from a picture into Excel format using Microsoft OneNote and no other 3rd party application.
The view will learn how to download and install SIMTOOLS and FORMLIST into Excel, how to use SIMTOOLS to generate a Monte Carlo simulation of 30 sales calls, and how to calculate the conditional probability based on the results of the Monte Carlo …
Graphs within dashboards are meant to be dynamic, representing data from a period of time that will change each time the dashboard is updated with new data. Rather than update each graph to point to a different set within a static set of data, t…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now