?
Solved

MYSQL stored procedure import fixed width text file

Posted on 2013-06-17
5
Medium Priority
?
831 Views
Last Modified: 2013-06-18
I am looking to import a large dataset from the IRS located here:
http://www.irs.gov/uac/SOI-Tax-Stats-Exempt-Organizations-Business-Master-File-Extract-(EO-BMF)
at the bottom of the page are file regional files, here is one of them:
http://www.irs.gov/file_source/pub/irs-soi/eo1.zip

The file is an ascii file with fixed width columns as defined here:
Position(s)      Contents

  1   9      Employer Identification Number (EIN).
 10   79      Primary Name of Organization.
 80   114      In Care of Name.
115   149      Street Address.
150   171      City.
172   173      State.
174   183      Zip Code.
184 - 187      Group Exemption Number.
188   189      Subsection Code.
190            Affiliation Code.
191   194      Classification Code(s).
195   200      Ruling Date.
201            Deductibility Code.
202   203      Foundation Code.
204   212      Activity Codes.
213            Organization Code.
214   215      Exempt Organization Status Code (New)
216 - 221      The Advance Ruling process is obsolete as of July 2008. These positions will be blank or zeroed out. For further information, please check the URL below: http://www.irs.gov/charities/charitable/article/0,,id=185568,00.html:
222 - 227      Tax Period.
228            Asset Code.
229            Income Code.
230 - 231      Filing Requirement Code.
232            PF Filing Requirement Code. (New)
233 - 235      Blanks.
236 - 237      Accounting Period.
238 - 250      Asset Amount.
251 - 263      Income Amount.
264      If the Income Amount is negative this contains a negative sign.
265 - 277      Form 990 Revenue Amount
278            If Revenue Amount is negative this contains a negative sign.
279 - 282      National Taxonomy of Exempt Entities (NTEE) Code.
283 - 317      Sort Name (Secondary Name Line).
318            0A - Line Feed

This data is updated monthly and I would like to have a stored procedure that I can run to grab the file and import it to a table with the appropriate columns without having to redefine all the column widths each time.

I am both a novice at stored procedures and have never used Load Data InFile (which is what the other forums suggested).

Could someone show me the template of what the stored procedure should look like so that I can see how to define the different column widths (just show a few and I can figure out the rest).  

What I imagine doing is uploading the files to the server then having a script on my server run that will call the stored procedure with a parameter of the file name to import from (there will be 4 files in total that I will be needing to import).  Since all the files have the identical column width structure and just have different file names that should allow me to repeat this processes as often as needed.
0
Comment
Question by:yoshcomputers
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
5 Comments
 

Author Comment

by:yoshcomputers
ID: 39254431
Here is the SQL that works...and does the fixed width that I need.  I just now want to store this a as a store procedure where I can just change the filename.

LOAD DATA LOCAL INFILE 'C:/Users/Yosh/Downloads/eo1/eo1.lst' INTO TABLE a_irs
(@var1)
SET
irs_ein=SUBSTR(@var1,1,9),
irs_name=SUBSTR(@var1,10,70),
irs_address=SUBSTR(@var1,115,35),
irs_city=SUBSTR(@var1,150,22),
irs_state=SUBSTR(@var1,172,2),
irs_zipcode=SUBSTR(@var1,174,10),
irs_groupexemptionnumber=SUBSTR(@var1,184,4),
irs_subsectioncode=SUBSTR(@var1,188,2),
irs_affiliationcode=SUBSTR(@var1,190,1),
irs_classificationcode=SUBSTR(@var1,191,4),
irs_exemptstatuscode=SUBSTR(@var1,214,2),
irs_nteecode=SUBSTR(@var1,279,4),
irs_sortname=SUBSTR(@var1,283,35);
0
 

Author Comment

by:yoshcomputers
ID: 39254704
I found out in Mysql you cannot have LOAD DATA INFILE in a stored procedure
0
 
LVL 49

Expert Comment

by:PortletPaul
ID: 39255563
this (brief) page discusses a workaround that appears relevant
http://kedar.nitty-witty.com/blog/using-load-data-infile-with-stored-procedure-workaround-mysql
0
 
LVL 27

Accepted Solution

by:
Zberteoc earned 1500 total points
ID: 39256305
As you already found out you CAN'T place the LOAD statement in a stored procedure. However you can place it in a script file, i.e. load_file.sql, and then have a job, either cron task on Linux or Sheduled Task on WIndows, to execute the script file. You can do that directly using MySQL if you can execute the script on the server or you can have a script in VBscrip, JavaScript or whatever language you want to execute it. The simplest way is to use MySQL directly and the syntax is simple:

mysql -h "server-name" -u "user" "-pXXXXXXXX" "database-name" < "load_file.sql"

Open in new window

0
 

Author Closing Comment

by:yoshcomputers
ID: 39256486
I appreciate the example, though it did leave something to the imagination to fill is some details on implementation.  It certainly got me on the right direction.
0

Featured Post

Optimize your web performance

What's in the eBook?
- Full list of reasons for poor performance
- Ultimate measures to speed things up
- Primary web monitoring types
- KPIs you should be monitoring in order to increase your ROI

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Does the idea of dealing with bits scare or confuse you? Does it seem like a waste of time in an age where we all have terabytes of storage? If so, you're missing out on one of the core tools in every professional programmer's toolbox. Learn how to …
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…
In this video, Percona Solutions Engineer Barrett Chambers discusses some of the basic syntax differences between MySQL and MongoDB. To learn more check out our webinar on MongoDB administration for MySQL DBA: https://www.percona.com/resources/we…
Suggested Courses

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question