Hadoop and dat files

I was not sure where to post this question. Moderator I my need your help in placing it in the right category.

I want to know if its possible to move .dat files into Hadoop
This should be a yes but want to make sure. Assuming the answer is yes what tool and how do I move data into the Hadoop Cluster.

The The next question how do I query it and with what.
LVL 8
Leo TorresSQL DeveloperAsked:
Who is Participating?
 
Leo TorresSQL DeveloperAuthor Commented:
Changed the Topics in hope of getting a response.
0
 
Tomas Helgi JohannssonCommented:
Hi!

Where does these .dat files orginate from ?
What is the file-structure used ? Are they JSON files or something else ?
You will need to ask yourself those questions before you can determine how you query these .dat files..

Regards,
    Tomas Helgi
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
Leo TorresSQL DeveloperAuthor Commented:
The dat contain Application data. Right now we have to use a service to read these files and place the data in SQL. The service is very prone to fail and the sql jobs are for sql 2000.

My idea was to enter data into HADOOP to enable the transfer of data.

Could a map-reduce job move this data?

Or is this idea a pipe dream?
0
 
Leo TorresSQL DeveloperAuthor Commented:
These dat files are the equivalent to sql data files the difference is that reading those is easy with the SQL engine. The service does not work work on SQL 2012.
0
 
Tomas Helgi JohannssonCommented:
HI!

What do you mean by "These dat files are the equivalent to sql data file." ?
How is the data structure inside the files ?
Most likely you could use a MapReduce approach. Check out the built-in FileInputFormats that your Hadoop engine provides. Otherwise you would need to customize that class.

https://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html
https://hadoopi.wordpress.com/2013/05/27/understand-recordreader-inputsplit/

Regards,
    Tomas Helgi
0
 
Leo TorresSQL DeveloperAuthor Commented:
These files are basically database tables used by the application .

Here is an example of what data looks like. There are some special Charters at the top.

<<bbx>>f     „                    #                         	                                                '               *     +     ,                              /               ÿ                                                                                                                                         .!            	áÊŒ     Ì~                       	áØ      Ö¬     Þä     Ì´     "ä       ¨      œ     ¥8     €À     Ù¤                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       þþþþ10001033    
00
FirstName
LastName

MRS
12345678
F
H
W
00
2


345689019M  
14020285        
111113332


9992740899

1111 SW John Doe AVENUE

City
FL
33013



MGpO!hAME[MGoAMEEMEZAMC7MEDA
004
MBw2:MTO
M].,5TMA


        

        
M00002-20050420
NEW
20050426
LastName              
FirstName    
 

Open in new window

0
 
Leo TorresSQL DeveloperAuthor Commented:
They are database files is what I meant. Think this is a better way to put it.

From what I have been told these are ISAM files
0
 
Leo TorresSQL DeveloperAuthor Commented:
if i am not mistaken I Think sqoop is for SQL I have used that one before. I am not sure how that would would with ISAM Files. Has anyone moved data from ISAM files? I will review your links.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.