[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

How do I concatenate many files into a single file?

Posted on 2014-08-28
10
Medium Priority
?
401 Views
Last Modified: 2014-08-28
I have got around 100 files. I would like to concatenate them into a single file. Just a note, these are all text files and there should not be any spaces in between file1 and file2 or file2 and file3 and so on. Also, contents of the file2, file3 and so on will start from the next line wherever the previous file end. For example,

file_20140801_1

100        20140101
101        20140102

file_20140802_2

1000     20130101
1001     20130102

file_20140803_3

2000    20140601
2001    20140602


When I concatenate above three files, it should look like below,

file

100        20140101
101        20140102
1000      20130101
1001      20130102
2000      20140601
2001      20140602

Considering above example, how do I concatenate around 100 files into a single file in UNIX. Please advise.
0
Comment
Question by:IT_ETL
10 Comments
 
LVL 38

Accepted Solution

by:
Gerwin Jansen, EE MVE earned 2000 total points
ID: 40290948
Are the files sorted in some way?

Basically you just cat every file to one new file:

cat `ls -ltr file_*` >> newfile
0
 
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 40290966
Correction: ls -tr

Or: ls -1 | sort
0
 

Author Comment

by:IT_ETL
ID: 40291164
The files are not sorted, can be concatenated in any order.
0
Get your Disaster Recovery as a Service basics

Disaster Recovery as a Service is one go-to solution that revolutionizes DR planning. Implementing DRaaS could be an efficient process, easily accessible to non-DR experts. Learn about monitoring, testing, executing failovers and failbacks to ensure a "healthy" DR environment.

 
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 40291187
Ok, based on your example: are all the filenames named like this:

file_20140801_1
file_20140802_2
file_20140803_3

etc?

Then the 'sort' would just be alphabetical and this should work for you:

cat `ls -1 file_* | sort` > newfile
0
 

Author Comment

by:IT_ETL
ID: 40291197
Thanks Gerwin. Above command concatenated all the files. Now I like to check couple of things.

1) How do I check number of files exist start with file* in a certain directory? I am expecting to see total count 111.
2) How do I check combined number of records from all the files equal to the number of records in a single file? I typed below commands,

wc -l file*
#I see number of records in each file plus total number of records on the bottom
wc -l single_file*
#I see total number of records in a single file

Number of record counts matched. Would you suggest any other commands to validate number of record counts are same?
0
 
LVL 31

Expert Comment

by:serialband
ID: 40291208
ls file*|wc

Your command wc -l file* list the number of lines in each file.

wc outputs 3 items.
Number of lines
Number of words
Number of characters
0
 
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 40291216
wc -l file*
# above command will count lines in every file matching file* pattern, with a summary line at the end

sample:
wc -l file*
 2 file_20140801_1
 2 file_20140802_2
 2 file_20140803_3
 6 total


wc -l newfile
# above command will count lines in the new file named newfile

sample:
wc -l new_file
6 new_file

Note that both commands above give the same output: 6 lines in total.
0
 

Author Comment

by:IT_ETL
ID: 40291257
ls file*|wc gives me below output,

$ ls file*|wc
     111     111    488
0
 
LVL 38

Expert Comment

by:Gerwin Jansen, EE MVE
ID: 40291298
ls file* | wc -l

is giving you the amount of files - you forgot the -l at the end?

$ ls file* | wc -l
3
0
 
LVL 48

Expert Comment

by:Tintin
ID: 40291719
No need for ls

cat file* >newfile
0

Featured Post

Configuration Guide and Best Practices

Read the guide to learn how to orchestrate Data ONTAP, create application-consistent backups and enable fast recovery from NetApp storage snapshots. Version 9.5 also contains performance and scalability enhancements to meet the needs of the largest enterprise environments.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Background Still having to process all these year-end "csv" files received from all these sources (including Government entities), sometimes we have the need to examine the contents due to data error, etc... As a "Unix" shop, our only readily …
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…
Suggested Courses
Course of the Month19 days, 5 hours left to enroll

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question