Solved

Bash shell script - Read file and add deliminator.

Posted on 2008-10-17
11
944 Views
Last Modified: 2013-12-26
I have a file (.dat) with fields that are not seperated by any deliminators, but by position and need to convert it to a csv file.  I need a bash script to read the .dat file, find the fields based on positions and write to a new csv file with the deliminator.

For example, my dat file contains
11222333344444
99888777766666
and need to convert it to:
11,222,3333,44444
99,888,7777,66666

Thanks in advance.
0
Comment
Question by:knc26
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 2
  • 2
  • +1
11 Comments
 
LVL 2

Expert Comment

by:tdiops
ID: 22742779

# change the numbers to represent the number of columns in each field
# add a new field delimiter, '\(.\{n\}\)' for more fields if necessary
 
mysplit() {
 sed 's/\(.\{2\}\)\(.\{3\}\)\(.\{4\}\)\(.\{5\}\)/\1,\2,\3,\4/'
}
 
sampledata() {
cat << %%%
11222333344444
99888777766666
%%%
}
 
sampledata | mysplit

Open in new window

0
 
LVL 40

Accepted Solution

by:
omarfarid earned 250 total points
ID: 22742867
try

cat file | while read line
do
   F1=`echo $line | cut -c 1-2`
   F2=`echo $line | cut -c 3-5`
   F3=`echo $line | cut -c 6-9`
   F4=`echo $line | cut -c 10-14`
   echo "$F1, $F2, $F3, $F4" >> newfile
done
0
 

Author Comment

by:knc26
ID: 22744600
I'm having a problem with the cut command.  When there is a field with more than one byte of space, cut command only includes one space.

For example
do
   F1=`echo $line | cut -c 1-2`
   F2=`echo $line | cut -c 3-5`
   F3=`echo $line | cut -c 6-9`
   F4=`echo $line | cut -c 10-14`
   echo "$F1, $F2, $F3, $F4" >> newfile
done

on: 112223  344444
gives me:
11,222,3 3,44444 (one blank) instead of 11,222,3  3,44444 (two blanks)
0
The Ultimate Checklist to Optimize Your Website

Websites are getting bigger and complicated by the day. Video, images, custom fonts are all great for showcasing your product/service. But the price to pay in terms of reduced page load times and ultimately, decreased sales, can lead to some difficult decisions about what to cut.

 
LVL 48

Expert Comment

by:Tintin
ID: 22744771
To use the cut solution, you need to ensure you have quotes around $line, ie:
#!/bin/sh
while read line
do
   F1=`echo "$line" | cut -c 1-2`
   F2=`echo "$line" | cut -c 3-5`
   F3=`echo "$line" | cut -c 6-9`
   F4=`echo "$line" | cut -c 10-14`
   echo "$F1,$F2,$F3,$F4" >> newfile
done <file

Open in new window

0
 
LVL 48

Expert Comment

by:Tintin
ID: 22744798
Please note that the while loop and cut solution is extremely slow.  The following sed solution is much shorter and many, many times quicker


sed "s/\(..\)\(...\)\(....\)\(.....\)/\1,\2,\3,\4/" file >newfile

Open in new window

0
 
LVL 2

Expert Comment

by:tdiops
ID: 22744853
Tintin: Check out the first response.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 22744874
tdiops, I did see your suggestion.  Note that the extended regexs aren't supported by all sed versions.  My solution will work with any sed version.
0
 

Author Comment

by:knc26
ID: 22744893
I think the cat command is removing the extra continous spaces.  Prior to performing the cut, I'm echoing each line and can see that multiple spaces within each record is being reduced to only one space.

Is this a behavior with the cat command?  I didn't see any cat option that would prevent this.  Is there another command that i can use instead of cat?
0
 
LVL 48

Expert Comment

by:Tintin
ID: 22744938
knc26.  I've already given you the reason and a cut solution to deal with multiple spaces.  Please re-read my second last post.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 22744944
How big are the files you are processing?  Please take note of my other post regarding how slow a while loop and cut is.
0
 
LVL 48

Expert Comment

by:Tintin
ID: 22745348
Here's a *very* graphic example as to how much quicker sed is.  Using just a small sample file (10000 lines), the while loop/cut solution takes 70 seconds to run compared to 0.02 of a second for the sed solution.  

That means the sed solution is 3500 times quicker!!!

$ cat script
#!/bin/sh
while read line
do
   F1=`echo "$line" | cut -c 1-2`
   F2=`echo "$line" | cut -c 3-5`
   F3=`echo "$line" | cut -c 6-9`
   F4=`echo "$line" | cut -c 10-14`
   echo "$F1,$F2,$F3,$F4" >> newfile
done <file


$ wc -l file
10000 file

$ time ./script
real    1m10.338s
user    0m8.740s
sys     0m41.060s

$ time sed "s/\(..\)\(...\)\(....\)\(.....\)/\1,\2,\3,\4/" file >newfile
real    0m0.022s
user    0m0.021s
sys     0m0.001s
0

Featured Post

[Webinar] How Hackers Steal Your Credentials

Do You Know How Hackers Steal Your Credentials? Join us and Skyport Systems to learn how hackers steal your credentials and why Active Directory must be secure to stop them. Thursday, July 13, 2017 10:00 A.M. PDT

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Let's say you need to move the data of a file system from one partition to another. This generally involves dismounting the file system, backing it up to tapes, and restoring it to a new partition. You may also copy the file system from one place to…
Over the years I've spent many an hour playing on hardened, DMZ'd servers, with only a sub-set of the usual GNU toy's to keep me company; frequently I've needed to save and send log or data extracts from these server back to my PC, or to others, and…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

691 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question