Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

how to retain spaces using while read..do..done command

Posted on 2006-04-17
32
Medium Priority
?
1,211 Views
Last Modified: 2013-12-26
Hi everyone,
  I'm not an expert in unix programming. So the terms which I used may not be really correct. Need help in something which I believe is quite simple...

I have to read in filea.txt

AAA       111        
BBB       222     YYYY
CCC       333     ZZZZ
DDD       444        
 
then append a serialised variable (running number) at the back of the file

and output to another new file.

The following is the codes which I'm using now but it seems that the string of empty spaces between AAA and 111 is trimmed to become only 1 space in my output file.

while read Line
 do
     counter=$((counter+=1))
       
     print $Line $counter
         
 done < filea.txt | cut -c1-110 > fileb.txt


This is my output file

AAA 111 01
BBB 222 YYYY 02
CCC 333 ZZZZ 03
DDD 444 04

What I wanted is the file to retain as the original input file with the serialised number at the back of the file.

Something like this:

AAA       111                01
BBB       222     YYYY     02
CCC       333     ZZZZ    03
DDD       444                04

Thanks in advanced!
0
Comment
Question by:chawtee
  • 19
  • 7
  • 3
  • +1
32 Comments
 
LVL 85

Accepted Solution

by:
ozo earned 200 total points
ID: 16467850
while read Line
do
     counter=$((counter+=1))
       
     echo "$Line" $counter
         
done < filea.txt | cut -c1-110 > fileb.txt
0
 

Author Comment

by:chawtee
ID: 16468077
Hi ozo,

I have changed my codes to the above you suggested. the spaces will not be trimmed anymore (Thanks alot for your help). However, the counter variable and week is not output to the file. Can help me spot the mistake?

. /dir_a/dir_b/list_of_directories.sh

filename="filea.dat"
filename1="fileb.dat"

cd $datadir (contained in list_of_directories.sh)

counter=0

week=`date +%V`
 
 while read Line
 do
     counter=$((counter+=1))
         
     print "$Line" $week $counter
         
 done < $datadir/$filename | cut -c1-109 > $datadir/$filename1

Thanks in advance!
0
 
LVL 85

Expert Comment

by:ozo
ID: 16468108
does
echo "$Line $week $counter"
output the week and counter?
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 

Author Comment

by:chawtee
ID: 16468116
I tried this:

echo "$Line $week $counter"

but the week and counter is still missing.
0
 

Author Comment

by:chawtee
ID: 16468126
wait. wait. I think I misread.

shoud be echo instead of print right?

I go try again.
0
 

Author Comment

by:chawtee
ID: 16468132
changed to echo function but still doesn't work.
0
 
LVL 85

Expert Comment

by:ozo
ID: 16468147
Are the lines in the file more than 109 characters long?
What happen if you take out the cut?
0
 

Author Comment

by:chawtee
ID: 16468156
no. the files not longer than 109. jus exactly 109 bytes long.
 
Thanks! I'll try and let u know in a while.
0
 

Author Comment

by:chawtee
ID: 16468168
The $week and $counter appeared. but there is a ? in between my original file and $week and counter.

Something like this:

AAA       111                ?16 01
BBB       222     YYYY     ?16 02
CCC       333     ZZZZ    ?16 03
DDD       444                ?16 04

Ths 16 is the week and 01/02/03/04 is the counter.
0
 
LVL 38

Assisted Solution

by:yuzh
yuzh earned 200 total points
ID: 16474189
To read the whole line without have to worry about how many char in a line,
you do:

IFS="\012"    #read in the whole line
exec 0<filea.txt
while read Line
 do
     counter=$((counter+=1))
       
     print $Line $counter
done
exit
0
 

Author Comment

by:chawtee
ID: 16474221
Hi ozo,

I've done it.

There are some ^M characters at the back of my input file and got to remove it using this command.

:,$ s/(ctrl) + v + m//g

The output is okay if the input data file is like that.

Input:
(record 1) AAA       111     XXXX
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444     TTTT

Output
(record 1) AAA       111     XXXX 16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444     TTTT 16 04

But my output becomes like this if my input file is like that-->

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444    


Output:
(record 1) AAA       111 16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444 16 04

My $week and $counter variable shifted places.

I wanted it to remain at the same position with records 2 and 3.

Can u help me? I have increased the points.
0
 
LVL 27

Assisted Solution

by:Nopius
Nopius earned 200 total points
ID: 16474243
The $week and $counter appeared. but there is a ? in between my original file and $week and counter.

It may be if you have CRLF at the end of input file (not just LF)

Try this:
filename="filea.dat"
filename1="fileb.dat"
cd $datadir (contained in list_of_directories.sh)
counter=0
week=`date +%V`
tr -d '\r' $datadir/$filename | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $week $counter
 done | cut -c1-109 > $datadir/$filename1
0
 

Author Comment

by:chawtee
ID: 16474282
Hi yuzh,

some of my records got some bytes removed by the program.

E.g:

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY01
(record 3) CCC       333     ZZZZ01
(record 4) DDD       444     31

Output:
(record 1) AAA       111     16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444     3 16 04

it seems that 1 and 01 is being removed by the method u suggested.

However, the long string of empty spaces is retained in my output file.

Thanks.
0
 

Author Comment

by:chawtee
ID: 16474290
Hi Nopius,

I got this error "only one string may be given when deleting without squeezing repeats"
after executing the solution u suggested.
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16474338
tr is different in different OS es (\r not always ^M, but sometimes it's a 'r'), that is my fault.
What about formatting:
'I wanted it to remain at the same position with records 2 and 3.'
From my point of view the output is as expected for your example:

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444    


Output:
(record 1) AAA       111 16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444 16 04

week and counter are on that position, you asked for (appended to the end of line).
Do you mean you need extra spaces? Or you need week and counter to be in specific column number?
Or they should follow by 222 and 333 ?
0
 

Author Comment

by:chawtee
ID: 16474347
Hi Nopius,

I wanted something like that:

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444    

Output:
(record 1) AAA       111             16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444             16 04

You mind typing codes for me to work on?

Cos I'm still a newbie to unix scripting...

I'm running my script on F-secure SSH.
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16474348
with tr, it may still work:
cat | tr -d '\r' | ...
:-)
tr doesn't accept filenames, only stdin,
0
 

Author Comment

by:chawtee
ID: 16474367
Hi Nopius,

replying to ur post on 04/18/2006 10:04AM SGT

I changed

tr -d '\r' $datadir/$filename | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $week $counter
 done | cut -c1-109 > $datadir/$filename1

to

tr -d 'r' $datadir/$filename | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $week $counter
 done | cut -c1-109 > $datadir/$filename1

(I remove the '\' in the '\r')

but I still encounter this error message --> tr: only one string may be given when deleting without squeezing repeats

0
 

Author Comment

by:chawtee
ID: 16474406
Hi Nopius,

after trying out ur suggestion posted at 04/18/2006 10:07AM SGT

I got this error.

/dir1/dir2/filea.dat: AAA:  not found.
/dir1/dir2/filea.dat[2]: BBB:  not found.
/dir1/dir2/filea.dat[3]: CCC:  not found.
/dir1/dir2/filea.dat[4]: DDD:  not found.
cat: read error: I/O error

My codes are as follow:

cat | tr -d '\r' | $datadir/$filename | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $year $week $counter
 done | cut -c1-109 > $datadir/$filename1

0
 
LVL 27

Expert Comment

by:Nopius
ID: 16474474
Here is a correct ^M removal:

cat $datadir/$filename | tr -d '\r' | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $year $week $counter
 done | cut -c1-109 > $datadir/$filename1

also it's better to use 'awk' and sprintf() if you  like to format your output string (add extra spaces when there is no field).
0
 

Author Comment

by:chawtee
ID: 16474507
Hi Nopius,

The codes u suggested works.

But I still cannot get what I wanted.

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444    

Output: (This is what I want)
(record 1) AAA       111             16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444             16 04

Output: (This is what I get)
(record 1) AAA       111 16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444 16 04

0
 

Author Comment

by:chawtee
ID: 16474526
Hi Nopius,

Some amendments to what I wrote jus now..

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444    

Output: (This is what I want)
(record 1) AAA       111             16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444             16 04

Output: (This is what I get)
(record 1) AAA       111 16 01
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444 16 04

Record 2 and 3 didn't display $week and $counter.

Is it because of the cut statement?
0
 

Author Comment

by:chawtee
ID: 16474538
Hi Nopius,

This is my codes. I tried removing the cut statement. The $week and $counter is outputted to my output file.

But $week and $counter in record 1 and 4 is still not in the same column as the $week and $counterin records 2 and 3.

How?

Thanks in advance!
0
 

Author Comment

by:chawtee
ID: 16474547
as mentioned above:

cat $datadir/$filename | tr -d '\r' | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $week $counter
 done > $datadir/$filename1
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16474699
this is probably what you want:

printf  '%-22s %s %s\n' "$Line" $year $week $counter

instead of:
print "$Line" $year $week $counter

0
 
LVL 38

Expert Comment

by:yuzh
ID: 16474980
run dos2unix to convert the input file then process the file.
0
 

Author Comment

by:chawtee
ID: 16475002
Hi yuzh, how do I do that?

I'm really sorry...

Need some spoon feeding...
0
 
LVL 38

Expert Comment

by:yuzh
ID: 16475217
>> how do I do that

dos2unix inputfile outputfile

some OS version allow you to do:
dos2unix fliename

man dos2unix
also have a look at http:Q_20149302.html

PS: in case you don't have dos2unix in your system, you can also do:
      have a look at the answer in http:Q_21218999.html
 
      if you need more help, please post your OS version.
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16475253
chawtee: dos2unix will remove ^M from your input file.
You have already done it by vi :s/^V^M//g
if you like to remove ^M all the time, use yuzh suggestion. Mine 'tr -d' will also work with appropriate \r quoting.
0
 

Author Comment

by:chawtee
ID: 16475446
Hi everyone!

I've manage to get what I wanted using cat and cut function, followed by a while loop to loop through all the records to assign the running number to every record.

Thanks alot for all your help!

Though no one really provided me the exact solution, I'll split the points to all of you who had tried to helped. Hope u guys don't mind... yeah?

Thanks once again.
0
 
LVL 27

Expert Comment

by:Nopius
ID: 16475464
Ok, chawtee. That was really easy question, and very long discussion :-)
0
 

Author Comment

by:chawtee
ID: 16475490
hehe... Thanks a million for your help again, Nopius.

0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Here is how to use MFC's automatic Radio Button handling in your dialog boxes and forms.  Beginner programmers usually start with a OnClick handler for each radio button and that's just not the right way to go.  MFC has a very cool system for handli…
Introduction: Displaying information on the statusbar.   Continuing from the third article about sudoku.   Open the project in visual studio. Status bar – let’s display the timestamp there.  We need to get the timestamp from the document s…
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
Despite its rising prevalence in the business world, "the cloud" is still misunderstood. Some companies still believe common misconceptions about lack of security in cloud solutions and many misuses of cloud storage options still occur every day. …
Suggested Courses

571 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question