how to retain spaces using while read..do..done command

Hi everyone,
  I'm not an expert in unix programming. So the terms which I used may not be really correct. Need help in something which I believe is quite simple...

I have to read in filea.txt

AAA       111        
BBB       222     YYYY
CCC       333     ZZZZ
DDD       444        
 
then append a serialised variable (running number) at the back of the file

and output to another new file.

The following is the codes which I'm using now but it seems that the string of empty spaces between AAA and 111 is trimmed to become only 1 space in my output file.

while read Line
 do
     counter=$((counter+=1))
       
     print $Line $counter
         
 done < filea.txt | cut -c1-110 > fileb.txt


This is my output file

AAA 111 01
BBB 222 YYYY 02
CCC 333 ZZZZ 03
DDD 444 04

What I wanted is the file to retain as the original input file with the serialised number at the back of the file.

Something like this:

AAA       111                01
BBB       222     YYYY     02
CCC       333     ZZZZ    03
DDD       444                04

Thanks in advanced!
chawteeAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ozoCommented:
while read Line
do
     counter=$((counter+=1))
       
     echo "$Line" $counter
         
done < filea.txt | cut -c1-110 > fileb.txt
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
chawteeAuthor Commented:
Hi ozo,

I have changed my codes to the above you suggested. the spaces will not be trimmed anymore (Thanks alot for your help). However, the counter variable and week is not output to the file. Can help me spot the mistake?

. /dir_a/dir_b/list_of_directories.sh

filename="filea.dat"
filename1="fileb.dat"

cd $datadir (contained in list_of_directories.sh)

counter=0

week=`date +%V`
 
 while read Line
 do
     counter=$((counter+=1))
         
     print "$Line" $week $counter
         
 done < $datadir/$filename | cut -c1-109 > $datadir/$filename1

Thanks in advance!
0
ozoCommented:
does
echo "$Line $week $counter"
output the week and counter?
0
Cloud Class® Course: Microsoft Azure 2017

Azure has a changed a lot since it was originally introduce by adding new services and features. Do you know everything you need to about Azure? This course will teach you about the Azure App Service, monitoring and application insights, DevOps, and Team Services.

chawteeAuthor Commented:
I tried this:

echo "$Line $week $counter"

but the week and counter is still missing.
0
chawteeAuthor Commented:
wait. wait. I think I misread.

shoud be echo instead of print right?

I go try again.
0
chawteeAuthor Commented:
changed to echo function but still doesn't work.
0
ozoCommented:
Are the lines in the file more than 109 characters long?
What happen if you take out the cut?
0
chawteeAuthor Commented:
no. the files not longer than 109. jus exactly 109 bytes long.
 
Thanks! I'll try and let u know in a while.
0
chawteeAuthor Commented:
The $week and $counter appeared. but there is a ? in between my original file and $week and counter.

Something like this:

AAA       111                ?16 01
BBB       222     YYYY     ?16 02
CCC       333     ZZZZ    ?16 03
DDD       444                ?16 04

Ths 16 is the week and 01/02/03/04 is the counter.
0
yuzhCommented:
To read the whole line without have to worry about how many char in a line,
you do:

IFS="\012"    #read in the whole line
exec 0<filea.txt
while read Line
 do
     counter=$((counter+=1))
       
     print $Line $counter
done
exit
0
chawteeAuthor Commented:
Hi ozo,

I've done it.

There are some ^M characters at the back of my input file and got to remove it using this command.

:,$ s/(ctrl) + v + m//g

The output is okay if the input data file is like that.

Input:
(record 1) AAA       111     XXXX
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444     TTTT

Output
(record 1) AAA       111     XXXX 16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444     TTTT 16 04

But my output becomes like this if my input file is like that-->

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444    


Output:
(record 1) AAA       111 16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444 16 04

My $week and $counter variable shifted places.

I wanted it to remain at the same position with records 2 and 3.

Can u help me? I have increased the points.
0
NopiusCommented:
The $week and $counter appeared. but there is a ? in between my original file and $week and counter.

It may be if you have CRLF at the end of input file (not just LF)

Try this:
filename="filea.dat"
filename1="fileb.dat"
cd $datadir (contained in list_of_directories.sh)
counter=0
week=`date +%V`
tr -d '\r' $datadir/$filename | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $week $counter
 done | cut -c1-109 > $datadir/$filename1
0
chawteeAuthor Commented:
Hi yuzh,

some of my records got some bytes removed by the program.

E.g:

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY01
(record 3) CCC       333     ZZZZ01
(record 4) DDD       444     31

Output:
(record 1) AAA       111     16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444     3 16 04

it seems that 1 and 01 is being removed by the method u suggested.

However, the long string of empty spaces is retained in my output file.

Thanks.
0
chawteeAuthor Commented:
Hi Nopius,

I got this error "only one string may be given when deleting without squeezing repeats"
after executing the solution u suggested.
0
NopiusCommented:
tr is different in different OS es (\r not always ^M, but sometimes it's a 'r'), that is my fault.
What about formatting:
'I wanted it to remain at the same position with records 2 and 3.'
From my point of view the output is as expected for your example:

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444    


Output:
(record 1) AAA       111 16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444 16 04

week and counter are on that position, you asked for (appended to the end of line).
Do you mean you need extra spaces? Or you need week and counter to be in specific column number?
Or they should follow by 222 and 333 ?
0
chawteeAuthor Commented:
Hi Nopius,

I wanted something like that:

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444    

Output:
(record 1) AAA       111             16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444             16 04

You mind typing codes for me to work on?

Cos I'm still a newbie to unix scripting...

I'm running my script on F-secure SSH.
0
NopiusCommented:
with tr, it may still work:
cat | tr -d '\r' | ...
:-)
tr doesn't accept filenames, only stdin,
0
chawteeAuthor Commented:
Hi Nopius,

replying to ur post on 04/18/2006 10:04AM SGT

I changed

tr -d '\r' $datadir/$filename | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $week $counter
 done | cut -c1-109 > $datadir/$filename1

to

tr -d 'r' $datadir/$filename | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $week $counter
 done | cut -c1-109 > $datadir/$filename1

(I remove the '\' in the '\r')

but I still encounter this error message --> tr: only one string may be given when deleting without squeezing repeats

0
chawteeAuthor Commented:
Hi Nopius,

after trying out ur suggestion posted at 04/18/2006 10:07AM SGT

I got this error.

/dir1/dir2/filea.dat: AAA:  not found.
/dir1/dir2/filea.dat[2]: BBB:  not found.
/dir1/dir2/filea.dat[3]: CCC:  not found.
/dir1/dir2/filea.dat[4]: DDD:  not found.
cat: read error: I/O error

My codes are as follow:

cat | tr -d '\r' | $datadir/$filename | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $year $week $counter
 done | cut -c1-109 > $datadir/$filename1

0
NopiusCommented:
Here is a correct ^M removal:

cat $datadir/$filename | tr -d '\r' | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $year $week $counter
 done | cut -c1-109 > $datadir/$filename1

also it's better to use 'awk' and sprintf() if you  like to format your output string (add extra spaces when there is no field).
0
chawteeAuthor Commented:
Hi Nopius,

The codes u suggested works.

But I still cannot get what I wanted.

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444    

Output: (This is what I want)
(record 1) AAA       111             16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444             16 04

Output: (This is what I get)
(record 1) AAA       111 16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444 16 04

0
chawteeAuthor Commented:
Hi Nopius,

Some amendments to what I wrote jus now..

Input:
(record 1) AAA       111    
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444    

Output: (This is what I want)
(record 1) AAA       111             16 01
(record 2) BBB        222     YYYY 16 02
(record 3) CCC       333     ZZZZ 16 03
(record 4) DDD       444             16 04

Output: (This is what I get)
(record 1) AAA       111 16 01
(record 2) BBB        222     YYYY
(record 3) CCC       333     ZZZZ
(record 4) DDD       444 16 04

Record 2 and 3 didn't display $week and $counter.

Is it because of the cut statement?
0
chawteeAuthor Commented:
Hi Nopius,

This is my codes. I tried removing the cut statement. The $week and $counter is outputted to my output file.

But $week and $counter in record 1 and 4 is still not in the same column as the $week and $counterin records 2 and 3.

How?

Thanks in advance!
0
chawteeAuthor Commented:
as mentioned above:

cat $datadir/$filename | tr -d '\r' | while read Line
 do
     counter=$((counter+=1))
     print "$Line" $week $counter
 done > $datadir/$filename1
0
NopiusCommented:
this is probably what you want:

printf  '%-22s %s %s\n' "$Line" $year $week $counter

instead of:
print "$Line" $year $week $counter

0
yuzhCommented:
run dos2unix to convert the input file then process the file.
0
chawteeAuthor Commented:
Hi yuzh, how do I do that?

I'm really sorry...

Need some spoon feeding...
0
yuzhCommented:
>> how do I do that

dos2unix inputfile outputfile

some OS version allow you to do:
dos2unix fliename

man dos2unix
also have a look at http:Q_20149302.html

PS: in case you don't have dos2unix in your system, you can also do:
      have a look at the answer in http:Q_21218999.html
 
      if you need more help, please post your OS version.
0
NopiusCommented:
chawtee: dos2unix will remove ^M from your input file.
You have already done it by vi :s/^V^M//g
if you like to remove ^M all the time, use yuzh suggestion. Mine 'tr -d' will also work with appropriate \r quoting.
0
chawteeAuthor Commented:
Hi everyone!

I've manage to get what I wanted using cat and cut function, followed by a while loop to loop through all the records to assign the running number to every record.

Thanks alot for all your help!

Though no one really provided me the exact solution, I'll split the points to all of you who had tried to helped. Hope u guys don't mind... yeah?

Thanks once again.
0
NopiusCommented:
Ok, chawtee. That was really easy question, and very long discussion :-)
0
chawteeAuthor Commented:
hehe... Thanks a million for your help again, Nopius.

0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
System Programming

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.