Bash script to make file name the first column in a set of csv formatted files and then output to one large file

Hi Team,

  Am just requesting a bash script  where I can have the filename of the file included as the first column and have this repeated in a set of many csv files in a particular directory and then send all the output to an aggregated file, something like this:

Projecta.csv
one, two, three, four, five
six,seven,eight,nine,ten
eleven,twelve,thirteen,fourteen,fifteen

Projectb.csv
one-1, two-1, three-1, four-1, five-1
six-1,seven-1,eight-1,nine-1,ten-1
eleven-1,twelve-1,thirteen-1,fourteen-1,fifteen-1

....many more csv files....and output to one large file like this....

total.txt
Projecta,one, two, three, four, five
Projecta,six,seven,eight,nine,ten
Projecta,eleven,twelve,thirteen,fourteen,fifteen
Projectb,one-1, two-1, three-1, four-1, five-1
Projectb,six-1,seven-1,eight-1,nine-1,ten-1
Projectb,eleven-1,twelve-1,thirteen-1,fourteen-1,fifteen-1

* I will then use another script to import that file into a mysql database.

Thanks very much for all the help.
rleyba828Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ozoCommented:
#!/bin/bash
for f in Project*.csv ; do
  while read line ; do
   echo $f,$line
  done < $f
done >> total.txt
0
rleyba828Author Commented:
hi ozo....Excellent.....it works perfectly!   one small request...

I downloaded from our server again,and I didn't realize  there are also " " in all the field names.....and can we also omit  the csv extension from being picked up?    the sample output is looking like this....


Projecta.csv,"one", "two", "three", "four", "five"
Projecta.csv, "six","seven","eight","nine",ten"
Projecta.csv,"eleven","twelve","thirteen","fourteen","fifteen"
Projectb.csv,"one-1", "two-1", "three-1", "four-1", "five-1"
Projectb.csv,"six-1","seven-1","eight-1","nine-1","ten-1"
Projectb.csv,"eleven-1","twelve-1","thirteen-1","fourteen-1","fifteen-1"

Can we modify the script so it will be like this?

"Projecta","one", "two", "three", "four", "five"
"Projecta", "six","seven","eight","nine",ten"
"Projecta","eleven","twelve","thirteen","fourteen","fifteen"
"Projectb","one-1", "two-1", "three-1", "four-1", "five-1"
"Projectb","six-1","seven-1","eight-1","nine-1","ten-1"
"Projectb","eleven-1","twelve-1","thirteen-1","fourteen-1","fifteen-1"

Thanks OZO!
0
wilcoxonCommented:
just changing the echo line should do it:
echo `echo $f | sed -e 's%^%"%' -e 's%.csv$%"%'`,$line

Open in new window


or, to avoid the backticks, this should work as well:
echo $f,$line | sed -e 's%^%"%' -e 's%.csv,%",%'

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ozoCommented:
    echo ${f%.csv},$line
suffices to remove the .csv
Inserting the " around each column in the line could be more difficult
    echo \"${f%.csv}\",\"${line//,/\",\"}\"
may almost work  if the line doesn't do anything  too tricky
0
rleyba828Author Commented:
thanks guys....working exactly as I want now.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.