Link to home
Start Free TrialLog in
Avatar of helpfinder
helpfinderFlag for Slovakia

asked on

Edit my bash script

Hi there,
I have working script, but I would like to modify it. Maybe somebody here could help me with this.
This is script I am using to export some data from one web page:

#!/bin/bash

date=$(date +"%Y-%m-%d_%H%M%S")

wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=<phone_number>

grep -oP '/inzerat/[0-9]+' bazos.tmp | cut -d '/' -f3 | awk '!seen[$0]++' | sed 's/$/"/' | sed 's/^/"/' > cislo_inzeratu.csv
grep '<span class=nadpis>' bazos.tmp | cut -d '>' -f3 | cut -d '<' -f1 | sed 's/$/"/' | sed 's/^/"/' > nazov_inzeratu.csv
grep '<b>' bazos.tmp | cut -d 'b' -f2 | cut -d '>' -f2 | cut -d '<' -f1 | grep -o '[0-9]* €' | sed 's/$/"/' | sed 's/^/"/' > cena_inzeratu.csv
grep '</td>' bazos.tmp | grep '<br>' | cut -d '>' -f3 | cut -d '<' -f1 | sed 's/$/"/' | sed 's/^/"/' > psc_inzeratu.csv
grep '</td>' bazos.tmp | grep 'x' | cut -d '>' -f2 | cut -d '<' -f1 | sed 's/$/"/' | sed 's/^/"/' > pocet_videni.csv
grep '<span class=nadpis>' bazos.tmp | cut -d '[' -f2 | cut -d ']' -f1 | sed 's/$/"/' | sed 's/^/"/' > datum_pridania.csv

paste -d "," cislo_inzeratu.csv nazov_inzeratu.csv cena_inzeratu.csv psc_inzeratu.csv datum_pridania.csv > bazos_$date.csv

rm bazos.tmp
rm cislo_inzeratu.csv
rm nazov_inzeratu.csv
rm cena_inzeratu.csv
rm psc_inzeratu.csv
rm pocet_videni.csv
rm datum_pridania.csv

Open in new window


in WGET part I am using parameter for phone number (in the script you can see as <phone_number>)
I need to use 2 different phone numbers and now I am using 2 separate scripts, only change is phone number. Is there any way how to merge these and use only one script with all data (from both phone numbers exported into one CSV file?

Second question -  is it possible to add another column (next comma separated values) represented current date? (date when export was made)

Many thanks
Avatar of Bill Prew
Bill Prew
You could use a function in the script, and then execute it twice, once for each phone number.  This should be pretty close as a starting point.

I also added in another column with the $date value in it for each line, use a different variable if you want a different format.

#!/bin/bash

function getphone {
    wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=$1

    grep -oP '/inzerat/[0-9]+' bazos.tmp | cut -d '/' -f3 | awk '!seen[$0]++' | sed 's/$/"/' | sed 's/^/"/' > cislo_inzeratu.csv
    grep '<span class=nadpis>' bazos.tmp | cut -d '>' -f3 | cut -d '<' -f1 | sed 's/$/"/' | sed 's/^/"/' > nazov_inzeratu.csv
    grep '<b>' bazos.tmp | cut -d 'b' -f2 | cut -d '>' -f2 | cut -d '<' -f1 | grep -o '[0-9]* €' | sed 's/$/"/' | sed 's/^/"/' > cena_inzeratu.csv
    grep '</td>' bazos.tmp | grep '<br>' | cut -d '>' -f3 | cut -d '<' -f1 | sed 's/$/"/' | sed 's/^/"/' > psc_inzeratu.csv
    grep '</td>' bazos.tmp | grep 'x' | cut -d '>' -f2 | cut -d '<' -f1 | sed 's/$/"/' | sed 's/^/"/' > pocet_videni.csv
    grep '<span class=nadpis>' bazos.tmp | cut -d '[' -f2 | cut -d ']' -f1 | sed 's/$/"/' | sed 's/^/"/' > datum_pridania.csv
    grep '<span class=nadpis>' bazos.tmp | echo $date > date_field.csv

    paste -d "," cislo_inzeratu.csv nazov_inzeratu.csv cena_inzeratu.csv psc_inzeratu.csv datum_pridania.csv date_field.csv>> bazos_$date.csv

    rm bazos.tmp
    rm cislo_inzeratu.csv
    rm nazov_inzeratu.csv
    rm cena_inzeratu.csv
    rm psc_inzeratu.csv
    rm pocet_videni.csv
    rm datum_pridania.csv
    rm date_field.csv
}

date=$(date +"%Y-%m-%d_%H%M%S")

getphone 1234567890
getphone 1234567890

Open in new window

»bp
Avatar of MURUGESAN N
MURUGESAN N
Flag of India image
Here goes the code I have tested
#!/bin/bash
#a)
#   Always use full path for any commands.
#Replace
#	date
#	..
#	wget..
#	..
#	grep..cut..awk..sed
#	..
#	paste
#	..
#	rm
#With:
#	/bin/date
#	..
#	/usr/bin/wget..
#	..
#	/bin/grep../usr/bin/cut../bin/awk../bin/sed
#	..
#	/usr/bin/paste
#	..
#	/bin/rm


#b)
#Variable names need not to be name of any existing command.
#Replace:
#	date=$(/bin/date +"%Y_%m_%d_%H%M%S")
#	..
#	..bazos_$date.csv
#With:
#	CURRENT_DATE=$(/bin/date +"%Y-%m-%d_%H%M%S")
#	..
#	.."bazos_$CURRENT_DATE.csv"



#c)
#Replace:
#	/usr/bin/wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=<phone_number>
#With:
#	/usr/bin/wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=$1
#$1 refer the first parameter passed to this script.

#d)
#Use wget only if one parameter passed to this script
#Replace:
#	/usr/bin/wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=$1
#With:
#	if test 0 -eq $#
#	then
#		echo "Usage:"
#		echo "$0 Valid_Phone_Number"
#		echo "Example:"
#		echo "$0 1234567890"
#	else
#		/usr/bin/wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=$1
#	fi

#e)
#Replace:
#	/bin/sed..stream1 | /bin/sed..stream2
#With:
#	/bin/sed "stream1;stream2"

#f)
#Before removing the file check existence of that file
#Example:
#	if test -f bazos.tmp
#	then
#		echo /bin/rm -f bazos.tmp
#		/bin/rm -f bazos.tmp
#	else
#		echo "bazos.tmp No such file"
#	fi

#g)
#Instead of removing files sequentially, we can remove using for loop
#Example:
#	for RemoveFiles in 1.txt 2.txt 3.txt
#	do
#		if test -f "$RemoveFiles"
#		then
#			echo "/bin/rm $RemoveFiles"
#			/bin/rm "$RemoveFiles"
#		fi
#	done

#h)
#To display output in terminal as well as to output file
#Replace:
#	/usr/bin/paste -d "," cislo_inzeratu.csv nazov_inzeratu.csv cena_inzeratu.csv psc_inzeratu.csv datum_pridania.csv > "bazos_$CURRENT_DATE.csv"
#With:
#	/usr/bin/paste -d "," cislo_inzeratu.csv nazov_inzeratu.csv cena_inzeratu.csv psc_inzeratu.csv datum_pridania.csv | /usr/bin/tee -a "bazos_$CURRENT_DATE.csv"

#i)
#I cannot read that page in readable format.
#Hence requesting author to reduce performance for the script.
#Replace
#	/bin/grep../usr/bin/cut../usr/bin/cut../usr/bin/cut../bin/grep
#With:
#	/bin/grep../bin/awk

CURRENT_DATE=$(/bin/date +"%Y-%m-%d_%H%M%S")
if test 0 -eq $#
then
	echo "Usage:"
	echo "$0 FirstParameter needs to be a valid phone number"
	echo "Example:"
	echo "$0 1234567890"
else
	/usr/bin/wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=$1
	/bin/grep -oP '/inzerat/[0-9]+' bazos.tmp | /usr/bin/cut -d '/' -f3 | /bin/awk '!seen[$0]++' | /bin/sed 's/$/"/;s/^/"/' > cislo_inzeratu.csv
	/bin/grep '<span class=nadpis>' bazos.tmp | /usr/bin/cut -d '>' -f3 | /usr/bin/cut -d '<' -f1 | /bin/sed 's/$/"/;s/^/"/' > nazov_inzeratu.csv
	/bin/grep '<b>' bazos.tmp | /usr/bin/cut -d 'b' -f2 | /usr/bin/cut -d '>' -f2 | /usr/bin/cut -d '<' -f1 | /bin/grep -o '[0-9]* €' | /bin/sed 's/$/"/;s/^/"/' > cena_inzeratu.csv
	/bin/grep '</td>' bazos.tmp | /bin/grep '<br>' | /usr/bin/cut -d '>' -f3 | /usr/bin/cut -d '<' -f1 | /bin/sed 's/$/"/;s/^/"/' > psc_inzeratu.csv
	/bin/grep '</td>' bazos.tmp | /bin/grep 'x' | /usr/bin/cut -d '>' -f2 | /usr/bin/cut -d '<' -f1 | /bin/sed 's/$/"/;s/^/"/' > pocet_videni.csv
	/bin/grep '<span class=nadpis>' bazos.tmp | /usr/bin/cut -d '[' -f2 | /usr/bin/cut -d ']' -f1 | /bin/sed 's/$/"/;s/^/"/' > datum_pridania.csv
	/usr/bin/paste -d "," cislo_inzeratu.csv nazov_inzeratu.csv cena_inzeratu.csv psc_inzeratu.csv datum_pridania.csv | /usr/bin/tee -a "bazos_$CURRENT_DATE.csv"
	for RemoveFiles in bazos.tmp \
	cislo_inzeratu.csv \
	nazov_inzeratu.csv \
	cena_inzeratu.csv \
	psc_inzeratu.csv \
	pocet_videni.csv \
	datum_pridania.csv
	do
		if test -f "$RemoveFiles"
		then
			echo "/bin/rm $RemoveFiles"
			/bin/rm "$RemoveFiles"
		fi
	done
	NumOfLine=''`/usr/bin/wc -l "bazos_$CURRENT_DATE.csv" | /bin/awk '{ print $1}'`''
	if test 0 -eq $NumOfLine
	then
		echo "For $1"
		echo "No one has user reviews. The user does not have any ads."
		echo "Hence removing emtpy output file bazos_$CURRENT_DATE.csv"
		echo "/bin/rm bazos_$CURRENT_DATE.csv"
		/bin/rm "bazos_$CURRENT_DATE.csv"
	else
		echo "Created following output file:"
		echo "bazos_$CURRENT_DATE.csv"
	fi
fi

Open in new window

Current output:
$ ./29021227_Query.sh
Usage:
./29021227_Query.sh FirstParameter needs to be a valid phone number
Example:
./29021227_Query.sh 1234567890
$ ./29021227_Query.sh 1234567890
--2017-05-09 15:43:24--  https://www.bazos.sk/hodnotenie.php?telefon=1234567890
Resolving www.bazos.sk... 88.86.119.241
Connecting to www.bazos.sk|88.86.119.241|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'bazos.tmp'

bazos.tmp                       [  <=>                                    ]  83.51K   265KB/s    in 0.3s

2017-05-09 15:43:25 (265 KB/s) - 'bazos.tmp' saved [85518]

"73200149","predam",,"830 00","8.5. 2017"
"73194420","Skoda 100",,"052 01","8.5. 2017"
"73184751","KUPIM STENIATKO BOLONSKEHO PSIKA",,"040 01","8.5. 2017"
"73171842","Skateboard",,"953 01","7.5. 2017"
"73171404","Cestný bycikel FAVORIT",,"953 01","7.5. 2017"
"73143229","landras.",,"985 01","6.5. 2017"
"73142989","dovoz zdarma 25-30kg za dobru cenu odber hned.",,"082 53","6.5. 2017"
"73142711","skvelá ponukám odstavčatá 25-30kg. mäsove typ.",,"044 13","6.5. 2017"
"73089978","VFU Brno, 1.ročnìk",,"966 22","4.5. 2017"
"73079507","Výzdoba na svadobné auto",,"949 01","4.5. 2017"
"73044108","Zahradne sedenie",,"900 61","3.5. 2017"
"72949986","mostová píla - denver skema logic cnc",,"040 01","30.4. 2017"
"72949816","Mostová píla -Emmedue discovery",,"040 01","30.4. 2017"
"72949643","Píla na kameň- cobalm ideal 15",,"040 01","30.4. 2017"
"72933937","spatne zrkadlo",,"941 01","30.4. 2017"
"72663123","Sapphire HD 7870 2GB GDDR5 XT WITH BOOST",,"974 11","29.4. 2017"
"72822954","Sáčky na materské mlieko AVENT a Moringa olejová",,"901 01","26.4. 2017"
"72818155","Dojčenská výživa Nutrilon ProExpert",,"901 01","26.4. 2017"
"72816487","Elektrický parný sterilizátor AVENT set 3v1",,"901 01","26.4. 2017"
"72815447","Elektrický ohrievač fliaš AVENT",,"901 01","26.4. 2017"
"72803858","Hygienický kôš na použité plienky MUNCHKIN",,"901 01","26.4. 2017"
"72802103","Hojdacie a vibracne kresielko Bright Starts",,"901 01","26.4. 2017"
"72789398","Karbidky Wehrmacht druhá svetová WW2",,"020 01","25.4. 2017"
"72788449","Zahradne sedenie",,"900 61","25.4. 2017"
"72785863","Predám tenisky Adidas velkosť 38.5",,"909 01","25.4. 2017"
"72785666","Predám originál Nike air max velkosť 38.5",,"909 01","25.4. 2017"
"72778074","Čierny dámsky trench kabát",,"031 01","25.4. 2017"
"72777523","Dievčenská/ dámska vzorovaná sukňa",,"031 01","25.4. 2017"
"72776806","Ružové dámske sako SX/S",,"031 01","25.4. 2017"
"72776546","Čierne dámske sako S",,"031 01","25.4. 2017"
"72770174","Prosim kto daruje",,"917 01","25.4. 2017"
"72707637","Longboard",,"979 01","23.4. 2017"
"72699055","Kto daruje hniezdo do postielky",,"076 17","23.4. 2017"
"72678762","7 čiapočiek pre dievčatko (0-6 mesiacov)",,"010 01","22.4. 2017"
"72678711","25 kusov oblečenia pre dievčatko: veľkosť 80 - 1rok",,"010 01","22.4. 2017"
"72678647","25 kusov oblečenia pre dievčatko (0-6 mesiacov)",,"010 01","22.4. 2017"
"72678510","40 kusov oblečenia pre chlapčeka (0-6 mesiacov)",,"010 01","22.4. 2017"
"72678391","14 kusov - body pre dievčatko (0-6 mesiacov)",,"010 01","22.4. 2017"
"72659352","Kúpim",,"017 01","22.4. 2017"
"72652113","Balík oblečenia",,"048 01","22.4. 2017"
"72651229","Svadobná vesta 48-52 - 52",,"018 01","22.4. 2017"
"72651142","krémové svadobné šaty 36",,"018 01","22.4. 2017"
"72645736","MINI balík dámskeho oblečenia",,"010 01","21.4. 2017"
"72645708","MINI balík dámskeho oblečenia",,"010 01","21.4. 2017"
"72645556","MINI balík dámskeho oblečenia - veľ. M",,"010 01","21.4. 2017"
"72645486","MINI balík dámskeho oblečenia - veľ. S",,"010 01","21.4. 2017"
"72645318","MINI balík dámskeho oblečenia (S/M)",,"010 01","21.4. 2017"
"72632038","Damsky lyziarsky set- nepouzivany vel. 40",,"040 22","21.4. 2017"
/bin/rm bazos.tmp
/bin/rm cislo_inzeratu.csv
/bin/rm nazov_inzeratu.csv
/bin/rm cena_inzeratu.csv
/bin/rm psc_inzeratu.csv
/bin/rm pocet_videni.csv
/bin/rm datum_pridania.csv
Created following output file:
bazos_2017-05-09_154324.csv

Open in new window







#In Bill_Prew.sh
#a)
#Replace:
#      getphone <phone_number_1>
#      getphone <phone_number_2>
#With:
#      getphone $1
#      getphone $2

#b)
#Replace:
#      paste -d "," cislo_inzeratu.csv nazov_inzeratu.csv cena_inzeratu.csv psc_inzeratu.csv datum_pridania.csv date_field>> bazos_$date.csv
#      ..
#      rm date_field
#With:
#      /usr/bin/paste -d "," cislo_inzeratu.csv nazov_inzeratu.csv cena_inzeratu.csv psc_inzeratu.csv datum_pridania.csv date_field.csv>> bazos_$date.csv
#      ..
#      if test -f date_field.csv
#      then
#            echo "/bin/rm date_field.csv"
#            /bin/rm date_field.csv
#      else
#            echo "date_field.csv No such file"
#      fi
Avatar of helpfinder
helpfinder
Flag of Slovakia image

ASKER

Many thanks for your answers,
@Bill Prew - unfortunately I am geting error
-bash: ./test.sh: /bin/bash^M: bad interpreter: No such file or directory

@Murugesan Nagarajan - script with your comments seemed promising, I removed all your comments and tried to run it. Unfortunatelly also here I received error:
./test2.sh: line 18: syntax error near unexpected token `cislo_inzeratu.csv'
'/test2.sh: line 18: `  cislo_inzeratu.csv \

not sure what is wrong here, if obviously you was able to get result.

btw I can not see in your posted result price (should be taken from "cena_inzeratu.csv"), but there is only empty column (,,) visible

Second thing, how to put second phone number?

Thanks
Avatar of Bill Prew
Bill Prew
The ^M makes me think that you have Windows style end-of-line on a unix machine.  Double check the format of the script file and ensure that it is in unix format with just line feeds at the end of lines, no carriage return.

~bp
Avatar of Bill Prew
Bill Prew
The only syntax error I did just notice was that the last two references in the paste and rm commands to "date_field" should be "date_field.csv".

And you will of course have to fill in the telephone numbers with real numbers at the bottom.

getphone <phone_number_1>
getphone <phone_number_2>

~bp
Avatar of helpfinder
helpfinder
Flag of Slovakia image

ASKER

Thank you Bill Prew,
now I am able to run the script, thanks. But still I have to think about it any maybe somehow fixt last details
This is how export looks like:

"73245072","Vymenim xbox one za ps3/xbox 360/ps vitu",,"010 01","9.5. 2017",2017-05-09_215223
"73244961","Playstation 3 super slim s 8 hrami",,"010 01","9.5. 2017",
"73237789","Hra na xbox 360 under defeat hd deluxe edition",,"010 01","9.5. 2017",
"73237770","Hra na xbox 360 xcom enemy unknown",,"010 01","9.5. 2017",
"73237749","Hra na xbox 360 project gotham racing4",,"010 01","9.5. 2017",
"73237729","Hra na xbox 360 driver san francisco",,"010 01","9.5. 2017",
"73237706","Hra na xbox 360 dead island goty",,"010 01","9.5. 2017",
"73237621","Hra na xbox 360 ninja gaiden 2",,"010 01","9.5. 2017",
"73237557","Ps4 hra kill zone shadow fall",,"010 01","9.5. 2017",
"73227539","Playstation 3 super slim 500gb so 6 hrami",,"010 01","9.5. 2017",
"73223176","Xbox classic crystal limited edition",,"010 01","9.5. 2017",
"73223137","Mikrofony,kamera,hry-playstation2/3",,"010 01","9.5. 2017",
"73065779","Ps2 hra king kong",,"010 01","4.5. 2017",
"73065762","Ps2 hra die hard",,"010 01","4.5. 2017",
"73065749","Ps2 hra scarface",,"010 01","4.5. 2017",
"73065728","Ps2 hra the godfather",,"010 01","4.5. 2017",
"73065699","Ps2 hra tomb raider anniversary",,"010 01","4.5. 2017",
"73065680","Ps2 hra tomb raider the angel of darkness",,"010 01","4.5. 2017",
"73065656","Ps2 hra x-men 2",,"010 01","4.5. 2017",
"73065632","Ps2 hra 007 quantum of solace",,"010 01","4.5. 2017",
"73065617","Ps2 hra nba 2k11",,"010 01","4.5. 2017",
"73065601","Ps2 hra fifa street 2",,"010 01","4.5. 2017",
"73065582","Ps2 hra fight night round 3",,"010 01","4.5. 2017",
"73065568","Ps2 hra need for speed undercover cz",,"010 01","4.5. 2017",
"73065549","Ps2 hra gran tourismo 4",,"010 01","4.5. 2017",
"73065529","Ps2 hra colin mcrae rally 2005",,"010 01","4.5. 2017",
"73065500","Ps2 hra midnight club 3",,"010 01","4.5. 2017",
"73065477","Ps2 hra wrc",,"010 01","4.5. 2017",
"73065461","Ps2 hra burnout revenge",,"010 01","4.5. 2017",
"73065440","Ps2 hra burnout 3",,"010 01","4.5. 2017",
"73065415","Ps2 hra heracles",,"010 01","4.5. 2017",
"73065387","Ps2 hra ratatouille",,"010 01","4.5. 2017",
"73065359","Ps2 hra legend of kay",,"010 01","4.5. 2017",
"73065335","Ps2 hra jak x",,"010 01","4.5. 2017",
"73065308","Ps2 hra jak 2",,"010 01","4.5. 2017",
"73065275","Ps2 hra jak and daxter",,"010 01","4.5. 2017",
"72392465","Eye toy kamera na ps2",,"010 01","14.4. 2017",
"73248848","steep na ps4",,"010 08","9.5. 2017",2017-05-09_215223
"73234976","dying light the following na ps4",,"010 08","9.5. 2017",
"73234264","star wars battlefront na ps4",,"010 08","9.5. 2017",
"73234145","need for speed na ps4",,"010 08","9.5. 2017",
"73234067","xbox one s 1tb+ 6 hier v zaruke",,"010 08","9.5. 2017",
"73233956","mafia 2 cz+mapa na ps3",,"010 08","9.5. 2017",
"73233924","metal gear solid 5 the phantom pain na ps3",,"010 08","9.5. 2017",
"73233888","metal gear solid 5 ground zeroes na ps3",,"010 08","9.5. 2017",
"73233854","prince of persia trilogy na ps3",,"010 08","9.5. 2017",
"73233808","le tour de france season 2015 na ps3",,"010 08","9.5. 2017",
"73233698","diablo 3 na ps3",,"010 08","9.5. 2017",
"73233602","skyrim+mapa na ps3",,"010 08","9.5. 2017",
"73233584","watch dogs cz na ps3",,"010 08","9.5. 2017",
"73233556","sniper ghost warrior na ps3",,"010 08","9.5. 2017",
"73233414","need for speed shift2 unleashed na ps3",,"010 08","9.5. 2017",
"73233378","need for speed hot pursuit na ps3",,"010 08","9.5. 2017",
"73233345","gta 4+mapa na ps3",,"010 08","9.5. 2017",
"73233271","need for speed most wanted limited edition na ps3",,"010 08","9.5. 2017",
"73232997","f1 2012 na ps3",,"010 08","9.5. 2017",
"73232969","dishonored cz na ps3",,"010 08","9.5. 2017",
"73232945","midnight club los angeles na ps3",,"010 08","9.5. 2017",
"73232909","need for speed the run na ps3",,"010 08","9.5. 2017",
"73232861","harry potter and the deathly hallows part2 na ps3",,"010 08","9.5. 2017",
"73232830","terminator salvation na ps3",,"010 08","9.5. 2017",
"73232787","angry birds star wars na ps3",,"010 08","9.5. 2017",
"73232692","b&w preference ps6",,"010 08","9.5. 2017",
"73232605","peterky",,"010 08","9.5. 2017",
"73232555","ovladac na xbox one",,"010 08","9.5. 2017",
"73232453","chatpad na xbox one",,"010 08","9.5. 2017",
"73232379","wrc 6 na xbox one",,"010 08","9.5. 2017",
"73232113","rise of the tomb raider na xbox one",,"010 08","9.5. 2017",
"73232037","gears of war 4 na xbox one",,"010 08","9.5. 2017",
"73231949","the evil within limited edition na xbox one",,"010 08","9.5. 2017",
"73231137","marantz cd player cd 6002 aj vymenim",,"010 08","9.5. 2017",
"73231086","spiderman shatteres dimensions na xbox 360",,"010 08","9.5. 2017",
"73231075","ultimate marvel vs capcom 3 na xbox 360",,"010 08","9.5. 2017",
"73231056","banjo kazooie na xbox 360",,"010 08","9.5. 2017",
"73231033","lips i love 80s na xbox 360",,"010 08","9.5. 2017",
"73231014","halo reach a fable 3 double pack na xbox 360",,"010 08","9.5. 2017",
"73230992","silent hill downpour na xbox 360",,"010 08","9.5. 2017",
"73230964","dragon age inquisition na xbox 360",,"010 08","9.5. 2017",
"73230901","resident evil origins collection na xbox one",,"010 08","9.5. 2017",
"73230399","ps3 fat 80gb v top stave",,"010 08","9.5. 2017",
"73228894","vintage stereo receiver luxman r-1030 aj vymena",,"010 08","9.5. 2017",

Open in new window


That means I have both phone numbers exported in one CSV, that´s great!
Only I am missing is Price (cena_inzeratu.csv) there is blank column (,,)
and also export date (last column) is present only twice in the output (probably once per one each phone number), desired would be each row contains also export date
Avatar of Bill Prew
Bill Prew
What phone numbers did you use for that test?

~bp
Avatar of helpfinder
helpfinder
Flag of Slovakia image

ASKER

and also to the last exported column date of the export shouild be the same as for "datum pridania" (insertion date) which seems to be d:M:yyyy and only date without time

numbers I am using are:
getphone 0940133715
getphone 0903443736
ASKER CERTIFIED SOLUTION
Avatar of Bill Prew
Bill Prew
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of MURUGESAN N
MURUGESAN N
Flag of India image
Here goes the code to convert unix2dos and dos2unix
#!/bin/bash
ScriptName=./test1.sh
/usr/bin/file "$ScriptName" | /bin/egrep -w "CRLF" | /bin/egrep -v "^$" >/dev/null 2>&1
Ret=$?
if test 0 -eq $Ret
then
	echo "Following file $ScriptName in Windows format:"
	echo -e "\n$ /usr/bin/file $ScriptName"
	/usr/bin/file "$ScriptName" | /bin/egrep -w "CRLF" | /bin/egrep -v "^$"
	echo -e "\n"
	echo "type yes or no (case insensitive) to continue or press Ctrl C"
	while [ 1 ]
	do
		read ConvertFile
		echo "$ConvertFile" | /bin/egrep -w -i "^yes$|^no$" >/dev/null 2>&1
		Ret=$?
		if test 0 -eq $Ret
		then
			echo "$ConvertFile" | /bin/egrep -w -i "^no$" >/dev/null 2>&1
			Ret=$?
			if test 0 -ne $Ret
			then
				echo "Converting file $ScriptName to unix format"
				echo -e "$ /bin/sed -i \"s/\\\r//g;\" $ScriptName"
				/bin/sed -i "s/\r//g;" "$ScriptName"
				echo -e "\n"
			else
				echo "Not converting file $ScriptName to unix format"
			fi
			break
		fi
		echo "type yes(case insensitive) to continue or press Ctrl C"
	done
fi

Open in new window


Instead of using the above script you can use (not using since it may or may not be available at all platforms => depends on administrator installer):
Converting unix 2 dos format
/usr/bin/unix2dos scriptname.sh

Open in new window

Converting dos 2 unix 2 format
/usr/bin/dos2unix scriptname.sh

Open in new window

Avatar of MURUGESAN N
MURUGESAN N
Flag of India image
>> getphone 0940133715
>> getphone 0903443736
Instead of using hard coded values, better use parameter passed inside the script
Example codes:
#Using First parameter
getphone $1
#Using Second parameter
getphone $2
#Using tenth parameter
getphone ${10}
#Using twentieth parameter
getphone ${20}

Open in new window

Avatar of MURUGESAN N
MURUGESAN N
Flag of India image
@helpfinder
Here goes updated code 
#!/bin/bash
unset -f Usage
Usage ()
{
	echo "Usage:"
	echo "$0 parameter or parameters needs to be a ten digit phone number"
	echo "Example:"
	echo "$0 0940133715"
	echo "OR"
	echo "$0 0903443736"
	echo "OR"
	echo "$0 0940133715 0903443736"
	Ret=1
	return $Ret
}
unset -f Retrieve_Phone_Data
Retrieve_Phone_Data ()
{
	if test 0 -eq $#
	then
		Usage
		Ret=$?
		return $Ret
	fi
	echo "$@" | /bin/egrep "^[0-9].*[0-9]$" >/dev/null 2>&1
	Ret=$?
	if test 0 -ne $Ret
	then
		Usage
		Ret=$?
		return $Ret
	fi
	for EachParamter in $@
	do
		NumOfDigits=''`echo -n "$EachParamter" | /usr/bin/wc -c`''
		if test 10 -ne $NumOfDigits
		then
			Usage
			Ret=$?
			return $Ret
		fi
	done
	CURRENT_DATE=$(/bin/date +"%Y-%m-%d_%H%M%S")
	for EachParamter in $@
	do
		echo "For $EachParamter:"
		echo "/usr/bin/wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=$EachParamter"
		/usr/bin/wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=$EachParamter
		/bin/grep -oP '/inzerat/[0-9]+' bazos.tmp | /usr/bin/cut -d '/' -f3 | /bin/awk '!seen[$0]++' | /bin/sed 's/$/"/;s/^/"/' > cislo_inzeratu.csv
		/bin/grep '<span class=nadpis>' bazos.tmp | /usr/bin/cut -d '>' -f3 | /usr/bin/cut -d '<' -f1 | /bin/sed 's/$/"/;s/^/"/' > nazov_inzeratu.csv
		/bin/grep '<b>' bazos.tmp | /usr/bin/cut -d 'b' -f2 | /usr/bin/cut -d '>' -f2 | /usr/bin/cut -d '<' -f1 | /bin/grep -o '[0-9]* €' | /bin/sed 's/$/"/;s/^/"/' > cena_inzeratu.csv
		/bin/grep '</td>' bazos.tmp | /bin/grep '<br>' | /usr/bin/cut -d '>' -f3 | /usr/bin/cut -d '<' -f1 | /bin/sed 's/$/"/;s/^/"/' > psc_inzeratu.csv
		/bin/grep '</td>' bazos.tmp | /bin/grep 'x' | /usr/bin/cut -d '>' -f2 | /usr/bin/cut -d '<' -f1 | /bin/sed 's/$/"/;s/^/"/' > pocet_videni.csv
		/bin/grep '<span class=nadpis>' bazos.tmp | /usr/bin/cut -d '[' -f2 | /usr/bin/cut -d ']' -f1 | /bin/sed 's/$/"/;s/^/"/' > datum_pridania.csv

		#You can use the following to display output in terminal
		#/usr/bin/paste -d "," cislo_inzeratu.csv nazov_inzeratu.csv cena_inzeratu.csv psc_inzeratu.csv datum_pridania.csv | /usr/bin/tee -a "bazos_$CURRENT_DATE.csv"
		#or
		/usr/bin/paste -d "," cislo_inzeratu.csv nazov_inzeratu.csv cena_inzeratu.csv psc_inzeratu.csv datum_pridania.csv >> "bazos_$CURRENT_DATE.csv"

		for RemoveFiles in bazos.tmp \
		cislo_inzeratu.csv \
		nazov_inzeratu.csv \
		cena_inzeratu.csv \
		psc_inzeratu.csv \
		pocet_videni.csv \
		datum_pridania.csv
		do
			if test -f "$RemoveFiles"
			then
				echo "/bin/rm $RemoveFiles"
				/bin/rm "$RemoveFiles"
			fi
		done
	done
	NumOfLine=''`/usr/bin/wc -l "bazos_$CURRENT_DATE.csv" | /bin/awk '{ print $1}'`''
	if test 0 -eq $NumOfLine
	then
		echo "For $EachParamter"
		echo "No one has user reviews. The user does not have any ads."
		echo "Hence removing emtpy output file bazos_$CURRENT_DATE.csv"
		echo "/bin/rm bazos_$CURRENT_DATE.csv"
		/bin/rm "bazos_$CURRENT_DATE.csv"
	else
		echo "Created following output file:"
		echo "bazos_$CURRENT_DATE.csv"
	fi
}
Retrieve_Phone_Data $@

Open in new window

Avatar of MURUGESAN N
MURUGESAN N
Flag of India image
Scriput output:
$ /usr/bin/file  ./29021227_Query.sh | /bin/egrep "CRLF" | /bin/egrep -v "^$"
./29021227_Query.sh: Bourne-Again shell script, Non-ISO extended-ASCII text executable, with CRLF line terminators
$ Ret=$?
$ echo $Ret
$ echo Since CRLF line terminators present I am converting to Linux format
Since CRLF line terminators present I am converting to Linux format
$ /bin/sed -i "s/\r//g;" ./29021227_Query.sh
$ /usr/bin/file  ./29021227_Query.sh | /bin/egrep "CRLF" | /bin/egrep -v "^$"
$ Ret=$?
$ echo $Ret
1
$ ./29021227_Query.sh 0940133715 0903443736
For 0940133715:
/usr/bin/wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=0940133715
--2017-05-10 09:20:02--  https://www.bazos.sk/hodnotenie.php?telefon=0940133715
Resolving www.bazos.sk... 88.86.119.241
Connecting to www.bazos.sk|88.86.119.241|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'bazos.tmp'

bazos.tmp                       [  <=>                                    ]  62.06K   132KB/s    in 0.5s

2017-05-10 09:20:03 (132 KB/s) - 'bazos.tmp' saved [63549]

/bin/rm bazos.tmp
/bin/rm cislo_inzeratu.csv
/bin/rm nazov_inzeratu.csv
/bin/rm cena_inzeratu.csv
/bin/rm psc_inzeratu.csv
/bin/rm pocet_videni.csv
/bin/rm datum_pridania.csv
For 0903443736:
/usr/bin/wget -O bazos.tmp https://www.bazos.sk/hodnotenie.php?telefon=0903443736
--2017-05-10 09:20:05--  https://www.bazos.sk/hodnotenie.php?telefon=0903443736
Resolving www.bazos.sk... 88.86.119.241
Connecting to www.bazos.sk|88.86.119.241|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'bazos.tmp'

bazos.tmp                       [   <=>                                   ]  71.99K   112KB/s    in 0.6s

2017-05-10 09:20:07 (112 KB/s) - 'bazos.tmp' saved [73722]

/bin/rm bazos.tmp
/bin/rm cislo_inzeratu.csv
/bin/rm nazov_inzeratu.csv
/bin/rm cena_inzeratu.csv
/bin/rm psc_inzeratu.csv
/bin/rm pocet_videni.csv
/bin/rm datum_pridania.csv
Created following output file:
bazos_2017-05-10_092002.csv

Open in new window



/bin/cat bazos_2017-05-10_092002.csv
"73245072","Vymenim xbox one za ps3/xbox 360/ps vitu",,"010 01","9.5. 2017"
"73244961","Playstation 3 super slim s 8 hrami",,"010 01","9.5. 2017"
"73237789","Hra na xbox 360 under defeat hd deluxe edition",,"010 01","9.5. 2017"
"73237770","Hra na xbox 360 xcom enemy unknown",,"010 01","9.5. 2017"
"73237749","Hra na xbox 360 project gotham racing4",,"010 01","9.5. 2017"
"73237729","Hra na xbox 360 driver san francisco",,"010 01","9.5. 2017"
"73237706","Hra na xbox 360 dead island goty",,"010 01","9.5. 2017"
"73237621","Hra na xbox 360 ninja gaiden 2",,"010 01","9.5. 2017"
"73237557","Ps4 hra kill zone shadow fall",,"010 01","9.5. 2017"
"73227539","Playstation 3 super slim 500gb so 6 hrami",,"010 01","9.5. 2017"
"73223176","Xbox classic crystal limited edition",,"010 01","9.5. 2017"
"73223137","Mikrofony,kamera,hry-playstation2/3",,"010 01","9.5. 2017"
"73065779","Ps2 hra king kong",,"010 01","4.5. 2017"
"73065762","Ps2 hra die hard",,"010 01","4.5. 2017"
"73065749","Ps2 hra scarface",,"010 01","4.5. 2017"
"73065728","Ps2 hra the godfather",,"010 01","4.5. 2017"
"73065699","Ps2 hra tomb raider anniversary",,"010 01","4.5. 2017"
"73065680","Ps2 hra tomb raider the angel of darkness",,"010 01","4.5. 2017"
"73065656","Ps2 hra x-men 2",,"010 01","4.5. 2017"
"73065632","Ps2 hra 007 quantum of solace",,"010 01","4.5. 2017"
"73065617","Ps2 hra nba 2k11",,"010 01","4.5. 2017"
"73065601","Ps2 hra fifa street 2",,"010 01","4.5. 2017"
"73065582","Ps2 hra fight night round 3",,"010 01","4.5. 2017"
"73065568","Ps2 hra need for speed undercover cz",,"010 01","4.5. 2017"
"73065549","Ps2 hra gran tourismo 4",,"010 01","4.5. 2017"
"73065529","Ps2 hra colin mcrae rally 2005",,"010 01","4.5. 2017"
"73065500","Ps2 hra midnight club 3",,"010 01","4.5. 2017"
"73065477","Ps2 hra wrc",,"010 01","4.5. 2017"
"73065461","Ps2 hra burnout revenge",,"010 01","4.5. 2017"
"73065440","Ps2 hra burnout 3",,"010 01","4.5. 2017"
"73065415","Ps2 hra heracles",,"010 01","4.5. 2017"
"73065387","Ps2 hra ratatouille",,"010 01","4.5. 2017"
"73065359","Ps2 hra legend of kay",,"010 01","4.5. 2017"
"73065335","Ps2 hra jak x",,"010 01","4.5. 2017"
"73065308","Ps2 hra jak 2",,"010 01","4.5. 2017"
"73065275","Ps2 hra jak and daxter",,"010 01","4.5. 2017"
"72392465","Eye toy kamera na ps2",,"010 01","14.4. 2017"
"73248848","steep na ps4",,"010 08","9.5. 2017"
"73234264","star wars battlefront na ps4",,"010 08","9.5. 2017"
"73234145","need for speed na ps4",,"010 08","9.5. 2017"
"73234067","xbox one s 1tb+ 6 hier v zaruke",,"010 08","9.5. 2017"
"73233956","mafia 2 cz+mapa na ps3",,"010 08","9.5. 2017"
"73233924","metal gear solid 5 the phantom pain na ps3",,"010 08","9.5. 2017"
"73233888","metal gear solid 5 ground zeroes na ps3",,"010 08","9.5. 2017"
"73233854","prince of persia trilogy na ps3",,"010 08","9.5. 2017"
"73233808","le tour de france season 2015 na ps3",,"010 08","9.5. 2017"
"73233698","diablo 3 na ps3",,"010 08","9.5. 2017"
"73233602","skyrim+mapa na ps3",,"010 08","9.5. 2017"
"73233584","watch dogs cz na ps3",,"010 08","9.5. 2017"
"73233556","sniper ghost warrior na ps3",,"010 08","9.5. 2017"
"73233414","need for speed shift2 unleashed na ps3",,"010 08","9.5. 2017"
"73233378","need for speed hot pursuit na ps3",,"010 08","9.5. 2017"
"73233345","gta 4+mapa na ps3",,"010 08","9.5. 2017"
"73233271","need for speed most wanted limited edition na ps3",,"010 08","9.5. 2017"
"73232997","f1 2012 na ps3",,"010 08","9.5. 2017"
"73232969","dishonored cz na ps3",,"010 08","9.5. 2017"
"73232945","midnight club los angeles na ps3",,"010 08","9.5. 2017"
"73232909","need for speed the run na ps3",,"010 08","9.5. 2017"
"73232861","harry potter and the deathly hallows part2 na ps3",,"010 08","9.5. 2017"
"73232830","terminator salvation na ps3",,"010 08","9.5. 2017"
"73232787","angry birds star wars na ps3",,"010 08","9.5. 2017"
"73232692","b&w preference ps6",,"010 08","9.5. 2017"
"73232605","peterky",,"010 08","9.5. 2017"
"73232555","ovladac na xbox one",,"010 08","9.5. 2017"
"73232453","chatpad na xbox one",,"010 08","9.5. 2017"
"73232379","wrc 6 na xbox one",,"010 08","9.5. 2017"
"73232113","rise of the tomb raider na xbox one",,"010 08","9.5. 2017"
"73232037","gears of war 4 na xbox one",,"010 08","9.5. 2017"
"73231949","the evil within limited edition na xbox one",,"010 08","9.5. 2017"
"73231137","marantz cd player cd 6002 aj vymenim",,"010 08","9.5. 2017"
"73231086","spiderman shatteres dimensions na xbox 360",,"010 08","9.5. 2017"
"73231075","ultimate marvel vs capcom 3 na xbox 360",,"010 08","9.5. 2017"
"73231056","banjo kazooie na xbox 360",,"010 08","9.5. 2017"
"73231033","lips i love 80s na xbox 360",,"010 08","9.5. 2017"
"73231014","halo reach a fable 3 double pack na xbox 360",,"010 08","9.5. 2017"
"73230992","silent hill downpour na xbox 360",,"010 08","9.5. 2017"
"73230964","dragon age inquisition na xbox 360",,"010 08","9.5. 2017"
"73230901","resident evil origins collection na xbox one",,"010 08","9.5. 2017"
"73230399","ps3 fat 80gb v top stave",,"010 08","9.5. 2017"
"73228894","vintage stereo receiver luxman r-1030 aj vymena",,"010 08","9.5. 2017"

Open in new window

SOLUTION
Avatar of MURUGESAN N
MURUGESAN N
Flag of India image
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of helpfinder
helpfinder
Flag of Slovakia image

ASKER

I still have € instead of € in my export result, but never mind. I can still do Replace in excel or so. Probably caused of character set in my Raspbian or something like that.
Basically the script from Bill Prew works perfectly - many thanks.

Many thanks also to you Murugesan Nagarajan, you definitely spent many time with this, really appreciate.
Avatar of MURUGESAN N
MURUGESAN N
Flag of India image
echo "Most welcome" | /usr/bin/wc