asked on

Powershell: download files from an url

Hello experts,

The following powershell script allows me to download files from an specific gateway provided by one of my editors:

#login information
$user= ""
#$user= Read-Host "Please enter a value"
#Password
$pass= ""
#$pass= read-host 'Enter Password' -AsSecureString
#ID of service
$IDapi= Read-Host "Please enter service number"
#Setup hmtl file
$api_html = "${IDapi}.html"
#Create folder in which will be located files to download
md ${IDapi}

#Reformat header
$pair = "${user}:${pass}"
$bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
$base64 = [System.Convert]::ToBase64String($bytes)
$basicAuthValue = "Basic $base64"
$headers = @{ Authorization = $basicAuthValue }

#Setup url
$url_setup = "https://url/getLog?nb=${IDapi}"

#Display url
$url_setup

#Recover html file (file which contains the various log listing
Invoke-WebRequest -Uri $url_setup -Headers $headers -OutFile .\$IDapi\$api_html

#Cleaning html file and build txt file
findstr.exe "filename" .\$IDapi\$api_html > .\$IDapi\${IDapi}_log.txt
$content = Get-Content ".\$IDapi\${IDapi}_log.txt" | foreach { $_ -replace '" />','' } 
Set-Content -Path ".\$IDapi\${IDapi}_log.txt" -Value $content
$content = Get-Content ".\$IDapi\${IDapi}_log.txt" | foreach { $_ -replace "	","" } 
Set-Content -Path ".\$IDapi\${IDapi}_log.txt" -Value $content
$content = Get-Content ".\$IDapi\${IDapi}_log.txt" | foreach { $_ -replace '<input type="hidden" name="filename" value="','' } 
Set-Content -Path ".\$IDapi\${IDapi}_log.txt" -Value $content

#Check listing to extract
sleep 2
notepad .\$IDapi\${IDapi}_log.txt
sleep 2
[System.Windows.Forms.MessageBox]::Show($THIS, "Please check your txt fie", "Extract Log")
pause

#Download file 
$content = Get-Content ".\$IDapi\${IDapi}_log.txt" 
 foreach ($line in $content) { 
 $url_setup = "https://url/downloadLog?filename=${line}&nb=${IDapi}&see=see"
 $url_setup
 Invoke-WebRequest -Uri $url_setup -Headers $headers -OutFile .\$IDapi\$line.log } 
exit

Open in new window

I would like to bring the following improvements:
1-If IDapi folder exists skip md action
2-Generate the reformat related to the HTML file, generate listing file without downloading HTML file
3-Define an AsOfDate variable example 20191016 and remove the various lines of the listing which don't contain this string
4-Change the approach related to the listing set up. Instead of asking to continue, I would like to remove this popup and make sure that listing has been set up automatically prior to the download files action
5-Remove the popup related $IDapi to report and be able to report multiples $IDapi in a txt file or directly in the .ps1 as variable.

If you have questions, please contact me.

Thank you for your help.

oBdA

This is heavily untested, but should work:

#login information
$user= ""
#$user= Read-Host "Please enter a value"
#Password
$pass= ""
#$pass = Read-Host 'Enter Password' -AsSecureString
$asOf = '20191016' 		# today's date: Get-Date -Format 'yyyyMMdd'
$apiAddress = 'domain.com'
$destinationFolder = '.'

#ID of service
$idApiList = @(
	'ServiceNumber1'
	'ServiceNumber2'
)
If (-not ($idApiList)) {
	$idApiString = Read-Host "Please enter the list of service numbers"
	$idApiList = $idApiString.Split(',', [StringSplitOptions]::RemoveEmptyEntries) | ForEach-Object {$_.Trim()}
}

# Reformat header
$bytes = [System.Text.Encoding]::ASCII.GetBytes("${user}:${pass}")
$base64 = [System.Convert]::ToBase64String($bytes)
$headers = @{Authorization = "Basic $($base64)"}

ForEach ($idApi in $idApiList) {
	# Create folder in which will be located files to download
	$workFolder = Join-Path -Path $destinationFolder -ChildPath $idApi
	If (-not (Test-Path -Path $workFolder)) {
		New-Item -Path $workFolder -ItemType Directory | Out-Null
	}

	# Setup url
	$urlGetLog = "https://$($apiAddress)/getLog?nb=$($idApi)"
	# Display url
	$urlGetLog

	$htmlFile = "$($workFolder)\$($idApi).html"
	$fileList = "$($workFolder)\$($idApi)_log.txt"
	# Recover html file (file which contains the various log listing
	Invoke-WebRequest -Uri $urlGetLog -Headers $headers -OutFile $htmlFile

	$content = Get-Content -Path $htmlFile
	$content = $content | Where-Object {$_ -like 'filename'}
	ForEach ($pattern in '" />', '\t', '<input type="hidden" name="filename" value="') {
		$content = $content -replace $pattern
	}
	$content = $content | Where-Object {$_ -like '*$($asOf)*'}
	Set-Content -Path $fileList -Value $content

	# Check listing to extract
	# [System.Windows.Forms.MessageBox]::Show($THIS, "Please check your txt file", "Extract Log")
	"Waiting for Notepad '$($fileList)' to be closed ..."
	Start-Process -FilePath notepad.exe -ArgumentList $fileList -Wait

	# Download files
	ForEach ($file in (Get-Content -Path $fileList)) {
		$urlGetFile = "https://$($apiAddress)/downloadLog?filename=$($file)&nb=$($idApi)&see=see"
		$urlGetFile
		Invoke-WebRequest -Uri $urlGetFile -Headers $headers -OutFile "$($workFolder)\$($file).log"
	} 
}

Open in new window

Luis Diaz

ASKER

Thank you oBdA, unable to test right now.
I reviewed requirement 2 in order to read HTML without downloading. Do you think this is possible?

oBdA

Well, there's probably no way to get the list without downloading it, but you don't have to write it to a file first.
Instead of lines 41-43, you can try this:

$page= Invoke-WebRequest -Uri $urlGetLog -Headers $headers -OutFile $htmlFile

$content = $page.RawContent

Open in new window

Luis Diaz

ASKER

Noted. Thank you for this update.

Luis Diaz

ASKER

oBdA: I tested the proposal but I have two little issues:
1-Files are generated at: UserProfile folder instead of current location. I supposed that I need to define CurrentDir folder like this

$CurrentDir = Split-Path $script:MyInvocation.MyCommand.Path
#Example: $Root = $CurrentDir + '\Root2'

Open in new window

But I don't know how
2-I tried to generate html file and it is generated: however the following:

ForEach ($pattern in '" />', '\t', '<input type="hidden" name="filename" value="') {
		$content = $content -replace $pattern
	}
	#$content = $content | Where-Object {$_ -like '*$($asOf)*'}
	Set-Content -Path $fileList -Value $content

Open in new window

generates me a FileList empty. So I suppose that the $pattern replacement is not working properly. Could you please help me to properly define.
I attached dummy html file.

Thank you for your help.
html-content.txt

oBdA

Try it like this:

#login information
$user= ""
#$user= Read-Host "Please enter a value"
#Password
$pass= ""
#$pass = Read-Host 'Enter Password' -AsSecureString
$asOf = '20191016' 		# today's date: Get-Date -Format 'yyyyMMdd'
$destinationFolder = Split-Path $MyInvocation.MyCommand.Path -Parent
#ID of service
$idApiList = @(
	'ServiceNumber1'
	'ServiceNumber2'
)
If (-not ($idApiList)) {
	$idApiString = Read-Host "Please enter the list of service numbers"
	$idApiList = $idApiString.Split(',', [StringSplitOptions]::RemoveEmptyEntries) | ForEach-Object {$_.Trim()}
}

$apiAddress = 'domain.com'
$urlGetLog = "https://$($apiAddress)/url/getLog?nb={0}"	## {0}: idApi
$urlGetFile = "https://$($apiAddress)/url/downloadLog?filename={1}&nb={0}&see=see"	## {0}: idApi; {1}: filename

# Reformat header
$bytes = [System.Text.Encoding]::ASCII.GetBytes("${user}:${pass}")
$base64 = [System.Convert]::ToBase64String($bytes)
$headers = @{Authorization = "Basic $($base64)"}

ForEach ($idApi in $idApiList) {
	# Create folder in which will be located files to download
	$workFolder = Join-Path -Path $destinationFolder -ChildPath $idApi
	If (-not (Test-Path -Path $workFolder)) {
		New-Item -Path $workFolder -ItemType Directory | Out-Null
	}
	$fileList = "$($workFolder)\$($idApi)_log.txt"
	
	# Recover html file (file which contains the various log listing
	$url = $urlGetLog -f $idApi
	Write-Host "Downloading log file list from '$($url)' ..."
	$html = Invoke-WebRequest -Uri $url -Headers $headers
	$fileNames = ($html.Content -split '\r?\n') | Where-Object {$_ -like '*<input*'} | ForEach-Object {
		$xml = [xml]$_
		If ($xml.input.name -eq 'filename') {
			$xml.input.value
		}
	}
	$fileNames | Set-Content -Path $fileList	## All identified files; for logging purposes only
	$downloadNames = $fileNames | Where-Object {$_ -like "*$($asOf)*"}

	# Download files
	ForEach ($name in $downloadNames) {
		$url = $urlGetFile -f $idApi, $name
		Write-Host "Downloading log file from '$($url)' ..."
		Invoke-WebRequest -Uri $url -Headers $headers -OutFile "$($workFolder)\$($name).log"
	} 
}

Open in new window

Luis Diaz

ASKER

Thank you for this proposal. I got the folders created however I don't have any "log.txt" file created.
Complicate to debug as I don't have any error message when I run the script.
Is there a way to add some debug lines to your proposal?
Thank you for your help.

Luis Diaz

ASKER

The following works for me, I don't know if there is a proper way to do it, please don't hesitate to help me to revise it.
I was thinking to:
1-to put a flag to download or not html file. Is there a way
2-Put service number in a txt located at current directory

#Bypass TLS 1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
#Graphique objects
[System.Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms")
#login information
$user= ""
#Password
$pass= ""
$asOf = '20191017' 		# today's date: Get-Date -Format 'yyyyMMdd'
$apiAddress = ''
$destinationFolder = Split-Path $MyInvocation.MyCommand.Path -Parent

#ID of service
$idApiList = @(
	''
    ''
)
If (-not ($idApiList)) {
	$idApiString = Read-Host "Please enter the list of service numbers"
	$idApiList = $idApiString.Split(',', [StringSplitOptions]::RemoveEmptyEntries) | ForEach-Object {$_.Trim()}
}

# Reformat header
$bytes = [System.Text.Encoding]::ASCII.GetBytes("${user}:${pass}")
$base64 = [System.Convert]::ToBase64String($bytes)
$headers = @{Authorization = "Basic $($base64)"}

ForEach ($idApi in $idApiList) {
	# Create folder in which will be located files to download
	$workFolder = Join-Path -Path $destinationFolder -ChildPath $idApi
	If (-not (Test-Path -Path $workFolder)) {
		New-Item -Path $workFolder -ItemType Directory | Out-Null
	}

	# Setup url
	$urlGetLog = "https://$($apiAddress)/getLog?nb=$($idApi)"
	# Display url
	$urlGetLog

	$htmlFile = "$($workFolder)\$($idApi).html"
	$fileList = "$($workFolder)\$($idApi)_log.txt"
	# Recover html file (file which contains the various log listing
	Invoke-WebRequest -Uri $urlGetLog -Headers $headers -OutFile $htmlFile
	#Get-Content $htmlFile | Where-Object { $_.Contains("value") }
    $content = Get-Content -Path $htmlFile | Where-Object { $_.Contains("filename") }
       #$page= Invoke-WebRequest -Uri $urlGetLog -Headers $headers -OutFile $htmlFile
       #$content = $page.RawContent
    	ForEach ($pattern in '" />', '\t', '<input type="hidden" name="filename" value="') {
    		$content = $content -replace $pattern
    	}
	$content = $content | Where-Object {$_.Contains("$($asOf)")}
	Set-Content -Path $fileList -Value $content

	# Download files
	ForEach ($file in (Get-Content -Path $fileList)) {
		$urlGetFile = "https://$($apiAddress)/downloadLog?filename=$($file)&nb=$($idApi)&see=see"
		$urlGetFile
		Invoke-WebRequest -Uri $urlGetFile -Headers $headers -OutFile "$($workFolder)\$($file).log"
	} 
}

Open in new window

oBdA

This allows you to get the ids inline, from file, or from keyboard by setting the variable $idApiSource as described.
For the "names" log file and the html source, there are now booleans you can set to (not) write the respective file.
It's based on https://www.experts-exchange.com/questions/29161242/Powershell-download-files-from-an-url.html# as it's less vulnerable to changes in the html source; if at some point, for example, the type and name attributes swap places, the string replace would fail, while the xml solution would still work.
Since the popups are removed, you don't need the System.Windows.Forms assembly anymore.
If you need it anyway later on (or in another script), note that the "LoadWithPartialName" method is deprecated. Use Add-Type instead:
Add-Type -AssemblyName System.Windows.Forms

# login information
$user= ""
#$user= Read-Host "Please enter a value"
# Password
$pass= ""
#$pass = Read-Host 'Enter Password' -AsSecureString
$asOf = '20191016' 		# today's date: Get-Date -Format 'yyyyMMdd'
$destinationFolder = Split-Path $MyInvocation.MyCommand.Path -Parent

$idApiSource = 'File'	## Sets the source from where to retrieve the IDs; set to Inline, File, or Manual
$idApiList_Inline = @(	## Inline service ID list
	'ServiceNumber1'
	'ServiceNumber2'
)
$idApiList_File = 'IDApiList.txt'	## Service ID input file name (expected in the script's folder), one ID per line

$htmlFile = "{0}.html"		## {0}: idApi; file name template with the html source
$saveHtmlFile = $true
$fileList = "{0}_log.txt"	## {0}: idApi; file name template with log file names parsed from the html
$saveFileList = $true

$apiAddress = 'domain.com'
$urlGetLog =  "https://$($apiAddress)/url/getLog?nb={0}"							## {0}: idApi
$urlGetFile = "https://$($apiAddress)/url/downloadLog?filename={1}&nb={0}&see=see"	## {0}: idApi; {1}: filename


Switch ($idApiSource) {
	'Inline' {
		$idApiList = $idApiList_Inline
	}
	'File' {
		$idApiList = Get-Content -Path "$($destinationFolder)\$($idApiList_File)"
	}
	default {
		$idApiString = Read-Host "Please enter the list of service numbers"
		$idApiList = $idApiString.Split(',', [StringSplitOptions]::RemoveEmptyEntries) | ForEach-Object {$_.Trim()}
	}
}

#Bypass TLS 1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# Reformat header
$bytes = [System.Text.Encoding]::ASCII.GetBytes("${user}:${pass}")
$base64 = [System.Convert]::ToBase64String($bytes)
$headers = @{Authorization = "Basic $($base64)"}

ForEach ($idApi in $idApiList) {
	# Create folder in which will be located files to download
	$workFolder = Join-Path -Path $destinationFolder -ChildPath $idApi
	If (-not (Test-Path -Path $workFolder)) {
		New-Item -Path $workFolder -ItemType Directory | Out-Null
	}
	
	# Recover html file (file which contains the various log listing
	$url = $urlGetLog -f $idApi
	Write-Host "Downloading log file list from '$($url)' ..."
	$html = Invoke-WebRequest -Uri $url -Headers $headers
	If ($saveHtmlFile) {
		$html.Content | Set-Content -Path "$($workFolder)\$($htmlFile -f $idApi)"	## Html source code; for logging purposes only
	}
	$fileNames = ($html.Content -split '\r?\n') | Where-Object {$_ -like '*<input*'} | ForEach-Object {
		$xml = [xml]$_
		If ($xml.input.name -eq 'filename') {
			$xml.input.value
		}
	}
	If ($saveFileList) {
		$fileNames | Set-Content -Path "$($workFolder)\$($fileList -f $idApi)"	## All identified files; for logging purposes only
	}
	$downloadNames = $fileNames | Where-Object {$_ -like "*$($asOf)*"}

	# Download files
	ForEach ($name in $downloadNames) {
		$url = $urlGetFile -f $idApi, $name
		Write-Host "Downloading log file from '$($url)' ..."
		Invoke-WebRequest -Uri $url -Headers $headers -OutFile "$($workFolder)\$($name).log"
	} 
}

Open in new window

Luis Diaz

ASKER

Thank you oBdA I will test it and keep you informed.

Luis Diaz

ASKER

OBdA,
I was wondering if there is a way to have the following improvements:
-Have a correspondance for service number folder name. I would like to rename them with a new name but for that I suppose to have a mapping to report revised name.
-Have a second variable as the one of $asof to exclude name of the listing which contains specific string:
Example: $exclude='ko'.
Thank you for your help.

oBdA

Service ID list must now be defined as csv (except for manual input, where the target folder will still be the service id)

# login information
$user= ""
#$user= Read-Host "Please enter a value"
# Password
$pass= ""
#$pass = Read-Host 'Enter Password' -AsSecureString
$asOf = '20191016' 		# today's date: Get-Date -Format 'yyyyMMdd'
$exclude = 'ko'
$destinationFolder = Split-Path $MyInvocation.MyCommand.Path -Parent

$idApiSource = 'File'	## Sets the source from where to retrieve the IDs; set to Inline, File, or Manual
## Inline service ID list as csv
$idApiList_Inline = @'
	"ID",				"Directory"
	"ServiceNumber1",	"Service1"
	"ServiceNumber2",	"Service2"
'@
$idApiList_File = 'IDApiList.csv'	## Service ID input file name (expected in the script's folder), as csv (see idApiList_Inline)

$htmlFile = "{0}.html"		## {0}: idApi; file name template with the html source
$saveHtmlFile = $true
$fileList = "{0}_log.txt"	## {0}: idApi; file name template with log file names parsed from the html
$saveFileList = $true

$apiAddress = 'domain.com'
$urlGetLog =  "https://$($apiAddress)/url/getLog?nb={0}"							## {0}: idApi
$urlGetFile = "https://$($apiAddress)/url/downloadLog?filename={1}&nb={0}&see=see"	## {0}: idApi; {1}: filename


Switch ($idApiSource) {
	'Inline' {
		$idApiList = $idApiList_Inline | ConvertFrom-Csv
	}
	'File' {
		$idApiList = Import-Csv -Path "$($destinationFolder)\$($idApiList_File)"
	}
	default {
		$idApiString = Read-Host "Please enter the list of service numbers"
		$idApiList = $idApiString.Split(',', [StringSplitOptions]::RemoveEmptyEntries) | Select-Object -Property @{n='ID'; e={$_.Trim()}}, @{n='Directory'; e={$_.Trim()}}
		}
	}
}

#Bypass TLS 1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# Reformat header
$bytes = [System.Text.Encoding]::ASCII.GetBytes("${user}:${pass}")
$base64 = [System.Convert]::ToBase64String($bytes)
$headers = @{Authorization = "Basic $($base64)"}

ForEach ($idApi in $idApiList) {
	# Create folder in which will be located files to download
	$workFolder = Join-Path -Path $destinationFolder -ChildPath $idApi.Directory
	If (-not (Test-Path -Path $workFolder)) {
		New-Item -Path $workFolder -ItemType Directory | Out-Null
	}
	
	# Recover html file (file which contains the various log listing
	$url = $urlGetLog -f $idApi.ID
	Write-Host "Downloading log file list from '$($url)' ..."
	$html = Invoke-WebRequest -Uri $url -Headers $headers
	If ($saveHtmlFile) {
		$html.Content | Set-Content -Path "$($workFolder)\$($htmlFile -f $idApi.ID)"	## Html source code; for logging purposes only
	}
	$fileNames = ($html.Content -split '\r?\n') | Where-Object {$_ -like '*<input*'} | ForEach-Object {
		$xml = [xml]$_
		If ($xml.input.name -eq 'filename') {
			$xml.input.value
		}
	}
	If ($saveFileList) {
		$fileNames | Set-Content -Path "$($workFolder)\$($fileList -f $idApi.ID)"	## All identified files; for logging purposes only
	}
	$downloadNames = $fileNames | Where-Object {$_ -like "*$($asOf)*"}
	If ($exclude) {
		$downloadNames = $downloadNames | Where-Object {$_ -notlike "*$($exclude)*"}
	}

	# Download files
	ForEach ($name in $downloadNames) {
		$url = $urlGetFile -f $idApi.ID, $name
		Write-Host "Downloading log file from '$($url)' ..."
		Invoke-WebRequest -Uri $url -Headers $headers -OutFile "$($workFolder)\$($name).log"
	} 
}

Open in new window

Luis Diaz

ASKER

I tested your last proposal but I have some little issues:

1-Extra } at line 42 and I removed it.
2-When csv file doesn't exist and I put service-folder name mapping I have the following error message:

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (:) [Import-Csv], FileNotFoundException
    + FullyQualifiedErrorId : FileOpenFailure,Microsoft.PowerShell.Commands.ImportCsvCommand

Open in new window

3-When csv file exists with the required information I don't have error but folders are not created
4-Download folder statement report the following message:

getLog?nb=' ...

Open in new window

And files are not downloaded
5-When I use the version reported at:
https://www.experts-exchange.com/questions/29161242/Powershell-download-files-from-an-url.html#a42962714
I got:

getLog?nb=138

Open in new window

138 which is the service number reported and related to $idApiList_Inline
Thank you for your help.

oBdA

The surplus curly bracket (line 40, actually) aside, I can't reproduce that.
As described in the comments, the variable $idApiSource must be set to indicate where you want your id source from. If this is set to "File", and the file doesn't exist, then there obviously will be an error. If you set it to Inline or Manual, the csv file is never accessed.

Here's a test run; note how at the start, the csv file is still named .txt, not .csv. $idApiSource is inline (IDs 123 and 321), and no error about the file.
Download location is obviously not the original site, but the correct ID is embedded in the URL.

PS C:\PS\EE29161242> gci | select Name

Name
----
Download.ps1
IDApiList.txt


PS C:\PS\EE29161242> .\Download.ps1
Reading ID list embedded into script.
Downloading log file list from 'http://www.acme.com/url/html-content.html?nb=123' ...
Downloading log file from 'https://domain.com/url/downloadLog?filename=viewcopy.log.20191016&nb=123&see=see' ...
Downloading log file list from 'http://www.acme.com/url/html-content.html?nb=321' ...
Downloading log file from 'https://domain.com/url/downloadLog?filename=viewcopy.log.20191016&nb=321&see=see' ...
PS C:\PS\EE29161242> gci | select Name

Name
----
Service123
Service321
Download.ps1
IDApiList.txt


PS C:\PS\EE29161242> ## Changed $idApiSource to 'File'
PS C:\PS\EE29161242> ren .\IDApiList.txt .\IDApiList.csv
PS C:\PS\EE29161242> gc .\IDApiList.csv
"ID",   "Directory"
"456",  "Service456"
"654",  "Service654"
PS C:\PS\EE29161242> .\Download.ps1
Reading ID list from file 'IDApiList.csv'
Downloading log file list from 'http://www.acme.com/url/html-content.html?nb=456' ...
Downloading log file from 'https://domain.com/url/downloadLog?filename=viewcopy.log.20191016&nb=456&see=see' ...
Downloading log file list from 'http://www.acme.com/url/html-content.html?nb=654' ...
Downloading log file from 'https://domain.com/url/downloadLog?filename=viewcopy.log.20191016&nb=654&see=see' ...
PS C:\PS\EE29161242> gci | select Name

Name
----
Service123
Service321
Service456
Service654
Download.ps1
IDApiList.csv

Open in new window

Here's the latest script; only difference to the last one is that the surplus curly bracket is removed, the inline ID list now uses three digits, and that it now writes to the console where the ID list is read from.

# login information
$user= ""
#$user= Read-Host "Please enter a value"
# Password
$pass= ""
#$pass = Read-Host 'Enter Password' -AsSecureString
$asOf = '20191016' 		# today's date: Get-Date -Format 'yyyyMMdd'
$exclude = 'ko'
$destinationFolder = Split-Path $MyInvocation.MyCommand.Path -Parent

$idApiSource = 'File'	## Sets the source from where to retrieve the IDs; set to Inline, File, or Manual
## Inline service ID list as csv
$idApiList_Inline = @'
	"ID",	"Directory"
	"123",	"Service123"
	"321",	"Service321"
'@
$idApiList_File = 'IDApiList.csv'	## Service ID input file name (expected in the script's folder), as csv (see idApiList_Inline)

$htmlFile = "{0}.html"		## {0}: idApi; file name template with the html source
$saveHtmlFile = $true
$fileList = "{0}_log.txt"	## {0}: idApi; file name template with log file names parsed from the html
$saveFileList = $true

$apiAddress = 'domain.com'
$urlGetLog =  "https://$($apiAddress)/url/getLog?nb={0}"							## {0}: idApi
$urlGetFile = "https://$($apiAddress)/url/downloadLog?filename={1}&nb={0}&see=see"	## {0}: idApi; {1}: filename


Switch ($idApiSource) {
	'Inline' {
		Write-Host 'Reading ID list embedded into script.'
		$idApiList = $idApiList_Inline | ConvertFrom-Csv
	}
	'File' {
		Write-Host "Reading ID list from file '$($idApiList_File)'"
		$idApiList = Import-Csv -Path "$($destinationFolder)\$($idApiList_File)" -ErrorAction Stop
	}
	default {
		$idApiString = Read-Host "Please enter the list of service numbers"
		$idApiList = $idApiString.Split(',', [StringSplitOptions]::RemoveEmptyEntries) | Select-Object -Property @{n='ID'; e={$_.Trim()}}, @{n='Directory'; e={$_.Trim()}}
	}
}

#Bypass TLS 1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# Reformat header
$bytes = [System.Text.Encoding]::ASCII.GetBytes("${user}:${pass}")
$base64 = [System.Convert]::ToBase64String($bytes)
$headers = @{Authorization = "Basic $($base64)"}

ForEach ($idApi in $idApiList) {
	# Create folder in which will be located files to download
	$workFolder = Join-Path -Path $destinationFolder -ChildPath $idApi.Directory
	If (-not (Test-Path -Path $workFolder)) {
		New-Item -Path $workFolder -ItemType Directory | Out-Null
	}
	
	# Recover html file (file which contains the various log listing
	$url = $urlGetLog -f $idApi.ID
	Write-Host "Downloading log file list from '$($url)' ..."
	$html = Invoke-WebRequest -Uri $url -Headers $headers
	If ($saveHtmlFile) {
		$html.Content | Set-Content -Path "$($workFolder)\$($htmlFile -f $idApi.ID)"	## Html source code; for logging purposes only
	}
	$fileNames = ($html.Content -split '\r?\n') | Where-Object {$_ -like '*<input*'} | ForEach-Object {
		$xml = [xml]$_
		If ($xml.input.name -eq 'filename') {
			$xml.input.value
		}
	}
	If ($saveFileList) {
		$fileNames | Set-Content -Path "$($workFolder)\$($fileList -f $idApi.ID)"	## All identified files; for logging purposes only
	}
	$downloadNames = $fileNames | Where-Object {$_ -like "*$($asOf)*"}
	If ($exclude) {
		$downloadNames = $downloadNames | Where-Object {$_ -notlike "*$($exclude)*"}
	}

	# Download files
	ForEach ($name in $downloadNames) {
		$url = $urlGetFile -f $idApi.ID, $name
		Write-Host "Downloading log file from '$($url)' ..."
		Invoke-WebRequest -Uri $url -Headers $headers -OutFile "$($workFolder)\$($name).log"
	} 
}

Open in new window

Luis Diaz

ASKER

Now it's work when I use 'Inline' parameter.
I removed the url part
$urlGetLog = "https://$($apiAddress)/url/getLog?nb={0}" ## {0}: idApi
$urlGetFile = "https://$($apiAddress)/url/downloadLog?filename={1}&nb={0}&see=see" ## {0}: idApi; {1}: filename

When I used 'File' parameter I got the following error message:

29161242_download-files-just-html-2.ps1:60 char:10
+     $html = Invoke-WebRequest -Uri $url -Headers $headers
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Open in new window

I attached csv file.

Notice that I put ; as following

$idApiList = $idApiString.Split(';', [StringSplitOptions]::RemoveEmptyEntries) | Select-Object -Property @{n='ID'; e={$_.Trim()}}, @{n='Directory'; e={$_.Trim()}}

Open in new window

Because csv file used a ; delimiter.
IDApiList.csv
I don't know how to debug this part.
If the best is to go back to txt I can do it.
Thank you in advance for your help.

oBdA

That's not the part for the File you changed, that's the part for the 'fallback' manual input.
The delimiter needs to be set for the Import-Csv command in line 37.

# login information
$user= ""
#$user= Read-Host "Please enter a value"
# Password
$pass= ""
#$pass = Read-Host 'Enter Password' -AsSecureString
$asOf = '20191016' 		# today's date: Get-Date -Format 'yyyyMMdd'
$exclude = 'ko'
$destinationFolder = Split-Path $MyInvocation.MyCommand.Path -Parent

$idApiSource = 'File'	## Sets the source from where to retrieve the IDs; set to Inline, File, or Manual
## Inline service ID list as csv
$idApiList_Inline = @'
	"ID",	"Directory"
	"123",	"Service123"
	"321",	"Service321"
'@
$idApiList_File = 'IDApiList.csv'	## Service ID input file name (expected in the script's folder), as csv (see idApiList_Inline)

$htmlFile = "{0}.html"		## {0}: idApi; file name template with the html source
$saveHtmlFile = $true
$fileList = "{0}_log.txt"	## {0}: idApi; file name template with log file names parsed from the html
$saveFileList = $true

$apiAddress = 'domain.com'
$urlGetLog =  "https://$($apiAddress)/getLog?nb={0}"							## {0}: idApi
$urlGetFile = "https://$($apiAddress)/downloadLog?filename={1}&nb={0}&see=see"	## {0}: idApi; {1}: filename


Switch ($idApiSource) {
	'Inline' {
		Write-Host 'Reading ID list embedded into script.'
		$idApiList = $idApiList_Inline | ConvertFrom-Csv
	}
	'File' {
		Write-Host "Reading ID list from file '$($idApiList_File)'"
		$idApiList = Import-Csv -Path "$($destinationFolder)\$($idApiList_File)" -Delimiter ';' -ErrorAction Stop
	}
	default {
		$idApiString = Read-Host "Please enter a comma separated list of service numbers"
		$idApiList = $idApiString.Split(',', [StringSplitOptions]::RemoveEmptyEntries) | Select-Object -Property @{n='ID'; e={$_.Trim()}}, @{n='Directory'; e={$_.Trim()}}
	}
}

#Bypass TLS 1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# Reformat header
$bytes = [System.Text.Encoding]::ASCII.GetBytes("${user}:${pass}")
$base64 = [System.Convert]::ToBase64String($bytes)
$headers = @{Authorization = "Basic $($base64)"}

ForEach ($idApi in $idApiList) {
	# Create folder in which will be located files to download
	$workFolder = Join-Path -Path $destinationFolder -ChildPath $idApi.Directory
	If (-not (Test-Path -Path $workFolder)) {
		New-Item -Path $workFolder -ItemType Directory | Out-Null
	}
	
	# Recover html file (file which contains the various log listing
	$url = $urlGetLog -f $idApi.ID
	Write-Host "Downloading log file list from '$($url)' ..."
	$html = Invoke-WebRequest -Uri $url -Headers $headers
	If ($saveHtmlFile) {
		$html.Content | Set-Content -Path "$($workFolder)\$($htmlFile -f $idApi.ID)"	## Html source code; for logging purposes only
	}
	$fileNames = ($html.Content -split '\r?\n') | Where-Object {$_ -like '*<input*'} | ForEach-Object {
		$xml = [xml]$_
		If ($xml.input.name -eq 'filename') {
			$xml.input.value
		}
	}
	If ($saveFileList) {
		$fileNames | Set-Content -Path "$($workFolder)\$($fileList -f $idApi.ID)"	## All identified files; for logging purposes only
	}
	$downloadNames = $fileNames | Where-Object {$_ -like "*$($asOf)*"}
	If ($exclude) {
		$downloadNames = $downloadNames | Where-Object {$_ -notlike "*$($exclude)*"}
	}

	# Download files
	ForEach ($name in $downloadNames) {
		$url = $urlGetFile -f $idApi.ID, $name
		Write-Host "Downloading log file from '$($url)' ..."
		Invoke-WebRequest -Uri $url -Headers $headers -OutFile "$($workFolder)\$($name).log"
	} 
}

Open in new window

Luis Diaz

ASKER

Ok, my mistake, sorry for that. Unable to test it right know. I will keep you informed.

Luis Diaz

ASKER

oBdA, I tested your last version and it works! Thank you again!
Just one remark: I removed . log at line 84 in order to avoid filename.log.log.
I have last additional requirements if possible:
1-Be able to put $false for asOf as it is done for $exclude

$downloadNames = $fileNames | Where-Object {$_ -like "*$($asOf)*"}

Open in new window

As it is done for:

	If ($exclude) {
		$downloadNames = $downloadNames | Where-Object {$_ -notlike "*$($exclude)*"}
	}

Open in new window

This will allows me to omit asOf if necessary through $asOf= $false.
2-Define an $include variable as it is done for $exclude.

	If ($include) {
		$downloadNames = $downloadNames | Where-Object {$_ -like "*$($include)*"}
	}

Open in new window

3-Remove files older than specific date at subfolders of $destinationFolder with flag approach as it is done for $exclude.

Thank you very much for your help.

ASKER CERTIFIED SOLUTION

oBdA

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Luis Diaz

ASKER

I tested and it works!
Thank you again for your help!