Link to home
Start Free TrialLog in
Avatar of mcrmg
mcrmg

asked on

help on script

Hi,

I have the following script that I got help from the EEs a few months ago. It has been working well until this month.

We receive many files monthly in TXT, CSV, XSL and XLSX. I use Powershell to combine them.

It is giving me an error  "excel cannot complete this task with available resources" on the last step. It does not show the error unless I open the final combined file.

It is difficult for me to provide the files because I do not think it is files related. Here are the steps I have done:

There are over 800 files in the folder. I located the file that was stopped and removed. The error stopped on another file. When I removed other files and kept the "supposed error files", the code went well. There is no pattern.
################################################################

$downloadFolder = 'June 2021 Downloads'

################################################################

$networkSource = "\\192.168.1.10\shared\${downloadFolder}\"

$sourceFolder = "C:\Source\"
$outputFolder = "C:\OUTPUT\"


Get-ChildItem -Path $sourceFolder -Include *.* -File -Recurse | foreach { $_.Delete()}

Copy-Item -Path "${networkSource}*LPE*.*" -Destination $sourceFolder



# del existing files
$outfile = "${outputFolder}combineFile.csv"
if(Test-path $outfile){
    Remove-Item $outfile -Confirm:$false
}






$Excel = New-Object -ComObject Excel.Application 
#---------only a few XLSX files------------------------------------------------------------------------
# convert XLSX into CSV
$filesXLSX = Get-ChildItem "${sourceFolder}" -Filter *.xlsx
foreach($file in ($filesXLSX)) {
    $newname = $file.FullName -replace '\.xlsx$', '.csv'
    $Workbook = $Excel.Workbooks.Open($file.FullName) 


    $Workbook.SaveAs($newname,6)
    $Workbook.Close($false)
}
$Excel.Quit()

# here will add CSV files (Converted from previous step into outfile)
$counter = 1
$files = Get-ChildItem "${sourceFolder}" -Filter *.csv
# Check first line if "Trans ID" exists, skip lines
    foreach ($f in $files){
        $firstLine = Get-Content -Path $f.FullName -TotalCount 1
        $secondLine = Get-Content -Path $f.FullName -TotalCount 2

            (Get-Content $f.FullName | Add-Content $outfile)
            echo "File opened CSV: $counter $f"

        $counter++

    }
#---------------------------------------------------------------------------------

$files = Get-ChildItem "${sourceFolder}" -Filter *.txt
# for some reason, the replaced file ("tab" to "," cannot be combined correctly RIGHT AFTER convertion, so we do the convertion first then combine
$counter = 1
foreach ($f in $files){
    if(Get-Content $f.FullName -First 1 | Select-String -Pattern "\t")
    {
        # replace "tab" with ","
        (Get-Content $f.FullName) -replace "\t", "," | Out-File $f.FullName

        ECHO "========== tab delimited file found ========== "  $f.Name 
    }


    (Get-Content $f.FullName | Add-Content $outfile) #########################################################

    echo "File opened TXT: $counter $f"

    $counter++
}






explorer.exe "${outputFolder}"

Open in new window


Please help. thank you
ASKER CERTIFIED SOLUTION
Avatar of Qlemo
Qlemo
Flag of Germany image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of mcrmg
mcrmg

ASKER

Thanks for the quick reply.

  1. Your last step is opening the folder in Explorer - this cannot bring up an Excel error message at all.
>> That is true. It just opens the folder.The error shows when I open the output file.
  1. The combining of CSV files is wrong, the headers are kept on append (and so added again and again). Looks like some code has been removed here.
>> True again. The headers will be appended. That is not an issue in my case. lol
  1. If the Excel error is caused by opening Excel in the first step, then adding a sleep between each file might help.If the error is a result of manually opening the combined CSV file in Excel, it might be too big.
>> The error displays when I open the output file manually. I doubt if the file is too big, it is about 20 some mb.

thanks

Avatar of mcrmg

ASKER

Is there a way to "clean/delete" everything after last row? thanks
A CSV is no Excel file. It does not have columns or rows, technically.
And the Excel export to CSV function should not generate content from empty rows.

See if you have still Excel instances running after converting but before you open the resulting file. Maybe Excel is not terminatting correctly, leaving a lot of memory junks around, and that instance is then used to open the CSV file.
Avatar of mcrmg

ASKER

@Qlemo , I think you were right on #3. (or something like that lol)

It looks like it works if I run the files in batches. Is there a way to modify the code to "spit" out an output file after processing, say, every 100 files. At the end then combine all the output files? thanks 
I don't see how that should help? If the final file is too big for somewhat reason ever, combining smaller file junks into it wouldn't give you a different outcome at the very end.

I would start with
a) checking as in my last comment
b) waiting 5 minutes after you see the Explorer window before opening the file.
Avatar of mcrmg

ASKER

a) checking as in my last comment >> I checked task manager, there was no Excel running
b) waiting 5 minutes after you see the Explorer window before opening the file. >> Still

Here is what I have tried:
1. I modified the code so it outputs a file when it processes every 100 files. I have 9 output files, each file can be opened up fine.

2. When I added the code to combine all 9 output files. The Final output file shows the same error when manually open up. Plus, not all the content from 9 output files were combined. (some files were cut off)

3. I then take this fine output file to my laptop, it opens up fine.

Very strange, I use the same code to run last month's folder, the output file is even bigger but runs fine.
I would start by cleaning up the code.. The comments in the code clearly show you, where you should use methods. Refactoring this allows easier reading and understanding the code as well as better debugging.

Using Excel automation requires that you clean up all used Excel objects and free them explicitly and call the garbage collector. For example, you don't clean up the workbook instances.

And for your logic: An Excel documents consists of worksheets. The SaveAs method only saves the active worksheet, the other sheets are lost.