Solved

Power shell script to read text file and populate data to SQL database tables

Posted on 2013-12-11
6
471 Views
Last Modified: 2014-02-06
I am new to power shell so need you help in this guys. I have attached a word file detailing the requirement.
Summary.docx
20130930-OPEQCNTRL
20130930-OPEQBAL
20130930-OPEQINT
20130930-OPEQLMT
TestOPDB.sql
OP-EQ-BAL.txt
OP-EQ-INT.txt
OP-EQ-LMT.txt
0
Comment
Question by:josyp
  • 4
  • 2
6 Comments
 
LVL 68

Expert Comment

by:Qlemo
ID: 39714232
Here is the frame to execute your stuff. But there are some flaws in your "dynamic" design - you need to know the file-to-table relation, or set up a rule like
20130930_OPEQBAL
DDDDDD_AABBCCC  => Table AA_BB_CCC

Open in new window

and your parameter file wouldn't help with that, or would it make more complicated than needed.

The parameter file isn't a bad idea, but it is much easier to keep just some vars in the beginning of the script, unless you need to run the same script with different parameter files.

And I would not use BCP. It is easy to access MSSQL from PS, so I would choose that approach (using DataReader or other .NET methods).
0
 

Author Comment

by:josyp
ID: 39720842
The frame that you pasted only contains two lines.
0
 
LVL 68

Expert Comment

by:Qlemo
ID: 39721102
Strange, I'm pretty sure I posted some code ... I remember I've checked the archive folder for the most recent control files and such.

Your question consists of several parts:
1. Using a parameter file or similar to configure paths and names
2. Waiting for a control file and the respective files (noted therein) to be written until 7:00 AM.
3. Comparing against an archive folder - only if current files are newer they should be considered
4. Parsing/converting the data files in a way they can be processed with BCP
5. Running the BCP import into respective tables

Part 4 is discussed in http://www.experts-exchange.com/Q_28318533.html.

Part 1 should be discussed in another thread as soon as we have crafted all the other stuff.

Parts 2 and 3 are what I had posted here as a code frame, now lost in space. I'll have to rebuild that.
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 68

Expert Comment

by:Qlemo
ID: 39721149
This is what I've put together so far. It does all the checking; doing the data import with BCP would be the next step now.
$DataFolder     = 'C:\Data'
$ArchiveFolder  = 'C:\Archive'
$CtrlFileSuffix = '_OPEQCNTRL'
$LogFile        = 'C:\Data\Log.txt'

$CtrlFile = "$(get-date -format yyyymmdd)$CtrlFileSuffix"

# Check for control file, exit if not created
# needs to get started after midnight!
while ((Get-Date).Hour -lt 7 -and !(Test-Path "$DataFolder\$CtrlFile"))
{
  Start-Sleep 20*60 # wait 20 minutes
}
if (!(Test-Path "$DataFolder\$CtrlFile"))
{
  "$(get-date) - Error: Control file not found" | Out-File -Append $LogFile
  return
}

# Log file exists, check against archive files
if (
    (Get-ChildItem "$ArchiveFolder\*$CtrlFileSuffix" -Name |
      sort -Descending |
      select -First 1
    ).Name -ge $CtrlFile
   )
{
  "$(get-date) - Error: No more recent control file found" | Out-File -Append $LogFile
  return
}

# Check for data files as noted in control file
$datafiles = Import-Csv "$DataFolder\$CtrlFile" -Header Type, Row, Name |
             ? { $_.Type -eq '02' } |
             Select -ExpandProperty Name

if ($missing = $datafiles | ? { !(Test-Path "$DataFolder\$_") })
{
  "$(get-date) - Error: Missing data files $($missing -join ',')" | Out-File -Append $LogFile
  return
}
Remove-Variable missing

# -- All checks done, BCP preps will start here, based on $datafiles ----

Open in new window

0
 

Author Comment

by:josyp
ID: 39735539
I don't see any thing happening with the above code tried to rename the control file to the current date and still not able to see anything happening, there is no log file being generated.
0
 
LVL 68

Accepted Solution

by:
Qlemo earned 500 total points
ID: 39739128
There was a bug in the checking stuff for the data files, and some other issues, so I've changed some more and introduced a logging function to see the log contents both on screen and in a file.
cls
Set-StrictMode -version Latest

<# Parameter section #>
$DataFolder     = 'C:\Data'
$ArchiveFolder  = 'C:\Archive'
$CtrlFileSuffix = '_OPEQCNTRL'
$LogFile        = 'C:\Data\Log.txt'
<# End #>

$CtrlFile = "$(get-date -format yyyyMMdd)$CtrlFileSuffix"

function write-log ([String]$msg)
{
  Write-Host $msg
  "$(get-date -format 'yyyy-MM-dd HH:mm:ss') - $msg" | Out-File -Append $LogFile
}

# Check for control file, exit if not created
# needs to get started after midnight!
while ((Get-Date).Hour -lt 7 -and !(Test-Path "$DataFolder\$CtrlFile"))
{
  Start-Sleep 20*60 # wait 20 minutes
}
if (!(Test-Path "$DataFolder\$CtrlFile"))
{
  write-log "Error: Control file '$DataFolder\$CtrlFile' not found"
  return
}

# Log file exists, check against archive files
if (
    (Get-ChildItem "$ArchiveFolder\*$CtrlFileSuffix" -Name |
      sort -Descending |
      select -First 1
    ) -ge $CtrlFile
   )
{
  write-log 'Error: No more recent control file found'
  return
}

# Check for data files as noted in control file
$datafiles = Import-Csv "$DataFolder\$CtrlFile" -Header Type, Row, Name -Delimiter '|' |
             ? { $_.Type -eq '02' } |
             Select -ExpandProperty Name

if ($missing = $datafiles | ? { !(Test-Path "$DataFolder\$_") })
{
  write-log "Error: Missing data file(s) $($missing -join ',')" 
  return
}
Remove-Variable missing

# -- All checks done, BCP preps will start here, based on $datafiles ----

Open in new window

Please note that you have posted files of format *-* here, while they should be *_* according to your description and the control file content.
Also note that http:/Q_28318533.html should use $datafiles, which contains the file names of the data files, if we put both scripts together.
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
how to import 7 54
Database Containment - Benefits 6 26
using t-sql EXISTS 8 23
select over clause 1 10
How to leverage one TLS certificate to encrypt Microsoft SQL traffic and Remote Desktop Services, versus creating multiple tickets for the same server.
In this article we will get to know that how can we recover deleted data if it happens accidently. We really can recover deleted rows if we know the time when data is deleted by using the transaction log.
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now