Solved

batch file that sorts a table biased on the content of the 2nd column to a csv file named by the most common value.

Posted on 2016-07-28
8
47 Views
Last Modified: 2016-08-03
so i have a sql database that i would like to extract data into a CSV file using a batch file that runs periodically

the data is as follows
NOTE: sample of data actual table is around 100k lines long
Base code,Ident A,Ident B,District,U ID,
PA,2,5,PHELPS,001
SA,4.3,SIVELLE,002
MAR,1,5,MARYS,003
MAR,2,5,MARYS,004
M4,5,3,MARYS,005
S6,6,7,PHELPS,006
S6,5,7,PHELPS,007
LM5,3,6,NODE,008

Extracted to files like below

MAR.csv
MAR,
M4,

S6.csv
S6,
PA,

SA.csv
SA,

LM5.csv
LM5,

I would like a batch solution, I can already set the database to be extracted to a csv if that makes it easier
using a Microsoft sql server 2008 r2 with the sql server management studio 2008 r2 aswell
0
Comment
Question by:csePixelated
  • 4
  • 4
8 Comments
 
LVL 68

Expert Comment

by:Qlemo
ID: 41733300
Just to get it correct: Each CSV file should contain Base Codes having the same District? And within each district the most common base code determines the file name?
0
 
LVL 1

Author Comment

by:csePixelated
ID: 41733342
yes
0
 
LVL 68

Accepted Solution

by:
Qlemo earned 500 total points
ID: 41733933
Gosh. You are loosing all relevant information. No clue what this export should be for.

Assuming following definition:
create table tst_EE
([Base Code] varchar(10),
 [Ident A] smallint,
 [Ident B] smallint,
 [District] varchar(10),
 [U ID] char(3)
 )

insert into tst_ee values
('PA' ,2,5,'PHELPS' ,'001'),
('SA' ,4,3,'SIVELLE','002'),
('MAR',1,5,'MARYS'  ,'003'),
('MAR',2,5,'MARYS'  ,'004'),
('M4' ,5,3,'MARYS'  ,'005'),
('S6' ,6,7,'PHELPS' ,'006'),
('S6' ,5,7,'PHELPS' ,'007'),
('LM5',3,6,'NODE'   ,'008')

Open in new window


There are several different approaches. I'm choosing a PowerShell solution getting pre-processed SQL data directly, because that does not require complex processing. The first two variables need to be adapted to your environment.
Because your "CSV" files contain just a single "column", I will generate a simple text file instead of a CSV.
$ConStr = 'Server=MsSqlServer\Instance; Database=TheDB; Integrated Security=sspi'
$path = 'C:\temp\EE\Results\'

# Get data including target filename (without extenstion and path)
$set = new-object system.data.dataset
(new-object System.Data.SqlClient.SqlDataAdapter (@"
  with
    vals as (select [Base Code], District, cnt=count(*) from tst_EE group by District, [Base Code]),
    rnks as (select [Base Code], District, rnk=dense_rank() over (partition by District order by cnt desc) from vals)
  select filename=rnks.[Base Code], vals.District, vals.[Base Code], vals.cnt
    from vals join rnks on vals.District = rnks.District and rnks.rnk = 1
  order by district, cnt desc
"@, $ConStr)).Fill($set) | out-null

# Process the results
$csv = $null
$data = @()
$set.Tables[0] | % {
  if ($csv -eq $_.filename) {
    $data += $_.{Base Code}
  } else {
    # need to write out the file now
    if ($data) { $data | Out-File "$path$csv.csv" }
    $csv = $_.filename
    $data = @($_.{Base Code})
    Write-Host "Processing district $($_.District) -> $csv.csv"
  }
  Write-Host "`t",$_.{Base Code},$_.District,$_.Cnt
}
# need to write out last file
$data | Out-File "$path$csv.csv"

Open in new window

There is some debug/progress output to the console included, though the script should be fast.
0
 
LVL 1

Author Comment

by:csePixelated
ID: 41735185
Qlemo, a ps1 script will indeed work, where do I plug in what table on the database I am referencing?
can the ip address be used in place of the servers name?
what do i put if it is the default sql instance?
sorry this is very new to me.
0
Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

 
LVL 68

Expert Comment

by:Qlemo
ID: 41735207
The table name I used is tst_EE. Just replace it. Or do you want to have a variable for it too?
0
 
LVL 1

Author Comment

by:csePixelated
ID: 41735218
No variable needed, I was simply confused, ty.
0
 
LVL 1

Author Comment

by:csePixelated
ID: 41735236
I am very new to Ps1 script, so i apologize in advance for all the questions.
ill admit i did not put the precise data down as the information i am dealing with cannot be posted to any degree, however i see no reason not to update you on the actual structure.

I believe i changed all the variables, I am running the .ps1 script from a windows 7 workstation, do i need to be running it from an sql server for it to work?

$ConStr = 'Server=cdiserv2\Instance; Database=MISCELLANEOUS; Integrated Security=sspi'
$path = 'Z:\Sync\Dealer Account Billing Sheets\LINE CODES\'

# Get data including target filename (without extenstion and path)
$set = new-object system.data.dataset
(new-object System.Data.SqlClient.SqlDataAdapter (@"
  with
    vals as (select [LineCode], Name, cnt=count(*) from "Line Codes" group by Name, [LineCode]),
    rnks as (select [LineCode], Name, rnk=dense_rank() over (partition by Name order by cnt desc) from vals)
  select filename=rnks.[LineCode], vals.Name, vals.[LineCode], vals.cnt
    from vals join rnks on vals.Name = rnks.Name and rnks.rnk = 1
  order by Name, cnt desc
"@, $ConStr)).Fill($set) | out-null

# Process the results
$csv = $null
$data = @()
$set.Tables[0] | % {
  if ($csv -eq $_.filename) {
    $data += $_.{LineCode}
  } else {
    # need to write out the file now
    if ($data) { $data | Out-File "$path$csv.csv" }
    $csv = $_.filename
    $data = @($_.{LineCode})
    Write-Host "Processing Name $($_.Name) -> $csv.csv"
  }
  Write-Host "`t",$_.{LineCode},$_.Name,$_.Cnt
}
# need to write out last file
$data | Out-File "$path$csv.csv"

Open in new window


be advised I do not use the SA login for sql, i use windows authentication
0
 
LVL 68

Assisted Solution

by:Qlemo
Qlemo earned 500 total points
ID: 41735325
The authentication is set up to use Windows Authentication (sspi).
You should be able to run the script on your workstation, at least if having the MSSQL Native Client installed.
0

Featured Post

Optimizing Cloud Backup for Low Bandwidth

With cloud storage prices going down a growing number of SMBs start to use it for backup storage. Unfortunately, business data volume rarely fits the average Internet speed. This article provides an overview of main Internet speed challenges and reveals backup best practices.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Safely Uninstall SQL Server 2008 R2 Express 3 60
Import filename in batch 2 24
SQL Server merge records in one table 2 12
Extract string portion 2 14
How to remove superseded packages in windows w60 or w61 installation media (.wim) or online system to prevent unnecessary space. w60 means Windows Vista or Windows Server 2008. w61 means Windows 7 or Windows Server 2008 R2. There are various …
This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
This Micro Tutorial hows how you can integrate  Mac OSX to a Windows Active Directory Domain. Apple has made it easy to allow users to bind their macs to a windows domain with relative ease. The following video show how to bind OSX Mavericks to …
In this video I am going to show you how to back up and restore Office 365 mailboxes using CodeTwo Backup for Office 365. Learn more about the tool used in this video here: http://www.codetwo.com/backup-for-office-365/ (http://www.codetwo.com/ba…

895 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now