sunny82
asked on
how to extract data from a table into a fixed length file
Hi,
I have to extract data from table A. Now A has about 200 columns. Now when I am writing data out into a file, every field should have a pre-specified length. How should I do it in perl? Any ideas?
like for example - column id should be 10 characters, name should be 30 characters and address should be 40. So in the output file, name should start from 11th position and address from 1st position
Regards,
Sayantan
I have to extract data from table A. Now A has about 200 columns. Now when I am writing data out into a file, every field should have a pre-specified length. How should I do it in perl? Any ideas?
like for example - column id should be 10 characters, name should be 30 characters and address should be 40. So in the output file, name should start from 11th position and address from 1st position
Regards,
Sayantan
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi,
Many thanks for your response. I used that and it works fine. But the problem is, I have over 200 columns in the table and for each column, I have to write sprintf("%10s%-30s%-40s", $id, $name, $address), corresponding to their widths, which becomes very unwieldy and difficult to maintin.
Is there a way to use config file such that I only need to change the config file once(if there is any field width change in any column) and I can generate the file accordingly?
The config file should just have column name and its width.
Many thanks for your response. I used that and it works fine. But the problem is, I have over 200 columns in the table and for each column, I have to write sprintf("%10s%-30s%-40s", $id, $name, $address), corresponding to their widths, which becomes very unwieldy and difficult to maintin.
Is there a way to use config file such that I only need to change the config file once(if there is any field width change in any column) and I can generate the file accordingly?
The config file should just have column name and its width.
If you put fields in this format in a file called config.txt:
id,10
name,30
address,40
You could then do your processing like this:open CONFIG, "<config.txt" or die "Can't open config file!";
while (<CONFIG>)
{
chomp;
($name, $length) = split(',');
$format .= "%${length}s"; # This will left-justify all fields
}
# Populate the @values array with the column values of a
# a row of your table. I'll just hard code it here, but you
# will probably read a record from the table.
@values = (123, 'My Name', 'My Address');
printf($format, @values);
;
ASKER
I am getting the output like this
%10 s%30 s123
below is my code
use strict;
my $core_config = "$ENV{'CFG'}/menir-core-co nfig-test. txt";
my $name;
my $length;
my $format;
open CONFIG, "<$core_config" or die "Can't open config file!";
while (<CONFIG>)
{
chomp;
($name, $length) = split(',');
$format .= "%${length}s"; # This will left-justify all fields
}
my $outfile = "$ENV{'DATA'}/CORE.txt";
my @row = (123, 'My Name', 'My Address');
printf($format, @row);
-------------------------- ---------- ----------
this is my config ->
id,10
name,30
address,10
%10 s%30 s123
below is my code
use strict;
my $core_config = "$ENV{'CFG'}/menir-core-co
my $name;
my $length;
my $format;
open CONFIG, "<$core_config" or die "Can't open config file!";
while (<CONFIG>)
{
chomp;
($name, $length) = split(',');
$format .= "%${length}s"; # This will left-justify all fields
}
my $outfile = "$ENV{'DATA'}/CORE.txt";
my @row = (123, 'My Name', 'My Address');
printf($format, @row);
--------------------------
this is my config ->
id,10
name,30
address,10
Hi Sunny,
Your version of the code works for me. All I changed was:
my $core_config = "$ENV{'CFG'}/menir-core-co nfig-test. txt";
to:
my $core_config = 'config.txt';
so it would work in my environment, and I got this output:
Question:
Does my version of the code work for you (if you just copy/paste it from my previous post)?
tel2
Your version of the code works for me. All I changed was:
my $core_config = "$ENV{'CFG'}/menir-core-co
to:
my $core_config = 'config.txt';
so it would work in my environment, and I got this output:
123 My Name My Address
Question:
Does my version of the code work for you (if you just copy/paste it from my previous post)?
tel2
Specifying the field width as part of the csv data is ok, but doesn't scale well.
A better approach would be to use an actual config file for the format specification. There are a number of config modules to choose from on CPAN. Here's an example using a YAML module/format.
config.yml
Test script:
Output:
D:\test>sunny82.pl
123 My Name My Address 12.50
A better approach would be to use an actual config file for the format specification. There are a number of config modules to choose from on CPAN. Here's an example using a YAML module/format.
config.yml
fields: # list of fields in desired order
- id
- name
- address
- salary
--- # field format specifications
id:
width: 10
justify: '-'
type: d
name:
width: 30
type: s
address:
width: 30
type: s
salary:
width: 12
type: f
precision: .2
Test script:
#!/usr/bin/perl
use 5.010;
use strict;
use warnings;
use YAML::Tiny;
my $conf = YAML::Tiny->read( 'config.yml' );
my $fields = $conf->[0]{fields};
my $format;
foreach my $field ( @$fields ) {
no warnings 'uninitialized';
$format .= '%' . $conf->[1]{$field}{justify}
. $conf->[1]{$field}{width}
. $conf->[1]{$field}{precision}
. $conf->[1]{$field}{type};
}
my @row = (123, 'My Name', 'My Address', 12.5);
printf($format, @row);
Output:
D:\test>sunny82.pl
123 My Name My Address 12.50
ASKER
Hi,
Sorry for replying a bit late.
@tel2: The output I am getting is "%10 s%30 s 123"
The code is exactly the same as your code, word to word.
Is it because of tis statement ?
$format .= "%${length}s"; # This will left-justify all fields
@FishMonger: @INC does not have YAML::Tiny. Can we use some other module?
Sorry for replying a bit late.
@tel2: The output I am getting is "%10 s%30 s 123"
The code is exactly the same as your code, word to word.
Is it because of tis statement ?
$format .= "%${length}s"; # This will left-justify all fields
@FishMonger: @INC does not have YAML::Tiny. Can we use some other module?
Hi Sunny,
"> The code is exactly the same as your code, word to word."
Perhaps, but the code you pasted above is not the same as mine.
Please copy/paste my code to your machine (again), run it, then copy/paste my code and the results here for analysis.
"> Is it because of tis statement ?
> $format .= "%${length}s"; # This will left-justify all fields"
Dunno. That statement works in my Perl 5.10 Linux environment I'm using, in both your version and my version of my code.
I doubt this is an issue, but what version of Perl are you using? Type "perl -v" and paste the output here.
Thanks.
tel2
"> The code is exactly the same as your code, word to word."
Perhaps, but the code you pasted above is not the same as mine.
Please copy/paste my code to your machine (again), run it, then copy/paste my code and the results here for analysis.
"> Is it because of tis statement ?
> $format .= "%${length}s"; # This will left-justify all fields"
Dunno. That statement works in my Perl 5.10 Linux environment I'm using, in both your version and my version of my code.
I doubt this is an issue, but what version of Perl are you using? Type "perl -v" and paste the output here.
Thanks.
tel2
What operating system are you running, Sunny? I'm assuming UNIX/Linux, below.
Further to my post above, here's an alternative way to write that format line (line #6 in my 2nd post):
$format .= '%' . $length . 's';
If that doesn't work in my script, please run each of the following one-liners from the Linux command line, and post the command and the output here:
perl -e '$length=12;$format .= "%${length}s";print $format'
perl -e '$length=12;$format .= "%" . $length . "s";print $format'
When you copy/paste my code back here for analysis (see my last post for details), please use this UNIX/Linux command to display the code before copying it:
cat -vet the-script-name-of-my-vers ion-of-the -code.pl
Hopefully that will show if there have been any strange characters inserted during the copy/paste to your system.
Further to my post above, here's an alternative way to write that format line (line #6 in my 2nd post):
$format .= '%' . $length . 's';
If that doesn't work in my script, please run each of the following one-liners from the Linux command line, and post the command and the output here:
perl -e '$length=12;$format .= "%${length}s";print $format'
perl -e '$length=12;$format .= "%" . $length . "s";print $format'
When you copy/paste my code back here for analysis (see my last post for details), please use this UNIX/Linux command to display the code before copying it:
cat -vet the-script-name-of-my-vers
Hopefully that will show if there have been any strange characters inserted during the copy/paste to your system.
ASKER