[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?
Solved

Pushing to a multi-dimensional hash?

Posted on 2014-04-07
9
Medium Priority
?
256 Views
Last Modified: 2014-04-08
Hello,

I have the following code which takes the process id from a matched IP Address and looks to find the rest of the flow in the logs file.  Unfortunately the process ID is not unique and is pulling data from other flows as well.  So I decided to match the process ID with the time (hour and minute), that way if it matches the pid and matches the time frame it will consider it as part of the flow.  I have extracted the pid and time into arrays but am having a hard time knowing how to relate them to each other so I can later search using them as the new criteria.  I'm assuming I would have to use a hash but but unsure how I would map to a multi-dimensional hash. Any help would be appreciated.

For a bigger picture of what I'm trying to do, I want to search for the IP "192.168.0.2" in a log file and have it give me all the flow for that IP which I would get by using the PID.  Unfortunately since the PIDs are not completely unique, I could have an IP "192.168.0.4" that has the same IP as "192.168.0.2" so the program currently would give me flows from both IP addresses.

Thank you!

Sample code:
# -------------------------------------------------------
# extract PIDs and Time from lines
# -------------------------------------------------------
my $infile3 = 'output.txt';
my @time;
my @pids;

open (INPUT2, $infile3) or die "Couldn't read $infile3.\n";

while (<INPUT2>) {
	if( /(\d\d):(\d\d):/ ) {
    	push @time, $1 .":" . $2;
		
		if (/ftpd\[(.*?)\]/) {
        	push @pids,  $1;
		}   	
    }

}
close(INPUT2);

# -------------------------------------------------------
# find flow based on PID that was found from criteria
# -------------------------------------------------------

# Put the PIDs into a hash.  Each line which matches will be stored.
my %pids = map { $_ => [] } @pids;


seek INPUT, 0, 0;

# Loop through the lines
while(my $line = <INPUT>) {
    # Look for a PID
    if(my($pid) = $line =~ m{ \[ \s*(\d+) \]: }x) {
        # Push it into the appropriate PID slot if it's on our list
        push @{$pids{$pid}}, $line if $pids{$pid};
    }
}

print Dumper \%pids;

Open in new window


output.txt:

Dec  1 23:59:10 ftp1 ftpd[4690]: FTP LOGIN FROM 192.168.0.2 [192.168.0.2], prod1
Dec  1 23:59:10 ftp1 ftpd[4692]: FTP LOGIN FROM 192.168.0.2 [192.168.0.2], prod
Dec  1 23:59:11 ftp1 ftpd[4693]: FTP LOGIN FROM 192.168.0.2 [192.168.0.2], test1
Dec  1 23:59:14 ftp1 ftpd[4696]: FTP LOGIN FROM 192.168.0.2 [192.168.0.2], test1
Dec  1 23:59:40 ftp1 ftpd[5320]: FTP LOGIN FROM 192.168.0.2 [192.168.0.2], test1

Open in new window


Current hash output:
         
 '4692' => [
                      'Dec  2 23:59:27 ftp1 ftpd[4692]: USER prod1
',
                      'Dec  2 23:59:29 ftp1 ftpd[4692]: PASS password
',
                      'Dec  2 23:59:29 ftp1 ftpd[4692]: FTP LOGIN FROM 192.168.0.2 [192.168.0.2], prod1
',
                      'Dec  2 23:59:33 ftp1 ftpd[4692]: PWD
',
                      'Dec  2 23:59:35 ftp1 ftpd[4692]: CWD /prod/data/
',
                      'Dec  2 23:59:38 ftp1 ftpd[4692]: TYPE Image
',
                      'Dec  2 23:59:40 ftp1 ftpd[4692]: PASV
',
                      'Dec  2 23:59:42 ftp1 ftpd[4692]: NLST
',
                      'Dec  2 23:59:45 ftp1 ftpd[4692]: QUIT
',
                      'Dec  2 23:59:45 ftp1 ftpd[4692]: FTP session closed
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: USER test1
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: PASS password
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: FTP LOGIN FROM 192.168.0.4 [192.168.0.4], test1
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: PWD
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: CWD /test/data/
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: TYPE Image
',
                      'Dec  2 10:39:13 ftp1 ftpd[4692]: PASV
',
                      'Dec  2 10:39:13 ftp1 ftpd[4692]: NLST
',
                      'Dec  2 10:39:13 ftp1 ftpd[4692]: FTP session closed
'
                    ],

Open in new window

0
Comment
Question by:dloszewski
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
9 Comments
 
LVL 84

Expert Comment

by:ozo
ID: 39984811
What would you want the output to be instead?
0
 

Author Comment

by:dloszewski
ID: 39984820
So right now as you can see in the hash output that I provided it is including flows from 23:59 from 192.168.0.2 as well as 10:39 from 192.168.0.4.  I only want the flow from 23:59 from 192.168.0.2 since the 192.168.0.2 is the only IP I"m searching for
0
 
LVL 84

Expert Comment

by:ozo
ID: 39984840
# Put the PIDs into a hash.  Each line which matches will be stored.
my %pids = map { $_ => {} } @pids;

open INPUT,"<Q_28407242.in" or die $!;
seek INPUT, 0, 0;

# Loop through the lines
while(my $line = <INPUT>) {
    # Look for a PID
    print $line;
    if(my($pid,$ip) = $line =~ m{ \[ \s*(\d+) \]:\s+FTP\s+LOGIN\s+FROM\s+(\S+)}x) {
        # Push it into the appropriate PID slot if it's on our list
        push @{$pids{$pid}{$ip}}, $line if $pids{$pid};
    }
}

print Dumper \%pids;

$VAR1 = {
          '4696' => {},
          '4693' => {},
          '4690' => {},
          '4692' => {
                      '192.168.0.2' => [
                                         'Dec  2 23:59:29 ftp1 ftpd[4692]: FTP LOGIN FROM 192.168.0.2 [192.168.0.2], prod1
'
                                       ],
                      '192.168.0.4' => [
                                         'Dec  2 10:39:12 ftp1 ftpd[4692]: FTP LOGIN FROM 192.168.0.4 [192.168.0.4], test1
'
                                       ]
                    },
          '5320' => {}
        };
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:dloszewski
ID: 39984871
But how about when it goes to input the following:

Dec 2 23:59:35 ftp1 ftpd[4692]: CWD /prod/data/

How will it know if it belongs to 192.168.0.2 or 192.168.0.4 since they have the same pid ?

I appreciate your help!
0
 

Author Comment

by:dloszewski
ID: 39985894
Ok, so I created two separate hashes, one to references the pids and the times and the second to reference the pids and all the lines that match that pid.  

How would I take out all the hash elements from the second hash that does not match the time for the key in the first one?

so for example, below you see 4696 occurs at 23:59 but the has below that has lines with a time of 10:39, how would I take out all the 10:39 out?:

$VAR1 = {
          '4696' => '23:59',
          '4693' => '23:59',
          '4690' => '23:59',
          '4692' => '23:59',
          '5320' => '23:59'
        };

Open in new window


'4692' => [
                      'Dec  2 23:59:27 ftp1 ftpd[4692]: USER prod1
',
                      'Dec  2 23:59:29 ftp1 ftpd[4692]: PASS password
',
                      'Dec  2 23:59:29 ftp1 ftpd[4692]: FTP LOGIN FROM 192.168.0.2 [192.168.0.2], prod1
',
                      'Dec  2 23:59:33 ftp1 ftpd[4692]: PWD
',
                      'Dec  2 23:59:35 ftp1 ftpd[4692]: CWD /prod/data/
',
                      'Dec  2 23:59:38 ftp1 ftpd[4692]: TYPE Image
',
                      'Dec  2 23:59:40 ftp1 ftpd[4692]: PASV
',
                      'Dec  2 23:59:42 ftp1 ftpd[4692]: NLST
',
                      'Dec  2 23:59:45 ftp1 ftpd[4692]: QUIT
',
                      'Dec  2 23:59:45 ftp1 ftpd[4692]: FTP session closed
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: USER test1
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: PASS password
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: FTP LOGIN FROM 192.168.0.4 [192.168.0.4], test1
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: PWD
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: CWD /test/data/
',
                      'Dec  2 10:39:12 ftp1 ftpd[4692]: TYPE Image
',
                      'Dec  2 10:39:13 ftp1 ftpd[4692]: PASV
',
                      'Dec  2 10:39:13 ftp1 ftpd[4692]: NLST
',
                      'Dec  2 10:39:13 ftp1 ftpd[4692]: FTP session closed
'
                    ],

Open in new window

0
 
LVL 84

Accepted Solution

by:
ozo earned 2000 total points
ID: 39985990
# Loop through the lines
my $ip;
while( <INPUT> ){
    # Look for a PID
    print;
    $ip=$1 if  m{FTP\s+LOGIN\s+FROM\s+(\S+)};
    if(my($pid) = m{ \[ \s*(\d+) \]: }x) {
        # Push it into the appropriate PID slot if it's on our list
        push @{$pids{$pid}{$ip}}, $_ if $pids{$pid};
    }
    undef $ip if m{FTP session closed};
}

print Dumper \%pids;


$VAR1 = {
          '4696' => {},
          '4693' => {},
          '4690' => {},
          '4692' => {
                      '' => [
                              'Dec  2 23:59:27 ftp1 ftpd[4692]: USER prod1
',
                              'Dec  2 23:59:29 ftp1 ftpd[4692]: PASS password
',
                              'Dec  2 10:39:12 ftp1 ftpd[4692]: USER test1
',
                              'Dec  2 10:39:12 ftp1 ftpd[4692]: PASS password
'
                            ],
                      '192.168.0.2' => [
                                         'Dec  2 23:59:29 ftp1 ftpd[4692]: FTP LOGIN FROM 192.168.0.2 [192.168.0.2], prod1
',
                                         'Dec  2 23:59:33 ftp1 ftpd[4692]: PWD
',
                                         'Dec  2 23:59:35 ftp1 ftpd[4692]: CWD /prod/data/
',
                                         'Dec  2 23:59:38 ftp1 ftpd[4692]: TYPE Image
',
                                         'Dec  2 23:59:40 ftp1 ftpd[4692]: PASV
',
                                         'Dec  2 23:59:42 ftp1 ftpd[4692]: NLST
',
                                         'Dec  2 23:59:45 ftp1 ftpd[4692]: QUIT
',
                                         'Dec  2 23:59:45 ftp1 ftpd[4692]: FTP session closed
'
                                       ],
                      '192.168.0.4' => [
                                         'Dec  2 10:39:12 ftp1 ftpd[4692]: FTP LOGIN FROM 192.168.0.4 [192.168.0.4], test1
',
                                         'Dec  2 10:39:12 ftp1 ftpd[4692]: PWD
',
                                         'Dec  2 10:39:12 ftp1 ftpd[4692]: CWD /test/data/
',
                                         'Dec  2 10:39:12 ftp1 ftpd[4692]: TYPE Image
',
                                         'Dec  2 10:39:13 ftp1 ftpd[4692]: PASV
',
                                         'Dec  2 10:39:13 ftp1 ftpd[4692]: NLST
',
                                         'Dec  2 10:39:13 ftp1 ftpd[4692]: FTP session closed
'
                                       ]
                    },
          '5320' => {}
        };
0
 

Author Comment

by:dloszewski
ID: 39986148
not working for me, I get the following:

$VAR1 = [
          '4690',
          '4692',
          '4693',
          '4696',
          '5320'
        ];
$VAR1 = [
          '23:59',
          '23:59',
          '23:59',
          '23:59',
          '23:59'
        ];
Not a HASH reference at C:/Users/dloszews/workspace/ftp_search/ftp_search.pl line 94, <INPUT> line 440955.

Open in new window


this is line 94:

   push @{$pids{$pid}{$ip}}, $_ if $pids{$pid};

Open in new window

0
 
LVL 84

Expert Comment

by:ozo
ID: 39986160
How are you setting %pids?

In http:#a39984840, I used
my %pids = map { $_ => {} } @pids;
0
 

Author Comment

by:dloszewski
ID: 39986182
looks to have done it, thanks
0

Featured Post

On Demand Webinar: Networking for the Cloud Era

Ready to improve network connectivity? Watch this webinar to learn how SD-WANs and a one-click instant connect tool can boost provisions, deployment, and management of your cloud connection.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Windows Script Host (WSH) has been part of Windows since Windows NT4. Windows Script Host provides architecture for building dynamic scripts that consist of a core object model, scripting hosts, and scripting engines. The key components of Window…
Whether you’re a college noob or a soon-to-be pro, these tips are sure to help you in your journey to becoming a programming ninja and stand out from the crowd.
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.
The goal of the video will be to teach the user the concept of local variables and scope. An example of a locally defined variable will be given as well as an explanation of what scope is in C++. The local variable and concept of scope will be relat…
Suggested Courses

656 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question