Solved

Howto: Capture data stream with telnet session on Linux Server

Posted on 2009-03-31
31
1,771 Views
Last Modified: 2013-12-16
Say, what telnet application on linux system (not running GUI) listening to a specific port can do following:

Quintum device sends CDR records via port: It requires a CHR Line feed to tell start sending its buffer.
The data is to be written to a file. Or even better still to a database in PostGreSQL.

any ideas please?
0
Comment
Question by:shaunwingin
  • 15
  • 10
  • 3
  • +1
31 Comments
 
LVL 76

Expert Comment

by:arnold
Comment Utility
You can use Socket in perl, C to effectively programatically handle the establishment of the connection and exchange data as you see fit.
You could look into using expect.
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
Hi,

Netcat can do it. Do I understand correctly, you want to connect to the quintum device on a port, and then send a CR, and record everything that is replied back into a file? If so:

echo | netcat host port > /tmp/file

E.g. if the host is 192.168.1.1 and it listens on 12345

echo | netcat 192.168.1.1 12345 > /tmp/file

Will record what it sends back to /tmp/file



0
 

Author Comment

by:shaunwingin
Comment Utility
Uberpappa, our description is 100% corrrect. How will netcat send a CR to the Quitnum to tell it to start sending?
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
THe command says:

echo | netcat host port > /tmp/file

Broken down:

echo

Which prints a CR

echo | netcat host port

Send the output of echo (CR) to netcat to forward to the remote host

echo | netcat host port > /tmp/file

and send whatever is returned to a file
0
 

Author Comment

by:shaunwingin
Comment Utility
tx - will try it. Is there any way to write the file into a database? The columns are fixed width.
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
Yes, it shouldn't be too difficult to do something like:

echo | netcat host port | populatedb.sh

Can you post a snippet of the file, so we can get a better understanding of what it contains?
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
Also, does it have to be PostgreSQL, or will MySQL also do?
0
 

Author Comment

by:shaunwingin
Comment Utility
It has to be PostgreSQL - I'm afraid... Can it be done?
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
It can, just a longer process... give me a sec and I'll post an example script...
0
 

Author Comment

by:shaunwingin
Comment Utility
How can I see if its listening.... Also the data is sent to a specific public ip address  - but comes from a dynamic ip address - both are public. How can I run nc to listen on the public ip address for all data that comes in on the port?
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
Create a database, and create a table with all the fields you require in the database:

CREATE TABLE quintum (
   Field1 CHAR(5),
   FIeld2 CHAR(10),
   Field3 CHAT(2)
);

As an example. The fields should correspond with your fixed width data.

Then, export to a file:

echo |netcat host port > /tmp/rawdata.txt

Inside the database, run the following script to create a temporary table and import the bulk text (not split into fields) in it:

CREATE TABLE temp_table (Line VARCHAR(200));

where 200 is large enough to contain the data from one line of the input. Then import the text file into it:

COPY temp_table (line) from '/tmp/rawdata.txt';

Now Split it into fields from the temp table to the new table:

INSERT INTO quintum SELECT SUBSTRING(Line FROM 1 FOR 5), SUBSTRING(Line FROM 6 FOR 10), SUBSTRING(Line FROM 17 FOR 2) from temp_table.

Then drop the temp table:

DROP TABLE temp_table;

The SUBSTRING statements are used to split the text into fields based on their fixed-widht placement.

Once you have the queries working, put them all into a .sql file, one after the other, then do:

echo | netcat host port > /tmp/rawdata.txt
psql database < /path/to/your/script.sql

And you're set to go.
0
 

Author Comment

by:shaunwingin
Comment Utility
Wow this looks a G8 script tx!
0
 

Author Comment

by:shaunwingin
Comment Utility
The file is created but no data is written...
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
Oh, that changes things a bit! :)

On the listening host, run:

netcat -l -p 12345 > /tmp/file

It will then record everything that is sent to port 12345 of that host to the file /tmp/file.
0
Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
Let me just make sure that I have this correct now:

Your quintum box (lets call it quintum.yourdomain.com listens on a port for a CR. Once it receives it, it opens a new connection to another host, call it otherhost.yourdomain.com which is supposed to be listening on a port for the result?
0
 

Author Comment

by:shaunwingin
Comment Utility
 -l      Used to specify that nc should listen for an incoming connection rather than initiate a connection to a remote host.  It is an
             error to use this option in conjunction with the -p, -s, or -z options.  Additionally, any timeouts specified with the -w option
             are ignored.
0
 

Author Comment

by:shaunwingin
Comment Utility
I tried:
netcat -l -p 9002 but returned an error....
Please see page 4 of attached file for process that Quintum uses. I've left password blank.
CDR.pdf
0
 
LVL 76

Expert Comment

by:arnold
Comment Utility
echo sends a lf/cr not just a cr.
try this modification to Uberpappa's info:
perl -e ' print  scalar chr(13);' | netcat 192.168.1.1 12345 > /tmp/file
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
Thanks Arnold, forgot about that.

After viewing that doc, the whole world around this question has changed. As I understand, the client connects, the server provides a welcome message, and asks for a password, the client supplies the password, the client supplies it, and then its IP and unit name and the server then responds with a CSV file, not a fixed widht file.

So back to the drawing board, this screams netcat with expect, or as suggested initially arnold, a perl/C script.

It does make the database import a lot easier though.

Let me try and dish up something with netcat and expect quickly.

0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
On second thought, it should be OK if we just send the password and other info sequentially:
#!/bin/bash
 

# CHANGE FROM HERE:
 

remoreserver="192.168.1.1"

remoteport=9002

rawdatafile=/tmp/rawdata.csv

tenorip="192.168.2.1"

tenorunit="tenor1"

password="secret"
 

# TO HERE
 

(

  echo "${password}"

  echo "${tenorip},${tenorunit}"

) | netcat remoteserver remoteport > ${rawdatafile}

Open in new window

0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
Typo:
#!/bin/bash

 

# CHANGE FROM HERE:

 

remoteserver="192.168.1.1"

remoteport=9002

rawdatafile=/tmp/rawdata.csv

tenorip="192.168.2.1"

tenorunit="tenor1"

password="secret"

 

# TO HERE

 

(

  echo "${password}"

  echo "${tenorip},${tenorunit}"

) | netcat ${remoteserver} ${remoteport} > ${rawdatafile}

Open in new window

0
 
LVL 76

Expert Comment

by:arnold
Comment Utility
http://perldoc.perl.org/perlipc.html#Sockets%3a-Client%2fServer-Communication

Using a perl script to connect and get the data will also make the option of having everything done within a single process available.

The link above provides different examples of using socket to establish a single connection or a pair of connection.

Give it a try, you will likely find other uses for the knowledge you get.

There is a Telnet module, Net::Telnet that you could use as well if you do not want to get into Socket setup etc.
0
 

Author Comment

by:shaunwingin
Comment Utility
tx Uberppa and Arnold. Uberppa, I'm not receiving data in the file. I ran tcpdump -i eth0:1 port 9002 -w isca
and getting a packet from the Quintum. Any ideas please? Perhaps you can setup a test on your side and see - I can put in your ip into the Quintum. Tx for teh great input.
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
we can try... mail me at uberpappa123 at gmail dot com
0
 
LVL 51

Expert Comment

by:ahoffmann
Comment Utility
how about using an expect script to send and receive the data from you special port, then simply call this script inside a `script' shell, see
  man script
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
As discussed on msn, I've written a script in php that will:

1) Telnet to a remote Server, and authenticate with the given password
2) Parse the Tenor name passed by the server
3) Check the last CDR that was received for this server, and send it to server to instruct it to only send from that CDR onwards
3) Read the CDR's that are returned, and log them into Postgres.

To make it work:

1) Create a database in postgres
2) Create a user and configure the permissions on the db as well as in pg_hba.conf so that the user can create records in the db.
3) Run the create.sql script against the db to create the table. (psql dbname < create.sql)
4) Change the first line in the importcdr script, which reads #!/usr/bin/php to point to wherever your php is on the server. You can check this with "which php".

Then run it as follows:

./importcdr serverip serverport <password>

If no password is configure, ommit it from the command. The port is 9003 and the serverip is the one you gave me on msn.

Hope it works! Let me know if anything on your side causes problems. This side its working 100%.

Scripts to follow......
0
 
LVL 7

Expert Comment

by:Morne Lategan
Comment Utility
The sql create script: create.sql

CREATE TABLE cdr (

	ip VARCHAR(20),

	tenor VARCHAR(50),

	seq INTEGER,

	called_number VARCHAR(50),

	duration_in_seconds INTEGER,

	time_initiated VARCHAR(20),

	time_connected VARCHAR(20),

	time_disconnected VARCHAR(20),

	disconnect_cause INTEGER,

	local_ip VARCHAR(20),

	remote_ip VARCHAR(20),

	originating_trunk_id VARCHAR(20),

	call_type INTEGER,

	call_number_type INTEGER,

	incoming_line INTEGER,

	incoming_channel INTEGER,

	outgoing_line INTEGER,

	outgoing_channel INTEGER,

	auto_switch_time VARCHAR(20),

	auto_switch_duration VARCHAR(20),

	bad_quality_events INTEGER,

	auto_switching_flag INTEGER,

	PRIMARY KEY (ip,tenor,seq,time_initiated)

);

Open in new window

0
 
LVL 7

Accepted Solution

by:
Morne Lategan earned 500 total points
Comment Utility
And the imprortcdr script
#!/usr/bin/php -q

<?php

	//-----------------------------------------------

	// DB CONFIGURATION

	//-----------------------------------------------

	define("DB_HOST","localhost");

	define("DB_DATABASE","somedatabase");

	define("DB_USER","someuser");

	define("DB_PWD","secret");
 

	//-----------------------------------------------

	// Connect to the DB

	//-----------------------------------------------

	$dbconn = pg_connect(

			" host="     .DB_HOST.

			" dbname="  .DB_DATABASE.

			" user="    .DB_USER.

			" password=".DB_PWD 

		);
 

	if (!$dbconn) {

		die('Unable to connect to DB: '.pg_last_error());

	}
 

	//-----------------------------------------------

	// print_usage ()

	//-----------------------------------------------

	// Print the script usage

	//-----------------------------------------------

	function print_usage () {

		echo "Usage: $argv[0] host port <password>\n";

	}
 

	//-----------------------------------------------

	// strip_crlf ()

	//-----------------------------------------------

	// Strip a string of CR and LF characters

	//-----------------------------------------------

	function strip_crlf ($line) {

		return str_replace("\r","",str_replace("\n","",$line));

	}
 

	//-----------------------------------------------

	// process_line ()

	//-----------------------------------------------

	// Insert a CDR line into the database

	//-----------------------------------------------

	function process_line($ip,$tenor,$line) {

		global $dbconn;

		$row=array();

		// Split the CSV line into db fields

		list(	

			$row["seq"],

			$row["called_number"],

			$row["duration_in_seconds"],

			$row["time_initiated"],

			$row["time_connected"],

			$row["time_disconnected"],

			$row["disconnect_cause"],

			$row["local_ip"],

			$row["remote_ip"],

			$row["originating_trunk_id"],

			$row["call_type"],

			$row["call_number_type"],

			$row["incoming_line"],

			$row["incoming_channel"],

			$row["outgoing_line"],

			$row["outgoing_channel"],

			$row["auto_switch_time"],

			$row["auto_switch_duration"],

			$row["bad_quality_events"],

			$row["auto_switching_flag"]

		) = split(",",strip_crlf($line));
 

		// And add the two thats not in there

                $row["ip"] = $ip;

                $row["tenor"] = $tenor;
 

		// Check to see if record already in DB:

		$query  = "SELECT COUNT(seq) FROM cdr WHERE ";

		$query .= "ip = '".$row["ip"]."' and ";

		$query .= "tenor = '".$row["tenor"]."' and ";

		$query .= "seq = ".$row["seq"]." and ";

		$query .= "time_initiated = '".$row["time_initiated"]."'";

		$qres   = pg_query($dbconn,$query);

		if ($qres) {

			$exists = pg_fetch_row($qres);

			// If at least one record exist

			if ($exists[0] > 0) {

				echo "Skipping seq ".$row["seq"]." already imported.\n";

			} else {

				// Insert it

				$res = pg_insert($dbconn,'cdr',$row);

				if (!res) {

					echo "Seq ".$row["seq"].": INSERTION FAILED.\n";

				} else {

					echo "Seq ".$row["seq"]." inserted.\n";

				}

			}

		} else {

			echo "Failing to query database for existing records!\n";

		}

	}
 

	//-----------------------------------------------

	// MAIN EXECUTION SECTION

	//-----------------------------------------------
 

	// Check arguments

	if ($argc < 3) {

	  print_usage();

	  exit;

	}

	$host=$argv[1];

	$port=$argv[2];

	if ($argc = 4) {

		$password=$argv[3];

	} else {

		$password="";

	}
 

	// Execute the telnet command, hooking pipes to stdin and stdout

	// Record stderror into a file

	$descriptorspec = array(

		0 => array("pipe", "r"),

		1 => array("pipe", "w"),

		2 => array("file", "/tmp/cdr-telnet-errors.log", "a")

	);

	$process = proc_open('/usr/bin/telnet '.$host.' '.$port,$descriptorspec,$pipes);
 

	// If the command succeeded:

	if (is_resource($process)) {

		$line = "";

		// Keep reading the command output char by char

		// until it contains "Password:"

		while (!feof($pipes[1]) and !strpos(strtolower($line),"password:")) {

			$line .= fread($pipes[1],1);

		}

		// Wait for a second

		sleep(1);

		// Then send the password

		fwrite($pipes[0],$password."\r\n");

		$line="";

		// Keep reading line by line until we have a comma in the

		// input

		while (!feof($pipes[1]) and !strpos($line,",")) {

			$line = fgets($pipes[1]);

		}

		// Then split it into ip and tenor

		list($ip,$tenor) = split(",",strip_crlf($line));

		// Report the IP/Tenor we're capturing for:

		echo "IP: $ip\n";

		echo "Tenor: $tenor\n";

		echo "Starting capture...\n";
 

		// Get the last successful CDR seq Number for

		// the IP/Tenor combination
 

		$res = pg_query($dbconn,'SELECT MAX(seq) FROM cdr WHERE ip=\''.$ip.'\' AND tenor=\''.$tenor.'\'');

		if ($res) {

			$lastcdrrow = pg_fetch_row($res);

			$lastcdr = $lastcdrrow[0];

			echo "Last CDR for tenor is: $lastcdr\n";

		} else {

			$lastcdr = "";

			echo "Warning: Unable to query last CDR seq for Tenor\n";

		}

		// Send the last successful sequence number

		// After a tenor reset, you'll need to mess

		// around here to get all the records.
 

		fwrite($pipes[0],$lastcdr."\r\n");

		// Read and process the reply line by line

		while (!feof($pipes[1])) {

			$line = fgets($pipes[1]);

			process_line($ip,$tenor,$line);

		}

		// Then close the command pipes and the command itself

		fclose($pipes[0]);

		fclose($pipes[1]);

		$return_value = proc_close($process);

	} else {

		// If the command failed:

		echo "Error: Telnet command failed. See /tmp/cdr-telnet-errors.log";

	}

?>

Open in new window

0
 

Author Comment

by:shaunwingin
Comment Utility
Most impressive and well written and commented script. Looking fwd to implementing as soon as get the time as not a Linux fundi and will need a good amount of time to implement....
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

How many times have you wanted to quickly do the same thing to a list but found yourself typing it again and again? I first figured out a small time saver with the up arrow to recall the last command but that can only get you so far if you have a bi…
If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now