Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1880
  • Last Modified:

Howto: Capture data stream with telnet session on Linux Server

Say, what telnet application on linux system (not running GUI) listening to a specific port can do following:

Quintum device sends CDR records via port: It requires a CHR Line feed to tell start sending its buffer.
The data is to be written to a file. Or even better still to a database in PostGreSQL.

any ideas please?
0
shaunwingin
Asked:
shaunwingin
  • 15
  • 10
  • 3
  • +1
1 Solution
 
arnoldCommented:
You can use Socket in perl, C to effectively programatically handle the establishment of the connection and exchange data as you see fit.
You could look into using expect.
0
 
Morne LateganCommented:
Hi,

Netcat can do it. Do I understand correctly, you want to connect to the quintum device on a port, and then send a CR, and record everything that is replied back into a file? If so:

echo | netcat host port > /tmp/file

E.g. if the host is 192.168.1.1 and it listens on 12345

echo | netcat 192.168.1.1 12345 > /tmp/file

Will record what it sends back to /tmp/file



0
 
shaunwinginAuthor Commented:
Uberpappa, our description is 100% corrrect. How will netcat send a CR to the Quitnum to tell it to start sending?
0
Build your data science skills into a career

Are you ready to take your data science career to the next step, or break into data science? With Springboard’s Data Science Career Track, you’ll master data science topics, have personalized career guidance, weekly calls with a data science expert, and a job guarantee.

 
Morne LateganCommented:
THe command says:

echo | netcat host port > /tmp/file

Broken down:

echo

Which prints a CR

echo | netcat host port

Send the output of echo (CR) to netcat to forward to the remote host

echo | netcat host port > /tmp/file

and send whatever is returned to a file
0
 
shaunwinginAuthor Commented:
tx - will try it. Is there any way to write the file into a database? The columns are fixed width.
0
 
Morne LateganCommented:
Yes, it shouldn't be too difficult to do something like:

echo | netcat host port | populatedb.sh

Can you post a snippet of the file, so we can get a better understanding of what it contains?
0
 
Morne LateganCommented:
Also, does it have to be PostgreSQL, or will MySQL also do?
0
 
shaunwinginAuthor Commented:
It has to be PostgreSQL - I'm afraid... Can it be done?
0
 
Morne LateganCommented:
It can, just a longer process... give me a sec and I'll post an example script...
0
 
shaunwinginAuthor Commented:
How can I see if its listening.... Also the data is sent to a specific public ip address  - but comes from a dynamic ip address - both are public. How can I run nc to listen on the public ip address for all data that comes in on the port?
0
 
Morne LateganCommented:
Create a database, and create a table with all the fields you require in the database:

CREATE TABLE quintum (
   Field1 CHAR(5),
   FIeld2 CHAR(10),
   Field3 CHAT(2)
);

As an example. The fields should correspond with your fixed width data.

Then, export to a file:

echo |netcat host port > /tmp/rawdata.txt

Inside the database, run the following script to create a temporary table and import the bulk text (not split into fields) in it:

CREATE TABLE temp_table (Line VARCHAR(200));

where 200 is large enough to contain the data from one line of the input. Then import the text file into it:

COPY temp_table (line) from '/tmp/rawdata.txt';

Now Split it into fields from the temp table to the new table:

INSERT INTO quintum SELECT SUBSTRING(Line FROM 1 FOR 5), SUBSTRING(Line FROM 6 FOR 10), SUBSTRING(Line FROM 17 FOR 2) from temp_table.

Then drop the temp table:

DROP TABLE temp_table;

The SUBSTRING statements are used to split the text into fields based on their fixed-widht placement.

Once you have the queries working, put them all into a .sql file, one after the other, then do:

echo | netcat host port > /tmp/rawdata.txt
psql database < /path/to/your/script.sql

And you're set to go.
0
 
shaunwinginAuthor Commented:
Wow this looks a G8 script tx!
0
 
shaunwinginAuthor Commented:
The file is created but no data is written...
0
 
Morne LateganCommented:
Oh, that changes things a bit! :)

On the listening host, run:

netcat -l -p 12345 > /tmp/file

It will then record everything that is sent to port 12345 of that host to the file /tmp/file.
0
 
Morne LateganCommented:
Let me just make sure that I have this correct now:

Your quintum box (lets call it quintum.yourdomain.com listens on a port for a CR. Once it receives it, it opens a new connection to another host, call it otherhost.yourdomain.com which is supposed to be listening on a port for the result?
0
 
shaunwinginAuthor Commented:
 -l      Used to specify that nc should listen for an incoming connection rather than initiate a connection to a remote host.  It is an
             error to use this option in conjunction with the -p, -s, or -z options.  Additionally, any timeouts specified with the -w option
             are ignored.
0
 
shaunwinginAuthor Commented:
I tried:
netcat -l -p 9002 but returned an error....
Please see page 4 of attached file for process that Quintum uses. I've left password blank.
CDR.pdf
0
 
arnoldCommented:
echo sends a lf/cr not just a cr.
try this modification to Uberpappa's info:
perl -e ' print  scalar chr(13);' | netcat 192.168.1.1 12345 > /tmp/file
0
 
Morne LateganCommented:
Thanks Arnold, forgot about that.

After viewing that doc, the whole world around this question has changed. As I understand, the client connects, the server provides a welcome message, and asks for a password, the client supplies the password, the client supplies it, and then its IP and unit name and the server then responds with a CSV file, not a fixed widht file.

So back to the drawing board, this screams netcat with expect, or as suggested initially arnold, a perl/C script.

It does make the database import a lot easier though.

Let me try and dish up something with netcat and expect quickly.

0
 
Morne LateganCommented:
On second thought, it should be OK if we just send the password and other info sequentially:
#!/bin/bash
 
# CHANGE FROM HERE:
 
remoreserver="192.168.1.1"
remoteport=9002
rawdatafile=/tmp/rawdata.csv
tenorip="192.168.2.1"
tenorunit="tenor1"
password="secret"
 
# TO HERE
 
(
  echo "${password}"
  echo "${tenorip},${tenorunit}"
) | netcat remoteserver remoteport > ${rawdatafile}

Open in new window

0
 
Morne LateganCommented:
Typo:
#!/bin/bash
 
# CHANGE FROM HERE:
 
remoteserver="192.168.1.1"
remoteport=9002
rawdatafile=/tmp/rawdata.csv
tenorip="192.168.2.1"
tenorunit="tenor1"
password="secret"
 
# TO HERE
 
(
  echo "${password}"
  echo "${tenorip},${tenorunit}"
) | netcat ${remoteserver} ${remoteport} > ${rawdatafile}

Open in new window

0
 
arnoldCommented:
http://perldoc.perl.org/perlipc.html#Sockets%3a-Client%2fServer-Communication

Using a perl script to connect and get the data will also make the option of having everything done within a single process available.

The link above provides different examples of using socket to establish a single connection or a pair of connection.

Give it a try, you will likely find other uses for the knowledge you get.

There is a Telnet module, Net::Telnet that you could use as well if you do not want to get into Socket setup etc.
0
 
shaunwinginAuthor Commented:
tx Uberppa and Arnold. Uberppa, I'm not receiving data in the file. I ran tcpdump -i eth0:1 port 9002 -w isca
and getting a packet from the Quintum. Any ideas please? Perhaps you can setup a test on your side and see - I can put in your ip into the Quintum. Tx for teh great input.
0
 
Morne LateganCommented:
we can try... mail me at uberpappa123 at gmail dot com
0
 
ahoffmannCommented:
how about using an expect script to send and receive the data from you special port, then simply call this script inside a `script' shell, see
  man script
0
 
Morne LateganCommented:
As discussed on msn, I've written a script in php that will:

1) Telnet to a remote Server, and authenticate with the given password
2) Parse the Tenor name passed by the server
3) Check the last CDR that was received for this server, and send it to server to instruct it to only send from that CDR onwards
3) Read the CDR's that are returned, and log them into Postgres.

To make it work:

1) Create a database in postgres
2) Create a user and configure the permissions on the db as well as in pg_hba.conf so that the user can create records in the db.
3) Run the create.sql script against the db to create the table. (psql dbname < create.sql)
4) Change the first line in the importcdr script, which reads #!/usr/bin/php to point to wherever your php is on the server. You can check this with "which php".

Then run it as follows:

./importcdr serverip serverport <password>

If no password is configure, ommit it from the command. The port is 9003 and the serverip is the one you gave me on msn.

Hope it works! Let me know if anything on your side causes problems. This side its working 100%.

Scripts to follow......
0
 
Morne LateganCommented:
The sql create script: create.sql

CREATE TABLE cdr (
	ip VARCHAR(20),
	tenor VARCHAR(50),
	seq INTEGER,
	called_number VARCHAR(50),
	duration_in_seconds INTEGER,
	time_initiated VARCHAR(20),
	time_connected VARCHAR(20),
	time_disconnected VARCHAR(20),
	disconnect_cause INTEGER,
	local_ip VARCHAR(20),
	remote_ip VARCHAR(20),
	originating_trunk_id VARCHAR(20),
	call_type INTEGER,
	call_number_type INTEGER,
	incoming_line INTEGER,
	incoming_channel INTEGER,
	outgoing_line INTEGER,
	outgoing_channel INTEGER,
	auto_switch_time VARCHAR(20),
	auto_switch_duration VARCHAR(20),
	bad_quality_events INTEGER,
	auto_switching_flag INTEGER,
	PRIMARY KEY (ip,tenor,seq,time_initiated)
);

Open in new window

0
 
Morne LateganCommented:
And the imprortcdr script
#!/usr/bin/php -q
<?php
	//-----------------------------------------------
	// DB CONFIGURATION
	//-----------------------------------------------
	define("DB_HOST","localhost");
	define("DB_DATABASE","somedatabase");
	define("DB_USER","someuser");
	define("DB_PWD","secret");
 
	//-----------------------------------------------
	// Connect to the DB
	//-----------------------------------------------
	$dbconn = pg_connect(
			" host="     .DB_HOST.
			" dbname="  .DB_DATABASE.
			" user="    .DB_USER.
			" password=".DB_PWD 
		);
 
	if (!$dbconn) {
		die('Unable to connect to DB: '.pg_last_error());
	}
 
	//-----------------------------------------------
	// print_usage ()
	//-----------------------------------------------
	// Print the script usage
	//-----------------------------------------------
	function print_usage () {
		echo "Usage: $argv[0] host port <password>\n";
	}
 
	//-----------------------------------------------
	// strip_crlf ()
	//-----------------------------------------------
	// Strip a string of CR and LF characters
	//-----------------------------------------------
	function strip_crlf ($line) {
		return str_replace("\r","",str_replace("\n","",$line));
	}
 
	//-----------------------------------------------
	// process_line ()
	//-----------------------------------------------
	// Insert a CDR line into the database
	//-----------------------------------------------
	function process_line($ip,$tenor,$line) {
		global $dbconn;
		$row=array();
		// Split the CSV line into db fields
		list(	
			$row["seq"],
			$row["called_number"],
			$row["duration_in_seconds"],
			$row["time_initiated"],
			$row["time_connected"],
			$row["time_disconnected"],
			$row["disconnect_cause"],
			$row["local_ip"],
			$row["remote_ip"],
			$row["originating_trunk_id"],
			$row["call_type"],
			$row["call_number_type"],
			$row["incoming_line"],
			$row["incoming_channel"],
			$row["outgoing_line"],
			$row["outgoing_channel"],
			$row["auto_switch_time"],
			$row["auto_switch_duration"],
			$row["bad_quality_events"],
			$row["auto_switching_flag"]
		) = split(",",strip_crlf($line));
 
		// And add the two thats not in there
                $row["ip"] = $ip;
                $row["tenor"] = $tenor;
 
		// Check to see if record already in DB:
		$query  = "SELECT COUNT(seq) FROM cdr WHERE ";
		$query .= "ip = '".$row["ip"]."' and ";
		$query .= "tenor = '".$row["tenor"]."' and ";
		$query .= "seq = ".$row["seq"]." and ";
		$query .= "time_initiated = '".$row["time_initiated"]."'";
		$qres   = pg_query($dbconn,$query);
		if ($qres) {
			$exists = pg_fetch_row($qres);
			// If at least one record exist
			if ($exists[0] > 0) {
				echo "Skipping seq ".$row["seq"]." already imported.\n";
			} else {
				// Insert it
				$res = pg_insert($dbconn,'cdr',$row);
				if (!res) {
					echo "Seq ".$row["seq"].": INSERTION FAILED.\n";
				} else {
					echo "Seq ".$row["seq"]." inserted.\n";
				}
			}
		} else {
			echo "Failing to query database for existing records!\n";
		}
	}
 
	//-----------------------------------------------
	// MAIN EXECUTION SECTION
	//-----------------------------------------------
 
	// Check arguments
	if ($argc < 3) {
	  print_usage();
	  exit;
	}
	$host=$argv[1];
	$port=$argv[2];
	if ($argc = 4) {
		$password=$argv[3];
	} else {
		$password="";
	}
 
	// Execute the telnet command, hooking pipes to stdin and stdout
	// Record stderror into a file
	$descriptorspec = array(
		0 => array("pipe", "r"),
		1 => array("pipe", "w"),
		2 => array("file", "/tmp/cdr-telnet-errors.log", "a")
	);
	$process = proc_open('/usr/bin/telnet '.$host.' '.$port,$descriptorspec,$pipes);
 
	// If the command succeeded:
	if (is_resource($process)) {
		$line = "";
		// Keep reading the command output char by char
		// until it contains "Password:"
		while (!feof($pipes[1]) and !strpos(strtolower($line),"password:")) {
			$line .= fread($pipes[1],1);
		}
		// Wait for a second
		sleep(1);
		// Then send the password
		fwrite($pipes[0],$password."\r\n");
		$line="";
		// Keep reading line by line until we have a comma in the
		// input
		while (!feof($pipes[1]) and !strpos($line,",")) {
			$line = fgets($pipes[1]);
		}
		// Then split it into ip and tenor
		list($ip,$tenor) = split(",",strip_crlf($line));
		// Report the IP/Tenor we're capturing for:
		echo "IP: $ip\n";
		echo "Tenor: $tenor\n";
		echo "Starting capture...\n";
 
		// Get the last successful CDR seq Number for
		// the IP/Tenor combination
 
		$res = pg_query($dbconn,'SELECT MAX(seq) FROM cdr WHERE ip=\''.$ip.'\' AND tenor=\''.$tenor.'\'');
		if ($res) {
			$lastcdrrow = pg_fetch_row($res);
			$lastcdr = $lastcdrrow[0];
			echo "Last CDR for tenor is: $lastcdr\n";
		} else {
			$lastcdr = "";
			echo "Warning: Unable to query last CDR seq for Tenor\n";
		}
		// Send the last successful sequence number
		// After a tenor reset, you'll need to mess
		// around here to get all the records.
 
		fwrite($pipes[0],$lastcdr."\r\n");
		// Read and process the reply line by line
		while (!feof($pipes[1])) {
			$line = fgets($pipes[1]);
			process_line($ip,$tenor,$line);
		}
		// Then close the command pipes and the command itself
		fclose($pipes[0]);
		fclose($pipes[1]);
		$return_value = proc_close($process);
	} else {
		// If the command failed:
		echo "Error: Telnet command failed. See /tmp/cdr-telnet-errors.log";
	}
?>

Open in new window

0
 
shaunwinginAuthor Commented:
Most impressive and well written and commented script. Looking fwd to implementing as soon as get the time as not a Linux fundi and will need a good amount of time to implement....
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

A proven path to a career in data science

At Springboard, we know how to get you a job in data science. With Springboard’s Data Science Career Track, you’ll master data science  with a curriculum built by industry experts. You’ll work on real projects, and get 1-on-1 mentorship from a data scientist.

  • 15
  • 10
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now