Howto: Capture data stream with telnet session on Linux Server
Say, what telnet application on linux system (not running GUI) listening to a specific port can do following:
Quintum device sends CDR records via port: It requires a CHR Line feed to tell start sending its buffer.
The data is to be written to a file. Or even better still to a database in PostGreSQL.
any ideas please?
Quintum device sends CDR records via port: It requires a CHR Line feed to tell start sending its buffer.
The data is to be written to a file. Or even better still to a database in PostGreSQL.
any ideas please?
Hi,
Netcat can do it. Do I understand correctly, you want to connect to the quintum device on a port, and then send a CR, and record everything that is replied back into a file? If so:
echo | netcat host port > /tmp/file
E.g. if the host is 192.168.1.1 and it listens on 12345
echo | netcat 192.168.1.1 12345 > /tmp/file
Will record what it sends back to /tmp/file
Netcat can do it. Do I understand correctly, you want to connect to the quintum device on a port, and then send a CR, and record everything that is replied back into a file? If so:
echo | netcat host port > /tmp/file
E.g. if the host is 192.168.1.1 and it listens on 12345
echo | netcat 192.168.1.1 12345 > /tmp/file
Will record what it sends back to /tmp/file
ASKER
Uberpappa, our description is 100% corrrect. How will netcat send a CR to the Quitnum to tell it to start sending?
THe command says:
echo | netcat host port > /tmp/file
Broken down:
echo
Which prints a CR
echo | netcat host port
Send the output of echo (CR) to netcat to forward to the remote host
echo | netcat host port > /tmp/file
and send whatever is returned to a file
echo | netcat host port > /tmp/file
Broken down:
echo
Which prints a CR
echo | netcat host port
Send the output of echo (CR) to netcat to forward to the remote host
echo | netcat host port > /tmp/file
and send whatever is returned to a file
ASKER
tx - will try it. Is there any way to write the file into a database? The columns are fixed width.
Yes, it shouldn't be too difficult to do something like:
echo | netcat host port | populatedb.sh
Can you post a snippet of the file, so we can get a better understanding of what it contains?
echo | netcat host port | populatedb.sh
Can you post a snippet of the file, so we can get a better understanding of what it contains?
Also, does it have to be PostgreSQL, or will MySQL also do?
ASKER
It has to be PostgreSQL - I'm afraid... Can it be done?
It can, just a longer process... give me a sec and I'll post an example script...
ASKER
How can I see if its listening.... Also the data is sent to a specific public ip address - but comes from a dynamic ip address - both are public. How can I run nc to listen on the public ip address for all data that comes in on the port?
Create a database, and create a table with all the fields you require in the database:
CREATE TABLE quintum (
Field1 CHAR(5),
FIeld2 CHAR(10),
Field3 CHAT(2)
);
As an example. The fields should correspond with your fixed width data.
Then, export to a file:
echo |netcat host port > /tmp/rawdata.txt
Inside the database, run the following script to create a temporary table and import the bulk text (not split into fields) in it:
CREATE TABLE temp_table (Line VARCHAR(200));
where 200 is large enough to contain the data from one line of the input. Then import the text file into it:
COPY temp_table (line) from '/tmp/rawdata.txt';
Now Split it into fields from the temp table to the new table:
INSERT INTO quintum SELECT SUBSTRING(Line FROM 1 FOR 5), SUBSTRING(Line FROM 6 FOR 10), SUBSTRING(Line FROM 17 FOR 2) from temp_table.
Then drop the temp table:
DROP TABLE temp_table;
The SUBSTRING statements are used to split the text into fields based on their fixed-widht placement.
Once you have the queries working, put them all into a .sql file, one after the other, then do:
echo | netcat host port > /tmp/rawdata.txt
psql database < /path/to/your/script.sql
And you're set to go.
CREATE TABLE quintum (
Field1 CHAR(5),
FIeld2 CHAR(10),
Field3 CHAT(2)
);
As an example. The fields should correspond with your fixed width data.
Then, export to a file:
echo |netcat host port > /tmp/rawdata.txt
Inside the database, run the following script to create a temporary table and import the bulk text (not split into fields) in it:
CREATE TABLE temp_table (Line VARCHAR(200));
where 200 is large enough to contain the data from one line of the input. Then import the text file into it:
COPY temp_table (line) from '/tmp/rawdata.txt';
Now Split it into fields from the temp table to the new table:
INSERT INTO quintum SELECT SUBSTRING(Line FROM 1 FOR 5), SUBSTRING(Line FROM 6 FOR 10), SUBSTRING(Line FROM 17 FOR 2) from temp_table.
Then drop the temp table:
DROP TABLE temp_table;
The SUBSTRING statements are used to split the text into fields based on their fixed-widht placement.
Once you have the queries working, put them all into a .sql file, one after the other, then do:
echo | netcat host port > /tmp/rawdata.txt
psql database < /path/to/your/script.sql
And you're set to go.
ASKER
Wow this looks a G8 script tx!
ASKER
The file is created but no data is written...
Oh, that changes things a bit! :)
On the listening host, run:
netcat -l -p 12345 > /tmp/file
It will then record everything that is sent to port 12345 of that host to the file /tmp/file.
On the listening host, run:
netcat -l -p 12345 > /tmp/file
It will then record everything that is sent to port 12345 of that host to the file /tmp/file.
Let me just make sure that I have this correct now:
Your quintum box (lets call it quintum.yourdomain.com listens on a port for a CR. Once it receives it, it opens a new connection to another host, call it otherhost.yourdomain.com which is supposed to be listening on a port for the result?
Your quintum box (lets call it quintum.yourdomain.com listens on a port for a CR. Once it receives it, it opens a new connection to another host, call it otherhost.yourdomain.com which is supposed to be listening on a port for the result?
ASKER
-l Used to specify that nc should listen for an incoming connection rather than initiate a connection to a remote host. It is an
error to use this option in conjunction with the -p, -s, or -z options. Additionally, any timeouts specified with the -w option
are ignored.
error to use this option in conjunction with the -p, -s, or -z options. Additionally, any timeouts specified with the -w option
are ignored.
ASKER
I tried:
netcat -l -p 9002 but returned an error....
Please see page 4 of attached file for process that Quintum uses. I've left password blank.
CDR.pdf
netcat -l -p 9002 but returned an error....
Please see page 4 of attached file for process that Quintum uses. I've left password blank.
CDR.pdf
echo sends a lf/cr not just a cr.
try this modification to Uberpappa's info:
perl -e ' print scalar chr(13);' | netcat 192.168.1.1 12345 > /tmp/file
try this modification to Uberpappa's info:
perl -e ' print scalar chr(13);' | netcat 192.168.1.1 12345 > /tmp/file
Thanks Arnold, forgot about that.
After viewing that doc, the whole world around this question has changed. As I understand, the client connects, the server provides a welcome message, and asks for a password, the client supplies the password, the client supplies it, and then its IP and unit name and the server then responds with a CSV file, not a fixed widht file.
So back to the drawing board, this screams netcat with expect, or as suggested initially arnold, a perl/C script.
It does make the database import a lot easier though.
Let me try and dish up something with netcat and expect quickly.
After viewing that doc, the whole world around this question has changed. As I understand, the client connects, the server provides a welcome message, and asks for a password, the client supplies the password, the client supplies it, and then its IP and unit name and the server then responds with a CSV file, not a fixed widht file.
So back to the drawing board, this screams netcat with expect, or as suggested initially arnold, a perl/C script.
It does make the database import a lot easier though.
Let me try and dish up something with netcat and expect quickly.
On second thought, it should be OK if we just send the password and other info sequentially:
#!/bin/bash
# CHANGE FROM HERE:
remoreserver="192.168.1.1"
remoteport=9002
rawdatafile=/tmp/rawdata.csv
tenorip="192.168.2.1"
tenorunit="tenor1"
password="secret"
# TO HERE
(
echo "${password}"
echo "${tenorip},${tenorunit}"
) | netcat remoteserver remoteport > ${rawdatafile}
Typo:
#!/bin/bash
# CHANGE FROM HERE:
remoteserver="192.168.1.1"
remoteport=9002
rawdatafile=/tmp/rawdata.csv
tenorip="192.168.2.1"
tenorunit="tenor1"
password="secret"
# TO HERE
(
echo "${password}"
echo "${tenorip},${tenorunit}"
) | netcat ${remoteserver} ${remoteport} > ${rawdatafile}
http://perldoc.perl.org/perlipc.html#Sockets%3a-Client%2fServer-Communication
Using a perl script to connect and get the data will also make the option of having everything done within a single process available.
The link above provides different examples of using socket to establish a single connection or a pair of connection.
Give it a try, you will likely find other uses for the knowledge you get.
There is a Telnet module, Net::Telnet that you could use as well if you do not want to get into Socket setup etc.
Using a perl script to connect and get the data will also make the option of having everything done within a single process available.
The link above provides different examples of using socket to establish a single connection or a pair of connection.
Give it a try, you will likely find other uses for the knowledge you get.
There is a Telnet module, Net::Telnet that you could use as well if you do not want to get into Socket setup etc.
ASKER
tx Uberppa and Arnold. Uberppa, I'm not receiving data in the file. I ran tcpdump -i eth0:1 port 9002 -w isca
and getting a packet from the Quintum. Any ideas please? Perhaps you can setup a test on your side and see - I can put in your ip into the Quintum. Tx for teh great input.
and getting a packet from the Quintum. Any ideas please? Perhaps you can setup a test on your side and see - I can put in your ip into the Quintum. Tx for teh great input.
we can try... mail me at uberpappa123 at gmail dot com
how about using an expect script to send and receive the data from you special port, then simply call this script inside a `script' shell, see
man script
man script
As discussed on msn, I've written a script in php that will:
1) Telnet to a remote Server, and authenticate with the given password
2) Parse the Tenor name passed by the server
3) Check the last CDR that was received for this server, and send it to server to instruct it to only send from that CDR onwards
3) Read the CDR's that are returned, and log them into Postgres.
To make it work:
1) Create a database in postgres
2) Create a user and configure the permissions on the db as well as in pg_hba.conf so that the user can create records in the db.
3) Run the create.sql script against the db to create the table. (psql dbname < create.sql)
4) Change the first line in the importcdr script, which reads #!/usr/bin/php to point to wherever your php is on the server. You can check this with "which php".
Then run it as follows:
./importcdr serverip serverport <password>
If no password is configure, ommit it from the command. The port is 9003 and the serverip is the one you gave me on msn.
Hope it works! Let me know if anything on your side causes problems. This side its working 100%.
Scripts to follow......
1) Telnet to a remote Server, and authenticate with the given password
2) Parse the Tenor name passed by the server
3) Check the last CDR that was received for this server, and send it to server to instruct it to only send from that CDR onwards
3) Read the CDR's that are returned, and log them into Postgres.
To make it work:
1) Create a database in postgres
2) Create a user and configure the permissions on the db as well as in pg_hba.conf so that the user can create records in the db.
3) Run the create.sql script against the db to create the table. (psql dbname < create.sql)
4) Change the first line in the importcdr script, which reads #!/usr/bin/php to point to wherever your php is on the server. You can check this with "which php".
Then run it as follows:
./importcdr serverip serverport <password>
If no password is configure, ommit it from the command. The port is 9003 and the serverip is the one you gave me on msn.
Hope it works! Let me know if anything on your side causes problems. This side its working 100%.
Scripts to follow......
The sql create script: create.sql
CREATE TABLE cdr (
ip VARCHAR(20),
tenor VARCHAR(50),
seq INTEGER,
called_number VARCHAR(50),
duration_in_seconds INTEGER,
time_initiated VARCHAR(20),
time_connected VARCHAR(20),
time_disconnected VARCHAR(20),
disconnect_cause INTEGER,
local_ip VARCHAR(20),
remote_ip VARCHAR(20),
originating_trunk_id VARCHAR(20),
call_type INTEGER,
call_number_type INTEGER,
incoming_line INTEGER,
incoming_channel INTEGER,
outgoing_line INTEGER,
outgoing_channel INTEGER,
auto_switch_time VARCHAR(20),
auto_switch_duration VARCHAR(20),
bad_quality_events INTEGER,
auto_switching_flag INTEGER,
PRIMARY KEY (ip,tenor,seq,time_initiated)
);ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Most impressive and well written and commented script. Looking fwd to implementing as soon as get the time as not a Linux fundi and will need a good amount of time to implement....
You could look into using expect.