projects
asked on
Tracking thousands of Internet devices
I have a customer who has 10's of thousands of devices connected all over the Internet.
They monitor using traditional commercial tools but the question came up about alternatives.
Specifically, either using something like nagios, Cacti, Zabbix, etc, vs writing something custom which isn't as complex as these tools, gives us the exact results we want without having to mess with an open source package.
What I am searching for and not finding is something like a daemon, a service which could read from the db and insert into the db it's results. Mainly, they want to monitor certain ports on every single device to make sure that services are up. We would build our own reports.
Second, if there is such a tool, that is not bloated with countless features and could monitor 10's of thousands of devices, which one would be the most simple to use and more important, could handle without needing an array of servers.
There are mentions of roll-your-own in this post but nothing that I can find on the net.
http://www.networkworld.com/article/2825879/network-management/7-free-open-source-network-monitoring-tools.html
If you have some leads on a roll-your-own, that's what I'm looking for. Something more of a process which runs as a service, simply reading from the db what it should test and updating/inserting it's results, able to handle a huge amount of devices.
Finally, no need to suggest commercial services. We've looked into them and none do exactly what we want. We would need to build our own reports.
They monitor using traditional commercial tools but the question came up about alternatives.
Specifically, either using something like nagios, Cacti, Zabbix, etc, vs writing something custom which isn't as complex as these tools, gives us the exact results we want without having to mess with an open source package.
What I am searching for and not finding is something like a daemon, a service which could read from the db and insert into the db it's results. Mainly, they want to monitor certain ports on every single device to make sure that services are up. We would build our own reports.
Second, if there is such a tool, that is not bloated with countless features and could monitor 10's of thousands of devices, which one would be the most simple to use and more important, could handle without needing an array of servers.
There are mentions of roll-your-own in this post but nothing that I can find on the net.
http://www.networkworld.com/article/2825879/network-management/7-free-open-source-network-monitoring-tools.html
If you have some leads on a roll-your-own, that's what I'm looking for. Something more of a process which runs as a service, simply reading from the db what it should test and updating/inserting it's results, able to handle a huge amount of devices.
Finally, no need to suggest commercial services. We've looked into them and none do exactly what we want. We would need to build our own reports.
ASKER
Hi, thanks for the reply.
I'm not being specific because I am being discreet. Also, I do mention I just need to monitor some ports so there isn't anything all that specific about that or the devices. Ports, over the Internet.
There are no issues with resources. No, we won't be using a DSL connection or some old PC. We have tons of high end hardware, lots of bandwidth, plenty of redundancy, none of those things are the question :)
What I am asking for in the question are some leads to something more daemon like, software, service, Linux based platform that could be a good starting point. That's what I am asking for.
Thanks.
I'm not being specific because I am being discreet. Also, I do mention I just need to monitor some ports so there isn't anything all that specific about that or the devices. Ports, over the Internet.
There are no issues with resources. No, we won't be using a DSL connection or some old PC. We have tons of high end hardware, lots of bandwidth, plenty of redundancy, none of those things are the question :)
What I am asking for in the question are some leads to something more daemon like, software, service, Linux based platform that could be a good starting point. That's what I am asking for.
Thanks.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks again and yes, we've already determined that we'd like to write our own but that's my question, some leads, maybe something that already exists as a starting point.
The Open Telecom Platform seems to be something interesting, that can handle some 2M connections at a time or something along those lines.
I'm looking for leads on a starting point or a few.
The Open Telecom Platform seems to be something interesting, that can handle some 2M connections at a time or something along those lines.
I'm looking for leads on a starting point or a few.
collectd might be an interesting study subject.
collectd is primarily used to obtain various performance data, but that could also include your type of service.
Anyway it is meant to massively monitor & log the data of monitoring to backend datastorage.
It also addresses lots of tiny updates (few byte values) on massively spread disk blocks problems as efficiently as possible.
http://collectd.org/
For port probing take a look at tcptraceroute, you will want many attempts spread of various devices, but the methods used should be helpfull, as it only does One syn check, not try to open a link. And your solution doesn;t need the incremental life times.
collectd is primarily used to obtain various performance data, but that could also include your type of service.
Anyway it is meant to massively monitor & log the data of monitoring to backend datastorage.
It also addresses lots of tiny updates (few byte values) on massively spread disk blocks problems as efficiently as possible.
http://collectd.org/
For port probing take a look at tcptraceroute, you will want many attempts spread of various devices, but the methods used should be helpfull, as it only does One syn check, not try to open a link. And your solution doesn;t need the incremental life times.
ASKER
The collectd project seems to be a daemon running on the remote devices, sending their local test results to a central server.
What I am looking for is a service on one or more servers which checks the availability of specific ports on many remote devices, inserting its findings into the local db.
What I am looking for is a service on one or more servers which checks the availability of specific ports on many remote devices, inserting its findings into the local db.
That is one mode of operation, it can be a server for collecting too.
It runs probes (local) and can either store the data local, or send to another collectd server whic can store it.
It is the probing stuff with local updating you may want to look at.
If you build a probe that polls a remote station you do have a framework for gathering & storing data...
It runs probes (local) and can either store the data local, or send to another collectd server whic can store it.
It is the probing stuff with local updating you may want to look at.
If you build a probe that polls a remote station you do have a framework for gathering & storing data...
ASKER
So long as we can get data from the tool, we can deal with the gathering and storing and everything else.
So take the probe running part from that code, build your own probe and store the data from it.
You like to see if certain services are running a cheap method is to probe like tcptraceroute does.
(for tcp ports) of UDP you can look at traceroute. For packet fabrication and how responses are processed.
You like to see if certain services are running a cheap method is to probe like tcptraceroute does.
(for tcp ports) of UDP you can look at traceroute. For packet fabrication and how responses are processed.
ASKER
We already use a customized version of mtr for traceroute. Much better than anything else for our purposes.
It seems that taking the probe part would mean we might as well write it all ourselves at that point.
I'll take a closer look at it but I think there are probably some other solutions out there that might work.
I read somewhere as mentioned above, that Open Telecom Platform could handle some 2M simultaneous connections or something.
That sounds like a possible starting point also.
It seems that taking the probe part would mean we might as well write it all ourselves at that point.
I'll take a closer look at it but I think there are probably some other solutions out there that might work.
I read somewhere as mentioned above, that Open Telecom Platform could handle some 2M simultaneous connections or something.
That sounds like a possible starting point also.
Ah i didn't know mtr... :-) yes that seems to be fit as well.
About the Open Telecom Platform...
Didn't know that one either, under that name..., but it is merged with the Erlang programming language nowadays. It is the opensourced PBX management system from Ericsson. see https://en.wikipedia.org/wiki/Open_Telecom_Platform
Curious i didn't notice that as i have an erlang jabber service running.
At least it is a solid base. http://www.erlang.org/
As about simultanious connections... Both TCP and UDP have a limit to what they can handle.
You can initiate at most 64Ki (- serverports) sessions / ip address. in practice about 40K max.
You need to take int account that it takes a few minutes before a closed socket is actually available again.
That is both for UDP as for TCP. This is caused by the 16 bit portnumber.
The problem may be that a high level object language that Erlang is may not support a scanning technique that doesn;t actualy open sockets, but abandons after sending the first SYN. (One doesn't open a socket, but merely send an SYN (possibly followed by a RST if an SYN/ACK is returned).
2M Connections also take resources from your system for buffers, sockets OS objects to access them( handles or descriptors), then again within your process runtime libraries and again in your programs.... so for 2M connections you will need to beef up.
About the Open Telecom Platform...
Didn't know that one either, under that name..., but it is merged with the Erlang programming language nowadays. It is the opensourced PBX management system from Ericsson. see https://en.wikipedia.org/wiki/Open_Telecom_Platform
Curious i didn't notice that as i have an erlang jabber service running.
At least it is a solid base. http://www.erlang.org/
As about simultanious connections... Both TCP and UDP have a limit to what they can handle.
You can initiate at most 64Ki (- serverports) sessions / ip address. in practice about 40K max.
You need to take int account that it takes a few minutes before a closed socket is actually available again.
That is both for UDP as for TCP. This is caused by the 16 bit portnumber.
The problem may be that a high level object language that Erlang is may not support a scanning technique that doesn;t actualy open sockets, but abandons after sending the first SYN. (One doesn't open a socket, but merely send an SYN (possibly followed by a RST if an SYN/ACK is returned).
2M Connections also take resources from your system for buffers, sockets OS objects to access them( handles or descriptors), then again within your process runtime libraries and again in your programs.... so for 2M connections you will need to beef up.
ASKER
Here is more information on an interesting use of OTP.
http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html
http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
We already know we'll be having to roll our own but there were helpful leads in this question.
Thanks.
Thanks.
You're not very specific in what devices and services are to be monitored, so it's a very complicated (if not impossible) to come up with a recommendation of a specific product.
From a basic consideration, every commercial software suite would have more features than you'll need, and therefore would probably have less performance for monitoring that much clients.
With that all in mind, I would recommend some roll-your-own (aka "custom made") software for that - it would be tailored down to your needs and give the best performance. And you have all features under control. Be sure to get the sources ...
But you have to be aware that monitoring such a big bunch of clients needs - depending on the watch cycle, the amount of checked services and the check depth - a considerable amount of both computing power and network bandwidth. To speak in pictures, you can't expect that task to be managed with an old office desktop machine database, a RasPi and a 6 MBit DSL connection.
Besides of that - depending on how reliable the monitoring should be - you'll need to consider redundance in network, database and checking machine(s) hardware as well as in internet connections ... but probably you could outsource that and use some rented datacenter web server machines ... so you could focus on the task and leave the redundancy etc. on the datacenter ...