Jasmin shahrzad
asked on
lxd monitoring tools
is someone try monitoring lxd containers tools?
ASKER
what about if some container using all memory? or disk is just growing. There are some shearing resource for all containers in one host. i have a xen server and one host with many lxd in xen i can see memory is top but i need to login to all lxd to see what happen.
Icinga & Nagios can also measure inside container by adding the right sentinels (nrpe, nsca) to help.
Those tools are meant to monitor hundreds / thousands of systems.
Those tools are meant to monitor hundreds / thousands of systems.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Aside: One other check you may have to run is to scan your system for ghost files.
These are files which have been deleted (no directory entry anymore) where file is still open (file handle connected to file).
You can find these files fairly easy, using one of these technigues.
If you're trying to debug oddball disk usage, keep these 2x commands handy.
In the case of LXD, you can run these commands at the machine level, then convert the host PID to the LXD container PID.
Or far easier, will be to write a script to use something like...
For all containers as $cname...
Then write the script check-for-ghost-files + arrange for this script to be accessible by all your containers.
My approach to this is to mount a tools directory in every container....
These are files which have been deleted (no directory entry anymore) where file is still open (file handle connected to file).
You can find these files fairly easy, using one of these technigues.
lsof -nP | grep '(deleted)'
find /proc/*/fd -ls | grep '(deleted)'If you're trying to debug oddball disk usage, keep these 2x commands handy.
In the case of LXD, you can run these commands at the machine level, then convert the host PID to the LXD container PID.
Or far easier, will be to write a script to use something like...
For all containers as $cname...
lxc exec $cname -- check-for-ghost-filesThen write the script check-for-ghost-files + arrange for this script to be accessible by all your containers.
My approach to this is to mount a tools directory in every container....
lxc config device add $cname tools disk source=/tools path=/tools
lsof -nP | grep '(deleted)' might not show all deleted files.
All MMAP'ed files (f.e. shared libraries* have no file handle They will show up with lsof -nP | grep DEL
Problem is that shows a lot more f.e. temporary named pipes or temp shared memory mappings so some grepping on filesystem may be needed.
All MMAP'ed files (f.e. shared libraries* have no file handle They will show up with lsof -nP | grep DEL
Problem is that shows a lot more f.e. temporary named pipes or temp shared memory mappings so some grepping on filesystem may be needed.
lxd monitoring is not really interesting... you need to monitor for
is my webserver running... (try some queries)
is my DNS running (try some queries)...
...
If LXD is running but the webserver inside died you still have zero knowledge....
IF hte webserver is still answering correctly then anything below/ above it also does... (webserver is running ==> container is running ==> host is running and all network is running as well).
Check on Icinga / nagio for verifying such services.