Link to home
Create AccountLog in
Avatar of @nir
@nirFlag for United States of America

asked on

Zabbix not sending alert on telegram when zabbix agent is down or unavailable.

Hello , Experts,
Need some help from zabbix experts. I'm not getting message on telegram when Zabbix agent is not available. All other problem messages on telegram is working fine.
Am I missing something ? Do I need to create a special template to a capture unavailability of zabbix agent.

 User generated image

I tried looking at solution on zabbix fourm. The closet match was 
https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/36671-want-to-be-alert-when-zabbix-agent-down-or-can-not-be-connected
but it was not helpful
My zabbix_agentd.conf look as below

User generated imageThank you


ASKER CERTIFIED SOLUTION
Avatar of madunix
madunix

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Avatar of @nir

ASKER

My zabbix_agentd.conf

# This is a configuration file for Zabbix agent daemon (Unix)

# To get more information about Zabbix, visit http://www.zabbix.com


############ GENERAL PARAMETERS #################


### Option: PidFile

#       Name of PID file.

#

# Mandatory: no

# Default:

# PidFile=/tmp/zabbix_agentd.pid


PidFile=/run/zabbix/zabbix_agentd.pid


### Option: LogType

#       Specifies where log messages are written to:

#               system  - syslog

#               file    - file specified with LogFile parameter

#               console - standard output

#

# Mandatory: no

# Default:

# LogType=file


### Option: LogFile

#       Log file name for LogType 'file' parameter.

#

# Mandatory: yes, if LogType is set to file, otherwise no

# Default:

# LogFile=


LogFile=/var/log/zabbix/zabbix_agentd.log


### Option: LogFileSize

#       Maximum size of log file in MB.

#       0 - disable automatic log rotation.

#

# Mandatory: no

# Range: 0-1024

# Default:

# LogFileSize=1


LogFileSize=0


### Option: DebugLevel

#       Specifies debug level:

#       0 - basic information about starting and stopping of Zabbix processes

#       1 - critical information

#       2 - error information

#       3 - warnings

#       4 - for debugging (produces lots of information)

#       5 - extended debugging (produces even more information)

#

# Mandatory: no

# Range: 0-5

# Default:

#DebugLevel=4


### Option: SourceIP

#       Source IP address for outgoing connections.

#

# Mandatory: no

# Default:

# SourceIP=


### Option: AllowKey

#       Allow execution of item keys matching pattern.

#       Multiple keys matching rules may be defined in combination with DenyKey.

#       Key pattern is wildcard expression, which support "*" character to match any number of any characters in certain position. It might be used in both key name and key arguments.

#       Parameters are processed one by one according their appearance order.

#       If no AllowKey or DenyKey rules defined, all keys are allowed.

#

# Mandatory: no


### Option: DenyKey

#       Deny execution of items keys matching pattern.

#       Multiple keys matching rules may be defined in combination with AllowKey.

#       Key pattern is wildcard expression, which support "*" character to match any number of any characters in certain position. It might be used in both key name and key arguments.

#       Parameters are processed one by one according their appearance order.

#       If no AllowKey or DenyKey rules defined, all keys are allowed.

#       Unless another system.run[*] rule is specified DenyKey=system.run[*] is added by default.

#

# Mandatory: no

# Default:

# DenyKey=system.run[*]


### Option: EnableRemoteCommands - Deprecated, use AllowKey=system.run[*] or DenyKey=system.run[*] instead

#       Internal alias for AllowKey/DenyKey parameters depending on value:

#       0 - DenyKey=system.run[*]

#       1 - AllowKey=system.run[*]

#

# Mandatory: no


### Option: LogRemoteCommands

#       Enable logging of executed shell commands as warnings.

#       0 - disabled

#       1 - enabled

#

# Mandatory: no

# Default:

# LogRemoteCommands=0


##### Passive checks related


### Option: Server

#       List of comma delimited IP addresses, optionally in CIDR notation, or DNS names of Zabbix servers and Zabbix proxies.

#       Incoming connections will be accepted only from the hosts listed here.

#       If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally

#       and '::/0' will allow any IPv4 or IPv6 address.

#       '0.0.0.0/0' can be used to allow any IPv4 address.

#       Example: Server=127.0.0.1,192.168.1.0/24,::1,2001:db8::/32,zabbix.example.com

#

# Mandatory: yes, if StartAgents is not explicitly set to 0

# Default:

# Server=


Server=10.0.0.250


### Option: ListenPort

#       Agent will listen on this port for connections from the server.

#

# Mandatory: no

# Range: 1024-32767

# Default:

# ListenPort=10050


### Option: ListenIP

#       List of comma delimited IP addresses that the agent should listen on.

#       First IP address is sent to Zabbix server if connecting to it to retrieve list of active checks.

#

# Mandatory: no

# Default:

# ListenIP=0.0.0.0


### Option: StartAgents

#       Number of pre-forked instances of zabbix_agentd that process passive checks.

#       If set to 0, disables passive checks and the agent will not listen on any TCP port.

#

# Mandatory: no

# Range: 0-100

# Default:

# StartAgents=3


##### Active checks related


### Option: ServerActive

#       Zabbix server/proxy address or cluster configuration to get active checks from.

#       Server/proxy address is IP address or DNS name and optional port separated by colon.

#       Cluster configuration is one or more server addresses separated by semicolon.

#       Multiple Zabbix servers/clusters and Zabbix proxies can be specified, separated by comma.

#       More than one Zabbix proxy should not be specified from each Zabbix server/cluster.

#       If Zabbix proxy is specified then Zabbix server/cluster for that proxy should not be specified.

#       Multiple comma-delimited addresses can be provided to use several independent Zabbix servers in parallel. Spaces are allowed.

#       If port is not specified, default port is used.

#       IPv6 addresses must be enclosed in square brackets if port for that host is specified.

#       If port is not specified, square brackets for IPv6 addresses are optional.

#       If this parameter is not specified, active checks are disabled.

#       Example for Zabbix proxy:

#               ServerActive=127.0.0.1:10051

#       Example for multiple servers:

#               ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]

#       Example for high availability:

#               ServerActive=zabbix.cluster.node1;zabbix.cluster.node2:20051;zabbix.cluster.node3

#       Example for high availability with two clusters and one server:

#               ServerActive=zabbix.cluster.node1;zabbix.cluster.node2:20051,zabbix.cluster2.node1;zabbix.cluster2.node2,zabbix.domain

#

# Mandatory: no

# Default:

# ServerActive=


ServerActive=10.0.0.250


### Option: Hostname

#       List of comma delimited unique, case sensitive hostnames.

#       Required for active checks and must match hostnames as configured on the server.

#       Value is acquired from HostnameItem if undefined.

#

# Mandatory: no

# Default:

# Hostname=


Hostname=Zabbix server


### Option: HostnameItem

#       Item used for generating Hostname if it is undefined. Ignored if Hostname is defined.

#       Does not support UserParameters or aliases.

#

# Mandatory: no

# Default:

# HostnameItem=system.hostname


### Option: HostMetadata

#       Optional parameter that defines host metadata.

#       Host metadata is used at host auto-registration process.

#       An agent will issue an error and not start if the value is over limit of 255 characters.

#       If not defined, value will be acquired from HostMetadataItem.

#

# Mandatory: no

# Range: 0-255 characters

# Default:

# HostMetadata=


### Option: HostMetadataItem

#       Optional parameter that defines an item used for getting host metadata.

#       Host metadata is used at host auto-registration process.

#       During an auto-registration request an agent will log a warning message if

#       the value returned by specified item is over limit of 255 characters.

#       This option is only used when HostMetadata is not defined.

#

# Mandatory: no

# Default:

# HostMetadataItem=


### Option: HostInterface

#       Optional parameter that defines host interface.

#       Host interface is used at host auto-registration process.

#       An agent will issue an error and not start if the value is over limit of 255 characters.

#       If not defined, value will be acquired from HostInterfaceItem.

#

# Mandatory: no

# Range: 0-255 characters

# Default:

# HostInterface=


### Option: HostInterfaceItem

#       Optional parameter that defines an item used for getting host interface.

#       Host interface is used at host auto-registration process.

#       During an auto-registration request an agent will log a warning message if

#       the value returned by specified item is over limit of 255 characters.

#       This option is only used when HostInterface is not defined.

#

# Mandatory: no

# Default:

# HostInterfaceItem=


### Option: RefreshActiveChecks

#       How often list of active checks is refreshed, in seconds.

#

# Mandatory: no

# Range: 60-3600

# Default:

# RefreshActiveChecks=120


### Option: BufferSend

#       Do not keep data longer than N seconds in buffer.

#

# Mandatory: no

# Range: 1-3600

# Default:

# BufferSend=5


### Option: BufferSize

#       Maximum number of values in a memory buffer. The agent will send

#       all collected data to Zabbix Server or Proxy if the buffer is full.

#

# Mandatory: no

# Range: 2-65535

# Default:

# BufferSize=100


### Option: MaxLinesPerSecond

#       Maximum number of new lines the agent will send per second to Zabbix Server

#       or Proxy processing 'log' and 'logrt' active checks.

#       The provided value will be overridden by the parameter 'maxlines',

#       provided in 'log' or 'logrt' item keys.

#

# Mandatory: no

# Range: 1-1000

# Default:

# MaxLinesPerSecond=20


############ ADVANCED PARAMETERS #################


### Option: Alias

#       Sets an alias for an item key. It can be used to substitute long and complex item key with a smaller and simpler one.

#       Multiple Alias parameters may be present. Multiple parameters with the same Alias key are not allowed.

#       Different Alias keys may reference the same item key.

#       For example, to retrieve the ID of user 'zabbix':

#       Alias=zabbix.userid:vfs.file.regexp[/etc/passwd,^zabbix:.:([0-9]+),,,,\1]

#       Now shorthand key zabbix.userid may be used to retrieve data.

#       Aliases can be used in HostMetadataItem but not in HostnameItem parameters.

#

# Mandatory: no

# Range:

# Default:


### Option: Timeout

#       Spend no more than Timeout seconds on processing

#

# Mandatory: no

# Range: 1-30

# Default:

# Timeout=3


### Option: AllowRoot

#       Allow the agent to run as 'root'. If disabled and the agent is started by 'root', the agent

#       will try to switch to the user specified by the User configuration option instead.

#       Has no effect if started under a regular user.

#       0 - do not allow

#       1 - allow

#

# Mandatory: no

# Default:

# AllowRoot=0


### Option: User

#       Drop privileges to a specific, existing user on the system.

#       Only has effect if run as 'root' and AllowRoot is disabled.

#

# Mandatory: no

# Default:

# User=zabbix

# NOTE: This option is overriden by settings in systemd service file!


### Option: Include

#       You may include individual files or all files in a directory in the configuration file.

#       Installing Zabbix will create include directory in /usr/local/etc, unless modified during the compile time.

#

# Mandatory: no

# Default:

# Include=


Include=/etc/zabbix/zabbix_agentd.d/*.conf


# Include=/usr/local/etc/zabbix_agentd.userparams.conf

# Include=/usr/local/etc/zabbix_agentd.conf.d/

# Include=/usr/local/etc/zabbix_agentd.conf.d/*.conf


####### USER-DEFINED MONITORED PARAMETERS #######


### Option: UnsafeUserParameters

#       Allow all characters to be passed in arguments to user-defined parameters.

#       The following characters are not allowed:

#       \ ' " ` * ? [ ] { } ~ $ ! & ; ( ) < > | # @

#       Additionally, newline characters are not allowed.

#       0 - do not allow

#       1 - allow

#

# Mandatory: no

# Range: 0-1

# Default:

# UnsafeUserParameters=0


### Option: UserParameter

#       User-defined parameter to monitor. There can be several user-defined parameters.

#       Format: UserParameter=<key>,<shell command>

#       See 'zabbix_agentd' directory for examples.

#

# Mandatory: no

# Default:

# UserParameter=


### Option: UserParameterDir

#       Directory to execute UserParameter commands from. Only one entry is allowed.

#       When executing UserParameter commands the agent will change the working directory to the one

#       specified in the UserParameterDir option.

#       This way UserParameter commands can be specified using the relative ./ prefix.

#

# Mandatory: no

# Default:

# UserParameterDir=


####### LOADABLE MODULES #######


### Option: LoadModulePath

#       Full path to location of agent modules.

#       Default depends on compilation options.

#       To see the default path run command "zabbix_agentd --help".

#

# Mandatory: no

# Default:

# LoadModulePath=${libdir}/modules


### Option: LoadModule

#       Module to load at agent startup. Modules are used to extend functionality of the agent.

#       Formats:

#               LoadModule=<module.so>

#               LoadModule=<path/module.so>

#               LoadModule=</abs_path/module.so>

#       Either the module must be located in directory specified by LoadModulePath or the path must precede the module name.

#       If the preceding path is absolute (starts with '/') then LoadModulePath is ignored.

#       It is allowed to include multiple LoadModule parameters.

#

# Mandatory: no

# Default:

# LoadModule=


####### TLS-RELATED PARAMETERS #######


### Option: TLSConnect

#       How the agent should connect to server or proxy. Used for active checks.

#       Only one value can be specified:

#               unencrypted - connect without encryption

#               psk         - connect using TLS and a pre-shared key

#               cert        - connect using TLS and a certificate

#

# Mandatory: yes, if TLS certificate or PSK parameters are defined (even for 'unencrypted' connection)

# Default:

# TLSConnect=unencrypted


### Option: TLSAccept

#       What incoming connections to accept.

#       Multiple values can be specified, separated by comma:

#               unencrypted - accept connections without encryption

#               psk         - accept connections secured with TLS and a pre-shared key

#               cert        - accept connections secured with TLS and a certificate

#

# Mandatory: yes, if TLS certificate or PSK parameters are defined (even for 'unencrypted' connection)

# Default:

# TLSAccept=unencrypted


### Option: TLSCAFile

#       Full pathname of a file containing the top-level CA(s) certificates for

#       peer certificate verification.

#

# Mandatory: no

# Default:

# TLSCAFile=


### Option: TLSCRLFile

#       Full pathname of a file containing revoked certificates.

#

# Mandatory: no

# Default:

# TLSCRLFile=


### Option: TLSServerCertIssuer

#               Allowed server certificate issuer.

#

# Mandatory: no

# Default:

# TLSServerCertIssuer=


### Option: TLSServerCertSubject

#               Allowed server certificate subject.

#

# Mandatory: no

# Default:

# TLSServerCertSubject=


### Option: TLSCertFile

#       Full pathname of a file containing the agent certificate or certificate chain.

#

# Mandatory: no

# Default:

# TLSCertFile=


### Option: TLSKeyFile

#       Full pathname of a file containing the agent private key.

#

# Mandatory: no

# Default:

# TLSKeyFile=


### Option: TLSPSKIdentity

#       Unique, case sensitive string used to identify the pre-shared key.

#

# Mandatory: no

# Default:

# TLSPSKIdentity=


### Option: TLSPSKFile

#       Full pathname of a file containing the pre-shared key.

#

# Mandatory: no

# Default:

# TLSPSKFile=


####### For advanced users - TLS ciphersuite selection criteria #######


### Option: TLSCipherCert13

#       Cipher string for OpenSSL 1.1.1 or newer in TLS 1.3.

#       Override the default ciphersuite selection criteria for certificate-based encryption.

#

# Mandatory: no

# Default:

# TLSCipherCert13=


### Option: TLSCipherCert

#       GnuTLS priority string or OpenSSL (TLS 1.2) cipher string.

#       Override the default ciphersuite selection criteria for certificate-based encryption.

#       Example for GnuTLS:

#               NONE:+VERS-TLS1.2:+ECDHE-RSA:+RSA:+AES-128-GCM:+AES-128-CBC:+AEAD:+SHA256:+SHA1:+CURVE-ALL:+COMP-NULL:+SIGN-ALL:+CTYPE-X.509

#       Example for OpenSSL:

#               EECDH+aRSA+AES128:RSA+aRSA+AES128

#

# Mandatory: no

# Default:

# TLSCipherCert=


### Option: TLSCipherPSK13

#       Cipher string for OpenSSL 1.1.1 or newer in TLS 1.3.

#       Override the default ciphersuite selection criteria for PSK-based encryption.

#       Example:

#               TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256

#

# Mandatory: no

# Default:

# TLSCipherPSK13=


### Option: TLSCipherPSK

#       GnuTLS priority string or OpenSSL (TLS 1.2) cipher string.

#       Override the default ciphersuite selection criteria for PSK-based encryption.

#       Example for GnuTLS:

#               NONE:+VERS-TLS1.2:+ECDHE-PSK:+PSK:+AES-128-GCM:+AES-128-CBC:+AEAD:+SHA256:+SHA1:+CURVE-ALL:+COMP-NULL:+SIGN-ALL

#       Example for OpenSSL:

#               kECDHEPSK+AES128:kPSK+AES128

#

# Mandatory: no

# Default:

# TLSCipherPSK=


### Option: TLSCipherAll13

#       Cipher string for OpenSSL 1.1.1 or newer in TLS 1.3.

#       Override the default ciphersuite selection criteria for certificate- and PSK-based encryption.

#       Example:

#               TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256

#

# Mandatory: no

# Default:

# TLSCipherAll13=


### Option: TLSCipherAll

#       GnuTLS priority string or OpenSSL (TLS 1.2) cipher string.

#       Override the default ciphersuite selection criteria for certificate- and PSK-based encryption.

#       Example for GnuTLS:

#               NONE:+VERS-TLS1.2:+ECDHE-RSA:+RSA:+ECDHE-PSK:+PSK:+AES-128-GCM:+AES-128-CBC:+AEAD:+SHA256:+SHA1:+CURVE-ALL:+COMP-NULL:+SIGN-ALL:+CTYPE-X.509

#       Example for OpenSSL:

#               EECDH+aRSA+AES128:RSA+aRSA+AES128:kECDHEPSK+AES128:kPSK+AES128

#

# Mandatory: no

# Default:

# TLSCipherAll=


####### For advanced users - TCP-related fine-tuning parameters #######


## Option: ListenBacklog

#       The maximum number of pending connections in the queue. This parameter is passed to

#       listen() function as argument 'backlog' (see "man listen").

#

# Mandatory: no

# Range: 0 - INT_MAX (depends on system, too large values may be silently truncated to implementation-specified maximum)

# Default: SOMAXCONN (hard-coded constant, depends on system)

# ListenBacklog=

TLSConnect=psk

TLSAccept=psk

TLSPSKIdentity=XXX-YY-ZZZ

TLSPSKFile=/etc/zabbix/mysec.psk



Avatar of @nir

ASKER

zabbix_server.log after disabling the agent

15865:20230903:164601.502 executing housekeeper
 15865:20230903:164601.642 housekeeper [deleted 12125 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.140091 sec, idle for 1 hour(s)]
 15865:20230903:174601.727 executing housekeeper
 15865:20230903:174601.848 housekeeper [deleted 12125 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.120487 sec, idle for 1 hour(s)]
 15888:20230903:183847.650 Zabbix agent item "service.info["vm3dservice",state]" on host "Win10Edu" failed: first network error, wait for 15 seconds
 15893:20230903:183902.268 resuming Zabbix agent checks on host "Win10Edu": connection restored
 15865:20230903:184601.940 executing housekeeper
 15865:20230903:184602.077 housekeeper [deleted 12125 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.136571 sec, idle for 1 hour(s)]
 15865:20230903:194602.119 executing housekeeper
 15865:20230903:194602.262 housekeeper [deleted 12128 hist/trends, 0 items/triggers, 0 events, 0 problems, 0 sessions, 0 alarms, 0 audit, 0 records in 0.142480 sec, idle for 1 hour(s)]
 15885:20230903:195831.225 Zabbix agent item "system.cpu.switches" on host "rasppi.test.lan" failed: first network error, wait for 15 seconds
 15893:20230903:195846.148 Zabbix agent item "system.cpu.util[,guest_nice]" on host "rasppi.test.lan" failed: another network error, wait for 15 seconds
 15893:20230903:195901.198 Zabbix agent item "system.cpu.load[all,avg1]" on host "rasppi.test.lan" failed: another network error, wait for 15 seconds
 15893:20230903:195916.250 temporarily disabling Zabbix agent checks on host "rasppi.test.lan": interface unavailable

Open in new window


Avatar of @nir

ASKER

Telegram Alert is not working ONLY for agent-down/agent unavaliable.
User generated image


Avatar of @nir

ASKER

Trigger Action via Telegram is enabled
User generated image


Avatar of madunix
madunix

There are network errors between your Zabbix server and some agents. 

User generated image


Ping Test: Can the Zabbix server ping the agent and vice versa?

Firewall: Verify that no firewalls prevent access to the Zabbix agent port, typically 10050.

Configuring the Zabbix Agent Verify that the zabbix_agentd.conf file is set up to accept connections from your Zabbix server.


User generated image


Hostname=Zabbix server: This should match this agent's hostname configured on your Zabbix server. If it doesn't, active checks will not work. 

Avatar of @nir

ASKER

Thank you @madunix, Yes I corrected the hostname.
Let me clarify my problem once again.

I have a webserver running and have created a webscernario to monitor website.
I logged into zabbix client and do systemctl stop apache2, I get the message in Telegram saying Webserver is down.
when I stop zabbix-agent, (systemctl stop zabbix-agent). I never get the message that says agent is down but I can see red icon saying zabbix agent not available in zabbix server dashboard

Open in new window

Do I need to create a template like webscenario for zabbix agent monitoring ?
Avatar of @nir

ASKER


After creating trigger and trigger action

with expression. It is working good. Thanks @madunix for pointing out at Trigger/Trigger actions


last(/hostname/agent.ping,#1)=0

Open in new window