System performance metrics log generation and transfer from rsyslog client to rsyslog server

Have a look at cacti.net
Using SNMP data collection.

If you are looking to setup monitoring with alerts, check out nagios as well.

Personally, I think you are making this harder than it has to be by trying to do this through rsyslog.  I suggest you install a Zabbix (free, open source monitoring tool) and install Zabbix clients (or use SNMP, WMI, or other supported protocols) to harvest performance data from each host system.

https://www.zabbix.com/

Syslog is meant to handle logrecords (messages from programs) and sort them according to some classification.

Performance data is not part of that. There are several packages for monitoring performance data.
sysstat being the oldest (AFAICT),  Zabbix is a popular one, collectd + rrdcache + some postprocessing can be useful.

The biggest problem with ANY performance monitoring system is the huge number of 4-8 byte updates that need to be spread around the disks.
So you need to evaluate the system that fits you workload the best.
Most solutions have a tendency to break disk performance. Als don't use the systems you use for recording the monitoring data for anything else...

Thank you Arnold, Gary and Noci for sharing your valuable suggestions.

We have installations of CACTI and earlier used Zabbix too.

Ideally we found these are enough for our internal monitoring purpose but one of our developer was suggesting ElasticSearch will display the same stats with better UI and search will be easy unlike the native tools. The idea is to make the UI more easy to understand for end users using Elasticsearch by collecting logs and forwarding to rsyslog where ElasticSearch will pickup and display the metrics.

I will further rethink on our strategy if we really need to this upon your suggestion

Once again thank you all for your proactive response.

Look at Nagios Log Server; You can certainly have a trial for Nagios Log Server, even for 60 days. You can find it here: https://www.nagios.com/products/nagios-log-server/
 

Much depends on what you need NS where you need it.

You could existing data collection to also get put into a DB

Not sure what elasticsearch provides that might be of interest.
Could you add context what additional, or duration is of interest.

Are you talking about getting baseline performance type data periodically that can be use to gauge growth of demand, as an alert, notification to begin planing out expansion of resources?

I.e. Based on current spec, growth, demand, the resources should last till the upgrade cycle.
But the need is to alert when resource use of this reaches 85% as an example
A combination of data, memory, CPU, storage, simultaneous users.....

VMs/, physicals combinations.....

Elastic search is a bean counter.   you push data records into it and some rules how to handle the data and then kibana can present graphs of counts of the data.
Elastic seawrch is oly the storage & retrieval engine of something often called an ELK-stack.   Filebeat can eb a data provider.  (there are other *beat) providers.
It is document (aka File) oriented and files have some kind of loosely (poorly defined) layout.   Only it handles the data much better then a file 

So elastic search on it's own is NOT the answer, there must have been some configurations and data collectors there as well.
For performance data collectd & grafana might be a better fit on it's own.

Hi all experts,

I would like to do a closing post, literally almost everyone helped me out appropriately by answering to this question as i have to select one as my solution, the one submitted by madunix is more near to my situation and I have selected his answer as my solution.

Once again thank you to all
Best Regards

You should be able to select multiple solutions as contributing to the solution.