[Web GUI] Metrics doesn't show up. Hosts are up and running

[veks]

Hi,
I've installed metrics server on main node, and metrics clients on all nodes (including the main node).
I've added hosts under the metrics setup.
And two of them are up, one is down (not sure why).

But, under the job and host metrics there's nothing.

Why?




Thanks!

[emcodem]

Hi veks,

sorry that metrics is still a little bit itchy.
Can you please tell me what version.txt in the same folder as server.exe says?
Also, please replace custom.ini here (and restart metrics_ui service after that):
tools\prometheus\grafana-7.4.3\conf

with this file (i think i messed this file up in the latest webint deliverable):

custom.ini

(32.62 KiB) Downloaded 254 times

The key points are:

Code: Select all

allow_embedding = true
disable_login_form = true
[auth.anonymous]
# enable anonymous access
enabled = true

If you added a host to the hosts list, as you already know, the client must be installed and running.

Check if this is the case by logging in to the main webinterface server where the metrics collector service/prometheus is installed and opening in the browser:
http://hostname:9182/metrics

You should get some text about cpu stats and such, if not either the servername you entered cannot be resolved by dns, metrics_client is not running or firewall blocks 9182 or such.

[veks]

Version is 1.3.0.1.
I'll test that out. tnx

[veks]

I've added your .ini file but it's the same.

Also, locally on each server I can access the metrics page, but another server can't access the page from another server.

Also I get this error when I want to restart and save:

Error saving retention time. {"readyState":4,"responseText":"Error restarting prometheus service: <br/>Please try to manually edit the service startup parameters e.g. --storage.tsdb.retention.time 1d using nssm.exe edit \"FFAStrans Metrics Collector\" Message, {\"killed\":false,\"code\":2,\"signal\":null,\"cmd\":\"net stop \\\"FFAStrans Metrics Collector\\\" && net start \\\"FFAStrans Metrics Collector\\\"\",\"stdout\":\".\",\"stderr\":\"The service name is invalid.\\r\\n\\r\\nMore help is available by typing NET HELPMSG 2185.\\r\\n\\r\\n\"}","status":500,"statusText":"Internal Server Error"}

[emcodem]

OK that splits up into at least 2 different problems:

Also, locally on each server I can access the metrics page, but another server can't access the page from another server.

We talk about this "page", right? http://hostname:9182/metrics
If it can only be reached from local, the only thing that could really influence is a firewall and of course DNS issues. If you can ping the exact "hostname" that you enter in the browser, firewall is the only possible problem left.

The service name is invalid.\\r\\n\\r

This is a strange situation because you should not be able to access the "Setup Page" unless prometheus is running on the same server as webinterface is running on. Did you maybe start it manually or do you have another prometheus server installed on this machine?

Could you confirm that on the server where webint is installed, you see these 4 services? (the Metrics Client is optional)

veks-metrics.png (4.02 KiB) Viewed 8346 times

[veks]

Hi,
it is running:
https://prnt.sc/A8-vk2iLkVKT
And yes, it's running on the same server, webinterface, and metrics server + client.

Thanks!

[veks]

Hi,
it is running:
https://prnt.sc/A8-vk2iLkVKT
And yes, it's running on the same server, webinterface, and metrics server + client.

Thanks!

It works, but only when accessed from the main server where webinterface is running.
But I can't access it from another PCs in the network. And that might be a problem with our firewall.

Thanks!

[veks]

@emcodem, is it possible in web interface to filter jobs per server?
As there's no exact way to see how many jobs each node has done in some period of time.

Thanks!

[emcodem]

ok i think i found something.
Please try editing custom.ini and default.ini here:
tools\prometheus\grafana-7.4.3\conf
set: allow_embedding to true.

It should be set in custom.ini but for some reason i needed to also set it in default.ini to make it work. After that restart grafana (metrics ui service).

Code: Select all

@emcodem, is it possible in web interface to filter jobs per server?
As there's no exact way to see how many jobs each node has done in some period of time.

I think we already have all needed metrics in the prometheus DB, all thats missing is a grafana dashobard for it.
You got some experience with grafana dashboards?

[veks]

ok i think i found something.
Please try editing custom.ini and default.ini here:
tools\prometheus\grafana-7.4.3\conf
set: allow_embedding to true.

It should be set in custom.ini but for some reason i needed to also set it in default.ini to make it work. After that restart grafana (metrics ui service).

Code: Select all

@emcodem, is it possible in web interface to filter jobs per server?
As there's no exact way to see how many jobs each node has done in some period of time.

I think we already have all needed metrics in the prometheus DB, all thats missing is a grafana dashobard for it.
You got some experience with grafana dashboards?

No, I don't have any experience with it

Tho, in grafana dashboard I don't see a way how to see a graph of all nodes at once, and not the summed graph view of for example CPU.
As I'd like to see which node uses most of the CPU at exact chosen time...

Ty