-
Notifications
You must be signed in to change notification settings - Fork 8
/
troubleshooting-uninstall-metrics.html.md.erb
144 lines (95 loc) · 6.71 KB
/
troubleshooting-uninstall-metrics.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
---
title: Troubleshooting and Uninstalling JMX Bridge
owner: PCF Metrics
---
<strong><%= modified_date %></strong>
This topic describes how to resolve common issues with the JMX Bridge for Pivotal Cloud Foundry (PCF) tile and how to uninstall the tile if necessary.
## <a id="troubleshot"></a> Troubleshoot
The following sections provide help with troubleshooting JMX Bridge for PCF.
### <a id='dependency'></a> Dependency Error During Install
You might see the following error if your installed version of Elastic Runtime is not compatible with the version of JMX Bridge that you are attempting to install:
`JMX Bridge requires 'cf' version '~> 1.8.0.0' as a dependency.`
<%= image_tag("images/jmx-ert-install-depedency.png") %>
Follow the instructions below that correspond to your installed Elastic Runtime/Pivotal Application Service (PAS) version:
* **Elastic Runtime v1.8.0 to v1.8.16**:
1. Download version v1.8.17 or later of the [Elastic Runtime tile](https://network.pivotal.io/products/elastic-runtime) and import it into your Ops Manager **Installation Dashboard**.
1. Click the newly added tile to review any configurable options. For more information, see [Installing Pivotal Cloud Foundry](https://docs.pivotal.io/pivotalcf/installing/index.html) and choose your IaaS.
1. Click **Review Pending Changes**, then **Apply Changes** to install both Elastic Runtime and JMX Bridge.
* **Elastic Runtime v1.8.17 or later**: Upgrade the JMX Bridge tile to v1.8.9 or later.
<p class="note"><strong>Note:</strong> A known issue exists with older versions of JMX Bridge and the newer versions of Elastic Runtime. JMX Bridge v1.8.2 through v1.8.8 only work with Elastic Runtime v1.8.0 through v1.8.16.</p>
### <a id='cyclical_dependency'></a> Cyclic Dependency Error During Install###
You might see the following error if you have Elastic Runtime/PAS and JMX Bridge installed.
<%= image_tag("images/jmx-ert-cyclic-depedency.png") %>
* **JMX Bridge v1.8.7 or later; Elastic Runtime v1.8**:
1. The Collector component is no longer compatible with JMX Bridge. All metrics that were previously flowing through the Collector are now flowing through the Firehose in Elastic Runtime v1.8.
1. Scale down the Collector in `Elastic Runtime > Resource Config` to `0`.
1. Click **Review Pending Changes**, click **Apply Changes**, and continue your JMX Bridge installation.
<p class="note"><strong>Note:</strong> This is only an issue for Elastic Runtime v1.8. The Collector does not exist in Elastic Runtime v1.9.</p>
### <a id='no-metrics'></a> Missing Metrics from PCF Installation or Firehose
If you do not see expected metrics from Elastic Runtime in the JMX provider, verify that you installed Elastic Runtime before JMX Bridge. If you installed JMX Bridge first, do the following:
1. SSH into the **opentsdb-firehose-nozzle** VM. For information about how to use the BOSH CLI to SSH into a VM, see [Advanced Troubleshooting with the BOSH CLI](https://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html).
1. Grant **sudo** access to the machine:
<pre class="terminal">$ sudo -i</pre>
1. Restart the `opentsdb-firehose-nozzle` job.
<pre class="terminal">$ monit restart opentsdb-firehose-nozzle</pre>
### <a id='bosh-metrics'></a> Missing BOSH Metrics
If you do not see expected metrics from BOSH, do the following:
1. Make sure the IP address in **JMX Bridge > Status > JMX Provider** matches the value entered in **BOSH Director > Director Config > Metrics IP Address**.
1. If the JMX Provider IP address matches the Metrics IP address and you see no BOSH metrics in the system, contact [Pivotal Support](https://support.pivotal.io/) for help.
### <a id='mbeans'></a> Validating JMX Bridge MBeans
If you do not see metrics from JMX Bridge in your third-party tooling integration as expected, first do the following to quickly determine whether there is an issue with either the JMX Bridge product or the tooling integration:
1. Verify [Java v6+](http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html) is installed.
1. Run `jconsole`:
<pre class="terminal">$ jconsole</pre>
1. Select **Remote Process** and enter the IP of the **JMX Provider** VM with port `44444`.
1. Fill in the username and password for the **JMX Provider** that was entered during installation of JMX Bridge.
1. Click **Connect**.
<%= image_tag("images/jmx-jconsole-connnect.png") %>
1. Allow **Insecure connection** if SSL was not enabled.
<%= image_tag("images/jmx-jsonsole-insecure.png") %>
You can now view all MBeans emitted by JMX Bridge.
<%= image_tag("images/jmx-jconsole-mbeans.png") %>
<p class="note"><strong>Note:</strong> If you have enabled SSL, see <a href="self-signed-ssl.html">Using SSL with a Self-Signed Certificate in JMX Bridge</a>.</p>
##### <a id='port-forwarding'></a> Set Up Port Forwarding for JMX
If you are connecting to `jconsole` from a location different from the install location (for example, deployed on AWS or GCP), you must set up port forwarding to access the MBeans.
1. Set up port forwarding on one tab of your console and keep it open:
<pre class="terminal">ssh -D 7777 username<span>@</span>pcf.domain.com -T</pre>
1. Start `jconsole` in a new tab and set up the `socksProxyPort` to the forwarded port:
<pre class="terminal">jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777</pre>
1. Navigate `jconsole` as normal.
### <a id='smoke-tests'></a> Smoke Tests
If errors occur when the smoke tests run, you can find the errors in the **ChangeLog** for the installation. Some common failures are listed below.
<table border='1' class='nice'>
<tr>
<th width="22%">Error</th>
<td><code>internalMetricsAreSent() Fails</code>
</td>
</tr>
<tr>
<th>Cause</th>
<td>The JMX Provider did not receive internal health metrics from the OpenTSDB Firehose Nozzle.</td>
</tr>
<tr>
<th>Solution</th>
<td>Restart the OpenTSDB Firehose Nozzle VM and check the logs to verify it is running correctly.</td>
</tr>
</table>
<table border='1' class='nice'>
<tr>
<th width="22%">Error</th>
<td><code>receivingFirehoseMetrics() Fails</code>
</td>
</tr>
<tr>
<th>Cause</th>
<td>The OpenTSDB Firehose Nozzle is not receiving metrics from the Firehose.</td>
</tr>
<tr>
<th>Solution</th>
<td>Restart the OpenTSDB Firehose Nozzle VM and check the logs to verify it is connected to the Firehose. If you see a lot of reconnect attempts in the logs then you likely need to scale up the number of OpenTSDB Firehose Nozzle instances in the <strong>Resource Config</strong> tab.</td>
</tr>
</table>
</tr>
</table>
## <a id='uninstall'></a> Uninstall JMX Bridge
To uninstall the JMX Bridge for PCF tile, see [Deleting a Product](https://docs.pivotal.io/pivotalcf/customizing/add-delete.html#del-prod).