[BUG] Failing PerformanceAnalyzerRCAHealthCheckIT integration test

[sruti1312]

What is the bug?
Performance Analyzer handler is not found on 9200 port causing the PerformanceAnalyzerRCAHealthCheckIT integration test to fail.

    org.opensearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/_plugins/_performanceanalyzer/metrics/?metrics=Disk_Utilization&agg=max&dim=&nodes=all], status line [HTTP/1.1 400 Bad Request]
    {"error":"no handler found for uri [/_plugins/_performanceanalyzer/metrics/?metrics=Disk_Utilization&agg=max&dim=&nodes=all] and method [GET]"}
        at __randomizedtesting.SeedInfo.seed([3C1D38B488D7B186:BA99E1ECBC7412DF]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:350)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:320)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:295)
        at app//org.opensearch.performanceanalyzer.PerformanceAnalyzerRCAHealthCheckIT.lambda$checkMetrics$0(PerformanceAnalyzerRCAHealthCheckIT.java:43)
        at app//org.opensearch.performanceanalyzer.util.WaitFor.waitFor(WaitFor.java:30)
        at app//org.opensearch.performanceanalyzer.PerformanceAnalyzerRCAHealthCheckIT.checkMetrics(PerformanceAnalyzerRCAHealthCheckIT.java:36)
        at app//org.opensearch.performanceanalyzer.PerformanceAnalyzerRCAHealthCheckIT.checkMetrics(PerformanceAnalyzerRCAHealthCheckIT.java:25)

partsrut@147dda7f4e9c performance-analyzer % curl "localhost:9200/_plugins/_performanceanalyzer/metrics/?metrics=Disk_Utilization&agg=max&dim=&nodes=all"
{"error":"no handler found for uri [/_plugins/_performanceanalyzer/metrics/?metrics=Disk_Utilization&agg=max&dim=&nodes=all] and method [GET]"}%   

https://github.com/opensearch-project/performance-analyzer-rca/runs/6313506397?check_suite_focus=true

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

What is the expected behavior?
A clear and concise description of what you expected to happen.

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

[jainankitk]

Okay, this was also missing _agent in the uri path:

org.opensearch.performanceanalyzer.PerformanceAnalyzerRCAHealthCheckIT > testLegacyRcaIsRunning STANDARD_OUT
    [2022-05-21T04:59:31,482][INFO ][o.o.p.PerformanceAnalyzerRCAHealthCheckIT] [testLegacyRcaIsRunning] before test
    [2022-05-21T04:59:31,482][INFO ][o.o.p.PerformanceAnalyzerRCAHealthCheckIT] [testLegacyRcaIsRunning] initializing REST clients against [http://localhost:9200]
    [2022-05-21T04:59:31,494][INFO ][o.o.p.PerformanceAnalyzerRCAHealthCheckIT] [testLegacyRcaIsRunning] Cluster is localhost:9200
    [2022-05-21T04:59:31,494][INFO ][o.o.p.PerformanceAnalyzerRCAHealthCheckIT] [testLegacyRcaIsRunning] initializing PerformanceAnalyzer client against [http://localhost:9200]
    [2022-05-21T05:01:31,526][INFO ][o.o.p.PerformanceAnalyzerRCAHealthCheckIT] [testLegacyRcaIsRunning] after test

Gradle Test Executor 2 finished executing tests.

> Task :integTest
Finished generating test XML results (0.02 secs) into: /Users/akjain/ws/performance-analyzer/build/test-results/integTest
Generating HTML test report...
Finished generating test html results (0.022 secs) into: /Users/akjain/ws/performance-analyzer/build/reports/tests/integTest
Stored cache entry for task ':integTest' with cache key 5202a2faa0f51332960afe813af650a4
Stopping `node{::integTest-0}`, tailLogs: false
Terminating opensearch process forcibly : commandLine:`/usr/local/Cellar/openjdk@11/11.0.12/libexec/openjdk.jdk/Contents/Home/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/Users/akjain/ws/performance-analyzer/build/testclusters/integTest-0/tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=logs -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Xms512m -Xmx512m -ea -esa -XX:MaxDirectMemorySize=268435456 -Dopensearch.path.home=/Users/akjain/.gradle/caches/transforms-3/87d9715b65fac598453b8f7553382a46/transformed/opensearch-2.0.0-SNAPSHOT.zip/opensearch-2.0.0-SNAPSHOT -Dopensearch.path.conf=/Users/akjain/ws/performance-analyzer/build/testclusters/integTest-0/config -Dopensearch.distribution.type=zip -Dopensearch.bundled_jdk=false -cp /Users/akjain/.gradle/caches/transforms-3/87d9715b65fac598453b8f7553382a46/transformed/opensearch-2.0.0-SNAPSHOT.zip/opensearch-2.0.0-SNAPSHOT/lib/* org.opensearch.bootstrap.OpenSearch` command:`/usr/local/Cellar/openjdk@11/11.0.12/libexec/openjdk.jdk/Contents/Home/bin/java` args:`'-Xshare:auto' '-Dopensearch.networkaddress.cache.ttl=60' '-Dopensearch.networkaddress.cache.negative.ttl=10' '-XX:+AlwaysPreTouch' '-Xss1m' '-Djava.awt.headless=true' '-Dfile.encoding=UTF-8' '-Djna.nosys=true' '-XX:-OmitStackTraceInFastThrow' '-Dio.netty.noUnsafe=true' '-Dio.netty.noKeySetOptimization=true' '-Dio.netty.recycler.maxCapacityPerThread=0' '-Dio.netty.allocator.numDirectArenas=0' '-Dlog4j.shutdownHookEnabled=false' '-Dlog4j2.disable.jmx=true' '-Djava.locale.providers=SPI,COMPAT' '-Xms1g' '-Xmx1g' '-XX:+UseG1GC' '-XX:G1ReservePercent=25' '-XX:InitiatingHeapOccupancyPercent=30' '-Djava.io.tmpdir=/Users/akjain/ws/performance-analyzer/build/testclusters/integTest-0/tmp' '-XX:+HeapDumpOnOutOfMemoryError' '-XX:HeapDumpPath=logs' '-XX:ErrorFile=logs/hs_err_pid%p.log' '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m' '-Xms512m' '-Xmx512m' '-ea' '-esa' '-XX:MaxDirectMemorySize=268435456' '-Dopensearch.path.home=/Users/akjain/.gradle/caches/transforms-3/87d9715b65fac598453b8f7553382a46/transformed/opensearch-2.0.0-SNAPSHOT.zip/opensearch-2.0.0-SNAPSHOT' '-Dopensearch.path.conf=/Users/akjain/ws/performance-analyzer/build/testclusters/integTest-0/config' '-Dopensearch.distribution.type=zip' '-Dopensearch.bundled_jdk=false' '-cp' '/Users/akjain/.gradle/caches/transforms-3/87d9715b65fac598453b8f7553382a46/transformed/opensearch-2.0.0-SNAPSHOT.zip/opensearch-2.0.0-SNAPSHOT/lib/*' 'org.opensearch.bootstrap.OpenSearch'`
:integTest (Thread[Execution worker for ':',5,main]) completed. Took 4 mins 16.075 secs.
Closing Git repo: /Users/akjain/ws/performance-analyzer/.git
Waiting for reaper to exit normally

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 5m 56s
10 actionable tasks: 6 executed, 4 up-to-date
Watched directory hierarchies: [/Users/akjain/ws/performance-analyzer]