[Bug]: Published core lib are desynced with snapshots and break plugin builds

[Yury-Fridlyand]

Describe the bug

#7306 and #7165 (and number of other changes before) causes major changes in core libraries. I don't know entire workflow of publishing snapshots, but it takes much more time than publish core libs. The delay could be up to 1 week. Along this time, no plugin code could be compiled/tested because core libs and snapshots are not in sync. All development is paralyzed.

To reproduce

^

Expected behavior

Stash newer core libs before publish until snapshot is ready. Publish both together only.

Screenshots

No response

Host / Environment

No response

Additional context

No response

Relevant log output

fatal error in thread [opensearch[pc.local][sql-worker][T#1]], exiting
java.lang.NoSuchMethodError: 'java.util.Map org.opensearch.action.admin.indices.mapping.get.GetMappingsResponse.mappings()'
	at org.opensearch.sql.opensearch.client.OpenSearchNodeClient.getIndexMappings(OpenSearchNodeClient.java:91) ~[?:?]
	at org.opensearch.sql.opensearch.request.system.OpenSearchDescribeIndexRequest.getFieldTypes(OpenSearchDescribeIndexRequest.java:85) ~[?:?]
	at org.opensearch.sql.opensearch.storage.OpenSearchIndex.getFieldTypes(OpenSearchIndex.java:117) ~[?:?]
	at org.opensearch.sql.analysis.Analyzer.visitRelation(Analyzer.java:156) ~[?:?]
	at org.opensearch.sql.analysis.Analyzer.visitRelation(Analyzer.java:106) ~[?:?]
	at org.opensearch.sql.ast.tree.Relation.accept(Relation.java:98) ~[?:?]
	at org.opensearch.sql.analysis.Analyzer.visitProject(Analyzer.java:365) ~[?:?]
	at org.opensearch.sql.analysis.Analyzer.visitProject(Analyzer.java:106) ~[?:?]
	at org.opensearch.sql.ast.tree.Project.accept(Project.java:71) ~[?:?]
	at org.opensearch.sql.analysis.Analyzer.analyze(Analyzer.java:133) ~[?:?]
	at org.opensearch.sql.executor.QueryService.analyze(QueryService.java:93) ~[?:?]
	at org.opensearch.sql.executor.QueryService.execute(QueryService.java:43) ~[?:?]
	at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:50) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:33) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:47) ~[?:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) [?:?]
fatal error in thread [opensearch[pc.local][sql-worker][T#1]], exiting
java.lang.NoSuchMethodError: 'java.util.Map org.opensearch.action.admin.indices.mapping.get.GetMappingsResponse.mappings()'
	at org.opensearch.sql.opensearch.client.OpenSearchNodeClient.getIndexMappings(OpenSearchNodeClient.java:91)
	at org.opensearch.sql.opensearch.request.system.OpenSearchDescribeIndexRequest.getFieldTypes(OpenSearchDescribeIndexRequest.java:85)
	at org.opensearch.sql.opensearch.storage.OpenSearchIndex.getFieldTypes(OpenSearchIndex.java:117)
	at org.opensearch.sql.analysis.Analyzer.visitRelation(Analyzer.java:156)
	at org.opensearch.sql.analysis.Analyzer.visitRelation(Analyzer.java:106)
	at org.opensearch.sql.ast.tree.Relation.accept(Relation.java:98)
	at org.opensearch.sql.analysis.Analyzer.visitProject(Analyzer.java:365)
	at org.opensearch.sql.analysis.Analyzer.visitProject(Analyzer.java:106)
	at org.opensearch.sql.ast.tree.Project.accept(Project.java:71)
	at org.opensearch.sql.analysis.Analyzer.analyze(Analyzer.java:133)
	at org.opensearch.sql.executor.QueryService.analyze(QueryService.java:93)
	at org.opensearch.sql.executor.QueryService.execute(QueryService.java:43)
	at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:50)
	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:33)
	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:47)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)

[nknize]

This is a big reason for the refactor of these core classes out of :server into the :libs and to decouple this mess of a plugin framework from the server logic. Downstream shouldn't be so dependent on the fat build. We'll get there. In the meantime there will be some growing pains like this.

[rishabh6788]

The snapshot publishing now happens as part of code commit on opensearch repo.
Moving this issue to core repo as snapshots are currently being handled by them.

[dblock]

@Yury-Fridlyand I am not sure I understand the source of the 1 week delay. When you say "core libs", what do you mean? Isn't that part of the snapshot publication (which happens immediately on merge)?

[Yury-Fridlyand]

Hi
core, common and other jars are published immediately, but server snapshot publish depends on extra CI checks and it could be published with significant delays.
I remember observing funny, but sad situation ~2 months ago, when first iteration if migrating code from core to common was done: snapshot publish didn't run, because CI failed, because plugin build failed, because plugins depend on snapshot which is not yet published.
This dependency loop was resolved recently, but two things remain:

1. Incompatible jars are published with same versions
2. Snapshot published with a delay

[dblock]

@Yury-Fridlyand @peterzhuamazon Where do we publish core and common JARs before snapshots and why do we need to publish them at all?

[minalsha]

@Yury-Fridlyand @peterzhuamazon could you please help address @dblock questions above?

[minalsha]

@Yury-Fridlyand / @peterzhuamazon Is this still an issue? Havent heard back here. W plan to close this issue if we dont hear back from you by 7/21. Thanks

[Yury-Fridlyand]

I can't answer the given question, but issue still persists.
Breaking changes that were published recently in #8157 made us suffer twice:

1. it is breaking
2. min distribution snapshot was published few days after core snapshot

So even with having a fix to comply with those changes, we weren't able to verify it or run tests without snapshot.