Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/main' into fix_ingest_bug
Browse files Browse the repository at this point in the history
  • Loading branch information
gaobinlong committed Aug 14, 2023
2 parents 2cea65b + db768d1 commit 3de7e8f
Show file tree
Hide file tree
Showing 60 changed files with 1,184 additions and 175 deletions.
3 changes: 2 additions & 1 deletion .github/workflows/backport.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ jobs:
installation_id: 22958780

- name: Backport
uses: VachaShah/backport@v2.1.0
uses: VachaShah/backport@v2.2.0
with:
github_token: ${{ steps.github_app_token.outputs.token }}
head_template: backport/backport-<%= number %>-to-<%= base %>
failure_labels: backport-failed
2 changes: 1 addition & 1 deletion .github/workflows/links.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- uses: actions/checkout@v2
- name: lychee Link Checker
id: lychee
uses: lycheeverse/lychee-action@v1.2.0
uses: lycheeverse/lychee-action@v1.8.0
with:
args: --accept=200,403,429 --exclude-mail **/*.html **/*.md **/*.txt **/*.json --exclude-file .lychee.excludes
fail: true
Expand Down
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Allow mmap to use new JDK-19 preview APIs in Apache Lucene 9.4+ ([#5151](https://github.com/opensearch-project/OpenSearch/pull/5151))
- Add events correlation engine plugin ([#6854](https://github.com/opensearch-project/OpenSearch/issues/6854))
- Introduce new dynamic cluster setting to control slice computation for concurrent segment search ([#9107](https://github.com/opensearch-project/OpenSearch/pull/9107))
- Implement on behalf of token passing for extensions ([#8679](https://github.com/opensearch-project/OpenSearch/pull/8679))
- [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them ([#9223](https://github.com/opensearch-project/OpenSearch/pull/9223))

### Dependencies
- Bump `log4j-core` from 2.18.0 to 2.19.0
Expand Down Expand Up @@ -86,6 +88,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Disallow compression level to be set for default and best_compression index codecs ([#8737]()https://github.com/opensearch-project/OpenSearch/pull/8737)
- Prioritize replica shard movement during shard relocation ([#8875](https://github.com/opensearch-project/OpenSearch/pull/8875))
- Introducing Default and Best Compression codecs as their algorithm name ([#9123]()https://github.com/opensearch-project/OpenSearch/pull/9123)
- Make SearchTemplateRequest implement IndicesRequest.Replaceable ([#9122]()https://github.com/opensearch-project/OpenSearch/pull/9122)

### Dependencies
- Bump `org.apache.logging.log4j:log4j-core` from 2.17.1 to 2.20.0 ([#8307](https://github.com/opensearch-project/OpenSearch/pull/8307))
Expand All @@ -108,6 +111,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump `org.apache.maven:maven-model` from 3.9.3 to 3.9.4 ([#9148](https://github.com/opensearch-project/OpenSearch/pull/9148))
- Bump `com.azure:azure-storage-blob` from 12.22.3 to 12.23.0 ([#9231](https://github.com/opensearch-project/OpenSearch/pull/9231))
- Bump `com.diffplug.spotless` from 6.19.0 to 6.20.0 ([#9227](https://github.com/opensearch-project/OpenSearch/pull/9227))
- Bump `org.xerial.snappy:snappy-java` from 1.1.8.2 to 1.1.10.3 ([#9252](https://github.com/opensearch-project/OpenSearch/pull/9252))
- Bump `com.squareup.okhttp3:okhttp` from 4.9.3 to 4.11.0 ([#9252](https://github.com/opensearch-project/OpenSearch/pull/9252))
- Bump `com.squareup.okio:okio` from 2.8.0 to 3.5.0 ([#9252](https://github.com/opensearch-project/OpenSearch/pull/9252))
- Bump `com.google.code.gson:gson` from 2.9.0 to 2.10.1 ([#9230](https://github.com/opensearch-project/OpenSearch/pull/9230))
- Bump `lycheeverse/lychee-action` from 1.2.0 to 1.8.0 ([#9228](https://github.com/opensearch-project/OpenSearch/pull/9228))
- Bump `snakeyaml` from 2.0 to 2.1 ([#9269](https://github.com/opensearch-project/OpenSearch/pull/9269))

### Changed
- Perform aggregation postCollection in ContextIndexSearcher after searching leaves ([#8303](https://github.com/opensearch-project/OpenSearch/pull/8303))
Expand All @@ -121,6 +130,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- [Remove] Deprecated Fractional ByteSizeValue support #9005 ([#9005](https://github.com/opensearch-project/OpenSearch/pull/9005))
- Make MultiBucketConsumerService thread safe to use across slices during search ([#9047](https://github.com/opensearch-project/OpenSearch/pull/9047))
- Change shard_size and shard_min_doc_count evaluation to happen in shard level reduce phase ([#9085](https://github.com/opensearch-project/OpenSearch/pull/9085))
- Add attributes to startSpan methods ([#9199](https://github.com/opensearch-project/OpenSearch/pull/9199))

### Deprecated

Expand Down
24 changes: 15 additions & 9 deletions DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -569,13 +569,19 @@ use Version checks accordingly (e.g., `Version.onOrAfter`, `Version.before`) to

#### Developer API

The Developer API consists of interfaces and foundation software implementations that enable external users to develop new OpenSearch features. This includes
obvious components such as the Plugin framework and less obvious components such as REST Action Handlers. When developing a new feature of OpenSearch it is important
to explicitly mark which implementation components may, or may not, be extended by external implementations. For example, all new API classes with `@opensearch.api`
signal that the new component may be extended by an external implementation and therefore provide backwards compatibility guarantees. Similarly, any class explicitly
marked with the `@opensearch.internal` annotation, or not explicitly marked by an annotation should not be extended by external implementation components as it does not
guarantee backwards compatibility and may change at any time. The `@deprecated` annotation should also be added to any `@opensearch.api` classes or methods that are
either changed or planned to be removed across minor versions.
The Developer API consists of interfaces and foundation software implementations that enable external users to develop new OpenSearch features. This includes obvious
components such as the Plugin and Extension frameworks and less obvious components such as REST Action Handlers. When developing a new feature of OpenSearch it is
important to explicitly mark which implementation components may, or may not, be extended by external implementations. For example, all new API classes with
`@PublicApi` annotation (or documented as `@opensearch.api`) signal that the new component may be extended by an external implementation and therefore provide
backwards compatibility guarantees. Similarly, any class explicitly marked with the `@InternalApi` (or documented as `@opensearch.internal`) annotation, or not
explicitly marked by an annotation should not be extended by external implementation components as it does not guarantee backwards compatibility and may change at
any time. The `@DeprecatedApi` annotation could also be added to any classes annotated with `@PublicApi` (or documented as `@opensearch.api`) or their methods that
are either changed (with replacement) or planned to be removed across major versions.

The APIs which are designated to be public but have not been stabilized yet should be marked with `@ExperimentalApi` (or documented as `@opensearch.experimental`)
annotation. The presence of this annotation signals that API may change at any time (major, minor or even patch releases). In general, the classes annotated with
`@PublicApi` may expose other classes or methods annotated with `@ExperimentalApi`, in such cases the backward compatibility guarantees would not apply to latter
(see please [Experimental Development](#experimental-development) for more details).

#### User API

Expand All @@ -592,8 +598,8 @@ and a log message to the OpenSearch deprecation log files using the `Deprecation
Rapidly developing new features often benefit from several release cycles before committing to an official and long term supported (LTS) API. To enable this cycle OpenSearch
uses an Experimental Development process leveraging [Feature Flags](https://featureflags.io/feature-flags/). This allows a feature to be developed using the same process as
a LTS feature but with additional guard rails and communication mechanisms to signal to the users and development community the feature is not yet stable, may change in a future
release, or be removed altogether. Any Developer or User APIs implemented along with the experimental feature should be marked with the `@opensearch.experimental` annotation to
signal the implementation is not subject to LTS and does not follow backwards compatibility guidelines.
release, or be removed altogether. Any Developer or User APIs implemented along with the experimental feature should be marked with `@ExperimentalApi` (or documented as
`@opensearch.experimental`) annotation to signal the implementation is not subject to LTS and does not follow backwards compatibility guidelines.

### Backports

Expand Down
2 changes: 1 addition & 1 deletion buildSrc/version.properties
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ spatial4j = 0.7
jts = 1.15.0
jackson = 2.15.2
jackson_databind = 2.15.2
snakeyaml = 2.0
snakeyaml = 2.1
icu4j = 70.1
supercsv = 2.4.0
log4j = 2.20.0
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.common.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Target;

/**
* Marks the public APIs as deprecated and scheduled for removal in one of the upcoming
* major releases. The types marked with this annotations could only be other {@link PublicApi}s.
*
* @opensearch.api
*/
@Documented
@Target({ ElementType.TYPE, ElementType.PACKAGE, ElementType.METHOD, ElementType.CONSTRUCTOR })
@PublicApi(since = "2.10.0")
public @interface DeprecatedApi {
/**
* Version since this API is deprecated
*/
String since();

/**
* Next major version when this API is scheduled for removal
*/
String forRemoval() default "";
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.common.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Target;

/**
* Experimental APIs that may not retain source and binary compatibility within major,
* minor or patch releases. The types marked with this annotations could only expose
* other {@link PublicApi} or {@link ExperimentalApi} types as public members.
*
* @opensearch.api
*/
@Documented
@Target({ ElementType.TYPE, ElementType.PACKAGE, ElementType.METHOD, ElementType.CONSTRUCTOR })
@PublicApi(since = "2.10.0")
public @interface ExperimentalApi {

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.common.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Target;

/**
* Internal APIs that have no compatibility guarantees and should be not used outside
* of OpenSearch core components.
*
* @opensearch.api
*/
@Documented
@Target({ ElementType.TYPE, ElementType.PACKAGE, ElementType.METHOD, ElementType.CONSTRUCTOR })
@PublicApi(since = "2.10.0")
public @interface InternalApi {

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.common.annotation;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Target;

/**
* Stable public APIs that retain source and binary compatibility within a major release.
* These interfaces can change from one major release to another major release
* (e.g. from 1.0 to 2.0). The types marked with this annotations could only expose
* other {@link PublicApi} or {@link ExperimentalApi} types as public members.
*
* @opensearch.api
*/
@Documented
@Target({ ElementType.TYPE, ElementType.PACKAGE, ElementType.METHOD, ElementType.CONSTRUCTOR })
@PublicApi(since = "2.10.0")
public @interface PublicApi {
/**
* Version when this API was released
*/
String since();
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

/**
* The OpenSearch API related annotations
*
* @opensearch.api
*/
@PublicApi(since = "2.10.0")
package org.opensearch.common.annotation;
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

import java.io.Closeable;
import java.io.IOException;
import org.opensearch.telemetry.tracing.attributes.Attributes;

/**
*
Expand Down Expand Up @@ -37,16 +38,21 @@ public DefaultTracer(TracingTelemetry tracingTelemetry, TracerContextStorage<Str

@Override
public SpanScope startSpan(String spanName) {
return startSpan(spanName, null);
return startSpan(spanName, Attributes.EMPTY);
}

@Override
public SpanScope startSpan(String spanName, SpanContext parentSpan) {
public SpanScope startSpan(String spanName, Attributes attributes) {
return startSpan(spanName, null, attributes);
}

@Override
public SpanScope startSpan(String spanName, SpanContext parentSpan, Attributes attributes) {
Span span = null;
if (parentSpan != null) {
span = createSpan(spanName, parentSpan.getSpan());
span = createSpan(spanName, parentSpan.getSpan(), attributes);
} else {
span = createSpan(spanName, getCurrentSpanInternal());
span = createSpan(spanName, getCurrentSpanInternal(), attributes);
}
setCurrentSpanInContext(span);
addDefaultAttributes(span);
Expand Down Expand Up @@ -74,8 +80,8 @@ private void endSpan(Span span) {
}
}

private Span createSpan(String spanName, Span parentSpan) {
return tracingTelemetry.createSpan(spanName, parentSpan);
private Span createSpan(String spanName, Span parentSpan, Attributes attributes) {
return tracingTelemetry.createSpan(spanName, parentSpan, attributes);
}

private void setCurrentSpanInContext(Span span) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
package org.opensearch.telemetry.tracing;

import java.io.Closeable;
import org.opensearch.telemetry.tracing.attributes.Attributes;

/**
* Tracer is the interface used to create a {@link Span}
Expand All @@ -27,12 +28,22 @@ public interface Tracer extends Closeable {
SpanScope startSpan(String spanName);

/**
* Started the {@link Span} with the given name and parent.
* Starts the {@link Span} with given name and attributes. This is required in cases when some attribute based
* decision needs to be made before starting the span. Very useful in the case of Sampling.
* @param spanName span name.
* @param attributes attributes to be added.
* @return scope of the span, must be closed with explicit close or with try-with-resource
*/
SpanScope startSpan(String spanName, Attributes attributes);

/**
* Starts the {@link Span} with the given name, parent and attributes.
* @param spanName span name.
* @param parentSpan parent span.
* @param attributes attributes to be added.
* @return scope of the span, must be closed with explicit close or with try-with-resource
*/
SpanScope startSpan(String spanName, SpanContext parentSpan);
SpanScope startSpan(String spanName, SpanContext parentSpan, Attributes attributes);

/**
* Returns the current span.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
package org.opensearch.telemetry.tracing;

import java.io.Closeable;
import org.opensearch.telemetry.tracing.attributes.Attributes;

/**
* Interface for tracing telemetry providers
Expand All @@ -21,9 +22,10 @@ public interface TracingTelemetry extends Closeable {
* Creates span with provided arguments
* @param spanName name of the span
* @param parentSpan span's parent span
* @param attributes attributes to be added.
* @return span instance
*/
Span createSpan(String spanName, Span parentSpan);
Span createSpan(String spanName, Span parentSpan, Attributes attributes);

/**
* provides tracing context propagator
Expand Down
Loading

0 comments on commit 3de7e8f

Please sign in to comment.