add rollup API spec

additional paths

Fixed spec (opensearch-project#246)

Signed-off-by: saimedhi <saimedhi@amazon.com>

Fixed search_pipeline spec (opensearch-project#253)

Signed-off-by: saimedhi <saimedhi@amazon.com>

Updated API name: search_pipeline.create to search_pipeline.put (opensearch-project#254)

Signed-off-by: saimedhi <saimedhi@amazon.com>

Fixed search_pipeline.get spec (opensearch-project#255)

Signed-off-by: saimedhi <saimedhi@amazon.com>

Filled in Missing Defaults (opensearch-project#249)

* Filled in Missing Defaults

Signed-off-by: Theo Truong <theotr@amazon.com>

* # Wordings

Signed-off-by: Theo Truong <theotr@amazon.com>

---------

Signed-off-by: Theo Truong <theotr@amazon.com>

Replace pull_request_target workflow with two workflows that upload/download an artifact. (opensearch-project#251)

* Fix: replace pull_request_target with a download/upload artifact.

Signed-off-by: dblock <dblock@amazon.com>

* Use upload/download-artifact@v4.

Signed-off-by: dblock <dblock@amazon.com>

---------

Signed-off-by: dblock <dblock@amazon.com>

Fix: var -> const. (opensearch-project#258)

Signed-off-by: dblock <dblock@amazon.com>

Adds tools linter. (opensearch-project#260)

Signed-off-by: dblock <dblock@amazon.com>

Fix: '@typescript-eslint/indent': 'warn'. (opensearch-project#262)

Signed-off-by: dblock <dblock@amazon.com>

Removed default values from param description (opensearch-project#264)

Signed-off-by: saimedhi <saimedhi@amazon.com>

Generate _opendistro endpoints through merger tool (opensearch-project#257)

* Generate _opendistro endpoints through merger tool

Signed-off-by: Theo Truong <theotr@amazon.com>

* # Renamed `replaced` with `superseded`

Signed-off-by: Theo Truong <theotr@amazon.com>

* # Rebased DEVELOPER_GUIDE.md

Signed-off-by: Theo Truong <theotr@amazon.com>

* # Set Tabsize from 4 to 2

Signed-off-by: Theo Truong <theotr@amazon.com>

---------

Signed-off-by: Theo Truong <theotr@amazon.com>

Add _plugins/_notifications/channels API spec (opensearch-project#256)

* Add _plugins/_notifications/channels API

Signed-off-by: Sokratis Papadopoulos <sokratis.papadopoulos@cern.ch>

Fix obvious lints. (opensearch-project#265)

* Fix cosmetic autoformat lints.
* Fixed @typescript-eslint/no-unused-vars.
* Fixed eqeqeq.
* Fixed @typescript-eslint/consistent-type-imports.
* Fixed no-useless-return.
* Fixed @typescript-eslint/array-type.
* Rebased with changes on main.

Signed-off-by: dblock <dblock@amazon.com>

Update list notification channels url for externalDocs (opensearch-project#267)

Signed-off-by: Sokratis Papadopoulos <sokratis.papadopoulos@cern.ch>
Co-authored-by: Sokratis Papadopoulos <sokratis.papadopoulos@cern.ch>

Updated/Corrected Docs (opensearch-project#270)

* Updated/Corrected Docs

- README.md
- CLIENT_GENERATOR_GUIDE.md
- DEVELOPER_GUIDE.md
- ./tools/README.md

Signed-off-by: Theo Truong <theotr@amazon.com>

* # minor corrections

Signed-off-by: Theo Truong <theotr@amazon.com>

* # minor corrections

Signed-off-by: Theo Truong <theotr@amazon.com>

---------

Signed-off-by: Theo Truong <theotr@amazon.com>

Add lychee github action  for links checking (opensearch-project#269)

Corrected content type for bulk operations (opensearch-project#275)

* Corrected content type for bulk operations

Signed-off-by: Theo Truong <theotr@amazon.com>

* # linting

Signed-off-by: Theo Truong <theotr@amazon.com>

---------

Signed-off-by: Theo Truong <theotr@amazon.com>

Validate _superseded_operations.yaml against its JSON schema (opensearch-project#276)

* Validate _superseded_operations.yaml against its JSON schema

Signed-off-by: Theo Truong <theotr@amazon.com>

* # lint

Signed-off-by: Theo Truong <theotr@amazon.com>

* # lint

Signed-off-by: Theo Truong <theotr@amazon.com>

---------

Signed-off-by: Theo Truong <theotr@amazon.com>

Fixed Linting for Tools (opensearch-project#278)

Signed-off-by: Theo Truong <theotr@amazon.com>

Removed Root file since it's not needed anymore (opensearch-project#279)

Signed-off-by: Theo Truong <theotr@amazon.com>

Fixed missing global params (opensearch-project#280)

Signed-off-by: Theo Truong <theotr@amazon.com>

Added validation for _info.yaml (opensearch-project#281)

DRY'ed JSON schema validation logic

Signed-off-by: Theo Truong <theotr@amazon.com>

Implement inline object schema validator (opensearch-project#282)

* Implement inline object schema validator and underlying visitor pattern

Signed-off-by: Thomas Farr <tsfarr@amazon.com>

* Fix spec lint error

Signed-off-by: Thomas Farr <tsfarr@amazon.com>

---------

Signed-off-by: Thomas Farr <tsfarr@amazon.com>

oops

fix lint

.github/workflows/build.yml

@@ -25,7 +25,7 @@ jobs:
         run: |-
           mkdir -p ../build
           npm install
-          export ROOT_PATH=../spec/opensearch-openapi.yaml
+          export ROOT_PATH=../spec
           export OUTPUT_PATH=../build/opensearch-openapi.yaml
           npm run merge -- $ROOT_PATH $OUTPUT_PATH
 

.github/workflows/coverage-comment.yml

@@ -0,0 +1,36 @@
+name: Comment with API Coverage
+
+on:
+  workflow_run:
+    workflows: ["Gather API Coverage"]
+    types:
+      - completed
+
+jobs:
+  comment:
+    runs-on: ubuntu-latest
+    if: >
+      github.event.workflow_run.event == 'pull_request' &&
+      github.event.workflow_run.conclusion == 'success'
+    steps:
+      - name: Download Coverage Report
+        uses: actions/download-artifact@v4
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          name: coverage
+          run-id: ${{ github.event.workflow_run.id }}
+
+      - name: 'Comment on PR'
+        uses: actions/github-script@v3
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          script: |
+            const fs = require('fs');
+            const data = JSON.parse(fs.readFileSync('./coverage.json'));
+            console.log(data);
+            await github.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: data.pull_request,
+              body: `API specs implemented for ${data.current}/${data.total} (${data.percent}%) APIs.`
+            });

.github/workflows/coverage.yml → .github/workflows/coverage-gather.yml

@@ -1,6 +1,6 @@
-name: API Coverage
+name: Gather API Coverage
 
-on: [push, pull_request_target]
+on: [push, pull_request]
 
 env:
   JAVA_VERSION: 11
@@ -14,14 +14,12 @@ jobs:
     steps:
       - name: Checkout Repo
         uses: actions/checkout@v2
-        with:
-           ref: ${{ github.event.pull_request.head.sha }}
       - name: Build Spec
         working-directory: ./tools
         run: |-
           mkdir -p ../build
           npm install
-          export ROOT_PATH=../spec/opensearch-openapi.yaml
+          export ROOT_PATH=../spec
           export OUTPUT_PATH=../build/opensearch-openapi.yaml
           npm run merge -- $ROOT_PATH $OUTPUT_PATH
       - name: Build and Run Docker Container
@@ -50,14 +48,15 @@ jobs:
           total=`jq -r '.paths | keys | length' build/local-openapi.json`
           percent=$((current * 100 / total))
           echo "API specs implemented for $current/$total ($percent%) APIs."
-          cat >>"$GITHUB_OUTPUT" <<EOL
-          current=$current
-          total=$total
-          percent=$percent
+          cat >>"coverage.json" <<EOL
+          {
+            "pull_request":${{ github.event.number }},
+            "current":$current,
+            "total":$total,
+            "percent":$percent
+          }
           EOL
-      - uses: peter-evans/create-or-update-comment@v4
-        if: github.event_name == 'pull_request_target'
+      - uses: actions/upload-artifact@v4
         with:
-          issue-number: ${{ github.event.number }}
-          body: |
-            API specs implemented for ${{ steps.coverage.outputs.current }}/${{ steps.coverage.outputs.total }} (${{ steps.coverage.outputs.percent }}%) APIs.
+          name: coverage
+          path: coverage.json

.github/workflows/links.yml

@@ -0,0 +1,17 @@
+name: Links
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  linkChecker:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Link Checker
+        uses: lycheeverse/lychee-action@v1
+        with:
+          fail: true
+          args: --base . --verbose --no-progress './**/*.yaml' './**/*.yml' './**/*.md' './**/*.json'  './**/*.ts' --exclude-path ./tools/package-lock.json

.github/workflows/lint.yml

@@ -24,4 +24,5 @@ jobs:
         with:
           node-version: 20.10.0
       - run: npm install
-      - run: npm run lint -- ../spec
+      - run: |
+          npm run lint:spec -- ../spec

.github/workflows/tools.yml

@@ -23,3 +23,4 @@ jobs:
           node-version: 20.10.0
       - run: npm install
       - run: npm run test
+      - run: npm run lint

.lycheeignore

@@ -0,0 +1 @@
+https://localhost:9200*

CLIENT_GENERATOR_GUIDE.md

@@ -19,13 +19,13 @@ In a client, the `search` operations are grouped in to a single API method, `cli
 
 In the [published OpenAPI spec](https://github.com/opensearch-project/opensearch-api-specification/releases), this grouping is denoted by `x-operation-group` vendor extension in every operation definition. The value of this extension is the name of the API action (like `search` or `indices.get_field_mapping`). Operations with the same `x-operation-group` value are guaranteed to have the same query string parameters, response body, and request body (for PUT/POST/DELETE operations). Common path parameters are also guaranteed to be the same. The only differences between operations are the HTTP method and the path. With that in mind, below are rules on how to combine operations of different HTTP methods and path compositions.
 
-- If an operation is marked with `x-ignorable: "true"`, then ignore the operation. Such an operation has been deprecated and has been replaced by a newer one. As far as the clients are concerned, ignorable operations do not exist.
+- If an operation is marked with `x-ignorable: true`, then ignore the operation. Such an operation has been deprecated and has been superseded by a newer one. As far as the clients are concerned, ignorable operations do not exist.
 - If two operations have identical HTTP methods, but different paths: use the path that best matches the path parameters provided.
 - If two operations have identical path, but different HTTP methods:
     - GET/POST: if the request body is provided then use POST, otherwise use GET
     - PUT/POST: Either works, but PUT is preferred when an optional path parameter is provided.
 
-The psuedo-code that combines the `search` operations into a single API method is as follows:
+The pseudocode that combines the `search` operations into a single API method is as follows:
 ```python
 def search(self, index=None, body=None):
     if index is None:
@@ -41,14 +41,21 @@ def search(self, index=None, body=None):
     return self.perform_request(method, path, body=body)
 ```
 
-
 ## Overloaded Name
-You will also encounter `x-overloaded-param: "metric"` for the `node_id` path parameter of `GET /_nodes/{node_id}` operation in `nodes.info` action. This is a special case where the path parameter is overloaded to accept either a node ID or a metric name. The `client.nodes.info` method when called with either `metric` or `node_id` (but not both), will use `GET /_nodes/{node_id}` operation (even though the path parameter name is `node_id`). When called with both `metric` and `node_id`, it will use `GET /_nodes/{node_id}/{metric}` operation.
+You will also encounter `x-overloaded-param: metric` for the `node_id` path parameter of the `GET /_nodes/{node_id}` operation in `nodes.info` action. This is a special case where the path parameter is overloaded to accept either a node ID or a metric name. When the user evokes the `client.nodes.info` method with either `metric` or `node_id` (but not both), the method will use the `GET /_nodes/{node_id}` operation. When evoked with both `metric` and `node_id`, it will use the `GET /_nodes/{node_id}/{metric}` operation.
 
 ## Handling Bulk Operations
-Some operations accept a bulk of data in the request body. For example, the `bulk` action accepts a bulk of index, update, and delete operations on multiple documents. Unlike other operations where the request body is a JSON object, the request body for bulk operations is a newline-seperated JSON string. The client will automatically convert the request body into a newline-seperated JSON objects. The request body of such operations will be denoted with `x-serialize: "bulk"` vendor extension.
+Some operations accept a bulk of data in the request body. For example, the `bulk` action accepts a bulk of index, update, and delete operations on multiple documents. Unlike other operations where the request body is a **JSON object**, the request body for bulk operations is an **NDJSON** (i.e a [Newline-delimited JSON](https://github.com/ndjson/ndjson-spec)). 
+
+In the spec, these request bodies have the `content` of `application/x-ndjson` and their schemas are of type array. In this situation, the client must serialize the request body accordingly, and set the `Content-Type` header to `application/x-ndjson`.
 
 ## Parameter Validation
 As of right now, most clients only validate whether required parameters are present. The clients do not validate the values of parameters against the enum values or regex patterns. This is to reduce performance overhead for the clients as the validation is already done on the server. However, the list of enum values and regex patterns are often written into the parameter description.
 
-Some clients also check for the validity of query string parameter names to guard the users from typos. If you decide to implement this feature, make sure that it's performant. Scripting languages like Python and Ruby require the code to be loaded into memory at runtime, and constructs used for this feature can be expensive to load, as far as micro-services are concerned.
+Some clients also check for the validity of query string parameter names to guard the users from typos. If you decide to implement this feature, make sure that it's performant. Scripting languages like Python and Ruby require the code to be loaded into memory at runtime, and constructs used for this feature can be expensive to load, as far as microservices are concerned.
+
+## Global Parameters
+All operations in the spec contain a set of parameters that are common across all operations. These parameters are denoted with `x-global: true` vendor extension. The generated clients should find a way to DRY these parameters in type definitions and method documentation.
+
+## Default Parameter Values
+Parameters can have default values either through schema or the `x-default` vendor extension. When both are present, `x-default` will take precedence.

DEVELOPER_GUIDE.md

@@ -3,8 +3,10 @@
   - [File Structure](#file-structure)
   - [Grouping Operations](#grouping-operations)
   - [Grouping Schemas](#grouping-schemas)
+  - [Superseded Operations](#superseded-operations)
+  - [Global Parameters](#global-parameters)
   - [OpenAPI Extensions](#openapi-extensions)
-  - [Linting](#linting)
+  - [Tools](#tools)
 
 # Developer Guide
 
@@ -20,10 +22,6 @@ The Specification is written in OpenAPI 3, so understanding the OpenAPI 3 specif
 
 To make editing the specification easier, we split the OpenAPI spec into multiple files that can be found in the [spec](spec) directory. The file structure is as follows:
 
-- The API Operations are grouped by namespaces in [spec/namespaces](spec/namespaces/) directory. Each `.yaml` file in this directory represents a namespace and holds all paths and operations of the namespace.
-- The data schemas are grouped by categories in [spec/schemas](spec/schemas/) directory. Each `.yaml` file in this directory represents a category.
-- The [spec/opensearch-openapi.yaml](spec/opensearch-openapi.yaml) file is the OpenAPI root file that ties everything together.
-
 ```
 spec
 │
@@ -43,9 +41,15 @@ spec
 │   ├── cat.aliases.yaml
 │   └── ...
 │
-└── opensearch-openapi.yaml
+├── _info.yaml
+├── _global_parameters.yaml
+└── _superseded_operations.yaml
 ```
-Every `.yaml` file is a valid OpenAPI 3 document. This means that you can use any OpenAPI 3 compatible tool to view and edit the files, and IDEs with OpenAPI support will provide you with autocompletion and validation in real-time.
+
+- The API Operations are grouped by namespaces in [spec/namespaces/](spec/namespaces) directory. Each file in this directory represents a namespace and holds all paths and operations of the namespace.
+- The data schemas are grouped by categories in [spec/schemas/](spec/schemas) directory. Each file in this directory represents a category.
+
+Every `.yaml` file in the namespaces and schemas folders is a OpenAPI 3 document. This means that you can use any OpenAPI 3 compatible tool to view and edit the files, and IDEs with OpenAPI support will also offer autocomplete and validation in realtime.
 
 ## Grouping Operations
 
@@ -64,25 +68,62 @@ For this reason, every operation *must* be accompanied by the `x-operation-group
 
 ## Grouping Schemas
 
-Schemas are grouped by categories to keep their names short and aid in client generation:
+Schemas are grouped by categories to keep their names short, and aid in client generation (where the schemas are translated into data types/classes, and divided into packages/modules). The schema file names can be in one of the following formats:
 
 - `_common` category holds the common schemas that are used across multiple namespaces and features.
 - `_common.<sub_category>` category holds the common schemas of a specific sub_category. (e.g. `_common.mapping`)
 - `<namespace>._common` category holds the common schemas of a specific namespace. (e.g. `cat._common`, `_core._common`)
 - `<namespace>.<action>` category holds the schemas of a specific sub_category of a namespace. (e.g. `cat.aliases`, `_core.search`)
 
+## Superseded Operations
+
+When an operation is superseded by another operation with **identical functionality**, that is a rename or a change in the URL, it should be listed in [_superseded_operations.yaml](./spec/_superseded_operations.yaml) file. The merger tool will automatically generate the superseded operation in the OpenAPI spec. The superseded operation will have `deprecated: true` and `x-ignorable: true` properties to indicate that it should be ignored by the client generator.
+
+For example, if the `_superseded_operations.yaml` file contains the following entry:
+```yaml
+/_opendistro/_anomaly_detection/{nodeId}/stats/{stat}:
+  superseded_by: /_plugins/_anomaly_detection/{nodeId}/stats/{stat}
+  operations:
+    - GET
+    - POST
+```
+Then, the merger tool will generate 2 superseded operations: 
+- `GET /_opendistro/_anomaly_detection/{nodeId}/stats/{stat}` 
+- `POST /_opendistro/_anomaly_detection/{nodeId}/stats/{stat}`
+
+from their respective superseding operations:
+
+- `GET /_plugins/_anomaly_detection/{nodeId}/stats/{stat}`
+- `POST /_plugins/_anomaly_detection/{nodeId}/stats/{stat}`
+
+if and only if the superseding operations exist in the spec. A warning will be printed on the console if they do not. 
+
+Note that the path parameter names do not need to match. So, if the actual superseding operations have path of `/_plugins/_anomaly_detection/{node_id}/stats/{stat_id}`, the merger tool will recognize that it is the same as `/_plugins/_anomaly_detection/{nodeId}/stats/{stat}` and generate the superseded operations accordingly with the correct path parameter names.
+
+## Global Parameters
+Certain query parameters are global, and they are accepted by every operation. These parameters are listed in the [spec/_global_parameters.yaml](spec/_global_parameters.yaml). The merger tool will automatically add these parameters to all operations.
+
 ## OpenAPI Extensions
 
-This repository includes several penAPI Specification Extensions to fill in any metadata not directly supported OpenAPI:
+This repository includes several OpenAPI Specification Extensions to fill in any metadata not natively supported by OpenAPI:
 
 - `x-operation-group`: Used to group operations into API actions.
 - `x-version-added`: OpenSearch version when the operation/parameter was added.
 - `x-version-deprecated`: OpenSearch version when the operation/parameter was deprecated.
 - `x-version-removed`: OpenSearch version when the operation/parameter was removed.
 - `x-deprecation-message`: Reason for deprecation and guidance on how to prepare for the next major version.
 - `x-ignorable`: Denotes that the operation should be ignored by the client generator. This is used in operation groups where some operations have been replaced by newer ones, but we still keep them in the specs because the server still supports them.
-- `x-global`: Denotes that the parameter is a global parameter that is included in every operation. These parameters are listed in the [root file](spec/opensearch-openapi.yaml).
+- `x-global`: Denotes that the parameter is a global parameter that is included in every operation. These parameters are listed in the [spec/_global_parameters.yaml](spec/_global_parameters.yaml).
+- `x-default`: Contains the default value of a parameter. This is often used to override the default value specified in the schema, or to avoid accidentally changing the default value when updating a shared schema.
+
+## Tools
+
+We authored a number of tools to merge and lint specs that live in [tools](tools). All tools have tests (run with `npm run test`) and a linter (run with `npm run lint`).
+
+### Merger
+
+The spec merger "builds", aka combines all `.yaml` files in a spec folder into a complete OpenAPI spec. A [workflow](./.github/workflows/build.yml) performs this task on the [spec folder](spec) of this repo then publishes the output into [releases](https://github.com/opensearch-project/opensearch-api-specification/releases).
 
-## Linting
-We have a linter that validates every `yaml` file in the `./spec` folder to assure that they follow the guidelines we have set. Check out the [Linter](tools/README.md#linter) tool for more information on how to run it locally. Make sure to run the linter before submitting a PR.
+### Linter
 
+The spec linter that validates every `.yaml` file in the `./spec` folder to assure that they follow the guidelines we have set. Check out the [Linter README](tools/README.md#spec-linter) for more information on how to run it locally. Make sure to run the linter before submitting a PR.

README.md

@@ -7,7 +7,8 @@
 - [Project Resources](#project-resources)
 - [Code of Conduct](#code-of-conduct)
 - [Developer Guide](#developer-guide)
-- [OpenSearch API Specs](#opensearch-api-specs)
+- [Client Generator Guide](#client-generator-guide)
+- [Published Spec](#published-spec)
 - [Security](#security)
 - [License](#license)
 - [Copyright](#copyright)
@@ -38,10 +39,17 @@ This project has adopted the [Amazon Open Source Code of Conduct](CODE_OF_CONDUC
 
 See [DEVELOPER_GUIDE](DEVELOPER_GUIDE.md).
 
-## OpenSearch API Specs
+## Client Generator Guide
+
+See [CLIENT_GENERATOR_GUIDE](CLIENT_GENERATOR_GUIDE.md).
+
+## Published Spec
 
 OpenSearch API Specs are hosted at https://opensearch-project.github.io/opensearch-api-specification/. See [PUBLISHING_SPECS](PUBLISHING_SPECS.md) for more information.
 
+Click [here](https://github.com/opensearch-project/opensearch-api-specification/releases/download/main/opensearch-openapi.yaml) to download the latest OpenSearch OpenAPI yaml file.
+
+
 ## Security
 
 If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/) or directly via email to aws-security@amazon.com. Please do **not** create a public GitHub issue.

_plugins/openapi.rb

@@ -4,7 +4,7 @@ def self.generate(_site, _payload)
 
     Dir.chdir('tools') do
       system 'npm install'
-      system 'npm run merge -- ../spec/opensearch-openapi.yaml ../_site/opensearch-openapi.yaml'
+      system 'npm run merge -- ../spec ../_site/opensearch-openapi.yaml'
     end
 
     @generated = true

coverage/README.md

@@ -2,4 +2,4 @@
 
 Builds the OpenAPI spec, and uses the [opensearch-api plugin](https://github.com/dblock/opensearch-api) and [openapi-diff](https://github.com/OpenAPITools/openapi-diff) to show the differences.
 
-API coverage is run on all pull requests via the [coverage workflow](../.github/workflows/coverage.yml).
+API coverage is run on all pull requests via the [coverage workflow](../.github/workflows/coverage-gather.yml).