-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add generate_library.sh
without post processing
#1916
Changes from 47 commits
53abd0e
434cbfc
5825bcc
c39cbea
10a1c6d
cca6770
89bbedc
3e85e33
fdf9260
f3b60a6
1b4746d
2856709
81cbdc2
d733c5a
92660ff
07f3ea6
49542f7
d7b3752
0c8aa78
3d612f8
280a571
d375848
734e1eb
9a5a7cd
35ad197
8477b74
5eb0223
b12f99e
e893df3
a7d8a55
2a9f8a8
51beff7
f7e4b72
32463da
b58a802
d7dcdca
4e525a6
64ee89e
56e418a
a056ae1
7ef8278
4f0ef4a
5805e84
880c27b
a5fe171
f6c5784
5d16f16
4f5689a
b38f907
1648254
b896cad
f7cb5f7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
on: | ||
push: | ||
branches: | ||
- main | ||
pull_request: | ||
paths: | ||
- library_generation/** | ||
|
||
workflow_dispatch: | ||
name: verify_library_generation_against_googleapis-gen | ||
jobs: | ||
verify_library_generation: | ||
runs-on: ubuntu-22.04 | ||
strategy: | ||
matrix: | ||
java: [ 8 ] | ||
steps: | ||
- uses: actions/checkout@v3 | ||
- uses: actions/setup-java@v3 | ||
with: | ||
java-version: ${{ matrix.java }} | ||
distribution: temurin | ||
cache: maven | ||
- name: Run integration tests | ||
run: | | ||
set -x | ||
library_generation/generate_library_integration_test.sh \ | ||
-p google/bigtable/v2 \ | ||
-d google-cloud-bigtable-v2-java \ | ||
--googleapis_gen_url https://cloud-java-bot:${{ secrets.CLOUD_JAVA_BOT_GITHUB_TOKEN }}@github.com/googleapis/googleapis-gen.git |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
# Generate GAPIC Client Library without post-processing | ||
|
||
The script, `generate_library.sh`, allows you to generate a GAPIC client library from proto files. | ||
|
||
## Environment | ||
|
||
Use Linux environment and install java runtime environment (8 or above). | ||
|
||
## Prerequisite | ||
In order to generate a GAPIC library, you need to pull `google/` from [googleapis](https://github.com/googleapis/googleapis) | ||
and put it into the directory containing `generate_library.sh` since protos in `google/` | ||
are likely referenced by protos from which the library are generated. | ||
|
||
In most cases, the `proto_path` is a subdirectory within `google/`. | ||
JoeWang1127 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## Parameters to run `generate_library.sh` | ||
|
||
You need to run the script with the following parameters. | ||
|
||
### proto_path | ||
A directory in the current working directory (refers as `$cwd`) and copy proto files into it. | ||
The absolute path of `proto_path` is `$cwd/$proto_path`. | ||
|
||
Use `-p` or `--proto_path` to specify the value. | ||
|
||
### destination_path | ||
A directory within `$cwd`. | ||
This is the path in which the generated library will reside. | ||
The absolute path of `destination_path` is `$cwd/$destination_path`. | ||
|
||
Use `-d` or `--destination_path` to specify the value. | ||
|
||
Note that you do not need to create `$detination_path` beforehand. | ||
|
||
The directory structure of the generated library is | ||
``` | ||
$destination_path | ||
JoeWang1127 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|_gapic-* | ||
| |_src | ||
| |_main | ||
| |_java | ||
| |_resources | ||
| |_test | ||
|_grpc-* | ||
| |_src | ||
| |_main | ||
| |_java | ||
| | ||
|_proto-* | ||
|_src | ||
|_main | ||
|_java | ||
|_proto | ||
``` | ||
You can't build the library as-is since it does not have `pom.xml` or `build.gradle`. | ||
To use the library, copy the generated files to the corresponding directory | ||
of a library repository, e.g., `google-cloud-java`. | ||
|
||
### version of gapic-generator-java | ||
You can find the released version of gapic-generator-java in [maven central](https://repo1.maven.org/maven2/com/google/api/gapic-generator-java/). | ||
|
||
Use `--gapic_generator_version` to specify the value. | ||
|
||
Note that you can specify a `SNAPSHOT` version as long as you have build the SNAPSHOT of gapic-generator-java in your maven | ||
local repository. | ||
|
||
### version of protobuf (optional) | ||
You can find the released version of protobuf in [GitHub](https://github.com/protocolbuffers/protobuf/releases/). | ||
The default value is defined in `gapic-generator-java-pom-parent/pom.xml`. | ||
|
||
Use `--protobuf_version` to specify the value. | ||
|
||
Note that if specified, the version should be compatible with gapic-generator-java. | ||
|
||
### version of grpc (optional) | ||
You can find the released version of grpc in [maven central](https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/). | ||
The default value is defined in `gapic-generator-java-pom-parent/pom.xml`. | ||
|
||
Use `--grpc_version` to specify the value. | ||
|
||
Note that if specified, the version should be compatible with gapic-generator-java. | ||
|
||
### transport (optional) | ||
One of GAPIC options passed to the generator. The value is either `grpc` or `grpc+rest`. | ||
The default value is `grpc`. | ||
|
||
Use `--transport` to specify the value. | ||
|
||
### rest_numeric_enums (optional) | ||
One of GAPIC options passed to the generator. The value is either `true` or `false`. | ||
The default value is `true`. | ||
|
||
Use `--rest_numeric_enums` to specify the value. | ||
|
||
### include_samples (optional) | ||
Whether generates code samples. The value is either `true` or `false`. | ||
The default value is `true`. | ||
|
||
Use `--include_samples` to specify the value. | ||
|
||
## An example to generate a client library | ||
``` | ||
library_generation/generate_library.sh \ | ||
-p google/cloud/confidentialcomputing/v1 \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you please add comments or more examples regarding There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I added an example of generating showcase client. |
||
-d google-cloud-confidentialcomputing-v1-java \ | ||
--gapic_generator_version 2.24.0 \ | ||
--protobuf_version 23.2 \ | ||
--grpc_version 1.55.1 \ | ||
--transport grpc+rest \ | ||
--rest_numeric_enums true \ | ||
--include_samples true | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
#!/usr/bin/env bash | ||
|
||
JoeWang1127 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
set -e | ||
|
||
# Wrap gapic-generator-java.jar because protoc requires the plugin to be executable. | ||
working_directory=$(dirname "$(readlink -f "$0")") | ||
exec java -classpath "$working_directory/gapic-generator-java-$gapic_generator_version.jar" com.google.api.generator.Main |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,160 @@ | ||
#!/usr/bin/env bash | ||
|
||
set -eo pipefail | ||
|
||
# parse input parameters | ||
while [[ $# -gt 0 ]] | ||
do | ||
key="$1" | ||
case $key in | ||
-p|--proto_path) | ||
proto_path="$2" | ||
shift | ||
;; | ||
-d|--destination_path) | ||
destination_path="$2" | ||
shift | ||
;; | ||
--gapic_generator_version) | ||
gapic_generator_version="$2" | ||
# export this variable so that it can be used in gapic-generator-java-wrapper.sh | ||
export gapic_generator_version | ||
shift | ||
;; | ||
--protobuf_version) | ||
protobuf_version="$2" | ||
shift | ||
;; | ||
--grpc_version) | ||
grpc_version="$2" | ||
shift | ||
;; | ||
--transport) | ||
transport="$2" | ||
shift | ||
;; | ||
--rest_numeric_enums) | ||
rest_numeric_enums="$2" | ||
shift | ||
;; | ||
--include_samples) | ||
include_samples="$2" | ||
shift | ||
;; | ||
*) | ||
echo "Invalid option: [$1]" | ||
exit 1 | ||
;; | ||
esac | ||
shift # past argument or value | ||
done | ||
|
||
working_directory=$(dirname "$(readlink -f "$0")") | ||
# source utility functions | ||
cd "$working_directory" | ||
source ./utilities.sh | ||
|
||
if [ -z "$protobuf_version" ]; then | ||
protobuf_version=$(get_protobuf_version "$gapic_generator_version") | ||
fi | ||
|
||
if [ -z "$grpc_version" ]; then | ||
grpc_version=$(get_grpc_version "$gapic_generator_version") | ||
fi | ||
|
||
if [ -z "$transport" ]; then | ||
transport="grpc" | ||
fi | ||
|
||
if [ -z "$rest_numeric_enums" ]; then | ||
rest_numeric_enums="true" | ||
fi | ||
|
||
if [ -z "$include_samples" ]; then | ||
include_samples="true" | ||
fi | ||
|
||
cd "$working_directory" | ||
mkdir -p "$destination_path" | ||
destination_path="$working_directory/$destination_path" | ||
##################### Section 0 ##################### | ||
# prepare tooling | ||
##################################################### | ||
cd "$working_directory" | ||
# the order of services entries in gapic_metadata.json is relevant to the | ||
# order of proto file, sort the proto files with respect to their name to | ||
# get a fixed order. | ||
proto_files=$(find "$proto_path" -type f -name "*.proto" | sort) | ||
folder_name=$(extract_folder_name "$destination_path") | ||
# download gapic-generator-java, protobuf and grpc plugin. | ||
download_tools "$gapic_generator_version" "$protobuf_version" "$grpc_version" | ||
##################### Section 1 ##################### | ||
# generate grpc-*/ | ||
##################################################### | ||
cd "$working_directory" | ||
"$protoc_path"/protoc "--plugin=protoc-gen-rpc-plugin=$working_directory/protoc-gen-grpc-java-$grpc_version-linux-x86_64.exe" \ | ||
"--rpc-plugin_out=:$destination_path/java_grpc.jar" \ | ||
$proto_files | ||
# unzip java_grpc.jar to grpc-*/src/main/java | ||
unzip_src_files "grpc" | ||
# remove empty files in grpc-*/src/main/java | ||
remove_empty_files "grpc" | ||
# remove grpc version in *ServiceGrpc.java file so the content is identical with bazel build. | ||
remove_grpc_version | ||
###################### Section 2 ##################### | ||
## generate gapic-*/, part of proto-*/, samples/ | ||
###################################################### | ||
cd "$working_directory" | ||
"$protoc_path"/protoc --experimental_allow_proto3_optional \ | ||
"--plugin=protoc-gen-java_gapic=$working_directory/gapic-generator-java-wrapper" \ | ||
"--java_gapic_out=metadata:$destination_path/java_gapic_srcjar_raw.srcjar.zip" \ | ||
"--java_gapic_opt=$(get_gapic_opts)" \ | ||
${proto_files} $(search_additional_protos) | ||
|
||
unzip -o -q "$destination_path/java_gapic_srcjar_raw.srcjar.zip" -d "$destination_path" | ||
# Sync'\''d to the output file name in Writer.java. | ||
unzip -o -q "$destination_path/temp-codegen.srcjar" -d "$destination_path/java_gapic_srcjar" | ||
# Resource name source files. | ||
proto_dir=$destination_path/java_gapic_srcjar/proto/src/main/java | ||
if [ ! -d "$proto_dir" ]; then | ||
# Some APIs don'\''t have resource name helpers, like BigQuery v2. | ||
# Create an empty file so we can finish building. Gating the resource name rule definition | ||
# on file existences go against Bazel'\''s design patterns, so we'\''ll simply delete all empty | ||
# files during the final packaging process (see java_gapic_pkg.bzl) | ||
mkdir -p "$proto_dir" | ||
touch "$proto_dir"/PlaceholderFile.java | ||
fi | ||
|
||
cd "$working_directory" | ||
# move java_gapic_srcjar/src/main to gapic-*/src. | ||
mv_src_files "gapic" "main" | ||
# remove empty files in gapic-*/src/main/java | ||
remove_empty_files "gapic" | ||
# move java_gapic_srcjar/src/test to gapic-*/src | ||
mv_src_files "gapic" "test" | ||
if [ "$include_samples" == "true" ]; then | ||
# move java_gapic_srcjar/samples/snippets to samples/snippets | ||
mv_src_files "samples" "main" | ||
fi | ||
##################### Section 3 ##################### | ||
# generate proto-*/ | ||
##################################################### | ||
cd "$working_directory" | ||
"$protoc_path"/protoc "--java_out=$destination_path/java_proto.jar" $proto_files | ||
# move java_gapic_srcjar/proto/src/main/java (generated resource name helper class) | ||
# to proto-*/src/main | ||
mv_src_files "proto" "main" | ||
# unzip java_proto.jar to proto-*/src/main/java | ||
unzip_src_files "proto" | ||
# remove empty files in proto-*/src/main/java | ||
remove_empty_files "proto" | ||
# copy proto files to proto-*/src/main/proto | ||
for proto_src in $proto_files; do | ||
mkdir -p "$destination_path/proto-$folder_name/src/main/proto" | ||
cp -f --parents "$proto_src" "$destination_path/proto-$folder_name/src/main/proto" | ||
done | ||
##################### Section 4 ##################### | ||
# rm tar files | ||
##################################################### | ||
cd "$destination_path" | ||
rm -rf java_gapic_srcjar java_gapic_srcjar_raw.srcjar.zip java_grpc.jar java_proto.jar temp-codegen.srcjar |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have to copy the whole
google/
folder? Or maybe only the common protos?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some protos reference
google/api
,google/iam
,google/type
. I'm not sure whether we have a exhausted list of protos such that only those protos can be referenced. So copygoogle
folder is a safe option.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not part of this PR, but I think we should come up with an exhaustive list before we switch to use it in our generation process. As
google/
folder contains protos from all services, which is a lot more than what we need, and it could easily confuse people in the future.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can do that, but do we have a guidance that a proto should not reference a proto outside of a list? If not, the list is likely changing and we are back to copy
google
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Theoretically, the service team can reference protos from anywhere, but cross service proto depedencies are not allowed per https://google.aip.dev/213, so the service protos should only reference common protos from a few selected folders. I think we may have two services that have cross service proto dependencies but all other services should obey it.
Now regarding common protos, we don't have a guidance on where they should be and it's possible that new common protos are added to new locations. But if it happens, I think it would be better to explicitly add it, similar to
explicitly declare a new dependency
vsgetting a new dependency transitively from a existing dependency
in Maven's world.We can discuss more in next project meeting.