-
Notifications
You must be signed in to change notification settings - Fork 366
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support clients for multiple versions of the same GCP service #10170
Comments
See #10278 (vmwareengine/v1) for another example of the proposed layout (without backwards-compatibility headers). For what it's worth, I used this as a temporary hack for scaffolding generation:
|
This is the script I used to generate #10857 #!/bin/bash
#
# Usage: ./relocate.sh apigateway kms osconfig
#
# If anything goes wrong, it will likely be in the "update builds" step.
#
# Verify the result with:
# ```
# ci/cloudbuild/build.sh -t cmake-install-pr
# bazel build //google/cloud/...
# ```
libraries=("$@")
function edit_config() {
library=$1
service_dirs=("\"\"") # Remember these. They are needed in Step 5.
vals=($(grep "service_proto_path: .*${library}.*v[0-9]" generator/generator_config.textproto -no | sed "s;service_proto_path:.*${library}/;;"))
# Assume that `product_path` immediately follows `service_proto_path`.
line_offset=1
for v in "${vals[@]}"; do
line=${v%:*}
subdir=${v#*:}
service_dirs+=("\"${subdir}/\"")
sed -i "$(($line+$line_offset))s;${library};${library}/${subdir};" generator/generator_config.textproto
# We need to increment the index for each `forwarding_product_path` line we add.
((line_offset+=1))
sed -i "$(($line+$line_offset))i \ \ forwarding_product_path: \"google/cloud/${library}\"" generator/generator_config.textproto
done
# Store in a file because dbolduc does not know how maps work.
printf "%s\n" "${service_dirs[@]}" | sort | uniq > dirs_${library}.tmp
}
function update_bazel() {
mapfile -t service_dirs < dirs_${library}.tmp
{
sed -n '1,18p' google/cloud/${library}/BUILD.bazel
echo "service_dirs = ["
for d in "${service_dirs[@]}"; do
echo " ${d}",
done
sed -n '22,39p' google/cloud/accessapproval/BUILD.bazel
sed -n '39,$p' google/cloud/${library}/BUILD.bazel
sed -n '62,$p' google/cloud/accessapproval/BUILD.bazel | sed "s/accessapproval/${library}/"
} > dern.tmp && mv dern.tmp google/cloud/${library}/BUILD.bazel
}
function update_cmake() {
mapfile -t service_dirs < dirs_${library}.tmp
{
sed '/set(DOXYGEN_PROJECT_NUMBER/q' google/cloud/${library}/CMakeLists.txt
sed -n '21,37p' google/cloud/accessapproval/CMakeLists.txt | sed "s/accessapproval/${library}/" | sed "s;\"\" \"v1/\";${service_dirs[*]};"
sed -n '/include(GoogleCloudCppDoxygen)/,$p' google/cloud/${library}/CMakeLists.txt | sed 's;"\*.h" "\*.cc" "internal/\*.h" "internal/\*.cc";${source_globs};' | sed 's;"mocks/\*.h";${mocks_globs};'
# Use tpu because accessapproval breaks the comment onto two lines.
sed -n '182,$p' google/cloud/tpu/CMakeLists.txt | sed "s;tpu;${library};g"
} > dern.tmp && mv dern.tmp google/cloud/${library}/CMakeLists.txt
}
# Checkout new branch
git checkout -b relocate-libraries
# Edit generator configuration
for library in "${libraries[@]}"; do
edit_config $library
done
git commit -m "refactor: versioned clients" generator/generator_config.textproto
# Regenerate libraries
ci/cloudbuild/build.sh -t generate-libraries-pr
git add google/
ci/cloudbuild/build.sh -t checkers-pr
git commit -m "generated changes" ci/ google/
# Remove stale code
for library in "${libraries[@]}"; do
rm google/cloud/$library/*.cc
rm -rf google/cloud/$library/internal
rm -rf google/cloud/$library/samples
done
git add google/
ci/cloudbuild/build.sh -t checkers-pr
git commit -m "remove stale code; regenerate docs" google/
# Update build files
for library in "${libraries[@]}"; do
update_bazel $library
update_cmake $library
rm dirs_${library}.tmp
done
ci/cloudbuild/build.sh -t checkers-pr
git commit -m "update build files" google/
# Update ABI dumps
ci/cloudbuild/build.sh -t check-api-pr
for library in "${libraries[@]}"; do
git add ci/abi-dumps/google_cloud_cpp_$library.expected.abi.dump.gz
done
git commit -m "update ABI dumps" |
For posterity, here is the final script I used: #!/bin/bash
#
# Usage: ./relocate.sh apigateway kms osconfig
#
# If anything goes wrong, it will likely be in the "update builds" step.
#
# Verify the result with:
# ```
# ci/cloudbuild/build.sh -t cmake-install-pr
# bazel build //google/cloud/...
# ```
set -euo pipefail
libraries=("$@")
function edit_config() {
library=$1
gc_dir=$library
if [[ $gc_dir == "dialogflow_cx" ]]; then gc_dir="dialogflow/cx"; fi
if [[ $gc_dir == "gameservices" ]]; then gc_dir="gaming"; fi
if [[ $gc_dir == "datamigration" ]]; then gc_dir="clouddms"; fi
if [[ $gc_dir == "profiler" ]]; then gc_dir="cloudprofiler"; fi
if [[ $gc_dir == "trace" ]]; then gc_dir="cloudtrace"; fi
service_dirs=("\"\"") # Remember these. They are needed in Step 5.
vals=($(grep "service_proto_path: .*/${gc_dir}/.*v[0-9]" generator/generator_config.textproto -no | sed "s;service_proto_path:.*${gc_dir}/;;"))
inserts=0
for v in "${vals[@]}"; do
og_line=${v%:*}
subdir=${v#*:}
service_dirs+=("\"${subdir}/\"")
# Find the index of the product_path. It is not always on the line after the
# `service_proto_path`, if there are `additional_proto_files` defined.
((line=$og_line+$inserts+1))
while [ $(sed -n "${line}p" generator/generator_config.textproto | grep product_path | wc -w) -eq 0 ]; do
((line+=1))
done
sed -i "${line}s;${library};${library}/${subdir};" generator/generator_config.textproto
# We also need to increment the index for each `forwarding_product_path` line we add.
((inserts+=1))
((line+=1))
sed -i "${line}i \ \ forwarding_product_path: \"google/cloud/${library}\"" generator/generator_config.textproto
done
# Store in a file because dbolduc does not know how maps work.
printf "%s\n" "${service_dirs[@]}" | sort | uniq > dirs_${library}.tmp
}
function check_typos() {
library=$1
if [ $(grep "google/cloud/$library/" .typos.toml | wc -w) -gt 0 ]; then
nvim .typos.toml
git commit -m "manually edit typos" .typos.toml
fi
}
function update_bazel() {
mapfile -t service_dirs < dirs_${library}.tmp
{
sed -n '1,18p' google/cloud/${library}/BUILD.bazel
echo "service_dirs = ["
for d in "${service_dirs[@]}"; do
echo " ${d}",
done
sed -n '22,39p' google/cloud/accessapproval/BUILD.bazel
sed -n '39,$p' google/cloud/${library}/BUILD.bazel
sed -n '62,$p' google/cloud/accessapproval/BUILD.bazel | sed "s/accessapproval/${library}/"
} > dern.tmp && mv dern.tmp google/cloud/${library}/BUILD.bazel
}
function update_cmake() {
mapfile -t service_dirs < dirs_${library}.tmp
{
sed '/set(DOXYGEN_PROJECT_NUMBER/q' google/cloud/${library}/CMakeLists.txt
sed -n '21,37p' google/cloud/accessapproval/CMakeLists.txt | sed "s/accessapproval/${library}/" | sed "s;\"\" \"v1/\";${service_dirs[*]};"
sed -n '/include(GoogleCloudCppDoxygen)/,$p' google/cloud/${library}/CMakeLists.txt | sed 's;"\*.h" "\*.cc" "internal/\*.h" "internal/\*.cc";${source_globs};' | sed 's;"mocks/\*.h";${mocks_globs};'
# Use tpu because accessapproval breaks the comment onto two lines.
sed -n '182,$p' google/cloud/tpu/CMakeLists.txt | sed "s;tpu;${library};g"
} > dern.tmp && mv dern.tmp google/cloud/${library}/CMakeLists.txt
}
# Checkout new branch
git checkout main
git pull --ff-only upstream main
git checkout -b relocate-${libraries[0]}
# Edit generator configuration
for library in "${libraries[@]}"; do
edit_config $library
done
git commit -m "refactor(${libraries[0]}): versioned clients" generator/generator_config.textproto
for library in "${libraries[@]}"; do
check_typos $library
done
# Regenerate libraries
ci/cloudbuild/build.sh -t generate-libraries-pr || true
git add google/
ci/cloudbuild/build.sh -t checkers-pr || true
git commit -m "generated changes" ci/ google/
# Remove stale code
for library in "${libraries[@]}"; do
rm google/cloud/$library/*.cc
rm -rf google/cloud/$library/internal
# Libraries with handwritten samples need finer tuning
rm -rf google/cloud/$library/samples
done
git add google/
ci/cloudbuild/build.sh -t checkers-pr || true
git commit -m "remove stale code; regenerate docs" google/
# Update build files
for library in "${libraries[@]}"; do
update_bazel $library
update_cmake $library
# Cleanup temporary files
rm dirs_${library}.tmp
done
ci/cloudbuild/build.sh -t checkers-pr || true
git commit -m "update build files" google/
# Update ABI dumps
mapfile -t libs < <(for library in ${libraries[@]}; do echo google_cloud_cpp_$library; done)
sed -i "131s/.*/libraries=(${libs[*]})/" ci/cloudbuild/builds/check-api.sh
ci/cloudbuild/build.sh -t check-api-pr || true
git add ci/abi-dumps
git restore ci/cloudbuild/builds/check-api.sh
git commit -m "update ABI dumps"
# Test and move on, lol.
for library in "${libraries[@]}"; do
bazel build //google/cloud/$library/...
done
git push -u origin relocate-${libraries[0]} |
Current StateAll libraries have versioned clients except for the following exceptions. Heavily handwritten:
Sorta versioned already:
@coryan suggests that we might want a single Obnoxious googleapis directory structure
google-cloud-cpp/generator/generator_config.textproto Lines 402 to 404 in a5572b1
Maybe we should just make it |
See https://github.com/googleapis/google-cloud-cpp/blob/main/doc/adr/2022-11-11-multiple-versions-of-GCP-service-in-one-library.md
We need to:
google/cloud/*_options_defaults.cc
#include "google/cloud/foo/*.h"
headersusing ::google::cloud::foo_vN;
directivesproduct_path
, and enable the flag.speech/v2
w/r/tservice_dirs
in bazel and cmakeSee this branch for what some of the changes should look like. (Next to add them to the generator, and break down the work into logical PRs) https://github.com/googleapis/google-cloud-cpp/compare/main...dbolduc:google-cloud-cpp:speech-reorg-dev?expand=1
The text was updated successfully, but these errors were encountered: