Check the localizations into the project nightly #16835

DHowett · 2024-03-06T22:12:15Z

Right now, the localization submission pipeline runs every night and
sends our localizable resources over to Touchdown. Later, release builds
pick up the localizations directly from Touchdown, move them into place,
and consume them.

This allowed us to avoid having localized content in the repository, but
it came with too many downsides:

Users could not contribute additional localizations very easily
We use the same release pipeline and Touchdown configuration for every
branch, so strings needed to either slightly match or entirely match
across an entire set of active release branches
Building from day to day can pull in different strings, making the
product not reproduceable
Calling TDBuild during release builds requires network access from the
build machine (so does restoring NuGet packages, but that's neither
here nor there)
Local developers and users could not test out other languages

This pull request moves all localization processing into the nightly
build phase and introduces support for checking loc in and submitting a
pull request. The pull request will not be created anew if one already
exists which has not been merged.

Anything we needed to do on release is now done overnight. This includes
moving loc files into the right places and merging the Cascadia
resources with the Context Menu resources (so that we can work around a
relatively lower amount of translations being chosen for the app versus
the context menu; see #12491 for more info.)

There are some smaller downsides to this approach and its
implementation:

The first commit is going to be huge
Right now, it only manages a single branch and uses a force push; if a
PR is not reviewed timely, it will be force-pushed and you cannot see
the day-to-day changes in the strings. Hopefully there won't be any.

I've taken great care to ensure that repeated runs of this new pipeline
will not result in unnecessary whitespace changes. This required
changing how we merge ContextMenu.resw into CascadiaPackage to always
use the .NET XmlWriter with specific flags.

NOTE that this does not allow users to contribute translation fixes
for the 10 languages which we are importing. We will still need to pull
changes out of those files and submit them as bugs to the localization
team separately, and hope they come back around in another nightly
build. However, there is no reason users cannot contribute
non-Touchdown languages.

…the-loc-build-checked-in-the-loc-files

DHowett · 2024-03-06T22:15:03Z

Open Q: we still have the context menu thing. I used to do it during the nightly, but that means local builds need to know about it.

If I move ContextMenu into the nightly, we can remove that phase from the build completely

DHowett · 2024-03-06T22:16:23Z

I didn't necessarily want to make the context menu strings part of the nightly loc PR, because they are 79 more files that we need to think about

This reverts commit 7f7a73d.

DHowett · 2024-03-07T15:56:34Z

build/pipelines/daily-loc-submission.yml

+    $existingPr = Get-GitHubPullRequest -HeadBranch "${{parameters.targetBranch}}"
+    If ($null -Eq $existingPr) {
+      $Now = Get-Date
+      New-GitHubPullRequest -Head "${{parameters.targetBranch}}" -Title "Localization Updates - $Now"


This cmdlet supports a base branch, but I couldn't get it to work properly in CI. So, I gave up. It always targets main now.

build/pipelines/daily-loc-submission.yml

DHowett · 2024-03-07T18:15:02Z

PR body is ready, and this is ready for review!

DHowett · 2024-03-07T18:15:41Z

src/cascadia/CascadiaPackage/Resources/en-US/Resources.resw

@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8"?>
+<?xml version="1.0" encoding="utf-8"?>


This file changed because we no longer run the script before every build... so it's gotta be finalized.

…y do want it

lhecker · 2024-03-07T21:12:29Z

build/scripts/Copy-ContextMenuResourcesToCascadiaPackage.ps1

+	# Reset paths to be absolute (for .NET)
+	$LanguageDir = (Get-Item $LanguageDir).FullName


Resolve-Path?

That would probably work as well.

lhecker · 2024-03-07T21:19:33Z

build/scripts/Copy-ContextMenuResourcesToCascadiaPackage.ps1

+	$writerSettings = [System.Xml.XmlWriterSettings]::new()
+	$writerSettings.NewLineChars = "`r`n"
+	$writerSettings.Indent = $true
+	$writer = [System.Xml.XmlWriter]::Create($ResPath, $writerSettings)
+	$XmlDocument.Save($writer)
+	$writer.Flush()
+	$writer.Close()


$XmlDocument.Save($ResPath)

If you look on the red side of the diff, I removed exactly that code 😄

I was seeing cases where it saved some lines with \n and some with \r\n. Yes. No, seriously.

Ah, I didn't even notice that.
This may be worth a comment IMO. When I read "consistency" I was wondering "what consistency". 😅

zadjii-msft

yep that's yaml alright

Right now, the localization submission pipeline runs every night and sends our localizable resources over to Touchdown. Later, release builds pick up the localizations directly from Touchdown, move them into place, and consume them. This allowed us to avoid having localized content in the repository, but it came with too many downsides: - Users could not contribute additional localizations very easily - We use the same release pipeline and Touchdown configuration for every branch, so strings needed to either slightly match or _entirely match_ across an entire set of active release branches - Building from day to day can pull in different strings, making the product not reproduceable - Calling TDBuild during release builds requires network access from the build machine (so does restoring NuGet packages, but that's neither here nor there) - Local developers and users could not test out other languages This pull request moves all localization processing into the nightly build phase and introduces support for checking loc in and submitting a pull request. The pull request will not be created anew if one already exists which has not been merged. Anything we needed to do on release is now done overnight. This includes moving loc files into the right places and merging the Cascadia resources with the Context Menu resources (so that we can work around a relatively lower amount of translations being chosen for the app versus the context menu; see #12491 for more info.) There are some smaller downsides to this approach and its implementation: - The first commit is going to be huge - Right now, it only manages a single branch and uses a force push; if a PR is not reviewed timely, it will be force-pushed and you cannot see the day-to-day changes in the strings. Hopefully there won't be any. I've taken great care to ensure that repeated runs of this new pipeline will not result in unnecessary whitespace changes. This required changing how we merge ContextMenu.resw into CascadiaPackage to always use the .NET XmlWriter with specific flags. NOTE that this does not allow users to _contribute_ translation fixes for the 10 languages which we are importing. We will still need to pull changes out of those files and submit them as bugs to the localization team separately, and hope they come back around in another nightly build. However, there is no reason users cannot contribute _non-Touchdown_ languages. (cherry picked from commit ab4b140) Service-Card-Id: 92019663 Service-Version: 1.18

Right now, the localization submission pipeline runs every night and sends our localizable resources over to Touchdown. Later, release builds pick up the localizations directly from Touchdown, move them into place, and consume them. This allowed us to avoid having localized content in the repository, but it came with too many downsides: - Users could not contribute additional localizations very easily - We use the same release pipeline and Touchdown configuration for every branch, so strings needed to either slightly match or _entirely match_ across an entire set of active release branches - Building from day to day can pull in different strings, making the product not reproduceable - Calling TDBuild during release builds requires network access from the build machine (so does restoring NuGet packages, but that's neither here nor there) - Local developers and users could not test out other languages This pull request moves all localization processing into the nightly build phase and introduces support for checking loc in and submitting a pull request. The pull request will not be created anew if one already exists which has not been merged. Anything we needed to do on release is now done overnight. This includes moving loc files into the right places and merging the Cascadia resources with the Context Menu resources (so that we can work around a relatively lower amount of translations being chosen for the app versus the context menu; see #12491 for more info.) There are some smaller downsides to this approach and its implementation: - The first commit is going to be huge - Right now, it only manages a single branch and uses a force push; if a PR is not reviewed timely, it will be force-pushed and you cannot see the day-to-day changes in the strings. Hopefully there won't be any. I've taken great care to ensure that repeated runs of this new pipeline will not result in unnecessary whitespace changes. This required changing how we merge ContextMenu.resw into CascadiaPackage to always use the .NET XmlWriter with specific flags. NOTE that this does not allow users to _contribute_ translation fixes for the 10 languages which we are importing. We will still need to pull changes out of those files and submit them as bugs to the localization team separately, and hope they come back around in another nightly build. However, there is no reason users cannot contribute _non-Touchdown_ languages. (cherry picked from commit ab4b140) Service-Card-Id: 92019665 Service-Version: 1.20

Right now, the localization submission pipeline runs every night and sends our localizable resources over to Touchdown. Later, release builds pick up the localizations directly from Touchdown, move them into place, and consume them. This allowed us to avoid having localized content in the repository, but it came with too many downsides: - Users could not contribute additional localizations very easily - We use the same release pipeline and Touchdown configuration for every branch, so strings needed to either slightly match or _entirely match_ across an entire set of active release branches - Building from day to day can pull in different strings, making the product not reproduceable - Calling TDBuild during release builds requires network access from the build machine (so does restoring NuGet packages, but that's neither here nor there) - Local developers and users could not test out other languages This pull request moves all localization processing into the nightly build phase and introduces support for checking loc in and submitting a pull request. The pull request will not be created anew if one already exists which has not been merged. Anything we needed to do on release is now done overnight. This includes moving loc files into the right places and merging the Cascadia resources with the Context Menu resources (so that we can work around a relatively lower amount of translations being chosen for the app versus the context menu; see #12491 for more info.) There are some smaller downsides to this approach and its implementation: - The first commit is going to be huge - Right now, it only manages a single branch and uses a force push; if a PR is not reviewed timely, it will be force-pushed and you cannot see the day-to-day changes in the strings. Hopefully there won't be any. I've taken great care to ensure that repeated runs of this new pipeline will not result in unnecessary whitespace changes. This required changing how we merge ContextMenu.resw into CascadiaPackage to always use the .NET XmlWriter with specific flags. NOTE that this does not allow users to _contribute_ translation fixes for the 10 languages which we are importing. We will still need to pull changes out of those files and submit them as bugs to the localization team separately, and hope they come back around in another nightly build. However, there is no reason users cannot contribute _non-Touchdown_ languages. (cherry picked from commit ab4b140) Service-Card-Id: 92019664 Service-Version: 1.19

DHowett and others added 18 commits May 17, 2023 15:11

okay just try it

34261dd

okay just try it ii

b723aaf

okay just try it iii

a69f569

foo bar baz

e544692

baz far fozz

f719185

Try to normalize line endings....... ffs

db0fb14

Merge remote-tracking branch 'origin/main' into dev/duhowett/what-if-…

e745011

…the-loc-build-checked-in-the-loc-files

Well let's just try it

a545a2e

I always get something wrong

26a90b1

Yeah i'm using tar for this what of it

852354a

Okay i hate it too

88bddb7

idk i guess no base then?

3ba5838

Foo

77cb4c3

Actually, don't edit the CascadiaPackage resources

7f7a73d

skip spellcheck for all non-en-us resources

1135e66

Set loc target to true

f639909

Spellbot consvc

f0168ad

No longer must we fetch localizations during release builds!

083b052

DHowett added 2 commits March 6, 2024 16:17

Revert "Actually, don't edit the CascadiaPackage resources"

18967d1

This reverts commit 7f7a73d.

On second thought...

c7a2ec1

DHowett commented Mar 7, 2024

View reviewed changes

DHowett added 2 commits March 7, 2024 10:11

Try to fix CRLF I guess

b40f738

ffs try the branch thing

f067b54

github-advanced-security bot found potential problems Mar 7, 2024

View reviewed changes

build/pipelines/daily-loc-submission.yml Fixed Show fixed Hide fixed

This comment has been minimized.

Sign in to view

OH I SEE, THE DOCS JUST LIE

914ac8c

This comment has been minimized.

Sign in to view

Context Menu Merge: fully standardize XML writing

ae8c38f

This comment has been minimized.

Sign in to view

DHowett added 2 commits March 7, 2024 10:59

Propagate the en-us context menu changes

a3b99e3

restore normlacy, speelboot

e00f944

DHowett commented Mar 7, 2024

View reviewed changes

Maybe the only en-us change was ContextMenu; in that case, we actuall…

806c6f9

…y do want it

DHowett changed the title ~~Check the localizations into the project~~ Check the localizations into the project nightly Mar 7, 2024

lhecker approved these changes Mar 7, 2024

View reviewed changes

zadjii-msft approved these changes Mar 8, 2024

View reviewed changes

DHowett merged commit ab4b140 into main Mar 8, 2024
21 checks passed

DHowett deleted the dev/duhowett/what-if-the-loc-build-checked-in-the-loc-files branch March 8, 2024 20:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check the localizations into the project nightly #16835

Check the localizations into the project nightly #16835

DHowett commented Mar 6, 2024 •

edited

Loading

DHowett commented Mar 6, 2024

DHowett commented Mar 6, 2024

DHowett Mar 7, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

DHowett commented Mar 7, 2024

DHowett Mar 7, 2024

lhecker Mar 7, 2024

DHowett Mar 7, 2024

lhecker Mar 7, 2024

DHowett Mar 7, 2024

lhecker Mar 7, 2024

zadjii-msft left a comment

		@@ -1,4 +1,4 @@
		<?xml version="1.0" encoding="utf-8"?>
		<?xml version="1.0" encoding="utf-8"?>

		# Reset paths to be absolute (for .NET)
		$LanguageDir = (Get-Item $LanguageDir).FullName

Check the localizations into the project nightly #16835

Check the localizations into the project nightly #16835

Conversation

DHowett commented Mar 6, 2024 • edited Loading

DHowett commented Mar 6, 2024

DHowett commented Mar 6, 2024

DHowett Mar 7, 2024

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

DHowett commented Mar 7, 2024

DHowett Mar 7, 2024

Choose a reason for hiding this comment

lhecker Mar 7, 2024

Choose a reason for hiding this comment

DHowett Mar 7, 2024

Choose a reason for hiding this comment

lhecker Mar 7, 2024

Choose a reason for hiding this comment

DHowett Mar 7, 2024

Choose a reason for hiding this comment

lhecker Mar 7, 2024

Choose a reason for hiding this comment

zadjii-msft left a comment

Choose a reason for hiding this comment

DHowett commented Mar 6, 2024 •

edited

Loading