Merge pull request #55 from CloudCannon/feat/translated-urls

Ability to translate page URLs

CHANGELOG.md

@@ -9,6 +9,10 @@
 
 ## Unreleased
 
+* Adds the ability to translate page URLs
+  * See [Docs > Translating URLs](https://rosey.app/docs/urls/)
+* Fixes an issue where some internal links with query strings or hash fragments would not be rewritten
+
 ## v2.0.5 (March 19, 2024)
 
 * Fixes an issue where `<template>` elements were being ignored by generate and build. 

docs/content/docs/_index.md

@@ -93,6 +93,7 @@ _site/
 index.liquid
 +rosey/
 +>> base.json
++>> base.urls.json
 {{< /tree >}}
 
 This `base.json` file contains all text that needs to be translated. For the layout we tagged above, this will look like the following:

docs/content/docs/checks.md

@@ -2,7 +2,7 @@
 title: "Rosey Checks File"
 nav_title: "Rosey Checks File"
 nav_section: Workflow
-weight: 6
+weight: 7
 ---
 
 The Rosey checks file highlights any discrepancies between your base translation file and your translated locale files, such as out of date or missing translations.

docs/content/docs/generate.md

@@ -26,6 +26,14 @@ The path to generate the Rosey base locale file to. Defaults to `rosey/base.json
 |-----------------|--------------|------------|
 | `--base <FILE>` | `ROSEY_BASE` | `base`     |
 
+### Base URLs
+
+The path to generate the Rosey base urls file to. Defaults to `rosey/base.urls.json`
+
+| CLI Flag             | ENV Variable      | Config Key  |
+|----------------------|-------------------|-------------|
+| `--base-urls <FILE>` | `ROSEY_BASE_URLS` | `base_urls` |
+
 ### Separator
 
 The separator to use between Rosey namespaces when generating keys. Defaults to `:`

docs/content/docs/urls.md

@@ -0,0 +1,58 @@
+---
+title: "Translated URLs"
+nav_title: "Translated URLs"
+nav_section: Workflow
+weight: 6
+---
+
+Rosey URL locale files can contain translated URLs for your website in a given language.
+
+## Creating translated URL locale files
+
+Creating URL locale files is not a step performed by Rosey. This part of the translation workflow is left open ended, usually integrating into an existing translation workflow for a company, or being programmatically created by transforming the input URLs.
+
+Rosey will look for URL locale files alongside the standard locale files. For a file at `rosey/locales/es.json`, Rosey will look for URLs at `rosey/locales/es.urls.json`.
+
+Locale files should be created based on the base URL file output from the [Rosey generate command](/docs/generate/). For the example base URL file:
+
+```json
+{
+    "version": 2,
+    "keys": {
+        "index.html": {
+            "original": "index.html"
+        },
+        "home/index.html": {
+            "original": "home/index.html"
+        },
+        "posts/hello-world.html": {
+            "original": "posts/hello-world.html"
+        }
+    }
+}
+```
+
+The `rosey/locales/ja-jp.urls.json` locale file should match the structure:
+
+```json
+{
+    "index.html": {
+        "original": "index.html",
+        "value": "index.html"
+    },
+    "home/index.html": {
+        "original": "home/index.html",
+        "value": "家/index.html"
+    },
+    "posts/hello-world.html": {
+        "original": "posts/hello-world.html",
+        "value": "投稿/こんにちは世界.html"
+    }
+}
+```
+
+Each of these keys is an object with `original` and `value` strings. The `value` string should contain the translated destination file, and will be used by Rosey when building your final multilingual site. 
+
+The output should always include the `.html` extension. Rosey will remove any trailing `index.html` filename where able.
+
+All internal links to these files will be updated within the target locale.

rosey/Cargo.toml

@@ -24,3 +24,5 @@ actix-files = "0.6"
 tokio = { version = "1.36", features = ["macros", "rt-multi-thread", "time"] }
 notify = "5.0.0"
 charabia = "0.6.0"
+path-slash = "0.2"
+url = "2"

rosey/features/build/rosey-build-links.feature

@@ -15,6 +15,8 @@ Feature: Rosey Links
       <h4><a href="/posts/hello-world.png">Hello World Asset</a></h4>
       <h5><a href="posts/hello-world/">Hello World Relative</a></h5>
       <h6><a href="/posts/hello-world.html">Hello World Extension</a></h6>
+      <h7><a href="/posts/hello-world/#title">Hello World Anchor</a></h7>
+      <h8><a href="/posts/hello-world.html?q=a">Hello World Query</a></h8>
       </body>
       </html>
       """
@@ -42,6 +44,12 @@ Feature: Rosey Links
     Then I should see a selector 'h6>a' in "dist/translated_site/blank/index.html" with the attributes:
       | href      | /blank/posts/hello-world.html |
       | innerText | Hello World Extension         |
+    Then I should see a selector 'h7>a' in "dist/translated_site/blank/index.html" with the attributes:
+      | href      | /blank/posts/hello-world/#title |
+      | innerText | Hello World Anchor              |
+    Then I should see a selector 'h8>a' in "dist/translated_site/blank/index.html" with the attributes:
+      | href      | /blank/posts/hello-world.html?q=a |
+      | innerText | Hello World Query                 |
 
 
   Scenario: Rosey doesn't update links already pointing to a locale

rosey/features/build/rosey-build-translated-links.feature

@@ -0,0 +1,119 @@
+Feature: Rosey Translated Links
+  Background:
+    Given I have the environment variables:
+      | ROSEY_SOURCE | dist/site            |
+      | ROSEY_DEST   | dist/translated_site |
+    Given I have a "dist/site/index.html" file with the content:
+      """
+      <html>
+      <body>
+      <h1><a href="/posts/hello-world">Hello World</a></h1>
+      <h2><a href="/posts/hello-world/">Hello World Trailing Slash</a></h2>
+      <h3><a href="/posts/hello-world/index.html">Hello World Direct Index Link</a></h3>
+      <h4><a href="posts/hello-world/">Hello World Relative</a></h4>
+      <h5><a href="/posts/foobar">Foobar</a></h5>
+      <h6><a href="/posts/foobar/">Foobar Trailing Slash</a></h6>
+      <h7><a href="/posts/foobar/index.html">Foobar Direct Index Link</a></h7>
+      </body>
+      </html>
+      """
+    And I have a "dist/site/posts/hello-world/index.html" file with the content:
+      """
+      <html>
+      <body>
+      <h1>Hello</h1>
+      <h2><a href="/posts/foobar/">Foobar Trailing Slash</a></h2>
+      </body>
+      </html>
+      """
+    And I have a "dist/site/posts/foobar/index.html" file with the content:
+      """
+      <html>
+      <body>
+      <h1>Hello</h1>
+      </body>
+      </html>
+      """
+    And I have a "rosey/locales/cool.json" file with the content:
+      """
+      {}
+      """
+    And I have a "rosey/locales/cool.urls.json" file with the content:
+      """
+      {
+        "index.html": {
+          "original": "index.html",
+          "value": "index.html"
+        },
+        "posts/hello-world/index.html": {
+          "original": "posts/hello-world/index.html",
+          "value": "radical-articles/sup-world/index.html"
+        },
+        "posts/foobar/index.html": {
+          "original": "posts/foobar/index.html",
+          "value": "radical-articles/foobar/foobar.html"
+        }
+      }
+      """
+
+  Scenario: Rosey updates internal links using translation keys
+    When I run my program with the flags:
+      | build |
+    Then I should see a selector 'h1>a' in "dist/translated_site/cool/index.html" with the attributes:
+      | href      | /cool/radical-articles/sup-world/ |
+      | innerText | Hello World                       |
+    Then I should see a selector 'h2>a' in "dist/translated_site/cool/index.html" with the attributes:
+      | href      | /cool/radical-articles/sup-world/ |
+      | innerText | Hello World Trailing Slash        |
+    Then I should see a selector 'h3>a' in "dist/translated_site/cool/index.html" with the attributes:
+      | href      | /cool/radical-articles/sup-world/ |
+      | innerText | Hello World Direct Index Link     |
+    Then I should see a selector 'h4>a' in "dist/translated_site/cool/index.html" with the attributes:
+      | href      | posts/hello-world/   |
+      | innerText | Hello World Relative |
+    Then I should see a selector 'h5>a' in "dist/translated_site/cool/index.html" with the attributes:
+      | href      | /cool/radical-articles/foobar/foobar.html |
+      | innerText | Foobar                                    |
+    Then I should see a selector 'h6>a' in "dist/translated_site/cool/index.html" with the attributes:
+      | href      | /cool/radical-articles/foobar/foobar.html |
+      | innerText | Foobar Trailing Slash                     |
+    Then I should see a selector 'h7>a' in "dist/translated_site/cool/index.html" with the attributes:
+      | href      | /cool/radical-articles/foobar/foobar.html |
+      | innerText | Foobar Direct Index Link                  |
+    Then I should see a selector 'h2>a' in "dist/translated_site/cool/radical-articles/sup-world/index.html" with the attributes:
+      | href      | /cool/radical-articles/foobar/foobar.html |
+      | innerText | Foobar Trailing Slash                     |
+
+  Scenario: Rosey updates meta urls using translation keys
+    When I run my program with the flags:
+      | build |
+    Then I should see a selector 'link' in "dist/translated_site/posts/hello-world/index.html" with the attributes:
+      | rel      | alternate                         |
+      | href     | /cool/radical-articles/sup-world/ |
+      | hreflang | cool                              |
+
+    Then I should see a selector 'link' in "dist/translated_site/en/posts/hello-world/index.html" with the attributes:
+      | rel      | alternate                         |
+      | href     | /cool/radical-articles/sup-world/ |
+      | hreflang | cool                              |
+    Then I should see a selector 'link' in "dist/translated_site/cool/radical-articles/sup-world/index.html" with the attributes:
+      | rel      | alternate              |
+      | href     | /en/posts/hello-world/ |
+      | hreflang | en                     |
+
+  Scenario: Rosey generates redirects with translation keys
+    When I run my program with the flags:
+      | build                     |
+      | --default-language "cool" |
+    Then I should see a selector 'a' in "dist/translated_site/posts/hello-world/index.html" with the attributes:
+      | href      | /cool/radical-articles/sup-world/     |
+      | innerText | Click here if you are not redirected. |
+
+  Scenario: Rosey generates canonicals with translation keys
+    When I run my program with the flags:
+      | build                     |
+      | --default-language "cool" |
+    Then I should see a selector 'link' in "dist/translated_site/posts/hello-world/index.html" with the attributes:
+      | rel      | canonical                         |
+      | href     | /cool/radical-articles/sup-world/ |
+      | hreflang | cool                              |

rosey/features/generate/rosey-generate-urls.feature

@@ -0,0 +1,31 @@
+Feature: Rosey Generate URLs
+  Background:
+    Given I have the environment variables:
+      | ROSEY_SOURCE | dist/site            |
+      | ROSEY_DEST   | dist/translated_site |
+
+  Scenario: Rosey generates a URLs file
+    Given I have a "dist/site/index.html" file with the content:
+      """
+      <html>
+      <body>
+      <p data-rosey="seal">Kiss From A Rose</p>
+      <p data-rosey="sting">Desert Rose</p>
+      </body>
+      </html>
+      """
+    And I have a "dist/site/about/index.html" file with the content:
+      """
+      <html>
+      <body>
+      <p data-rosey="seal">Kiss From A Rose</p>
+      <p data-rosey="seal">Kiss From A Rose</p>
+      </body>
+      </html>
+      """
+    When I run my program with the flags:
+      | generate |
+    Then I should see "rosey/base.urls.json" containing the values:
+      | version                         | int:2            |
+      | keys.index\.html.original       | index.html       |
+      | keys.about/index\.html.original | about/index.html |

rosey/src/lib.rs

@@ -107,6 +107,7 @@ impl RoseyOptions {
                 separator: matches.get("separator", base.separator),
                 locales: working_dir.join(matches.get("locales", base.locales)),
                 base: working_dir.join(matches.get("base", base.base)),
+                base_urls: working_dir.join(matches.get("base-urls", base.base_urls)),
                 default_language: matches.get("default-language", base.default_language),
                 redirect_page: matches
                     .get_opt("redirect-page", base.redirect_page)
@@ -200,6 +201,17 @@ impl RoseyTranslation {
         }
     }
 
+    pub fn insert_uncounted(&mut self, key: String, value: String) {
+        match self {
+            RoseyTranslation::V1(keys) => {
+                keys.insert(key, value);
+            }
+            RoseyTranslation::V2(keys) => {
+                keys.insert(key, RoseyTranslationEntry::new(value));
+            }
+        }
+    }
+
     pub fn get(&self, key: &str) -> Option<&String> {
         match self {
             RoseyTranslation::V1(keys) => keys.get(key),
@@ -302,6 +314,10 @@ impl RoseyLocale {
         self.keys.insert(key, value, page);
     }
 
+    pub fn insert_uncounted(&mut self, key: String, value: String) {
+        self.keys.insert_uncounted(key, value);
+    }
+
     pub fn output(&mut self, version: u8) -> String {
         match version {
             2 => serde_json::to_string_pretty(self).unwrap(),

rosey/src/main.rs

@@ -58,6 +58,15 @@ async fn main() {
                             example_defaults.base.display()
                         )),
                 )
+                .arg(
+                    Arg::with_name("base-urls")
+                        .long("base-urls")
+                        .value_name("PATH")
+                        .help(&format!(
+                            "The file to generate the Rosey base URLs file to. \n ─ Defaults to '{}'",
+                            example_defaults.base_urls.display()
+                        )),
+                )
                 .arg(
                     Arg::with_name("separator")
                         .long("separator")

rosey/src/options.rs

@@ -15,6 +15,7 @@ pub struct RoseyPublicConfig {
     pub tag: String,
     pub separator: String,
     pub base: PathBuf,
+    pub base_urls: PathBuf,
     pub locales: PathBuf,
     pub languages: Option<Vec<String>>,
     pub exclusions: String,
@@ -35,6 +36,7 @@ impl Default for RoseyPublicConfig {
             tag: "data-rosey".into(),
             separator: ":".into(),
             base: "rosey/base.json".into(),
+            base_urls: "rosey/base.urls.json".into(),
             locales: "rosey/locales".into(),
             languages: None,
             exclusions: r#"\.(html?|json)$"#.into(),
@@ -69,6 +71,7 @@ impl Display for RoseyPublicConfig {
         writeln!(f, "   - Source:              {}", self.source.display())?;
         writeln!(f, "   - Destination:         {}", self.dest.display())?;
         writeln!(f, "   - Base locale file:    {}", self.base.display())?;
+        writeln!(f, "   - Base urls file:      {}", self.base_urls.display())?;
         writeln!(f, "   - Locales directory:   {}", self.locales.display())?;
         match &self.images_source {
             Some(s) => writeln!(f, "   - Images source:       {}", s.display())?,