Truncatewords breaks in markdownify

[SzymonSel]

• {{ Model.Content.Person.About.Markdown | truncate: 400 | markdownify | raw }} work as expedcted
• {{ assigned_author.Content.Person.About.Markdown | truncatewords: 40 | markdownify | raw }} doesn't render markdown
• {{ assigned_author.Content.Person.About.Markdown | split: " " | slice: 0, 40 | join: " " | append: "..." | markdownify | raw }} working workaround

When hacking a workaround, I've got a hunch it's something to do with the "..." ie. When I place the append filter at the end, then the markdown breaks.

Not sure if this is a Fluid or OrchardCore issue.

[MikeAlhayek]

@SzymonSel are you able to provide a report for this? A test case will help identifying the issue.

truncatewords is part of Liquid

[SzymonSel]

Sorry, perhaps I wasn't too clear. What I meant was, that I'm not sure whether it's markdownify or truncateword the culprit.

A test case scenario:

• Create a ContentType with a MarkdownField ie MarkdownTest
• Create a Liquid Template ie. Content__MyTestContentType
• Provide the template code as such:
  • {{ Model.ContentItem.Content.MyTestContentType.MarkdownTest.Markdown | truncate: 400 | markdownify | raw }} work as expedcted
  • {{ Model.ContentItem.Content.MyTestContentType.MarkdownTest.Markdown | truncatewords: 40 | markdownify | raw }} doesn't render markdown
  • {{ Model.ContentItem.Content.MyTestContentType.MarkdownTest.Markdown | split: " " | slice: 0, 40 | join: " " | append: "..." | markdownify | raw }} working workaround

[sebastienros]

Instead, can you just show us what value is provided as the input and output of each filter?

Example: "my friend like _bread_" | truncate: 40 and what you get vs. what you expect, showing what the issue actually is. Same for the filters.

Then we could see what the issue is.

"doesn't render markdown" doesn't help

[github-actions]

We triaged this issue and set the milestone according to the priority we think is appropriate (see the docs on how we triage and prioritize issues).

This indicates when the core team may start working on it. However, if you'd like to contribute, we'd warmly welcome you to do that anytime. See our guide on contributions here.

[SzymonSel]

Ok, ok! geesh!
Example:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, 
>sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 

Output 1, truncate: 100:

Lorem ipsum dolor sit amet, consectetur adipiscing elit,
>sed do eiusmod tempor incididunt ut l...

Output 2, truncatewords: 12:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, >sed do eiusmod tempor...

Output 3, hacked. This is what I would expect trunctatewords + markdownify to do:

Lorem ipsum dolor sit amet, consectetur adipiscing elit,
>sed do eiusmod tempor...

[sebastienros]

So it looks like truncate_words is not doing anything?

Have you tried the actual filter named truncatewords, I don't think truncate_words exists ;)

[sebastienros]

https://shopify.github.io/liquid/filters/truncatewords/

[SzymonSel]

Yes, you are correct :) I've corrected my example - the problem still stands, though slightly different

[sebastienros]

Output1 results look fine
Output2 results look fine

Output 3, hacked. This is what I would expect trunctatewords + markdownify to do:

So in this case it's truncatewords: 12 | markdownify ? And you made up what you expect? What do you actually get?

[SzymonSel]

No. Output 2 doesnt get "interpreted" and just displas the raw, truncated text as if mardownify didn't work.

[sebastienros]

No. Output 2 doesnt get "interpreted" and just displas the raw, truncated text as if mardownify didn't work.

This is not compatible with what you wrote in the issue:

I have still no idea if you are using markdownify in these examples, and if it's the case then what is the output from truncate? These should be two different concerns, and only one should be described as the issue. Your examples suggest they only use truncatewords. Skype me if you want to show what you are doing and getting.

[SzymonSel]

Ok, I'll try again. I was referering to the original examples. From the top:

Example value of Model.ContentItem.Content.MyTestContentType.MarkdownTest.Markdown:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, 
>sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 

1. {{ Model.ContentItem.Content.MyTestContentType.MarkdownTest.Markdown | truncate: 100 | markdownify | raw }}
   Output 1:

   Lorem ipsum dolor sit amet, consectetur adipiscing elit,

   sed do eiusmod tempor incididunt ut l...

2. {{ Model.ContentItem.Content.MyTestContentType.MarkdownTest.Markdown | truncatewords: 12 | markdownify | raw }}
   Output 2:

   Lorem ipsum dolor sit amet, consectetur adipiscing elit, >sed do eiusmod tempor incididunt ut l...
   

3. {{ Model.ContentItem.Content.MyTestContentType.MarkdownTest.Markdown | split: " " | slice: 0, 12 | join: " " | append: "..." | markdownify | raw }}
   Output 3:

   Lorem ipsum dolor sit amet, consectetur adipiscing elit,

   sed do eiusmod tempor...

[sebastienros]

truncatewords removed the line break, making the markdown invalid.

Repro-ing:

{% assign md = "Lorem ipsum dolor sit amet, consectetur adipiscing elit,\n>sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. " %}
{{ md | replace: "\n", "###"}}
---
{{ md | truncatewords: 12  | replace: "\n", "###"}}

Outputs:

Lorem ipsum dolor sit amet, consectetur adipiscing elit,###>sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 
---
Lorem ipsum dolor sit amet, consectetur adipiscing elit, >sed do eiusmod tempor...

Verified on https://shopify.dev/docs/api/liquid by pasting the example above.

truncatewords, like truncate will mess-up the html/markdown. You can't assume it will be valid after that, even if it was not removing libre breaks, it could remove a __ or #, (or break an HTML tag). The solution is to remove any tagging then apply the truncation.

[SzymonSel]

Well, but truncate didn't mess up anything - but I guess it's a matter of finding an example which will break it.

So to conclude: truncate and truncatewords don't mix and match with markdownify unless someone feals lucky ;)

[sebastienros]

Exactly. For HTML that's why there is a filter to remove all the tags. You could contribute the same thing for markdown if you want.

[sebastienros]

check this warning on the Shopify website for truncatewords:

Caution

HTML tags are treated as words, so you should strip any HTML from truncated content. If you don't strip HTML, then closing HTML tags can be removed, which can result in unexpected behavior.

[SzymonSel]

True true. Thanks for your time Seb!
I'll make a note to myself to make the contribution in the future not so far away :)

[sebastienros]

Well, but truncate didn't mess up anything

It did! It removed the line break right before the markdown quote > and thus it's not a valid markdown quote anymore.

[SzymonSel]

"Output 1" looks fine to me, it's "Output 2" that's lost a line break

[sebastienros]

Sorry, didn't realize you mentioned truncate specifically, correct. It seems it doesn't remove line breaks by design, checked on shopify. It will still be an issue with formatted text.