My filter adds unwanted classes (bug in Pandoc?)

[pagiraud]

Hi,

Finishing and trying to add an option to a Lua filter, I stumbled upon a problem that I reduced to some sort of MWE. Maybe it’s me doing wrong things (I’m not an expert at all) but this seems quite strange to me.

Consider this markdown file (note the foo class):

---
title: "Untitled"
---

## Some title

![Some caption](teaching.jpg){.foo}

## Without caption

![](teaching.jpg)

One more paragraph.

And then this filter:

local classes = {"aclass", "anotherclass"}
local function bblocks (blocks)
  local newClasses = {}
  local originalClasses = {}
  -- Go from end to start to avoid problems with shifting indices.
  for i = #blocks-1, 1, -1 do
    originalClasses = {}
    newClasses = classes
    if blocks[i].t == "Figure" then
      originalClasses = blocks[i].content[1].content[1].classes
      for k,v in pairs(originalClasses) do table.insert(newClasses, v) end
      blocks[i].content[1].content[1].classes = newClasses
      print("For my Figure:")
      for k,v in pairs(blocks[i].content[1].content[1].classes) do print(v) end
    elseif blocks[i].t == "Para" then
      originalClasses = blocks[i].content[1].classes
      for k,v in pairs(originalClasses) do table.insert(newClasses, v) end
      blocks[i].content[1].classes = newClasses
      print("For my Para:")
      for k,v in pairs(blocks[i].content[1].classes) do print(v) end
    end
  end
  return blocks
end

return {{Blocks = bblocks}}

Then, when I invoke:

pandoc --lua-filter ./testbug.lua -f markdown -t native test.md -o test.json

I get the expected output in the console:

For my Para:
aclass
anotherclass
For my Figure:
aclass
anotherclass
foo

But the generated AST contains also a foo class in the Para block:

[ Header
    2
    ( "some-title" , [] , [] )
    [ Str "Some" , Space , Str "title" ]
, Figure
    ( "" , [] , [] )
    (Caption
       Nothing [ Plain [ Str "Some" , Space , Str "caption" ] ])
    [ Plain
        [ Image
            ( "" , [ "aclass" , "anotherclass" , "foo" ] , [] )
            [ Str "Some" , Space , Str "caption" ]
            ( "teaching.jpg" , "" )
        ]
    ]
, Header
    2
    ( "without-caption" , [] , [] )
    [ Str "Without" , Space , Str "caption" ]
, Para
    [ Image
        ( "" , [ "aclass" , "anotherclass" , "foo" ] , [] )
        []
        ( "teaching.jpg" , "" )
    ]
, Para
    [ Str "One"
    , Space
    , Str "more"
    , Space
    , Str "paragraph."
    ]
]

Since the console output is correct and Blocks are evaluated last to first (so the one with the wanted foo class is the last evaluated), I can’t figure what I’m doing wrong…

[jgm]

Just from a quick glance:

      for k,v in pairs(originalClasses) do table.insert(newClasses, v) end

newClasses scopes over the whole for loop, so you just end up accumulating things, right?

[pagiraud]

Don’t these two lines at the beginning of the loop avoid things to accumulate?

    originalClasses = {}
    newClasses = classes

Initializing originalClasses and newClasses at the beginning of the loop and not of the function doesn’t change anything.

[alerque]

You are not actually initializing newClasses. You're re-assigning it by reference to classes, but when you mutate it after that the original table for classes gets updated.

$ lua
Lua 5.4.7  Copyright (C) 1994-2024 Lua.org, PUC-Rio
> D = require("pl.pretty").dump
> a = { "one" }
> b = a
> b[#b+1] = "two"
> D(a)
{
  "one",
  "two"
}

[pagiraud]

Of course… thank you so much the hint.

So, here is my new (and working) code:

local classes = {"aclass", "anotherclass"}
local function bblocks (blocks)
  --local newClasses = {}
  --local originalClasses = {}
  -- Go from end to start to avoid problems with shifting indices.
  for i = #blocks-1, 1, -1 do
    local originalClasses = {}
    local newClasses = {}
    if blocks[i].t == "Figure" then
      originalClasses = blocks[i].content[1].content[1].classes
      for k,v in pairs(classes) do table.insert(newClasses, v) end
      for k,v in pairs(originalClasses) do table.insert(newClasses, v) end
      blocks[i].content[1].content[1].classes = newClasses
      print("For my Figure:")
      for k,v in pairs(blocks[i].content[1].content[1].classes) do print(v) end
    elseif blocks[i].t == "Para" then
      originalClasses = blocks[i].content[1].classes
      for k,v in pairs(classes) do table.insert(newClasses, v) end
      for k,v in pairs(originalClasses) do table.insert(newClasses, v) end
      blocks[i].content[1].classes = newClasses
      print("For my Para:")
      for k,v in pairs(blocks[i].content[1].classes) do print(v) end
    end
  end
  return blocks
end

Still, I don’t understand why the output in the console and what is actually generated by Pandoc are different (I must have missed something in the way Pandoc works).

[bpj]

How are they different?

If they show the classes in different order it may be because you use pairs rather than ipairs in the loops around the print statements. While ipairs iterates over all numeric indices in order pairs iterates over all indices in "random" order. Always use ipairs when you want to iterate over a table as a list.

[pagiraud]

Well, to sum up, this is the way they are different (in my original code, it’s ok in the new one):

	Console	Generated AST
First block (Figure)	aclass, anotherclass, foo	aclass, anotherclass, foo
Second block (Para)	aclass, anotherclass	aclass, anotherclass, foo

I now understand that in the console, it just worked by chance, because my image without the class foo was the last in my markdown file: since my loop went backwards, it was computed before the image with the foo class. But why doesn’t it work in the same way for the generated AST? I thought my print commands simply showed me what the AST was fed with…