toJSON and toJSONPretty messes up secret from Vault

[nagypeterjob]

Consul Template version

v0.25.1

Configuration

{{` { `}}
   {{ if (eq .Values.vault.secretPath "") }}{{` {{- with secret "secrets/` }}{{ .Values.namespace }}/{{ .Values.serviceName }}{{`" -}}`}}{{ else }}{{` {{- with secret "`}}{{ .Values.vault.secretPath }}{{`" -}}`}}{{ end }}
        {{` "service": {{ .Data.data | toJSONPretty }} `}}
   {{` {{ end }} `}}
   {{` , `}}
    {{` {{- with secret "secrets/`}}{{ .Values.namespace }}{{`/common" }} `}}
        {{` "common": {{ .Data.data | toJSONPretty }} `}}
     {{` {{ end }} `}}
     {{` } `}}
{{- end }}

Sample data

# Include sample data you reference in the template from Consul or Vault here.
{
  "mongo_uri": mongodb://username:password@shard0.maln2.mongodb.net:27017,shard1.maln2.mongodb.net:27017,shard3.maln2.mongodb.net:27017/collection?ssl=true&authSource=user&readPreference=secondaryPreferred&replicaSet=replicaset"
}

Expected behavior

The expected behavior is that the rendered secret (including toJSONPretty call) contains the value for key mongo_url in the exact same format as provided via Vault.

What should have happened?

The mongo connection string should not be HTML escaped.

Actual behavior

Actually the rendered secret messes up the value for mongo_url. Right now, the value for mongo_url key will look like this:

"mongo_uri": mongodb://username:password@shard0.maln2.mongodb.net:27017,shard1.maln2.mongodb.net:27017,shard3.maln2.mongodb.net:27017/collection?ssl=true\u0026authSource=user\u0026readPreference=secondaryPreferred\u0026replicaSet=replicaset"

What actually happened?

toJSON and toJSONPretty generates json by calling the Golang standard library function json.Marshal. It is well known that json.Marshal HTML escapes the produced json string. Though It is still valid json, I wonder if in the context of Vault it is considered to be a "valid" and "desirable" side effect.

As a resolution we could:

1. use a custom json encoder
2. use SetEscapeHTML as in link
3. provide a function parameter through which on can turn on/off HTML escape side effect

Do you think it is a valid problem?

[eikenb]

Thanks for filing the issue @nagypeterjob.

To quickly restart the issue to be sure... the problem is the toJSON (and Pretty) use json.Marshal and it HTML escapes strings which breaks the URL.

I can see this as an issue, but I'm not sure if just changing toJSON and toJSONPretty would work as it seems like this could break things. Maybe an option can be worked in somehow. I'm going to label this an enhancement for now assuming we'll need to add some way to enable this (vs. changing existing behavior). Feel free to chime in if you have a differing opinion or some ideas on the implementation.

Thanks.

[nagypeterjob]

Thanks for acknowledging that this is indeed an issue @eikenb!

I am more than happy to make a PR after we make a decision of the right solution.

1. As you have said, changing existing behaviour feels weird and might break things.
2. I would probably create new functions, something like toUnescapedJSON or toUnsafeJSON. It would look like this:

func toUnescapedJSON(t interface{}) (string, error) {
    buf := &bytes.Buffer{}
    encoder := json.NewEncoder(buf)
    encoder.SetEscapeHTML(false)
    err := encoder.Encode(t) // Encode leaves a \n character at the end of the sequence, we might want to trim the result.
    if err != nil {
       return "", errors.Wrap(err, "toUnescapedJSON")
    }
    return buffer.String(), err
}

We could also create a toUnescapedJSONPretty function by extending the code above with encoder.SetIndent.

encoder.SetIndent("", "    ")

What do you think?

[eikenb]

That sounds reasonable. Please include tests. Thanks.

[nagypeterjob]

I have finished the PR as we agreed to @eikenb. Feel free to take a look at it if you have the time. Thanks in advance 🤞

[eikenb]

Hey @nagypeterjob, I took some time off for the holidays but will take a look at this as soon as I get a chance. Thanks!

[eikenb]

Missed that this didn't get auto-closed with the merge. Closing now.