New Restrictions on Contig Names not Strict Enough

[jamesemery]

I see that in section 1.4.7 of VCFv4.3, it states that contig names can mostly follow restrictions placed upon them by the SAM spec except for a few explicitly disallowed characters. It would seem that the restricted characters are not enough to preserve parsing of VCF files. Specifically, the SAM format restrictions specify (”[!-)+-<>-][!-]”) which translates into any printable character so long as it isn't in the first place. We place some restrictions on this, namely specifying that the characters ”<>[]:” are additionally non-valid. Since the contig name has to appear in a header line this runs the risk of making it impossible to parse a header string. An example contig header line might be:

##contig=<ID=ctg1,length=81195210,Description="Foo">

Unfortunately both the = and the , characters are legal according to the new restrictions so technically the string "ctg1,length=81195210" COULD be a valid contig name, which means that parsing is ambiguous. Something needs to be done about these two characters at least, the way I see it there are a couple options:

1. Restrict the contig name to disallow = and , would fix this problem at the expense of further restricting contigs allowed by the SAM spec.
2. Escape the ID field in the contig header line somehow, for example with a second set of <> or "" characters making the header line look something like this:
   ##contig=<ID=<ctg1>,length=81195210,Description="Foo"> which would break forward compatibility and add some complexity into parsing.
3. Encode dangerous characters in the header somehow, perhaps by percent encoding of the header contig ID field. The VCFv4.3 spec only seems to explicitly specify percent encoding for INFO and FORMAT field values. Perhaps it could be expanded/clarified to also apply to header ID line values (except for INFO and FORMAT where a regex is already imposed)?

[yfarjoun]

For what it's worth, I support the first option. and I would even strengthen that to support disallowing other "annoying" characters (annoying for command-line parsing purposes) such as:

"`:*()?|\/!&

In fact, I would support disallowing all non-alphanumerics except for a select few:

-_+

I know that this is incompatible with SAM, and we should find a way to deal with that.

[tfenne]

I think that before making anything more restrictive it's worth doing a survey of what's actually being used. E.g. disallowing anything that's in the broadly used hs38DH (including : and *) would cause a lot of people a lot of unnecessary pain.

[yfarjoun]

I am, of-course well aware of the : and * in hs38DH...and would love for there to be a solution that would not cause a lot of pain....I guess I was venting some frustration about that.