forward tokenization failed within roxygen2 examples

[r2evans]

I think this is actually two errors (handling backslashes and handling braces), but I think they are both being caused by a tokenization problem.

Setup:

#' .. content for \description{} (no empty lines) ..
#'
#' .. content for \details{} ..
#' @param x 
#' @return
#' @examples
fun <- function(x) {
}

After @examples, add \dontrun{ and hit return. The #' doesn't auto-fill, and I get the message:

ess-jump-token: Internal error: Forward tokenization failed:
#’ \dontrun{

If I manually enter the closing brace:

#' ...
#' \dontrun{
#' }

and hit enter after \dontrun{, it still gives me the tokenization failure.

If I manually input the mid-brace line, ala

#' ...
#' \dontrun{
#'
#' }

the enter-and-indent is causing the mid-brace roxy-prefixes to indent as well:

#' ...
#' \dontrun{
  #'
  #' 
#' }

Notes:

• occurs with both #' and ##'
• both in and not-in a package sub-directory
• if I type in #' \dontrun (no brace), it fails with the tokenization error
• if I type in #' dontrun{ (no backslash), no error but it indents the next #' line

Versions:

GNU Emacs 25.1.1 (x86_64-w64-mingw32) of 2016-09-17
ess-version: 16.09 [elpa: 20160927.954] (loaded from c:/Users/r2/.emacs.d/elpa/ess-20160927.954/)

[lionel-]

It should be fixed, thanks. Please don't expect indentation to work perfectly in roxygen blocks by the way, that is a difficult problem. Eventually we'll provide an edit-to-popup-buffer command, as in org-mode, to edit the example block.

[r2evans]

For clarity: I "expect" nothing. You guys are maintaining a very non-trivial tool providing fluid access to a backend process, dealing with expressions, steps, and a whole lot of styling, sometimes resulting in conflicts and/or non-deterministic expectations. I'm grateful that it does what it does!

I don't currently rely on indentation within roxy blocks; the issue here was that the roxy tokens were being indented (which is different from code indentation, afaict) and enter-and-indent was failing. After that, any other coolness within roxy editing is gravy.

Thanks!

BTW, speaking of tokenization and such, are you aware that roxygen2 is adding markdown support? References: r-lib/roxygen2#431, r-lib/roxygen2#496, and r-lib/roxygen2#505. I don't know if it'll be a dramatic change or just a nuance to the tokenization. Would it make sense to break roxy blocks into a different mode using polymode-like mechanisms? Just a thought, thanks for ESS.

[lionel-]

are you aware that roxygen2 is adding markdown support?

Yes and I'm absolutely thrilled! No more weird latex :)

Would it make sense to break roxy blocks into a different mode using polymode-like mechanisms?

I think that would be difficult because polymode is designed to work with vertically separated chunks, not horizontally separated (roxy prefix vs code). Best approach is probably popup-to-edit-buffer.

[r2evans]

I'm pretty stoked about @md (and cmark too). Once cmark/pandoc natively support figure/table linking (still WIP, I believe), it'll have everything I "need", for the most part. Getting that notation into roxy would be a natural extension.

Gotcha on the polymode thing; I thought about that, but am not certain how the section-demarcation is happening internally. I know that the triple-tick notation works well (and is vertically-deterministic), I thought perhaps that the polymode determination of block-mode could be done by creating a list of line boundaries, and finding the roxy token is just one way to demarcate another boundary of lines. No worries. Thanks again!