Semantic Tokens API #86415
Comments
|
Since microsoft/vscode-languageserver-node#367 proposed the protocol to be server pushing notification to the client, would it be fine for treeDataProvider.onDidChangeTreeDataEvent.fire(item)vs semanticTokenProvider.onDidChangeSemanticTokensEvent.fire(document)In such case, the client is capable of requesting vscode to re-render the semantic highlighting once notification from server is received. |
|
Hi, I've implemented a provider prototype combining java language server and semantic tokens api in Demo: |
|
Am I correct that So, for example, if I "goto definition" to the new file, the editor will first ask to only color the small visible part of the document (for faster response), and then it'll ask the second query to color the whole file (so that scrolling doesn't show boring colorless text)? This is a super-important optimization, 👍 Also I'd like to suggest that maybe, if we have ranges, we don't need
I don't like the presence of
|
|
How do I flag escape characters and/or escaped characters? I do not see it in the current list of token types or token modifiers. vscode/src/vs/platform/theme/common/tokenClassificationRegistry.ts Lines 372 to 408 in be0aca7 |
|
Note: This is 1-2 weeks old. Some issues may no longer be relevant. General feedback: After trying out the little experiment above, here are some issues I ran across:
API thoughts:
|
|
@mattacosta Thank you for your feedback. Ping @dbaeumer for the language server and @aeschli for the token types / styling specific feedback. Regarding the API specific feedback:
Thanks again for the great feedback! |
|
Regardind
Which type did you miss from semantic tokens ? If it is the builder it is exposed via I will export |
|
@dbaeumer Perhaps @mattacosta is referring to this.
I ended up doing this instead to move on. import { SemanticTokens } from 'vscode-languageserver-protocol/lib/protocol.sematicTokens.proposed'; |
|
Is |
|
@dbaeumer Yeah, I saw that. But no, it doesn't.
I mean, based on the syntax highlighting I'm not sure we should be doing I don't really understand TypeScript namespaces and modules well though so... 🤷♂ |
|
@dbaeumer No, not the builder. I needed the interface exposing the handler methods. export interface SemanticTokens {
semanticTokens: {
on(handler: ServerRequestHandler<Proposed.SemanticTokensParams, Proposed.SemanticTokens, Proposed.SemanticTokensPartialResult, void>): void;
onEdits(handler: ServerRequestHandler<Proposed.SemanticTokensEditsParams, Proposed.SemanticTokensEdits | Proposed.SemanticTokens, Proposed.SemanticTokensEditsPartialResult | Proposed.SemanticTokensEditsPartialResult, void>): void;
onRange(handler: ServerRequestHandler<Proposed.SemanticTokensRangeParams, Proposed.SemanticTokens, Proposed.SemanticTokensPartialResult, void>): void;
};
}The export interface SemanticTokens {
resultId?: string;
data: number[];
}The former is required because the connection only has the let connection = createConnection(ProposedFeatures.all, ...);
connection.language.semanticTokens.on(...); // Still inferred as `Language & SemanticTokens`
let provider = new SemanticProvider(connection);
class SemanticProvider {
// TypeScript needs an explicit intersection type for `connection.language` either here in
// the parameter list for the generic version or later in a type cast.
constructor(connection: Connection) {
// connection.language.semanticTokens.on(...);
// ~~~~~~~~~~~~~~ does not exist
(connection.language as Language & SemanticTokens).semanticTokens.on(...);
}
} |
|
@rcjsuen So close! You're supposed to just import import { Proposed } from 'vscode-languageserver';
function handler(...): Proposed.SemanticTokens {
// do stuff
} |
|
@mattacosta I see the problem but I would like to propose a different approach (which I use in these cases). You can always capture an inferred type in TypeScript using the typeof operator. So you can do the following type ConnectionType = typeof connection;
class SemanticProvider {
constructor(connection: ConnectionType) {
}
}or if you don't want to capture it you can inline it as well class SemanticProvider {
constructor(conn: typeof connection) {
}
}would that work for you ? |
|
I don't believe so. My connection isn't accessible like that in reality. It's more like: abstract class Server {
protected readonly connection: typeof ???; // no value to use here
constructor() {
this.connection = this.createConnection();
// setup logging, init handlers, etc
}
protected abstract createConnection(): ???;
// ...other methods that use the connection property...
}
class SemanticServer extends Server {
constructor() {
super();
this.semanticTokenProvider = new SemanticTokenProvider(this.connection);
}
}
class SyntaxServer extends Server {
// took a page from typescript; folding, selectionrange, etc
} |
|
I'm trying to integrate semantic tokens in the Dart extension. Overall it works pretty well, but one thing I noticed is that nested regions don't appear to work. In Dart, this comes up quite often with interpolation in string literals or in documentation comments. Here the entire string literal would be a |
|
@simolus3 we have the same issue in rust-analyzer here: rust-lang/rust-analyzer#3447 |
|
@simolus3 This was alexdima's response to my comment a few weeks ago on this:
Also, for both interpolated strings and documentation comments, those can (and really should) be done on the TM grammar side of things. I see no reason to change the semantic highlighting API for them unless there's something really weird about Dart/Rust. |
alexdima
added
feature-request
|
@alexdima, I see that this issue is in a "March 2020" milestone. Should we take that to mean that it's targeted at being available outside of the proposedAPI system by the end of March? We're very interested in using this feature in our Razor extension, but understand that
|
|
@alexdima, I experimented with adding semantic token support to the C# extension and am very happy with the results. One thing that I would like to bring up is that in order to get the full Visual Studio experience the Dark+ and Light+ themes need some tweaks. I have added the tweaks I've made to my PR - dotnet/vscode-csharp#3667. Would the team be open to incorporating these changes into the the released themes? I also noticed that the keyword token is using the 'keyword.control' scope. Perhaps it would be worth having both in the default tokens since the distinction is such a big part of the VS Code aesthetic. |
|
@aeschli can you comment on #86415 (comment) |
|
@JoeRobich Please file a separate issue with your theme change suggestions so we can go through them together. We are very careful not to make changes to the default themes, but let's discuss this in the new issue. |
|
A heads up if anyone was using /**
* A semantic tokens builder can help with creating a `SemanticTokens` instance
* which contains delta encoded semantic tokens.
*/
export class SemanticTokensBuilder {
constructor(legend?: SemanticTokensLegend);
/**
* Add another token.
*
* @param line The token start line number (absolute value).
* @param char The token start character (absolute value).
* @param length The token length in characters.
* @param tokenType The encoded token type.
* @param tokenModifiers The encoded token modifiers.
*/
push(line: number, char: number, length: number, tokenType: number, tokenModifiers: number): void;
/**
* Add another token. Use only when providing a legend.
*
* @param range The range of the token. Must be single-line.
* @param tokenType The token type.
* @param tokenModifiers The token modifiers.
*/
push(range: Range, tokenType: string, tokenModifiers?: string[]): void;
/**
* Finish and create a `SemanticTokens` instance.
*/
build(resultId?: string): SemanticTokens;
} |
|
https://github.com/microsoft/vscode-languageserver-node/blob/e5d7ad881b1a51b486e3f0e4aa0fbc25dad2be58/protocol/src/protocol.semanticTokens.proposed.ts#L18 < I wonder, doesn't this lack an |
|
@etc0de I think the best is to open a new issue |
ghost
mentioned this issue
|
@alexdima ok I made one here: microsoft/language-server-protocol#968 |
and others
This issue tracks the API proposal for semantic tokens.
Sample: semantic-tokens-sample
API: vscode.proposed.d.ts
Tokens representation
A file can contain many tokens, perhaps even hundreds of thousands of tokens. Therefore, to improve the memory consumption around describing semantic tokens, we have decided to avoid allocating an object for each token and we represent tokens from a file as an array of integers. Furthermore, the position of each token is expressed relative to the token before it because most tokens remain stable relative to each other when edits are made in a file.
In short, each token takes 5 integers to represent, so a specific token
iin the file consists of the following array indices:5*i-deltaLine: token line number, relative to the previous token5*i+1-deltaStart: token start character, relative to the previous token (relative to 0 or the previous token's start if they are on the same line)5*i+2-length: the length of the token. A token cannot be multiline.5*i+3-tokenType: will be looked up inSemanticTokensLegend.tokenTypes. We currently ask thattokenType< 65536.5*i+4-tokenModifiers: each set bit will be looked up inSemanticTokensLegend.tokenModifiersHow to encode tokens
Here is an example for encoding a file with 3 tokens in a uint32 array:
For this example, we will choose the following legend which must be passed in when registering the provider:
tokenTypeandtokenModifiersas integers using the legend. Token types are looked up by index, so atokenTypevalue of1meanstokenTypes[1]. Multiple token modifiers can be set by using bit flags, so atokenModifiervalue of3is first viewed as binary0b00000011, which means[tokenModifiers[0], tokenModifiers[1]]becausebits 0 and 1 are set. Using this legend, the tokens now are:
startCharof the second token is made relative to thestartCharof the first token, so it will be10 - 5. The third token is on a different line than the second token, so thestartCharof the third token will not be altered:Updating tokens
Instead of always returning all the tokens in a file, it is possible for a
DocumentSemanticTokensProviderto implement this method (updateSemanticTokens) and then return incremental updates to the previously provided semantic tokens.How tokens change when the document changes
Let's look at how tokens might change.
Continuing with the above example, suppose a new line was inserted at the top of the file.
That would make all the tokens move down by one line (notice how the line has changed for each one):
The integer encoding of the tokens does not change substantially because of the delta-encoding of positions:
It is possible to express these new tokens in terms of an edit applied to the previous tokens:
Furthermore, let's assume that a new token has appeared on line 4:
The integer encoding of the tokens is:
Again, it is possible to express these new tokens in terms of an edit applied to the previous tokens:
NOTE: When doing edits, it is possible that multiple edits occur until VS Code decides to invoke the semantic tokens provider.
NOTE: If the provider cannot compute
SemanticTokensEdits, it can "give up" and return all the tokens in the document again.NOTE: All edits in
SemanticTokensEditscontain indices in the old integers array, so they all refer to the previous result state.The text was updated successfully, but these errors were encountered: