-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bundle or improve the nimlangserver installation #9
Comments
My impression is that most extensions for the mainstream languages use the package manager of the particular language to install the language server binary. I suspect this is done because managing binaries in the extension is difficult in practice. This technique doesn't prevent version pinning and forced upgrades if the package manager is capable enough (which Nimble should be). |
Counterpoint: It's possible to bundle platform-specific binaries by providing platform-specific VSIX files: |
Even though we need to make a decision, there is no need to block the first release because of this. |
it is also possible to bundle all binaries in a single vsix and choose a binary at runtime, which might be a more simple process that said, many extensions download their deps instead of bundling (ie not using the package manager) - here's a simple example: https://github.com/haskell/vscode-haskell/blob/b14b3735d4582de485342cc8f5c51b2892caea5b/src/hlsBinaries.ts#L145-L162 |
bundling a binary for the current platform only: bundling binaries for all platforms: downloading a binary: nimble install: Overall, I think "bundling a binary for the current platform only" provides the best user experience, so I vote for that. |
i'm all for @nickysn's suggestion "bundling a binary for the in the meantime it would be great if ready compiled suggestion posted : nim-lang/langserver#92 |
On how to bundle maximally compatible binaries that don't depend on distro specifics: #46 (comment) |
That would probably work on most distros, however it will still fail on some of the more weird ones, such as NixOS (https://nixos.wiki/wiki/Packaging/Binaries), so it still doesn't cover all distros.
No, it's not difficult to maintain, it's easy to maintain, because the distro packagers do the work for you. And they update your libraries, when they have security vulnerabilities. Surely, there's a downside - it's slow. That's why language package managers exist, because developers want to use the latest, greatest bleeding edge libraries, and when they find a bug in a library, they don't want to wait 3 years, until the fixed version of the library gets into Debian stable. Developers also prefer more fine grained control over their libraries, because they use fast evolving code, with an unstable API, that breaks their code when you change versions of libraries. So, there's really two different worlds - there's the world of the language package managers, where there's the convenience and speed of having a huge selection of libraries, that are evolving at a very rapid rate. And there's the world of the distro package managers, where you have the stability and security of a curated, maintained set of packages. So, different worlds, different goals, different advantages and disadvantages. So, where's this heading? Eventually, nimlangserver is going to become stable and it'll be shipped by most distributions. In fact, it's starting to happen right now with the 1.2.0 release. Why the whole rant? You're thinking fast and bleeding edge. And that's where we're at right now. But eventually, we're heading towards slow and stable. In fact, there's not much going on in the language server - the LSP protocol is stable, and most of the real work is done in nimsuggest, so it's only a thin layer, that will become stable soon - sooner than expected. On the other hand, Linux ABI breakages will probably continue, so I'm thinking long term (years from now) to rely on the distro package managers to provide a more stable version, rather than shipping a binary. On Windows and macOS it's quite the opposite - I think there the best option is to bundle a binary with the extension and use it by default. On Linux - also ship a binary, but maybe use whatever comes first in the path? But what if there's an outdated version installed via nimble in ~/.nimble/bin? I guess the real question is, which version to use, if there are several versions installed on the computer. |
Hey, it's Sunday, we're allowed to rant ;)
Slow is not where the inefficiency comes from: the increased complexity lies in having to maintain compatibility across version boundaries (aka "having options") - when you know that extension and langserver are the same version, you have a fundamentally more simple problem to manage because one determines the other and there's no friction and version compatibility issues between them - you're able to, like in the case we're discussing, remove the complexity of (mostly meaningless) options and choices that otherwise cause unnecessary maintenance. This is why bundling a binary is easier to deal with: fewer moving parts. The distro argument is exactly the same: either you have to deal with vscode version, extension version, nimlangserver version and distro/libpcre version - or you can remove the last two and have an objectively more simple product - both as a user and as a developer having to support it. The complexity of managing a distro and its versioning mismatches when managing a "product" or application come at a cost - it is this cost that in a very darwinian way mostly has obsoleted distros with users having migrated to docker and similar "containerised" or isolated solutions for applications, leaving distros with the responsibility to ship base libraries to bootstrap the system but not much else. Some hold on to the distro, because it does indeed have some advantages but it turns out that the "upgrade library for security reasons" happens far more rarely than "upgrade library breaks the application" or "library is missing, my binary is not working" and the outcome is that ecosystems like vscode, firefox, go, rust, javascript etc have moved away from that model. Even if there's an argument to be had that security fixes are important, the ongoing cost of having to deal with mundane failures is still higher - at the end of the day. When there's a security fix, the application will get updated as well because that's where the majority of users are. That Coming back to this issue, we're targeting vscode and the absolutely most simple thing for the extension is to support one version, and one version only and the (interim) solution (while the binary distribution is being worked on) is to make a PR that makes sure that the "local nimble install" option is used for it (or that a binary is downloaded without involving nimble). nimlangserver will keep evolving because lsp as a whole is evolving together with vscode, as is nim - indeed, it is because the evolution is mostly driven by vscode (which drives lsp development) that tying nimlangserver to the extension makes sense. Every such time it evolves, you can either relive the distro version mismatch, with its maintainer having gone on vacation or moved on to a different problem, or you can remove the distro from the equation entirely.
This is very simple: use the binary built by the extension which lives in its own local folder using the options in nimble that already exist for this purpose. If someone is special and wants to use a different binary, they can copy the binary there. |
Long discussion. Surely, @arnetheduck loves to debate. Let's skip the offtopic and focus on solutions. Do we all agree to ship a platform-specific binary with the extension, instead of using nimble (even if it's a nimble "local" install)? If yes, let's start from there. |
Im not sure about it but dont have strong opinions on it neither. Another pro that it has besides the one you mentioned:
Also, using nimble it's a good exercise of eating our own dog food |
How important is that, compared to the other things?
Yes, but:
|
It just adds up, like the other points.
It is not like it were mutually exclusive (i.e. using Nimble vs setup for new users). IMO it's a matter of providing better ux/feedback. |
And how would that better ux/feedback look like? A console window showing nimble downloading and compiling stuff? |
I mean, is there any part of nimble that was designed to allow GUI integration? |
We could start with just providing feedback of what's doing and show the actual error if something goes wrong.
I dont know if there is work in this direction done but providing info of in what stage Nimble is or what's left sounds like a great addition to Nimble |
Do you want (and do you have time) to work on these things? If I were to work on this issue, I would choose to embed the binary. I think it offers a better user experience - the extension works instantly, without any downloads, compilation time (the nimble build is quite slow - it also downloads and compiles the Nim compiler from source). |
fwiw, it does this only if the installed nim version does not match the expected version - also, the intent is that nimble fetches an already-compiled nim in the case but that hasn't made it to nimble yet. |
I can do it but I have to work on some nimble issues first. I would start with changing the installation to be local to the extension and providing better feedback as it's a smaller change than to deal with the binaries and automatise releases. |
The nimble.lock file specifies this version:
This makes the user's installed nim version almost never match the expected version. And the intent to add the ability for nimble to download an already-compiled nim is nice in principle, but:
So, in practice, the user experience will continue to suck for a long time, if we choose to continue down this road. |
ok, that's a bit weird - for an app like nls, I'd expect it to lock 2.0.2 or any other released version since it shouldn't need any |
This is something to change as well and we'll be moving away from it in general - ie when nimble "drives" the nim installation, it shouldn't be packaged with Nim - it's also a problem of the build, ie when nimble builds nim, for some reason it also builds nimble which then replaces whatever nimble is installed which is backwards. Finally, there's no real reason to tie the nimble release schedule to that of nim (unless nim starts releasing much more frequently).
indeed - the current nimble has a lot of aspirational / broken features that lack polish or outright don't wor - these need to be addressed holistically before the extension can provide the excellent experience we want (or the extension needs to do a lot of workarounds that eventually should not be needed). At some point, the extension will likely have to require a minimum nimble version for example to provide some of its functionality - strictly though, it is |
this is simply a matter of priorities and addressing the issue at the core before introducing workarounds in "leaf" projects like vscode-nim (which also cost development effort) - it's good to have the end-game in mind when deciding what to do with this issue today - for example, the fact that it takes an extra 3 minutes to set up nimlangserver because the build builds nim isn't really a problem that needs addressing in vscode-nim - eventually, as nimble improves, this will solve itself in vscode-nim as well. |
Who is going to provide it, then? If we're going to bundle it with the extension, we might as well bundle nimlangserver instead and skip the entire compilation step. If it won't be bundled with the extension, then who is going to provide it? |
But having the nimlangserver binary bundled with the extension is also a possibly desirable "end-game". It will always be the fastest and the least trouble for the majority of the users. Just because nimble will eventually suck less in 3 years time (in addition to the development efforts, you need to include the time needed for users to update their nimble version as well), it will still take more time to compile nimlangserver from source, compared to not compiling it. |
Yeah, well, generally I think bundling or downloading binaries provides the better UX for vscode-nim simply because reduces the number of moving parts - this path requires some work on shipping binaries however (such as getting rid of pcre and building langserver statically via musl as is customary for shipping binaries). That said, if the build via nimble is kept in any shape or form (for example as a fallback in case there is no binary available for the platform or where there are "problems" for fringe distros like nix that have their own standards), the general comments above still apply, and there exist low-hanging fruit ways of making it more pleasurable (that also are broadly beneficial to the ecosystem outside of whatever vscode-nim does). Ie the "output window" is a general feature that can be used to log the download of the binary or the build, and as vscode becomes smarter, it will also make sense it it to understand more about That said, there exists a fundamental difference between langserver and nimble that is important to the bundling-or-not decision: nimlangserver is an implementation detail of vscode-nim where there is no expectation that the user will interact with it directly "outside" of vscode (or equivalent editor - sharing langserver between editors is quite the fringe use case that I believe doesn't merit consideration / isn't worth the complexity) whereas |
Getting rid of pcre is entirely independent from whether we should ship a binary or build from source via nimble, because nimble does jack shit about the libpcre.so dependency, since libpcre.so isn't a nimble package. So, if you don't have libpcre.so installed on your system, you're getting an error anyway, whether it's a bundled binary, or compiled with nimble. The only way this dependency problem can be resolved automatically via a package manager is if we used the distro's package manager, but that's distro-specific and lot of extra work, and I think that should be done by distro packagers, not by us. Doesn't mean we shouldn't get rid of libpcre.so in the long run. I'm just saying we don't need to wait for this to be done, before we can ship a binary. Regards musl - I have a bad opinion about this C library, since I've looked at its sources, at least in the form, used in the WASI SDK, when I was working on Free Pascal for the WebAssembly/WASI platform. For the record, Free Pascal doesn't use a libc, it has its own independent RTL, but I wanted to test compatibility when linking with C code, and the official SDK for WASI used musl, so that's why I had to look at it. Maybe it's just the WASM fork, maybe for Linux it's better quality, I don't know, but the code that I saw sucked. :( So, I would use musl only as a last resort, in case linking with glibc really does create problems between distros. I'm planning to test our nimlangserver binaries on several popular distros in a VM and see if it works.
Yeah, a dual option - bundled binary or if that fails, nimble install is also possible end game. It's just more options to support, but it's doable.
That's why I don't think relying on nimble to install nimlangserver is a good idea. Nimble is a development tool, designed for building from source. Most of the time, we install binaries, we don't compile everything from source. There are Linux distros that build everything from source, and they have dedicated followers, however the majority of users simply don't use them, because they don't have the time to waste. |
the simple rule when using glibc is to compile on an old distro (like ubuntu 18 LTS), usually via a docker image (instead of whatever github actions provides) - this tends to work for the 95% use case because it covers any distro newer than that - musl is a popular way for the last 5% and given that Nim doesn't rely on any advanced glibc functionality, I think it's a safe option to consider. |
Now errors and outputs is logged in the Nim output windows as they happen. This PR also prepares the local installation environment. Unfortunately it errors, so Nimble needs to be improved. cc nim-lang#9
Now errors and outputs is logged in the Nim output windows as they happen. This PR also prepares the local installation environment. Unfortunately it errors, so Nimble needs to be improved. cc #9
In the PR above (needs testing on other platforms before merging) The installation now happens locally and the output is transferred to the "Nim" output windows. So if something goes wrong the user knows. |
* Installs the langserver locally #9 * Changes the order so it prioritises the local install over the global Maybe we can use an option for those users that prefer to use the global installation * fixes win support
Currently, the VS Code extension searches for nimlangserver in the PATH and if it doesn't find it, it attempts to install it via
nimble install nimlangserver --accept
. There are two potential problems with this:An alternative approach that we could try is to bundle a nimlangserver binary with the VS Code extension itself. This way, when a new version of nimlangserver is released, we can issue an updated VS Code extension in the marketplace. The downside to this approach is that we need to bundle a binary for each supported platform (i.e. at least Linux, Windows, macOS for x86_64/aarch64, maybe also i386 and arm?) This will make it difficult to build the VSIX installer package on a single platform, because it would require setting up a crosscompiler environment, or manual work (e.g. building the binaries on different computers/operating systems and collecting the binaries).
The text was updated successfully, but these errors were encountered: