Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debian build is broken #8141

Closed
raoulb opened this issue Jan 14, 2021 · 19 comments
Closed

Debian build is broken #8141

raoulb opened this issue Jan 14, 2021 · 19 comments
Assignees
Milestone

Comments

@raoulb
Copy link
Contributor

raoulb commented Jan 14, 2021

What version of Hugo are you using (hugo version)?

$ hugo version
Hugo Static Site Generator v0.80.0/extended linux/amd64 BuildDate: 2021-01-01T02:12:40Z

Does this issue reproduce with the latest release?

Yes.

Issues

Javascript building/bundling fails with sources and dependencies all in assets/js. Here is a minimal example I constructed based on the test project linked in the Hugo documentation.

With Hugo 0.80.0 the build fails:

$ hugo version
Hugo Static Site Generator v0.80.0/extended linux/amd64 BuildDate: 2021-01-01T02:12:40Z
$ hugo
Start building sites … 
Total in 12 ms
Error: Error building site: JSBUILD: failed to transform "js/main.js" (application/javascript): "/home/r/hugoTestProjectJSModImports/assets/js/main.js:1:23": Could not resolve "./lib"

While with Hugo 0.79.1 it works as expected:

$ hugo version
Hugo Static Site Generator v0.79.1/extended linux/amd64 BuildDate: 2020-12-22T10:11:47Z
$ hugo
Start building sites … 

                   | EN  
-------------------+-----
  Pages            |  2  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     |  0  
  Processed images |  0  
  Aliases          |  0  
  Sitemaps         |  1  
  Cleaned          |  0  

Total in 11 ms

If this is not a regression but intended new behaviour, then the documentation might need further clarification.

BTW: Calling esbuild manually works on the same sources:

esbuild --version
0.8.28
esbuild --bundle --minify --outfile=out.js main.js 
cat out.js 
(()=>{function o(){return"Hello from lib in the main project!!"}console.log("Hello from main entry point.");function l(){console.log(o())}window.hello=l;})();
@bep bep self-assigned this Jan 14, 2021
@bep bep added this to the v0.81 milestone Jan 14, 2021
@bep
Copy link
Member

bep commented Jan 14, 2021

I will have a look at it (I esp. appreciate the example repo).

The only related change in 0.80 was a upgrade from Esbuild 0.8.15 to 0.8.17. I do have some (I think) pretty good tests in this area (also a manual repo), so I'm a little surprised. A side not is that I tried to pull in the lates ESBuild (he works fast), and that broke my test repo.

@bep
Copy link
Member

bep commented Jan 14, 2021

I cannot reproduce your issue (on MacOS, but I don't see how that can differ) -- the error you get looks like the error I get when I pull in the latest esbuild.

@moorereason
Copy link
Contributor

@raoulb,
I'm unable to reproduce this on my system:

$ hugo-v0.80.0-extended env
Hugo Static Site Generator v0.80.0-792EF0F4/extended linux/amd64 BuildDate: 2020-12-31T13:46:18Z
GOOS="linux"
GOARCH="amd64"
GOVERSION="go1.15.1"

$ hugo-v0.80.0-extended
Start building sites …

                   | EN
-------------------+-----
  Pages            |  2
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  0
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0

Total in 9 ms

$ hugo-v0.79.1-extended env
Hugo Static Site Generator v0.79.1-EDB9248D/extended linux/amd64 BuildDate: 2020-12-19T15:50:19Z
GOOS="linux"
GOARCH="amd64"
GOVERSION="go1.15.1"

$ hugo-v0.79.1-extended
Start building sites …

                   | EN
-------------------+-----
  Pages            |  2
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  0
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0

Total in 15 ms

It's interesting to me that your version string is different from mine. I downloaded the tar.gz extended releases from GH. Where are you getting your hugo binaries? Can you try the ones in the tar.gz on GH?

@bep bep removed the Bug label Jan 14, 2021
@bep
Copy link
Member

bep commented Jan 14, 2021

OK, then I guess that the version @raoulb is built without the versioning of Go Modules. Which I hope means that he built it himself?

@raoulb
Copy link
Contributor Author

raoulb commented Jan 14, 2021

I use the official packages from Debian sid/unstable from here.

@moorereason
Copy link
Contributor

Cc: @anthonyfok

@raoulb
Copy link
Contributor Author

raoulb commented Jan 14, 2021

@moorereason

I'm unable to reproduce this on my system

Oh, interesting and kind of good news.

Where are you getting your hugo binaries? Can you try the ones in the tar.gz on GH?

Ok. I tried the both versions for the tar.gz files and they work.

$ ~/hugo_0.80.0_Linux-64bit/hugo version
Hugo Static Site Generator v0.80.0-792EF0F4 linux/amd64 BuildDate: 2020-12-31T13:37:58Z
$ ~/hugo_0.80.0_Linux-64bit/hugo 
Start building sites … 

                   | EN  
-------------------+-----
  Pages            |  2  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     |  0  
  Processed images |  0  
  Aliases          |  0  
  Sitemaps         |  1  
  Cleaned          |  0  

Total in 10 ms

and

$ ~/hugo_extended_0.80.0_Linux-64bit/hugo version
Hugo Static Site Generator v0.80.0-792EF0F4/extended linux/amd64 BuildDate: 2020-12-31T13:46:18Z
$ ~/hugo_extended_0.80.0_Linux-64bit/hugo 
Start building sites … 

                   | EN  
-------------------+-----
  Pages            |  2  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     |  0  
  Processed images |  0  
  Aliases          |  0  
  Sitemaps         |  1  
  Cleaned          |  0  

Total in 9 ms

Now I could try the deb packages provided here on GH if this helps.

@bep bep added the Bug label Jan 14, 2021
@raoulb
Copy link
Contributor Author

raoulb commented Jan 14, 2021

I cannot reproduce your issue (on MacOS, but I don't see how that can differ) -- the error you get looks like the error I get when I pull in the latest esbuild.

Oh, good point. Maybe the Debian package builds against an own, 'external' version of ESBuild.

At least this is what I see here with the build time dependency:

adep: golang-github-evanw-esbuild-dev (>= 0.8.17)
    extremely fast JavaScript bundler and minifier (library) 

if I'm reading this correctly.

@bep bep changed the title Possible Regression in ESBuild Version issue with Debian build Jan 14, 2021
@raoulb
Copy link
Contributor Author

raoulb commented Jan 14, 2021

@moorereason @anthonyfok

Now I could try the deb packages provided here on GH if this helps.

Ok, the package hugo_extended_0.80.0_Linux-64bit.deb from GH also works fine.
So this is actually a pure Debian issue it seems. Should I open another issue there?

@moorereason
Copy link
Contributor

We've pinned esbuild at v0.8.17, but the esbuild package from Debian is v0.8.28.

@moorereason
Copy link
Contributor

Tested the latest esbuild and can confirm that it fails to build your test site with the error message in the OP: Could not resolve "./lib".

@bep bep changed the title Version issue with Debian build Debian build is broken Jan 14, 2021
@moorereason
Copy link
Contributor

Last update 😄

The test site in the OP starts failing with esbuild v0.8.21. If I change the import path to ./assets/js/lib, it builds successfully.

@bep
Copy link
Member

bep commented Jan 14, 2021

@moorereason this issue is not about ESBuild (which is a known issue -- we had a buildbot PR that luckily failed in test some weeks ago). This is about the broken Debian build.

@anthonyfok
Copy link
Member

anthonyfok commented Jan 14, 2021

Dear all,

@raoulb, Thank you for reporting the issue, especially timely because Debian 11 (bullseye) will go into freezing soon, and it is good that you let us know of the issue now instead of later.

@bep and @moorereason, thank you both for your excellent investigative work. Indeed, it was my laziness and oversight when updating Debian's copy of esbuild to a too-new version that led to this issue. Now that @moorereason has found out which version of esbuild (v0.8.21) has the behaviour change, I can start looking into a fix for my Debian build later today. Thank you @moorereason for saving me a lot of time.

@bep
Copy link
Member

bep commented Jan 14, 2021

@anthonyfok are you somehow running with different versions of the dependencies in the Debian build (I assume because they're shared by others?)? Do the tests pass?

@anthonyfok
Copy link
Member

anthonyfok commented Jan 16, 2021

Hi @bep, sorry for the late reply.

are you somehow running with different versions of the dependencies in the Debian build

Yes. I had stupidly upgraded golang-github-evanw-esbuild from 0.8.15-1 to 0.8.28-1. Usually, I follow the versions listed in Hugo's go.mod quite strictly, but this last time, I was not able to upgrade to 0.8.17 directly with the usual command because there had been too many newer esbuild releases that 0.8.17 got pushed from https://github.com/evanw/esbuild/releases to the "Next" page, and the pattern matching in debian/watch file did not work. There are ways around that, but I either forgot how or got too lazy, and decided to just go ahead and package 0.8.28, and that's how I got the Debian package into trouble.

(I assume because they're shared by others?)

At the moment, golang-github-evanw-esbuild-dev is not used by other packages. Hugo is currently the only Debian package that depends on it.

Do the tests pass?

Yes, they do, but the Debian build isn't running all the tests, but go test -vet=off -v -p 4 -test.short -timeout 120m

dh_auto_test -- -test.short -timeout 120m
	cd _build && go test -vet=off -v -p 4 -test.short -timeout 120m github.com/gohugoio/hugo github.com/gohugoio/hugo/bufferpool github.com/gohugoio/hugo/cache/filecache github.com/gohugoio/hugo/cache/namedmemcache github.com/gohugoio/hugo/codegen github.com/gohugoio/hugo/commands github.com/gohugoio/hugo/common/collections github.com/gohugoio/hugo/common/constants github.com/gohugoio/hugo/common/herrors github.com/gohugoio/hugo/common/hexec github.com/gohugoio/hugo/common/hreflect github.com/gohugoio/hugo/common/hugio github.com/gohugoio/hugo/common/hugo github.com/gohugoio/hugo/common/loggers github.com/gohugoio/hugo/common/maps github.com/gohugoio/hugo/common/math github.com/gohugoio/hugo/common/para github.com/gohugoio/hugo/common/terminal github.com/gohugoio/hugo/common/text github.com/gohugoio/hugo/common/types github.com/gohugoio/hugo/common/urls github.com/gohugoio/hugo/compare github.com/gohugoio/hugo/config github.com/gohugoio/hugo/config/privacy github.com/gohugoio/hugo/config/services github.com/gohugoio/hugo/create github.com/gohugoio/hugo/deploy github.com/gohugoio/hugo/deps github.com/gohugoio/hugo/docshelper github.com/gohugoio/hugo/helpers github.com/gohugoio/hugo/htesting github.com/gohugoio/hugo/htesting/hqt github.com/gohugoio/hugo/hugofs github.com/gohugoio/hugo/hugofs/files github.com/gohugoio/hugo/hugofs/glob github.com/gohugoio/hugo/hugolib github.com/gohugoio/hugo/hugolib/filesystems github.com/gohugoio/hugo/hugolib/paths github.com/gohugoio/hugo/identity github.com/gohugoio/hugo/langs github.com/gohugoio/hugo/langs/i18n github.com/gohugoio/hugo/lazy github.com/gohugoio/hugo/livereload github.com/gohugoio/hugo/markup github.com/gohugoio/hugo/markup/asciidocext github.com/gohugoio/hugo/markup/asciidocext/asciidocext_config github.com/gohugoio/hugo/markup/blackfriday github.com/gohugoio/hugo/markup/blackfriday/blackfriday_config github.com/gohugoio/hugo/markup/converter github.com/gohugoio/hugo/markup/converter/hooks github.com/gohugoio/hugo/markup/goldmark github.com/gohugoio/hugo/markup/goldmark/goldmark_config github.com/gohugoio/hugo/markup/highlight github.com/gohugoio/hugo/markup/internal github.com/gohugoio/hugo/markup/markup_config github.com/gohugoio/hugo/markup/mmark github.com/gohugoio/hugo/markup/org github.com/gohugoio/hugo/markup/pandoc github.com/gohugoio/hugo/markup/rst github.com/gohugoio/hugo/markup/tableofcontents github.com/gohugoio/hugo/media github.com/gohugoio/hugo/metrics github.com/gohugoio/hugo/minifiers github.com/gohugoio/hugo/modules github.com/gohugoio/hugo/modules/npm github.com/gohugoio/hugo/navigation github.com/gohugoio/hugo/output github.com/gohugoio/hugo/parser github.com/gohugoio/hugo/parser/metadecoders github.com/gohugoio/hugo/parser/pageparser github.com/gohugoio/hugo/publisher github.com/gohugoio/hugo/related github.com/gohugoio/hugo/releaser github.com/gohugoio/hugo/resources github.com/gohugoio/hugo/resources/images github.com/gohugoio/hugo/resources/images/exif github.com/gohugoio/hugo/resources/internal github.com/gohugoio/hugo/resources/jsconfig github.com/gohugoio/hugo/resources/page github.com/gohugoio/hugo/resources/page/page_generate github.com/gohugoio/hugo/resources/page/pagemeta github.com/gohugoio/hugo/resources/postpub github.com/gohugoio/hugo/resources/resource github.com/gohugoio/hugo/resources/resource_factories/bundler github.com/gohugoio/hugo/resources/resource_factories/create github.com/gohugoio/hugo/resources/resource_transformers/babel github.com/gohugoio/hugo/resources/resource_transformers/htesting github.com/gohugoio/hugo/resources/resource_transformers/integrity github.com/gohugoio/hugo/resources/resource_transformers/js github.com/gohugoio/hugo/resources/resource_transformers/minifier github.com/gohugoio/hugo/resources/resource_transformers/postcss github.com/gohugoio/hugo/resources/resource_transformers/templates github.com/gohugoio/hugo/resources/resource_transformers/tocss/dartsass github.com/gohugoio/hugo/resources/resource_transformers/tocss/scss github.com/gohugoio/hugo/source github.com/gohugoio/hugo/tpl github.com/gohugoio/hugo/tpl/cast github.com/gohugoio/hugo/tpl/collections github.com/gohugoio/hugo/tpl/compare github.com/gohugoio/hugo/tpl/crypto github.com/gohugoio/hugo/tpl/data github.com/gohugoio/hugo/tpl/debug github.com/gohugoio/hugo/tpl/encoding github.com/gohugoio/hugo/tpl/fmt github.com/gohugoio/hugo/tpl/hugo github.com/gohugoio/hugo/tpl/images github.com/gohugoio/hugo/tpl/inflect github.com/gohugoio/hugo/tpl/internal github.com/gohugoio/hugo/tpl/internal/go_templates/cfg github.com/gohugoio/hugo/tpl/internal/go_templates/fmtsort github.com/gohugoio/hugo/tpl/internal/go_templates/htmltemplate github.com/gohugoio/hugo/tpl/internal/go_templates/testenv github.com/gohugoio/hugo/tpl/internal/go_templates/texttemplate github.com/gohugoio/hugo/tpl/internal/go_templates/texttemplate/parse github.com/gohugoio/hugo/tpl/internal/resourcehelpers github.com/gohugoio/hugo/tpl/js github.com/gohugoio/hugo/tpl/lang github.com/gohugoio/hugo/tpl/math github.com/gohugoio/hugo/tpl/openapi/openapi3 github.com/gohugoio/hugo/tpl/os github.com/gohugoio/hugo/tpl/partials github.com/gohugoio/hugo/tpl/path github.com/gohugoio/hugo/tpl/reflect github.com/gohugoio/hugo/tpl/resources github.com/gohugoio/hugo/tpl/safe github.com/gohugoio/hugo/tpl/site github.com/gohugoio/hugo/tpl/strings github.com/gohugoio/hugo/tpl/templates github.com/gohugoio/hugo/tpl/time github.com/gohugoio/hugo/tpl/tplimpl github.com/gohugoio/hugo/tpl/tplimpl/embedded github.com/gohugoio/hugo/tpl/transform github.com/gohugoio/hugo/tpl/urls github.com/gohugoio/hugo/transform github.com/gohugoio/hugo/transform/livereloadinject github.com/gohugoio/hugo/transform/metainject github.com/gohugoio/hugo/transform/urlreplacers github.com/gohugoio/hugo/watcher

See https://buildd.debian.org/status/fetch.php?pkg=hugo&arch=amd64&ver=0.80.0-1&stamp=1609753546&raw=0 for the full build (and test) log.

I have decided to fix the bug by including a local copy of esbuild 0.8.17 into the Hugo Debian package for the time being. The fix will appear in hugo_0.80.0-2 soon.

@eclipseo
Copy link
Contributor

Any plans on making it work with the latest ESBuild?

@anthonyfok
Copy link
Member

Hello @raoulb and all:

Here is an update:

  1. The incompatibility wtih ESBuild >= v0.8.21 was thankfully caught by this test (GitHub Actions): https://github.com/gohugoio/hugo/runs/1627365570?check_suite_focus=true (titled "Do not merge: deps: Update to ESBuild v0.8.27"). The tests that failed are TestJSBuildWithNPM and TestJSBuild.

  2. As @moorereason reported, the incompatibility started with ESBuild v0.8.21, most likely due to Plugins don't seem to apply to entrypoints evanw/esbuild#546 "Plugins don't seem to apply to entrypoints" and its fix evanw/esbuild@05eaca4.

  3. As @eclipseo suggested, I agree that making Hugo work with the latest ESBuild is a possibility. Let's open a new issue for that, so I can close this issue regarding version mismatch in Debian packaging.

  4. The error reported in this issue was not detected during Debian build because:

    1. TestJSBuildWithNPM and TestJSBuild are run only when CI=true (or non-empty), but CI was not set in recent Debian packaging.
    2. With CI=true set, TestJSBuildWithNPM and TestJSBuild would still have failed during Debian build because Internet access during build is disallowed.
  5. Fortunately, Debian does run autopkgtest CI where Internet access is allowed, so the latest upload to Debian takes advantage of this to hopefully catch errors like this. Here is the most recent changelog entries:


hugo (0.80.0-3) unstable; urgency=medium

  • Replace local copy of github.com/cli/safeexec
    with dependency on golang-github-cli-safeexec-dev (>= 1.0.0)
  • Replace local copy of github.com/bep/godartsass
    with dependency on golang-github-bep-godartsass-dev (>= 0.11.0)
  • debian/rules: Restore environment variable CI=true for dh_auto_test
    so that tests that would have caught errors such as
    Debian build is broken #8141 would not be missed
  • Remove obsolete 0006-skip-TestHugoModulesTargetInSubFolder.patch
  • Merge patches which skip tests that access the Internet
    into 0005-skip-tests-that-access-the-internet.patch,
    and allow those tests to be run during autopkgtest.
  • Add debian/tests/autopkgtest-pkg-go.conf
    to install git and npm as extra dependencies for autopkgtest
  • debian/rules: Build non-extended version (without LibSass) on riscv64
    as go1.15 does not yet support cgo on riscv64, and extended build FTBFS.
    TODO: Revisit this when go1.16 is released.
    See all: add cgo support to the riscv port golang/go#36641

-- Anthony Fok foka@debian.org Wed, 27 Jan 2021 06:21:10 -0700

hugo (0.80.0-2) unstable; urgency=medium

  • Include a local copy of github.com/evanw/esbuild v0.8.17
    because golang-github-evanw-esbuild 0.8.28-1 which I prematurely
    upgraded to is incompatible with Hugo 0.80.0.
    Fixes Debian build is broken #8141

-- Anthony Fok foka@debian.org Sat, 16 Jan 2021 04:28:36 -0700


@raoulb, thank you again for reporting this issue.

I believe the issue is resolved in the Debian package hugo_0.80.0-2, and made a bit more futureproof in 0.80.0-3, hence I'm closing this issue.

@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 24, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants