murdock and tests using `utf-8` characters #11691

cladmi · 2019-06-13T11:46:37Z

Description

Tests using utf-8 characters fails in murdock native but work my developer machine.

tests/lua_loader: run the test in CI tests/lua_loader: run the test in CI #11686
tests/pkg_u8g2: run the test in CI tests/pkg_u8g2: run the test in CI #11687
tests/pthread_rwlock: run the test in CI tests/pthread_rwlock: run the test in CI #11688

For the boards tests, it looks like they work correctly on the nodes but the report is not described as an utf-8 document so show invalid output in the reports by default.

Related to #9095 (comment)

Steps to reproduce the issue

Enable the given tests in CI and check the output.

Expected results

Tests work in native correctly.
Tests report show the utf-8 characters by default on boards.

Actual results

Tests using utf-8 fail with UnicodeEncodeError: 'ascii' codec can't encode characters in position for native
Encoding for file tests results is utf-8.

Versions

This is murdock specific so not RIOT.

The text was updated successfully, but these errors were encountered:

kaspar030 · 2019-06-13T12:21:45Z

Tests using utf-8 characters fails in murdock native but work my developer machine.

Can you test on your developer machine if they work with LANG_* set to C?

kaspar030 · 2019-06-13T12:26:56Z

And maybe test if adding "export LANG=C.UTF-8" to the top of ./.murdock fixes the problem on CI?

kaspar030 · 2019-06-13T12:28:13Z

For the boards tests, it looks like they work correctly on the nodes but the report is not described as an utf-8 document so show invalid output in the reports by default.

Let's keep this issue seperate. It is a webserver configuration issue and not related to utf-8, but to non-utf-8 being present in the board's output.

cladmi · 2019-06-13T12:36:59Z

LC_ALL=C make -C tests/pthread_rwlock/ flash test fails with the same error.
LC_ALL=C.UTF-8 make -C tests/pthread_rwlock/ flash test works.

I think LANG does not configure this, both C and C.UTF-8 work without issues.

And maybe test if adding "export LANG=C.UTF-8" to the top of ./.murdock fixes the problem on CI?

Will try in tests/pkg_u8g2: run the test in CI #11687.

A good question being where should it be set by default ?
In the script, in the build system ?

Let's keep this issue seperate. It is a webserver configuration issue and not related to utf-8, but to non-utf-8 being present in the board's output.

Agreed it is webserver configuration. I put it here if the locale configuration should have been something configured server side.

The issue is also present if only utf-8 characters are present.

https://ci.riot-os.org/RIOT-OS/RIOT/11688/4879a310043103bcc23bb0077c0341f14d4fb10f/output/run_test/tests/pthread_rwlock/samr21-xpro:llvm.txt

cladmi · 2019-06-13T12:41:51Z

Locally I successfully tested

LC_ALL=C  make -C tests/pthread_rwlock/ flash test

with

diff --git a/Makefile.include b/Makefile.include
index f9ed21b6a..149ac6d78 100644
--- a/Makefile.include
+++ b/Makefile.include
@@ -582,6 +582,7 @@ reset:
 .PHONY: test test/available
 TESTS ?= $(foreach file,$(wildcard $(APPDIR)/tests/*[^~]),\
                         $(shell test -f $(file) -a -x $(file) && echo $(file)))
+test: export LC_ALL=C.UTF-8
 test: $(TEST_DEPS)
        $(Q) for t in $(TESTS); do \
                $$t || exit 1; \

However on a board it fails to run term and I need to also add the variable for term.

cladmi · 2019-06-13T12:49:19Z

And maybe test if adding "export LANG=C.UTF-8" to the top of ./.murdock fixes the problem on CI?

Will try in tests/pkg_u8g2: run the test in CI #11687.

I should also still try to make the test fail to see how the error is handled.

kaspar030 · 2019-06-13T13:04:13Z

A good question being where should it be set by default ?
In the script, in the build system ?

I don't think we should override the user's locale system. Only obscure distributions do not configure utf-8 locales for users... Thus the build system is IMO not the right place.

Murdock is a different story. I think the build workers actually do configure utf-8, but on the RPi worker node, this doesn't happen. We could either set that up in the RPi systemd unit file, or just in .murdock.
I think I'd prefer the latter, as I prefer the murdock environment as uniform as possible.
(I remember having this in there somewhere, in some context, but I don't know where...)

cladmi · 2019-06-13T14:18:22Z

I don't think we should override the user's locale system. Only obscure distributions do not configure utf-8 locales for users... Thus the build system is IMO not the right place.

I wanted to overwrite it only for test and term.

If not overwritten, then we should add as requirement that the system running the tests is configured to use UTF-8, otherwise tests can fail.
Which would mean that the CI workers must be updated and not the .murdock script.

cladmi · 2019-07-11T12:45:46Z

From local testing, it looks like it should be configured in the riotdocker image.

Currently the locale in the container is "POSIX" whatever that means

locale                                                                                                                                                                                                                                                                                    
LANG=                                                                                                                                                                                                                                                                                     
LANGUAGE=                                                                                                                                                                                                                                                                                 
LC_CTYPE="POSIX"                                                                                                                                                                                                                                                                          
LC_NUMERIC="POSIX"                                                                                                                                                                                                                                                                        
LC_TIME="POSIX"                                                                                                                                                                                                                                                                           
LC_COLLATE="POSIX"                                                                                                                                                                                                                                                                        
LC_MONETARY="POSIX"                                                                                                                                                                                                                                                                       
LC_MESSAGES="POSIX"                                                                                                                                                                                                                                                                       
LC_PAPER="POSIX"                                                                                                                                                                                                                                                                          
LC_NAME="POSIX"                                                                                                                                                                                                                                                                           
LC_ADDRESS="POSIX"                                                                                                                                                                                                                                                                        
LC_TELEPHONE="POSIX"                                                                                                                                                                                                                                                                      
LC_MEASUREMENT="POSIX"                                                                                                                                                                                                                                                                    
LC_IDENTIFICATION="POSIX"                                                                                                                                                                                                                                                                 
LC_ALL=

By doing a local test inspired from #11220 it fails in the current docker image with a samr21-xpro.
I could not run native though due to a native_cpu_init: getcontext: Operation not permitted issue so ignored it.

BOARD=samr21-xpro make -C tests/lua_loader/ flash
DOCKER_FLAGS='--rm --privileged -i --group-add plugdev --group-add dialout --device=$(PORT)' BUILD_IN_DOCKER=1 DOCKER="sudo docker" BOARD=samr21-xpro make -C tests/lua_loader/ flash

The test fails due to the encoding errors.

print((require"m2").a)
2019-07-11 14:40:12,635 - INFO # chega de repente
print((require"c1").X)
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.6/logging/__init__.py", line 996, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 48: ordinal not in range(128)

When adding -e LC_ALL=C.UTF-8 it works:

DOCKER_FLAGS='--rm --privileged -i --group-add plugdev --group-add dialout --device=$(PORT) -e LC_ALL=C.UTF-8' BUILD_IN_DOCKER=1 DOCKER="sudo docker" BOARD=samr21-xpro make -C tests/lua_loader/ test

DOCKER_FLAGS='--rm --privileged -i --group-add plugdev --group-add dialout --device=$(PORT) -e LC_ALL=C.UTF-8' BUILD_IN_DOCKER=1 DOCKER="sudo docker" BOARD=samr21-xpro make -C tests/lua_loader/ test
make: Entering directory '/home/harter/work/git/RIOT/tests/lua_loader'
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm --privileged -i --group-add plugdev --group-add dialout --device=/dev/ttyACM0 -e LC_ALL=C.UTF-8 -t -u "$(id -u)" \
    -v '/usr/share/zoneinfo/Europe/Berlin:/etc/localtime:ro' -v '/home/harter/work/git/RIOT:/data/riotbuild/riotbase' -e 'RIOTBASE=/data/riotbuild/riotbase' -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' -e 'BUILD_DIR=/data/riotbuild/riotbase/build' -e 'RIOTPROJECT=/data/riotbuild/riotbase' -e 'RIOTCPU=/data/riotbuild/riotbase/cpu' -e 'RIOTBOARD=/data/riotbuild/riotbase/boards' -e 'RIOTMAKE=/data/riotbuild/riotbase/makefiles' -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache   \
    -e 'BOARD=samr21-xpro' \
    -w '/data/riotbuild/riotbase/tests/lua_loader/' \
    'riot/riotbuild:latest' make test
python3 -m easy_install --user pyserial
Searching for pyserial
Reading https://pypi.python.org/simple/pyserial/
Downloading https://files.pythonhosted.org/packages/0d/e4/2a744dd9e3be04a0c0907414e2a01a7c88bb3915cbe3c8cc06e209f59c30/pyserial-3.4-py2.py3-none-any.whl#sha256=e0770fadba80c31013896c7e6ef703f72e7834965954a78e71a3049488d4d7d8
Best match: pyserial 3.4
Processing pyserial-3.4-py2.py3-none-any.whl
Installing pyserial-3.4-py2.py3-none-any.whl to /data/riotbuild/.local/lib/python3.6/site-packages
Adding pyserial 3.4 to easy-install.pth file
Installing miniterm.py script to /data/riotbuild/.local/bin

Installed /data/riotbuild/.local/lib/python3.6/site-packages/pyserial-3.4-py3.6.egg
Processing dependencies for pyserial
Finished processing dependencies for pyserial
/data/riotbuild/riotbase/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200"
Twisted not available, please install it if you want to use pyterm's JSON capabilities
2019-07-11 14:40:57,643 - INFO # Connect to serial port /dev/ttyACM0
Welcome to pyterm!
Type '/exit' to exit.
2019-07-11 14:41:00,825 - INFO # main(): This is RIOT! (Version: 2019.10-devel-25-g6b094)
2019-07-11 14:41:00,832 - INFO # I am a module, hi!
print((require"m1").a)
2019-07-11 14:41:00,895 - INFO # Quando uma lua
print((require"m2").a)
2019-07-11 14:41:00,957 - INFO # chega de repente
print((require"c1").X)
2019-07-11 14:41:01,019 - INFO # E se deixa no céu,
print((require"c2").X)
2019-07-11 14:41:01,081 - INFO # como esquecida

make: Leaving directory '/home/harter/work/git/RIOT/tests/lua_loader'

I used this minimal diff to put test in docker.

git diff
diff --git a/Makefile.include b/Makefile.include
index 37509aacb..6339bba60 100644
--- a/Makefile.include
+++ b/Makefile.include
@@ -600,10 +600,15 @@ TESTS ?= $(foreach file,$(wildcard $(APPDIR)/tests/*[^~]),\
 # See #11762.
 TEST_DEPS += $(TERMDEPS)
 
+ifeq ($(BUILD_IN_DOCKER),1)
+test: ..in-docker-container
+else
 test: $(TEST_DEPS)
+       python3 -m easy_install --user pyserial
        $(Q) for t in $(TESTS); do \
                $$t || exit 1; \
        done
+endif
 
 test/available:
        $(Q)test -n "$(strip $(TESTS))"
diff --git a/makefiles/docker.inc.mk b/makefiles/docker.inc.mk
index d7249b5f5..ffb05efd1 100644
--- a/makefiles/docker.inc.mk
+++ b/makefiles/docker.inc.mk
@@ -5,6 +5,7 @@ export DOCKER_FLAGS ?= --rm
 # List of Docker-enabled make goals
 export DOCKER_MAKECMDGOALS_POSSIBLE = \
   all \
+  test \
   buildtest \
   scan-build \
   scan-build-analyze \

kaspar030 · 2019-07-11T13:18:31Z

From local testing, it looks like it should be configured in the riotdocker image.

+1, will PR a fix.

cladmi · 2019-09-16T10:40:34Z

I added a PR to try re-enabling these tests #12246

cladmi added Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) Area: tests Area: tests and testing framework Area: CI Area: Continuous Integration of RIOT components labels Jun 13, 2019

cladmi assigned kaspar030 and smlng Jun 13, 2019

kaspar030 mentioned this issue Jul 11, 2019

add ENV LC_ALL|LANG=C.UTF-8 RIOT-OS/riotdocker#75

Merged

This was referenced Jul 12, 2019

tests/lua_loader: run the test in CI #11686

Merged

tests/pkg_u8g2: run the test in CI #11687

Merged

tests/pthread_rwlock: run the test in CI #11688

Merged

This was referenced Sep 16, 2019

tools/docheck: fix for non-GNU OSes #12240

Closed

tests: re-enable utf-8 tests on native #12246

Merged

miri64 closed this as completed in #12246 Sep 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

murdock and tests using `utf-8` characters #11691

murdock and tests using `utf-8` characters #11691

cladmi commented Jun 13, 2019

kaspar030 commented Jun 13, 2019

kaspar030 commented Jun 13, 2019

kaspar030 commented Jun 13, 2019

cladmi commented Jun 13, 2019

cladmi commented Jun 13, 2019

cladmi commented Jun 13, 2019

kaspar030 commented Jun 13, 2019

cladmi commented Jun 13, 2019

cladmi commented Jul 11, 2019

kaspar030 commented Jul 11, 2019

cladmi commented Sep 16, 2019

murdock and tests using utf-8 characters #11691

murdock and tests using utf-8 characters #11691

Comments

cladmi commented Jun 13, 2019

Description

Steps to reproduce the issue

Expected results

Actual results

Versions

kaspar030 commented Jun 13, 2019

kaspar030 commented Jun 13, 2019

kaspar030 commented Jun 13, 2019

cladmi commented Jun 13, 2019

cladmi commented Jun 13, 2019

cladmi commented Jun 13, 2019

kaspar030 commented Jun 13, 2019

cladmi commented Jun 13, 2019

cladmi commented Jul 11, 2019

kaspar030 commented Jul 11, 2019

cladmi commented Sep 16, 2019

murdock and tests using `utf-8` characters #11691

murdock and tests using `utf-8` characters #11691