implementing NFT with threading support with emscripten #2

kalwalt · 2019-07-22T09:10:54Z

In this PR i test the NFT with threading support. Basically i modified some functions in ARToolKitJS.cpp with the threading methods in trackingSub.c :

THREAD_HANDLE_T *trackingInitInit( KpmHandle *kpmHandle );
int trackingInitStart( THREAD_HANDLE_T *threadHandle, ARUint8 *imagePtrLuma );
int trackingInitGetResult( THREAD_HANDLE_T *threadHandle, float trans[3][4], int *page );
int trackingInitQuit( THREAD_HANDLE_T **threadHandle_p );

The project is not finished, they are some modifications to do... but even if it is not finished, i got 80-90 fps and 20-25 fps while detecting the NFT marker/image.
Note: not all the examples will works, i'm testing now only the nft_threejs.html example.
I'm developing for now only the minified version, i not added the Pthread support to the debug version due to this issue explained in this emscripten article:

Pthreads + memory growth (ALLOW_MEMORY_GROWTH) is especially tricky, see wasm design issue #1271. This currently causes JS accessing the wasm memory to be slow - but this will likely only be noticeable if the JS does heavy work (wasm runs at full speed, so moving work over can fix this). This also requires that your JS be aware that the HEAP* views may need to be updated - use the GROWABLE_HEAP_* helper functions which automatically handle that for you.

When it will be ready, will be merged in PR #1

to do list:

build the debug lib version with Pthreads support (if possible)
solve the asm.js issue: Invalid asm.js: Invalid member of stdlib
testing all the examples
removing not necessary code ( old code with webWorkers)
adding threading only to NFT
adding filtering to the trans Matrix
restoring the wasm LIb?
solving the issue with Firefox see implementing NFT with threading support with emscripten #2 (comment)

Note:

If someone want to try and test the code you can go to https://kalwalt.github.io/kalwalt-interactivity-AR/

But Please read carefully this before:

You need to enable the SharedArrayBuffer option in your browser. This was disabled due to Spectre issue read this article !!!

Update 16/2/2023

updated the code to Emscripten emsdk 3.1.20,

removed libjpeg stuff/code -> added -s USE_LIBJPEG=1
removed useless transferDataToHeap method
minor changes

Anyway, as said in this comment the code fails to load the camera parameter, and then fails to detect and track the NFT marker.

kalwalt · 2019-07-22T12:23:10Z

With the last commit i get this error:

three.min.js:359 THREE.Matrix3: .getInverse() can't invert matrix, determinant is 0

what does it means? the matrix is not filled in the right way? look:

int trackingInitGetResult( THREAD_HANDLE_T *threadHandle, float trans[3][4], int *page );

the trans is a 3 * 4 as you can see in the code:

jsartoolkit5/emscripten/ARToolKitJS.cpp

Lines 104 to 128 in 41eb40a

    
               float trans[3][4]; 
        
           float trackingTrans[3][4]; 
        
           kpmResultNum = trackingInitGetResult( arc->threadHandle, trackingTrans, &pageNo); 
        
           ARLOGi("kpmResultNum is: %d\n", kpmResultNum); 
        
               for( i = 0; i < kpmResultNum; i++ ) { 
        
                   //if (kpmResult[i].pageNo == markerIndex && kpmResult[i].camPoseF == 0 ) { 
        
           		if (pageNo == markerIndex ) { 
        
                   // if( flag == -1 || err > kpmResult[i].error ) { // Take the first or best result. 
        
           			if( flag == -1 ) { // Take the first or best result. 
        
                        flag = i; 
        
                        err = kpmResult[i].error; 
        
           					ARLOGe("error in the tracking"); 
        
                    } 
        
                } 
        
               } 
        
           flag = kpmResultNum; 
        
           ARLOGi("flag is: %d\n", flag); 
        
               if (flag > -1) { 
        
                   for (j = 0; j < 3; j++) { 
        
                   	for (k = 0; k < 4; k++) { 
        
                   		trans[j][k] = trackingTrans[j][k]; 
        
                   	} 
        
                   }

kalwalt · 2019-07-22T12:59:05Z

from this stackoverflow article mrdoob comment:

Matrix3.getInverse(): can't invert matrix, determinant is 0 usually happens when either the scale.x, scale.y or scale.z are 0. Make sure you're not scaling the object to 0.

kalwalt · 2019-07-22T21:23:06Z

The NFT marker is detected, i see the scan in the console, but the sphere is not displayed. I will see this issue in the next days.
A real improvement in fps, at least on my Desktop machine!

evaristoc · 2019-07-23T15:33:25Z

If it is calculating a determinant, it is looking something like a scaling factor. If I am not wrong should be applicable only to square matrices. Is trackingTrans[3][4] the target matrix for the determinant operation? (Are matrices written as M[m][n] in C++?)

evaristoc · 2019-07-23T15:36:40Z

@kalwalt
This number : kpmResultNum : do you think is a determinant?

evaristoc · 2019-07-23T16:23:25Z

@kalwalt

I will try to check the code these days to see if I spot something. Apparently the trackingTrans matrix is the pose matrix.

I want to get and idea what the target of the threading is. At least one of the targeted functions appears to be trackingInitMain. The target (master?) data seems to be the descriptors (a struc) : trackingHandle.

If I understood correctly, I would say it makes sense to run a threading over that part.

kalwalt · 2019-07-23T17:15:11Z

If it is calculating a determinant, it is looking something like a scaling factor. If I am not wrong should be applicable only to square matrices. Is trackingTrans[3][4] the target matrix for the determinant operation? (Are matrices written as M[m][n] in C++?)

Unfortunately I'm not very skilled in math, biìut looking in the code it should be the pose matrix?

This number : kpmResultNum : do you think is a determinant?

I don't think, it was part of the previous code, i will delete it.

I want to get and idea what the target of the threading is. At least one of the targeted functions appears to be trackingInitMain. The target (master?) data seems to be the descriptors (a struc) : trackingHandle.

i think i have implemented as in mainLoop in the nftSimple example:

https://github.com/artoolkitx/artoolkit5/blob/a8baa4b95be381776af40d49b677bd5757f780c8/examples/nftSimple/nftSimple.c#L560-L598

but the data after this routine pass trough
the markersNFT struct from ARMarkerNFT.c (see in the nftSimple example few lines after L598) the trans matrix data pass trough the arFilterTransMat function; maybe it is a necessary step?

kalwalt · 2019-07-23T19:12:06Z

emscripten/ARToolKitJS.cpp

-        if (flag > -1) {
+
+				//ARLOGi("flag is: %d\n", arc->detectedPage);
+        if (pageNo >= 0 && pageNo == arc->detectedPage) {


the problem is this we do not pass any data to the EM_ASM
if i change the variable in the conditional for example:
if (pageNo >= 0) we get data but with that Threejs error, and the sphere appear and disappears. MAybe we should use another condition or what else?

don't remember why i put this:
if (pageNo >= 0 && pageNo == arc->detectedPage) {

maybe it should be

if (arc->detectedPage >= -1) { //// or better if (arc->detectedPage > -1) {

I would rather ask why appears and disappears. As you put it, it is a hack that might work for some but not all cases. I would prefer to understand why pageNo should be as condition.

However, this might not have sense: pageNo >= 0 && pageNo == arc->detectedPage. I can see some contradictory behaviour there.

If you leave it as arc->detectedPage > -1, does it mean that pageNo >= 0? For what I read from the code, detectedPage was a sort of flag for different states. What meant -1 status for detectedPage?

What about:

if (pageNo >= 0 && arc->detectedPage SOMETHING -1) {

This is the flagging I found for detectedPage:

int detectedPage = -2; // -2 Tracking not inited, -1 tracking inited OK, >= 0 tracking online on page.

evaristoc · 2019-07-23T20:25:22Z

@kalwalt
pose matrix
I guess but not sure the operations with the pose (transformation) matrix are being left at minimal: the actual shape is 4x4 but the last row default to [0,0,0,1]. It is possible that is a common practice to skip that row. Something to confirm? Also the vector that multiply it has 1 as last value ([x,y,z,1]T). Check https://en.wikipedia.org/wiki/Transformation_matrix for more details.

kpmResultNum
I will check this later?

markersNFT struct
Haven't been able to read the whole code yet. Those are apparently the chosen corners (the key points that will be listed for comparison). For what I saw in the code, the operation where markersNTF struc was used suggested a rectification of the position of the markers so they can be compared. I guess this is an homographic correction.

The code apparently never bother about detecting whether there was a variation of the values of the transformation matrix : it apparently just rectify at every frame, no matter the values. So if I understand correctly that part of the script always runs.

It is likely necessary: if I am not wrong the detector algorithm needs to rectify the test image (frame) in order to compare the found points in a similar orientation, position, etc than in the training image before declaring a good matching. Once the points have been detected, the homographic correction might be used to correct the volume projection over the target area in the test image. There is something similar happening with marker-based detection, @kalwalt, but the training marker is deployed differently and the algos might be easier.

However, this is NOT a step that might require threading. A for-loop will do. I think there are other parts that will be more suitable to, to mention:

the method devoted to the search of the corners
the one that create the descriptors
the one that compares (the matching one)
the part that choose good points and reduce the number of outliers (if there is one implemented)

evaristoc · 2019-07-23T20:37:15Z

@kalwalt

However now that I see it you might not have access to those methods though... hmmmm.... And they might be already optimized.

So probably the point comparison might be a good target, as well as the creation of the strucs when a painful for-loop is implemented.

Can you rather work on a simpler approach by for example making some for-loops more threaded instead? See that those for-loops are almost always there, for every frame.

Just a suggestion.

evaristoc · 2019-07-23T20:57:30Z

Contact you these days? Take care!

kalwalt · 2019-07-23T21:22:43Z

@evaristoc thanks for the feedback! I have not time to answer to all the questions/topics but I want to make an observation: the kpmMatching and kpmGetResult functions (inside the threading function too) are responsable for the matching and retrieveing trans data and previously in the getNFTMarkerInfo used this two functions without threading, maybe i overcomplicated the code, But i have no time to test my idea now.... very late for me, i will try tomorrow if i will have time.

kalwalt · 2019-07-27T15:21:00Z

With the latest commits i got always the message Tracking lost. that means the ar2Tracking function receive some empty args, but it is not possible, for this reason i'm very confused....

kalwalt · 2019-07-27T21:02:59Z

@evaristoc @nicolocarpignoli It seems that ar2Tracking was filled with wrong args:
it needs arc->videoFrame instead of arc->videoLuma , I tested with the Chrome Browser and the FPS are not so bad: 35fps while detecting the pinball image and more 50fps without.
Please test it and let me know what do you think!

But there are other refinement in the code to do, and other issues to solve...

kalwalt · 2019-07-27T22:06:18Z

Unfortunately this not works with Firefox:

Current environment does not support SharedArrayBuffer, pthreads are not available! artoolkitNft.min.js:formatted:7495
ExitStatus: Program terminated with exit(-1)

gpt3-bot · 2019-07-27T22:28:00Z

SharedArrayBuffer is supported by Firefox but is off by default for now as a result of Spectre. Should be available again to all main branch here shortly, now that Microsoft and Intel seem to have worked out a fix. For now you can turn it on.

…

On Jul 27, 2019, at 3:06 PM, Walter Perdan ***@***.***> wrote: Unfortunately this not works with Firefox: Current environment does not support SharedArrayBuffer, pthreads are not available! artoolkitNft.min.js:formatted:7495 ExitStatus: Program terminated with exit(-1) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

kalwalt · 2019-09-22T18:47:42Z

@nicolocarpignoli @evaristoc @ThorstenBux @shawmakesmusic now it works!! it was a problem of scaling the model and repositioning it !!
I will do the same for the other example.

kalwalt · 2019-09-22T21:21:38Z

I should also test the binary version of the models, probably more efficient in loading time, and needs to test also with mobile devices...
I would know if there are some good news for the NFT marker creator @Carnaux

nicolocarpignoli · 2019-09-23T07:03:14Z

@kalwalt Great!

Carnaux · 2019-09-23T20:04:25Z

@kalwalt Unfortunately I didn't have time in August and September, but in October I will have a lot more time for projects. The last thing I was doing was trying to read the jpeg image because the current implementation of the marker creator is focused to local, not web, so I can't read some properties of the jpeg file using JavaScript. Maybe @ThorstenBux can help me to understand the data inside the iset and fset files.

kalwalt · 2019-09-23T20:55:36Z

@Carnaux I imagined you were busy, but no problem! Unfortunately i have not so much time but I will try to move forward. regarding the problem for jpeg images I will begin to look at the problem, maybe I will open a specific issue in your repository.

Carnaux · 2019-09-23T21:15:31Z

@kalwalt I will open one and describe what is needed and what I was trying.

kalwalt · 2019-09-23T21:16:23Z

@Carnaux just opened one right now!

nicolocarpignoli · 2019-09-24T21:10:44Z

@kalwalt are you able to host a version of it on Github.io? It would be a great way to test quickly with different mobile phones too. If you don't know how to to that just ask i can help you it's simple

kalwalt · 2019-09-24T21:15:11Z

@nicolocarpignoli I can do it. and this is a good idea. 😄

kalwalt · 2019-09-24T21:22:09Z

@nicolocarpignoli with Github.io you can only host the master or the gh-pages or docs folder, you can not host other branch apart them. I will host on my https://github.com/kalwalt/kalwalt-interactivity-AR. I will do tomorrow as now is too late and i am very tired.

nicolocarpignoli · 2019-09-25T08:20:49Z

yes no worries host it wherever you want :)

kalwalt · 2019-09-25T21:07:06Z

@nicolocarpignoli and everybody here... i hosted one of the examples (CesiumMan) at my page https://kalwalt.github.io/kalwalt-interactivity-AR/nft_threads_cesium.html will host also the other examples when i will have time.

kalwalt · 2019-10-01T20:40:44Z

i added some other example to test to my https://kalwalt.github.io/kalwalt-interactivity-AR/ if someone want to try.

kalwalt · 2019-10-07T21:09:12Z

The asm.js issue: Invalid asm.js: Invalid member of stdlib seems disappeared, don't know why i would like to know what caused maybe the last changes?

Edit: but i tested on my https://kalwalt.github.io/kalwalt-interactivity-AR/
running things locally it not happens but instead i get that error on my webpage.

kalwalt · 2023-02-16T19:44:58Z

I adapted the code to emscripten emsdk 3.1.20, the code can be build but when i run the nft_threejs example the camera parameter can not be corrected loaded. For precision: it load the binary file but unfortunately this is not sufficient. That's very strange, maybe something related to emscripten pthread? One possible issue is the loadCamera function in the --pre-js, if we remove it from artoolkit.api.js and create the necessary code for loading outside maybe it can be solved?
I do not intend to add threading support to jsartoolkit5, but maybe in a future to jsartoolkitNFT.
I will add some useful comment in the code for future reference.

kalwalt · 2023-02-16T20:05:07Z

emscripten/ARToolKitJS.cpp

-	 * Transfer data
-	 */
-
-	void transferDataToHeap(int id, ARUint8  *data) {


I m not sure but it seems that is not correct, i should check and maybe i will restore it.

kalwalt · 2023-02-16T20:06:05Z

tools/makem.js

 //FLAGS += ' -s ERROR_ON_UNDEFINED_SYMBOLS=0';
 //FLAGS += ' -s NO_BROWSER=1 '; // for 20k less
 FLAGS += ' --memory-init-file 0 '; // for memless file
-FLAGS += ' -s BINARYEN_TRAP_MODE=clamp'


with newer emscripten (upstream) version it's not needed anymore.

kalwalt · 2023-02-16T20:07:42Z

tools/makem.js

@@ -117,10 +115,10 @@ var FLAGS = '' + OPTIMIZE_FLAGS;
 FLAGS += ' -Wno-warn-absolute-paths ';
 FLAGS += ' -s TOTAL_MEMORY=' + MEM + ' ';
 FLAGS += ' -s USE_ZLIB=1';
+FLAGS += ' -s USE_LIBJPEG=1 ';


we don't needed anymore to include libjpeg source code since it's embedded inside Emscripten, just adding this flag...

implementing NFT with threading support

162bc15

kalwalt added the enhancement New feature or request label Jul 22, 2019

kalwalt self-assigned this Jul 22, 2019

kalwalt mentioned this pull request Jul 22, 2019

Fixing nft #1

Closed

15 tasks

trackingInitGetResult instead of kpmGetResult

41eb40a

kalwalt added 2 commits July 22, 2019 15:46

adding quit tracking and other stuf for the transpose matrix

ec03ab4

improved getNFTMarkerInfo with threaded tracking

6f46ad1

kalwalt commented Jul 23, 2019

View reviewed changes

kalwalt added 3 commits July 24, 2019 21:04

tested a different conditional

a3e19c3

adding pthreads also in the bytecode libs

f5694a9

simplifying part of the code

218cb8f

kalwalt force-pushed the nft-with-threads branch from 296fc2e to 218cb8f Compare July 26, 2019 22:25

this solved the issue: arc->videoFrame instead of arc->videoLuma !!!

0914d82

kalwalt added 2 commits July 27, 2019 23:38

cleaning the code

359292b

restoring the previous conditional

67b0413

fix for CesiumMan example

e9a88e3

kalwalt mentioned this pull request Sep 23, 2019

About jpeg and future development of NFT marker creator Carnaux/NFT-Marker-Creator#6

Closed

kalwalt mentioned this pull request Oct 8, 2019

Multiple NFT marker kalwalt/kalwalt-interactivity-AR#2

Closed

kalwalt mentioned this pull request Oct 28, 2019

Can't detect nft marker on smartphone camera kalwalt/kalwalt-interactivity-AR#3

Closed

kalwalt added C/C++ coding with C and C++ language code design javascript all about javascript code NFT nft markerless technology labels Dec 7, 2019

kalwalt mentioned this pull request Jan 27, 2020

Wasm compilation with threads artoolkitx/jsartoolkit5#88

Closed

kalwalt mentioned this pull request May 23, 2020

add threejs + wasm worker + gltf model #48

Open

kalwalt added 2 commits February 15, 2023 23:29

fix gitmodules

e50e52b

fix to build with emsdk 3.1.20

759c846

kalwalt commented Feb 16, 2023

View reviewed changes

implementing NFT with threading support with emscripten #2

Are you sure you want to change the base?

implementing NFT with threading support with emscripten #2

Conversation

kalwalt commented Jul 22, 2019 • edited Loading

Note:

Update 16/2/2023

kalwalt commented Jul 22, 2019 • edited Loading

kalwalt commented Jul 22, 2019

kalwalt commented Jul 22, 2019

evaristoc commented Jul 23, 2019 • edited Loading

evaristoc commented Jul 23, 2019

evaristoc commented Jul 23, 2019 • edited Loading

kalwalt commented Jul 23, 2019

kalwalt Jul 23, 2019 • edited Loading

Choose a reason for hiding this comment

kalwalt Jul 23, 2019

Choose a reason for hiding this comment

kalwalt Jul 23, 2019

Choose a reason for hiding this comment

evaristoc Jul 23, 2019 • edited Loading

Choose a reason for hiding this comment

evaristoc Jul 23, 2019

Choose a reason for hiding this comment

evaristoc Jul 23, 2019

Choose a reason for hiding this comment

evaristoc commented Jul 23, 2019 • edited Loading

evaristoc commented Jul 23, 2019 • edited Loading

evaristoc commented Jul 23, 2019

kalwalt commented Jul 23, 2019 • edited Loading

kalwalt commented Jul 27, 2019

kalwalt commented Jul 27, 2019 • edited Loading

kalwalt commented Jul 27, 2019

gpt3-bot commented Jul 27, 2019 via email

kalwalt commented Sep 22, 2019

kalwalt commented Sep 22, 2019

nicolocarpignoli commented Sep 23, 2019

Carnaux commented Sep 23, 2019 • edited Loading

kalwalt commented Sep 23, 2019

Carnaux commented Sep 23, 2019

kalwalt commented Sep 23, 2019

nicolocarpignoli commented Sep 24, 2019 • edited Loading

kalwalt commented Sep 24, 2019

kalwalt commented Sep 24, 2019 • edited Loading

nicolocarpignoli commented Sep 25, 2019

kalwalt commented Sep 25, 2019

kalwalt commented Oct 1, 2019 • edited Loading

kalwalt commented Oct 7, 2019 • edited Loading

kalwalt commented Feb 16, 2023

kalwalt Feb 16, 2023

Choose a reason for hiding this comment

kalwalt Feb 16, 2023

Choose a reason for hiding this comment

kalwalt Feb 16, 2023

Choose a reason for hiding this comment

kalwalt commented Jul 22, 2019 •

edited

Loading

kalwalt commented Jul 22, 2019 •

edited

Loading

evaristoc commented Jul 23, 2019 •

edited

Loading

evaristoc commented Jul 23, 2019 •

edited

Loading

kalwalt Jul 23, 2019 •

edited

Loading

evaristoc Jul 23, 2019 •

edited

Loading

evaristoc commented Jul 23, 2019 •

edited

Loading

evaristoc commented Jul 23, 2019 •

edited

Loading

kalwalt commented Jul 23, 2019 •

edited

Loading

kalwalt commented Jul 27, 2019 •

edited

Loading

Carnaux commented Sep 23, 2019 •

edited

Loading

nicolocarpignoli commented Sep 24, 2019 •

edited

Loading

kalwalt commented Sep 24, 2019 •

edited

Loading

kalwalt commented Oct 1, 2019 •

edited

Loading

kalwalt commented Oct 7, 2019 •

edited

Loading