Temporarily pausing execution of a lua_State with other lua threads running on its stack. lua_break()?

[HaroldCindy]

Similar to #208 / #239.

The lua_break() construct is nice because it seems like it would allow longjmp()-style yields without requiring participation from intermediary coroutine consumers to yield "all the way up" to the original lua_resume(). i.e., there's no chance that it would be picked up by intermediary callers of coroutine.resume() between the lua_break() call and the topmost lua_resume() call.

I use this mechanism to trigger yields in a VM_INTERRUPT() handler so I can run threads on the global stack for a given time slice before pre-empting. This works fine when those threads don't have nested uses of coroutine.resume() and friends, but breaks down otherwise.

Given something like

ret = "FAIL"
local NESTING = 3


local function nested_coro(i)
    if i >= NESTING then
        print("before_breaker")
        -- this is just a C function with `return lua_break(L);`
        breaker()
        print("after_breaker")
        ret = "OK"
    else
        print("entering")
        coroutine.wrap(function()
            nested_coro(i + 1)
        end)()
        print("exiting")
    end
end

print("start_nested")
nested_coro(0)
print("end_nested")
print(ret)

and an execution loop like

int main()
{
    // ...
    int status = LUA_OK;
    do
    {
        status = lua_resume(L, nullptr, 0);

        if (status == LUA_BREAK)
        {
            fprintf(stderr, "main status: %d\n", lua_status(L));
            fprintf(stderr, "okay, gonna go again\n");
            sleep(1);
        }
    } while(status == LUA_BREAK);
    // ...
    return status;
}

I get an output like:

start_nested
entering
entering
entering
before_breaker
main status: 6
okay, gonna go again
main status: 6
okay, gonna go again

Looking at the states of the lua_States tracked by the heap, they all correctly have state = LUA_BREAK, but calling lua_resume() on the top-level lua_State doesn't really do anything. I can get some of the code to run by manuallylua_resume()ing the lowest-level coroutine with the appropriate from (and perhaps continuing to do this until I reach the top-level coroutine again.) I suppose I could use the debuginterrupt hook to keep track of the order of threads that got interrupted by my lua_break() call, then call lua_resume() on them in sequence, but this doesn't seem ideal.

Is there a better way of achieving this kind of transparent halting / resuming of lua_States with potentially deep nested coroutine stacks?

[HaroldCindy]

It would appear that the culprit is ldo.cpp:resume()

luau/VM/src/ldo.cpp

Lines 364 to 365 in ce8495a

	// run continuations from the stack; typically resumes Lua code and pcalls
	resume_continue(L);

, where it calls out to ldo.cpp:resume_continue()

luau/VM/src/ldo.cpp

Lines 310 to 314 in ce8495a

	// Continuation can break again
	if (L->status == LUA_BREAK)
	break;
	luau_poscall(L, L->top - n);

, which then executes coroutine.wrap()'s continuation, which of course returns LUA_BREAK again in

luau/VM/src/lcorolib.cpp

Lines 164 to 170 in ce8495a

	static int auxwrapcont(lua_State* L, int status)
	{
	lua_State* co = lua_tothread(L, lua_upvalueindex(1));
	// if coroutine still hasn't yielded after the break, break current thread again
	if (co->status == LUA_BREAK)
	return interruptThread(L, co);

.

That explains the current behavior, but I'm not entirely sure how LUA_BREAK is meant to be used, is it only intended to work correctly with singlestep? 🤔

[HaroldCindy]

Calling lua_resume() (by way of auxresume()) inside of the various C continuations of lcorolib.cpp in the LUA_BREAK case seems to correct the issue, but I'm unclear if this approach is sound or desired. The existing comments in the continuation functions seem to indicate that auxresume() was intended to be called, given the "still hasn't yielded after the break" comment, though. Previously the continuation never actually attempted to resume coroutines, even if they had LUA_BREAK status.

For ref, something like

diff --git a/VM/src/lcorolib.cpp b/VM/src/lcorolib.cpp
index 3c79b1f8..80f92b95 100644
--- a/VM/src/lcorolib.cpp
+++ b/VM/src/lcorolib.cpp
@@ -22,7 +22,7 @@ static int costatus(lua_State* L)
 static int auxresume(lua_State* L, lua_State* co, int narg)
 {
     // error handling for edge cases
-    if (co->status != LUA_YIELD)
+    if (co->status != LUA_YIELD && co->status != LUA_BREAK)
     {
         int status = lua_costatus(L, co);
         if (status != LUA_COSUS)
@@ -126,9 +126,17 @@ static int coresumecont(lua_State* L, int status)
     lua_State* co = lua_tothread(L, 1);
     luaL_argexpected(L, co, 1, "thread");
 
-    // if coroutine still hasn't yielded after the break, break current thread again
+    // This continuation might be for a thread that was in break state, try resuming it
     if (co->status == LUA_BREAK)
-        return interruptThread(L, co);
+    {
+        auxresume(L, co, 0);
+
+        // if coroutine still hasn't yielded after the break, break current thread again
+        if (co->status == LUA_BREAK)
+        {
+            return interruptThread(L, co);
+        }
+    }
 
     int r = auxresumecont(L, co);
 
@@ -166,9 +174,17 @@ static int auxwrapcont(lua_State* L, int status)
 {
     lua_State* co = lua_tothread(L, lua_upvalueindex(1));
 
-    // if coroutine still hasn't yielded after the break, break current thread again
+    // This continuation might be for a thread that was in break state, try resuming it
     if (co->status == LUA_BREAK)
-        return interruptThread(L, co);
+    {
+        auxresume(L, co, 0);
+
+        // if coroutine still hasn't yielded after the break, break current thread again
+        if (co->status == LUA_BREAK)
+        {
+            return interruptThread(L, co);
+        }
+    }
 
     int r = auxresumecont(L, co);

Passes the Luau test suite and gives me the desired output of

start_nested
entering
entering
entering
before_breaker
main status: 6
okay, gonna go again
after_breaker
exiting
exiting
exiting
end_nested
OK