pygls does not handle async user shutdown handler correctly

[alcarney]

If I register an async handler for the shutdown request, pygls does not guarantee that it will finish executing before responding to the client. This often leads to the client sending the exit notification and pygls calling sys.exit() before my code cleaned up everything it needs to.

I'm finding this is leading to a lot of orphaned esbonio processes hanging around on my machine! 😅

I think the issue is down to how pygls handles the user registering features that "shadow" built in features.

pygls/pygls/protocol/lsp_meta.py

Lines 15 to 29 in d3a1421

	@functools.wraps(base_func)
	def decorator(self, *args, **kwargs):
	ret_val = base_func(self, *args, **kwargs)
	try:
	user_func = self.fm.features[method_name]
	self._execute_notification(user_func, *args, **kwargs)
	except KeyError:
	pass
	except Exception:
	logger.exception(
	'Failed to handle user defined notification "%s": %s', method_name, args
	)
	return ret_val

If we look at how _execute_notification is implemented

pygls/pygls/protocol/json_rpc.py

Lines 144 to 153 in d3a1421

	def _execute_notification(self, handler, *params):
	"""Executes notification message handler."""
	if asyncio.iscoroutinefunction(handler):
	future = asyncio.ensure_future(handler(*params))
	future.add_done_callback(self._execute_notification_callback)
	else:
	if is_thread_function(handler):
	self._server.thread_pool.apply_async(handler, (*params,))
	else:
	handler(*params)

It uses asyncio.ensure_future to schedule the coroutine to be executed, but then will return almost immediately - meaning there's no guarantee it will complete before pygls responds to the client.

[alcarney]

I think I have an idea that will fix this, but it will probably have to go into #418.

If we allow pygls' built-in features to yield, there's an opportunity to run the user's code in the middle of the method, before resuming the built-in to run to completion.

Not only would this fix this issue, but it might help with #381, where pygls can do some initial setup e.g. initialize the workspace, set client capabilities on the server object etc. and then yield to the user's code - allowing them to register some extra features, before resuming to compute the server's capabilities and eventually respond to the client