Issue running starlette: TypeError: _run_asgi2() takes 2 positional arguments but 4 were given

[cancan101]

I am getting the following traceback when trying to launch an asgi server with starlette:
I am using this run command: newrelic-admin run-program gunicorn -w 4 -k uvicorn.workers.UvicornWorker my_package:app

Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 391, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/app/.heroku/python/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/app/.heroku/python/lib/python3.8/site-packages/newrelic/api/asgi_application.py", line 387, in nr_async_asgi
    return await coro
  File "/app/.heroku/python/lib/python3.8/site-packages/newrelic/common/async_proxy.py", line 148, in __next__
    return self.send(None)
  File "/app/.heroku/python/lib/python3.8/site-packages/newrelic/common/async_proxy.py", line 120, in send
    return self.__wrapped__.send(value)
  File "/app/.heroku/python/lib/python3.8/site-packages/newrelic/api/asgi_application.py", line 387, in nr_async_asgi
    return await coro
  File "/app/.heroku/python/lib/python3.8/site-packages/newrelic/common/async_proxy.py", line 148, in __next__
    return self.send(None)
  File "/app/.heroku/python/lib/python3.8/site-packages/newrelic/common/async_proxy.py", line 120, in send
    return self.__wrapped__.send(value)
  File "/app/.heroku/python/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "<string>", line 5, in wrapper
  File "/app/.heroku/python/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/app/.heroku/python/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/app/.heroku/python/lib/python3.8/site-packages/newrelic/hooks/framework_starlette.py", line 108, in middleware_wrapper
    return await FunctionTraceWrapper(wrapped, name=name)(*args, **kwargs)
  File "/app/.heroku/python/lib/python3.8/site-packages/newrelic/api/function_trace.py", line 150, in literal_wrapper
    return wrapped(*args, **kwargs)
TypeError: _run_asgi2() takes 2 positional arguments but 4 were given

Using:

newrelic==5.24.0.153
starlette==0.13.8

[a-feld]

Interesting @cancan101 , thanks for filing an issue!

From that stack trace, it looks like what's happening is the starlette errors middleware (using ASGI v3 protocol) is calling an ASGI v2 application, which is incompatible, causing the crash.

• Does the application work correctly without the newrelic-admin command?
• Do you have a copy of the _run_asgi2 function that's being called?
• Any more information about your stack that you can share (middlewares, any starlette extensions, etc.)?

[cancan101]

Does the application work correctly without the newrelic-admin command?

yes

Do you have a copy of the _run_asgi2 function that's being called?

I don't have that any where in my project code

Any more information about your stack that you can share (middlewares, any starlette extensions, etc.)?

I think this is being caused by the Sentry asgi middleware: https://github.com/getsentry/sentry-python/blob/dbd7ce89b24df83380900895307642138a74d27a/sentry_sdk/integrations/asgi.py#L91-L94

[a-feld]

That definitely does seem to be the cause! How did you add the sentry middleware?

This is the app I'm running with and I can't reproduce the crash:

from sentry_sdk.integrations.asgi import SentryAsgiMiddleware
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.responses import PlainTextResponse
from starlette.routing import Route


async def pong(request):
    return PlainTextResponse("*")


routes = [
    Route("/", pong),
]
app = Starlette(routes=routes, middleware=[Middleware(SentryAsgiMiddleware)])

[cancan101]

I am also using this middleware which seems to be causing the issue: https://github.com/mfreeborn/fastapi-sqlalchemy/blob/39e7e8bb5d98a9de5fc365adf4bdc661d824b614/fastapi_sqlalchemy/middleware.py#L18-L45
I have it added after sentry in the list.

[cancan101]

It seems like FunctionWrapper from new relic is being wrapped around the DB middleware causing sentry to then mis-classify it as asgi2

[cancan101]

Ok the issue doesn't seem to be specific to the DB middleware, it is just having any other middleware after the sentry middleware that triggers the issue.

[cancan101]

If there is no other middleware specified then the next middleware after sentry is starlette.exceptions.ExceptionMiddleware which seems to be special cased by new relic:

newrelic-python-agent/newrelic/hooks/framework_starlette.py

Lines 219 to 226 in ca22180

	def instrument_starlette_exceptions(module):
	wrap_function_wrapper(module, "ExceptionMiddleware.__call__", error_middleware_wrapper)
	wrap_function_wrapper(module, "ExceptionMiddleware.http_exception",
	wrap_exception_handler)
	wrap_function_wrapper(module, "ExceptionMiddleware.add_exception_handler",
	wrap_add_exception_handler)

[a-feld]

Thanks for that info, I've reproduced the issue and I'm investigating.

from sentry_sdk.integrations.asgi import SentryAsgiMiddleware
from starlette.applications import Starlette
from starlette.responses import PlainTextResponse
from starlette.routing import Route


async def pong(request):
    return PlainTextResponse("*")


routes = [
    Route("/", pong),
]
app = Starlette(routes=routes)


@app.middleware("http")
async def middleware_with_decorator(request, call_next):
    response = await call_next(request)
    return response


app.add_middleware(SentryAsgiMiddleware)

[a-feld]

We've encountered this kind of problem before because ASGI2/3 detection via attribute introspection doesn't work very well with wrapt function wrappers. We had to fix this in uvicorn by deferring the assignment of the wrapper until the interface modes were resolved.

We're just discussing now how to address these kinds of issues.

[a-feld]

Maybe @GrahamDumpleton has ideas on how to solve this issue? 😄

There may be some things that can be done in wrapt, such as inspecting if the wrapper function is a coroutine function and then propagating attributes / flags upward.

[a-feld]

The TL/DR as I understand it currently:

import asyncio


def foo():
    return bar()


async def bar():
    return await asyncio.sleep(0)


# >>> False
print("asyncio.iscoroutinefunction(foo)", asyncio.iscoroutinefunction(foo))

# >>> True
print("asyncio.iscoroutine(foo())", asyncio.iscoroutine(foo()))

[GrahamDumpleton]

Neither the wrapt ObjectProxy or FunctionWrapper which derives from it have any support for proxying async functions. Thus if using the inspect functions on a wrapper and you ask whether the wrapped function is async, you will get the wrong result if it is.

The callable(), inspect.iscoroutine() and inspect.iscoroutinefunction() functions are problematic in a proxy object because if you wrap something which isn't any of them, yet add the proxy functions, the tests on the wrapper will always say the wrapped object is a function or async function even if they aren't.

This is why the base ObjectProxy doesn't have a proxy for __call__() and if you know you are wrapping a function you must use FunctionWrapper which does. There is no separate AsyncFunctionWrapper object for async functions, but even if there was it would be complicated to know when to use it versus a normal function wrapper in a generic system where over versions of the software being wrapped, what is used can change.

The only possible solution have thought might be able to use to resolve the problem is to have the constructor for the proxy object perform a check whether a normal function, async function is passed and add extra proxy functions to the wrapper instance in that case for async.

Complicating this though is that you also have async generators and async context managers, so you need to look for the presence of each specific possible async dunder method and create a proxy function for each one as found. This will create a bit of overhead on construction and overhead may mount up over time if done on the generic project.

For FunctionWrapper at least, since should know only dealing with a function, it may be enough to look for just __await__() and add a proxy function for it to the wrapper instance. I have no idea right now how async instance methods complicate this though.

So suggest adding to the newrelic FunctionWrapper equivalent a constructor which checks for __await__ on what is being wrapped, and if there is, add a proxy function for it. Not sure if this is enough though for async instance methods as may need to be add support down in wrapt directly in the special _FunctionWrapperBase and BoundFuntionWrapper types.

Anyway, since not really been devoting any time to improving wrapt, I have been ignoring the whole async issues for a very long time now. So this isn't the only related issue.

• AttributeError: __aexit__ when using wrapt.ObjectProxy on async context managers. GrahamDumpleton/wrapt#73
• Recipe for universal decorator that accepts async function GrahamDumpleton/wrapt#150

Ability to put time in on this is a bit complicated right now.

[a-feld]

Thanks for the note @GrahamDumpleton ❤️ ! These are some really good ideas, we'll discuss on the team to decide which way we'll resolve this. We'll also discuss/explore if we can contribute back to wrapt, keeping in mind any limited time you have for review 🚀 .

[a-feld]

@cancan101 While we investigate, you should be able to work around this issue by forcing all requests to be treated as ASGI v3 by adding this patch:

SentryAsgiMiddleware._run_asgi2 = SentryAsgiMiddleware._run_asgi3

This forces the _run_asgi2 call to run the _run_asgi3 method instead.

Example app:

from sentry_sdk.integrations.asgi import SentryAsgiMiddleware
from starlette.applications import Starlette
from starlette.responses import PlainTextResponse
from starlette.routing import Route


async def pong(request):
    return PlainTextResponse("*")


routes = [
    Route("/", pong),
]
app = Starlette(routes=routes)


@app.middleware("http")
async def middleware_with_decorator(request, call_next):
    response = await call_next(request)
    return response


SentryAsgiMiddleware._run_asgi2 = SentryAsgiMiddleware._run_asgi3
app.add_middleware(SentryAsgiMiddleware)

[cancan101]

I wonder if this was a similar issue that Starlette ran into and if so if there is any inspiration to be taken from their solution:? encode/starlette#1106

[a-feld]

The problem is that the sentry middleware is classifying our wrapper (an ASGI v3 app) as an ASGI v2 app. Therefore, any fix around detection logic would have to be applied to the sentry middleware itself.

It might be worth putting in a feature request with Sentry to specify the ASGI type via the constructor rather than relying on auto detection in all cases. uvicorn provides a switch to set the ASGI interface (via the --interface [auto|asgi3|asgi2|wsgi] flag), for example.

[GrahamDumpleton]

What exactly is the supposed difference between ASGI 2 and 3 calling interface. As far as I can tell, wrapt actually seems to be giving the correct results.

import inspect
import asyncio

from wrapt import decorator

async def func1():
    await asyncio.sleep(1)

@decorator
def wrapper(wrapped, instance, args, kwargs):
    return wrapped(*args, **kwargs)

@wrapper
async def func2():
    print('WRAPPER')
    await asyncio.sleep(1)

def _looks_like_asgi3(app):
    # type: (Any) -> bool
    """
    Try to figure out if an application object supports ASGI3.
    This is how uvicorn figures out the application version as well.
    """
    if inspect.isclass(app):
        return hasattr(app, "__await__")
    elif inspect.isfunction(app):
        return asyncio.iscoroutinefunction(app)
    else:
        call = getattr(app, "__call__", None)  # noqa
        return asyncio.iscoroutinefunction(call)

async def main():
    print("FUNC1")
    print("ISCOROUTINEFUNCTION", inspect.iscoroutinefunction(func1))
    print("LOOKSLIKEASGI3", _looks_like_asgi3(func1))
    result = func1()
    print("ISCOROUTINE", inspect.iscoroutine(result))
    await result

    print("FUNC2")
    print("ISCOROUTINEFUNCTION", inspect.iscoroutinefunction(func2))
    print("LOOKSLIKEASGI3", _looks_like_asgi3(func2))
    result = func2()
    print("ISCOROUTINE", inspect.iscoroutine(result))
    await result

asyncio.run(main())

This yields:

FUNC1
ISCOROUTINEFUNCTION True
LOOKSLIKEASGI3 True
ISCOROUTINE True
FUNC2
ISCOROUTINEFUNCTION True
LOOKSLIKEASGI3 True
ISCOROUTINE True
WRAPPER

So the results are same when the decorator is applied.

Anyone have an example of ASGI 2 code so can check with it?

Right now looks like issue may be elsewhere.

[GrahamDumpleton]

For reference:

• https://asgi.readthedocs.io/en/latest/specs/main.html?highlight=legacy#legacy-applications

[a-feld]

This was the reproduction case I used if you wanted to have a look @GrahamDumpleton 😄
#117 (comment)

The repro needs to be run with newrelic using newrelic-admin

[GrahamDumpleton]

Should add one comment. The reason that wrapt gives the correct results is that inspect.iscoroutinefunction() and inspec.iscoroutine() internally use isinstance() checks. With wrapt isinstance() checks actually work against the original wrapped type, as well as the wrapper, so the lookups done for subsequent to those checks appears to work. So my original comments saying they likely wouldn't work was wrong just because of how they were implemented.

[a-feld]

For full transparency, it's totally possible that I completely mis-classified the problem, in which case I'm sure Graham will take me to school here 😅
I can try to give this a second look in the morning, but can't make any promises since there's some other work that I've been committed to at the moment 😛 .

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[LyleScott]

This is still an issue for us with similar circumstances.

[jojozhang-ap]

This is still an issue for us too, while we are using ASGI middleware for sentry and New Relic as monitoring service.

[mrbaboon]

Is this truly a wontfix? We are currently running into this with one of our FastAPI projects, and don't want to have to make a choice between NewRelic and Sentry.

[andreagrandi]

Why has this been marked as wontfix? We are having the same issue and even trying the alternative method (here: https://docs.newrelic.com/docs/apm/agents/python-agent/installation/python-agent-advanced-integration/#manual-integration ) we have exactly the same error. Basically we have no way to use New Relic in our application because we are also using Sentry.

[mrlubos]

I think sentry-python v1.8.0 release fixes this!

[alex-kessler-katapult]

still an open issue. don't want to drop newrelic.

[anton-demydov-zoral]

We have the same issue

[KingNoosh]

We're facing this too, trying to evaluate New Relic and Datadog at the moment.

[umaannamalai]

We have a PR (linked to this issue) up now to patch this that should go out in our next release. The branch is available for earlier testing as well.

[Ak-x]

Hello all, the PR has been merged. Closing this issue unless someone reports they are still experiencing it.

Here is the latest version which includes the fix: Python 8.5.0