Issue #401: Cleanup for React packages for publishing #457

jahabeebs · 2024-06-10T06:18:11Z

@LukeLalor @dbrewster

closes #401

Description

Adds tsup to bundle eidolon-components and eidolon-client
Bumps turbo to 2.0
eslint-config was not working at all, I had to follow the instructions in the turbo docs to get this working and now it properly lints every package.
From eidolon-ui2 I did have to remove the "extends": "@repo/typescript-config/react-library.json" because it was causing issues and I adapted the tsconfig from the starter repo in the next.js examples repo (https://github.com/vercel/vercel/blob/main/examples/nextjs/tsconfig.json)
A ton of tsconfig and tsconfig lint changes for the published packages and eidolon-ui2. I can try to trim them down if you think there are too many
Adds outputs to turbo.json in webui and rename from pipeline to tasks ("pipeline" is not part of the turbo schema)
Separated the server and client components of eidolon-components and creates separate paths for them because they were causing server/client violations in eidolon-ui2
Added changeset workflow file (to .github/workflows) and cli dependency (to webui), not sure the best way to test this

I need help to:

Install the changeset bot to the eidolon repo: https://github.com/apps/changeset-bot
see docs here: https://github.com/changesets/changesets/blob/main/docs/automating-changesets.md
Best way to test the full changeset workflow? I don't want to try to publish and steal the npm namespaces

webui/packages/eidolon-components/src/provider/eidolon_provider.tsx

jahabeebs · 2024-06-19T06:05:04Z

@LukeLalor @dbrewster I think it's just about finished, I've updated the description with as much detail as I could about my changes (unfortunately I couldn't find a way to change any less files) but we can chat about it on Thursday

# Conflicts: # webui/apps/eidolon-ui2/package.json # webui/package.json # webui/pnpm-lock.yaml

jahabeebs · 2024-06-27T06:52:28Z

Remaining issues:

Test webui:

[Polly] [adapter:fetch] Using the fetch adapter in Node has been deprecated. Please use the node-http adapter instead.
PASS test/server/process-server-handler.test.ts

Playwright tests:
"fetch failed" when front end tries to access server--no api requests succeeding even though the server is healthy

logs:

2024-06-27 01:50:31 INFO:     Started server process [1]
2024-06-27 01:50:31 INFO:     Waiting for application startup.
2024-06-27 01:50:31 /usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:394: UserWarning: <built-in function callable> is not a Python type (it may be an instance of an object), Pydantic will allow any object with no validation since we cannot even enforce that the input is an instance of the given type. To get rid of this error wrap the type with `pydantic.SkipValidation`.
2024-06-27 01:50:31   warn(
2024-06-27 01:50:33 --- Logging error ---
2024-06-27 01:50:33 Traceback (most recent call last):
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/kernel.py", line 71, in get_resources
2024-06-27 01:50:33     ret[k] = tu[0].promote(kind)
2024-06-27 01:50:33              ^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/resources/resources_base.py", line 32, in promote
2024-06-27 01:50:33     return clazz.model_validate(self.model_dump())
2024-06-27 01:50:33            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/pydantic/main.py", line 551, in model_validate
2024-06-27 01:50:33     return cls.__pydantic_validator__.validate_python(
2024-06-27 01:50:33            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/reference_model.py", line 160, in _validate
2024-06-27 01:50:33 INFO - Overriding builtin resource 'Reference.ToolCallLLMWrapper'
2024-06-27 01:50:33 INFO - Building machine 'DEFAULT'
2024-06-27 01:50:33     spec.model_validate(self.model_extra or {})
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/pydantic/main.py", line 551, in model_validate
2024-06-27 01:50:33     return cls.__pydantic_validator__.validate_python(
2024-06-27 01:50:33            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/builtins/logic_units/web_search.py", line 75, in <lambda>
2024-06-27 01:50:33     default_factory=lambda: os.environ["CSE_ID"],
2024-06-27 01:50:33                             ~~~~~~~~~~^^^^^^^^^^
2024-06-27 01:50:33   File "<frozen os>", line 679, in __getitem__
2024-06-27 01:50:33 KeyError: 'CSE_ID'
2024-06-27 01:50:33 
2024-06-27 01:50:33 During handling of the above exception, another exception occurred:
2024-06-27 01:50:33 
2024-06-27 01:50:33 Traceback (most recent call last):
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/logging/__init__.py", line 1110, in emit
2024-06-27 01:50:33     msg = self.format(record)
2024-06-27 01:50:33           ^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/logging/__init__.py", line 953, in format
2024-06-27 01:50:33     return fmt.format(record)
2024-06-27 01:50:33            ^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_client/util/logger.py", line 15, in format
2024-06-27 01:50:33     return super().format(record)
2024-06-27 01:50:33            ^^^^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/logging/__init__.py", line 687, in format
2024-06-27 01:50:33     record.message = record.getMessage()
2024-06-27 01:50:33                      ^^^^^^^^^^^^^^^^^^^
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/logging/__init__.py", line 377, in getMessage
2024-06-27 01:50:33     msg = msg % self.args
2024-06-27 01:50:33           ~~~~^~~~~~~~~~~
2024-06-27 01:50:33 TypeError: not all arguments converted during string formatting
2024-06-27 01:50:33 Call stack:
2024-06-27 01:50:33   File "/usr/local/bin/eidolon-server", line 8, in <module>
2024-06-27 01:50:33     sys.exit(main())
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/bin/agent_http_server.py", line 242, in main
2024-06-27 01:50:33     uvicorn.run(
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 577, in run
2024-06-27 01:50:33     server.run()
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 65, in run
2024-06-27 01:50:33     return asyncio.run(self.serve(sockets=sockets))
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
2024-06-27 01:50:33     return runner.run(main)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
2024-06-27 01:50:33     return self._loop.run_until_complete(task)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/uvicorn/lifespan/on.py", line 86, in main
2024-06-27 01:50:33     await app(scope, self.receive, self.send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
2024-06-27 01:50:33     return await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2024-06-27 01:50:33     await super().__call__(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
2024-06-27 01:50:33     await self.middleware_stack(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 151, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 571, in __call__
2024-06-27 01:50:33     return await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 103, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 103, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 77, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 103, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 103, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 51, in __call__
2024-06-27 01:50:33     await self.app(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
2024-06-27 01:50:33     await self.middleware_stack(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 765, in app
2024-06-27 01:50:33     await self.lifespan(scope, receive, send)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 732, in lifespan
2024-06-27 01:50:33     async with self.lifespan_context(app) as maybe_state:
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
2024-06-27 01:50:33     return await anext(self.gen)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/bin/agent_http_server.py", line 173, in start_os
2024-06-27 01:50:33     machine: AgentMachine = machine_spec.instantiate()
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/reference_model.py", line 195, in instantiate
2024-06-27 01:50:33     return self._get_reference_class()(*args, **kwargs)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/agent_machine.py", line 60, in __init__
2024-06-27 01:50:33     for name, r in AgentOSKernel.get_resources(AgentResource).items():
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/kernel.py", line 73, in get_resources
2024-06-27 01:50:33     register_resource_promote_error(k, tu[0].kind, kind, e)
2024-06-27 01:50:33   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/resource_load_error_handler.py", line 31, in register_resource_promote_error
2024-06-27 01:50:33     logger.exception(f"Error promoting resource {name} of kind {resourceKind} to {kind}", e)
2024-06-27 01:50:33 Message: "Error promoting resource search_agent of kind Agent to <class 'eidolon_ai_sdk.system.resources.agent_resource.AgentResource'>"
2024-06-27 01:50:33 Arguments: (KeyError('CSE_ID'),)
2024-06-27 01:50:33 INFO - Starting agent 'conversational_agent'
2024-06-27 01:50:33 INFO - Starting agent 'scraping_agent'
2024-06-27 01:50:33 INFO - Starting agent 'research_agent'
2024-06-27 01:50:33 INFO - Starting agent 'speech_agent'
2024-06-27 01:50:33 INFO - Server Started in 2.34s

dbrewster · 2024-06-27T18:36:23Z

That stack trace is normal as one of the services, the web research service, can't be started, but it shouldn't affect anything.
It looks like the server never saw the request from the UI. Is the environment variable / config file correct for the UI? I think you ran into this when you were building the docker compose command line

jahabeebs · 2024-06-28T06:50:34Z

That stack trace is normal as one of the services, the web research service, can't be started, but it shouldn't affect anything. It looks like the server never saw the request from the UI. Is the environment variable / config file correct for the UI? I think you ran into this when you were building the docker compose command line

I fixed the webui component test that was failing (had to change an import)

The only other issue is the front end/back end communication thing:

When I use docker-compose.yml locally the front end is able to make calls to the back end and can fetch the chatbot processes, but as soon as I try to start a new chat or access a new chat there's a front-end error that I can't see because of minification (the server stays healthy, it's just a client-side error). I tried setting NODE_ENV=development everywhere I could to be able to see the actual error but I haven't been able to get past that minification for some reason

When I run the front end with the pnpm run dev command (rather than build) + a docker back end the chatbot works perfectly and the front end doesn't crash

So for some reason the built version of the front end is crashing but not the development version...even in React strict mode

…401-package-monorepos

jahabeebs · 2024-07-03T03:36:29Z

@dbrewster Unfortunately I still haven't figured out the source of this minified JS error that I get after building everything. Even when I pass in NODE_ENV=development to docker-compose and the webui dockerfile it still minifies on build. Here's a video of what happens (can trigger the crash also by pressing add chat). When running the front end in dev mode rather than built it works just fine.

Minified react error 130: https://react.dev/errors/130

A possible cause of the issues is me splitting up the eidolon-components package into client and server dists. Technically we can't have class components as server components (like AgentHandler, FileHandler, FilesHandler, see here for details: vercel/next.js#47232 and here: palantir/blueprint#6263), however I tried 1) refactoring them as functional components (ended up with same minified error) and 2) making them client components (eidolon-ui2 wouldn't even build, get an error "TypeError: o.TX is not a constructor" when building the routes

eidolon_ui2_crashes.mov

jahabeebs added 5 commits June 9, 2024 19:24

package monorepos

6535b86

packaging monorepos

91b98c2

no reference

95282f7

clean up eidolon-client

fc5ce09

cache

a630eba

vercel bot deployed to Preview June 10, 2024 06:19 View deployment

more cleanup

8c43c09

vercel bot deployed to Preview June 12, 2024 05:53 View deployment

LukeLalor reviewed Jun 12, 2024

View reviewed changes

webui/packages/eidolon-components/src/provider/eidolon_provider.tsx Outdated Show resolved Hide resolved

fix next.js build issue, add to tsconfig, remove excess types

0a79c8d

vercel bot deployed to Preview June 14, 2024 06:38 View deployment

fix eslint issues

419b491

vercel bot deployed to Preview June 17, 2024 04:48 View deployment

migrate turbo

16382a1

vercel bot deployed to Preview June 17, 2024 04:49 View deployment

fix more build issues, convert eidolon-ui2 to module

e546366

vercel bot deployed to Preview June 17, 2024 05:40 View deployment

revert ui2 back to standard tsconfig

b53a5f7

vercel bot deployed to Preview June 19, 2024 04:05 View deployment

still making whole bundle a client component

8dc78b1

vercel bot deployed to Preview June 19, 2024 04:29 View deployment

fix client/server component issues

ae14e7e

vercel bot deployed to Preview June 19, 2024 04:49 View deployment

test publishing

1654b1a

vercel bot deployed to Preview June 19, 2024 05:32 View deployment

cleanup

0b0f620

vercel bot deployed to Preview June 19, 2024 05:47 View deployment

Merge branch 'refs/heads/main' into feature/401-package-monorepos

89ca308

# Conflicts: # webui/apps/eidolon-ui2/package.json # webui/package.json # webui/pnpm-lock.yaml

vercel bot deployed to Preview June 26, 2024 03:01 View deployment

vercel bot deployed to Preview June 27, 2024 05:06 View deployment

troubleshoot tsup config not found

901e39c

vercel bot deployed to Preview June 27, 2024 05:13 View deployment

fix build

f18306f

vercel bot deployed to Preview June 27, 2024 05:26 View deployment

fix tests

dbf4f99

vercel bot deployed to Preview June 27, 2024 05:48 View deployment

no test?

d673d5f

vercel bot deployed to Preview June 27, 2024 05:55 View deployment

need eidolon client dist

a05e612

vercel bot deployed to Preview June 27, 2024 06:02 View deployment

copy dist correctly

364d0e0

vercel bot deployed to Preview June 27, 2024 06:07 View deployment

strict mode

8dc2459

vercel bot deployed to Preview June 28, 2024 05:51 View deployment

clean up exports

be11b97

vercel bot deployed to Preview June 28, 2024 06:29 View deployment

fix path

82a91b5

vercel bot deployed to Preview June 28, 2024 06:39 View deployment

Merge remote-tracking branch 'refs/remotes/origin/main' into feature/…

cdf4c7f

…401-package-monorepos

vercel bot deployed to Preview June 28, 2024 06:53 View deployment

Merge branch 'main' into feature/401-package-monorepos

668cc3f

vercel bot deployed to Preview July 3, 2024 05:53 View deployment

Merge branch 'main' into feature/401-package-monorepos

361a74c

vercel bot deployed to Preview July 17, 2024 05:45 View deployment

dbrewster closed this Jul 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue #401: Cleanup for React packages for publishing #457

Issue #401: Cleanup for React packages for publishing #457

jahabeebs commented Jun 10, 2024 •

edited

Loading

jahabeebs commented Jun 19, 2024 •

edited

Loading

jahabeebs commented Jun 27, 2024

dbrewster commented Jun 27, 2024

jahabeebs commented Jun 28, 2024

jahabeebs commented Jul 3, 2024 •

edited

Loading

Issue #401: Cleanup for React packages for publishing #457

Issue #401: Cleanup for React packages for publishing #457

Conversation

jahabeebs commented Jun 10, 2024 • edited Loading

Description

I need help to:

jahabeebs commented Jun 19, 2024 • edited Loading

jahabeebs commented Jun 27, 2024

dbrewster commented Jun 27, 2024

jahabeebs commented Jun 28, 2024

jahabeebs commented Jul 3, 2024 • edited Loading

jahabeebs commented Jun 10, 2024 •

edited

Loading

jahabeebs commented Jun 19, 2024 •

edited

Loading

jahabeebs commented Jul 3, 2024 •

edited

Loading