Initial proposal: dynamic schema stitching at runtime.

[mattblackdev]

@theodorDiaconu I'm the random dude that messaged you in the Meteor forum DM about dynamic schemas. As you know, I have a need to generate dynamic schemas at runtime. This is my initial attempt at making it possible to do runtime schema stitching with your libs. This pull request isn't the full solution but merely a humble proposal. I wanted to at least show some attempt and get feedback. There are three parts to this so far: 2 commits here and another PR in grapher-schema-directives repo.

1. I tested the idea on the grapher-performance repo and learned I could leverage your graphql-load package to merge new schemas & resolvers. (Awesome work on that btw!) So this PR allows for a Loader instance to be passed in via Config. I figured I need to pass an instance instead of using the default/static one so I can have full control over what is loaded, and add and remove things at runtime.

2. The next thing I learned was calling initialize() a second time will rebuild the whole apollo server. I think this is okay, but grapher will attempt to call addLinks on existing collections with existing links. So there is another pull request in the directives repo that merely checks if a link exists already before adding. I'll talk more about it on that pull request.

3. When initialize runs a second time it adds a new graphQLServer handler via WebApp.connectHandlers.use(). I found that the "stack" in connect is not really a "stack". So the original graphQLServer is getting called and not the new one. To get around this I added a line that simply clears the stack via WebApp.connectHandlers.stack = []. Clearly this is not the best idea, but I haven't found another way yet.

Again, this is just a proof of concept. And I hope we can leverage this PR to build the right API for this use case. Perhaps there could be something like refreshServer({ loader }).

Finally, all I've done as far as testing this is a few changes in the grapher-performance repo:

// server/load.js
const loader = new Loader()

loader.load([
  EntitiesModule,
  {
    typeDefs: [QueryType],
  },
])

loader.load({
  resolvers: QueryGrapherResolver,
})

initialize({ LOADER: loader })

Meteor.setTimeout(() => {
  loader.load({
    typeDefs: `type Query { sayHello: String }`,
    resolvers: {
      Query: { sayHello: () => 'Hello' },
    },
  })
  initialize()
}, 10 * 1000)

And boom, new merged schema and resolver visible in graphiql. My next step will be to generate the grapher resolvers automatically, and since they map so well:

// server/queries/QueryGrapher.resolver.js
// ...
    users(_, args, { db }, ast) {
      return db.users.astToQuery(ast).fetch();
    },

    comments(_, args, { db }, ast) {
      return db.comments.astToQuery(ast).fetch();
    },
// ...

It should be pretty painless. I wouldn't be surprised if you're already working on that!

Anyways, I hope you see the value in adding this as you've pretty much already done 95% of the work. Let me know what you think about adding a refreshServer({ loader }) function and any other advice on the changes in these two PRs.

Thanks!

[theodorDiaconu]

That's so badass, but aren't you concerned of downtime as the schema updates ? I think the bit concerning. Also, I don't see this as a super commonly used scenario, however it isn't bad either.

[mattblackdev]

I think because its just swapping middleware, it's not an issue? Unless a request can get dropped between two synchronous lines of code:

// server/core/main-server.js:128

// this removes all handlers
WebApp.connectHandlers.stack = [];
// this binds the specified paths to the Express server running Apollo + GraphiQL
WebApp.connectHandlers.use(graphQLServer);

However, I would like to eventually revisit how to only replace the old graphQLServer handler without wiping out the whole "stack" array of middleware. I think it could be done by wrapping the graphQLServer with middleware of its own that would do the swapping... actually might do this sooner than later. I'm also not sure if there should be any clean up of the old graphQLServer instance, but maybe garbage collection is all thats needed.

But do you have any thoughts on a real "refresh" api? Instead of calling initialize again? I'm sure there's a lot of work that initialize does that could be skipped especially when it gets to the grapher part. But I just don't know grapher internals at all right now.

[theodorDiaconu]

@mattblackdev there is really not a plan for the refresh api, I think your use-case is so rare I don't even know if it's worth to have in this repo (this repo goal is to just get you quickly started with apollo, and serve 95% of cases, but you are in the 5%), but you could have another package, that hacks this package. And I can expose/make these code-changes inside the repo to allow the things you wanna do.

[theodorDiaconu]

@mattblackdev this feature may be very very good. I want to feature your package in this package. Any updates on it ?

[mattblackdev]

@theodorDiaconu Actually, I learned about Prisma and development took a turn. Now I'm using our meteor app to orchestrate a prisma instance. Which basically makes it like an internal graph.cool service.

I'm not 100% sure that's the right way to go either and might come back to this. One drawback is it's a whole other database and server to manage. But it has a lot of benefits for the users in the long run. Regardless, I'm curious what do you like in particular about the possibility of a this feature?

A few things I've learned:

• Prisma is badass, but written in Scala lol.

• graphql-middleware is a thing(i.e. graphql-shield) and seems to be getting adopted by the community. Schema directives are SUPER powerful, yet are not the ideal way to implement runtime resolver generation (even though I've done it that way already) because as you said they mix concerns in the SDL.

• graphql-yoga is cool but I couldn't get it to roll with meteor. Apollo-server 2.0 beta is out but I also couldn't get it to role with meteor.

• I think once we see some good implementations of graphql-middleware and yoga's so called "document-middleware", this dynamic stuff will be a lot simpler. Also, when prisma get's a mongodb connector, integration with meteor might be really interesting.

-Matt

[theodorDiaconu]

@mattblackdev in which direction did you go ? I'm curious to know!

[mattblackdev]

@theodorDiaconu The Prisma thing was cool but it felt like overkill and we weren't thrilled about managing 2 more servers and dealing with MySQL.

So, I did another attempt with VulcanJS. It uses extended SimpleSchema definitions for collections and has GraphQL generation built it. I was able to serialize the definitions and load them from the DB when the server starts. That would generate permissions/auth/validation, CRUD resolvers and even UI like lists and forms. It actually worked really great and Vulcans future is bright, but I had to stop working on it. Next step would be to hack it so we could bounce the graphql server at runtime.

At work we went back to iterating on our production app because of some user feedback. Hopefully will get back to this soon. I still keep an eye on this project. Props to you for updating to apollo server 2.0.

Cheers!

[mattblackdev]

@theodorDiaconu graphql-middleware is here, thought you’d be interested. check it out: https://www.prisma.io/blog/graphql-middleware-zie3iphithxy/

Btw, I really think Vulcan should adopt your grapher+apollo setup.

[theodorDiaconu]

@mattblackdev wow the middleware is impressive very nice. Regarding Vulcan, I have scheduled a discussion with Sacha.

[mattblackdev]

Hell yeah! Thanks, Matthew

…

On Aug 17, 2018, 2:43 AM -0400, Theodor Diaconu ***@***.***>, wrote: @mattblackdev wow the middleware is impressive very nice. Regarding Vulcan, I have scheduled a discussion with Sacha. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[theodorDiaconu]

I am starting to realise that if this gets done. We can dynamically update the whole API at RUNTIME. This is crazy!!!!

[jamesgibson14]

I have been dynamically generating my entire Meteor app for 5 years now. And it IS Amazing!!! I can create/update templates, routes, methods, etc on the fly. However I haven't been able to switch to Apollo yet because of this issue,

[theodorDiaconu]

@jamesgibson14 how do you do that? can you show us a demo I'm curious!

[jamesgibson14]

Well from a high level I just observe a "Modules" collection and evaluate the Module.code which is just a string of code either javascript or html/blaze/vue. I use Ace/Monaco Editor to edit those modules. which then triggers a new eval.

There are a lot of little hacks with Meteor I learned along the way to get routes and methods working.
This is the code for an actual module that creates a Meteor method:

delete Meteor.server.method_handlers.testMethod;
const testMethod = new ValidatedMethod({
  name: 'testMethod',
  validate: function (args) {
    console.log('args123456', args);
    
  },
  run: function (args) {
    
    
   return args
  }
});

As soon as I eval this I can call it from the client.

The problem with apollo is the express routes, I can evaluate a new route but it is hard to loop through the current routes array and remove the route you are trying to replace. I have hacked WebApps in Meteor to name the routes and then I can delete it, but I can't do that with Apollo... yet. I believe in this issue I would have to rebuild the entire WebApps.handlers.routes array for every eval, which is not out of the question.