[discuss] Move to domain-based directory structure #71566
Comments
joshdover
added
discuss
Team:Core
|
Pinging @elastic/kibana-platform (Team:Platform) |
|
Pinging @elastic/kibana-operations (Team:Operations) |
|
Pinging @elastic/kibana-app-arch (Team:AppArch) |
|
cc @epixa because I know "organize by domain" was a point you were previously passionate about :) |
Is this something we already know we'll be doing, or only a proposal? TBH, I'm not that big of a fan for a couple reasons:
In short, that's only my opinion, but I think that having the suggested domain structure, while keeping it under the existing |
|
I like the goals of this effort and the outcome, but I wonder if we could alleviate some of @pgayvallet's concerns by tweaking the execution plan. What about these steps:
I think step 3 will be faster, and we avoid @pgayvallet's concern of having 'non-domain' folders scanned unnecessarily. Also, @spalger just pulled this PR together for me: #71731, which relies on these plugin folders. |
|
@pgayvallet has some good practical points and I'd love to hear other opinions from the team and contributors outside the team too. This is already implied in the proposal, but to call it out directly: regardless of what we adopt, at some point moving away from layer-centric is a scale imperative. What does our repository structure look like in 2 years if we do nothing vs adopt a different structure? Reading the goals section, I think all the goals have a common theme of supporting a larger codebase and many more contributors than we have today. Take x-pack: 2 years ago, x-pack folder had 17 plugins. Today master has 63, so a bit less than 4x. 2 years from now we're easily 100+ (yikes!) and if the growth trend continues as-is it'll be significantly higher. At some point layer-centric organization becomes impractical, so thank you @joshdover and the team for the clear proposal and trying to get ahead of this. |
Nothing is certain yet, but we know we need a solution to scale our source code repository.
The idea (though not explicitly stated above) is that directories should be able to contain all of the code that is related to a single domain. That code may include plugins, packages, types, documentation, etc. So the idea of moving out of the
I actually don't think part is required in order to implement the rest. Depending on how we go about this, instead we could:
(1) sounds like more manual work than it is since I don't expect these domains to change often. However I do think that having a strict separation between what goes in the build vs. what doesn't may be helpful in other contexts. But I agree it's not necessary right now.
This seems like a perfect workable plan to me, but it does require we move plugin code twice. I'm not sure what is more effort though. Maybe @elastic/kibana-operations can shed light on what adjustments we would need to make to the build system. |
|
I share the opinions that @joshdover expressed in this comment. I'd also like to add that not tying the directory structure to the concept of plugins gives us flexibility in defining what should be a plugin, without having to drastically restructure the code again. The only thing that absolutely has to be a plugin at this point is "x-pack", because it has to be conceptually installed and separate from the OSS build. We've historically used plugins at a much more granular level, and used them to enforce encapsulation when this isn't absolutely necessary. |
It is relevant to the UI applications only? How it's different from the
Is core === platform + Security plugin + Spaces plugin? If so, we might want to call it
The testing part is not fully covered in the proposal. I'd argue the all the functional tests that we have in
@joshdover How restructuring codebase affects the plugin IDs? Right now we have the same domains in OSS and x-pack folders. Do we want to formalize plugin naming schema? |
There isn't anything stopping us from supporting domain-specific development mode in the I just expect that we will try to make the change and fix the things that are broken... I'm not sure how much will break, but based on my experience I don't think that anything could break that would be too much effort to fix. I have a lot of confidence that once we have consensus on where we want to go we can get there by just starting a branch and moving forward. Sorry, I know that's not fantastic for planning exactly how many steps this will take but it's the best I can commit to. |
|
@constancecchen @sqren @XavierM @nreese @cjcenizal would be curious to hear some opinions from other teams on the merits or potential issues of domain-based folders for your domains. |
|
I think the proposal makes a lot of sense. I really like the idea of |
|
Thanks for the tag! 👍 / no strong objections here - from the Enterprise Search team's POV, this isn't a significant change from our current planned plugin directory structure. Based on the list in the top comment, sounds like we'd fall under x-pack like so:
Hopefully that looks sane. |
|
We spent some time thinking about domains while reorganizing the Management apps and I think we arrived at some pretty good ones:
For the apps owned by ES UI, I think they'd make more sense grouped into
|
I think a domain-based structure makes sense and having all observability code in a single location is a good thing. I don't foresee any problems migrating to this structure. |
|
During previous discussions, we decided that You brought up a good point @restrry about the tests not being covered. I agree that we should ideally make these part of the appropriate domains. It might take us a bit to get here, but it makes sense to me. @cjcenizal, I agree that the domains you all came up for the Management sections are great. @joshdover and I discussed whether or not it'd make sense to introduce a Perhaps we should consider renaming the |
|
Looks good to me from the Kibana App plugins. I've closed our Kibana App disucssion ticket (#70452) in favor of this one. The only thing missing from my side in the above list is the |
|
Combining a few suggestions & discussions above, I've tried to regroup the "core" and "application_services" domains under a single "platform" domain which contains a few sub-domains.
There's still quite a few cross-cutting domains in the What I like about having a "platform" domain is that it makes it quite clear where most shared services live, whether they are plugins or not. Of course other domains may expose APIs, but the primary place for purely shared APIs can live under "platform".
Agreed, this was not very clear. I intend for the I agree also that we need to be able to move the e2e tests out of the top-level
I tend to agree here. I think the UI groupings make sense from a user-perspective, but that the domains in our codebase do not necessarily have to map to what the user sees. To take a step back, the goal of this proposal is to scale our codebase to more developers. Our premise is that using good software design practices will help accomplish that. By organizing by domain, we can facilitate appropriate code sharing as well as proper encapsulation between domains. How we build software, organize our teams, and communicate across the company will be heavily influenced by this structure (and vice versa). This proposal allows teams to have their own "mini-platform" that is specific to their domain without sharing prematurely with the rest of the Kibana ecosystem. Sharing prematurely can lead to new, unintentional dependencies across the codebase on components that were not intended to be shared. These dependencies become a maintenance burden that can slow down new development. It's also important that as our codebase grows, new and old developers alike can find services relevant to the task at hand. Having clearly labelled domains (and sub-domains!) is incredibly helpful, especially when we have a growing list of 100+ plugins. The "data" grouping in the screenshot doesn't really map well to Kibana's codebase. It's too broad a term to describe what is actually inside of it, which are plugins related to user data indices. I also agree with @cjcenizal's point that "elasticsearch" may be too broad as well. We should consider labels that both describe what is inside the domain and how it is related to the rest of Kibana from an architecture standpoint (not necessarily a user standpoint). The other domains you listed, CJ, make sense to me, with Watcher being the wild card here. I don't feel too strongly about where that goes. I agree it's architecturally distinct from Kibana's Alerting platform, however it is related from a domain perspective. If doesn't share much code with anything else, it seems fine to put it into |
|
When discussing with @cjcenizal outside of this GitHub issue, he brought up a good point that it's not immediately obvious why the proposed domain-oriented hierarchy satisfies the goals which were originally outlined. Below, I've expounded upon why I think the proposed hierarchy satisfies these goals, in an effort to provide more context. Create a new repository structure that can grow with our codebase and organizationCurrently, there are three primary categories of Kibana code: a plugin, a This is a rather rigid structure, that limits us from making certain changes like redefining what should be modeled as a plugin or what to model as a @kbn package. A lot of times these decisions end up being made for pragmatic reasons; for example, something becomes a @kbn package so it can be statically imported everywhere, including in developer tools or tests. These decisions require that the code be placed in specific folders, and incur other trade-offs. Short-term, a majority of code will still be modeled as a plugin. However, long-term, I'd like to eliminate the differences between something being a Make exploring the codebase simpler and improve the discoverability of shared services and other foundational building blocksThe domain-oriented hierarchy should make it much easier to develop a mental model about Kibana's architecture. Grouping all of the plugins which contain shared-services as part of the Where-as grouping all plugins for a specific Solution within the same top-level domain will make it obvious which plugins are Solution-specific, and should only be used from within that Solution. When a "component" is large-enough and isolated from everything else, it should be a "top-level domain", for example Alerting. If there are a bunch of smaller "components" which are all rather isolated, but conceptually are for a similar purpose, they should be grouped under another "top-level domain", for example all of the CLIs. Group related code in the same area of the codebaseIdeally, all code with a sub-domain should exhibit high-cohesion. There are many different types of cohesion; however, a domain-oriented hierarchy like the one that's proposed in this issue allows us the freedom to structure our modules and plugins in this manner. This does depend on us getting the hierarchy right, and there will likely need to be some adjustments made over time. Promote encapsulation between modules and reduced coupling across the codebaseThis is primarily a long-term goal, which we won't necessarily realize by just putting the plugins into a folder-hierarchy. Long-term, we should provide more control over which plugin APIs and static-code should be accessible publicly vs internal to a sub-domain. Exposing public APIs comes at a maintenance cost for both the "producer" and the "consumer". Whenever a public API changes, it either has to be done in a backward-compatible manner, which requires some forethought from the producer; or it is a breaking change, and requires changes to all consumers. Therefore, from the maintenance perspective, it's generally best to minimize what has to be a public API. Currently, this encapsulation is generally done using a plugin. However, as soon as a plugin exposes a method on its setup or start contracts it becomes public and any other plugin can then depend on it. This comes at a disadvantage in situations where certain plugins are highly integrated with each and rely on different APIs than other types of consumers. Therefore, plugins that would benefit from having internal plugin APIs should be grouped together within the domain hierarchy. This way once we figure out the exact method of facilitating this access controls between plugins in sub-domains, it can be done non-invasively. Pave the way for a new, modular toolchain based on distinct NPM packagesAt this time, it's very difficult to implement a modular build. The lack of isolation between plugins and the frequent static code sharing makes this incredibly challenging. We've used parallelization as much as possible to make the build fast, but at some point, this just doesn't work any longer. If we get the isolation between the various domains and sub-domains right, we should commonly see the code within a subset of the domain hierarchy changing at the same time when implementing a feature. This will allow the future modular build toolchain to only have to rebuild the parts of the code that changed and improve the developer experience. Nothing in the proposed domain-hierarchy jumps out at me as something that changes in complete isolation, or would frequently change at the same time as another subdomain. However, there's likely something that I'm overlooking. Getting the domain-hierarchy right from the start isn't a necessity, and we can figure it out over-time. |
|
Thanks for sharing your thoughts @joshdover and @kobelb! This is great context and I love this direction. I spoke with @alisonelizabeth and we consolidated our thoughts a bit. Analysis/explanation of
|
I believe the answer here is that you shouldn't. Code shouldn't be shared before it's contributed to the domain it belongs in. If the domain owners don't have the time to support and maintain the code, then the code should be kept private. There are two aspects here:
It seems the situation with es_ui_shared was a mixture of both: the services are considered "beta" or "experimental", and they were built by a team that probably shouldn't be the final owners and maintainers of such a library. We don't have a process for 1, though I do think it will be important to establish one. The process for 2 involves cooperation between the two teams, and if the domain owners can't take it on, then the code should be kept private. I think this is covered inside https://github.com/elastic/engineering/blob/master/kibana_dev_principles.md#dont-share-code-prematurely Public APIs have overhead to maintain, and risks to consume. We evaluate third party packages because of these risks. Will bugs be addressed? How many maintainers of the package are there? How many users? What are the chances this package will stop being maintained? Even if the code is great, our public services should have enough people to support and maintain it to give consumers confidence in those answers. |
|
@stacey-gammon I agree with you. I think we're all generally aligned with these principles. At the same time, I think we need some processes, conventions, or tooling to support them. For example, what's the right way to share code between two teams while preventing a third team from also deciding to try it out? People want to do the right thing, and there are things we can do to help them. Even if's as simple as a convention like having a consistently labeled "Shared code" section in a plugin's or domain's root-level README. I'm conscious that we're wandering off the original topic of this issue though -- let me know if you'd like me to create a new one to keep the topics separate. |
I think what I'm trying to get at is, there isn't. You shouldn't be sharing code between two domains, unless it's the prescribed, supported solution, owned by the appropriate domain owners. I'm saying "domain" instead of team, because if two teams are working within a single domain, we want to support sharing it. I do agree we need some tooling for sharing code within a domain, or subdomain, without sharing it publicly. This "organize by domain" folder structure should help us, if we can come up with a way to let code be exported from a plugin but not a domain. @joshdover - is there an issue for this yet? As we continue to grow, we'll continue to reflect and iterate on our tooling and recommendations. Currently the stance is that all code shared from a plugin is public. Now we are seeing, based on how often very small plugins are created, we should support code that is "public only within a domain". The original thinking was that we'd have a handful of large plugins that covered domains, each which contained various nested services. As it turned out, that just isn't how developers are using the system. Instead, we are seeing many, very small plugins. We reflected, and now are iterating. I hope that the |
|
Thanks @stacey-gammon. You answered my comment #72034 (comment).
If we take the form lib for example. There might be 2/3 teams doing many forms that would benefit from it. It seems that there will never be an appropriate domain owner as it does not fall in the "useful for all" category. If I hear you well, you are suggesting in this concrete scenario to copy/paste the whole form lib to the other team and let them maintain it (and have 2 copies of it in the repo)? IMO there might be some middle ground: the way we are currently working with the Siem team. They are consuming the form lib but don't request any changes. If they'd like an improvement, they suggest it but are not blocked by it and let us plan when/if we address it. At the same time, if they find a bug, they communicate it to us and that's awesome because it might be a bug we hadn't noticed. I do agree though that this type of collaboration does require a stable API, tests and docs, which in this example we did not fully follow (I admit). |
Since we are already in this situation with the forms library, and multiple teams are using it, I think it's okay to leave exposed and owners as the elasticsearch team (all of this is about creating recommendations, not rules). I see three better spots for it instead of es-ui-shared:
I leave up to you and the App Arch team to decide what to do with it ultimately.
I think App Arch would be the clear domain owner of this. App Arch services and utilities don't have to be "useful for all" to be something they want to take on. The heuristic, imo, should be something like "Will maintaining and supporting this library improve Kibana developer experience and efficiency enough to offset the costs, while taking into account other priorities?". The answer would take into account:
|
|
@cjcenizal I think the domains you came up with make more sense to me now. And +1 to using cohesion as a strong signal that components should belong to the same domain. As for For If there are no objections, I will update the top-level description to reflect the new Elasticsearch domains CJ outlined above.
No issue, but I'm not sure there needs to be. The way domains expose code to other domains is no different than other NPM modules. They should define an
These suggestions regarding the forms lib make sense to me as well. Let's get that code in the right long-term home and iterate from there. If the team does not have the bandwidth to accomdate all requests for changes from other teams, then other teams should either feel empowered to make the changes themselves (which introduces the danger of the forms lib becoming incoherent) or to make a copy ("fork") of the lib inside their own plugin. Obviously having many copies can get out of hand, but there are downsides to trying to make a one-size-fits-all solution as well. Making practical decisions is the goal here, not to have a universal rule. |
|
Love this initiative, will be really helpful in improving developer experience. Also big ++ to moving integration/functional tests at domain level. It's really hard to naviagte around code in current structure. |
|
Great initiative! I'd like to add another example similar to Maybe we can think of ways how team responsibilities, code ownership and expected target audiences of code can be part of the concept. Some very rough thoughts about the previously mentioned code sharing between plugins:
Hope these thoughts make sense and are useful 😅 |
@joshdover Somehow in the discussion so far I had overlooked this implication. So this means that the top-level directory of each domain would represent an API boundary for sharing static code, e.g.: // src/platform/public/index.ts
export * from '../core/public';
export * from '../data/public';
...etc
// somewhere in my plugin
✅ import { CoreStart, RandomThingFromDataPlugin } from 'src/platform/public';
❌ import { RandomThingFromKibanaUtils } from 'src/platform/kibana_utils/public';I think there a few questions we'd need to answer around how this is implemented in practice, however that's probably a discussion for a different thread. |
I think this pattern is a separate consideration, but this proposal definitely is designed with an eye towards improving module encapsulation. Having clearer boundaries between modules & domains would definitely help and I think linting that would make sense. It may also be needed to migrate to the new Bazel toolchain. If we decided to go down this route, I think we'd probably need to introduce this gradually and it will most likely be a forcing function to refactor some modules to better fit with this pattern. IMO, those cases are probably code smells anyways and this exercise would probably improve the overall quality of the codebase. |
|
@joshdover the Kibana Telemetry team had planned to start grouping our plugins into a structure proposed in the description of this issue but hadn't considered moving the combined grouping to |
Let's wait for us to decide this issue first to avoid any unnecessary churn in the codebase. That said, I'd like to bring this proposal to a "final comment period" of sorts. It seems like we have a consensus on this general direction and the high-level organization of these domains. If anyone has any fundamental flaws or concerns with this plan, please raise them now. If no fundamental flaws are raised by EOD Friday September 4th, I propose we close this issue and begin planning the work to make this a reality. |
|
@joshdover thanks for the info.
|
I think we should put these under EDIT: I updated the structure above to include these |
|
With no objections, I'm closing this issue. The Platform team will coordinate an execution plan in a separate follow up issue. |
This issue is to discuss moving the Kibana repository to domain-based file directory structure.
Goals
Non-goals
Proposal
Currently, the bulk of Kibana's source code is organized in a flat structure inside the
src/pluginsandx-pack/pluginsdirectories, where each directory is a plugin. Building on the capabilities added in #68811, I propose that we create a new, hierarchical directory structure that is domain-centric rather than layer-centric. If today's structure describes what layer of code (eg. a plugin) is contained in a directory, this new structure would describe what domain of responsibility (eg. alerting) is contained in a directory.This proposal involves:
pluginsdirectoriessrcdirectoryThe domains:
application_servicesis a domain of services used to build other applicationscoreis a domain of services that define or significantly augment how Kibana worksalertingappears complex enough to be its own domain. It is both an application service and an application itselfmanagementappears to be similar toalertingin that it is a service and an application, though this could probably be moved intoapplication_servicesif desiredProposed structure
dev_toolsplugin)homeplugin)managementplugin)Execution
Once we have a consensus on a general structure, we should start moving to this incrementally. I propose that we start with the
application_servicesdomains. We can use this first domain to figure out any kinks in moving that much code and use what we learn to tackle the next domains.We can then delegate the domains to individual teams to make the final moves.
Outstanding questions
no-restricted-pathsESLint rule to be compatible with this new structure?The text was updated successfully, but these errors were encountered: