[WIP] What: Cloud Native Guide Posts Issue #961

[TheFoxAtWork]

Why:

• Capture milestones from successful projects
• provide guidance and recommendations to projects seeking graduation
• Provide criteria to guide TOC on determining when sandbox projects should depart from sandbox

This change addresses the need by:

• Modifying archiving.md to reflect Sandbox Departure link
• Modify sandbox.md to set Sandbox Departure criteria
• Modify README.md to capture Milestones link
• create project milestones for community review, improvement, and feedback

Signed-off-by: Emily Fox themoxiefoxatwork@gmail.com

[xmulligan]

Suggested change

	* Project has a clearly discoverable governance doc that covers the basics of decision making to include the commit bit
	* Project has a clearly discoverable governance doc that covers the basics of decision making and how to earn the commit bit

[craigbox]

"the commit bit" is probably not a well understood term. I don't know if the CNCF has a defined term ("maintainer" is definitely overloaded), but I think "how to earn permission to approve pull requests" or something similar would help.

(I assume the SCM for CNCF projects does not have to include GitHub and a more inclusive terms than Pull Requests might be appropriate)

[lizrice]

I've been thinking about Roadmaps. In the past I've been in favour of them as a requirement but I think I'm changing my mind because I'm not convinced they are as valuable as we'd like to think.

Background / food for thought: the Linux kernel does not have a published roadmap. Everyone involved knows what's going through mailing list discussions.

When we say things like a roadmap that's aligned with releases, combined with a regular cadence of releases, this feels dangerously close to requiring open source projects to make commitments that end users might come to think they can rely on.

And it doesn't seem very aligned with the ethos of open source to require developers to sign up to fixed commitments. Sure, in our community most people are being paid to work on projects but not everyone is. And if someone from Company A makes a commitment to deliver a particular feature in a certain release, but then Company A decides they should spend their time elsewhere, who from the project is supposed to pick up the slack to make sure the commitment is reached?

It might work for very large projects, but the more I think about it, the more it seems like a recipe for burnout amongst contributors, and disappointment amongst end users.

Projects generally use issues to communicate what features and fixes people are working on. A feature that meets some set of criteria (up to the project) makes it into the next release. Isn't that sufficient?

And if we require more, isn't that rather overstepping the idea that we don't impose any particular governance on projects?

[monadic]

+100 - we really need to stop turning CNCF into a giant bureaucracy that treats all projects the same. This kills innovation stone dead and will put so many people off. Also, please stop imposing requirements on project maintainers that add a lot of work and will lead to burnout, without a very strong compensating reason and balance

[TheFoxAtWork]

@lizrice Thanks for bringing this up, I dont disagree. We currently have roadmaps integrated through various portions of the lifecycle so we'll need to evaluate what the value is of these in both understanding the project's longer goals as well as the direction they are heading, however they are always point in time for evaluations along the lifecycle.

Your point on perceived commitment is spot on as well, and I'm not confident "messaging" here will resolve this disconnect. We'll need to consider more about how we can view the direction of the project at an evaluation, without dictating how the project expresses that.

[TheFoxAtWork]

@monadic Thank you for the feedback. This PR on milestones does not establish new or additional requirements on projects, it is designed to serve as guiding points along their [project's] journey — from sandbox to beyond graduation which they may leverage. However, as @craigbox mentioned later, these may be seen as requirements. This is something will need to rework in this PR to avoid.
Regarding burnout, agreed. The CNCF lifecycle is structured to evaluate projects for stability, maturity, and adoption potential (to varying degrees) over the years, as we've learned more about project success, we're beginning to distill steps projects have taken along their path to success. Not as requirements, more as a catalog of what has and hasnt worked and more context around why.

Would providing more reasoning/clarity/what-it-does assist with the milestones? Such that as a project maintainer looking to move to incubation from sandbox (for instance) has ample information about how these may be beneficial to decide if documenting a roadmap is worthwhile for their project, or perhaps including a vision/direction, or neither, works for their needs?

[jberkus]

I've suggested different language for this point.

[lizrice]

Preparing for graduation, we (Cilium) have been getting recommendations and requirements to do a lot of security-related audits and assessments:

• 3rd party security audit - clear graduation requirement
• Open SSF (aka Core Infrastructure) best practices badge - clear graduation requirement
• TAG Security review self assessment and/or joint review - not documented requirements. Seems a duplication of effort for the project security folks who are already assisting the 3rd party audit.
• CLOmonitor badge - includes some security-related checks, recommended by CNCF
• OpenSSF scorecard - recommended by 3rd party audit, overlaps with CLOmonitor
• SLSA assessment - recommended by 3rd party audit

They’re all good things on their own, but taken together they’re repetitive and overlapping, and are causing quite a lot of frustration ("we're drowning in checklists and paperwork".) It would be really helpful to have a clear steer on which of these are actually required.

[TheFoxAtWork]

There's a lot here we can get better clarity on.

The TAG Security Review/Assessment process was heavily reworked a few years ago to structure it such that projects may complete portions of the overall "Security Package" aligned with where they are at in the CNCF "lifecycle". This revision was designed to prevent the security review (which often doesn't have enough reviewers and as a result does take longer) from becoming a bottleneck for a project pursuing graduation. If projects completed the "self-assessment" during Sandbox, then optional "joint-review" during Incubation (well before considering graduation) these two security documents form a security package that can be leveraged by Security Auditors to assist in the formal audit. The joint-review builds on top of the self-assessment, but both documents do NOT replace a Security Audit, while joint-reviews do have some areas of overlap with Security Audits, the Security Audits go more in depth in other areas. They should be viewed as cumulatively complementary to one another, however a joint-review is NOT a requirement for graduation (nor is the self-assessment for incubation).

I am not comfortable placing TAG Security and the joint-reviews in a position where all projects come to them b/c they are now the executors of a graduation requirement. It's not feasible with the amount of projects in the landscape and the amount of security reviewers we have.

There as a been a lot of evolution in open source and supply chain security these past few years, with increasing maturity of various assessments that look at different portions of a project - from its architecture to its repository configuration and release processes. We've not yet revisited how these changes should be weighed or considered in the CNCF lifecycle and I believe this is worth pursuing by community through the Security TAG.

[tomkerkhove]

Preparing for graduation, we (Cilium) have been getting recommendations and requirements to do a lot of security-related audits and assessments:

• 3rd party security audit - clear graduation requirement
• Open SSF (aka Core Infrastructure) best practices badge - clear graduation requirement
• TAG Security review self assessment and/or joint review - not documented requirements. Seems a duplication of effort for the project security folks who are already assisting the 3rd party audit.
• CLOmonitor badge - includes some security-related checks, recommended by CNCF
• OpenSSF scorecard - recommended by 3rd party audit, overlaps with CLOmonitor
• SLSA assessment - recommended by 3rd party audit

They’re all good things on their own, but taken together they’re repetitive and overlapping, and are causing quite a lot of frustration ("we're drowning in checklists and paperwork".) It would be really helpful to have a clear steer on which of these are actually required.

Agreed

[tomkerkhove]

Some practical things that I think would be nice to include are:

• The fact that new proposals should end up in CNCF-based Google workspace (see Provide CNCF-hosted Google workspace for DD docs #980)
• If we should use issues vs pull requests to open proposals. If we stick with pull requests - What should be in them? Because typically people include a whole proposal in the PR which then gets ignored/must be ported over to the DD doc which costs a lot of time
  • Been through this twice to write a full proposal (which takes a lot of time) only to see it get ignored and need to create a Google doc afterwards

[TheFoxAtWork]

I had an excellent discussion with a community member yesterday about the amount of security asks of projects and the current throughput of Security TAG in reviews, specifically regarding the joint-reviews.
It was recommended that we extract the threat modeling process (when its ready - this is still in development) from the joint review and focus on this for projects immediately after moving to incubation. The threat modeling portion is designed to take up 2-3 hours of a projects time and is likely useful for projects immediately following incubation.

What do others think about this as a milestone?

[dims]

+1 @TheFoxAtWork

[craigbox]

I would caution that even though it might not be the stated intention, these may well end up interpreted as "unwritten criteria" for moving between maturity levels, especially in light of #992.

[TheFoxAtWork]

The fact that new proposals should end up in CNCF-based Google workspace (see #980)

@tomkerkhove this would be best in the graduation criteria/process documentation as this was a recent change. @amye is this something you can ensure is integrated?

Been through this twice to write a full proposal (which takes a lot of time) only to see it get ignored and need to create a Google doc afterwards

Completely agree this is time consuming and inconsistent. @amye whatever updates we make for this should reduce duplicative effort. We the template for graduation with a lot of information in it, but perhaps its worth figuring out where this goes in a DD or a DD refresh versus initiating a process.

[TheFoxAtWork]

Per discussion on today's TOC/TAG Call here is a google doc of the primary changes for better discussion/facilitation of this PR: https://docs.google.com/document/d/1aL0mxInyz-qtRT2QM7kdr9riM36uihILtgM4TR35cPQ/edit?usp=sharing

[lizrice]

Suggested change

	* Project plans activities to increase project adoption trajectory 3 adopters
	* Project plans activities to increase project adoption

[lizrice]

This one worries me A LOT. It clearly incentivizes projects and companies to do the wrong thing, in order to meet these "milestones".

For example, fearing that they are being seen to have "too many" contributions going into the open source project, the major backer of a project decides to hold their engineering efforts back into closed-source add-ons. Or use outsourcing companies and partners rather than hire people to work full-time on the project. Or decide not to contribute a project to the CNCF because it's going to be such an uphill struggle for the project to see itself as successful according to these criteria.

None of these outcomes are good for anyone in our community.

(Also, why arbitrarily 5? And what does "regular" mean? We could spend time debating these points - but that is time that could be better spent on actually progressing / supporting projects.)

[raravena80]

Makes sense, I suggested a slightly different variation of this here: https://docs.google.com/document/d/1aL0mxInyz-qtRT2QM7kdr9riM36uihILtgM4TR35cPQ. I think it's hard to determine a fixed number that can work for all the variety of different projects. IMO a more 'open' or 'flexible' language would work better.

[TheFoxAtWork]

These are good counterpoints @lizrice . We also don't want to have a single entity (company or organization or individual) be the majority decision maker of a projects technical direction, if that entity is acquired or disappears, the project can be left out to dry, and in some case unable to recover because the primary pool of contribution has evaporated.

@raravena80 @lizrice - How do we strike a balance in these milestones — as guiding points projects could leverage to in order to become more resilient, mature, and widely adopted versions — that could help projects avoid the scenarios you present as well as a increase the likelihood of project's survivability with primary entity's complete absence? What actionable goals/ideals/structure/governance could be provided for projects to increase their resilience and survival likelihood in the event a core contributing entity ceases to be or continue?

Ricardo mentioned open and flexible language here, would the following adjustments meet the intent?

Suggested change

	* Project has more than 5 regular contributors, more than half of which are from different organizations
	* Project has multiple contributors who regularly participate in the growth and development of the project in accordance with the projects synchronous and asynchronous cadence
	* Project contribution governance defines timeliness and methods for recovering roles and permissions of contributors that drop from regular participation as _defined by the project_ (roles such as issue triage, community management, PR reviewer, maintainer, release lead, etc. and regularity such as participation in two of three releases, facilitation of 4 community meetups annually, etc.)

[jberkus]

I can't say I'm keen on removing specifics from this. Folks in the community manager role for a project like having clear targets. Once a project gets to applying Graduated, if they don't have at least five regular contributors, the application isn't likely to go well.

Note that these are milestones, not updates to the specific requirements.

[raravena80]

👍 I believe all these can be covered in the March 21st TOC meeting.

[monadic]

Hi, I say this with care and respect for the intentions of the people on this thread. If the CNCF wants to continue to be a home for innovative and leading projects, then it needs to listen to people who work on them and make them successful. In my view, the kind of rules being proposed here will greatly narrow what sorts of projects can thrive in CNCF and make it ultimately unappealing. Please can we arrange a TOC call with appropriate representation to discuss.

-alexis

[TheFoxAtWork]

@monadic thank you for engaging, we can certainly schedule this for a discussion on a TOC call! It is apparent that the guiding points identified here are still being viewed as rules, requirements, or other forms of constraint which is not what they should be used for.

I would love to hear more about your experience to see which, if any, of these describe the path for moving levels of the projects you've been involved in. Scaling that experience and making it approachable for other projects — if they would like to leverage it — would be beneficial for new projects and maybe even existing projects seeking to grow and move levels. When we have a date, I'll update here. Thank you again!

[monadic]

thanks @TheFoxAtWork

this is something we spent a lot of time thinking about when creating CNCF - what makes a successful OSS project? the answer is that they come from different places but a consistent theme is small companies innovating ** and sustaining ** them. this has numerous consequences. I love multipolar projects too.

[TheFoxAtWork]

Looks like this actually already on the Agenda for the meeting next week (march 21st) @monadic (apologies for missing it earlier)

[monadic]

8am PT? please can you advertise on toc & alums email lists

[craigbox]

It is apparent that the guiding points identified here are still being viewed as rules, requirements, or other forms of constraint which is not what they should be used for.

The points under discussion today are being discussed as a criteria for graduation in #992. I've commented at length there.

(I would love to be involved in an in-person discussion, but not enough to get up at 4am I'm afraid.)

[lizrice]

I have some concrete ideas for this milestones document, to de-couple the advice on growing a project from the TOC’s assessment of its maturity.

• Make this document about guidance to projects on how to grow. Some indication of what success looks like could be helpful, but be careful to make it applicable to all projects (so, no arbitrary numbers.)
• Remove all mention of the sandbox / incubation / graduation levels so that people aren’t tempted to draw their own conclusions about whether they should be considered by the TOC when moving levels.
• Remove the points in this document that appear as guidance to the TOC on how to assess projects. Anything the TOC requires from all projects at a particular maturity level should be documented in the criteria, and anything else should be a matter of judgement.

I’m aiming to be there for the discussion coming up in this week’s TOC meeting - looking forward to it.
—

Folks in the community manager role for a project like having clear targets.

As someone with experience on both the TOC and in community management for Cilium, I can tell you that no-one wants yet more checklists where it’s unclear how important they are - our experience with the myriad security assessments advised to us during the course of our graduation application is an example. If something looks like it might be considered, projects feel obliged to document them in their DD just in case it affects their application. And (sadly) people with some axe to grind will bring up the lack of some specific “milestone” during the public comment period. Neither of these outcomes are remotely helpful to the project or the user community - they just create additional work for the projects and for the TOC.

Once a project gets to applying Graduated, if they don't have at least five regular contributors, the application isn't likely to go well.

Most Sandbox projects have at least five regular contributors, so this number isn’t helpful as a target for graduation. What’s the right number? For massive (Kubernetes-esque) projects it’s probably one or two orders higher than this; for a small but critical security library, five might be totally appropriate.

There is no “one size fits all” set of measurements that is appropriate for the needs of all projects. That’s why levelling assessments can’t be boiled down to a set of checkboxes, and it’s why previous incarnations of the TOC have deliberately resisted setting arbitrary thresholds, in favour of exercising judgement on a per-project basis.

[JustinCappos]

I'll also mention a few more boundary cases so they are on everyone's radar. Specifically, that it's hard to assess projects which are more specification focused given guidelines which are more often to assess an implementation.

For example, when considering a specification project, should you count maintainers / approvers for the specification as those controlling entities? Should this also include the implementation maintainers?

A healthy rate of churn for a successful and active specification may only involve a few changes a year.

What if there are different compliant implementations from different groups that individually do not have a high degree of maintainer diversity? Would this be considered reasonable diversity, given one can always pick which group's implementation to use?

[TheFoxAtWork]

Checking in on this PR to determine if any further changes suggested have not yet been addressed. Please take a few moments to review and determine if I've incorporated your changes. Thank you all in advance

[nikhita]

PR has been open for a while. LGTM, let's

Having guide posts will help with evaluating sandbox annual reviews and providing a structured way to help sandbox projects figure out a growth path.

[TheFoxAtWork]

PR needs refreshed to align with recent changes in the TOC repo. Will update later.