[Design] De-embedding ALUR and Friends

[Mornix]

Rationale

Drasil knows too much about AssumpChunk, ReqChunk, and Changes. This propagates through most of the Language. This should be abstracted away.

Existing Implementation

Introduction

This proposal is considering three different chunks: AssumpChunk, ReqChunk, and Change(Chunk), with primary design being driven by AssumpChunk.

AssumpChunk Definition

data AssumpChunk = AC 
    { _aid :: String
    , assuming :: Sentence
    , _refName :: RefName -- HACK for refs?. No spaces/special chars allowed
    , _attribs :: Attributes
    }

Where aid becomes the uid for HasUID.
assuming is the assumption.
refName is used to allow "links" to the assumption.
attribs is used to store a list of Attributes. This field is going to be largely ignored in its current state because of its removal (#537), and instead shortname will be considered as that is all attribs is used for in these chunks.
shortname is the displayed text that is hyperlinked in the resulting document.

Similarities

Both ReqChunk and Change contain similar fields as AssumpChunk, however both add a "switch" to toggle functional/nonfunctional and likely/unlikely respectively.

Field Uses

To make this cleaner, the "switch" fields are going to be ignored from ReqChunk and Changes, and common names will be introduced for the remaining fields for the rest of this document.

Fields

id = aid (AssumpChunk), id (ReqChunk), id (Changes)
body = assuming (AssumpChunk), requires (ReqChunk), chng (Changes)
refName is named the same across all three.
shortname is an adaptation of Attributes across all three.

High Level Use

id is used both to implement HasUID and equality of chunks.
body stores a Sentence which becomes the "body" of the generated output.
refName becomes the hyperlink the backends use to reference a given chunk. There are restrictions on this field as indicated by the comment in the AssumpChunk definition above.
shortname becomes the text which refName will hyperlink.

Use Observations

AssumpChunk

id and refName seem to always be instantiated to the same value.
DocumentLanguage wraps assump as mkAssump such that refName and shortname are instantiated to the same value (as of this commit).

ReqChunk

id is "some" unique value. (ex. req1, req2, ..., reqn)
DocumentLanguage wraps frc constructor as mkRequirements which makes the refName field the same as the shortname.

Changes

id is "some" unique value. (ex. likeChg1, likeChg2, ..., likeChgn)
Document language wraps lc as mkLklyChnk such that shortname and refName are the same value.

Embedding Propagation

Where does this chunk embedding propagate to in the Language?

1. ReferenceDB
   Each different chunk "type" (ie. AssunpChunk, ReqChunk, and Changes) becomes its own field in ReferenceDB.
2. Document
   Each one of the chunks has its own constructor in Document
3. LayoutObj
   The ALUR constructor for LayoutObj is a hack and is how these chunks eventually end up being rendered.

De-embedding Proposal

With all the uses enumerated, it is possible to see how the chunks are used and how these chunks can be de-embedded from the backend.

Backend Data Structure

Based on the three chunks, and how they're instantiated and used, a "new" generic chunk could be created as:

data ReferableChunk = 
	{ body :: Sentence
	, sn :: Maybe String
	}

body is carried over from the old chunks, and holds the bulk of the information associated with a requirement, assumption, change, or really any "list" item.
sn is the shortname referred to above. If sn is Nothing we will see how the hyperlink text is generated below.

Where are id and refName?

• id, as seen from the uses mentioned above, is not important in what its value is, only that it is unique. This could be auto-generated or derived from shortname (which I'm assuming is unique, please correct me if my analysis is wrong.)
• refName this is an odd field as it is the same as shortname but places restrictions on its value such that it is compatible for linking by the TeX and HTML backends. I believe refName is a derivable value from shortname as well, and each backend should perform the transformation to an acceptable character set allowed for hyperlinks. It should not be the responsibility (or care) of the user that the field is an acceptable character set.

Example Use

The examples as they exist, could be easily converted to use ReferableChunk with the existing information.

Document Use

The simplest implementation is groups of ReferableChunks are wrapped in Enumerations with ListType Desc.

An even better implementation is to add a new ListType which is a hybrid between Desc and Numeric called something such as FormattedNumeric.
FormattedNumeric with fields prefix :: String and suffix :: String such that instantiation may look like (in Haskell-ish pseudocode):
Enumeration (FormattedNumeric "A" "") ....
Resulting in assumptions being formatted as

A1: <Desc of A1>
A2: <Desc of A2>

This would allow backends to use existing infrastructure for counting (and abstracting the need to explicitly count elements in the Enumeration as with Desc) while being more complex than just a simple list and allowing formatting around the numeric value.

This FormattedNumeric ListType can also acts as a shortname iff sn is Nothing as long as Enumeration are not directly used, but instead Document gains a constructor RefChunkGroup [ReferableChunk] String String instead. If this new constructor is used there would be an intermediate step so all chunks have their shortname known before separating them and transforming them into just an Enumeration. This approach would allow for all ALUR type chunks to be reference-able by default and maintain a meaningful name regardless if one is specified.

Backend Generation

This proposal aids in #541 as it largely removes the need for ALUR. Ideally the ReferableChunk's are rendered using existing LayoutObjinfrastructure. The only change to the backend, is performing an as-needed transformation onshortname` into something that is

1. A valid hyperlink for the given backend
2. Still unique

Implementation Order

1. ReferableChunk
2. Update Document and friends to handle ReferableChunk
3. Update TeX and HTML backend to generate hyperlinks from shortname of ReferableChunk
4. Update ReferenceDB to handle ReferableChunk
5. Plug it all in
6. Remove ALUR and friends

Alternatives

1. Don't autogenerate shortname if not explicitly given. Instead raise an error if it is referenced.
   This option was considered, however a lot of the caseStudies use reference labels such as A1 which is something that can't be autogenerated in a reliable way.
2. Reintroduce refName in ReferableChunk but heavily restrict the character set. Make that the uid instead of shortname.
   The reason this was not chosen was the hyperlink largely "doesn't matter". As long as it is unique for the generated document and a valid hyperlink. shortname implements uniqueness and the backend implements transformation to acceptable hyperlink (and a second uniqueness check on the transformed shortname).

Conclusion

This proposal seems to cover all aspects needed to de-embed ALUR chunks from Drasil while retaining and improving their functionality. While I have tried to be thorough with analyzing how ALUR components are embedded, used, and how they should be used (caseStudies), I may have missed something along the way that may affect this proposal.

As always any feedback, suggestions, or criticism on incorrect choices, bad design, or anything I've missed is welcome.

[JacquesCarette]

A lot of this is great. But you'll need to do another pass, as there are a few misconceptions, which then leads in a weird direction. Without further ado:

• we want to store all of our chunks in databases (different 'tables' in the DB will have different kinds of information). For that we need a primary key -- and that is always the uid. So that is always needed. Furthermore, uids are always supposed to be unique.
• these 3 chunks really convey 5 different 'meanings': assumption, functional requirement, non-functional requirement, likely change, unlikely change.
• those 5 meanings are quite important! So whatever Sentence is the actual instance (of an assumption, likely change, etc) needs to be 'tagged' with this extra meaning.
• the point of de-embedding is to allow this 'extra meaning' to be data (and eventually live in Data.Drasil) that can be changed/extensible/etc.
• refName is indeed currently used the name of the hyperlink, while shortname is the user-facing version of the same. So refName does need to be unique-to-the-document -- but not globally unique. shortname does not even necessarily need to be unique.
• because right now ALUR exists, it means that each of the 5 meanings could, in theory, be displayed differently. That is probably overkill. But we don't want all 5 to have to be formatted identically. So we do need to have access to some of the 'meaning' information at formatting time; or at least, we need some kind of token on which the formatter can dispatch.
• so the backend-datastructure will need that id (it can't be derived). It may actually need some additional information that will get turned into shortname and refName.
• it most importantly needs to be tagged with the 'meaning'. This is the part that really needs design.
• I like your ideas around ListType and FormattedNumeric. I think the model here should be somewhat like XML and CSS. The Referable is like a fairly generic XML tag. But it could have a 'class', and the formatting can trigger on the class. (Think of the concept 'Assumption' as being such a class).

[szymczdm]

Well, there go all the nicely formatted comments I was writing up.

[JacquesCarette]

I'm sure some of them make different points!

[szymczdm]

Only thing I have to add to @JacquesCarette's comments is that the FormattedNumeric list is a great idea, but if we're not keeping track (somehow) of which number gets assigned to which assumption, ref, etc. we lose some of the traceability. This can be easily remedied (as mentioned below), but it requires that we can differentiate the meanings of these chunks (ex. is it a Functional requirement or assumption?)

If you look at the RefDB stuff, you'll see that the assumptions, requirements, etc. are currently stored as pairs that keep track of the chunk and its numeric identifier. Then we can always have the option of creating references that look like "See A1 for more details" or "SomeActualAssumption for more details".

Also, the current implementation is by no means great, since (I believe) it's got an implicit ordering that must be used for creating both the refDB entry and the associated list. So, for example, if we had:

ourAssumps = [a1, a2, a3]

then we need to explicitly pass ourAssumps into both the refDB constructor and the assumption list constructor. If we don't, the order could be changed. This could be solved by automating the refDB entry based on the list construction (or something along those lines of automatically grabbing them, ordering them, and keeping track of the order).

[Mornix]

Thanks both of you for the feedback.

I was hoping to have something more concrete worked out by now, but I haven’t traced through everything or worked out all the details just yet, so this is more informal rather than an update to the original proposal (for the time being). Thought I’d post an update so if there’s any issue I can correct course before I finish working out all the details.

With the feedback and the introduction of Label the structure would be:

data ReferableChunk = 
	{ id: String
	, body :: Sentence
	, labeluid: String
	, semuid: String
	}

labeluid is the uid of the new Label type as discussed in the May 29th Drasil meeting.
semuid is the uid of a new type, ChunkSemantics, which contains the semantic information currently embedded in the individual chunk types.

At the moment, I haven’t fully worked out what belongs in ChunkSemantics or how it will encode that information, but @JacquesCarette has provided good suggestions of including formatting and styling as a part of this.
ChunkSemantics is assuming the formatting and styling of all Assumptions, or Changes will be the same (ie. be consistent).

These ChunkSemantics are stored in a DB to perform lookup as needed.
As for the ReferableChunks I notice that each type of chunk (as they currently exist) is in its own list in the RefDB. I’m assuming that this should be the case with ReferableChunks being in different lists grouped by ChunkSemantics uid so it is still possible to lookup a chunk by number as is currently an option pointed out by @szymczdm.

I also liked @szymczdm suggestion of automating the refDB entry as well to help ensure consistency.
My thought for that is (in pseudocode)

assumps = [assump1, assump2, ...]
assumpgroup = group (ChunkSemantics ...) assumps

assumpn is a ReferableChunk with ChunkSemantic uid being empty.
group is not a great name and attaches the uid of the new ChunkSemanticsto each assumption and groups them in a RefDB.

All feedback is welcome.

[JacquesCarette]

Thinking of this as a ReferableChunk isn't quite right. It is referable, but that is not the 'meaning'. Really what we need is something closer to Concept, NamedIdea, etc. The point is that, for now, things like assumptions are just "ideas" (which I use here informally) which are tagged as being in a particular "class" (again, informal). The 'payload' of an assumption is a Sentence; as it is an "idea", it has an id; as we will always refer to these, it needs to be referable, so needs a labeluid; and because it is tagged, it needs to be able to refer to that tag.

We desperately need to introduce a UID type synonym in Drasil, and stop using String directly. We could go as far as making UID a newtype wrapper, to stop people doing String manipulate (outside the facilities provided by some module meant for just that).

So the above makes me think of ConceptChunk and its dom. The idea is the same: the "domain" of a Concept is a (set of) classifiers [think 'type'].

In other words, I think an assumption could be "just" a referable Concept with dom being itself the Concept "assumption" (which itself might have a domain that classifies it as being a concept from the Requirements domain). [I also note that the implementation of dom is wrong, see below]

So I think this discussion bring out the following things that need done (first!):

• introduce UID as (at least) a type synonym (probably best first step) and try to remove String from more of Language.Drasil.
• change the domain implementation for ConceptChunk from being a list of ConceptChunks (which introduces all sorts of difficulties because of the recursive nature of the definition), to being a list of UIDs of what should be verified to be Concepts.
• make sure that ConceptChunk can be referred to.

[Mornix]

Thanks for the feedback.

I’m going to add myself to the assignees of this issue and start working on those first steps mentioned now. I’ll also work to update the proposal accordingly.

[Mornix]

Edited June 27th 2018 to be more accurate based on what was discussed during the Drasil meeting on June 26th, 2018

Preface: Prior to the below proposal, Maybe ShortName/Maybe Label should be removed from ConceptChunk. Instead a chunk very similar to ConceptChunk but more concrete should be introduced, call this chunk, ConceptInstance. This new ConceptInstance has a Label, no Maybe.

Now that most of the infrastructure is present to build ConceptChunks as outlined by comment 6, the next three things to look at are:

1. How should the examples and document language use ConceptChunk and ConceptInstance to replace the more targeted chunks? (Or use chunks for the first time!)
2. How should document language help generate Contents and Sections based on the Concept(Chunk|Instance) "trees"?
3. What should replace ALUR in the backend to facilitate generalization?

Replacing *Chunks with ConceptChunk and ConceptInstance

At this point, I see the SRS domain gaining the following ConceptChunks (in a loose sense):

1. A root CommonConcept:
   (A CommonConcept because I don't think of this as a Chunk which will appear in the output. I think of this more as a needed root node to associate the immediate descendants.)

• noun phrase (np) = "software requirements specification"
• abbreviation (abbr) = "SRS"
• definition (defn) = "" (I don't see this field being used at the moment and haven't thought about what the definition of this Concept is, but I am welcome to suggestions on what the definition should be.)
• domain (dom) = empty

2. Root ACR (Assumptions, Changes, Requirements) "categories" as ConceptChunks.

• np = "requirements"
• abbr = "R"
• defn = The definition that goes immediately following the heading before any lists of descendants (Assumptions) or more specific categories (Changes, Requirements). For example for requirements this may be "This section provides the functional requirements, the business tasks that the software is expected to complete, and the non-functional requirements, the qualities that the software is expected to exhibit."
• dom = The root SRS CommonConcept.

3. Deep CR "categories" as ConceptChunks.
   (Likely/Unlikely, Functional/Nonfunctional)

• np = likely changes"
• abbr = "LC"
• defn = ""
• dom = The "Change" ConceptChunk.

And finally individual requirements, assumptions, or changes would have these deep(est) ConceptChunks as their domain and be ConceptInstances.

Many of the needed ideas to back these concepts are already defined in Data.Drasil.Concepts. The idea behind the domain trees is to help generate the sections, using NounPhrase as the title of the section (or subsection). The abbr field is to be used in tables (like the traceability matrix) to maintain the R1...Rn, LC1...LCn, etc which are currently present.

Contents from Concept(Chunk|Instance) trees.

This is still a bit of a work-in-progress, but it would be convenient to provide as input to a function the general Requirements ConceptChunk and have the tree walked, and Contents and Sections generated from that, with subsections Functional Requirements and Nonfunctional Requirements generated as part of the process.

A signature such as
mkSecCC :: Concept ->[SecCons] -> (Concept -> SecCons) -> (Concept -> Contents) -> [SecCons] -> Section

Where:

• Concept is a Concept such as the "broad" "category" requirements.
• first list of SecCons is the Sections and Contents prepended to the resulting Section.
• the first function is for intermediate nodes between the Concept passed in, and any leaf node Concepts.
• The second function formats the leaf nodes as Contents.
• The second list of SecCons is anything that should be appended to the resulting Section.

mkConCC :: Concept -> (Concept -> Contents) -> [Contents]

A "convenience" function which takes a Concept, such as that of functional requirements, finds all immediate leaf nodes and calls the second argument over each leaf Concept.

To help facilitate this, the Contents type should remove the Assumption, Requirements, Changes constructors and add support for referencing individual items in Enumeration.

Replacing LayoutObj's ALUR

Replacing ALUR would simply be updating LayoutObj's List constructor to handle individual item references, removing ALUR, and using the new infrastructure present with referable lists. Referable lists in LayoutObj allows for both output of Concept(Chunk|Instance)s similar to the *Chunks used now, and helps facilitate the goal of #541 to improve the renderer's generated document code.

[JacquesCarette]

Read this. Digesting it. Will hopefully comment on it later today.

[Mornix]

@JacquesCarette Comment 8 updated.

[JacquesCarette]

Yes, I think this is close enough to start implementing. Certainly the parts having to do with Concept(Chunk|Instance) and the 3-level hierarchy you present.

As far as replacing ALUR, what I would actually do would be first implement a simple form of a replacement (without taking out ALUR), and then try to use this to change one of the smallest examples. This is more than likely to shake out a few issues [without breaking everything]. For example, I have some doubts about mkSecCC. In general, I agree, as it does go along the right 'spirit' of considering the examples as first pulling together a whole lot of knowledge into a "database" of sorts (i.e. the system information data-structure), and then using various 'document reciples' that go from this database to a generic layout-oriented DSL (aka LayoutObj), which can then be rendered in multiple concrete display languages.

Main things:

• I think the design is good and you can start to implement it
• Be very careful to implement everything incrementally, which might require you to implement new features before deprecating the old features.
• I would be quite happy to merge in each small piece of the implementation incrementally as well.

[JacquesCarette]

Yes, I think so.

[Mornix]

Yesterday during the Drasil meeting, @szymczdm and I talked about adding referencing to individual items in enumerations. We concluded that Labels should be a Maybe at the per-item level and if use patterns (like a particular enumeration being completely labelled or completely unlabelled) emerged, changes could be made then.

[Mornix]

Commit 62939f7 contains a demo of ConceptInstances to define the functional requirements for NoPCM. The extended commit message explains the reasoning between the slight differences observed in the screenshots.

Below are screenshots comparing the HTML output of the old requirements and the new ConceptInstance ones:

New

Old

[JacquesCarette]

I actually sort of prefer the New ordering of information. I definitely dislike that the reason it had to change is because of (hopefully temporary) weaknesses in the infrastructure.

[Mornix]

This is a small proposal to add support for prefixes to ShortNames when ConceptInstances are referenced.

Problem

When ConceptInstances are referenced through a function, such as, makeRef, there is no means to lookup or generate the correct prefix for the reference at that moment.

Context

ConceptInstance as it stands

Currently, ConceptInstance contains a UID which “points” to a domain. The domains themselves are not stored and thus cannot be looked up at the moment. Domain lookups can be easily remedied by including them in ChunkDB.

Once domains are tracked, the issue still remains that a ChunkDB is not available at the moment of referencing.

The Proposed Solution

While it’s not possible to resolve a domain at the moment of referencing, the ShortName type is flexible enough, and currently mentions, it should not be thought of as a String. This provides the necessary means to introduce two new constructors to the type:


FromChunk :: UID -> ShortName
Prefix :: ShortName -> ShortName -> ShortName

FromChunk provides the means to store the needed information at referencing time to later retrieve the needed information from the domain.

Prefix compliments FromChunk by taking two ShortNames and making the first the prefix of the second.

The two suggested constructors would turn the structure of a ShortName for ConceptInstance into something such as
(Prefix (FromChunk “funcReqDom”) (r ^. shortname))

The resolution would be performed in the drasil-printers package spec function.

Benefits

The benefit of this method is the design maintains a clear separation of concerns by not pollution either ConceptInstance with a new field or the referencing logic by necessitating a ChunkDB.

This design also provides means to later expose as a printing parameter the ability to toggle prefixes on-and-off.

Drawbacks

This proposal has two major drawbacks as I see it:

1. RefType requires a form of “hack” to disable the prefixing by customRef.
2. The constraint HasDefTable would be needed for a few functions in printers’ import.

Alternatives

I can see two alternatives at the moment:

1. Generate the prefix from cic ConceptInstance constructor because the immediate parent domain is completely present.

I don’t like the idea of this because it seems “too early” to decide what formatting would be applied to the reference stage. This also has the drawback of including the prefix even in the enumeration or requiring an additional field.

2. Add a makeRef-like constructor which requires a ChunkDB or some form of domain when referencing.

This option feels like it is leaking implementation details of ConceptInstance while also adding ambiguity and special casing to the user-exposed referencing logic.

Any feedback is appreciated.

/cc #970 #975

[JacquesCarette]

I like this a lot. [Except that I would rename Prefix to Concat]. And the issue #976 would also be covered by this. And I think #971 too.

I think we'll eventually do away with RefType, so I think that hack will disappear. I don't think the second drawback is a big deal.

[samm82]

And apparently #470 as well

[JacquesCarette]

This issue is getting ridiculously long. [Though it has lots of good stuff in it!] Maybe we should think of documenting the parts that still need to be done in new issues and close this one? It feels like the bulk of the work, not just the design, has been done and merged in.

[Mornix]

Agreed. Tracking what's remaining in #985.