Agile approach

by Juanjo Hierro.  Chief Architect of FI-WARE.

Prolog

About Themes, Epics, User Stories and Work Items

Entries in Backlogs used in FI-WARE can be of any of the following types:

• Theme
• Epic
• Feature
• User-Story
• Work Item

The Backlog in a product development project is fundamentally about "functionality that has to be developed" in the product. Seeking for clarity, and because I have found it helpful when explaining Agile concepts, I have explained that entries in the backlog indeed map into "work to be done". Actually, a lot of the work you have to do when developing a software product has to do with developing the software that implements the functionality the users expect from your product and it is the description of functionality to be supported in your product what we intend to capture through Themes, Epics, Features or User-Stories. The rest of the work may be categorized as Work Items (creating specific parts the documentation, solving a support ticket, running an analysis or defining part of the API spec before we actually implement it, etc).

Themes, Epics, Features or User-Stories describe functionality at different level of granularity. Themes describe functionality at a very high level. Themes are further refined into Epics, Features and User-stories. When should I label a given backlog entry as an Epic, a Feature or a User-Story ?

User Stories comply with "INVEST" properties which means they should be "Independent, Negotiable, Valuable, Estimatable, Small and Testable". Some resources that may be helpful in finding out what is meant by INVEST can be found at [1], [2] and even [3] but in general:

• A User Story has to be something sufficiently limited in scope as to be affordable in one Sprint, unitary tests included. ("Small" property)
• A User Story should be detailed enough as to be able to define a test for it. ("Testable" property)
• A User stories should include enough information enabling a developer to make a rough estimation of the resources needed for designing, developing, and testing the functionality within one Sprint ("Estimatable" property).
• There is no need to cover all details nor to have everything finalized before starting actual development. In other words, you shouldn't enter into that level of detail at which you are probably wasting time and unnecessarily delaying development. Just have the details that make you confident that development may fit in the duration of the Sprint. There should be details that may be worked out while developing. ("Negotiable" property)

"Independent" and "Valuable" are also relevant characteristics but I guess there may be many Epics and Features that would fulfill those properties. That's why I would make emphasis on the previous ones.

Therefore, based on the above, you should ask your development teams the following questions to determine whether a new entry you want to insert in the backlog should qualify as an User-Story or not:

• Can you commit to get it developed by the end of the next Sprint ?
• Would you be able to define a test for it ?

If their answer to these questions is "No" or you see the doubt in their eyes … it's not an User-Story.

Note that labeling an entry as User-Story in the backlog depends very much on a fundamental parameter of your project: the duration of a Sprint. In this respect, Sprints are defined to last one month in FI-WARE. What could be labeled as a User-Story would vary a lot if we had defined a different duration for Sprints.

You can consider a Feature just as a special kind of Epic. What Epics can be labeled as Features ? Those describing functionality that you feel confident you will be able to develop in the course of a Product Release. Therefore, labeling an entry as Feature in the backlog depends very much on another fundamental parameter of your project: the duration of a Release. Duration of a Release should be a multiple of the duration of a Sprint and use to be 60 to 120 working days long. In this respect, Releases (also named as minor Releases) in FI-WARE last three months. We have also introduced the concept of Major Releases which last a determined number of minor Releases. Therefore, we use a two digit notation to identify a Release (e.g., Release 1.3)

Note that being able to state that you expect a given functionality described as an Epic to be implemented in your product by the duration of a Release simply means that you have enough information about the functionality as to feel confident that you will be able to refine it to the level of User-Stories which, in turn, can be addressed in development sprints, all during the course of the Release.

Why are we dealing with Features in addition to Epics ? Because it gives hints about what functionality we expect to address within a certain period o time and this is important for the overall management of a large product development project like FI-WARE and helps to share valuable information with customers about our product roadmap. Users need to have some hints on what functionality is going to be available when, so that they can plan their developments better. Other than this, you may think on Features as just Epics you expect to develop before some given date. Of course, Agile is about re-planning the work to be done whenever you feel necessary and priorities change. That means that a Feature you initially planned for Release 2.1 of your product may get delayed and postponed for a later Release.

A Theme, is much more high-level or abstract than an Epic. The frontier between Themes and Epics is fuzzy, but is not that relevant as the frontier between Epics and Features or User Stories. Deriving a Feature allows us to map the corresponding functionality into a product Release. Deriving a User-Story means you can stop refining and start actual development of a concrete functionality in your product. Whenever you derive a new backlog entry, either as result of refining previously existing Epics or Features or just because it captures new functionality, you should ask yourself: "Is the description of this functionality detailed enough as to make it affordable in the course of a Sprint ? Does it own the desired INVEST properties ?". If the answer is "Yes", then you should label it as an User-Story. If not, then you should ask yourself: "Is the description of this functionality detailed enough as to make it affordable in the course of a Release ?". If the answer is "Yes", then you should label it as a Feature. Otherwise, it is an Epic. When you derive a User-Story directly from an Epic, we recommend to create a Feature with the same description. You may think in a first approach that this is unnecessary duplication of entries, but this will allow to keep a complete description of planned features for all Releases, which is important in large products/systems.

Note than a given Epic may be refined into several entries, each linked to a finer-grained description of the target functionality, but still to be considered as an Epic. Last but not least, an Epic may also be refined into a mixture of several Epics, Features and User Stories. This is natural consequence of first deriving the finer-grained entries and then categorize them properly.

A Work item refers to work that needs to be done in the course of a Sprint, trying to meet a concrete goal. It may relate to work to be done in order to refine an existing Epic (or a feature) and derive actual Features (or User-stories) derived from it. In general, any work you need to address to progress development but may be difficult to describe in terms of "functionality supported by the product". Work Items help to report activities performed by your development team you want to report because it consumes a significant amount of resources and you want to leave a trace of that. They are also extremely helpful in monitoring progress at a higher-level, which is something that is always needed in development of a large product/system.

You may derive a mixture of finer-grained Epics plus a list of Features, User-Stories and Work Items when you refine an existing Epic. Similarly, you may obtain finer-grained Features, plus a list of User Stories and Work Items when you refine an existing Feature.

To illustrate this, let's consider a concrete example. You may find out that your system needs to provide some sort of Directory functionality. In a first approach, you may have identified an Epic linked to this functionality. Here you are the description of the Goal linked to that Epic:

As an user I can create/delete/modify entries in a Directory and then retrieve entries using different methods

You may afterwards turn this Epic into three Features A, B and C:

• A = As an user I can create/delete/modify entries in the Directory.
• B = As an user I can discover entries that match some given criteria, expressed in a query language similar to SQL.
• C = As an user I can subscribe to events linked to creation/deletion/modification of entries in the Directory. Those events are received spontaneously, through a callback service I can register linked to my subscription. Subscriptions may have a lifetime, be paused, resumed and canceled.

(note: you should, of course, provide a more detailed description for each of these features in real life, but let's keep it simplified here for convenience)

You may decide that such Features will be exposed through a well defined API. Therefore, all the three Features may get refined into:

• a WorkItem M = Specifying a well-defined REST interface 'I' through which operations linked to features A and B are exposed
• another WorkItem N = Extending specifications of interface 'I' to include operations supporting feature C
• three UserStories O, P and Q corresponding to implementation of the operations in the REST interface 'I' linked to features A, B and C respectively.

When dealing with realization of the backlog through subsequent Sprints, it may happen that you decided to address WorkItem M in Sprint X of the Release 1.1 and then in Sprint X+1, also part of the same Release, you decided to address the implementation of UserStories O and P, leaving WorkItem N and UserStory Q to the Release 1.2. This illustrates a major concept in how Agile works differently than the traditional waterfall models: we are not waiting for finalizing the complete specification of interface 'I' to start implementing part of it.

Creating the first version of the FI-WARE backlogs

In defining how we were going to use Agile in FI-WARE, we have had to deal with a fundamental characteristic of the project: it is not about developing from the scratch but from a set of selected products (assets) resulted from previous projects, many of which hadn't been developed using Agile. If we had started development of every FI-WARE Generic Enabler (that's the way we name components in FI-WARE) from the scratch, the FI-WARE GE backlogs would contain Themes/Epics/Features/User-stories that, all together, would summarize the whole functionality of FI-WARE. But this is not the case.

When creating the backlog for a given FI-WARE GE, we have considered that it should contain the Themes/Epics/Features/User-stories that, at the start of the FI-WARE project, map to "functionality to be developed" in the reference implementation of the GE we are building based on a number of baseline products. We won't capture all the functionality that was already implemented in the baseline products. That would mean trying to carry out a kind of "reverse engineering" of baseline products, deriving User-Stories from what those baseline products already support. This, apart from meaning a huge effort would not be be that useful to development teams and would simply delay our development. Something which would precisely go against a cornerstone axiom in Agile: trying to do things that are useful for the development teams. Thus, a potential user who wants to have a detailed picture of all target functionality to be supported by a given FI-WARE GE should study:

• the FI-WARE Product Vision, in order to understand what is overall expected for the GE
• the documentation about the products that have been adopted as baseline for the reference implementation of the GE (available on Materializing the FI-WARE Vision): this should give the user a clear picture of what functionality has already been covered and therefore will be supported in FI-WARE
• the backlog linked to the GE (also available on Materializing the FI-WARE Vision: this will allow the user to understand what's going on and is planned on the roadmap

In summary: the FI-WARE backlogs were created to drive developments to be carried out after the FI-WARE project started. They are not about documenting what he have done during many years in our respective labs.

Management of the FI-WARE Backlog

The FI-WARE project uses a Product Backlog to drive the development of the reference implementations of Generic Enablers (GE) in FI-WARE. You can find a full description of all the Themes, Epics, Features and User-Stories linked to each FI-WARE Chapter and GE visiting the FI-WARE Wiki section on Materializing the FI-WARE Vision.

FI-WARE deals with management of the lifecycle of Backlog entries using FusionForge trackers. Each FI-WARE chapter owns a Backlog Management tracker (access only allowed to FI-WARE Chapter members):

• Cloud Hosting Backlog Management tracker
• Data/Context Management Backlog Management tracker
• IoT Services Enablement Backlog Management tracker
• Apps/Services Ecosystem and Delivery Backlog Management tracker
• Security Backlog Management tracker
• Interface to Networks and Devices Backlog Management tracker

Any ticket created in a given Chapter tracker is linked to a Theme/Epic/Feature/User-story described on the Wiki. Work Items to be carried out also have a ticket on the proper Chapter tracker. Note that Work Items are not documented on the Wiki. We have taken this decision in FI-WARE because Themes/Epics/Features/User-stories are the only Backlog entries that are relevant to FI-WARE users.

There is a comprehensive set of tutorials explaining FI-WARE project members:

• How to assign identifiers to FI-WARE Backlog entries (convention to follow)
• How to upload the full description of backlog entries to the Wiki
• How to create and configure trackers in FusionForge
• How to create entries in the "Backlog Management" Tracker of a FI-WARE Chapter
• How to follow-up a tracker in FusionForge

Requesting the addition of new entries to the FI-WARE Backlog

FI-WARE users can issue request for the addition of new Themes, Epics or Features in FI-WARE, linked to existing Generic Enablers or proposals for new ones. They do so by creating tickets on the FI-WARE Theme/Epic/Feature Requests tracker

In order to get access to the FI-WARE backlog one needs to:

1. Register an account at the FI-WARE FusionForge
2. join the FI-WARE project in FusionForge
3. Provide a link (in the ticket) to a complete backlog entry draft in the Uncategorized Enabler backlog on this wiki
4. Issue a ticket on the FI-WARE Theme/Epic/Feature Requests tracker

If the ticket is correct (authorized user, formally correct, in the Uncategorized Enabler backlog, etc) a Unique ID is created and assigned to the entry by the FI-WARE Chief Architect or Deputy Chief Architect. A task to handle the ticket is then created and assigned to one of the FI-WARE Chapter Leads/Architects. They will double-check the entry and decide if additional technical clarification and modification is required. This may involve several interactions between the ticket Issuer, the FI-WARE Chapter Lead/Architect, and the FI-WARE Chief Architects in accordance to actual demand. Each and any of these steps is documented in the task on FI-WARE Forge. As a result of this process, a new Theme, Epic or Feature may be added to the FI-WARE Backlog. Since tickets and associated tasks are visible to the Issuer, FI-WARE Chapter Lead/Architect, FI-WARE Chief Archtitects, and even to registered Observers, status and progress is completely transparent.

References

[1] – xp123.com/articles/invest-in-good-stories-and-smart-tasks/

[2] – agilesoftwaredevelopment.com/blog/vaibhav/good-user-story-invest

[3] – en.wikipedia.org/wiki/INVEST_%28mnemonic%29