Business Readiness Rating : a framework for evaluating Open Source Software

I remember having read something about the Open Business Readiness Rating initiative but never got down to reading their work. I just did and found that - while not revolutionnary - it could be a very good tool to assess our own software - especially regarding the enterprise content management segment of the market. Since MODx 1.0 and more particularly Tattoo might just aim this target, I think it could be interresting to give this a look.

Let me quote some parts of the whitepaper I found really interresting (which you can download on their site, 22 pages. You can also download the assessment template or see assesments for Mambo or FireFox for instance).

Sorry for the long post, I tried to summarize what this is about as good as I could :

Today, open source software is increasingly considered for business use.

(...)

Despite these benefits, certain issues can hinder adoption. The number of open source projects is vast. Projects range from low-quality individual efforts to high-quality enterprise solutions. On SourceForge alone, over 100,000 open source projects are listed, with many more on other open source repositories like CodeHaus, Tigris, Java.net, ObjectWeb, and OpenSymphony.

Users and potential adopters of open source software face the following challenges:

• Selection. For some software categories, the choices are virtually limitless.
• Support. Most open source packages are not professionally supported.
• Longevity. Since most open source projects are not backed by commercial companies, the availability of future releases depends on community efforts.
• Volatility. Many open source projects follow the “Release Early, Release Often” paradigm to obtain traction in the community. Thus the only constant thing in the open source world is change. Many potential adopters of open source software are not ready to track and implement the rapid updates and changes of software packages prevalent in the open source world.
• Low quality code for immature projects. During conception, an open source project is usually developed by one or a few hobbyists or cash-strapped IT staffers who are passionate about creating something. These early developers may lack the resources or necessary experience to deliver a complete product. Projects may be orphaned rapidly once initial developers have created a component that mostly solves their problem. New projects may be developed without formal software engineering or testing.

(...)

There is a real need for a widely adopted, standardized method to assess the maturity of open source software. (In Appendix 2 of this paper, we list many of the characteristics of mature open source software.)

(...)

In practice, many software evaluation projects are done ad hoc, without a formal assessment methodology. Ad hoc methods may be incorrect or incomplete in their assessment, and it is extremely difficult to validate the correctness of the evaluation. Inaccurate and incomplete evaluation mechanisms can lead to faulty decisions and product choices, which makes ad hoc assessment risky.

(...)

Additionally, an open and standard assessment model that is widely adopted and non-controversial would allow open source software users to share assessment results. Why standard? A standardized model allows common understand¬ing of the assessment ratings. Why open? An open model promotes trust in the assessment process. It also ensures that the assessment model is flexible with respect to future changes. Validation of an open assessment model for correct¬ness is straightforward: potential adopters of model will look at it, comment on it, and improve it.

Such a model should include the crucial requirements of a good software rating model — that it be Complete, Simple, Adaptable, and Consistent (CSAC):

• COMPLETE. The primary requirement for any product rating model is the ability of the model to highlight every prominent characteristic of the product, whether favorable or not. This is necessary so that the rating for any product is never misleading.
• SIMPLE. To gain wide acceptance, the model must be easily understood and relatively easy to use. Furthermore, the rating and terminology should be customer friendly. However, the model’s completeness takes a higher priority.
• ADAPTABLE. Due to rapid changes in the software industry, any software rating model created today may be irrelevant in the future. During the conception stage, it is impossible to capture all future potential uses of the model. Therefore, we strive to build our model with adaptability in mind — and to keep it open. That way, when the model requires an extension, it will be easy to add one without much disruption of the current model.
• CONSISTENT. The scales and ratings that the model produces should be consistent across the model’s different target uses. Comparable ratings for two software packages from two categories should signify equal business readiness.

(...)

During the initial Quick Assessment phase, a simple filter lets potential adopters quickly rule in or rule out software components with confidence.
We identified several viability indicators for use as filters in this phase, including:

• What is the licensing/legal situation of the software?
• Does it comply with standards?
• Are there referenceable adopters or users for it?
• Is a supporting or stable organization associated with the development efforts?
• What is its implementation language?
• Does it support internationalization and localization in your desired language?
• Are there third-party reviews of the software?
• Have books been published about the software?
• Is it being followed by industry analysts, such as Gartner or IDC?

(...)



(...)

We have defined 12 categories for assessing software:

Functionality : How well will the software meet the average user’s requirements?

• Usability : How good is the UI? How easy to use is the software for end-users?
• How easy : is the software to install, configure, deploy, and maintain?
• Quality: Of what quality are the design, the code, and the tests? How complete and error-free are they?
• Security : How well does the software handle security issues? How secure is it?
• Performance : How well does the software perform?
• Scalability : How well does the software scale to a large environment?
• Architecture : How well is the software architected? How modular, portable, flexible, extensible, open, and easy to integrate is it?
• Support : How well is the software component supported?
• Documentation : Of what quality is any documentation for the software?
• Adoption : How well is the component adopted by community, market, and industry?
• Community : How active and lively is the community for the software?
• Professionalism : What is the level of the professionalism of the development process and of the project organization as a whole?

We define an assessment of software from a specific aspect as a Category Rating. A category rating is obtained by grouping together several metrics that measure the same aspects. How the rating in one category is calculated may differ from how another category is measured, but the results should use the same scale (1 to 5). One metric may contribute to several categories in different ways: for example, Fedora’s release cycle of six months indicates a high level of community “liveness” but a low level of stability.

(....)

The Quick Assessment phase and defining and ranking of metrics and categories according to their importance for the software’s functional orientation leads us to the actions and steps taken in each assessment phase of the model to calculate the software’s Business Readiness Rating.



Phase 1 – Quick Assessment

• Identify a list of components to be evaluated.
• Measure each component against the quick assessment criteria.
• Remove any components that do not satisfy user requirements from the list.

Phase 2 – Target usage assessment

Category weights

• Rank the 12 categories according to importance (1 – highest, 12 – lowest).
• Take the top 7 (or fewer) categories for that component, and assign a percentage of importance for each, totaling 100% over the chosen categories.
• Metric weights
• For each metric within a category, rank the metric according to importance to business readiness.
• For each metric within a category, assign a percentage of importance, totaling 100% over all the metrics within one category.

Phase 3 – Data collection and processing

• Gather data for each metric used in each category rating, and calculate the applied weighting for each metric.
• Phase 4 – Data translation
• Use category ratings and the functional orientation weighting factors to calculate the Business Readiness Rating score.
• Publish the software’s Business Readiness Rating score.

Very interesting and cool David... thanks!

Thanks Ryan

I realize this might look theoretical and with a general scope, but I think I can use some of what’s in there to market MODx better