The field of software engineering made notable strides in the 1990s when the Department of Defense promulgated via its contracting operations a Capability Maturity Model supported by a Software Engineering Center (*SEI) at Carnegie-Mellon University. Arguably, the model and resulting forces were more belief-based than experimentally validated, but “process improvement through measurement” became a motivating mantra. For more detail see the over-edited Wikipedia article on CMM.
This post is aimed at accessibility researchers and at managers and developers of products with an accessibility requirement, explicitly or not. Visually impaired readers of this post may find some ammunition for accessibility complaints and for advice to organizations they work with.
The 5 Levels of Maturity Model
Here are my interpretations of the 5 levels of capability maturity focused on web accessibility features:
Chaotic, Undefined. Level 1
Each web designer followed his or her own criteria for good web pages, with no specific institutional target for accessibility. Some designers may know W3C standards or equivalents but nothing requires the designers to use them.
Repeatable but still undefined Level 2
Individual web designers can. through personal and group experience, estimate page size, say in units of HTML elements and attributes. Estimation enables better pricing against requirements. Some quality control is in place, e.g. using validation tools, maybe user trials, but the final verdict on suitability of web sites for clients rests in judgements of individual designers. Should those designers leave the organization, their replacements have primarily prior products but not necessarily any documented experience to repeat the process or achieve comparable quality.
Defined Level 3
Here, the organization owns the process which is codified and used for measurement of both project management and product quality. For example, a wire frame or design tool might be not a designer option but rather a process requirement subject to peer review. Standards such as W3c might be applied but are not as significant for capability maturity as that SOME process is defined and followed.
Managed Level 4
At this level, each project can be measured for both errors in product and process with the goal of improvement. Bug reports and accessibility complaints should lead to identifiable process failures and then changes.
Optimizing Level 5
Beyond Managed Level 4, processes can be optimized for new tools and techniques using measurements and data rather than guesswork. For example, is “progressive enhancement” an improvement or not?” can be analytically framed in terms of bug reports, customer complaints, developer capabilities, product lines expansion, and many other qualities.
How well does CMM apply to accessibility?
Personally, I’m not at all convinced a CMM focus would matter in many environments, but still it’s a possible way to piggy back on a movement that has influenced many software industry thinkers and managers.
Do standards raise process quality?
It seems obvious to me that standards such as W3C raise awareness of product quality issues that force process definition and also provide education on meeting the standards. But is a well defined standard either necessary or sufficient for high quality processes?
An ALT tag standard requires some process point where ALT text is constructed and entered into HTML. A process with any measurement of product quality will involve flagging missing ALT texts which leads to process improvement because it’ is so patently silly to have required rework on such a simple task. Or are ALT tags really that simple? A higher level of awareness of how ALT tags integrate with remaining text and actually help visually impaired page users requires more sensitivity and care and review and user feedback. The advantage of standards is that accessibility and usability qualities can be measured in a research context with costs then amortized across organizations and transformed into education expenses. So, the process improvement doesn’t immediately or repeatably lead to true product quality, but does help as guidance.
Does CMM apply in really small organizations?
Many web development projects are contracted through small one-person or part-time groups. Any form of measurement represents significant overhead on getting the job done. For this, CMM spawned the Personal and Team Software Processes for educational and industrial improvements. Certainly professionals who produce highly accessible web sites have both acquired education and developed some form of personal discipline that involved monitoring quality and conscious improvement efforts.
Should CMM influence higher education?
On the other hand, embedded web development may inherit its parent organization quality and development processes, e.g. a library or IT division of a university. Since the abysmal level of accessibility across universities and professional organizations suggest lack of attention and enforcement of standards is a major problem. My recorded stumbling around Computer Science websites surfaced only one organization that applied standards I followed to navigate web pages effectively, namely, University of Texas, which has a history of accessibility efforts. Not surprisingly, an accessibility policy reinforced with education and advocacy and enforcement led small distributed departmental efforts to better results. Should by lawsuit or even education commitment to educational fairness for persons with disability suddenly change the law of the land, at least one institution stands out as a model of both product and process quality.
Organizations can define really awful processes
A great example of this observation is Unrepentant’s blog and letter to DoJ about PDF testimonies. Hours of high-minded social justice and business case talk was represented in PDF of plaint text on Congressional websites. Not only is PDF a pain for visually impaired people, no matter how much it applies accessibility techniques, the simple fact of requiring an application external to the browser, here Adobe Reader, is a detriment to using the website on many devices such as my Levelstar Icon or smart phones. My bet is that sure enough there’s a process on Congressional websites, gauged to minimize effort by exporting WORD docts into PDF and then a quick upload. The entire process is wrong-headed when actual user satisfaction is considered, e.g. how often are citizens with disabilities and deviant devices using or skipping reading valuable testimony and data? Indeed, WCAG standards hint, among many other items, that, surprise, web pages use HTML that readily renders strings of texts quite well for reading across a wide variety of devices, including assistive technology.
The message here is that a Level 3 process such as “export testimony docs as PDF” is detrimental to accessibility without feedback and measurement of actual end usage. The Unrepentant blogger claims only a few hours of work required for a new process producing HTML, which I gratefully read by listening on the device of my choice in a comfortable location and, best of all, without updating the damned Adobe reader.
Quality oriented organizations are often oblivious about accessibility
The CMM description in the URL at the start of this article is short and readable but misses the opportunity to include headings, an essential semantic markup technique. I had to arrow up and down this page to extract the various CMM levels rather than apply a heading navigation as in this blog post. Strictly speaking the article is accessible by screen reader but I wouldn’t hire the site’s web designer if accessibility were a requirement because there’s simply much more usability and universality well worth applying.
I have also bemoaned the poor accessibility of professional computing organization websites>. Until another generation of content management systems comes along, it’s unlikely to find improvement in these websites although a DoJ initiative could accelerate this effort.
CMM questions for managers, developers, educators, buyers, users
So, managers, are your web designers and organization at the capability level you desire?
How would you know?
- Just sample a few pages in WAVE validator from WebAim.org. Errors flagged by WebAim are worth asking web developers? do these errors matter? how did they occur? what should be changed or added to your process, if any? But not all errors are equally important, e.g. unlabelled forms may cause abandoned transactions and lost sales while missing ALT tags just indicate designer ignorance. And what if WAVE comes up clean? Now you need to validate the tool against your process to know if you’re measuring the right stuff. At the very least, every manager or design client has a automated feedback in seconds from tools like WAVE and a way to hold web developers accountable for widespread and easily correctable flaws.
- Ask for the defined policy. would an objective like W3C standards suffice? Well, that depends on costs within the organization’s process, including both production and training replacements.
- Check user surveys and bug reports. Do these correspond to the outputs of validation tools such as WebAim’s WAVE?
- Most important, check for an accessibility statement and assure you can live with its requirements and that they meet social and legal standards befitting your organizational goals.
Developers, are you comfortable with your process?
Level 1 is often called “ad hoc” or “chaotic” for a reason, a wake up call. For many people, a defined process seems constraining of design flexibility and geek freedom. For others, a process gets out of the way many sources of mistakes and interpersonal issues about ways of working. Something as trivial as a missing or stupid ALT tag hardly seems worthy of contention yet a process that respects accessibility must at some point have steps to insert, and review ALT text, requiring only seconds in simple cases and minutes if a graphic lacks purpose or context, with many more minutes if the process mis-step shows up only in a validator or user test. Obviously processes can have high payoffs or receive the scolding from bloggers like Unrepentant and me if the process has the wrong goal.
Buyers of services or products for web development, is CMM a cost component?
Here’s where high leverage can be attained or lost. Consider procuring a more modern content management system. Likely these vary in the extent to which they export accessible content, e.g. making it easier or harder to provide semantic page outlines using headings. There are also issues of accessibility of the CMS product functions to support developers with disabilities.
In the context of CMM, a buyer can ask the same questions as a manager about a contractor organizations’ process maturity graded against an agreed upon accessibility statement and quality assessment.
Users and advocates, does CMM help make your case?
If we find pages with headings much, much easier to navigate but a site we need to use lacks headings, it’s constructive to point out this flaw. It seems obvious that a web page with only an H4 doesn’t have much process behind its production, but is this an issue of process failure, developer education, or missing requirements? If, by any chance, feedback and complaints are actually read and tracked, a good manager would certainly ask about the quality of the organization’s process as well as that of its products.
Educators,does CMM thinking improve accessibility and usability for everyone?
Back to software engineering, getting to Level 5 was a BFD for many organizations, e.g. related to NASA or international competition with India enterprises. Software engineering curricula formed around CMM and government agencies used it to force training and organizational change. The SEI became a major force and software engineering textbooks had a focus for several chapters on project management and quality improvement. Frankly, as a former software engineering educator, I tended to skim this content to get to testing which I considered more interesting and concrete and relevant.
By the way, being sighted at the time, I didn’t notice the omission of accessibility as a requirement or standards body of knowledge. I have challenged Computing Education blogger and readers to include accessibility somewhere in courses, but given the combination of accreditation strictures and lack of faculty awareness, nothing is likely to happen. Unless, well, hey, enforcement just might change these attitudes. My major concern is that computing products will continue to be either in the “assistive technology ghetto” or costly overhauls because developers were never exposed to accessibility.
Looking for exemplars, good or bad?
Are there any organizations that function at level 5 for accessibility and how does that matter for their internal costs and for customer satisfaction as well as legal requirements?
Please comment if your organization has ever considered issues like CMM and where you consider yourself in a comparable level.