An Evaluation of CMMI Process Areas for
Small to Medium Sized
Software Development Organisations
F.G.Wilkie, D.McFall and F.McCaffery
Centre for Software Process Technologies
Faculty of Engineering
University of Ulster
Newtownabbey
Co-Antrim BT37 0QB
Northern Ireland
(email: fg.wilkie@, d.mcfall@, f.mccaffery@ulster.ac.uk)
Keywords: CMMI, Light-weight appraisal, SME, Software engineering practices
Abstract
In this paper we describe the results of CMMI software process appraisal work with six small-to-medium sized software development companies. Our analysis of six CMMI Process Areas appraised within each of these organisations is presented. Commonly practiced or not practiced elements of the model are identified, leading to the notion of perceived value associated with each specific CMMI practice. A finer grained framework which encompasses the notion of perceived value within specific practices is presented. We argue that such a framework provides incentive to small-to-medium sized enterprises starting process improvement programmes.
1.0 Introduction
The Centre for Software Process Technologies [CSPT 1, Wilkie et al 2004] is a research and knowledge transfer group funded jointly by the University of Ulster and a Northern Ireland governmental organisation charged with the economic development of this geographical region. The CSPT is tasked with motivating and supporting a culture of software process improvement within the Northern Ireland software industry.

The Software Engineering Institute has been transitioning from the Capability Maturity Model (SW-CMM) [SEI 2] to the Capability Maturity Model Integrated (CMMI) [Chrissis et al 2003] for the past two years or so. From the point of view of the CSPT, the CMMI is a welcome transition because it includes two representations: a continuous representation and a staged representation, whereas the older CMM was purely a staged model. The continuous representation is a more attractive proposition for small-to-medium sized (SMEs) companies which characterise the NI software industry – particularly where most of these companies have no compelling reason to achieve any particular maturity level rating, but would rather see the benefits from a software process improvement programme in perhaps a more gradual, progressive manner. Such an approach fits better with the continuous representation than with a staged model.
As part of the product suite for CMMI, the CMMI Steering Group and CMMI Product Team consisting of professionals from government, industry and the Software Engineering Institute has published the requirements for three categories of method which employ the CMMI [SEI 1]. Methods developed to comply with these requirements are known as ARC (Appraisal Requirements for CMMI) class-C, class- B and class-A. The Software Engineering Institute has developed its own class-A compliant method which is called SCAMPISM. The requirements for class-A methods tend to result in large methods which require a sizable effort from the appraised organisation both in terms of preparation for such an appraisal as well as considerable external effort from an appraisal team.
For much of the Northern Ireland software industry, class-A methods would not currently be appropriate. The CSPT approach is to build up awareness and understanding in the aims and objectives of software process improvement in a more gradual way – trying to keep the overhead associated with such measures small to begin with. For this reason the CSPT has developed its own appraisal method which follows the intent of the ARC 1.1 for a class-C method. Our method is called Express Process Appraisal (EPA).
SM SCAMPI, CMMI and CMM Integration are service marks of Carnegie Mellon University.

In a pilot appraisal programme, EPA has been used to appraise six software development companies in Northern Ireland. We evaluated all specific practices regardless of their capability level, but did not appraise any generic practices. This has enabled us to understand the characterisation of the process areas in use rather than their degree of institutionalisation. The results of all six appraisals indicate that every company fits somewhere into capability level 0-1 in most of the appraised process areas. The intention of this paper is to evaluate the structure of the CMMI continuous representation (version 1.1) [Chrissis et al 2003] from the point of view of its base and advanced specific practices in a number of process areas. The results of our evaluation have enabled us to make some observations about the structure of five CMMI process areas and to suggest some changes to this structure that would be helpful when appraising software organisations that have had little or no prior exposure to CMMI.
Our thesis is that by inspecting current industry practice for a group of organisations with little prior knowledge of CMMI, it is possible to make judgements about the relative levels of perceived value associated with individual practices within CMMI process areas. This information can be used to refine our understanding of the various specific practices detailed within the CMMI for the purpose of developing more appropriate appraisal procedures for use with such small-to-medium sized organisations.
It is our contention that perceived value is a judgement criterion used in determining activities that organisations pursue. If a prospective software engineering practice has a low perceived value (at a particular point in time) then that practice will not be pursued at that time. The perceived value is the combination of two parts. Firstly the perceived cost/effort required as an upfront, and in some cases ongoing, investment to put the practice in place. Secondly, the perceived benefit to be derived from making this investment. Activities (or practices) that are common in organisations are those that the organisation has deemed to have high perceived value. From the level of common practice in the software industry (through our sample data set), we can build a finer-grained understanding of the relative perceived value of the set of CMMI practices.

Of course, there may be several alternative reasons why a company implements particular practices and ignores others. For example, (i) some practices may be forced through regulatory requirements; (ii) some practices may be easy to implement and (iii) some practices may be prominent because of tradition. Half of our company sample were ISO 9001 accredited and would therefore have been influenced by its practice requirements. Undoubtedly some practices instilled in these companies will have a higher perceived value as a consequence of the influence of this standard on their business domains. Apart from ISO 9001, none of the six companies in our pilot scheme was forced to introduce any practices through other regulatory requirements.
In our experience, the economics of modern software development means that companies operate lean processes essential for business survival. As such, few practices are adopted simply because they are either easy to implement or because they have traditionally been part of the fabric of the software engineering domain. Such practices may be of low cost to introduce, but if they yield a low perceived benefit then, overall, their perceived value will be low. We therefore believe that in the highly competitive software industry of today, perceived value is at the root of decisions regarding software development practices. From an industry survey of 56 companies conducted by our research centre [McFall 2003, McCaffery 2004], the influences and motivations leading to the sets of practices followed by the companies in our dataset are deemed representative of those encountered by a wide range of software companies.
Section 2 provides a short profile of the appraised companies. Section 3 outlines the EPA method. Section 4 presents the results of our work and suggests extensions to our existing method to cater for organisations with little or no prior involvement in CMMI. Section 5 relates our work to that of others, and section 6 offers our concluding remarks.

2.0 The Data Set: Profile of Appraised Companies
Six software development companies participated in the pilot appraisal programme
over a five month period. On any categorising scheme such as (i) company size, (ii)
business sector, (iii) number of years in business, (iv) product- or services orientation,
(v) multi-national or indigenous, the six companies represent a random sample. Furthermore, the companies were selected by an independent sponsoring organisation without consulting the CSPT. The only discernable linkage between all the companies was a desire for some initial process appraisal which they viewed as an evaluation exercise.
Three of the companies are ISO900 1 accredited while the other three are not. Apart from the ISO 9001 accreditation, no other external standards were required of any of these organisations. None of the companies had any prior experience of software process improvement or particularly of either SW-CMM or CMMI.
Four of the companies can be considered mainly product-based while the other two are mainly services-orientated. The companies ranged in size from 8, 12, 25, 30, 65 and 130 software engineers. Four of the companies are based in one geographical region while the other two are based in another geographical region separated by approximately 75 miles. Two of the companies sourced many of their staff from the same donor organisation. There was little evidence of staff mixing between the other four companies. One of the companies is a subsidiary of a multinational while the other five can be considered indigenous. Two of the companies operate in the telecoms sector, while the other four all operate in different domains.
The fact that all of these organisations have been in business, developing software, for several years suggests that from an engineering and management standpoint, these companies are all doing at least “enough” of the right activities to survive – the essentials are in place.
Based on this profile of the appraised companies, combined with a larger detailed survey of 56 software companies we conducted [McCaffery 2004], we believe that our dataset, whilst small, is nevertheless representative of small-to-medium sized software development companies. Therefore, we believe that the results presented later are typical of SME’s with little or no prior experience of software process improvement.

3.0 Express Process Appraisal (EPA)
This section contains a short description of the EPA method and is intended to show the level of rigor, professionalism and consistency with which all six company appraisals were conducted.
The EPA method was developed by 5 staff in the CSPT. All these staff attended the official Software Engineering Institute’s (SEI) ‘Introduction to CMMI’ course and two of the staff subsequently completed the official SEI ‘Intermediate Concepts of CMMI’ course. One of the CSPT staff also had team member experience on both CMM and CMMI appraisal teams led by fully (SEI) qualified Lead Appraisers.
The team of 5 CSPT staff developed sets of stock questions from the CMMI. These stock questions enable us to ensure adequate coverage of the model during questioning sessions. Many additional questions are asked as an interview proceeds, based upon the answers provided to the stock questions. The stock questions also ensure that each appraised organisation is treated in a consistent manner (particularly important with light-weight appraisal methods) so that the appraisal results across companies can be compared one with another.
Six of the process areas associated with maturity level 2 in eth staged representation were applied: Requirements Management, Configuration Management, Project Planning, Project Monitoring & Control, Measurement & Analysis and Process & Product Quality Assurance. These are 6 of the 7 process areas associated with maturity level 2 and therefore the justification for starting a process improvement exercise with them is already well established – they constitute the engineering management basis of an organisation.
The method was applied by two appraisers who gathered evidence from the appraised
company through a series of interviews with company staff and also through some
document review. All observations were presented back to the development engineers
who participated in the original interviews. This helped validate the integrity of the observations.
Overall, EPA requires about 42 person-hours of the appraised organisation’s time and a similar amount of appraiser time. The method, whilst light-weight, is nevertheless much more thorough than a simple survey would be and, we believe, is ARC class-C compliant.
There are, by necessity, limitations to the capabilities of the method. It requires only one form of objective evidence – gathered through interview sessions. Therefore, if interviewees wished to, they could simply provide the “right” answers, rather than being totally truthful. Through careful staging of the interviews with no attribution and complete confidentiality, we hoped to eliminate this possibility. Both of the appraisal team members were from outside the organisation and this also helped ensure that staff believed in the established confidentiality.
The method, as implemented in the pilot scheme, is limited to a predefined set of six process areas. As such, it could be argued that, it takes little account of the business or situation specific issues. However, the six chosen process areas, coming from the maturity level 2 stable, are fundamental disciplines for any software development organisation and should therefore be of high importance.

4.0 CMMI observations from appraised companies
This section presents our findings across the 6 appraised companies with respect to all 6 process areas within the appraisal scope. These process areas contain a mixture of base and advanced practices. We present observations only for those practices where a strong positive or negative result was obtained across the sample set. We have analysed our findings for each specific practice across each of the 6 companies and also present some analysis of these observations albeit based on the limited sample size.
The figures (1-5) in the following subsections provide a score between 0-100% for each specific practice in each of five process areas under investigation. The score represents a cumulative measure across the 6 companies in the sample with each bar in the graph being a measure of how well the companies met the needs of that specific practice. A score of 0% means that the specific practice was not supported at all by any of the companies, whereas a score of 100% means that the CMMI practice was fully supported by all companies. The scores were obtained by rating the answers to questions on each specific practice as “Not-At-All” – value 0, “Partially” – value 0.33, “Largely” – value 0.67 and “Fully” – value 1.0. Each specific practice was explored by asking a set of between 1 and 5 questions. The scores for each question in a specific practice were averaged to give a single score value for that specific practice for that company. Then these individual company scores were added into the overall score for that practice which is shown in the appropriate graph. The reader can determine the relative contribution of each company to the result from the segmented representation of the bars on the graphs.
To ensure that confidentiality is maintained, neither the companies nor their relative overall performance can be identified from the presented results
4.1 The Requirements Management process area
Figure 1 shows the average score across the 6 companies in our data set for each of the specific practices in the Requirements Management process area.

	100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%		

SP 1.1-1	SP 1.2-2	SP 1.3-1	SP 1.4-2	SP 1.5-1
Requirements Management Specific Practices
CMMI Practice Number	CMMI Practice description	
SP 1.1-1	Obtain an understanding of requirements	
SP 1.2-1	Obtain commitment to requirements	
SP 1.3-1	Manage requirements changes	
SP 1.4-1	Maintain bidirectional traceability of requirements	
SP 1.5-1	Identify inconsistencies between project work and requirements	

Figure 1 – Average scores for practices in CMMI Requirements Management
All appraised companies appear strong in SP 1.1-1 (Obtaining an understanding of requirements). Requirements were well documented with appropriate criteria in place to determine the acceptability of these requirements. However, every organisation failed to demonstrate an adequate scheme for evaluating the impact of changes to their requirements (SP 1.3-1) and few of the organisations maintained a clear scheme of bi-directional traceability to requirements (SP1 .4-2). They also failed to record and monitor the rate and sources of change requests (SP 1.3-1) with consequent impact on their ability to determine trends and thereby refine requirements management processes. Alarmingly, half of the appraised companies failed to ensure that their plans/schedules always accurately reflected the current state of their requirements

(SP1 .5-1). Obtaining commitments to requirements (SP1 .2-2) was strongly followed in most of the appraised companies.
Our observations suggest that SP 1.3-1 appears to be a more sophisticated practice than the other base practices in this process area and that SP1 .4-2 is justified in being deemed an advanced practice within CMMI version 1.1. At first sight, it appears that the advance practice SP 1.2-2 appears to be more commonly practiced than some of the basic practices in this process area. However, 5 of the 6 appraised companies are product based and, within such companies, obtaining commitment to requirements is perhaps less difficult than for services-based organisations.
4.2 The Configuration Management process area
Figure 2 shows the average score across the 6 companies in our data set for each of the specific practices in the Configuration Management process area.

SP1.1-1 SP1.2-1 SP1.3-1 SP2.1-1 SP2.2-1 SP3.1-1 SP3.2-1

100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

Configuration Management Specific Practices
CMMI Practice Number	CMMI Practice description	
SP 1.1-1	Identify configuration items	
SP 1.2-1	Establish a configuration management system	
SP 1.3-1	Create or release baselines	
SP 2.1-1	Track change requests	
SP 2.2-1	Control configuration items	
SP 3.1-1	Establish configuration management records	
SP 3.2-1	Perform configuration audits	

Figure 2 – Average scores for practices in CMMI Configuration Management

Most companies identify configuration items (SP 1.1-1) and have an established system for controlling these items (SP 1.2-1) – albeit relying on tools more than process. However, sufficient usage of the reporting facilities in most CM tools was not evident and audits of configuration management records are almost never performed (SP3 .2-1). Half of the companies in our data set were not controlling their configuration items adequately (SP2.2-1). On the positive side, most companies are using baselines to control the release of software (SP 1.3-1).
Company reliance on tool support in this area is to the detriment of more human (manual) aspects of the CM process. For this process area, the specific practices associated with each of the three specific goals appear to reflect three different degrees of commonly observed practice in our data set. We see the practices of the first specific goal (Establish Baselines) being more commonly applied than those of the second specific goal (Track and Control Changes) which in turn are more commonly applied than those of the third specific goal (Establish Integrity).
4.3 The Project planning process area
Figure 3 shows the average score across the 6 companies in our data set for each of the specific practices in the Project Planning process area.

	100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%		

SP SP SP SP SP SP SP SP SP SP SP SP SP SP
1.1-1 1.2-1 1.3-1 1.4-1 2.1-1 2.2-1 2.3-1 2.4-1 2.5-1 2.6-1 2.7-1 3.1-1 3.2-1 3.3-1
Project Planning Specific Practices
CMMI Practice Number	CMMI Practice description	
SP 1.1-1	Estimate the scope of the project	
SP 1.2-1	Establish estimates of work product and task attributes	
SP 1.3-1	Define project lifecycle	
SP 1.4-1	Determine estimates of effort and cost	
SP 2.1-1	Establish the budget and schedule	
SP 2.2-1	Identify project risks	
SP 2.3-1	Plan for data management	
SP 2.4-1	Plan for project resources	
SP 2.5-1	Plan for needed knowledge and skills	
SP 2.6-1	Plan stakeholder involvement	
SP 2.7-1	Establish the project plan	
SP 3.1-1	Review plans that affect the project	
SP 3.2-1	Reconcile work and resource levels	
SP 3.3-1	Obtain plan commitment	

Figure 3 – Average scores for practices in CMMI Project Planning
All appraised companies establish budgets and schedules (SP2. 1-1) and decompose their projects into manageable sets of tasks with clearly defined boundaries and scope (SP 1.1-1) using appropriate tool support. However, re-planning as a result of deviations was much less evident. For some companies, the notion of a plan and a schedule are one and the same thing. We found that all the companies communicated effectively with their stakeholders (SP2.6-1) and this helped to obtain necessary project commitments (SP3.3-1). The main weakness we found was in the area of establishing estimates of work product and task attributes (SP 1.2-1). Interestingly,

most companies appeared to determine estimates of effort and costs (SP 1.4-1) but the basis of these was unsound. The connection of SP1.2-1 with SP1.4-1 was often not made. There appeared to be little or no comparison of actual effort against estimated effort at any stage. Amongst the smaller companies (less than 25 staff) there was little explicit planning for needed knowledge and skills (SP2.5- 1) – the managers in such companies seemed to implicitly know who possessed the necessary skills to undertake a given task.
Tasks involving reflection on past performance tend to be problematical. The lack of past project historical data tends to discourage size estimation and in this sense is something of a vicious circle. Companies had great difficulty understanding what constitutes appropriate units of size measure for their various tasks. The sense of difficulty in performing estimation seemed to result in a resignation that any degree of accuracy would be almost unattainable and therefore such practices were deemed to be of low perceived value. For product-based companies, erroneous estimates often lead to reduced features in scheduled releases which may not always have serious consequences. Hence, estimates were often not deemed critical.

4.4 The Project Monitoring & Control process area
Figure 4 shows the average score across the 6 companies in our data set for each of the specific practices in the Project Monitoring and Control process area.
100%
90%

	80% 70% 60% 50% 40% 30% 20% 10% 0%		

SP 1.1-1 SP 1.2-1 SP 1.3-1 SP 1.4-1 SP 1.5-1 SP 1.6-1 SP 1.7-1 SP 2.1-1 SP 2.2-1 SP 2.3-1
Project Monitoring & Control Specific Practices
CMMI Practice Number	CMMI Practice description	
SP 1.1-1	Monitor project planning parameters	
SP 1.2-1	Monitor commitments	
SP 1.3-1	Monitor project risks	
SP 1.4-1	Monitor data management	
SP 1.5-1	Monitor stakeholder involvement	
SP 1.6-1	Conduct progress reviews	
SP 1.7-1	Conduct milestone reviews	
SP 2.1-1	Analyse issues	
SP 2.2-1	Take corrective action	
SP 2.3-1	Manage corrective action	

Figure 4 – Average scores for practices in CMMI Project Monitoring & Control
Most appraised companies have good mechanisms for both periodic and event driven review of progress (SP 1.6-1 and 1.7-1). Issues identified from reviews are analysed effectively with appropriate corrective actions being taken (SP2.1-1 and SP2.2-1). Whilst the core planning parameters appear to be in place for most companies (SP 1.1- 1) and are carefully reviewed, the scope of the parameters deemed important was limited. The limited set of parameters used, whilst apparently adequate for the status quo, would not provide sufficient insight into the operation of the companies under (i) extreme circumstances and (ii) where an improvement programme was launched.

In general, from figure 4, it appears that most companies are better at ‘reacting’ (associated with SP1.5-1, 1.6-1, 1.7-1, 2.1-1 and 2.2-1) than ‘discovery’ (SP 1.1-1, 1.2-1, 1.3-1, 1.4-1 and 2.3-1) – a common characteristic of low maturity organisations. In particular SP2.3-1, which is concerned with managing corrective actions to ensure that they are completed effectively, was usually weak – companies not having sufficient measures to determine effectiveness.
4.5 The Process & Product Quality Assurance process area
Figure 5 shows the average score across the 6 companies in our data set for each of the specific practices in the Process and Product Quality Assurance process area.

	100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%		

SP 1.1-1	SP 1.2-1	SP 2.1-1	SP 2.2-1
Process - Product Quality Assurance
CMMI Practice Number	CMMI Practice description	
SP 1.1-1	Objectively evaluate processes	
SP 1.2-1	Objectively evaluate work products and services	
SP 2.1-1	Communicate and ensure resolution of non-compliance issues	
SP 2.2-1	Establish records	

Figure 5 – Average scores for practices in CMMI Process and Product QA
The meaning of “QA” varies widely. This was evident in our sample of companies. For some, QA is synonymous with testing. For most, QA is constrained to the evaluation of work products and services. Only one company in our sample included the evaluation of both product and process related aspects of its operation. The poor performance of most companies in monitoring process usage in particular, eliminates their ability to identify and therefore deal with non-compliance issues (SP2. 1-1).

4.6 The Measurement & Analysis process area
Of the six process areas we evaluated in this study, measurement & analysis was by far the weakest to the extent that we stopped trying to quantify the answers we received. For this reason, no bar chart is presented in this section.
Organisational business goals, generated from a strategic plan, should inform the tactics of the company. These tactics are monitored and controlled through operational measures taken on a periodic basis. For most of the appraised companies, core operational measures were apparent for a very limited set of tactics. These tactics generally concerned basic survival – maintaining the essence of the business, such as ensuring that milestones are reached at appropriate times.
All appraised companies reported having a strategic plan; however the limited measures being made at the operational level suggests either a strategic naivety or a failure to develop comprehensive tactics for the overall strategic plan. We observed little evidence of documented measurements objectives (SP 1.1-1). Most appraised companies were unsure of the links between their business goals and the measures they make. That said, some measures were made, stored, analysed and appeared to be reported through periodic progress reviews (practices related to specific goal 2), but little of this was documented (practices related to specific goal 1). This lack of formality inhibits the traceability of business goals through tactics and down to operations.
4.7 Extensions to EPA
Our thesis is that organisations conduct activities on the basis of that activity’s perceived value to them. Therefore, by analysing the degree to which companies support each of the CMMI practices, we can say something about the perceived value of each CMMI practice to the software industry. This is a very important factor for two reasons. Firstly, it enables the appraisal to be focussed on practices which are most likely to be deemed valuable to the organisation and thereby the whole appraisal

exercise is viewed more positively – essential for companies that are often very sceptical about process appraisal. Secondly, it assists the appraisal team when encouraging a company to embark on a programme of process improvement because the relevance and benefit to be gained by implementing practices is more attractive.
Our focus is strictly based on practices within the CMMI. All CMMI practices have been carefully considered and therefore their actual value to an organisation is implicit. However, the question is whether a practice of low perceived value actually yields low real value or not. In some cases there may be a need for education to better align the ‘perception’ with the ‘actual’.
For each of the CMMI specific practices discussed in the previous section, we present their perceived value (in table 1) – leading to a finer grained model that can be used to introduce and stimulate process improvement in smaller steps, which is of particular relevance to low maturity organisations, where selling improvements must be on the basis of perceived value.
CMMI Practices have been categorised as having either HIGH, MEDIUM or LOW perceived value. This scheme is based on the results presented earlier in figures 1-5. Practices having a score of more than 70% are deemed to be HIGH, whilst those having less than 40% compliance are deemed to be LOW. All other practices falling between 40-70% are deemed MEDIUM. The thresholds of 40 and 70% were chosen because they provide reasonable discrimination between practices.
The perceived values act as a guide for an SEPG charged with implementing process improvement because it will be easier to encourage the use of practices that have a higher perceived value. Practices with a low perceived value may require more effort to convince software engineering managers of their worth. So the categories can inform the plans developed by an SEPG. Of course, irrespective of the CMMI, each practice has an actual value which will often be different from its perceived value. The actual value will vary from company to company – it is, in many cases, very dependent on environmental considerations such as company size, business domain and business goals. It is the gap between the perceived value and the actual value that an SEPG needs to be concerned about because this will determine the overall benefit
which the company could realise by implementing that practice. The difficulty is in determining the actual value. A limitation of the CMMI for SMEs is the implication that all specific practices are of equal, high, value. The use of a small number of so- called ‘advanced’ practices in the CMMI (version 1.1) does little or nothing to dispel the mystery surrounding which practices are most valuable.
Table 1 presents our analysis of the perceived values for specific practices. These results are averaged across the six companies evaluated.

Requirements Management	
Perceived Value	Goal 1		
High	SP1.1-1
SP1.2-2		
Medium	SP1.5-1		
Low	SP1.3-1
SP1.4-2		
Configuration Management	
Perceived Value	Goal 1	Goal 2	Goal 3	
High	SP1.1-1
SP1.3-1	---	---	
Medium	SP1.2-1	SP2.1-1
SP2.2-1	SP3.1-1	
Low	---	---	SP3.2-1	
Project Planning	
Perceived Value	Goal 1	Goal 2	Goal 3	
High	SP1.1-1	SP2.1-1
SP2.7-1	SP3.3-1	
Medium	SP1.3-1
SP1.4-1	SP2.2-1 SP2.3-1 SP2.4-1 SP2.6-1	SP3.1-1
SP3.2-1	
Low	SP1.2-1	SP2.5-1	----	
Project Monitoring and Control	
Perceived Value	Goal 1	Goal 2		
High	SP1.6-1
SP1.7-1	SP2.1-1
SP2.2-1		
Medium	SP1.1-1 SP1.2-1 SP1.3-1 SP1.5-1	---		
Low	SP1.4-1	SP2.3-1		
Process and Product Quality Assurance	
Perceived Value	Goal 1	Goal 2		
High	---	---		
Medium	SP1.1-1
SP1.2-1	---		
Low	---	SP2.1-1
SP2.2-1		

Table 1 – Industry perceived values of CMMI specific practices

It is clear from table 1 that within most goals there are practices spread over the three categories. For example, 7 out of the 11 goals appraised involve some practices which are deemed to have low perceived value. In 9 out of the 11 goals, the perceived high value practices are outnumbered by the medium and low value practices. Four of the 11 goals do not entail any perceived high value practices. All this means that the goal- driven appraisal encourages organisations to be appraised against practices which they do not value. Consequently, the appeal of the appraisal is reduced and subsequent improvement activities are likely to be unwelcome.
The implications of these findings suggest extending the EPA method through optional inclusion of three appraisal passes. The first pass (EPA phase 1) focuses more precisely on practices that have a high perceived value. Thereby encouraging low maturity companies – motivated by some likely positive feedback in the results. The second pass (EPA phase 2) would cover high and medium value practices, followed later by a third pass (EPA phase 3) to include the perceived low-value practices. Thus building identified improvements in small, tolerable steps.
5.0 Related Work
Process assessment methods generally draw upon one of two process models: (i) SPICE and (ii) CMMI [Chrissis et al 2003]. During the late nineties the SPIRE programme (Software Process Improvement in Regions of Europe) [SPIRE 1998] applied the SPICE model to a variety of software development organisations of less than 50 staff (i.e. SMEs). Experiences with the SPICE model led to cherry-picking amongst the SPICE process areas that were directly related to the company’s business goals. Our work, however, is commenting on the priorities of practice within CMMI which is at a much finer-grained level than the reported tailoring outcomes from the SPIRE programme.
Anacleto et al 2004a,b has considered process assessment methods for small software
companies. They provide a framework for comparing assessment methods and have
compared many methods using it. They suggest that assessment methods “either

provide a method for a fixed set of processes or in correspondence with the specific characteristics and goals of an organisation select a set of processes to be investigated.” Our EPA method has been used in the former style where we preselected a set of 6 process areas to appraise. However, EPA could equally well be employed with a set of process areas determined in consultation with the organisation to be appraised. Those process areas being drawn from the full suite of 25 within the continuous representation of CMMI and within the constraints of EPA where the evidence gathering sessions must be concluded within one working day.
Paulk (1998) evaluated how the SW-CMM could be used in small software development organisations. He set out to determine if there were key processes or goals that should not be applied to small organisations and to determine what motivates abuse of the SW-CMM. He points out that the primary business objective of small companies is to survive. Even though such companies may believe that benefit is possible from process improvement, they often have insufficient resource to implement any such programme. He considers that all SW-CMM key process areas and goals are always relevant to any environment but that professional judgement should be exercised as to acceptable alternative practices. In our work, we do not suggest that any of the CMMI process areas or associated goals can be ignored, however, we have identified three tiers to the CMMI practices associated with these goals that exist through common practice within our sample set. Paulk also suggests that “unreasonable constraints on tailoring can lead to significant resistance to following the process”. The implication here is that, as Hoffman (1998) expresses1 “don’t require processes that don’t make sense”. The extension of this from our work is simply “don’t immediately require practices that don’t currently appear to add sufficient value”. By following the conclusion from Paulk’s work, that software process improvement should be done to help the business, our three-phased method is driven by the need of businesses to be motivated for software process improvement.
Consolidating performance scores across the six companies in our sample, we find
that, overall, the six process areas from strongest CMMI compliance to weakest are:
(i) Project Monitoring & Control; (ii) Project Planning; (iii) Requirements
1 As cited in Paulk (1998).

Management; (iv) Configuration Management; (v) Process & Product Quality Assurance and (vi) Measurement & Analysis. It is interesting to compare our consolidated results with results reported by the Software Engineering Institute [SEI 3] from SCAMPISM appraisal returns. The process area profiles for organisations appraised at maturity level 1 (some 18 organisations) indicate the following order with respect to the percentage of appraisals reporting full satisfaction in each of these six process areas (from highest to lowest): (i) Project Monitoring & Control; (ii) Project Planning; (iii) Requirements Management; (iv) Configuration Management; (v) Measurement & Analysis and (vi) Process & Product Quality Assurance. It appears that the trends we observed in our small sample set are very similar to those reported from full SCAMPISM appraisals. The fact that the SEI sample set is larger than our own, combined with their results being based on a much more rigorous method (SCAMPISM), we believe further qualifies the validity of our work.
6.0 Summary and Conclusions
From our initial pilot work with EPA, certain trends have emerged in the appraisal results. For example, small software companies tend to focus on product quality assurance rather than process quality assurance – relying more heavily on individual developer competence as opposed to process.
Medium-sized software development organisations appear to consider process to a greater extent – but often not as fully as the CMMI would inform. Such organisations may benefit from some additional process, but must always be convinced of the benefits from its introduction.
During this current time of severe economic constraint in the software industry, most companies appear to spend too little time reflecting on their own performance. Root- cause analysis is often neglected, inhibiting performance improvement. The culture of software process improvement tends either to be absent, or to be pursued in a fairly hap-hazard manner.

Most companies preferred the on-site interview sessions to be contained within one working day. It was possible to cover 6 process areas to capability level 1 within this one day constraint. However, by distilling out three fractions from the 6 process areas, it would be possible to extend the scope of an appraisal to more process areas, covering only high and medium value practices, and still complete the evidence gathering within the one day constraint. Of course, such an approach is not ARC class-C compliant until the full extent of the CMMI is embraced – as would be the case by completing evidence gathering associated with practices of high, medium and low perceived value.
A limitation of the CMMI for small-to-medium sized enterprises is the implication that all specific practices are of equal, high value. This is not the industry perception observed in our current study. The number of advanced practices in the current version (V1. 1) of the CMMI is very small and there is no explicit suggestion that advanced practices are more valuable than base practices anyway. Furthermore, there is no differentiation within base (or advanced) practices on the issue of value. We believe that there is a need for more guidance as to the value of each specific practice to different types and sizes of organisations if the CMMI is to gain widespread appeal. For any CMMI practice, the perceived benefit may not match the actual benefit as a result of the gap between perceived and actual value. Understanding the issues in these terms greatly helps in “selling” model details of process improvement into organizations.
Many software development companies that do not have any desire or need to be CMMI compliant may well be satisfied with EPA phase 1 or phase 2. The introduction of these phases may encourage more companies to embark on a process improvement programme, which is the ultimate objective.
7.0 Acknowledgements
This work was partly funded by the Centre for Software Process Technologies at the University of Ulster which is supported by the EU Programme For Peace And Reconciliation in Northern Ireland and The Border Region Of Ireland (PEACE II).

The authors also acknowledge sponsorship of the pilot appraisal programme by Invest NI and also the participation of the six commercial companies involved in the pilot.
8.0 References
Anacleto 2004a Anacleto, A, von Wangenheim, CG, Salviano, CF, Savi, R. “A method for Process Assessment in small software companies”, 4th International SPICE Conference on Process Assessment and Improvement, Lisbon, Portugal, pp.69-76 (April 2004).
Anacleto 2004b Anacleto, A, von Wangenheim, CG, Salviano, CF, Savi, R. “Experiences gained from applying ISO/IEC 15504 to small software companies in Brazil”, 4th International SPICE Conference on Process Assessment and Improvement, Lisbon, Portugal, pp.33- 37 (April 2004).
Chrissis 2003	Chrissis, MB, Konrad, M, Shrum, S. “CMMI: Guidelines for
Process Integration and Product Improvement”, Addison Wesley, (2003), ISBN 0-321-15496-7.
CSPT 1	The Centre for Software Process Technologies, University of Ulster,
www.cspt.ulster.ac.uk.
Hoffman 1998 Hoffmann, L. “Small Projects and the CMM” in “Key Practices to the CMM: Inappropriate for Small Projects?” panel, Rita Hadden moderator, Proceedings of the 1998 Software Engineering Process Group Conference, Chicago, IL, (March 1998).
McCaffery 2004 McCaffery FH, Wilkie FG, McFall DJ, Lester NG, (Apr 2004) "Northern Ireland Software Industry Survey", Proceedings of Fourth International SPICE Conference on Process Assessment and Improvement, Lisbon, Portugal, 28-29 April 2004, SPICE User Group (Lisbon, Portugal), ISBN 972-9071-73-X, Pages 159-16 1
McFall 2003	McFall, D., Wilkie, F.G. McCaffery, F. Lester, N.G. and Sterritt, R.
“Software Processes and Process Improvement in Northern Ireland”, 16th International Conference on Software & Systems Engineering and their Applications, Paris, France, December 2003, ISSN: 1637- 5033, p1-10.
Paulk 1998	Paulk, MC. “Using the Software CMM in Small Organisations”.
Joint 1998 Proceedings of the Pacific Northwest Software Quality Conference and the Eighth International Conference on Software Quality, Portland, OR, pp.250-361, (October 1998).

SEI 1	“Appraisal Requirements for CMMI, Version 1.1 (ARC, V1. 1)” by
the CMMI Product Team, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA. Technical Report CMU/SEI‑
2001 -TR-034, ESC-TR-200 1-034, pp 1-49, (2001).
SEI 2	“The Capability Maturity Model: Guidelines for Improving the
Software Process”, by Carnegie Mellon University Software Engineering Institute, Addison Wesley Longman, (1994), ISBN 0- 201-54664-7.
SEI 3	“Process Maturity Profile: CMMI v1.1, SCAMPISM v1.1 Class A
Appraisal Results”, The Software Engineering Institute, August 2004, www.sei.cmu.edu/sema/profile.html
SPIRE 1998	“The SPIRE Handbook: Better, Faster, Cheaper software
development in small organisations”, ISBN 1-874303-02-9, Centre for Software Engineering Ltd, Dublin, 1998.
Wilkie 2004	Wilkie, F.G., McFall, D. and McCaffery, F. “The Centre for
Software Process Technologies: A model for process improvement in geographical regions with small software industries”, in proceedings of 16th Software Engineering Process Group Conference (2004), 8-11 March, Orlando, Florida, 5 pages.