Software Evaluation
An IT system will often be ordered and designed complete so that hardware, software, operating procedures and documents used within the system form an integrated whole. There will however be situations where either new software is installed to extend the range of tasks carried out by the system or existing software is replaced or upgraded.
The organisation may have a choice of a number of different applications from various suppliers, each of which might claim to perform the required task. Some method is needed whereby the organisation can evaluate the different packages under offer to decide which one most appropriately matches their particular needs.
Note that this evaluation process is important for all types of software purchase. Aspects of it will apply to software that is being specially written in the same way as it applies to of the shelf packages.
The process of evaluation begins with the identification of the end users needs. No proper decision can be made unless there is a clear idea of the exact tasks that must be performed by the software. Knowing what functionality the user requires will give an idea of the type of package required; the users experience and the way in which the package will operate will indicate the type of user interface needed. Once this starting point is established it will be possible to identify software that matches the user's needs.
Once the user's requirements are known, the following methods can be used to identify software that may be applicable to a particular problem:
- Use Expertise - hire a consultant or training agency or by contact experienced users possibly via a user group on Internet
- Personal Experience - obtain a trial version. Some manufacturers will supply a cut down or time-limited version of software to allow evaluation to take place.
- Written Experience - consult magazines (IT or trade), training manuals or books.
Evaluation Criteria
These methods will allow a decision to be made as to whether or not a particular package will fulfill a particular need. In most situations a number of packages will be found that appear to match the user requirement. Some method is needed of measuring which of the available packages is best suited to the task. This is done by measuring the performance of the different packages against a set of evaluation criteria. The weighting given to each criterion will vary according to the individual circumstances.
| Functionality. | Does the software perform the functions required. It may not be necessary for the software to provide all desirable functions but it must at least deliver functions identified as essential. The match for functionality will refer back to the agreed problem specification. Additional functionality may be useful for planned future needs but it could also confuse the user by presenting too many choices. |
| Performance. | Given that the software provides various functions, how well does it do this. The best way of measuring performance is to have a series of standardised benchmark tasks which each package can be applied to. These benchmarks should reflect the type of task undertaken in both normal and extreme operation. |
| User Interface. | How easy is the user interface to operate and can existing expertise be utilized? Is there on line help and is it context sensitive? |
| Compatibility. | How compatible is the software with existing software. A lack of compatibility will lead to problems in information flow. Will existing data need to be converted? Can data be imported and exported and if so what standard formats are supported - do these match requirements? |
| Training and Support. | What standard of documentation is available. Are there training resources for the software and what quality of user support is provide and at what cost? |
| Robustness. | How robust is the package - what version is it? Is it prone to crash? |
| Cost Benefit. | The cost of a software package is, on its own, irrelevant. What is most important is the cost compared to the financial gains that use of the software will result in. Gains may be measurable in terms of reduced staffing or improved productivity or they may be indirect for example better customer relations produced by a more rapid response to enquiries. |
Evaluation Report
Once the different software options have been evaluated then an evaluation report will be produced. The function of this report is to provide the basis for making the final selection of the software to be used.
The report will begin by outlining the context of the problem. It will contain a section identifying the methodology used in evaluating the software and provide a summary of the actual evaluation - giving figure or using charts where appropriate.
Finally the report will make recommendations based on the evaluation and the perceived needs of the user. Each recommendation will be supported by a justification, which will relate the needs of the user to the evaluation performance of the software.
©LEV