ti trovi in: Home Page - Biblioteca - Documentazione - Study on the guidelines for website accessibility -

3 - Subjective check

3.3 - Evaluation process

The process of evaluating the degree to which an environment corresponds to the essential criteria focuses on the iteration points indicated in the methodology and is carried out as follows:

1. human factor analysis by one or more experts;
2. tests with users;
3. data interpretation by the expert and concluding report with indication of the quality level achieved.

 

3.3.1 - Analysis carried out by human factor experts

The evaluation by one or more experts will consist essentially of a cognitive walk-through by which the expert constructs usage scenarios for the environment and interface that simulate user behaviour at the cognitive level.

Essentially, this involves evaluating where the environment facilitates or obstructs a given action by the user to achieve an objective.

The expert will therefore need to know what services and information the environment is intended to provide and the actions the user needs to carry out to achieve these objectives through the interface. He will also need information on potential users and the experience and knowledge requested for them to interact with the environment.

This part of the evaluation concludes with the allocation to each of the 11 criteria of a score on a rising scale from 1 to 5 where:

1 = no match between the environment and the criterion under examination;

2 = poor match between the environment and the criterion under examination;

3 = adequate match between the environment and the criterion under examination;

4 = good match between the environment and the criterion under examination;

5 = very good match between the environment and the criterion under examination;

In general, human factor experts must have the following training and work experience:

• university education including one year of training in one or more of the following aspects of ergonomics:
  • environmental ergonomics;
  • hardware ergonomics;
  • cognitive ergonomics;
  • macroergonomics.
• proven professional experience with at least one year under the supervision of an expert in human factor analysis, ideally with relevant certification.

 

3.3.2 - Evaluation carried out with users

The second part of the evaluation process envisages the creation of a user panel that includes disabled users using their own assistive technologies.

The panel carries out a series of tests based on interaction with the environment. The tests can be carried out:

• freely: users navigate the site without specific tasks;
• by objectives: users carry out specific tasks.

In carrying out the tests, the user panel is guided by the human factor expert.

When the test involves free navigation, the expert notes the users’ comments and observations on their conduct on a form.

When the test involves specific tasks, the expert records the type of task, the amount of time required to perform it and any errors committed, and notes users’ comments and observations on their conduct.

This part also concludes with an evaluation using a subjective scale analogous to the one described in the previous point.

 

3.3.3 - Data processing and final report

The final stage of the evaluation is to draw up a report in which the expert indicates:

• the subjective scale-based evaluation resulting from the cognitive walk-through;
• his qualitative considerations;
• the data concerning user performance in the assigned tasks: performance measurements, comments, observations on user behaviour;
• responses to evaluation questionnaires completed by users;
• the overall evaluation of the quality level achieved, in accordance with the following schema:
  • average overall value less than 2 = no quality;
  • average overall value greater than or equal to 2 but less than 3 = first quality level;
  • average overall value greater than or equal to 3 but less than 4 = second quality level;
  • average overall value greater than or equal to 4 = third quality level.