5. User Testing and Feedback
This chapter will examine some of the information obtained from user-testing, and
some of the modifications made to the package as a result.
5.1 General Results and Changes
5.1.1 Hardware & Computer Experience
The original intention was that the package should run on a Macintosh with a mouse,
since all desktop Macintosh systems have one. When the initial pilot studies took
place the chosen computer system was a Macintosh Portable. This machine had an
integral track ball instead of a mouse, as do most current Macintosh notebook models.
It was found that for the target user group a track ball was easier to use than a
mouse, and could in fact be used by those who had severe physical disabilities,
or by the very young.
As development work on the package progressed, and the use of QuickTime and colour
pictures was introduced, the Macintosh Portable became unsuitable, since it could
support neither of these, and a newer Macintosh Duo230 notebook with a grey-scale
display was used instead. There were some problems with the keyboard, particularly
the spacebar not always being responsive. This was easily solved, since if the spacebar
did not leave a space, a second, firmer press of it would leave one. Also there were
potential problems with a smaller track ball. A smaller track ball was a consequence
of the physical size of the computer, which was about 25% of that of the Portable.
The track ball had to be smaller to fit the machine. The problem with the track ball
was overcome through training. The small track ball turned out to be easy to use if
the child had practise at it. For a small group of children who did have problems, a
larger separate track ball was used which could be plugged into the computer.
Throughout the piloting only one child was found who had had no computer experience.
No obvious difficulties were encountered by children with learning disabilities
when using either the package or the equipment.
5.1.2 Use of Consistent Gender Throughout the Package
Originally, some of the modules of the the package presented the child using it with scenes which
depicted children of either gender. In some of the scenes used in the Emotions module, it
was found that some boys commented on the presence of girls in the scenes, and that the girls
would cry, whereas the boys would respond in a more "macho" fashion. In order to correct this
bias, it was decided to present figures of the same gender as the child throughout the scenes, and
this is now automatic following the introduction module.
5.1.3 Size of Screen Buttons (OK etc.)
Some children had difficulty with accurate positioning of the cursor, particularly those
children with physical disabilities. For this reason the size of some of the screen buttons
was increased to help the hit-rate. Owing to the decision that the package must work on a
small-screen "classic" Macintosh , it was not possible to make the buttons too big, since
that would use up too much of the screen that could be better utilised for other kinds of display.
Recently, Apple Computer have discontinued sales of classic Macintoshes, and the package has
been adapted to utilise slightly more screen space if it is available (though this is an option
that is under user control).
5.1.4 Wording of Questions
It is very important in the writing of the package, the drafting of the user-messages, and the
production of the documentation, to use language in a consistent manner when describing the
different elements of the package. Since most of the package elements owe their allegiance to
the fields of Psychology or Computer Science, it is possible to fall into the trap of using
jargon, which means something completely different to the user. For example, it is alleged in
computer support circles that some inexperienced computer users have spent a fair amount of time
searching for the "any" key when prompted to "press any key" by a computer program!
Fortunately, having a development team with diverse computer experience helped to remove some
of the opportunities for jargon to creep in. When the package was adapted to use QuickTime
sign-language video, some of the questions to the child were rewritten in order to clarify the
"jargon" that had been used. One message to the child used to be
"Click on the OK button"
and was rewritten as
"Click on the OK box"
since it was discovered that there was confusion over the concept of "buttons" on screen,
keys on the keyboard, and the button on the mouse or track ball. Some of the QuickTime
video could also be used to indicate where things on the screen are, with respect to
the signer. In one of the video clips the signer is asking about a place, and as she
does so, she points over her shoulder to the position of the picture on the display.
5.1.5 Modules/Menu-Items/Package-Structure Names
As some of the modules were used it became apparent that they could be applied
in slightly different ways from those initially expected. For example, originally the
family module was designed to allow the child to select the family in their current
home. As other needs became apparent it was then adapted to allow the selection of
individuals, and then to do so without the requirement of there being an associated
house. This meant that the concept "family" was no longer applicable. As a result of
this the family module was renamed the people module. A selection of people in the
package is now termed a set rather than a family. The structure that holds information
about the members of a set includes space for a name and possibly a picture for a
setting associated with the set.
(see section 4.2 for a list of package modules and short description of each one.)
fig. 5.1 The current Emotions Palette.
5.2.1 Problems with Recognising Expressions
At an early stage in the project, pencil and paper tests were performed where children
were asked to draw pictures of some of the seven different emotions utilised in the package
(figure 5.1). The children were given words describing the expressions and asked to draw an
expression based on the description. The children's drawings (figure 5.2) then provided the
basis for the facial expressions used in the Emotions Modules.
fig 5.2 Examples of children's drawings of "happy", "ordinary" and "sad" faces.
The next stage was to ask groups of children to use a word or a couple of words to describe
the seven different expressions. This was done and the resulting phrases were classified
according to the following criteria:
hit - child successfully identified the expression and gave a meaningful
consistent - while not necessarily being the usual choice, there is an
obvious relation with the usual one.
idiosyncratic - not obviously related to the usual meaning for such an expression.
The interpretation of some of the expressions gave cause for concern, especially the neutral
face and the frightened face, and these had to be adapted (see section 6.4.2). Examples of
the descriptions of the expressions are given in table 5.1, and charts of the overall
results of this classification are presented in figures 5.3 to 5.6.
A Lot Happy
Not Too Happy
Not That Happy
A Little Bit Happy
table 5.1 List of emotions attributed to the original versions of the "Very Happy" and "Normal" faces.
fig. 5.3 Pie chart of classification of children's responses to the original "Neutral" face.
fig. 5.4 Pie chart of classification of children's responses to the modified "Neutral" face.
fig. 5.5 Pie chart of classification of children's responses to the "Very Happy" face.
fig. 5.6 Pie chart of classification of children's responses to the "Angry" face.
Thus the emotions module gives an insight into the language of the child, since some of
the subjects use idiosyncratic words("boss" and "joyful" for face one, "feeling sick"
for face six, "jerked" and "crazed" for face seven), and also gives an indication of
how quickly they make up their minds when their reaction time is measured. It was
also possible to establish if they were anxious about using computers. Some were
very careful about their response ("reasonably happy" for face two in figure 5.1),
some subjects would give more than one word or phrase to describe their selected face.
5.2.2 Scenes Used as a Discussion Point
Appendix A gives a complete listing of all the standard scenes used in the Emotions
module. The school scene presented in the Emotions module has been used to get the child to
talk about various aspects of the curriculum. A response of "I don't like school" has been
associated with a scared expression, and has been found to be associated with difficulties
that the child has in certain subjects. One child did not like copying information from the
board because, having read it and then looked down to his book, he found it difficult to
remember what he had seen.
The scene with the dog is an ice-breaker. It gives the child a chance to talk about their
own dog. Sometimes a child has asked if the dog in the picture is meant to represent their
own dog, or a stranger's dog. The choice of interpretation is really up to the child, and
does not affect the results of the rest of the package.
Some scenes produced results that were highly consistent - 98% chose the very-happy expression
for the birthday scene. Of those that did not, one subject in a children's home did not get
presents from everyone, and another got presents they did not like.
5.2.3 Some Scenes Thought of as a Matching Task
All scenes contain a view of a child without an expression. Some scenes also contain
representations of other children with expressions. When presented with such
scenes, some children perceive the task to be to choose an expression for the child without
one, that matches the ones on the rest of the children in the scene. Although this is worth
noting, it is not thought of as being a problem by the team at Liverpool.
5.3.1 Replacement of the Scrapbook
Despite the fact that the scrapbook, used to permit the choice of a house, is similar
to standard Apple interfaces, it was found to be sufficiently awkward for children to render
it impractical for use within this project. A particular problem is that children find it
difficult to remember all the possibilities when they make their choice. Basing choice on
comparisons is much simpler when the things that are being compared are all in view, and
this is a standard HCI technique. Another problem is that some of the proffered alternatives
are too specific. Some children selected a house based on a particular aspect of it,
ignoring one which was of a more appropriate type, but which lacked a certain feature.
This rendered the scrapbook an inaccurate tool. For these reasons the tool was replaced,
and a simpler hierarchical tool is used instead. Specific details of the new tool are
given in section 6.5.3.
5.3.2 Difficulties with Building-Name Question
The question "What name do you call this place?" caused frequent problems in
that children failed to understand what was required. The question was included to
allow a child to label a place with a name such as "Jon's house" or "Uncle Frank's
House" or to give the house name if it had one. It was thought better to allow a degree
of freedom in interpreting the question rather than seek to elicit a specific response
which might be inappropriate, and so the question was left as it was.
5.3.3 Personalising Building Representations
A pen and paper exercise was carried out to discover what important features
children would use to describe their house. By examining the information gathered we
will be able to present a set of items for personalising a picture to make our
representations more life-like and hence more acceptable to children. Three examples
of the responses appear below.
Example 1. A girl aged 10.
1. A noisy area. A lot of people walk past.
2. It's above a laundrette.
3. It has two bedrooms.
4. My home is quite near to the school.
5. It has five rooms altogether.
6. It has got a big landing.
7. It has got a big living room.
8. It's a big flat.
9. Got one pet.
10. It has a balcony that you climb on to out of a window to hang your washing out.
Example 2. A girl aged 10.
1. Got a wall round it.
2. Painted black and white.
3. A Widnes sign outside.
4. 3 bedrooms.
6. Big garden.
8. Field next to it.
9. A Lane next to it.
Example 3. A boy aged 10.
1. It's a bit big.
2. It's only got 5 rooms.
3. It's not very tidy because my books are everywhere.
4. It's got a marvellous view.
5. It's in the 4th floor of the I M Marsh Campus.
6. It's the last flat on the fourth floor.
7. The Living room isn't very big.
8. The kitchen is small.
9. The bedroom is bigger than the kitchen.
10. It's number is 4.
From the results illustrated in the examples above, some children refer to physical
attributes of the place, and some to where a place is located.
5.4.1 Selection of Appropriate Representations
Some children tended to choose representations that were more "life-like" than
might be expected from those on offer (e.g. Dad is bald so the "grandfather" figure is
chosen rather than the male adult figure, sister has a "perm" so the "Grandmother"
figure is selected - figure 5.7). Others chose representations in a strict chronological
order and discussed why they were choosing the ones they did (e.g. the medium boy figure
because he is younger than large boy figure, even though the real-life person represented
by the medium boy figure is taller - see figure 5.8).
fig. 5.7 Grandmother figure from the standard family.
fig. 5.8 small, medium and large boy figures in the standard family.
5.4.2 Positioning of Figures
Some children commented that they were unhappy for the baby figure to be on the floor,
preferring it to be held in someone's care.
5.4.3 Confusion over Instructions to the Child
Confusion arose over the task where children thought that they were allowed to use a
figure to represent a family member only once. The verbal instructions from the interviewer
were modified to include the fact that a figure could be chosen more than once to represent
more than one different person.
5.4.4 Requests for Extra Representations
Shortcomings in the representations were pointed out - that there were no ethnic
figures and no-one in a wheelchair.
As well as the addition of a cat and dog, there have been requests for other pets, including
fish, birds and a snake.
5.4.5 Sensitivity of Questions
The opening question to the module is "Who lives in this place?". This must be used
carefully: one child's father had just recently left and she was upset about this.
5.4.6 Enhanced Reporting Accuracy
One child who had previously been interviewed by a member of the Liverpool team in the
usual non-computer-based way, was interviewed with this computer-based system, and reported
extra family members. These were "old men" figures and "baby" figures. When asked about the
additions, it was discovered that his mother had fostered a number of children, and next
door was a Senior-Citizen's Home, some of whose residents the child thought of as grandparents.
5.4.7 Inclusion of Deceased Family Members
Some children want to include deceased family members, including siblings and grandparents.
One boy who had two brothers wanted to include a sister who had died, but who had been very
important to him. The other brothers, however, did not mention the deceased sister.
5.4.8 Inclusion of Self in the Family
When creating the family, some children automatically include themselves, while others ask
if they should. In order to check whether the child was included or not, a module was written
which asks the child whether they have included themselves, and invites them to should they
wish to. If they have already included themselves it prompts them to indicate which of the
family members they have picked to represent them.
The original question asked in the emotions and people startup section was "Have you ever
felt like this with a member of your family?" Some children would not admit to feeling
bad with some people, though there was evidence that there had been difficulties.
5.6.1 Pen and Paper Studies to Determine Suitable Pain Representations
A number of pen and paper studies were carried out with groups of children to assess the
suitability of representations for pain-types. The question asked was
"Remember the last time you were in pain. Write down what caused it, then shade or mark
the place you had the pain".
Three example responses are shown in figure 5.9. A number of different marks to represent
pain were produced by children. Some of these were adapted for use in the pain palette
in the package. See figure 5.10 for a depiction of the default pain palette.
|I fell off a wall and I fell and got my leg caught in the wheel of my bike.
||I was climbing a conker tree
||I had migraine and I had pains in my waist.
fig. 5.9 Three example responses to the pain questionnaire.
fig 5.10 pain palette & description of pain-types.
|7. 45 degree line
||8. fading spikes
The pain-types offered have changed in a number of ways as the package developed -
this included a rectangle and a 45deg. line which were added as a result of work
with orthopaedic children who requested some alternative pain-type shapes. The
pain-types may be used by the child in whatever way they think is appropriate, and
there is no right or wrong choice. Table 5.2 lists the team's suggested descriptions
for the pain-types offered in the default palette.
table 5.2 Default pain-types: suggested description.
||Description when still
||Description when animated
||2. spots, small lumps.
||2. itchy, ticklish.
||3. hard lump.
||3. throbbing pain.
||acute jabbing pain.
|7. 45 degree line
||no animated representation
|8. Fading spikes
||fading, sharp pain.
5.6.2 Size/Intensity Palette
The Size/Intensity palette was originally intended to convey the size of
the area of pain, with the intention of introducing a separate palette for pain
intensity based on selecting a colour. From tests, it was discovered that the size
palette is used both for size and intensity. No separate intensity palette has yet
5.6.3 Laterality Problems
There is a problem of laterality - does the child see the picture of the front
and back body views as being the view in a mirror, transposing the left and right
sides? After indicating the pain site with the package, children have been asked to
point to the same area on their own bodies. Some children transpose the left and right
sides, but for purposes of this study this has not been a problem. If it was thought
to be useful, the left and right sides of the body-views could be labelled to aid
5.6.4 Children's Descriptions of Our Pain-Types
Children used the module, and were asked to give more information about the pain
sites that they had marked. Their descriptions of pains were matched with the representation
that they chose and the results appear in table 5.3. Unlike the emotions in the emotions
palette, there is no concept of right or wrong for the choice of pain representations.
table 5.3 - literal descriptions given by children for pain-types in the pain palette
like electric shock
pin in bone
shoulder pain, cramp
smack into wall
bang on cot sides
agony, touched muscle
bed sore in the morning
stomach, digging into me
bone come out of leg
pin in leg
drip in hand
sore elbows-pressure sore
fall off bike
sore elbow - a pressure sore
5.6.5 Throb Control - Slide Control vs. Buttons
The slide control, as with the scrapbook, is a Macintosh interface element which has proved to
be an inappropriate choice for children. Although the slide control facilitates a range of
choices, it is difficult to use it to adjust the throb speed it in a way that is self-explanatory.
An alternative throb control is discussed in section 6.9.2.
5.6.6 Effects of Reporting Historical and Current Pain
Very often, children said that they weren't currently in pain even though in one
data-gathering exercise the children in question were in the orthopaedic ward of a hospital,
and had pins and bolts through their legs. To combat the general under-reporting of pain,
the children were asked first about pains that had occurred historically. Having discussed
and labelled all of those, the subject of current pain was considered. Although this increased
the level of reporting of current pain there were still children who denied ever having been in
pain. One way of prompting them to realise that this was not true was to first ask if they had
ever had stomach ache, toothache, or a sore-throat. Informal results show that if the child
considers historical pain before considering current pain there is more pain reporting compared
to when current pain is considered first and historical pain last.
Sometimes pains that are recorded as being current are obviously minor compared to other pain that
a child must be feeling, yet the child would not report the obvious major source of pain, such as
pins and needles reported by children in traction in an orthopaedic ward.
5.6.7 Use of the Tool by Children With Learning Difficulties
The somatic experiences tool was also usable by children with learning difficulties.
One child who had been repeatedly taking his clothes off over a period of several days, reported
an itchy pain site using the module which he had not reported to anyone else. It transpired that
he had neuro-fibromatosis and had developed a new site of "spots", which none of his carers had
noticed, and which he had not mentioned to anyone. The spots caused discomfort when his clothes
rubbed on them. The same child also reported a highly inflamed ingrown toe-nail, about which
no-one had previously known.
5.6.8 Physical vs. Emotional Pain
Very few children referred to emotional pain. Out of 80 children using the tool, only
one reported the loss of a grandmother as a pain.
5.6.9 Repeated Use of the Tool
When the tool was used on further occasions by the same children they appeared anxious to
remember what pain shapes and sites they chose on previous occasions so that they could choose
them again. Some children considered very carefully before choosing representations. It was
expected that when children with resolving pain used the tool again, they would choose the same
pain representations, but in decreasing size. In reality this rarely happened, but since the
population of children with repeating use of the tool is small, the results are inconclusive.
Piloting some of the package modules has produced some interesting and thought-provoking results.
Some of the results have made apparent the need for changes to some aspects of the interface, and
others have shown aspects of the interface to be very successful. The next chapter examines some
of the changes that have been made to the package, from minor modifications in most cases, to the
complete redesign of the interface for two modules (Scrapbook and Emotions and People).
back to Chapter 4. forward to Chapter 6