5. User Testing and Feedback

This chapter will examine some of the information obtained from user-testing, and some of the modifications made to the package as a result.


5.1 General Results and Changes

5.1.1 Hardware & Computer Experience

The original intention was that the package should run on a Macintosh with a mouse, since all desktop Macintosh systems have one. When the initial pilot studies took place the chosen computer system was a Macintosh Portable. This machine had an integral track ball instead of a mouse, as do most current Macintosh notebook models. It was found that for the target user group a track ball was easier to use than a mouse, and could in fact be used by those who had severe physical disabilities, or by the very young.

As development work on the package progressed, and the use of QuickTime and colour pictures was introduced, the Macintosh Portable became unsuitable, since it could support neither of these, and a newer Macintosh Duo230 notebook with a grey-scale display was used instead. There were some problems with the keyboard, particularly the spacebar not always being responsive. This was easily solved, since if the spacebar did not leave a space, a second, firmer press of it would leave one. Also there were potential problems with a smaller track ball. A smaller track ball was a consequence of the physical size of the computer, which was about 25% of that of the Portable. The track ball had to be smaller to fit the machine. The problem with the track ball was overcome through training. The small track ball turned out to be easy to use if the child had practise at it. For a small group of children who did have problems, a larger separate track ball was used which could be plugged into the computer.

Throughout the piloting only one child was found who had had no computer experience. No obvious difficulties were encountered by children with learning disabilities when using either the package or the equipment.

5.1.2 Use of Consistent Gender Throughout the Package

Originally, some of the modules of the the package presented the child using it with scenes which depicted children of either gender. In some of the scenes used in the Emotions module, it was found that some boys commented on the presence of girls in the scenes, and that the girls would cry, whereas the boys would respond in a more "macho" fashion. In order to correct this bias, it was decided to present figures of the same gender as the child throughout the scenes, and this is now automatic following the introduction module.

5.1.3 Size of Screen Buttons (OK etc.)

Some children had difficulty with accurate positioning of the cursor, particularly those children with physical disabilities. For this reason the size of some of the screen buttons was increased to help the hit-rate. Owing to the decision that the package must work on a small-screen "classic" Macintosh , it was not possible to make the buttons too big, since that would use up too much of the screen that could be better utilised for other kinds of display. Recently, Apple Computer have discontinued sales of classic Macintoshes, and the package has been adapted to utilise slightly more screen space if it is available (though this is an option that is under user control).

5.1.4 Wording of Questions

It is very important in the writing of the package, the drafting of the user-messages, and the production of the documentation, to use language in a consistent manner when describing the different elements of the package. Since most of the package elements owe their allegiance to the fields of Psychology or Computer Science, it is possible to fall into the trap of using jargon, which means something completely different to the user. For example, it is alleged in computer support circles that some inexperienced computer users have spent a fair amount of time searching for the "any" key when prompted to "press any key" by a computer program! Fortunately, having a development team with diverse computer experience helped to remove some of the opportunities for jargon to creep in. When the package was adapted to use QuickTime sign-language video, some of the questions to the child were rewritten in order to clarify the "jargon" that had been used. One message to the child used to be
"Click on the OK button"
and was rewritten as
"Click on the OK box"
since it was discovered that there was confusion over the concept of "buttons" on screen, keys on the keyboard, and the button on the mouse or track ball. Some of the QuickTime video could also be used to indicate where things on the screen are, with respect to the signer. In one of the video clips the signer is asking about a place, and as she does so, she points over her shoulder to the position of the picture on the display.

5.1.5 Modules/Menu-Items/Package-Structure Names

As some of the modules were used it became apparent that they could be applied in slightly different ways from those initially expected. For example, originally the family module was designed to allow the child to select the family in their current home. As other needs became apparent it was then adapted to allow the selection of individuals, and then to do so without the requirement of there being an associated house. This meant that the concept "family" was no longer applicable. As a result of this the family module was renamed the people module. A selection of people in the package is now termed a set rather than a family. The structure that holds information about the members of a set includes space for a name and possibly a picture for a setting associated with the set.


5.2 Results From the Emotions Module


(see section 4.2 for a list of package modules and short description of each one.)


fig. 5.1 The current Emotions Palette.




5.2.1 Problems with Recognising Expressions

At an early stage in the project, pencil and paper tests were performed where children were asked to draw pictures of some of the seven different emotions utilised in the package (figure 5.1). The children were given words describing the expressions and asked to draw an expression based on the description. The children's drawings (figure 5.2) then provided the basis for the facial expressions used in the Emotions Modules.





fig 5.2 Examples of children's drawings of "happy", "ordinary" and "sad" faces.



The next stage was to ask groups of children to use a word or a couple of words to describe the seven different expressions. This was done and the resulting phrases were classified according to the following criteria:

hit - child successfully identified the expression and gave a meaningful description.

consistent - while not necessarily being the usual choice, there is an obvious relation with the usual one.

idiosyncratic - not obviously related to the usual meaning for such an expression.
The interpretation of some of the expressions gave cause for concern, especially the neutral face and the frightened face, and these had to be adapted (see section 6.4.2). Examples of the descriptions of the expressions are given in table 5.1, and charts of the overall results of this classification are presented in figures 5.3 to 5.6.


Hit Consistent Idiosyncratic
Very Happy
(face 1) 		Very Happy
Joyful
Boss
A Lot Happy
Dead Happy
Delighted
Really Cheerful 		Happy
Excited
Laughing
Cheerful
Smiling
Jolly 		Proud
Normal
(face 3) 		OK
Serious
Straight Face
Normal
Alright 		Bored
Solemn
Not Too Happy
Not That Happy
Not Happy
A Little Bit Happy
Reasonably Happy
Not Sure
Quite Sad
Calm
Fed Up
Dull
Fine 		Miserable
Puzzled
Sad
Grumpy
Lonesome
Unhappy
Worried
Happy
Not Happy
Sick
Alarm
Left Out
Terrified
Strange
Pleasant
Blase

table 5.1 List of emotions attributed to the original versions of the "Very Happy" and "Normal" faces.



fig. 5.3 Pie chart of classification of children's responses to the original "Neutral" face.



fig. 5.4 Pie chart of classification of children's responses to the modified "Neutral" face.



fig. 5.5 Pie chart of classification of children's responses to the "Very Happy" face.



fig. 5.6 Pie chart of classification of children's responses to the "Angry" face.



Thus the emotions module gives an insight into the language of the child, since some of the subjects use idiosyncratic words("boss" and "joyful" for face one, "feeling sick" for face six, "jerked" and "crazed" for face seven), and also gives an indication of how quickly they make up their minds when their reaction time is measured. It was also possible to establish if they were anxious about using computers. Some were very careful about their response ("reasonably happy" for face two in figure 5.1), some subjects would give more than one word or phrase to describe their selected face.

5.2.2 Scenes Used as a Discussion Point

Appendix A gives a complete listing of all the standard scenes used in the Emotions module. The school scene presented in the Emotions module has been used to get the child to talk about various aspects of the curriculum. A response of "I don't like school" has been associated with a scared expression, and has been found to be associated with difficulties that the child has in certain subjects. One child did not like copying information from the board because, having read it and then looked down to his book, he found it difficult to remember what he had seen.

The scene with the dog is an ice-breaker. It gives the child a chance to talk about their own dog. Sometimes a child has asked if the dog in the picture is meant to represent their own dog, or a stranger's dog. The choice of interpretation is really up to the child, and does not affect the results of the rest of the package.

Some scenes produced results that were highly consistent - 98% chose the very-happy expression for the birthday scene. Of those that did not, one subject in a children's home did not get presents from everyone, and another got presents they did not like.

5.2.3 Some Scenes Thought of as a Matching Task

All scenes contain a view of a child without an expression. Some scenes also contain representations of other children with expressions. When presented with such scenes, some children perceive the task to be to choose an expression for the child without one, that matches the ones on the rest of the children in the scene. Although this is worth noting, it is not thought of as being a problem by the team at Liverpool.


5.3 Results From the Buildings Module



5.3.1 Replacement of the Scrapbook

Despite the fact that the scrapbook, used to permit the choice of a house, is similar to standard Apple interfaces, it was found to be sufficiently awkward for children to render it impractical for use within this project. A particular problem is that children find it difficult to remember all the possibilities when they make their choice. Basing choice on comparisons is much simpler when the things that are being compared are all in view, and this is a standard HCI technique. Another problem is that some of the proffered alternatives are too specific. Some children selected a house based on a particular aspect of it, ignoring one which was of a more appropriate type, but which lacked a certain feature. This rendered the scrapbook an inaccurate tool. For these reasons the tool was replaced, and a simpler hierarchical tool is used instead. Specific details of the new tool are given in section 6.5.3.

5.3.2 Difficulties with Building-Name Question

The question "What name do you call this place?" caused frequent problems in that children failed to understand what was required. The question was included to allow a child to label a place with a name such as "Jon's house" or "Uncle Frank's House" or to give the house name if it had one. It was thought better to allow a degree of freedom in interpreting the question rather than seek to elicit a specific response which might be inappropriate, and so the question was left as it was.

5.3.3 Personalising Building Representations

A pen and paper exercise was carried out to discover what important features children would use to describe their house. By examining the information gathered we will be able to present a set of items for personalising a picture to make our representations more life-like and hence more acceptable to children. Three examples of the responses appear below.

Example 1. A girl aged 10.
1. A noisy area. A lot of people walk past.
2. It's above a laundrette.
3. It has two bedrooms.
4. My home is quite near to the school.
5. It has five rooms altogether.
6. It has got a big landing.
7. It has got a big living room.
8. It's a big flat.
9. Got one pet.
10. It has a balcony that you climb on to out of a window to hang your washing out.

Example 2. A girl aged 10.
1. Got a wall round it.
2. Painted black and white.
3. A Widnes sign outside.
4. 3 bedrooms.
5. Garage.
6. Big garden.
7. Shed.
8. Field next to it.
9. A Lane next to it.
10. Extension.

Example 3. A boy aged 10.
1. It's a bit big.
2. It's only got 5 rooms.
3. It's not very tidy because my books are everywhere.
4. It's got a marvellous view.
5. It's in the 4th floor of the I M Marsh Campus.
6. It's the last flat on the fourth floor.
7. The Living room isn't very big.
8. The kitchen is small.
9. The bedroom is bigger than the kitchen.
10. It's number is 4.

From the results illustrated in the examples above, some children refer to physical attributes of the place, and some to where a place is located.

5.4 Results From the Families Module



5.4.1 Selection of Appropriate Representations

Some children tended to choose representations that were more "life-like" than might be expected from those on offer (e.g. Dad is bald so the "grandfather" figure is chosen rather than the male adult figure, sister has a "perm" so the "Grandmother" figure is selected - figure 5.7). Others chose representations in a strict chronological order and discussed why they were choosing the ones they did (e.g. the medium boy figure because he is younger than large boy figure, even though the real-life person represented by the medium boy figure is taller - see figure 5.8).


fig. 5.7 Grandmother figure from the standard family.


fig. 5.8 small, medium and large boy figures in the standard family.

5.4.2 Positioning of Figures

Some children commented that they were unhappy for the baby figure to be on the floor, preferring it to be held in someone's care.


5.4.3 Confusion over Instructions to the Child

Confusion arose over the task where children thought that they were allowed to use a figure to represent a family member only once. The verbal instructions from the interviewer were modified to include the fact that a figure could be chosen more than once to represent more than one different person.

5.4.4 Requests for Extra Representations

Shortcomings in the representations were pointed out - that there were no ethnic figures and no-one in a wheelchair.
As well as the addition of a cat and dog, there have been requests for other pets, including fish, birds and a snake.

5.4.5 Sensitivity of Questions

The opening question to the module is "Who lives in this place?". This must be used carefully: one child's father had just recently left and she was upset about this.

5.4.6 Enhanced Reporting Accuracy

One child who had previously been interviewed by a member of the Liverpool team in the usual non-computer-based way, was interviewed with this computer-based system, and reported extra family members. These were "old men" figures and "baby" figures. When asked about the additions, it was discovered that his mother had fostered a number of children, and next door was a Senior-Citizen's Home, some of whose residents the child thought of as grandparents.

5.4.7 Inclusion of Deceased Family Members

Some children want to include deceased family members, including siblings and grandparents. One boy who had two brothers wanted to include a sister who had died, but who had been very important to him. The other brothers, however, did not mention the deceased sister.

5.4.8 Inclusion of Self in the Family

When creating the family, some children automatically include themselves, while others ask if they should. In order to check whether the child was included or not, a module was written which asks the child whether they have included themselves, and invites them to should they wish to. If they have already included themselves it prompts them to indicate which of the family members they have picked to represent them.


5.5 Results From The Emotions and People Module



The original question asked in the emotions and people startup section was "Have you ever felt like this with a member of your family?" Some children would not admit to feeling bad with some people, though there was evidence that there had been difficulties.


5.6 Results From the Somatic Experiences Module



5.6.1 Pen and Paper Studies to Determine Suitable Pain Representations

A number of pen and paper studies were carried out with groups of children to assess the suitability of representations for pain-types. The question asked was
"Remember the last time you were in pain. Write down what caused it, then shade or mark the place you had the pain".
Three example responses are shown in figure 5.9. A number of different marks to represent pain were produced by children. Some of these were adapted for use in the pain palette in the package. See figure 5.10 for a depiction of the default pain palette.

I fell off a wall and I fell and got my leg caught in the wheel of my bike. I was climbing a conker tree I had migraine and I had pains in my waist.

fig. 5.9 Three example responses to the pain questionnaire.



1. spikes 2. spots 3. lump
4. lightning 5. cloud 6. rectangle
7. 45 degree line 8. fading spikes
fig 5.10 pain palette & description of pain-types.


The pain-types offered have changed in a number of ways as the package developed - this included a rectangle and a 45deg. line which were added as a result of work with orthopaedic children who requested some alternative pain-type shapes. The pain-types may be used by the child in whatever way they think is appropriate, and there is no right or wrong choice. Table 5.2 lists the team's suggested descriptions for the pain-types offered in the default palette.

Type Description when still Description when animated
1. Spikes sharp, piercing. jabbing.
2. Spots 2. spots, small lumps. 2. itchy, ticklish.
3. Lump 3. hard lump. 3. throbbing pain.
4. Lightning sharp, piercing. acute jabbing pain.
5. Cloud diffuse. tickly, diffuse.
6. Rectangle graze. graze.
7. 45 degree line cut. no animated representation
8. Fading spikes sharp, piercing. fading, sharp pain.
table 5.2 Default pain-types: suggested description.



5.6.2 Size/Intensity Palette

The Size/Intensity palette was originally intended to convey the size of the area of pain, with the intention of introducing a separate palette for pain intensity based on selecting a colour. From tests, it was discovered that the size palette is used both for size and intensity. No separate intensity palette has yet been tried.

5.6.3 Laterality Problems

There is a problem of laterality - does the child see the picture of the front and back body views as being the view in a mirror, transposing the left and right sides? After indicating the pain site with the package, children have been asked to point to the same area on their own bodies. Some children transpose the left and right sides, but for purposes of this study this has not been a problem. If it was thought to be useful, the left and right sides of the body-views could be labelled to aid identification.


5.6.4 Children's Descriptions of Our Pain-Types

Children used the module, and were asked to give more information about the pain sites that they had marked. Their descriptions of pains were matched with the representation that they chose and the results appear in table 5.3. Unlike the emotions in the emotions palette, there is no concept of right or wrong for the choice of pain representations.

Pain 1
Fracture
stinging
dislocation
cracked skull
physio
like electric shock
pin in bone
shoulder pain, cramp
arthritis
headache
smack into wall
bang on cot sides
banged plaster
agony, touched muscle 		Pain 2
fracture
sore throat
twisted ankle
skin graft
tummy (constipation)
bed sore in the morning
tonsils 		Pain 3
stitches
scar
boob lump
cramp
headache
tummy ache
Pain 4
ache
broken bone
stitches out
sting
migraine
stomach, digging into me
bone come out of leg
bone graft
shoulder strain
cramp
fall
pin prick
pin in leg
Perthes disease 		Pain 5
drip in hand
back ache
ache
dull pain 		Pain 6
sore elbows-pressure sore
fall off bike
Pain 7
sore elbow - a pressure sore
stiff finger
itch
table 5.3 - literal descriptions given by children for pain-types in the pain palette



5.6.5 Throb Control - Slide Control vs. Buttons

The slide control, as with the scrapbook, is a Macintosh interface element which has proved to be an inappropriate choice for children. Although the slide control facilitates a range of choices, it is difficult to use it to adjust the throb speed it in a way that is self-explanatory. An alternative throb control is discussed in section 6.9.2.

5.6.6 Effects of Reporting Historical and Current Pain

Very often, children said that they weren't currently in pain even though in one data-gathering exercise the children in question were in the orthopaedic ward of a hospital, and had pins and bolts through their legs. To combat the general under-reporting of pain, the children were asked first about pains that had occurred historically. Having discussed and labelled all of those, the subject of current pain was considered. Although this increased the level of reporting of current pain there were still children who denied ever having been in pain. One way of prompting them to realise that this was not true was to first ask if they had ever had stomach ache, toothache, or a sore-throat. Informal results show that if the child considers historical pain before considering current pain there is more pain reporting compared to when current pain is considered first and historical pain last.

Sometimes pains that are recorded as being current are obviously minor compared to other pain that a child must be feeling, yet the child would not report the obvious major source of pain, such as pins and needles reported by children in traction in an orthopaedic ward.

5.6.7 Use of the Tool by Children With Learning Difficulties

The somatic experiences tool was also usable by children with learning difficulties. One child who had been repeatedly taking his clothes off over a period of several days, reported an itchy pain site using the module which he had not reported to anyone else. It transpired that he had neuro-fibromatosis and had developed a new site of "spots", which none of his carers had noticed, and which he had not mentioned to anyone. The spots caused discomfort when his clothes rubbed on them. The same child also reported a highly inflamed ingrown toe-nail, about which no-one had previously known.

5.6.8 Physical vs. Emotional Pain

Very few children referred to emotional pain. Out of 80 children using the tool, only one reported the loss of a grandmother as a pain.

5.6.9 Repeated Use of the Tool

When the tool was used on further occasions by the same children they appeared anxious to remember what pain shapes and sites they chose on previous occasions so that they could choose them again. Some children considered very carefully before choosing representations. It was expected that when children with resolving pain used the tool again, they would choose the same pain representations, but in decreasing size. In reality this rarely happened, but since the population of children with repeating use of the tool is small, the results are inconclusive.


5.7 Summary



Piloting some of the package modules has produced some interesting and thought-provoking results. Some of the results have made apparent the need for changes to some aspects of the interface, and others have shown aspects of the interface to be very successful. The next chapter examines some of the changes that have been made to the package, from minor modifications in most cases, to the complete redesign of the interface for two modules (Scrapbook and Emotions and People).
back to Chapter 4. forward to Chapter 6