Facial image processing system

Claim 2

1 . A method of determining a head-pose measurement of a person, said head pose measurement including a rotational displacement and translation displacement, said method including the steps of: (a) capturing images of the face and head of said person from at least two viewpoints; (b) computing an initial estimate of head-pose; (c) computing the three dimensional location of two or more facial features; and (d) determining a rotational displacement and translation displacement which best match a head-model to the computed location of said two or more facial features.

Claim 3

2 . A method of determining a head-pose measurement of a person as claimed in claim 1 wherein step (b) includes the sub-step of, localising the face within two or more of the captured images.

Claim 4

3 . A method of determining a head-pose measurement of a person as claimed in claim 1 or claim 2 wherein step (c) includes the substeps of: identifying feature search regions, based on said initial estimate of head pose, within each captured image that may contain an identifiable facial feature; measuring the position of two or more facial features within two or more respective feature search regions in each of said captured images;

Claim 5

4 . A method of passively determining the direction of eye gaze of a person, said method including the steps of: (a) capturing images of the face and head of said person from at least two viewpoints; (b) localising the face within two or more of the captured images; (c) computing an initial estimate of head-pose; (d) identifying feature search regions, based on said initial estimate of head pose, within each captured image that may contain an identifiable facial feature; (e) measuring the position of two or more facial features within two or more respective feature search regions in each of said captured images; (f) computing the three-dimensional location of each of said two or more facial features; (g) determining a head pose measurement, said head pose measurement including a rotational displacement and translation displacement which best match a head-model to the computed location of said two or more facial features; (h) using said head pose measurement to determine the location the centre of one eyeball; (i) using said head pose measurement to find an eye region corresponding with said eyeball, within each captured image, and determining the location of the centre of the iris of said eye; (j) using the determined position of said centre of said eyeball and the determined location of the centre of the iris of said eye, to calculate an eye-gaze direction for said eye.

Claim 6

4 . A method of passively determining the direction of eye gaze of a person, said method including the steps of: (a) capturing images of the face and head of said person from at least two viewpoints; (b) localising the face within two or more of the captured images; (c) computing an initial estimate of head-pose; (d) identifying feature search regions, based on said initial estimate of head pose, within each captured image that may contain an identifiable facial feature; (e) measuring the position of two or more facial features within two or more respective feature search regions in each of said captured images; (f) computing the three-dimensional location of each of said two or more facial features; (g) determining a head pose measurement, said head pose measurement including a rotational displacement and translation displacement which best match a head-model to the computed location of said two or more facial features; (h) using said head pose measurement to determine the location the centre of one eyeball; (i) using said head pose measurement to find an eye region corresponding with said eyeball, within each captured image, and determining the location of the centre of the iris of said eye; (j) using the determined position of said centre of said eyeball and the determined location of the centre of the iris of said eye, to calculate an eye-gaze direction for said eye.

Claim 7

5 . A method of passively determining the direction of eye gaze of a person, wherein the direction of eye gaze of said person is calculated by averaging the direction of eye gaze of said person's two eyes, wherein the direction of eye gaze for each eye is determined using the method of claim 4 .

Claim 8

6 . A method of determining the eye closure amount of an eye located on a person's face, said eye having an eyeball, a top eyelid, and a bottom eyelid, said top and bottom eyelids meeting at a pair of eye corners; wherein eye closure amount is defined as the ratio of eye corner separation to eyelid distance, said method including the steps of: (a) capturing images of the face and head of said person from at least two viewpoints; (b) determining a head pose measurement, from said captured images; (c) using said head pose measurement to determine an eye region within said captured image; (d) determining the location of said eye corners within said eye region; (e) fitting at least two oriented edge detectors to said top eyelid and to said bottom eyelid, such that one of said oriented edge detectors of said top eyelid intersects with one oriented edge detectors of said bottom eyelid at each eye corner; (f) approximating the position of said top eyelid, and said bottom eyelid, by fitting a parabola to said at least two oriented edge detectors corresponding to each eyelid, such that each parabola is tangential to the at least two oriented edge detectors of said parabola's respective eyelid; (g) determining said eye corner separation by calculating the distance between said eye corners; (h) determining the maximum eyelid distance by calculating the maximum distance between said parabolas on the portion of said parabolas between said eye corners, in a direction perpendicular to a line joining the eye corners; (i) calculating the eye closure amount by taking the ratio of the determined eye corner separation to maximum eyelid distance.

Claim 9

6 . A method of determining the eye closure amount of an eye located on a person's face, said eye having an eyeball, a top eyelid, and a bottom eyelid, said top and bottom eyelids meeting at a pair of eye corners; wherein eye closure amount is defined as the ratio of eye corner separation to eyelid distance, said method including the steps of: (a) capturing images of the face and head of said person from at least two viewpoints; (b) determining a head pose measurement, from said captured images; (c) using said head pose measurement to determine an eye region within said captured image; (d) determining the location of said eye corners within said eye region; (e) fitting at least two oriented edge detectors to said top eyelid and to said bottom eyelid, such that one of said oriented edge detectors of said top eyelid intersects with one oriented edge detectors of said bottom eyelid at each eye corner; (f) approximating the position of said top eyelid, and said bottom eyelid, by fitting a parabola to said at least two oriented edge detectors corresponding to each eyelid, such that each parabola is tangential to the at least two oriented edge detectors of said parabola's respective eyelid; (g) determining said eye corner separation by calculating the distance between said eye corners; (h) determining the maximum eyelid distance by calculating the maximum distance between said parabolas on the portion of said parabolas between said eye corners, in a direction perpendicular to a line joining the eye corners; (i) calculating the eye closure amount by taking the ratio of the determined eye corner separation to maximum eyelid distance.

Claim 10

7 . A method of detecting a blink of a person's eye, said method including the following steps: calculating eye closure value using a method as claimed in claim 4; forming a temporal sequence of eye closure values; and comparing said temporal sequence of eye closure values with a blink template, wherein said blink template comprises a sequence of eye closure values corresponding to blink, in order to determine a correlation value between said blink template and said model temporal sequence of eye closure value, wherein a blink is determined to have occurred if said correlation value is greater than a predetermined threshold.

Claim 11

8 . A method of detecting a blink of a person's eye using a plurality of independent measurements, wherein each of said plurality of measurements produces a time varying output value, and at least one of said plurality of measurements is a measurement of eye closure value which has been calculated using a method as claimed in claim 6 , said method including the step of: combining the output value of two or more of said plurality of independent measurements in a weighted manner to calculate a blink detection value, wherein a blink is determined to have occurred if said blink detection value is greater than a predetermined threshold blink detection value.

Claim 12

9 . A method of detecting a blink of a person's eye as claimed in claim 8 wherein at least one of said plurality of independent measurements is selected from: an eye region template correlation coefficient; an optical flow measurement in said eye region; eye region image rate of change; colour segmentation within the eye region; and iris template correlation coefficient

Claim 13

10 . A method of determining an eye gaze direction of an observer comprising the steps of: (a) capturing at least one image of the observer and determining a head pose angle of said observer; (b) utilising said head pose angle to locate an expected eye position of said observer; and (c) analysing said expected eye position to locate at least one eye of said observer, and observing the location of the eye to determine an eye gaze direction.

Claim 14

10 . A method of determining an eye gaze direction of an observer comprising the steps of: (a) capturing at least one image of the observer and determining a head pose angle of said observer; (b) utilising said head pose angle to locate an expected eye position of said observer; and (c) analysing said expected eye position to locate at least one eye of said observer, and observing the location of the eye to determine an eye gaze direction.

Claim 15

11 . A method as claimed in claim 10 wherein said step (a) includes capturing an image in stereo and processing said image in stereo.

Claim 16

12 . A method as claimed in any previous claim 10 or claim 11 wherein said step (a) incudes the steps of: (i) determining a set of distinguishable feature locations on an observer's face; (ii) sorting candidate feature locations of a current head pose to determine a likely head pose.

Claim 17

13 . A method as claimed in any one of claims 10 to claim 12 wherein said method degrades gracefully in the presence of occlusion of the observer's face.

Claim 18

14 . A method as claimed in claim 13 wherein said occlusion includes the wearing of spectacles or sunglasses by a user.

Claim 19

15 . A method as claimed in any previous claim wherein said method further includes determining the three dimensional position of a centre of rotation of an observer's eye.

Claim 20

16 . A method as claimed in any previous claim wherein said method further determines eye closure and blinking.

Claim 21

17 . A method of detecting a flexible feature on a person's face, said method including the step of comparing the variance of said feature's position from an expected position of said facial feature, based on a head-pose measurement, with a threshold variance to determine whether the feature is flexible.

Claim 22

17 . A method of detecting a flexible feature on a person's face, said method including the step of comparing the variance of said feature's position from an expected position of said facial feature, based on a head-pose measurement, with a threshold variance to determine whether the feature is flexible.

Claim 23

18 . A method as claimed in claim 17 wherein said head-pose measurement is determined using the method of any one of claims 1 to 3 .

Claim 24

19 . A method of analysis of a person performing a task wherein the method of any one of the previous claims is used to analyse an action of said subject.

Claim 25

20 . A method as claimed in any one of claims 1 to 17 wherein said method utilises a measure feature location to determine if individual features on an observer's face are flexible or fixed.

Claim 26

21 . A method of animating an image, wherein the movement of a subject is tracked using a method according to any one of the previous claims, and wherein said image is animated in response to said movement.

Claim 27

22 . A method for measuring operator fatigue including the step of: tracking any one or more of: head-pose; eye gaze; eye closure amount; or blink rate of the operator.

Claim 28

22 . A method for measuring operator fatigue including the step of: tracking any one or more of: head-pose; eye gaze; eye closure amount; or blink rate of the operator.

Claim 29

23 . A method for measuring operator fatigue wherein head-pose is measured using the method of any one of claims 1 to 3 .

Claim 30

24 . A method for measuring operator fatigue wherein eye gaze is measured using the method of any one of claims 4 or 5 or 10 to 14 .

Claim 31

25 . A method for measuring operator fatigue wherein eye closure amount is measured using the method of claim 6 .

Claim 32

26 . A method for measuring operator fatigue wherein a blink rate is determined using the method of any one of claims 7 to 9 .

Claim 33

27 . An apparatus when implementing a method of any previous claim.

Claim 34

28 . A method of detecting eye, gaze substantially as hereinbefore described with reference to the accompanying drawings.

Claim 35

28 . A method of detecting eye, gaze substantially as hereinbefore described with reference to the accompanying drawings.

Claim 36

29 . A method of detecting head-pose substantially as hereinbefore described with reference to the accompanying drawings.

Claim 37

29 . A method of detecting head-pose substantially as hereinbefore described with reference to the accompanying drawings.

Claim 38

30 . A method of blink detection substantially as hereinbefore described with reference to the accompanying drawings.

Claim 39

30 . A method of blink detection substantially as hereinbefore described with reference to the accompanying drawings.

Claim 40

31 . A method of determining eye closure amount substantially as hereinbefore described with reference to the accompanying drawings.

Claim 41

31 . A method of determining eye closure amount substantially as hereinbefore described with reference to the accompanying drawings.