US7155036B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Face detection under varying rotation
Assignee: Sony Corp
•
Filing: 2001-02-21
•
Granted: 2006-12-26
•
Expires: 2021-02-21
Abstract
Abstract A feature based face recognition method and apparatus. Global amplitude and direction is calculated for regions defined by a sliding window over an input gray scale image. These values are low pass filtered and a search is carried out for candidates for an eye. Once an eye candidate is identified by comparing the amplitude with a threshold, the other eye is presumed to be located a fixed distance away at the angle of the eye. The mouth is then determined to be in a predetermined location perpendicular to the angle of the eyes. A face contour is estimated by defining two ellipses between which face contours having angles similar to the ellipse are found. Multiple detections are possible and those detections are merged by averaging. The image is resized and the process repeats until the image has been reduced to its minimum size.
Inventors
- 👤 Pingshan Li
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 12:00
CPC Classifications
G; G06; G06V; G06V40/00; G06V40/10; G06V40/16; G06V40/161
Claims Analysis
Total Claims: 59
Claim 1
Independent Claim
150 words
A method of detecting a face in a matrix of pixels representing an image, the method comprising: computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image; inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices; wherein, upon establishing a first facial feature candidate sub-matrix, a second facial feature candidate sub-matrix is assumed to lie in a direction determined by the vector direction of the first facial feature candidate sub-matrix; establishing a matching distance and angle criteria for relationships among each of the first and second candidate sub-matrices and a third facial feature candidate submatrix; and identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.
Claim 2
Independent Claim
150 words
A method of detecting a face in a matrix of pixels representing an image, the method comprising: computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image; inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices; wherein, upon establishing a first facial feature candidate sub-matrix, a second facial feature candidate sub-matrix is assumed to lie in a direction determined by the vector direction of the first facial feature candidate sub-matrix; establishing a matching distance and angle criteria for relationships among each of the first and second candidate sub-matrices and a third facial feature candidate submatrix; and identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.
Claim 3
Dependent Claim
18 words
The method according to claim 1 , further comprising low pass filtering the image prior to the inspecting.
Claim 4
Dependent Claim
18 words
The method according to claim 1 , wherein a facial feature size is assumed to be M×M pixels.
Claim 5
Dependent Claim
58 words
The method according to claim 1 , wherein a first facial feature candidate includes a first eye, and wherein the second candidate facial feature includes a second eye, and wherein the second eye is assumed to be located approximately two feature sizes away from the first eye along the vector direction of the first facial feature candidate sub-matrix.
Claim 6
Dependent Claim
41 words
The method according to claim 3 , wherein a facial feature candidate includes a first eye, and wherein a candidate for a second eye is assumed to be located approximately 2M away along a direction determined by the candidate first eye.
Claim 7
Dependent Claim
55 words
The method according to claim 4 , wherein the third facial feature candidate includes a mouth, and wherein the mouth is assumed to be located approximately 3 feature sizes away from a center point between the two candidate eyes in a direction perpendicular to a direction defined by a line between the two candidate eyes.
Claim 8
Dependent Claim
19 words
The method according to claim 5 , further comprising computing an eyeline distance L between the two candidate eyes.
Claim 9
Dependent Claim
53 words
The method according to claim 7 , wherein the third facial feature candidate includes a mouth, and wherein the mouth is assumed to be located approximately 1.5L away from a center point between the two candidate eyes in a direction perpendicular to a direction defined by a line between the two candidate eyes.
Claim 10
Dependent Claim
73 words
The method according to claim 6 , wherein a fourth facial feature candidate includes a face contour, and wherein: the facial contour is assumed to be located between an inner and an outer ellipse, the inner and outer ellipses being centered at approximately the center point between the two candidate eyes; and the inner ellipse having a vertex at approximately the location of the mouth candidate and covertices at approximately the candidate eyes.
Claim 11
Dependent Claim
64 words
The method according to claim 9 , wherein the outer ellipse has a vertex at approximately 1.5 times the distance from the candidate mouth to the center point between the two candidate eyes and covertices at approximately equaling the distance between the two candidate eyes from the center point between the two candidate eyes, the vertex and covertices of the two ellipses being aligned.
Claim 12
Dependent Claim
73 words
The method according to claim 8 , wherein a fourth facial feature candidate includes a face contour, and wherein: the facial contour is assumed to be located between an inner and an outer ellipse, the inner and outer ellipses being centered at approximately the center point between the two candidate eyes; and the inner ellipse having a vertex at approximately the location of the mouth candidate and covertices at approximately the candidate eyes.
Claim 13
Dependent Claim
52 words
The method according to claim 11 , wherein the outer ellipse has a vertex at approximately 1.5 times the distance from the center point between the two candidate eyes and the candidate mouth and covertices at approximately two times the feature size, the vertex and covertices of the two ellipses being aligned.
Claim 14
Dependent Claim
45 words
The method according to claim 1 , further comprising determining that a plurality of sets of overlapping faces are detected, and merging the overlapping detections by finding an average location of the overlapped faces and defining a detected face as residing at the average location.
Claim 15
Dependent Claim
31 words
The method according to claim 1 , further comprising: resizing the matrix of pixels defining the image; and repeating the computing, inspecting, establishing and identifying on the resized matrix of pixels.
Claim 16
Dependent Claim
15 words
The method according to claim 1 , wherein the amplitude comprises a feature amplitude value.
Claim 17
Dependent Claim
166 words
A computer readable medium storing computer program which, when executed on a programmed processor, carry out a method of detecting a face in a matrix of pixels representing an image, the method comprising: computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image; inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices; wherein, upon establishing a first facial feature candidate sub-matrix, a second facial feature candidate sub-matrix is assumed to lie in a direction determined by the vector direction of the first facial feature candidate sub-matrix; establishing a matching distance and angle criteria for relationships among each of the first and second candidate sub-matrices and a third facial feature candidate submatrix; and identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.
Claim 18
Dependent Claim
166 words
A computer readable medium storing computer program which, when executed on a programmed processor, carry out a method of detecting a face in a matrix of pixels representing an image, the method comprising: computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image; inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices; wherein, upon establishing a first facial feature candidate sub-matrix, a second facial feature candidate sub-matrix is assumed to lie in a direction determined by the vector direction of the first facial feature candidate sub-matrix; establishing a matching distance and angle criteria for relationships among each of the first and second candidate sub-matrices and a third facial feature candidate submatrix; and identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.
Claim 19
Dependent Claim
20 words
The computer readable medium according to claim 16 , further comprising low pass filtering the image prior to the inspecting.
Claim 20
Dependent Claim
58 words
The computer readable medium according to claim 17 , wherein a facial feature size is assumed to be M×M pixels, and wherein the first facial feature candidate includes a first eye, and wherein a candidate for a second eye is assumed to be located approximately two feature sizes away along a direction determined by the first eye candidate.
Claim 21
Dependent Claim
56 words
The computer readable medium according to claim 18 , wherein the third facial feature candidate includes a mouth, and wherein the mouth is assumed to be located a predetermined distance away from a center point between the two candidate eyes in a direction perpendicular to a direction defined by a line between the two candidate eyes.
Claim 22
Dependent Claim
75 words
The computer readable medium according to claim 19 , wherein a fourth facial feature candidate includes a face contour, and wherein: the facial contour is assumed to be located between an inner and an outer ellipse, the inner and outer ellipses being centered at approximately the center point between the two candidate eyes; and the inner ellipse having a vertex at approximately the location of the mouth candidate and covertices at approximately the candidate eyes.
Claim 23
Dependent Claim
47 words
The computer readable medium according to claim 20 , further comprising determining that a plurality of sets of overlapping faces are detected, and merging the overlapping detections by finding an average location of the overlapped faces and defining a detected face as residing at the average location.
Claim 24
Dependent Claim
17 words
The computer readable medium according to claim 16 , wherein the amplitude comprises a feature amplitude value.
Claim 25
Dependent Claim
165 words
A face detection apparatus for detecting a face in a matrix of pixels representing an image, comprising: input means for receiving an input image; a programmed processor carrying out a process of: computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image; inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices; wherein, upon establishing a first facial feature candidate sub-matrix, a second facial feature candidate sub-matrix is assumed to lie in a direction determined by the vector direction of the first facial feature candidate sub-matrix; establishing a matching distance and angle criteria for relationships among each of the first and candidate sub-matrices and a third facial feature candidate submatrix ; and identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.
Claim 26
Dependent Claim
165 words
A face detection apparatus for detecting a face in a matrix of pixels representing an image, comprising: input means for receiving an input image; a programmed processor carrying out a process of: computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image; inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices; wherein, upon establishing a first facial feature candidate sub-matrix, a second facial feature candidate sub-matrix is assumed to lie in a direction determined by the vector direction of the first facial feature candidate sub-matrix; establishing a matching distance and angle criteria for relationships among each of the first and candidate sub-matrices and a third facial feature candidate submatrix ; and identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.
Claim 27
Dependent Claim
21 words
The apparatus according to claim 23 further comprising a low pass filter that filters the input image prior to the inspecting.
Claim 28
Dependent Claim
44 words
The apparatus according to claim 23 , wherein the first feature candidate includes a first eye, and wherein a candidate for a second eye is assumed to be located approximately two feature sizes away along the vector direction of the facial feature candidate sub-matrix.
Claim 29
Dependent Claim
56 words
The apparatus according to claim 25 , wherein a fourth facial feature candidate includes a mouth, and wherein the candidate mouth is assumed to be located approximately 3 feature sizes away from a center point between the two candidate eyes in a direction perpendicular to a direction defined by a line between the two candidate eyes.
Claim 30
Dependent Claim
61 words
The apparatus according to claim 25 , wherein a fourth facial feature candidate includes a mouth, and wherein the mouth is assumed to be located approximately 1.5 times the distance between the two candidate eyes away from a center point between the two candidate eyes in a direction perpendicular to a direction defined by a line between the two candidate eyes.
Claim 31
Dependent Claim
73 words
The apparatus according to claim 27 , wherein a fifth facial feature candidate includes a face contour, and wherein: the facial contour is assumed to be located between an inner and an outer ellipse, the inner and outer ellipses being centered at approximately the center point between the two candidate eyes; and the inner ellipse having a vertex at approximately the location of the mouth candidate and covertices at approximately the candidate eyes.
Claim 32
Dependent Claim
64 words
The apparatus according to claim 27 , wherein the outer ellipse has a vertex at approximately 1.5 times the distance from the candidate mouth to the center point between the two candidate eyes and covertices at approximately equaling the distance between the two candidate eyes from the center point between the two candidate eyes, the vertex and covertices of the two ellipses being aligned.
Claim 33
Dependent Claim
72 words
The apparatus according to claim 29 , wherein a facial feature candidate includes a face contour, and wherein: the facial contour is assumed to be located between an inner and an outer ellipse, the inner and outer ellipses being centered at approximately the center point between the two candidate eyes; and the inner ellipse having a vertex at approximately the location of the mouth candidate and covertices at approximately the candidate eyes.
Claim 34
Dependent Claim
52 words
The apparatus according to claim 30 , wherein the outer ellipse has a vertex at approximately 1.5 times the distance from the center point between the two candidate eyes and the candidate mouth and covertices at approximately two times the feature size, the vertex and covertices of the two ellipses being aligned.
Claim 35
Dependent Claim
47 words
The apparatus according to claim 23 , wherein the processor further determines that a plurality of sets of overlapping faces are detected, and merging the overlapping detections by finding an average location of the overlapped faces and defining a detected face as residing at the average location.
Claim 36
Dependent Claim
38 words
The apparatus according to claim 23 , wherein the processor further carries out a process of: resizing the matrix of pixels defining the image; and repeating the computing, inspecting, establishing and identifying on the resized matrix of pixels.
Claim 37
Dependent Claim
15 words
The apparatus according to claim 23 , wherein the amplitude comprises a feature amplitude value.
Claim 38
Independent Claim
50 words
A method of face detection in an image, comprising: identifying a first candidate facial feature with a feature size; computing a first direction associated with the first candidate facial; and searching for a second candidate facial feature at a first location determined at least in part by the first direction.
Claim 39
Independent Claim
50 words
A method of face detection in an image, comprising: identifying a first candidate facial feature with a feature size; computing a first direction associated with the first candidate facial; and searching for a second candidate facial feature at a first location determined at least in part by the first direction.
Claim 40
Dependent Claim
56 words
The method according to claim 35 , further comprising: finding the second candidate facial feature; computing a geometric relationship between the first facial feature and the second feature, the geometric relationship including a second direction; and searching for a third candidate facial feature at a third location determined by the geometric relationship including the second direction.
Claim 41
Dependent Claim
40 words
The method according to claim 36 , further comprising: finding the third candidate facial feature; creating a bounded geometric construct about the first, second and third features; and searching for a facial contour within the bounds of the geometric construct.
Claim 42
Dependent Claim
20 words
The method according to claim 35 , wherein the first and second candidate facial features comprise first and second eyes.
Claim 43
Dependent Claim
25 words
The method according to claim 35 , wherein the first candidate facial feature comprises an eye and wherein the first direction comprises an eye direction.
Claim 44
Dependent Claim
53 words
The method according to claim 36 , wherein the first and second candidate facial features comprise first and second candidate eyes, and wherein the geometric relationship includes an eyeline having length L equaling a distance between the first and second candidate eyes, and wherein the second direction comprises a direction of the eyeline.
Claim 45
Dependent Claim
45 words
The method according to claim 40 , wherein the third candidate facial feature comprises a candidate mouth, and wherein the third location is situated a predetermined distance from the eyeline along a line perpendicular to the eyeline and passing through a center of the eyeline.
Claim 46
Dependent Claim
34 words
The method according to claim 37 , wherein the bounded geometric construct comprises an inner and an outer ellipse, and wherein searching comprises searching for the facial contour between the inner and outer ellipse.
Claim 47
Dependent Claim
37 words
The method according to claim 42 , wherein the searching further comprises searching for points between the inner and outer ellipse that have direction within a specified angle of tangent angles of the inner and outer ellipses.
Claim 48
Dependent Claim
17 words
The method according to claim 35 , wherein the identifying comprises finding dark regions of the image.
Claim 49
Dependent Claim
102 words
The method according to claim 36 , wherein the first and second candidate eyes are tested against the following criteria to determine that they are eye candidates: D (θ lefteye ,θ righteye )<15°; A lefteye <α·A middle ; A righteye <α·A middle ; G lefteye <β·G middle ; G righteye <β·G middle ; where D is the distance function, θ is an angle defining the direction, A is an amplitude of the candidate feature, G is a low pass filtered amplitude, and α and β are constants, and wherein the subscript middle denotes quantities associated with a midpoint between the two eyes.
Claim 50
Dependent Claim
21 words
The method according to claim 45 , wherein α is approximately equal to 2 and β is approximately equal to ⅞.
Claim 51
Dependent Claim
107 words
The method according to claim 40 , wherein the first and second candidate eyes and the candidate mouth are tested against the following criteria to determine that they are eye candidates and mouth candidate: D [θeyeline,θmouth]<10° A mouth >λ·( A lefteye +A righteye ); ( G mouth <γ·G rightface and G mouth <μ·G leftface ) or ( G mouth <γ·G leftface and G mouth <μ·G rightface ); G lefteye <η·G leftface ; G righteye <η·G rightface ; where G leftface and G rightface are face location points located L/2 directly below the eyes along a line perpendicular to the eyeline and where λ,γ,μ, and η are constants.
Claim 52
Dependent Claim
25 words
The method according to claim 47 , wherein λ is approximately ⅛, λ is approximately 8/7, μ is approximately ⅞ and η is approximately ⅞.
Claim 53
Dependent Claim
148 words
The method according to claim 43 , a center point of the eyeline is (x c , y c ) and the distance between the two eyes is 2r eyes and wherein the inner ellipses are given by: { x = r eyes cos ( θ + θ eyeline ) + x c y = D mouth sin ( θ + θ eyeline ) + y c ; and wherein the outer ellipse is given by: { x = 2 r eyes cos ( θ + θ eyeline ) + x c y = 1.5 D mouth sin ( θ + θ eyeline ) + y c ; wherein, if a line is drawn from the center of the ellipse to the outer ellipse, this line is expected to cross the candidate face contour.
Claim 54
Dependent Claim
68 words
The method according to claim 49 , wherein candidate points on the candidate face contour are expected to meet the criterion of: D (θ face ,θ ellipse )<30° where D is the distance function, the angle the face contour where the line crosses is denoted by θ face and the angle of the tangent line of the ellipse at the point where the line crosses θ ellipse .
Claim 55
Dependent Claim
31 words
The method according to claim 50 , further comprising counting a number of points meeting the criterion and comparing the number to a threshold to determine existence of a face contour.
Claim 56
Independent Claim
178 words
A method of detecting a face in a matrix of pixels representing an image, the method comprising: computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image; inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices; wherein, upon establishing a first facial feature candidate sub-matrix, a second facial feature candidate sub-matrix is assumed to lie in a direction determined by the vector direction of the first facial feature candidate sub-matrix; wherein the second facial feature candidate sub-matrix has a vector direction substantially equal to the direction of the first facial feature candidate sub-matrix vector direction plus 180 degrees; establishing a matching distance and angle criteria for relationships between each of the first and second candidate sub-matrices and a third facial feature candidate submatrix; and identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.
Claim 57
Independent Claim
178 words
A method of detecting a face in a matrix of pixels representing an image, the method comprising: computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image; inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices; wherein, upon establishing a first facial feature candidate sub-matrix, a second facial feature candidate sub-matrix is assumed to lie in a direction determined by the vector direction of the first facial feature candidate sub-matrix; wherein the second facial feature candidate sub-matrix has a vector direction substantially equal to the direction of the first facial feature candidate sub-matrix vector direction plus 180 degrees; establishing a matching distance and angle criteria for relationships between each of the first and second candidate sub-matrices and a third facial feature candidate submatrix; and identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.
Claim 58
Independent Claim
163 words
A method of detecting a face in a matrix of pixels representing an image, the method comprising: computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image; inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices; wherein, upon establishing a first facial feature candidate sub-matrix, a second facial feature candidate sub-matrix is assumed to lie in a direction determined by the vector direction of the first facial feature candidate sub-matrix, wherein the direction is determined without regard for a horizontal or vertical direction; establishing a matching distance and angle criteria for relationships between each of the first and second candidate sub-matrices and a third facial feature candidate submatrix; and identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.
Claim 59
Independent Claim
163 words
A method of detecting a face in a matrix of pixels representing an image, the method comprising: computing a vector having amplitude and direction for each of a plurality of sub-matrices, the sub-matrices having size estimated to be a facial feature size in the image; inspecting each sub-matrix to determine if the amplitude exceeds a threshold, wherein sub-matrices having amplitude exceeding the threshold are deemed facial feature candidate sub-matrices; wherein, upon establishing a first facial feature candidate sub-matrix, a second facial feature candidate sub-matrix is assumed to lie in a direction determined by the vector direction of the first facial feature candidate sub-matrix, wherein the direction is determined without regard for a horizontal or vertical direction; establishing a matching distance and angle criteria for relationships between each of the first and second candidate sub-matrices and a third facial feature candidate submatrix; and identifying sets of candidate sub-matrices matching the distance and angle criteria to detect the presence of a face in the image.
References Cited (Backward Citations)
US5680481A
Cited
US5781650A
Cited
US5835616A
Cited
US5852669A
Cited
US6456328B1
Cited
US5832115A
Cited
US6072893A
Cited
US6633655B1
Cited
US6526161B1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2009-12-25 | 2010-06-26 | 2010-12-26 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2013-12-25 | 2014-06-26 | 2014-12-26 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2017-12-25 | 2018-06-26 | 2018-12-26 | $7700.00 | Window Closed / Lapsed |