US6788809B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
System and method for gesture recognition in three dimensions using stereo imaging and color vision
Assignee: Intel Corp
•
Filing: 2000-06-30
•
Granted: 2004-09-07
•
Expires: 2020-06-30
Abstract
Abstract A system and method for recognizing gestures. The method comprises obtaining image data and determining a hand pose estimation. A frontal view of a hand is then produced. The hand is then isolated the background. The resulting image is then classified as a type of gesture. In one embodiment, determining a hand pose estimation comprises performing background subtraction and computing a hand pose estimation based on an arm orientation determination. In another embodiment, a frontal view of a hand is then produced by performing perspective unwarping and scaling. The system that implements the method may be a personal computer with a stereo camera coupled thereto.
Inventors
- 👤 Radek Grzeszczuk
- 👤 Gary R. Bradski
- 👤 Michael H. Chu
- 👤 Jean-Yves Bouget
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-27 05:39
CPC Classifications
G; G06; G06F; G06F3/00; G06F3/01; G06F3/017; G06V; G06V20/00; G06V20/60; G06V20/64; G06V40/00; G06V40/10; G06V40/107
Claims Analysis
Total Claims: 36
Claim 1
Independent Claim
123 words
A method for recognizing gestures comprising: obtaining an image data; determining a hand pose estimation based on computing a center of the hand, computing an orientation of the hand in relation to a camera reference frame, performing background subtraction, determining an arm orientation, and computing the hand pose estimation based on the arm orientation; producing a frontal view of a hand; isolating the hand from the background; and classifying a gesture of the hand; wherein computing a center of the hand includes defining a hand region as a cylinder centered along the 3D line with dimensions large enough to include a typical hand, selecting pixels from within the hand region as hand pixels, and averaging the location of all of the hand pixels.
Claim 2
Independent Claim
123 words
A method for recognizing gestures comprising: obtaining an image data; determining a hand pose estimation based on computing a center of the hand, computing an orientation of the hand in relation to a camera reference frame, performing background subtraction, determining an arm orientation, and computing the hand pose estimation based on the arm orientation; producing a frontal view of a hand; isolating the hand from the background; and classifying a gesture of the hand; wherein computing a center of the hand includes defining a hand region as a cylinder centered along the 3D line with dimensions large enough to include a typical hand, selecting pixels from within the hand region as hand pixels, and averaging the location of all of the hand pixels.
Claim 3
Dependent Claim
73 words
The method of claim 1 wherein computing an orientation of the hand further comprises: defining a hand reference frame with an x component a y component and a z component such that the x component is aligned with the 3D line, the y component is perpendicular to the x component, the y component is parallel to the viewing plane, and the z component is perpendicular to the x component and the y component.
Claim 4
Independent Claim
62 words
A method for recognizing gestures comprising: obtaining an image data; determining a hand pose estimation; producing a frontal view of a hand based on performing perspective unwarping to produce an unwarped frontal view of the hand and scaling the unwarped frontal view of the hand into a template image; isolating the hand from the background; and classifying a gesture of the hand.
Claim 5
Independent Claim
62 words
A method for recognizing gestures comprising: obtaining an image data; determining a hand pose estimation; producing a frontal view of a hand based on performing perspective unwarping to produce an unwarped frontal view of the hand and scaling the unwarped frontal view of the hand into a template image; isolating the hand from the background; and classifying a gesture of the hand.
Claim 6
Dependent Claim
26 words
The method of claim 3 wherein performing perspective unwarping comprises: mathematically moving a virtual camera to a canonical location with respect to a hand reference frame.
Claim 7
Dependent Claim
46 words
The method of claim 4 wherein mathematically moving the virtual camera comprises: rotating the virtual camera to align a reference frame of the virtual camera with a reference frame of the hand; translating the virtual camera to a fixed distance from the orientation of the hands.
Claim 8
Dependent Claim
26 words
The method of claim 3 wherein scaling comprises: choosing a fixed correspondence between the dimensions of the template image and the dimensions of a typical hand.
Claim 9
Dependent Claim
25 words
The method of claim 1 wherein isolating the hand comprises: initializing a hand color probability density function; and refining the hand color probability density function.
Claim 10
Dependent Claim
23 words
The method of claim 7 wherein initializing comprises: using the hand pixels to initialize and evaluate the hue-saturation histogram of the hand color.
Claim 11
Dependent Claim
62 words
The method of claim 8 wherein refining comprises: choosing a part of a color space that contains a majority of the hand pixels to define a hand color; selecting those pixels in the image surrounding the hand that are of a color corresponding to the hand color; discarding the hand pixels which are not of the color corresponding to the hand color.
Claim 12
Dependent Claim
20 words
The method of claim 3 wherein classifying a gesture comprises: matching the hand template against a plurality of gesture templates.
Claim 13
Independent Claim
67 words
A method for recognizing gestures comprising: obtaining an image data; determining a hand pose estimation; producing a frontal view of a hand; isolating the hand from the background; classifying a gesture of the hand; and matching the hand template against a plurality of gesture templates based on computing geometric moments of a first order, a second order and a third order; and applying a Mahalanobis distance metric.
Claim 14
Independent Claim
67 words
A method for recognizing gestures comprising: obtaining an image data; determining a hand pose estimation; producing a frontal view of a hand; isolating the hand from the background; classifying a gesture of the hand; and matching the hand template against a plurality of gesture templates based on computing geometric moments of a first order, a second order and a third order; and applying a Mahalanobis distance metric.
Claim 15
Independent Claim
75 words
A method for recognizing gestures comprising: obtaining an image data; performing background subtraction; computing a hand pose estimation based on an arm orientation determination; performing perspective unwarping to produce an unwarped frontal view of a hand; scaling the unwarped frontal view of the hand into a template image; isolating the hand from the background using color segmentation; and classifying a gesture of the hand by matching the hand with a plurality of template hand images.
Claim 16
Independent Claim
75 words
A method for recognizing gestures comprising: obtaining an image data; performing background subtraction; computing a hand pose estimation based on an arm orientation determination; performing perspective unwarping to produce an unwarped frontal view of a hand; scaling the unwarped frontal view of the hand into a template image; isolating the hand from the background using color segmentation; and classifying a gesture of the hand by matching the hand with a plurality of template hand images.
Claim 17
Dependent Claim
70 words
The method of claim 12 wherein the image data comprises: a color image comprised of a red value, a green value and a blue value for each pixel of a captured image, and a depth data comprised of an x value in a camera reference frame, a y value in the camera reference frame, and a z value in the camera reference frame for each pixel of the captured image.
Claim 18
Dependent Claim
41 words
The method of claim 13 wherein performing background subtraction comprises: selecting as a foreground arm image those pixels of the depth data where the difference between a mean background depth and the current depth is larger than an empirically defined threshold.
Claim 19
Dependent Claim
71 words
The method of claim 14 wherein determining an arm orientation comprises: computing a three-dimensional (3D) line that defines the arm orientation by fitting a first two dimensional (2D) line to the image data in the image plane and fitting a second 2D line to the image data in the depth dimension in the plane containing the first 2D line such that the second 2D line is perpendicular to the viewing plane.
Claim 20
Dependent Claim
32 words
The method of claim 15 wherein computing the hand pose estimation comprises: computing a center of the hand; and computing an orientation of the hand in relation to the camera reference frame.
Claim 21
Independent Claim
114 words
A system comprising: a stereo camera coupled to a computer, the computer comprising a processor and a storage device to read from a machine readable medium, the machine readable medium containing instructions which, when executed by the processor, cause the computer to perform operations comprising: obtaining an image data; performing background subtraction; computing a hand pose estimation based on an arm orientation determination; performing perspective unwarping to produce an unwarped frontal view of a hand; scaling the unwarped frontal view of the hand into a template image; isolating the hand from the background using color segmentation; and classifying a gesture of the hand by matching the hand with a plurality of template hand images.
Claim 22
Independent Claim
114 words
A system comprising: a stereo camera coupled to a computer, the computer comprising a processor and a storage device to read from a machine readable medium, the machine readable medium containing instructions which, when executed by the processor, cause the computer to perform operations comprising: obtaining an image data; performing background subtraction; computing a hand pose estimation based on an arm orientation determination; performing perspective unwarping to produce an unwarped frontal view of a hand; scaling the unwarped frontal view of the hand into a template image; isolating the hand from the background using color segmentation; and classifying a gesture of the hand by matching the hand with a plurality of template hand images.
Claim 23
Dependent Claim
70 words
The system of claim 17 wherein the image data comprises: a color image comprised of a red value, a green value and a blue value for each pixel of a captured image, and a depth data comprised of an x value in a camera reference frame, a y value in the camera reference frame, and a z value in the camera reference frame for each pixel of the captured image.
Claim 24
Dependent Claim
41 words
The system of claim 17 wherein performing background subtraction comprises: selecting as a foreground arm image those pixels of the depth data where the difference between a mean background depth and the current depth is larger than an empirically defined threshold.
Claim 25
Dependent Claim
71 words
The system of claim 17 wherein determining an arm orientation comprises: computing a three-dimensional (3D) line that defines the arm orientation by fitting a first two dimensional (2D) line to the image data in the image plane and fitting a second 2D line to the image data in the depth dimension in the plane containing the first 2D line such that the second 2D line is perpendicular to the viewing plane.
Claim 26
Dependent Claim
32 words
The system of claim 17 wherein computing the hand pose estimation comprises: computing a center of the hand; and computing an orientation of the hand in relation to a camera reference frame.
Claim 27
Dependent Claim
17 words
The method of claim 3 wherein the image data comprises: a color image and a depth data.
Claim 28
Dependent Claim
64 words
The method of claim 22 wherein the color image comprises a red value, a green value and a blue value for each pixel of a captured image, and the depth data comprises an x value in a camera reference frame, a y value in the camera reference frame, and a z value in the camera reference frame for each pixel of the captured image.
Claim 29
Dependent Claim
30 words
The method of claim 3 wherein determining a hand pose estimation comprises: performing background subtraction; determining an arm orientation; and computing the hand pose estimation based on the arm orientation.
Claim 30
Independent Claim
87 words
A method for recognizing gestures comprising: obtaining an image data; determining a hand pose estimation based on performing background subtraction, determining an arm orientation, and computing the hand pose estimation based on the arm orientation, wherein performing background subtraction includes selecting as a foreground arm image those pixels where the difference between a mean background depth and the current depth is larger than an empirically defined threshold; producing a frontal view of a hand; isolating the hand from the background; and classifying a gesture of the hand.
Claim 31
Independent Claim
87 words
A method for recognizing gestures comprising: obtaining an image data; determining a hand pose estimation based on performing background subtraction, determining an arm orientation, and computing the hand pose estimation based on the arm orientation, wherein performing background subtraction includes selecting as a foreground arm image those pixels where the difference between a mean background depth and the current depth is larger than an empirically defined threshold; producing a frontal view of a hand; isolating the hand from the background; and classifying a gesture of the hand.
Claim 32
Independent Claim
134 words
A method for recognizing gestures comprising obtaining an image data; determining a hand pose estimation based on performing background subtraction, determining an arm orientation, and computing the hand pose estimation based on the arm orientation, wherein determining an arm orientation includes fitting a first two dimensional (2D) line to the image data in the image plane; fitting a second 2D line to the image data in the depth dimension in the plane containing the first 2D line such that the second 2D line is perpendicular to the viewing plane, and combining the first 2D line and the second 2D line into a three-dimensional (3D) line such that the 3D line defines the arm orientation; producing a frontal view of a hand; isolating the hand from the background; and classifying a gesture of the hand.
Claim 33
Independent Claim
134 words
A method for recognizing gestures comprising obtaining an image data; determining a hand pose estimation based on performing background subtraction, determining an arm orientation, and computing the hand pose estimation based on the arm orientation, wherein determining an arm orientation includes fitting a first two dimensional (2D) line to the image data in the image plane; fitting a second 2D line to the image data in the depth dimension in the plane containing the first 2D line such that the second 2D line is perpendicular to the viewing plane, and combining the first 2D line and the second 2D line into a three-dimensional (3D) line such that the 3D line defines the arm orientation; producing a frontal view of a hand; isolating the hand from the background; and classifying a gesture of the hand.
Claim 34
Dependent Claim
34 words
The method of claim 26 wherein fitting a first 2D line comprises: employing an iterative reweighted least square method; and wherein fitting a second 2D line comprises: employing an iterative reweighted least square method.
Claim 35
Dependent Claim
18 words
The method of claim 27 wherein employing an iterative reweighted least square method comprises: using a Welsch M-estimator.
Claim 36
Dependent Claim
32 words
The method of claim 24 wherein computing the hand pose estimation comprises: computing a center of the hand; and computing an orientation of the hand in relation to a camera reference frame.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2007-09-07 | 2008-03-08 | 2008-09-07 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2011-09-07 | 2012-03-08 | 2012-09-07 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2015-09-07 | 2016-03-08 | 2016-09-07 | $7700.00 | Window Closed / Lapsed |