US6771303B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Video-teleconferencing system with eye-gaze correction
Assignee: Microsoft Technology Licensing LLC
•
Filing: 2002-04-23
•
Granted: 2004-08-03
•
Expires: 2022-04-23
Abstract
Abstract Correcting for eye-gaze in video communication devices is accomplished by blending information captured from a stereoscopic view of the conferee and generating a virtual image of the conferee. A personalized face model of the conferee is captured to track head position of the conferee. First and second video images representative of a first conferee taken from different views are concurrently captured. A head position of the first conferee is tracked from the first and second video images. Matching features and contours from the first and second video images are ascertained. The head position as well as the matching features and contours from the first and second video images are synthesized to generate a virtual image video stream of the first conferee that makes the first conferee appear to be making eye contact with a second conferee who is watching the virtual image video stream.
Inventors
- 👤 Zhengyou Zhang
- 👤 Ruigang Yang
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 11:17
CPC Classifications
H; H04; H04N; H04N7/00; H04N7/14; H04N7/141; H04N7/142; H04N7/144
Claims Analysis
Total Claims: 46
Claim 1
Independent Claim
98 words
A method, comprising: concurrently capturing first and second video images representative of a first conferee taken from different views; tracking a head position of the first conferee from the first and second video images; ascertaining features and contours from the first video image that match features and contours from the second video image, wherein the contours from the first and second video images approximated by assigning polygonal lines; and synthesizing the head position, the features, and the contours from the first and second video images that match to generate a virtual image video stream of the first conferee.
Claim 2
Independent Claim
98 words
A method, comprising: concurrently capturing first and second video images representative of a first conferee taken from different views; tracking a head position of the first conferee from the first and second video images; ascertaining features and contours from the first video image that match features and contours from the second video image, wherein the contours from the first and second video images approximated by assigning polygonal lines; and synthesizing the head position, the features, and the contours from the first and second video images that match to generate a virtual image video stream of the first conferee.
Claim 3
Dependent Claim
19 words
The method as recited in claim 1 , further comprising storing a personalized face model of the first conferee.
Claim 4
Dependent Claim
54 words
The method as recited in claim 1 , further comprising storing a personalized face model of the first conferee and evaluating the first and second video images with respect to the personalized face model of the first conferee to monitor feature points from the first and second video images to track the head position.
Claim 5
Dependent Claim
49 words
The method as recited in claim 1 , wherein ascertaining features from the first video image that matches features from the second video image comprises rectifying the first and second video images and locating features from the first and second video images that reside on epipolar lines that match.
Claim 6
Dependent Claim
49 words
The method as recited in claim 1 , wherein ascertaining contours from the first video image that matches contours from the second video image comprises rectifying the first and second video images and locating contours from the first and second video images that reside on epipolar lines that match.
Claim 7
Dependent Claim
56 words
The method as recited in claim 1 , wherein the synthesizing of the head position the features and the contours from the first and second video images comprises morphing the head position as well as the features and contours from the first and second images to generate the virtual image video stream of the first conferee.
Claim 8
Dependent Claim
56 words
The method as recited in claim 1 , wherein synthesizing the head position, the features, and the contours from the first and second video images comprises blending multi-texture features associated with the head position, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.
Claim 9
Dependent Claim
21 words
One or more computer-readable media comprising computer- executable instructions that, when executed, perform the method as recited in claim 1 .
Claim 10
Independent Claim
68 words
A method, comprising: storing a personalized face model of a first conferee; concurrently capturing first and second video images representative of the first conferee taken from different views; evaluating the first and second video images with respect to the personalized face model of the first conferee to ascertain three dimensional information; and synthesizing the three dimensional information to generate a virtual image video stream of the first conferee.
Claim 11
Independent Claim
68 words
A method, comprising: storing a personalized face model of a first conferee; concurrently capturing first and second video images representative of the first conferee taken from different views; evaluating the first and second video images with respect to the personalized face model of the first conferee to ascertain three dimensional information; and synthesizing the three dimensional information to generate a virtual image video stream of the first conferee.
Claim 12
Dependent Claim
48 words
The method as recited in claim 9 , wherein evaluating the first and second video images with respect to the personalized face model of the first conferee further comprises tracking feature points from the first and second video images to monitor a head position of the first conferee.
Claim 13
Dependent Claim
29 words
The method as recited in claim 9 , further comprising ascertaining features and contours from the first video image that match features and contours from the second video image.
Claim 14
Dependent Claim
50 words
The method as recited in claim 9 , further comprising ascertaining features and contours from the first video image that match features and contours from the second video image and synthesizing the three dimensional information wit features and contours to generate the virtual image video stream of the first conferee.
Claim 15
Dependent Claim
95 words
The method as recited in claim 9 , further comprising ascertaining features and contours from the first video image that match features and contours from the second video image and synthesizing the three dimensional information with features and contours to generate the virtual image video stream of the first conferee, and wherein synthesizing the three dimensional information, the features, and the contours from the first and second video images comprises morphing the head, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.
Claim 16
Dependent Claim
101 words
The method as recited in claim 9 , further comprising ascertaining features and contours from the first video image that match features and contours from the second video image and synthesizing the three dimensional information with features and contours to generate the virtual image video stream of the first conferee, and wherein synthesizing the three dimensional information, the features, and the contours from the first and second video images comprises blending multi-texture features associated with the head positions as the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.
Claim 17
Dependent Claim
21 words
One or more computer-readable media comprising computer- executable instructions that, when executed, perform the method as recited in claim 9 .
Claim 18
Independent Claim
107 words
A system, comprising: means for concurrently capturing first and second video images representative of a first conferee taken from different views; means for tracking a head position of the first conferee from the first and second video images; means for ascertaining features and contours from the first video image that match features and contours from the second video image, wherein the contours from the first and second video images are approximated by assigning polygonal lines; and means for synthesizing the head position, the features, and the contours from the first and second video images that match to generate a virtual image video stream of the first conferee.
Claim 19
Independent Claim
107 words
A system, comprising: means for concurrently capturing first and second video images representative of a first conferee taken from different views; means for tracking a head position of the first conferee from the first and second video images; means for ascertaining features and contours from the first video image that match features and contours from the second video image, wherein the contours from the first and second video images are approximated by assigning polygonal lines; and means for synthesizing the head position, the features, and the contours from the first and second video images that match to generate a virtual image video stream of the first conferee.
Claim 20
Dependent Claim
25 words
The system as recited in claim 16 , further comprising means for storing a personalized face model of the first conferee in a memory device.
Claim 21
Dependent Claim
62 words
The system as recited in claim 16 , further comprising means for storing a personalized face model of the first conferee in a memory device and means for evaluating the first and second video images with respect to the personalized face model of the first conferee to monitor feature points from the first and second video images to track the head position.
Claim 22
Dependent Claim
56 words
The system as recited in claim 16 , wherein the means for ascertaining features from the first video image that matches features from the second video image comprises means for rectifying the first and second video images and means for locating features from the first and second video images that reside an epipolar lines that match.
Claim 23
Dependent Claim
56 words
The system as recited in claim 16 , wherein the means for ascertaining contours from the first video image that matches contours from the second video image comprises means for rectifying the first and second video images and means for locating contours from the first and second video images that reside on epipolar lines that match.
Claim 24
Dependent Claim
57 words
The system as recited in claim 16 , wherein the means for synthesizing the head position, the features, and the contours from the first and second video images comprises means for morphing the head positions, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.
Claim 25
Dependent Claim
61 words
The system as recited in claim 16 , wherein the means for synthesizing the head position, the features, and the contours from the first and second video images comprises means for blending multi-texture features associated with the head position, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.
Claim 26
Dependent Claim
131 words
A video-teleconferencing system, comprising: a head pose tracking module, configured to receive first and second video images representative of a first conferee concurrently taken from different views and track head position of the first conferee; a stereo module, configured to receive the first and second video images representative of the first conferee concurrently taken from different views and match features and contours of the first conferee observed from the first and second video images, wherein the contours from the first and second video images are approximated by assigning polygonal lines; and a view synthesis module, configured to synthesize the head position the matching features and the matching contours of the first conferee observed from the first and second video images to generate a virtual image video stream of the first conferee.
Claim 27
Dependent Claim
131 words
A video-teleconferencing system, comprising: a head pose tracking module, configured to receive first and second video images representative of a first conferee concurrently taken from different views and track head position of the first conferee; a stereo module, configured to receive the first and second video images representative of the first conferee concurrently taken from different views and match features and contours of the first conferee observed from the first and second video images, wherein the contours from the first and second video images are approximated by assigning polygonal lines; and a view synthesis module, configured to synthesize the head position the matching features and the matching contours of the first conferee observed from the first and second video images to generate a virtual image video stream of the first conferee.
Claim 28
Dependent Claim
23 words
The system as recited in claim 23 , wherein the stereo module is further configured to rectify the first and second video images.
Claim 29
Dependent Claim
44 words
The system as recited in claim 23 , wherein the stereo module is further configured to perform contour matching of the first and second video images by subtracting the background from the first and second images to extract the contours of the first conferee.
Claim 30
Dependent Claim
60 words
The system as recited in claim 23 , wherein the view system module is configured to synthesize the head position, the features and the contours from the first and second video images by is morphing the head position, the features and the contours from the first and second images to generate the virtual image video stream of the first conferee.
Claim 31
Dependent Claim
63 words
The system as recited in claim 23 , wherein the view system module is configured to synthesize the head position, the features, and the contours from the first and second video images by blending multi-texture features associated with the head position, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.
Claim 32
Dependent Claim
126 words
One or more computer-readable media having stored thereon computer executable instructions that, when executed by one or more processors, causes the one or more processors of a computer system to: concurrently capture first and second video images representative of a first conferee taken from different views; track a head position of the first conferee from the first and second video images; ascertain features and contours from the first video image that match features and contours from the second video image, wherein the contours from the first and second video images are approximated by assigning polygonal lines; and synthesize the head position, the features, and the contours from the first and second video images that match to generate a virtual image video stream of the first conferee.
Claim 33
Dependent Claim
126 words
One or more computer-readable media having stored thereon computer executable instructions that, when executed by one or more processors, causes the one or more processors of a computer system to: concurrently capture first and second video images representative of a first conferee taken from different views; track a head position of the first conferee from the first and second video images; ascertain features and contours from the first video image that match features and contours from the second video image, wherein the contours from the first and second video images are approximated by assigning polygonal lines; and synthesize the head position, the features, and the contours from the first and second video images that match to generate a virtual image video stream of the first conferee.
Claim 34
Dependent Claim
33 words
One or more computer-readable media as recited in claim 28 , further comprising computer executable instructions that, when executed, direct the computer system to store a personalized face model of the first conferee.
Claim 35
Dependent Claim
68 words
One or more computer-readable media as recited in claim 28 , further comprising computer executable instructions that, when executed, direct the computer system to store a personalized face model of the first conferee and evaluate the first and second video images with respect to the personalized face model of the first conferee to monitor feature points from the first and second video images to track the head position.
Claim 36
Dependent Claim
64 words
One or more computer-readable media as recited in claim 28 , further comprising computer executable instructions that, when executed, direct the computer system to, rectify the first and second video images and locate features from the first and second video images that reside on epipolar lines that match when ascertaining features from the first video image that matches features from the second video image.
Claim 37
Dependent Claim
64 words
One or more computer-readable media as recited in claim 28 , further comprising computer executable instructions that, when executed, direct the computer system to, rectify the first and second video images and locate contours from the first and second video images that reside on epipolar lines that match when ascertaining contours from the first video image that matches contours from the second video image.
Claim 38
Dependent Claim
71 words
One or more computer-readable media as recited in claim 28 , further comprising computer executable instructions that, when executed, direct the computer system to morph the head position as well as the features and contours from the first and second images to generate the virtual image video stream of the first conferee when synthesizing the head position as well as the features and contours from the first and second video images.
Claim 39
Dependent Claim
71 words
One or more computer-readable media as recited in claim 28 , further comprising computer executable instructions that, when executed, direct the computer system to blend multi-texture features associated with the head position, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee, when synthesizing the head position, the features, and the contours from the first and second video images.
Claim 40
Dependent Claim
118 words
One or more computer-readable media having stored thereon computer executable instructions that, when executed by one or more processors, causes the one or more processors of a computer system to: store a personalized face model of a first conferee; concurrently, capture first and second video images representative of the first conferee taken from different views; evaluate the first and second video images with respect to the personalized face model of the first conferee to ascertain three dimensional information; and synthesize the three dimensional information to generate a virtual image video stream of the first conferee that makes the first conferee appear to be making eye contact with a second conferee who is watching the virtual image video stream.
Claim 41
Dependent Claim
118 words
One or more computer-readable media having stored thereon computer executable instructions that, when executed by one or more processors, causes the one or more processors of a computer system to: store a personalized face model of a first conferee; concurrently, capture first and second video images representative of the first conferee taken from different views; evaluate the first and second video images with respect to the personalized face model of the first conferee to ascertain three dimensional information; and synthesize the three dimensional information to generate a virtual image video stream of the first conferee that makes the first conferee appear to be making eye contact with a second conferee who is watching the virtual image video stream.
Claim 42
Dependent Claim
62 words
One or more computer-readable media as recited in claim 35 , further comprising computer executable instructions that, when executed, direct the computer system to, track feature points from the first and second video images to monitor a head position of the first conferee when evaluating the first and second video images with respect to the personalized face model of the first conferee.
Claim 43
Dependent Claim
43 words
One or more computer-readable media as recited in claim 35 , further comprising computer executable instructions that, when executed, direct the computer system to, ascertain features and contours from the first video image that match features and contours from the second video image.
Claim 44
Dependent Claim
64 words
One or more computer-readable media as recited in claim 35 , further comprising computer executable instructions that, when executed, direct the computer system to ascertain features and contours from the first video image that match features and contours from the second video image and synthesizing the three dimensional information with features and contours to generate the virtual image video stream of the first conferee.
Claim 45
Dependent Claim
102 words
One or more computer-readable media as recited in claim 35 , further comprising computer executable instructions that, when executed, direct the computer system to ascertain features and contours from the first video image that match features and contours from the second video image and synthesize the three dimensional information with features and contours, and morph the head position as well as the features and contours from the first and second images to generate the virtual image video stream of the first conferee when synthesizing the three dimensional information as well as the features and contours from the first and second video images.
Claim 46
Dependent Claim
65 words
One or more computer-readable media as recited in claim 35 , further comprising computer executable instructions that, when executed, direct the computer system to, ascertain features and contours from the first video image that match features and contours from the second video image and synthesize the three dimensional information with the features and contours to generate the virtual image video steam of the first conferee.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2007-08-03 | 2008-02-02 | 2008-08-03 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2011-08-03 | 2012-02-02 | 2012-08-03 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2015-08-03 | 2016-02-02 | 2016-08-03 | $7700.00 | Window Closed / Lapsed |