US4980762A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for image processing to obtain three dimensional motion and depth
Assignee: Massachusetts Institute of Technology
•
Filing: 1989-10-13
•
Granted: 1990-12-25
•
Expires: 2009-10-13
Abstract
Abstract An image processing system provides the capability of extracting three dimensional motion information from two dimensional images. The image processing system achieves such extraction by analyzing selected image points. The selected image points correspond to the projection of points on a three dimensional object. The object is moving over time (i.e. translating and rotating with respect to the imaging device). The selected image points are processed in parallel by a plurality of processors to first determine the translational velocity. Next, the rotational velocity of the patch is determined. Lastly, the relative depth of each image point is determined in light of the previously determined translational and rotational velocity. The image processing system accomplishes this task by using an imager and a processing means. The system has potential applications in filming and in robotics.
Inventors
- 👤 David J. Heeger
- 👤 Allan Jepson
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-25 10:09
CPC Classifications
G; G06; G06T; G06T7/00; G06T7/20; G06T7/246
Claims Analysis
Total Claims: 50
Claim 1
Independent Claim
114 words
A method of determining a correct direction of three dimensional translational velocity of an object from a sequence over time of two object, comprising the steps of: (a) selecting a plurality of image points from the sequence of images; (b) performing a transformation on image intensities of the selected plurality of image points to yield transformation vectors; and (c) finding a correct direction of translational velocity such that the transformation vectors are in a range of a coefficient matrix, where the coefficient matrix is a set of column vectors that span a space of all possible transformation vectors corresponding to rigid objects with the correct direction of translational velocity and any arbitrary rotational velocity.
Claim 2
Independent Claim
114 words
A method of determining a correct direction of three dimensional translational velocity of an object from a sequence over time of two object, comprising the steps of: (a) selecting a plurality of image points from the sequence of images; (b) performing a transformation on image intensities of the selected plurality of image points to yield transformation vectors; and (c) finding a correct direction of translational velocity such that the transformation vectors are in a range of a coefficient matrix, where the coefficient matrix is a set of column vectors that span a space of all possible transformation vectors corresponding to rigid objects with the correct direction of translational velocity and any arbitrary rotational velocity.
Claim 3
Dependent Claim
29 words
A method as recited in claim 1 wherein the selected image points are selected so that it is certain that the columns of the coefficient matrix are linearly independent.
Claim 4
Dependent Claim
22 words
A method as recited in claim 1 wherein the performing a transformation step comprises determing image velocity vectors at the selected points.
Claim 5
Dependent Claim
42 words
A method as recited in claim 1 wherein the step of finding a correct direction of translational velocity comprises trying various values for the direction of translational velocity to determine whether the transformation vectors are in a range of the coefficient matrix.
Claim 6
Dependent Claim
30 words
A method as recited in claim 1 wherein the finding of a correct direction of translational velocity comprises employing a search strategy to locate the correct direction of translation velocity.
Claim 7
Dependent Claim
15 words
A method as recited in claim 1 wherein the object is a three dimensional object.
Claim 8
Dependent Claim
41 words
A method as recited in claim 1 wherein the step of finding a correct direction of translational velocity further comprises finding a direction of translational velocity so that an orthogonal complement of the coefficient matrix is orthogonal to the transformation vectors.
Claim 9
Dependent Claim
24 words
A method as recited in claim 1 wherein positions of the selected image points and the direction of translational velocity determine the coefficient matrix.
Claim 10
Dependent Claim
20 words
A method as recited in claim 8 wherein different choices of the direction of translational velocity produce different coefficient matrices.
Claim 11
Independent Claim
140 words
A method of determining a correct direction of three dimensional translational velocity T of an object from a sequence of two dimensional images taken over time that are projections of the object, comprising the steps of: (a) selecting a plurality of image points in each image of the sequence of two dimensional images that correspond to projections of points on the objects; (b) performing a transformation on image intensities of the selected plurality of image points to yield vectors θ which are velocities of the image points in two dimensions; and (c) finding a correct direction of T such that θ is in a range of C(T), where C(T) is a set of column vectors that span a space of all possible θ for rigid objects with translational velocity in the correct direction of T and any arbitrary rotational velocity.
Claim 12
Independent Claim
140 words
A method of determining a correct direction of three dimensional translational velocity T of an object from a sequence of two dimensional images taken over time that are projections of the object, comprising the steps of: (a) selecting a plurality of image points in each image of the sequence of two dimensional images that correspond to projections of points on the objects; (b) performing a transformation on image intensities of the selected plurality of image points to yield vectors θ which are velocities of the image points in two dimensions; and (c) finding a correct direction of T such that θ is in a range of C(T), where C(T) is a set of column vectors that span a space of all possible θ for rigid objects with translational velocity in the correct direction of T and any arbitrary rotational velocity.
Claim 13
Dependent Claim
27 words
A method as recited in claim 10 wherein the selected image points are selected so that it is certain that the columns of C(T) are linearly independent.
Claim 14
Dependent Claim
40 words
A method as recited in claim 10 wherein the step of finding a correct direction of T comprises employing a trial and error technique of trying various directions of T to determine whether θ is in a range of C(T).
Claim 15
Dependent Claim
37 words
A method as recited in claim 10 wherein the step of finding a correct direction of T comprises employing a search strategy that searches selected values in a solution space to locate a proper direction of T.
Claim 16
Independent Claim
168 words
A method of determining a correct direction of three dimensional translational velocity T of three dimensional object images taken from consecutive time frames wherein the images are projections of an object, comprising the steps of: (a) selecting a plurality of image points in each image of the sequence of two dimensional images that correspond to projections of points on the objects such that the image points selected for all the correspond to object points that share a same direction of three dimensional translational velocity T; (b) calculating vectors θ that characterize two dimensional image velocity at the selected image points; (c) finding a correct direction of T such that θ is in a range of C(T), where C(T) is a set of column vectors that span a space of all possible θ for rigid objects with translational velocity in the correct direction of T and any arbitrary rotational velocity, and C(T) is determined by positions of the selected image image points and by the correct direction of T.
Claim 17
Independent Claim
168 words
A method of determining a correct direction of three dimensional translational velocity T of three dimensional object images taken from consecutive time frames wherein the images are projections of an object, comprising the steps of: (a) selecting a plurality of image points in each image of the sequence of two dimensional images that correspond to projections of points on the objects such that the image points selected for all the correspond to object points that share a same direction of three dimensional translational velocity T; (b) calculating vectors θ that characterize two dimensional image velocity at the selected image points; (c) finding a correct direction of T such that θ is in a range of C(T), where C(T) is a set of column vectors that span a space of all possible θ for rigid objects with translational velocity in the correct direction of T and any arbitrary rotational velocity, and C(T) is determined by positions of the selected image image points and by the correct direction of T.
Claim 18
Independent Claim
137 words
A method of determining a correct direction of three dimensional translational velocity T of an object from a sequence of two dimensional images over time that are projections of the object, comprising the steps: (a) selecting a plurality of image points from the sequence of images; (b) performing a transformation on image intensities of the selected plurality of image points to yield vectors θ; and (c) defining C(T) as a set of vectors that span the orthogonal complement of C(T) where C(T) is a set of column vectors that span a space of all possible θ for rigid objects with a translational velocity in the correct direction of T and any arbitrary rotational velocity; and (d) selecting a correct direction of T that provides a zero value for a norm of a product of C(T) and θ.
Claim 19
Independent Claim
137 words
A method of determining a correct direction of three dimensional translational velocity T of an object from a sequence of two dimensional images over time that are projections of the object, comprising the steps: (a) selecting a plurality of image points from the sequence of images; (b) performing a transformation on image intensities of the selected plurality of image points to yield vectors θ; and (c) defining C(T) as a set of vectors that span the orthogonal complement of C(T) where C(T) is a set of column vectors that span a space of all possible θ for rigid objects with a translational velocity in the correct direction of T and any arbitrary rotational velocity; and (d) selecting a correct direction of T that provides a zero value for a norm of a product of C(T) and θ.
Claim 20
Dependent Claim
30 words
A method as recited in claim 15 wherein the step of selecting a correct direction of T comprises determining the product of C(T) and θ for various directions of T.
Claim 21
Dependent Claim
29 words
A method as recited in claim 15 wherein the step of selecting a direction of T comprises applying a systematic search strategy to locate a correct direction of T.
Claim 22
Dependent Claim
36 words
A method as recited in claim 15 further comprising precomputing all possible values of C(T) for a discrete range of directions of T so that selection of a correct direction of T can be performed easily.
Claim 23
Dependent Claim
26 words
A method as recited in claim 15 wherein the selected image points are selected so it is assured that the columns of C(T) are linearly independent.
Claim 24
Dependent Claim
24 words
A method as recited in claim 15 wherein the step of performing a transformation comprises determining image velocity vectors of the selected image points.
Claim 25
Dependent Claim
20 words
A method as recited in claim 15 where C(T) depends on positions of the selected image points and on T.
Claim 26
Independent Claim
140 words
A method of determining three dimensional motion and depth of an object from a sequence of two dimensional images over time that are projections of the object, comprising the steps of: (a) selecting a plurality of image points from the sequence of images; (b) performing a transformation on image intensities of the selected plurality of image points to yield vectors θ; (c) finding a unit vector of translation velocity of the three dimensional motion denoted as T such that θ is in a range of C(T) where C(T) is a set of column vectors that span a space of all possible θ for rigid objects with translational velocity in a direction of T and any arbitrary rotational velocity; and (d) determining rotational velocity of the three dimensional motion and three dimensional depth in light of the determined value of T.
Claim 27
Independent Claim
140 words
A method of determining three dimensional motion and depth of an object from a sequence of two dimensional images over time that are projections of the object, comprising the steps of: (a) selecting a plurality of image points from the sequence of images; (b) performing a transformation on image intensities of the selected plurality of image points to yield vectors θ; (c) finding a unit vector of translation velocity of the three dimensional motion denoted as T such that θ is in a range of C(T) where C(T) is a set of column vectors that span a space of all possible θ for rigid objects with translational velocity in a direction of T and any arbitrary rotational velocity; and (d) determining rotational velocity of the three dimensional motion and three dimensional depth in light of the determined value of T.
Claim 28
Dependent Claim
56 words
An image processing system comprising: (a) an imager for receiving two dimensional projections of three dimensional motion and depth of an object; (b) a processing means for processing a series of images of the object to determine actual corresponding three dimensional motion and depth of points on the object from image data provided by the imager.
Claim 29
Dependent Claim
56 words
An image processing system comprising: (a) an imager for receiving two dimensional projections of three dimensional motion and depth of an object; (b) a processing means for processing a series of images of the object to determine actual corresponding three dimensional motion and depth of points on the object from image data provided by the imager.
Claim 30
Dependent Claim
16 words
An image processing system as recited in claim 23 wherein the imager is a video camera.
Claim 31
Dependent Claim
29 words
An image processing system as recited in claim 23 wherein the processing means determines rotational velocity, relative depth and direction of translational velocity for the points on the object.
Claim 32
Dependent Claim
18 words
An image processing system as recited in claim 23 wherein the imager is an array of imaging elements.
Claim 33
Dependent Claim
21 words
An image processing system as recited in claim 26 wherein the imager is comprised of an array of charge coupled devices.
Claim 34
Dependent Claim
19 words
An image processing system as recited in claim 23 wherein the processing means comprises a plurality of processing elements.
Claim 35
Dependent Claim
18 words
An image processing system as recited in claim 28 wherein the plurality of processing elements process in parallel.
Claim 36
Dependent Claim
21 words
An image processing system as recited in claim 29 wherein each processing element acts on a selected region of the images.
Claim 37
Independent Claim
65 words
A method of combining film sequences using an image processing system, comprising the steps of: (a) filming three dimensional motion in a first setting; (b) extracting the three dimensional motion from the two dimensional images; and (c) repeating the three dimensional motion when filming in a second setting so that film from the first setting may be readily combined with film from the first setting.
Claim 38
Independent Claim
65 words
A method of combining film sequences using an image processing system, comprising the steps of: (a) filming three dimensional motion in a first setting; (b) extracting the three dimensional motion from the two dimensional images; and (c) repeating the three dimensional motion when filming in a second setting so that film from the first setting may be readily combined with film from the first setting.
Claim 39
Dependent Claim
64 words
An image processing system comprising: (a) a camera means for filming in a first setting and a second setting; and (b) a processing means for extracting three dimensional motion experienced in the first setting from the film in the first setting so that such motion can be repeated when filming in the second setting to allow film from both settings to be readily combined.
Claim 40
Dependent Claim
64 words
An image processing system comprising: (a) a camera means for filming in a first setting and a second setting; and (b) a processing means for extracting three dimensional motion experienced in the first setting from the film in the first setting so that such motion can be repeated when filming in the second setting to allow film from both settings to be readily combined.
Claim 41
Dependent Claim
71 words
A robotic system comprising; (a) a robot that is capable of motion in a three dimensional space, wherein the robot has an imager for viewing the three dimensional space as the robot moves; (b) an image processing system coupled to the robot for extracting three dimensional motion information for the robot from two dimensional images from the imager so that the robot can track its motion in the three dimensional space.
Claim 42
Dependent Claim
71 words
A robotic system comprising; (a) a robot that is capable of motion in a three dimensional space, wherein the robot has an imager for viewing the three dimensional space as the robot moves; (b) an image processing system coupled to the robot for extracting three dimensional motion information for the robot from two dimensional images from the imager so that the robot can track its motion in the three dimensional space.
Claim 43
Dependent Claim
110 words
An intelligent camera that records two dimensional images of three dimensional objects and is able to process the two dimensional images to determine the corresponding translational velocity and rotational velocity of the objects as well as to determine relative depth maps of the objects, comprising: (a) a visual receiver for receiving the two dimensional images; (b) a recorder for recording the two dimensional images of objects received by the visual receiver; and (c) a processor programmed to process the two dimensional images recorded by the recorder to determine the corresponding translational velocity and rotational velocity of the objects as well as to determine the relative depth maps of the objects.
Claim 44
Dependent Claim
110 words
An intelligent camera that records two dimensional images of three dimensional objects and is able to process the two dimensional images to determine the corresponding translational velocity and rotational velocity of the objects as well as to determine relative depth maps of the objects, comprising: (a) a visual receiver for receiving the two dimensional images; (b) a recorder for recording the two dimensional images of objects received by the visual receiver; and (c) a processor programmed to process the two dimensional images recorded by the recorder to determine the corresponding translational velocity and rotational velocity of the objects as well as to determine the relative depth maps of the objects.
Claim 45
Dependent Claim
30 words
An intelligent camera as recited in claim 34 wherein the camera determines and records three dimensional motion information for each of the objects moving rigidly with respect to the camera.
Claim 46
Dependent Claim
28 words
An intelligent camera as recited in claim 34 wherein the camera determines and records separate relative depth maps for each object moving rigidly with respect to the camera.
Claim 47
Independent Claim
100 words
A method of tracking three dimensional motion over time for an object using an image processing system, comprising the steps of: (a) taking a first sequence of two dimensional images of the object; (b) processing the first sequence of images so as to be able to extract the translational and rotational velocity of the object for the sequence; (c) taking a second sequence of two dimensional images of the object at a subsequent time; (d) updating the translational and rotational velocity of the object by processing the second sequence of images in view of the previous translational and rotational velocity.
Claim 48
Independent Claim
100 words
A method of tracking three dimensional motion over time for an object using an image processing system, comprising the steps of: (a) taking a first sequence of two dimensional images of the object; (b) processing the first sequence of images so as to be able to extract the translational and rotational velocity of the object for the sequence; (c) taking a second sequence of two dimensional images of the object at a subsequent time; (d) updating the translational and rotational velocity of the object by processing the second sequence of images in view of the previous translational and rotational velocity.
Claim 49
Dependent Claim
18 words
A method as recited in claim 37 wherein the step of updating comprises employing standard sequential estimation techniques.
Claim 50
Dependent Claim
18 words
A method as recited in claim 38 wherein the standard sequential estimation technique employed is a Kalman filter.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 1993-12-24 | 1994-06-25 | 1994-12-25 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 1997-12-24 | 1998-06-25 | 1998-12-25 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2001-12-24 | 2002-06-25 | 2002-12-25 | $7700.00 | Window Closed / Lapsed |