US7162056B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Systems and methods for the automated sensing of motion in a mobile robot using visual data
Assignee: FUTURE VISION HOLDINGS Inc
•
Filing: 2003-08-15
•
Granted: 2007-01-09
•
Expires: 2023-08-15
Abstract
Abstract The invention is related to methods and apparatus that detect motion by monitoring images from a video camera mounted on a mobile robot, such as an autonomously navigated mobile robot. Examples of such robots include automated vacuum floor sweepers. Advantageously, embodiments of the invention can automatically sense a robot's motional state in a relatively reliable and cost-efficient manner. Many configurations of robots are configured to include at least one video camera. Embodiments of the invention permit the use of a video camera onboard a robot to determine a motional state for the robot. This can advantageously permit the motional state of a robot to be determined at a fraction of the cost of additional sensors, such as a laser, an infrared, an ultrasonic, or a contact sensor.
Inventors
- 👤 Michael Christopher Burl
- 👤 Paolo Pirjanian
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-05 14:34
CPC Classifications
G; G05; G05D; G05D1/00; G05D1/02; G05D1/021; G05D1/0231; G05D1/0246; G05D1/0253; G06; G06T; G06T7/00; G06T7/20; G06T7/246; G05D1/0227; G05D1/0238; G05D1/024; G05D1/0242; G05D1/0255; G05D1/0268; G05D1/0272; G05D1/0274
Claims Analysis
Total Claims: 48
Claim 1
Independent Claim
342 words
A method of determining a motional state of a mobile robot, the method comprising: retrieving pixel data for images taken at intervals from a camera that is coupled to the mobile robot; comparing pixel data for a first image to pixel data for a second image to generate a measure of a difference between the two images, wherein comparing comprises: filtering the first image pixel data with a gradient magnitude filter, where the gradient magnitude filter computes at least a spatial gradient; low-pass filtering the gradient magnitude filtered first image pixel data to generate a gradient-magnitude filtered smoothed first image pixel data; comparing the gradient-magnitude filtered first image pixel data to a first threshold; generating a binary map of the first image pixel data at least partly in response to the comparison of the gradient-magnitude filtered smoothed first image pixel data with the first threshold; filtering the second image pixel data with the gradient magnitude filter; low-pass filtering the gradient magnitude filtered second image pixel data to generate a gradient-magnitude filtered smoothed second image pixel data; comparing the gradient-magnitude filtered smoothed second image pixel data to a second threshold; generating a binary map of the second image pixel data at least partly in response to the comparison of the gradient-magnitude filtered second image pixel data to the second threshold; and comparing the binary map of the first image pixel data to the binary map of the second image pixel data to identify data for pixels that are different between the first image and the second image; using the comparison of the pixel data to count the number of pixel data identified as changed; comparing the count to a third predetermined threshold; determining the motional state of the mobile robot at least partly in response to the count; and automatically providing an indication to a user that an undesired stoppage of the mobile robot has occurred if: the motional state of the mobile robot indicates that the mobile robot is not in motion; and the mobile robot is intended to be in motion.
Claim 2
Independent Claim
342 words
A method of determining a motional state of a mobile robot, the method comprising: retrieving pixel data for images taken at intervals from a camera that is coupled to the mobile robot; comparing pixel data for a first image to pixel data for a second image to generate a measure of a difference between the two images, wherein comparing comprises: filtering the first image pixel data with a gradient magnitude filter, where the gradient magnitude filter computes at least a spatial gradient; low-pass filtering the gradient magnitude filtered first image pixel data to generate a gradient-magnitude filtered smoothed first image pixel data; comparing the gradient-magnitude filtered first image pixel data to a first threshold; generating a binary map of the first image pixel data at least partly in response to the comparison of the gradient-magnitude filtered smoothed first image pixel data with the first threshold; filtering the second image pixel data with the gradient magnitude filter; low-pass filtering the gradient magnitude filtered second image pixel data to generate a gradient-magnitude filtered smoothed second image pixel data; comparing the gradient-magnitude filtered smoothed second image pixel data to a second threshold; generating a binary map of the second image pixel data at least partly in response to the comparison of the gradient-magnitude filtered second image pixel data to the second threshold; and comparing the binary map of the first image pixel data to the binary map of the second image pixel data to identify data for pixels that are different between the first image and the second image; using the comparison of the pixel data to count the number of pixel data identified as changed; comparing the count to a third predetermined threshold; determining the motional state of the mobile robot at least partly in response to the count; and automatically providing an indication to a user that an undesired stoppage of the mobile robot has occurred if: the motional state of the mobile robot indicates that the mobile robot is not in motion; and the mobile robot is intended to be in motion.
Claim 3
Dependent Claim
16 words
The method as defined in claim 1 , wherein the method is performed in real time.
Claim 4
Dependent Claim
34 words
The method as defined in claim 1 , wherein the gradient magnitude filter further comprises a low-pass filter such that the gradient magnitude filter computes both the spatial gradient and the low-pass-filtering for smoothing.
Claim 5
Dependent Claim
17 words
The method as defined in claim 3 , wherein the low-pass filter corresponds to a Gaussian filter.
Claim 6
Dependent Claim
78 words
The method as defined in claim 3 , wherein filtering the first image pixel data further comprises: filtering the first image pixel data in a first direction and in a second direction with a gradient filter that is characterized by the following convolution kernel, [−1−4−5 0 5 4 1]/32, where results of filtering are separately maintained for the first direction and for the second direction; and combining the separately maintained data by computing a magnitude of the data.
Claim 7
Dependent Claim
20 words
The method as defined in claim 1 , further comprising low-pass filtering image pixel data prior to computing a gradient.
Claim 8
Dependent Claim
38 words
The method as defined in claim 6 , wherein a low-pass filter is characterized by the following convolution kernel: [1 4 6 4 1]/16; and a gradient filter is characterized by the following convolution kernel: [−1 0 1]/2.
Claim 9
Dependent Claim
72 words
The method as defined in claim 1 , further comprising: receiving an indication that the mobile robot is intended to be in motion in a first direction; determining that the motional state of the mobile robot is not in motion while receiving the indication that the mobile robot is intended to be in motion; and changing a path of travel for the mobile robot at least partly in response to the determination.
Claim 10
Dependent Claim
27 words
The method as defined in claim 1 , further comprising subsampling the pixel data such that fewer than all the available pixels from the camera are used.
Claim 11
Dependent Claim
29 words
The method as defined in claim 1 , wherein the third predetermined threshold varies at least partly in response to a number of pixels used to compute the count.
Claim 12
Dependent Claim
38 words
The method as defined in claim 1 , wherein the first threshold and the second threshold are adaptive to filtering by the gradient magnitude filter of the first image pixel data and the second image pixel data, respectively.
Claim 13
Dependent Claim
77 words
The method as defined in claim 11 , wherein the first threshold is selected to be about half the maximum value for the gradient magnitude computed for a pixel by the gradient magnitude filter after filtering of the first image pixel data, and where the second threshold is selected to be about half the maximum value for the gradient magnitude computed for a pixel by the gradient magnitude filter after filtering of the second image pixel data.
Claim 14
Dependent Claim
37 words
The method as defined in claim 1 , wherein the comparison of the binary map of the first image pixel data to the binary map of the second image pixel data comprises a sloppy exclusive-or (XOR) computation.
Claim 15
Dependent Claim
61 words
The method as defined in claim 13 , wherein the sloppy XOR is computed by comparing a pixel from the binary map of the first image pixel data with a location of (x,y) to a plurality of pixels in the binary map of the second image pixel data, wherein the plurality of pixels includes a pixel with a location of (x,y).
Claim 16
Dependent Claim
20 words
The method as defined in claim 1 , wherein the motional state is determined only with observation of visual data.
Claim 17
Dependent Claim
25 words
The method as defined in claim 1 , where the mobile robot is autonomous, and where the method is performed entirely within the mobile robot.
Claim 18
Dependent Claim
48 words
The method as defined in claim 1 , further comprising: sending pixel data from the mobile robot to a remote computer; performing at least part of filtering operations in the remote computer; and receiving an indication of the motional state of the mobile robot from the remote computer.
Claim 19
Independent Claim
183 words
A method of determining a motional state of a mobile robot, the method comprising: receiving pixel data for video images, where the video images are taken from a camera mounted to the mobile robot; processing the pixel data for the video images to identify amounts of spatial gradient within a video image; characterizing pixels of a video image into at least a first group and a second group, wherein the pixels of the first group correspond to a higher spatial gradient than the pixels of the second group; using the characterization of the pixels to compare a first video image to a second video image to detect the motional state of the mobile robot, wherein the motional state is selected from a set of possible motional states, the set comprising at least “in motion” and “not in motion;” and automatically providing an indication to a user that an undesired stoppage of the mobile robot has occurred if: the motional state of the mobile robot indicates that the mobile robot is not in motion; and the mobile robot is intended to be in motion.
Claim 20
Independent Claim
183 words
A method of determining a motional state of a mobile robot, the method comprising: receiving pixel data for video images, where the video images are taken from a camera mounted to the mobile robot; processing the pixel data for the video images to identify amounts of spatial gradient within a video image; characterizing pixels of a video image into at least a first group and a second group, wherein the pixels of the first group correspond to a higher spatial gradient than the pixels of the second group; using the characterization of the pixels to compare a first video image to a second video image to detect the motional state of the mobile robot, wherein the motional state is selected from a set of possible motional states, the set comprising at least “in motion” and “not in motion;” and automatically providing an indication to a user that an undesired stoppage of the mobile robot has occurred if: the motional state of the mobile robot indicates that the mobile robot is not in motion; and the mobile robot is intended to be in motion.
Claim 21
Dependent Claim
51 words
The method as defined in claim 18 , wherein characterizing further comprises: arranging the pixels of a video image into a binary map such that a first logic state indicates association with the first group with the higher spatial gradient; and a second logic state indicates association with the second group.
Claim 22
Dependent Claim
71 words
The method as defined in claim 19 , wherein using the characterization comprises: comparing a binary value for one pixel from the binary map of the first image to a group of binary values for a group of pixels in the second image; wherein the group of pixels in the second image includes a pixel in the same location as the one pixel from the binary map of the first image.
Claim 23
Dependent Claim
38 words
The method as defined in claim 20 , where comparing the binary value to the group of binary values further comprises computing a sloppy Exclusive-OR (“sloppy XOR”) operation between the binary value and the group of binary values.
Claim 24
Dependent Claim
57 words
The method as defined in claim 18 , further comprising: inspecting pixel data to evaluate whether enough useful spatial gradients exist for robust detection of the motional state; and inhibiting a motional state of “not moving” at least partly in response to a determination that the detection of the motional state is not likely to be reliable.
Claim 25
Dependent Claim
24 words
The method as defined in claim 18 , wherein the motional state is determined only by analysis of visual data observed by the camera.
Claim 26
Dependent Claim
25 words
The method as defined in claim 18 , where the mobile robot is autonomous, and where the method is performed entirely within the mobile robot.
Claim 27
Dependent Claim
49 words
The method as defined in claim 18 , further comprising: sending pixel data from the mobile robot to a remote computer; performing at least part of pixel characterizing operations in the remote computer; and receiving an indication of the motional state of the mobile robot from the remote computer.
Claim 28
Dependent Claim
30 words
The method as defined in claim 18 , wherein processing the pixel data further comprises processing the pixel data to identify amounts of spatial gradient within the entire video image.
Claim 29
Independent Claim
184 words
A method of controlling a behavior of a mobile robot based on a mismatch between an intended motional state and a perceived motional state in a mobile robot, the method comprising: receiving an indication of the intended motional state, where the motional state is selected from the group including moving and not moving; using visual data from a camera that is coupled to the mobile robot to perceive the motional state of the mobile robot, where the perceived motional state of the mobile robot is selected from the group including moving and not moving, wherein the mobile robot uses simultaneous localization and mapping (SLAM) techniques for navigation; comparing the intended motional state to the perceived motional state to detect whether a mismatch exists between the intended motional state and the perceived motional state; and automatically changing the behavior of the mobile robot at least partly in response to a detected mismatch if: the intended motional state of the mobile robot indicates that the mobile robot is moving; and the perceived motional state of the mobile robot indicates that the mobile robot is not moving.
Claim 30
Independent Claim
184 words
A method of controlling a behavior of a mobile robot based on a mismatch between an intended motional state and a perceived motional state in a mobile robot, the method comprising: receiving an indication of the intended motional state, where the motional state is selected from the group including moving and not moving; using visual data from a camera that is coupled to the mobile robot to perceive the motional state of the mobile robot, where the perceived motional state of the mobile robot is selected from the group including moving and not moving, wherein the mobile robot uses simultaneous localization and mapping (SLAM) techniques for navigation; comparing the intended motional state to the perceived motional state to detect whether a mismatch exists between the intended motional state and the perceived motional state; and automatically changing the behavior of the mobile robot at least partly in response to a detected mismatch if: the intended motional state of the mobile robot indicates that the mobile robot is moving; and the perceived motional state of the mobile robot indicates that the mobile robot is not moving.
Claim 31
Dependent Claim
65 words
The method as defined in claim 27 , wherein the intended motional state is moving, where changing the behavior comprises changing a navigated path, further comprising: instructing the mobile robot to travel in a direction approximately opposite to that previously requested for at least a distance sufficient to clear an obstruction; instructing the mobile robot to yaw; and instructing the mobile robot to resume traveling.
Claim 32
Dependent Claim
28 words
The method as defined in claim 27 , wherein the intended motional state is moving, and where changing the behavior comprises shutting off motors to conserve battery life.
Claim 33
Dependent Claim
27 words
The method as defined in claim 27 , wherein the mobile robot further comprises floor cleaning brushes, wherein changing the behavior further comprises shutting off cleaning brushes.
Claim 34
Dependent Claim
28 words
The method as defined in claim 27 , wherein the mobile robot further comprises a vacuum cleaner, wherein changing the behavior further comprises shutting off the vacuum cleaner.
Claim 35
Dependent Claim
24 words
The method as defined in claim 27 , wherein the intended motional state is moving, and where changing the behavior comprises setting an alert.
Claim 36
Dependent Claim
34 words
The method as defined in claim 27 , wherein changing the behavior comprises: resetting an estimate of pose (location and heading); and disabling mapping functions for recognition of new landmarks until localization is achieved.
Claim 37
Dependent Claim
29 words
The method as defined in claim 27 , wherein changing the behavior comprises disabling mapping functions for recognition of new landmarks until passage of a predetermined amount of time.
Claim 38
Dependent Claim
23 words
The method as defined in claim 34 , wherein changing the behavior further comprising resetting an estimate of pose, including location and heading.
Claim 39
Dependent Claim
31 words
The method as defined in claim 27 , wherein changing the behavior comprises disabling mapping functions for recognition of new landmarks until a predetermined number of unmatched landmarks have been observed.
Claim 40
Dependent Claim
23 words
The method as defined in claim 36 , wherein changing the behavior further comprising resetting an estimate of pose, including location and heading.
Claim 41
Dependent Claim
45 words
The method as defined in claim 27 , wherein the intended motional state is moving, where changing the behavior comprises: using the indication to determine that the mobile robot has been knocked over; and initiating procedures to restore the mobile robot to an upright position.
Claim 42
Dependent Claim
22 words
The method as defined in claim 27 , wherein the motional state is perceived using only the visual data from the camera.
Claim 43
Dependent Claim
25 words
The method as defined in claim 27 , where the mobile robot is autonomous, and where the method is performed entirely within the mobile robot.
Claim 44
Dependent Claim
49 words
The method as defined in claim 27 , further comprising: transferring visual data from the mobile robot to a remote computer; performing at least part of filtering operations in the remote computer; and transferring an indication of the perceived motional state of the mobile robot from the remote computer.
Claim 45
Independent Claim
78 words
A method of detecting that a mobile robot has been kidnapped, the method comprising: receiving an indication that the mobile robot is not instructed to be moving; receiving data for video images from a camera coupled to the mobile robot; comparing data from different video images to determine whether or not the mobile robot is in motion; and determining that the mobile robot has been kidnapped when the video images indicate that the mobile robot is in motion.
Claim 46
Independent Claim
78 words
A method of detecting that a mobile robot has been kidnapped, the method comprising: receiving an indication that the mobile robot is not instructed to be moving; receiving data for video images from a camera coupled to the mobile robot; comparing data from different video images to determine whether or not the mobile robot is in motion; and determining that the mobile robot has been kidnapped when the video images indicate that the mobile robot is in motion.
Claim 47
Dependent Claim
205 words
A circuit for a mobile robot that is configured to determine a motional state of the mobile robot, the circuit comprising: a means for receiving pixel data for video images, where the video images are taken from a camera mounted to the mobile robot; a means for processing the pixel data for the video images to identify amounts of spatial gradient within a video image; a means for characterizing pixels of a video image into at least a first group and a second group, wherein the pixels of the first group correspond to a higher spatial gradient than the pixels of the second group; and a means for using the characterization of the pixels to compare a first video image to a second video image to detect the motional state of the mobile robot, wherein the motional state is selected from a set of possible motional states, the set comprising at least “in motion” and “not in motion;” a means for automatically providing an indication to a user that an undesired stoppage of the mobile robot has occurred if: the motional state of the mobile robot indicates tat the mobile robot is not in motion, and the mobile robot is intended to be in motion.
Claim 48
Dependent Claim
205 words
A circuit for a mobile robot that is configured to determine a motional state of the mobile robot, the circuit comprising: a means for receiving pixel data for video images, where the video images are taken from a camera mounted to the mobile robot; a means for processing the pixel data for the video images to identify amounts of spatial gradient within a video image; a means for characterizing pixels of a video image into at least a first group and a second group, wherein the pixels of the first group correspond to a higher spatial gradient than the pixels of the second group; and a means for using the characterization of the pixels to compare a first video image to a second video image to detect the motional state of the mobile robot, wherein the motional state is selected from a set of possible motional states, the set comprising at least “in motion” and “not in motion;” a means for automatically providing an indication to a user that an undesired stoppage of the mobile robot has occurred if: the motional state of the mobile robot indicates tat the mobile robot is not in motion, and the mobile robot is intended to be in motion.
References Cited (Backward Citations)
US6362589B1
Cited
US4628453A
Cited
US5081585A
Cited
US5001635A
Cited
US5109425A
Cited
US4969036A
Cited
US5706355A
Cited
US5387947A
Cited
US5652489A
Cited
US5838828A
Cited
US6408109B1
Cited
US5995884A
Cited
US6496592B1
Cited
US6321147B1
Cited
US6462498B1
Cited
US20030007682A1
Cited
US6809490B2
Cited
US20040017937A1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2010-01-08 | 2010-07-10 | 2011-01-09 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2014-01-08 | 2014-07-10 | 2015-01-09 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2018-01-08 | 2018-07-10 | 2019-01-09 | $7700.00 | Window Closed / Lapsed |