US6301385B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for segmenting images prior to coding
Assignee: AT&T Properties LLC
•
Filing: 1998-12-17
•
Granted: 2001-10-09
•
Expires: 2018-12-17
Abstract
Abstract To segment moving foreground from background, where the moving foreground is of most interest to the viewer, this method uses three detection algorithms as the input to a neural network. The multiple cues used are focus, intensity, and motion. The neural network consists of a two-layered neural network. Focus and motion measurements are taken from high frequency data, edges; whereas, intensity measurements are taken from low frequency data, object interiors. Combined, these measurements are used to segment a complete object. Results indicate that moving foreground can be segmented from stationary foreground and moving or stationary background. The neural network segments the entire object, both interior and exterior, in this integrated approach. Results also demonstrate that combining cues allows flexibility in both type and complexity of scenes. Integration of cues improves accuracy in segmenting complex scenes containing both moving foreground and background. Good segmentation yields bit rate savings when coding the object of interest, also called the video object in MPEG4. This method combines simple measurements to increase segmentation robustness.
Inventors
- 👤 Tsuhan Chen
- 👤 Cassandra Turner Swain
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:18
CPC Classifications
G; G06; G06T; G06T7/00; G06T7/20; G06V; G06V10/00; G06V10/20; G06V10/26; H; H04; H04N; H04N19/00; H04N19/20
Claims Analysis
Total Claims: 23
Claim 1
Independent Claim
136 words
A method for segmenting an image comprising: a) applying at least three segmentation techniques to an image producing independent segmentation maps, the segmentation techniques including focus, motion, and intensity measurements, the focus and motion measurements producing edge information on foreground portions of the image, and the intensity measurement producing interior information on the foreground portions of the image; b) providing a neural network with the segmentations maps produced from the segmentation techniques, wherein each map is processed independently from and without combination with another map, and each map is provided as a separately weighted, dedicated, and independent input to the neural network; and c) wherein the neural network integrates the dedicated, independent and separately weighted segmentation maps as a final segmentation map output without post-processing filling, the output including segmented, filled foreground portions of the image.
Claim 2
Independent Claim
136 words
A method for segmenting an image comprising: a) applying at least three segmentation techniques to an image producing independent segmentation maps, the segmentation techniques including focus, motion, and intensity measurements, the focus and motion measurements producing edge information on foreground portions of the image, and the intensity measurement producing interior information on the foreground portions of the image; b) providing a neural network with the segmentations maps produced from the segmentation techniques, wherein each map is processed independently from and without combination with another map, and each map is provided as a separately weighted, dedicated, and independent input to the neural network; and c) wherein the neural network integrates the dedicated, independent and separately weighted segmentation maps as a final segmentation map output without post-processing filling, the output including segmented, filled foreground portions of the image.
Claim 3
Independent Claim
154 words
A method for segmenting a sequence of images prior to coding the image sequence comprising the steps of: a) detecting motion within foreground and background portions of the image sequence; b) detecting focus within the image sequence; c) detecting intensity within the image sequence, the motion and focus detection producing measurements including edge information on foreground portions of the image sequence, and the intensity detection producing a measurement including interior information on foreground portions of the image sequence; d) calculating segments using a neural network employing detected measurements from steps a) through c) wherein each measurement is processed independently from and without combination with another measurement, and each measurement is provided as a separately weighted, dedicated, and independent input to the neural network, the neural network integrating the dedicated, independent and separately weighted measurements as a final segmented output image without post-processing filling, the output image including segmented, filled foreground portions of the image.
Claim 4
Independent Claim
154 words
A method for segmenting a sequence of images prior to coding the image sequence comprising the steps of: a) detecting motion within foreground and background portions of the image sequence; b) detecting focus within the image sequence; c) detecting intensity within the image sequence, the motion and focus detection producing measurements including edge information on foreground portions of the image sequence, and the intensity detection producing a measurement including interior information on foreground portions of the image sequence; d) calculating segments using a neural network employing detected measurements from steps a) through c) wherein each measurement is processed independently from and without combination with another measurement, and each measurement is provided as a separately weighted, dedicated, and independent input to the neural network, the neural network integrating the dedicated, independent and separately weighted measurements as a final segmented output image without post-processing filling, the output image including segmented, filled foreground portions of the image.
Claim 5
Dependent Claim
23 words
The method according to claim 2 , further comprising the steps of: e) acquiring two consecutive images for use in the detecting steps.
Claim 6
Dependent Claim
42 words
The method according to claim 2 , wherein the step of detecting motion comprises: (i) detecting a difference between pixels in successive frames; and (ii) determining that a pixel is in motion if the difference for that pixel exceeds a predetermined threshold.
Claim 7
Dependent Claim
42 words
The method according to claim 2 , wherein the step of detecting focus comprises: (i) calculating the magnitude of the Sobel edge detection over an nxn pixel square; and (ii) dividing the magnitude of the Sobel edge detection by the edge width.
Claim 8
Dependent Claim
21 words
The method according to claim 2 , wherein the step of detecting intensity comprises determining a gray level of the pixel.
Claim 9
Independent Claim
170 words
A method for processing an image sequence to segment foreground from background, comprising the steps of: a) acquiring successive images in the sequence; b) simultaneously measuring motion, focus and intensity of pixels within successive images, wherein step of simultaneously measuring motion includes measuring motion within foreground and background portions of the images, the focus and motion measurements producing edge information on foreground portions of the image sequence, and the intensity measurement producing interior information on the foreground portions of the image sequence; c) inputting the motion, focus and intensity measurements to a neural network wherein each measurement is processed independently from and without combination with another measurement, and each measurement is provided as a separately weighted, dedicated, and independent input to the neural network, d) calculating foreground and background segmentation using the motion, focus, and intensity measurements with the neural network; and e) drawing a segment map based on the calculated foreground and background segments without post-processing filling, the segment map including segmented, filled foreground portions of the image sequence.
Claim 10
Independent Claim
170 words
A method for processing an image sequence to segment foreground from background, comprising the steps of: a) acquiring successive images in the sequence; b) simultaneously measuring motion, focus and intensity of pixels within successive images, wherein step of simultaneously measuring motion includes measuring motion within foreground and background portions of the images, the focus and motion measurements producing edge information on foreground portions of the image sequence, and the intensity measurement producing interior information on the foreground portions of the image sequence; c) inputting the motion, focus and intensity measurements to a neural network wherein each measurement is processed independently from and without combination with another measurement, and each measurement is provided as a separately weighted, dedicated, and independent input to the neural network, d) calculating foreground and background segmentation using the motion, focus, and intensity measurements with the neural network; and e) drawing a segment map based on the calculated foreground and background segments without post-processing filling, the segment map including segmented, filled foreground portions of the image sequence.
Claim 11
Dependent Claim
25 words
The method according to claim 7 , further comprising the steps of: f) training the neural network using two initial frames and a hand-segmented result.
Claim 12
Dependent Claim
24 words
The method according to claim 8 , further comprising the step of: g) speeding up the training step f) using an adaptive learning rate.
Claim 13
Dependent Claim
42 words
The method according to claim 7 , wherein the step of detecting motion comprises: (i) detecting a difference between pixels in successive images; and (ii) determining that a pixel is in motion if the difference for that pixel exceeds a predetermined threshold.
Claim 14
Dependent Claim
42 words
The method according to claim 7 , wherein the step of detecting focus comprises: (i) calculating the magnitude of the Sobel edge detection over an n×n pixel square; and (ii) dividing the magnitude of the Sobel edge detection by the edge width.
Claim 15
Dependent Claim
21 words
The method according to claim 7 , wherein the step of detecting intensity comprises determining a gray level of the pixel.
Claim 16
Independent Claim
186 words
An apparatus for segmenting foreground and background from a sequence of images; a) a motion detection detecting motion of pixels within foreground and background portions of the image sequence and outputting a motion map; b) a focus detector detecting pixels that are in focus and outputting a focus map; c) an intensity detector detecting those pixels that have high intensity and those with low intensity and outputting an intensity map, the focus and motion detection producing measurements including edge information on foreground portions of the image sequence, and the intensity detection producing a measurement including interior information on the foreground portions of the image sequence; d) a neural network being coupled to the motion detector, the focus detector and the intensity detector, weighing and integrating the outputs from these detectors and outputting a segmentation map without post-processing filling, the output segmentation map including segmented, filled foreground portions of the image sequence, and e) wherein each output of the detectors is processed independently from and without combination with another output, and each output is provided as a separately weighted, dedicated, and independent input to the neural network.
Claim 17
Independent Claim
186 words
An apparatus for segmenting foreground and background from a sequence of images; a) a motion detection detecting motion of pixels within foreground and background portions of the image sequence and outputting a motion map; b) a focus detector detecting pixels that are in focus and outputting a focus map; c) an intensity detector detecting those pixels that have high intensity and those with low intensity and outputting an intensity map, the focus and motion detection producing measurements including edge information on foreground portions of the image sequence, and the intensity detection producing a measurement including interior information on the foreground portions of the image sequence; d) a neural network being coupled to the motion detector, the focus detector and the intensity detector, weighing and integrating the outputs from these detectors and outputting a segmentation map without post-processing filling, the output segmentation map including segmented, filled foreground portions of the image sequence, and e) wherein each output of the detectors is processed independently from and without combination with another output, and each output is provided as a separately weighted, dedicated, and independent input to the neural network.
Claim 18
Dependent Claim
17 words
The apparatus according to claim 13 , wherein the neural network comprises a two layer neural network.
Claim 19
Dependent Claim
34 words
The apparatus according to claim 13 , wherein the neural network further comprises a hidden layer with two neurons and an output layer with one neuron, and each of the neurons employ sigmoid functions.
Claim 20
Dependent Claim
34 words
The apparatus according to claim 15 , wherein the two neurons in the hidden layer are coupled to the one neuron in the output layer using a ninth weight and a tenth weight, respectively.
Claim 21
Dependent Claim
37 words
The apparatus according to claim 16 , wherein the one neuron in the output layer includes three inputs, one from each of the two neurons in the hidden layer and a bias input, having an eleventh weight.
Claim 22
Independent Claim
267 words
A device for segmenting objects within a sequence of images prior to coding of the images for transmission or storage of the images comprising: a) means for digitizing the image sequence to obtain a sequence of digitized images; b) means for segmenting an image based on motion of an object in foreground and background portions within the image, said motion segmenting means being coupled to said means for digitizing and outputting a motion segmentation map; c) means for segmenting an image using focus measurements, said focus segmenting means being coupled to said means for digitizing and outputting a focus segmentation map; d) means for segmenting an image using brightness measurements, said brightness segmenting means being coupled to said means for digitizing and outputting a brightness segmentation map, the motion producing a measurement including edge information on foreground portions of the image, the focus measurements including edge information on the foreground portions of the image, and the brightness measurements including interior information on the foreground portions of the image; e) a neural network for calculating a final segmentation map using segmentation maps output by the motion segmenting means, the brightness segmenting means and the focus segmenting means, and applying the outputted maps to the neural network, wherein each map is processed independently from and without combination with another map, and each map is provided as a separately weighted, dedicated, and independent input to the neural network, the neural network integrating the dedicated, independent and separately weighted maps as the final segmentation map without post-processing filling, and the final segmentation map including segmented, filled foreground portions of the image.
Claim 23
Independent Claim
267 words
A device for segmenting objects within a sequence of images prior to coding of the images for transmission or storage of the images comprising: a) means for digitizing the image sequence to obtain a sequence of digitized images; b) means for segmenting an image based on motion of an object in foreground and background portions within the image, said motion segmenting means being coupled to said means for digitizing and outputting a motion segmentation map; c) means for segmenting an image using focus measurements, said focus segmenting means being coupled to said means for digitizing and outputting a focus segmentation map; d) means for segmenting an image using brightness measurements, said brightness segmenting means being coupled to said means for digitizing and outputting a brightness segmentation map, the motion producing a measurement including edge information on foreground portions of the image, the focus measurements including edge information on the foreground portions of the image, and the brightness measurements including interior information on the foreground portions of the image; e) a neural network for calculating a final segmentation map using segmentation maps output by the motion segmenting means, the brightness segmenting means and the focus segmenting means, and applying the outputted maps to the neural network, wherein each map is processed independently from and without combination with another map, and each map is provided as a separately weighted, dedicated, and independent input to the neural network, the neural network integrating the dedicated, independent and separately weighted maps as the final segmentation map without post-processing filling, and the final segmentation map including segmented, filled foreground portions of the image.
References Cited (Backward Citations)
US4783833A
Cited
US4937878A
Cited
US4876457A
Cited
US5103488A
Cited
US5727081A
Cited
US5687251A
Cited
US5867593A
Cited
US5768409A
Cited
US5715325A
Cited
US5864630A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2004-10-08 | 2005-04-09 | 2005-10-09 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2008-10-08 | 2009-04-09 | 2009-10-09 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2012-10-08 | 2013-04-09 | 2013-10-09 | $7700.00 | Window Closed / Lapsed |