Patent US6898331B2 successfully refreshed from Google Patents.
US6898331B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Image fusion system and method
Assignee: BAE Systems Controls Inc
•
Filing: 2002-08-28
•
Granted: 2005-05-24
•
Expires: 2022-08-28
Abstract
Abstract A contrast-based image fusion system and method of processing multiple images to form a processed or fused image including regions selected from one or more images. Images are divided into image regions. Portions of the images are filtered if necessary. A contrast map is generated for each image via a convolution kernel resulting in a contrast map with contrast values for each image region. Contrast values are compared and image regions are selected based on a selection criteria or process such as greater or maximum contrast. The selected image regions form the fused image. If necessary, the luminance of one or more portions of the fused image is adjusted. One sensor is selected as a reference sensor, and an average intensity of each region of the reference sensor image is determined across the reference sensor image. The intensity of one or more regions in the final image is adjusted by combining the determined average luminance values and intensity values of the final image.
Inventors
- 👤 Carlo Tiana
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 16:49
CPC Classifications
G; G06; G06V; G06V10/00; G06V10/20; G06T; G06T5/00; G06T5/50; G06T15/00; G06T15/50; G06T15/503; G06T7/00; G06T7/20; H; H04; H04N; H04N5/00; H04N5/74; G06T2207/00; G06T2207/20; G06T2207/20212; G06T2207/20221
Claims Analysis
Total Claims: 68
Claim 1
Independent Claim
147 words
A method of forming a processed image using a plurality of images, each image generated by a respective sensor, comprising: dividing each image into a plurality of image regions; generating a contrast map for each image, each contrast map including a contrast value for each image region; applying a selection process to said contrast value for selecting an image region for use in said processed image; and forming said processed image with the selected image regions, wherein contrast values of contrast maps of respective first, second, and third sensors are compared together to form said processed image, the method further comprising: identifying contrast values from first and second sensor images to form an intermediate contrast map; wherein applying the selection process comprises applying a selection process to the contrast values of the intermediate contrast map and contrast values of a contrast map of a third sensor image.
Claim 2
Independent Claim
147 words
A method of forming a processed image using a plurality of images, each image generated by a respective sensor, comprising: dividing each image into a plurality of image regions; generating a contrast map for each image, each contrast map including a contrast value for each image region; applying a selection process to said contrast value for selecting an image region for use in said processed image; and forming said processed image with the selected image regions, wherein contrast values of contrast maps of respective first, second, and third sensors are compared together to form said processed image, the method further comprising: identifying contrast values from first and second sensor images to form an intermediate contrast map; wherein applying the selection process comprises applying a selection process to the contrast values of the intermediate contrast map and contrast values of a contrast map of a third sensor image.
Claim 3
Dependent Claim
34 words
The method of claim 1 , wherein dividing the images into the plurality of image regions further comprises dividing each image on a pixel-by-pixel basis, into blocks of pixels, or into arbitrary shaped regions.
Claim 4
Dependent Claim
13 words
The method of claim 1 , wherein each sensor detects a different wavelength.
Claim 5
Dependent Claim
20 words
The method of claim 1 , wherein the plurality of sensors includes an infrared (IR) sensor and a radar sensor.
Claim 6
Dependent Claim
20 words
The method of claim 1 , wherein the plurality of sensors includes an infrared (IR) and an ultraviolet (UV) sensor.
Claim 7
Dependent Claim
20 words
The method of claim 1 , wherein the plurality of sensors includes a radar sensor and an ultraviolet (UV) sensor.
Claim 8
Dependent Claim
27 words
The method of claim 1 , wherein the plurality of images are generated by two or more infrared (IR) sensors, each IR sensor detecting a different wavelength.
Claim 9
Dependent Claim
25 words
The method of claim 1 , wherein applying the selection process includes comparing competing contrast values of two corresponding image regions from two respective images.
Claim 10
Dependent Claim
20 words
The method of claim 8 , wherein said selection process operates to select the greater of the competing contrast values.
Claim 11
Dependent Claim
21 words
The method of claim 8 , wherein comparing competing contrast values further comprises comparing corresponding contrast values of overlapping image regions.
Claim 12
Dependent Claim
24 words
The method of claim 1 , wherein the first and second sensors are infrared (IR) sensors and the third sensor is a radar sensor.
Claim 13
Dependent Claim
17 words
The method of claim 1 , wherein a sensor image displays a view from a moving vehicle.
Claim 14
Dependent Claim
20 words
The method of claim 12 , wherein the moving vehicle is an aircraft, a watercraft, an automobile, or a train.
Claim 15
Dependent Claim
20 words
The method of claim 1 , further comprising adjusting an intensity of one or more regions of said processed image.
Claim 16
Dependent Claim
14 words
The method of claim 14 , further comprising weighting the degree of intensity adjustment.
Claim 17
Dependent Claim
19 words
The method of claim 14 , wherein adjusting the intensity further comprises adjusting the intensity across said processed image.
Claim 18
Dependent Claim
65 words
The method of claim 14 , wherein adjusting the intensity across said processed image further comprises: selecting one sensor as a reference sensor; determining at least one average intensity value for each region of the reference sensor image; and adjusting the intensity of one or more regions in said processed image by combining the determined average intensity values and intensity values of said processed image.
Claim 19
Dependent Claim
27 words
The method of claim 17 , wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the radar sensor.
Claim 20
Dependent Claim
28 words
The method of claim 17 , wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the infrared (IR) sensor.
Claim 21
Dependent Claim
47 words
The method of claim 17 , wherein adjusting the intensity of one or more regions in said processed image further comprises adjusting the intensity of a line in said processed image corresponding to a line in the reference sensor image for which the average intensity was determined.
Claim 22
Dependent Claim
58 words
The method of claim 20 , wherein adjusting the intensity of one or more lines in said processed image further comprises adjusting the intensity of a line in said processed image that is adjacent to a line in said processed image corresponding to the same line in the reference sensor image for which the average intensity was determined.
Claim 23
Dependent Claim
23 words
The method of claim 1 , before generating the contrast map for each image, further comprising filtering regions of one or more images.
Claim 24
Dependent Claim
21 words
The method of claim 22 , wherein filtering further comprises spatially filtering regions of each image by weighting selected image regions.
Claim 25
Dependent Claim
27 words
The method of claim 23 , wherein one sensor comprises a radar sensor, and wherein spatial filtering is performed by filtering image regions above a radar horizon.
Claim 26
Dependent Claim
29 words
The method of claim 1 , wherein each of the sensors detects substantially the same scene such that each of the images is associated with substantially the same scene.
Claim 27
Dependent Claim
43 words
The method of claim 25 , wherein each of the sensors detects a range of wavelengths that is different than a range of wavelengths detected by the other sensors, such that each of the images is associated with a different range of wavelengths.
Claim 28
Independent Claim
207 words
A method of forming a processed image using a plurality of images, each image generated by a respective sensor, comprising: dividing each image into a plurality of image regions; generating a contrast map for each image, each contrast map including a contrast value for each image region; applying a selection process to said contrast value for selecting an image region for use in said processed image; and forming said processed image with the selected image regions; wherein generating the contrast map further comprises performing a convolution to determine the contrast value of the contrast map; wherein performing the convolution further comprises performing the convolution with a Kernel Kc, wherein [{Kc*S 1 (x,y), Kc*S 2 (x,y)}] represents the convolution; K c = - 1 2 2 - 1 2 - 1 2 2 - 1 2 2 1 + 2 2 - 1 2 - 1 2 2 - 1 2 - 1 2 2 ≅ - 0.354 - 0.500 - 0.354 - 0.500 3.414 - 0.500 - 0.354 - 0.500 - 0.354 S 1 represents image regions of a first image; S 2 represents image regions of a second image; and (x,y) represent spatial coordinates of the images.
Claim 29
Independent Claim
207 words
A method of forming a processed image using a plurality of images, each image generated by a respective sensor, comprising: dividing each image into a plurality of image regions; generating a contrast map for each image, each contrast map including a contrast value for each image region; applying a selection process to said contrast value for selecting an image region for use in said processed image; and forming said processed image with the selected image regions; wherein generating the contrast map further comprises performing a convolution to determine the contrast value of the contrast map; wherein performing the convolution further comprises performing the convolution with a Kernel Kc, wherein [{Kc*S 1 (x,y), Kc*S 2 (x,y)}] represents the convolution; K c = - 1 2 2 - 1 2 - 1 2 2 - 1 2 2 1 + 2 2 - 1 2 - 1 2 2 - 1 2 - 1 2 2 ≅ - 0.354 - 0.500 - 0.354 - 0.500 3.414 - 0.500 - 0.354 - 0.500 - 0.354 S 1 represents image regions of a first image; S 2 represents image regions of a second image; and (x,y) represent spatial coordinates of the images.
Claim 30
Dependent Claim
34 words
The method of claim 27 , wherein dividing the images into the plurality of image regions further comprises dividing each image on a pixel-by-pixel basis, into blocks of pixels, or into arbitrary shaped regions.
Claim 31
Dependent Claim
13 words
The method of claim 27 , wherein each sensor detects a different wavelength.
Claim 32
Dependent Claim
25 words
The method of claim 27 , wherein applying the selection process includes comparing competing contrast values of two corresponding image regions from two respective images.
Claim 33
Dependent Claim
20 words
The method of claim 30 , wherein said selection process operates to select the greater of the competing contrast values.
Claim 34
Dependent Claim
21 words
The method of claim 30 , wherein comparing competing contrast values further comprises comparing corresponding contrast values of overlapping image regions.
Claim 35
Independent Claim
160 words
A system for combining a plurality of images to form a final image, comprising: a plurality of sensors that generate respective images; a processor configured to divide each image into a plurality of image regions, generate a contrast map for each image, each contrast map including a contrast value for each image region, apply a selection criterion to said contrast value for selecting an image region for use in said processed image, and form said processed image with the selected image regions; wherein contrast values of contrast maps of a respective first, second, and third sensors are compared together to form the final image; wherein the processor is further configured to identify contrast values from first and second sensor images to form an intermediate contrast map; and wherein the processor applies the selection criterion by applying a selection process to the contrast values of the intermediate contrast map and contrast values of a contrast map of a third sensor image.
Claim 36
Independent Claim
160 words
A system for combining a plurality of images to form a final image, comprising: a plurality of sensors that generate respective images; a processor configured to divide each image into a plurality of image regions, generate a contrast map for each image, each contrast map including a contrast value for each image region, apply a selection criterion to said contrast value for selecting an image region for use in said processed image, and form said processed image with the selected image regions; wherein contrast values of contrast maps of a respective first, second, and third sensors are compared together to form the final image; wherein the processor is further configured to identify contrast values from first and second sensor images to form an intermediate contrast map; and wherein the processor applies the selection criterion by applying a selection process to the contrast values of the intermediate contrast map and contrast values of a contrast map of a third sensor image.
Claim 37
Dependent Claim
27 words
The system of claim 33 , wherein the processor is configured to divide each image into individual pixels, into blocks of pixels, or into arbitrary shaped regions.
Claim 38
Dependent Claim
13 words
The system of claim 33 , wherein each sensor detects a different wavelength.
Claim 39
Dependent Claim
20 words
The system of claim 33 , wherein the plurality of sensors includes an infrared (IR) sensor and a radar sensor.
Claim 40
Dependent Claim
20 words
The system of claim 33 , wherein the plurality of sensors includes an infrared (IR) and an ultraviolet (UV) sensor.
Claim 41
Dependent Claim
20 words
The system of claim 33 , wherein the plurality of sensors includes a radar sensor and an ultraviolet (UV) sensor.
Claim 42
Dependent Claim
25 words
The system of claim 33 , wherein the plurality of sensors includes two or more infrared (IR) sensors, each IR sensor detecting a different wavelength.
Claim 43
Dependent Claim
26 words
The system of claim 33 , wherein the processor is further configured to compare competing contrast values of two corresponding image regions from two respective images.
Claim 44
Dependent Claim
21 words
The system of claim 40 , wherein the processor is further configured to select the greater of the competing contrast values.
Claim 45
Dependent Claim
20 words
The system of claim 40 , wherein the processor is configured to compare corresponding contrast values of overlapping image regions.
Claim 46
Dependent Claim
24 words
The system of claim 33 , wherein the first and second sensors are infrared (IR) sensors and the third sensor is a radar sensor.
Claim 47
Dependent Claim
17 words
The system of claim 33 , wherein a sensor image displays a view from a moving vehicle.
Claim 48
Dependent Claim
20 words
The system of claim 44 , wherein the moving vehicle comprises an aircraft, a watercraft, an automobile, or a train.
Claim 49
Dependent Claim
25 words
The system of claim 33 , wherein the processor is further configured to adjust an intensity of one or more regions of said processed image.
Claim 50
Dependent Claim
19 words
The system of claim 46 , wherein the processor is configured to adjust the intensity across said processed image.
Claim 51
Dependent Claim
18 words
The system of claim 47 , wherein the processor is configured to weight the degree of intensity adjustment.
Claim 52
Dependent Claim
67 words
The system of claim 47 , wherein the processor is further configured to select one sensor as a reference sensor, determine at least one average intensity value for each region of the reference sensor image across the reference sensor image, and adjust the intensity of one or more regions in said processed image by combining the determined average intensity values and intensity values of said processed image.
Claim 53
Dependent Claim
27 words
The system of claim 49 , wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the radar sensor.
Claim 54
Dependent Claim
28 words
The system of claim 49 , wherein the sensors include a radar sensor and an infrared (IR) sensor, and wherein the reference sensor comprises the infrared (IR) sensor.
Claim 55
Dependent Claim
38 words
The system of claim 49 , wherein the processor is configured to adjust the intensity of a line in said processed image corresponding to a line in the reference sensor image for which the average intensity was determined.
Claim 56
Dependent Claim
49 words
The system of claim 49 , wherein the processor is configured to adjust the intensity of a line in said processed image that is adjacent to a line in said processed image corresponding to the same line in the reference sensor image for which the average intensity was determined.
Claim 57
Dependent Claim
18 words
The system of claim 33 , wherein the processor is configured to filter one or more image regions.
Claim 58
Dependent Claim
24 words
The system of claim 54 , wherein the processor is configured to filter the one or more image regions by weighting selected image regions.
Claim 59
Dependent Claim
29 words
The system of claim 54 , wherein one sensor comprises a radar sensor, and wherein the processor is further configured to spatially filter image regions above a radar horizon.
Claim 60
Dependent Claim
30 words
The system of claim 33 , wherein each of the sensors detects substantially the same scene such that each of the respective images is associated with substantially the same scene.
Claim 61
Dependent Claim
43 words
The system of claim 33 , wherein each of the sensors detects a range of wavelengths that is different than a range of wavelengths detected the other sensors, such that each of the respective images is associated with a different range of wavelengths.
Claim 62
Independent Claim
219 words
A system for combining a plurality of images to form a final image, comprising: a plurality of sensors that generate respective images; and a processor configured to divide each image into a plurality of image regions, generate a contrast map for each image, each contrast map including a contrast value for each image region, apply a selection criterion to said contrast value for selecting an image region for use in said processed image, and form said processed image with the selected image regions; wherein the processor is configured to generate the contrast map by performing a convolution to determine the contrast value of the contrast map; and wherein the processor is configured to perform the convolution with a Kernel Kc, wherein [{Kc*S 1 (x,y), Kc*S 2 (x,y)}] represents the convolution; K c = - 1 2 2 - 1 2 - 1 2 2 - 1 2 2 1 + 2 2 - 1 2 - 1 2 2 - 1 2 - 1 2 2 ≅ - 0.354 - 0.500 - 0.354 - 0.500 3.414 - 0.500 - 0.354 - 0.500 - 0.354 S 1 represents image regions of a first image; S 2 represents image regions of a second image; and (x,y) represent spatial coordinates of the images.
Claim 63
Independent Claim
219 words
A system for combining a plurality of images to form a final image, comprising: a plurality of sensors that generate respective images; and a processor configured to divide each image into a plurality of image regions, generate a contrast map for each image, each contrast map including a contrast value for each image region, apply a selection criterion to said contrast value for selecting an image region for use in said processed image, and form said processed image with the selected image regions; wherein the processor is configured to generate the contrast map by performing a convolution to determine the contrast value of the contrast map; and wherein the processor is configured to perform the convolution with a Kernel Kc, wherein [{Kc*S 1 (x,y), Kc*S 2 (x,y)}] represents the convolution; K c = - 1 2 2 - 1 2 - 1 2 2 - 1 2 2 1 + 2 2 - 1 2 - 1 2 2 - 1 2 - 1 2 2 ≅ - 0.354 - 0.500 - 0.354 - 0.500 3.414 - 0.500 - 0.354 - 0.500 - 0.354 S 1 represents image regions of a first image; S 2 represents image regions of a second image; and (x,y) represent spatial coordinates of the images.
Claim 64
Dependent Claim
27 words
The system of claim 59 , wherein the processor is configured to divide each image into individual pixels, into blocks of pixels, or into arbitrary shaped regions.
Claim 65
Dependent Claim
13 words
The system of claim 59 , wherein each sensor detects a different wavelength.
Claim 66
Dependent Claim
26 words
The system of claim 59 , wherein the processor is further configured to compare competing contrast values of two corresponding image regions from two respective images.
Claim 67
Dependent Claim
21 words
The system of claim 62 , wherein the processor is further configured to select the greater of the competing contrast values.
Claim 68
Dependent Claim
20 words
The system of claim 62 , wherein the processor is configured to compare corresponding contrast values of overlapping image regions.
References Cited (Backward Citations)
US4661986A
Cited
US4674125A
Cited
US5686960A
Cited
US5325449A
Cited
US5510772A
Cited
US5881163A
Cited
US6208766B1
Cited
US5778108A
Cited
US6137498A
Cited
US6163309A
Cited
US6320979B1
Cited
US6201899B1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2008-05-23 | 2008-11-22 | 2009-05-24 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2012-05-23 | 2012-11-22 | 2013-05-24 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2016-05-23 | 2016-11-22 | 2017-05-24 | $7700.00 | Window Closed / Lapsed |