US8711249B2
Active / In Force
US
⏱️ 0.4 years remaining
Method of and apparatus for image denoising
Assignee: Sony Corp
•
Filing: 2007-03-29
•
Granted: 2014-04-29
•
Expires: 2027-03-29
Abstract
Abstract An image denoising system and method of implementing the image denoising system is described herein. Noise is decomposed within each channel into frequency bands, and sub-band noise is propagated. Denoising is then able to occur at any node in a camera pipeline after accurately predicting noise that is signal level-dependent, frequency dependent and has inter-channel correlation. A methodology is included for estimating image noise in each color channel at a sensor output based on average image level and camera noise parameters. A scheme is implemented for detecting a peak-white image level for each color channel and predicting image level values for representative colors. Based on a noise model and camera parameters, noise levels are predicted for each color channel for each color patch and these noise levels are propagated to the denoising node. A three dimentional LUT correlates signal level to noise level. Then, a denoising threshold is adaptively controlled.
Inventors
- 👤 Farhan A. Baqai
- 👤 Akira Matsui
- 👤 Kenichi Nishio
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:05
CPC Classifications
H; H04; H04N; H04N1/00; G; G06; G06T; G06T5/00; G06T5/70; H04N23/00; H04N23/80; H04N23/84; H04N23/843; H04N23/88; H04N9/00; H04N9/64; H04N9/646
Claims Analysis
Total Claims: 39
Claim 1
Independent Claim
92 words
A method of denoising an image programmed in a memory of a device comprising: a. acquiring an image wherein the image contains noise; b. splitting the noise into a plurality of sub-bands of noise; c. determining a threshold for each sub-band of noise wherein the threshold is a multiple of a predicted noise amount, wherein the predicted noise amount is determined based on a computation comprising a conversion of a pixel value using a lookup table and interpolation; and d. filtering out the noise below the threshold for each sub-band of noise.
Claim 2
Independent Claim
92 words
A method of denoising an image programmed in a memory of a device comprising: a. acquiring an image wherein the image contains noise; b. splitting the noise into a plurality of sub-bands of noise; c. determining a threshold for each sub-band of noise wherein the threshold is a multiple of a predicted noise amount, wherein the predicted noise amount is determined based on a computation comprising a conversion of a pixel value using a lookup table and interpolation; and d. filtering out the noise below the threshold for each sub-band of noise.
Claim 3
Dependent Claim
15 words
The method as claimed in claim 1 further comprising acquiring one or more camera parameters.
Claim 4
Dependent Claim
21 words
The method as claimed in claim 2 further comprising computing input noise matrices by incorporating the one or more camera parameters.
Claim 5
Dependent Claim
16 words
The method as claimed in claim 3 further comprising propagating band-wise noise from the noise matrices.
Claim 6
Dependent Claim
22 words
The method as claimed in claim 4 further comprising generating a signal-to-noise look up table to correlate signal values and noise values.
Claim 7
Dependent Claim
20 words
The method as claimed in claim 5 further comprising predicting a pixel-by-pixel noise amount using the signal-to-noise look up table.
Claim 8
Dependent Claim
14 words
The method as claimed in claim 1 wherein the threshold is a pixel-by-pixel threshold.
Claim 9
Dependent Claim
17 words
The method as claimed in claim 7 wherein the noise is filtered out by the pixel-by-pixel threshold.
Claim 10
Dependent Claim
32 words
The method as claimed in claim 2 wherein the one or more camera parameters include at least one of amplifier gain, auto-white balancing gains, edge enhancement gain and color correction matrices parameters.
Claim 11
Dependent Claim
23 words
The method as claimed in claim 2 wherein the one or more parameters are available at a processor when implemented on a camera.
Claim 12
Dependent Claim
31 words
The method as claimed in claim 2 wherein the one or more parameters are available via header information in an image when implemented with image rendering software on a computing device.
Claim 13
Dependent Claim
22 words
The method as claimed in claim 4 wherein propagating band-wise noise further includes band-splitting the input noise matrices into N sub-bands each.
Claim 14
Dependent Claim
14 words
The method as claimed in claim 1 wherein the steps a-c occur in real-time.
Claim 15
Dependent Claim
66 words
The method as claimed in claim 6 wherein predicting the pixel-by-pixel noise amount further includes: a. inputting a pixel value into the look up table; b. converting the input pixel value to the predicted noise amount; c. mapping signal value on an input signal to band-wise noise matrices using the look up table; and d. interpolating and extrapolating band-wise noise matrices according to the signal value.
Claim 16
Dependent Claim
19 words
The method as claimed in claim 5 further comprising updating the look up table when a shooting condition changes.
Claim 17
Independent Claim
139 words
A method of denoising an image programmed in a memory of a device comprising: a. acquiring one or more camera parameters; b. computing input noise matrices by incorporating the one or more camera parameters; c. propagating band-wise noise from the input noise matrices, wherein propagating the band-wise noise is repeated for a number of patches in the image; d. generating a signal-to-noise look up table to correlate signal values and noise values, wherein the signal-to-noise look up table is only updated when a shooting condition changes; e. predicting a pixel-by-pixel noise amount using the signal-to-noise look up table, wherein the predicted pixel-by-pixel noise amount is determined based on a computation; f. controlling a pixel-by-pixel threshold, wherein the pixel-by-pixel threshold is a multiple of the pixel-by-pixel noise amount; and g. filtering out noise in the image using the pixel-by-pixel threshold.
Claim 18
Independent Claim
139 words
A method of denoising an image programmed in a memory of a device comprising: a. acquiring one or more camera parameters; b. computing input noise matrices by incorporating the one or more camera parameters; c. propagating band-wise noise from the input noise matrices, wherein propagating the band-wise noise is repeated for a number of patches in the image; d. generating a signal-to-noise look up table to correlate signal values and noise values, wherein the signal-to-noise look up table is only updated when a shooting condition changes; e. predicting a pixel-by-pixel noise amount using the signal-to-noise look up table, wherein the predicted pixel-by-pixel noise amount is determined based on a computation; f. controlling a pixel-by-pixel threshold, wherein the pixel-by-pixel threshold is a multiple of the pixel-by-pixel noise amount; and g. filtering out noise in the image using the pixel-by-pixel threshold.
Claim 19
Dependent Claim
32 words
The method as claimed in claim 16 wherein the one or more camera parameters include at least one of amplifier gain, auto-white balancing gains, edge enhancement gain and color correction matrices parameters.
Claim 20
Dependent Claim
23 words
The method as claimed in claim 16 wherein the one or more parameters are available at a processor when implemented on a camera.
Claim 21
Dependent Claim
31 words
The method as claimed in claim 16 wherein the one or more parameters are available via header information in an image when implemented with image rendering software on a computing device.
Claim 22
Dependent Claim
21 words
The method as claimed in claim 16 wherein propagating band-wise noise further includes band-splitting an input noise matrix into N sub-bands.
Claim 23
Dependent Claim
14 words
The method as claimed in claim 16 wherein the steps a-d occur in real-time.
Claim 24
Dependent Claim
66 words
The method as claimed in claim 16 wherein predicting the pixel-by-pixel noise amount further includes: a. inputting a pixel value into the look up table; b. converting the input pixel value to a predicted noise amount; c. mapping signal value on an input signal to band-wise noise matrices using the look up table; and d. interpolating and extrapolating band-wise noise matrices according to the signal value.
Claim 25
Dependent Claim
19 words
The method as claimed in claim 16 further comprising updating the look up table when a shooting condition changes.
Claim 26
Independent Claim
113 words
A system for denoising an image comprising: a. an image processing pipeline comprising a noise reduction block coupled to the set of imaging components for reducing noise in the image, wherein the noise reduction block splits the noise into sub-bands and utilizes thresholds received from the noise reduction control unit to filter the noise; and b. a noise reduction control unit coupled to the image processing pipeline, wherein the noise reduction control unit determines threshold values based on a multiple of a predicted noise amount to reduce the noise, wherein the predicted noise amount is determined based on a computation comprising a conversion of a pixel value using a lookup table and interpolation.
Claim 27
Independent Claim
113 words
A system for denoising an image comprising: a. an image processing pipeline comprising a noise reduction block coupled to the set of imaging components for reducing noise in the image, wherein the noise reduction block splits the noise into sub-bands and utilizes thresholds received from the noise reduction control unit to filter the noise; and b. a noise reduction control unit coupled to the image processing pipeline, wherein the noise reduction control unit determines threshold values based on a multiple of a predicted noise amount to reduce the noise, wherein the predicted noise amount is determined based on a computation comprising a conversion of a pixel value using a lookup table and interpolation.
Claim 28
Dependent Claim
55 words
The system as claimed in claim 24 wherein the image processing pipeline further comprises: a. an imager for acquiring the image; b. a set of imaging components coupled to the imager for processing the image; and c. a set of post-processing components coupled to the noise reduction block for processing the image after noise reduction.
Claim 29
Dependent Claim
20 words
The system as claimed in claim 24 wherein the noise reduction control unit generates the thresholds based on the sub-bands.
Claim 30
Dependent Claim
18 words
The system as claimed in claim 24 wherein the noise reduction control unit is implemented on a processor.
Claim 31
Dependent Claim
30 words
The system as claimed in claim 24 wherein the noise reduction control unit implements an imager noise model, noise propagation and predicted band-wise noise matrices to determine the threshold values.
Claim 32
Independent Claim
182 words
A system for denoising an image comprising: a. an image processing pipeline, wherein the image processing pipeline further comprises: i. an imager for acquiring the image; ii. a set of imaging components coupled to the imager for processing the image; iii. a noise reduction block coupled to the set of imaging components for reducing the noise in the image; and iv. a set of post-processing components coupled to the noise reduction block for processing the image after noise reduction; and b. a processor implementing a noise reduction control unit function coupled to the image processing pipeline, wherein the processor determines threshold values to reduce noise wherein the threshold values are multiples of predicted noise amounts, wherein the noise reduction block splits the noise into sub-bands and utilizes thresholds received from the noise reduction control unit to filter the noise, wherein a three dimensional signal-to-noise lookup table is used to correlate signal value and noise value, wherein the predicted noise amounts are determined based on a computation comprising a conversion of a pixel value using the three dimensional signal-to-noise lookup table and interpolation.
Claim 33
Independent Claim
182 words
A system for denoising an image comprising: a. an image processing pipeline, wherein the image processing pipeline further comprises: i. an imager for acquiring the image; ii. a set of imaging components coupled to the imager for processing the image; iii. a noise reduction block coupled to the set of imaging components for reducing the noise in the image; and iv. a set of post-processing components coupled to the noise reduction block for processing the image after noise reduction; and b. a processor implementing a noise reduction control unit function coupled to the image processing pipeline, wherein the processor determines threshold values to reduce noise wherein the threshold values are multiples of predicted noise amounts, wherein the noise reduction block splits the noise into sub-bands and utilizes thresholds received from the noise reduction control unit to filter the noise, wherein a three dimensional signal-to-noise lookup table is used to correlate signal value and noise value, wherein the predicted noise amounts are determined based on a computation comprising a conversion of a pixel value using the three dimensional signal-to-noise lookup table and interpolation.
Claim 34
Dependent Claim
16 words
The system as claimed in claim 29 wherein the processor generates thresholds based on the sub-bands.
Claim 35
Dependent Claim
27 words
The system as claimed in claim 29 wherein the processor implements an imager noise model, noise propagation and predicted band-wise noise matrices to determine the threshold values.
Claim 36
Independent Claim
95 words
A method of denoising an image comprising: a. detecting an average image level at a sensor for each color channel in a set of color channels; b. estimating an average noise level for each color channel in the set of color channels; c. predicting noise at a node where denoising is performed; and d. controlling an adaptive threshold for denoising, wherein the threshold is a multiple of a predicted noise amount, wherein the predicted noise amount is determined based on a computation comprising a conversion of a pixel value using a lookup table and interpolation.
Claim 37
Independent Claim
95 words
A method of denoising an image comprising: a. detecting an average image level at a sensor for each color channel in a set of color channels; b. estimating an average noise level for each color channel in the set of color channels; c. predicting noise at a node where denoising is performed; and d. controlling an adaptive threshold for denoising, wherein the threshold is a multiple of a predicted noise amount, wherein the predicted noise amount is determined based on a computation comprising a conversion of a pixel value using a lookup table and interpolation.
Claim 38
Independent Claim
119 words
A method of denoising an image comprising: a. detecting an average image level at an image sensor for each color channel in a set of color channels; b. predicting image levels for a set of representative colors; c. computing a noise level of each color patch and each color channel; d. predicting noise at a node where denoising is performed; e. generating a three dimensional look up table to correlate signal level to noise level; and f. adaptively controlling a threshold setting for denoising, wherein the threshold is a multiple of a predicted noise amount, wherein the predicted noise amount is determined based on a computation comprising a conversion of a pixel value using a lookup table and interpolation.
Claim 39
Independent Claim
119 words
A method of denoising an image comprising: a. detecting an average image level at an image sensor for each color channel in a set of color channels; b. predicting image levels for a set of representative colors; c. computing a noise level of each color patch and each color channel; d. predicting noise at a node where denoising is performed; e. generating a three dimensional look up table to correlate signal level to noise level; and f. adaptively controlling a threshold setting for denoising, wherein the threshold is a multiple of a predicted noise amount, wherein the predicted noise amount is determined based on a computation comprising a conversion of a pixel value using a lookup table and interpolation.
References Cited (Backward Citations)
US3971065A
Cited
US4142071A
Cited
US5012333A
Cited
US6691086B2
Cited
JPH0556309A
Cited
JPH07249101A
Cited
US6098038A
Cited
US6148103A
Cited
US20040019492A1
Cited
US6226321B1
Cited
US7209567B1
Cited
US20030152283A1
Cited
US7072511B2
Cited
US6480610B1
Cited
US6813389B1
Cited
US20020015528A1
Cited
US20060204124A1
Cited
WO2001074058A1
Cited
US20020034337A1
Cited
WO2002005262A2
Cited
CN1460323A
Cited
US6377637B1
Cited
US7133824B2
Cited
US20040212692A1
Cited
WO2003041390A3
Cited
WO2003041390A2
Cited
US20030138141A1
Cited
JP2005509167A
Cited
US20030161549A1
Cited
JP2003284084A
Cited
US20030202693A1
Cited
US20030235343A1
Cited
US20050062861A1
Cited
JP2004088149A
Cited
US20050099515A1
Cited
CN1675919A
Cited
JP2004208079A
Cited
US7636436B2
Cited
US20040252208A1
Cited
WO2005029846A1
Cited
JP2005303803A
Cited
WO2007002406A2
Cited
US20070040914A1
Cited
US20070104387A1
Cited
US7885462B2
Cited
US20090021648A1
Cited
US20090161982A1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2017-04-28 | 2017-10-28 | 2018-04-29 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2021-04-28 | 2021-10-28 | 2022-04-29 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2025-04-28 | 2025-10-28 | 2026-04-29 | $7700.00 | Window Closed / Lapsed |