Patent EP0684585A2 successfully refreshed from Google Patents.
EP0684585A2
Expired / Lapsed
EP
⏱️ 0.0 years remaining
Image forming method and apparatus
Assignee: Canon Inc
•
Filing: 1995-04-21
•
Granted: —
•
Expires: 2015-04-21
Abstract
Abstract Upon observation of multi-viewpoint images, even when the viewpoint moves continuously, the image cannot normally be displayed in correspondence with the continuous movement of the viewpoint, and an image from a viewpoint closest to the current viewpoint is displayed. To prevent this, each two adjacent images of the multi-viewpoint images are interpolated by detecting corresponding pixels and generating an interpolated image in accordance with the correspondences among pixels. The interpolated image is generated in such a manner that an epipolar plane image is generated by juxtaposing corresponding lines of the multi-viewpoint image, straight lines are detected from the epipolar plane image, and the intersecting points between interpolated lines and the detected straight lines are detected as interpolated pixels. In this case, as for a point for which a straight line cannot be detected, a straight line passing the point is estimated, thereby generating an interpolated pixel. When a plurality of straight lines are detected for a pixel of interest, the priority level is assigned to the pixel of interest in correspondence with the number of detected straight lines. When the priority level is smaller than that of another pixel through which the detected straight line passes, the straight line is ignored, and an interpolated pixel is not generated.
Inventors
- 👤 Takahiro C/O Canon K.K. Oshino
- 👤 Hideyuki C/O Canon K.K. Tamura
- 👤 Akihiro C/O Canon K.K. Katayama
- 👤 Hiroyuki C/O Canon K.K. Yamamoto
- 👤 Koichiro C/O Canon K.K. Tanaka
Bibliographic Details
Application Number
—
Jurisdiction
EP
Data Source
google_patents
Last Synchronized
2026-10-26 11:46
CPC Classifications
G; G06; G06T; G06T15/00; G06T15/10; G06T15/20; G06T15/205; G06T7/00; G06T7/97; G06V; G06V10/00; G06V10/10; G06V10/12; G06V10/14; G06V10/147
Claims Analysis
Total Claims: 64
Claim 1
Independent Claim
55 words
An image processing method comprising: the detection step of detecting correspondences of pixels among multi-viewpoint images; the first generation step of generating pixels constituting an interpolated images on the basis of the detected correspondences; and the construction step of constructing the interpolated image on the basis of the pixels generated in the first generation step.
Claim 2
Dependent Claim
55 words
The method according to claim 1, further comprising: the estimation step of estimating correspondences of non-processed pixels, from which correspondences cannot be detected in the detection step, among the multi-viewpoint images; and the second generation step of further generating pixels constituting the interpolated image on the basis of the correspondences estimated in the estimation step.
Claim 3
Dependent Claim
77 words
The method according to claim 2, wherein the estimation step includes the step of detecting a line segment consisting of pixels having a value closest to a value of the non-processed pixels from line segments passing the non-processed pixel on an epipolar plane image constituted by the multi-viewpoint images, and the second generation step includes the step of setting an average value of the pixels constituting the line segment to be the value of the non-processed pixels.
Claim 4
Dependent Claim
53 words
The method according to claim 3, wherein the estimation step includes the step of detecting, from line segments each having a length of five pixels and having a non-processed pixel as a center, a line segment which has a minimum sum of differences between pixels constituting the line segment and the non-processed pixel.
Claim 5
Dependent Claim
98 words
The method according to claim 2, wherein the detection step includes the step of generating an epipolar plane image from the multi-viewpoint images, and detecting pixels constituting a linear image passing a pixel of interest on the epipolar plane image, and the first generation step includes the step of inserting an interpolated line between each two adjacent lines of the epipolar plane image, and assigning, to a pixel located at an intersecting point between a straight line connecting pixels detected in the detection step, and the interpolated line, an average value of the linear image passing the pixel.
Claim 6
Dependent Claim
59 words
The method according to claim 5, wherein the detection step includes the step of detecting a straight line passing the pixel of interest as the linear image when a pixel with a value which has a square of a difference from a value of the pixel of interest falling within a predetermined value is present on the straight line.
Claim 7
Dependent Claim
24 words
The method according to claim 6, wherein the detection step includes the step of detecting the linear image while increasing the predetermined value stepwise.
Claim 8
Dependent Claim
61 words
The method according to claim 5, wherein the detection step includes the step of storing pixels together with priority levels corresponding to a detection order of the pixels when the pixels constituting the linear image are detected, and the first generation step includes the step of generating pixels constituting the interpolated image in the order from pixels with lower priority levels.
Claim 9
Dependent Claim
41 words
The method according to claim 1, further comprising the correction step of performing correction processing of images of the multi-viewpoint images, and wherein the detection step includes the step of detecting correspondences from the multi-viewpoint images corrected in the correction step.
Claim 10
Dependent Claim
20 words
The method according to claim 1, further comprising the output step of outputting the image constructed in the construction step.
Claim 11
Dependent Claim
41 words
The method according to claim 1, further comprising the viewpoint detection step of detecting a position of a viewpoint, and wherein the first generation step includes the step of generating an image from the viewpoint detected in the viewpoint detection step.
Claim 12
Dependent Claim
47 words
The method according to claim 1, wherein the first generation step includes the step of generating stereoscopic images which have viewpoint positions deviated by a distance corresponding to a binocular parallax, and the output step includes the step of displaying the stereoscopic images on a stereoscopic display.
Claim 13
Dependent Claim
66 words
The method according to claim 1, wherein the detection step includes the step of setting a score corresponding to the number of correspondences of a pixel of interest for the pixel of interest, and ignoring a correspondence between the pixel of interest and a corresponding pixel when the score is lower than a score of the pixel of interest, which is set for the corresponding pixel.
Claim 14
Dependent Claim
26 words
The method according to claim 13, wherein the score is set to become lower as the number of correspondences of the pixel of interest is larger.
Claim 15
Dependent Claim
104 words
The method according to claim 13, wherein the detection step includes the step of generating an epipolar plane image from the multi-viewpoint images, and detecting a linear image passing a pixel of interest on the epipolar plane image as a correspondence for the pixel of interest, and the first generation step includes the step of inserting an interpolated line between each two adjacent lines of the epipolar plane image, and assigning, to a pixel located at an intersecting point between a straight line connecting pixels detected in the detection step, and the interpolated line, an average value of the linear image passing the pixel.
Claim 16
Dependent Claim
59 words
The method according to claim 15, wherein the detection step includes the step of detecting a straight line passing the pixel of interest as the linear image when a pixel with a value which has a square of a difference from a value of the pixel of interest falling within a predetermined value is present on the straight line.
Claim 17
Dependent Claim
24 words
The method according to claim 16, wherein the detection step includes the step of detecting the linear image while increasing the predetermined value stepwise.
Claim 18
Dependent Claim
61 words
The method according to claim 15, wherein the detection step includes the step of storing pixels together with priority levels corresponding to a detection order of the pixels when the pixels constituting the linear image are detected, and the first generation step includes the step of generating pixels constituting the interpolated image in the order from pixels with lower priority levels.
Claim 19
Dependent Claim
31 words
The method according to claim 14, further comprising the prediction step of predicting maximum and minimum values of an inclination of a straight line to be detected in the detection step.
Claim 20
Dependent Claim
103 words
An image processing method for processing multi-viewpoint images, comprising: the viewpoint detection step of detecting a position of a viewpoint; the generation step of generating, on the basis of the multi-viewpoint images, an image which has, as a virtual viewpoint, an intersecting point between a line of sight from the viewpoint detected in the viewpoint detection step and a viewpoint array corresponding to the multi-viewpoint images; and the correction step of correcting a distortion of an image caused by a deviation between the viewpoint detected in the viewpoint detection step and the virtual viewpoint on the basis of an amount of the deviation.
Claim 21
Dependent Claim
69 words
The method according to claim 20, wherein the generation step includes the detection step of detecting correspondences among pixels between each two adjacent images of the multi-viewpoint images, the interpolated image generation step of generating pixels constituting an interpolated image on the basis of the detected correspondences, and the construction step of constructing the interpolated image on the basis of the pixels generated in the interpolated image generation step.
Claim 22
Dependent Claim
46 words
The method according to claim 20, wherein the correction step includes the step of correcting a distortion, in an up-and-down direction of an image on the basis of a distance from a viewpoint to an object for the multi-viewpoint image, and the amount of the deviation.
Claim 23
Dependent Claim
46 words
The method according to claim 20, wherein the correction step includes the step of correcting a distortion, in a right-and-left direction of an image on the basis of a distance from a viewpoint to an object for the multi-viewpoint image, and the amount of the deviation.
Claim 24
Dependent Claim
42 words
The method according to claim 20, further comprising the correction step of performing correction processing of images of the multi-viewpoint images, and wherein the detection step includes the step of detecting the correspondences from the multi-viewpoint images corrected in the correction step.
Claim 25
Dependent Claim
20 words
The method according to claim 20, further comprising the output step of outputting the image constructed in the construction step.
Claim 26
Dependent Claim
46 words
The method according to claim 25, wherein the generation step includes the step of generating stereoscopic images which have viewpoint positions deviated by a distance corresponding to a binocular parallax, and the output step includes the step of displaying the stereoscopic images on a stereoscopic display.
Claim 27
Dependent Claim
101 words
An image processing method comprising: the detection step of detecting correspondences of pixels among multi-viewpoint images; the first generation step of generating pixels constituting an interpolated image on the basis of the detected correspondences; the estimation step of estimating correspondences of non-processed pixels, from which correspondences cannot be detected in the detection step, among the multi-viewpoint images; the second generation step of further generating pixels constituting the interpolated image on the basis of the correspondences estimated in the estimation step; and the construction step of constructing the interpolated image on the basis of the pixels generated in the first generation step.
Claim 28
Dependent Claim
114 words
An image processing method comprising: the detection step of detecting correspondences of pixels among multi-viewpoint images; the generation step of generating pixels constituting an interpolated image on the basis of the detected correspondences; and the construction step of constructing the interpolated image on the basis of the pixels generated in the generation step, wherein the detection step includes the step of ignoring a correspondence between a pixel of interest and a corresponding pixel when the corresponding pixel which has the correspondence with the pixel of interest also has a correspondence with another pixel, and the number of correspondences for the pixel of interest is larger than the number of correspondences for the corresponding pixel.
Claim 29
Dependent Claim
52 words
An image processing apparatus comprising: detection means for detecting correspondences of pixels among multi-viewpoint images; first generation means for generating pixels constituting an interpolated images on the basis of the detected correspondences; and construction means for constructing the interpolated image on the basis of the pixels generated by said first generation means.
Claim 30
Dependent Claim
53 words
The apparatus according to claim 29, further comprising: estimation means for estimating correspondences of non-processed pixels, from which correspondences cannot be detected by said detection means, among the multi-viewpoint images; and second generation means for further generating pixels constituting the interpolated image on the basis of the correspondences estimated by said estimation means.
Claim 31
Dependent Claim
69 words
The apparatus according to claim 29, wherein said estimation means detects a line segment consisting of pixels having a value closest to a value of a non-processed pixel from line segments passing the non-processed pixel on an epipolar plane image constituted by the multi-viewpoint images, and said second generation means sets an average value of the pixels constituting the line segment to be the value of the non-processed pixel.
Claim 32
Dependent Claim
49 words
The apparatus according to claim 31, wherein said estimation means detects, from line segments each having a length of five pixels and having a non-processed pixel as a center, a line segment which has a minimum sum of differences between pixels constituting the line segment and the non-processed pixel.
Claim 33
Dependent Claim
90 words
The apparatus according to claim 30, wherein said detection means generates an epipolar plane image from the multi-viewpoint images, and detects pixels constituting a linear image passing a pixel of interest on the epipolar plane image, and said first generation means inserts an interpolated line between each two adjacent lines of the epipolar plane image, and assigns, to a pixel located at an intersecting point between a straight line connecting pixels detected by said detection means, and the interpolated line, an average value of the linear image passing the pixel.
Claim 34
Dependent Claim
55 words
The apparatus according to claim 33, wherein said detection means detects a straight line passing the pixel of interest as the linear image when a pixel with a value which has a square of a difference from a value of the pixel of interest falling within a predetermined value is present on the straight line.
Claim 35
Dependent Claim
20 words
The apparatus according to claim 34, wherein said detection means detects the linear image while increasing the predetermined value stepwise.
Claim 36
Dependent Claim
53 words
The apparatus according to claim 33, wherein said detection means stores pixels together with priority levels corresponding to a detection order of the pixels when the pixels constituting the linear image are detected, and said first generation means generates pixels constituting the interpolated image in the order from pixels with lower priority levels.
Claim 37
Dependent Claim
36 words
The apparatus according to claim 29, further comprising correction means for performing correction processing of images of the multi-viewpoint images, and wherein said detection means detects correspondences from the multi-viewpoint images corrected by said correction means.
Claim 38
Dependent Claim
19 words
The apparatus according to claim 29, further comprising output means for outputting the image constructed by said construction means.
Claim 39
Dependent Claim
36 words
The apparatus according to claim 29, further comprising viewpoint detection means for detecting a position of a viewpoint, and wherein said first generation means generates an image from the viewpoint detected by said viewpoint detection means.
Claim 40
Dependent Claim
39 words
The apparatus according to claim 29, wherein said first generation means generates stereoscopic images which have viewpoint positions deviated by a distance corresponding to a binocular parallax, and said output means displays the stereoscopic images on a stereoscopic display.
Claim 41
Dependent Claim
62 words
The apparatus according to claim 29, wherein said detection means sets a score corresponding to the number of correspondences of a pixel of interest for the pixel of interest, and ignores a correspondence between the pixel of interest and a corresponding pixel when the score is lower than a score of the pixel of interest, which is set for the corresponding pixel.
Claim 42
Dependent Claim
26 words
The apparatus according to claim 41, wherein the score is set to become lower as the number of correspondences of the pixel of interest is larger.
Claim 43
Dependent Claim
96 words
The apparatus according to claim 41, wherein said detection means generates an epipolar plane image from the multi-viewpoint images, and detects a linear image passing a pixel of interest on the epipolar plane image as a correspondence for the pixel of interest, and said first generation means inserts an interpolated line between each two adjacent lines of the epipolar plane image, and assigns, to a pixel located at an intersecting point between a straight line connecting pixels detected by said detection means, and the interpolated line, an average value of the linear image passing the pixel.
Claim 44
Dependent Claim
55 words
The apparatus according to claim 43, wherein said detection means detects a straight line passing the pixel of interest as the linear image when a pixel with a value which has a square of a difference from a value of the pixel of interest falling within a predetermined value is present on the straight line.
Claim 45
Dependent Claim
20 words
The apparatus according to claim 44, wherein said detection means detects the linear image while increasing the predetermined value stepwise.
Claim 46
Dependent Claim
53 words
The apparatus according to claim 45, wherein said detection means stores pixels together with priority levels corresponding to a detection order of the pixels when the pixels constituting the linear image are detected, and said first generation means generates pixels constituting the interpolated image in the order from pixels with lower priority levels.
Claim 47
Dependent Claim
30 words
The apparatus according to claim 41, further comprising prediction means for predicting maximum and minimum values of an inclination of a straight line to be detected by said detection means.
Claim 48
Dependent Claim
100 words
An image processing apparatus for processing multi-viewpoint images, comprising: viewpoint detection means for detecting a position of a viewpoint; generation means for generating, on the basis of the multi-viewpoint images, an image which has, as a virtual viewpoint, an intersecting point between a line of sight from the viewpoint detected by said viewpoint detection means and a viewpoint array corresponding to the multi-viewpoint images; and correction means for correcting a distortion of an image caused by a deviation between the viewpoint detected by said viewpoint detection means and the virtual viewpoint on the basis of an amount of the deviation.
Claim 49
Dependent Claim
61 words
The apparatus according to claim 48, wherein said generation means comprises detection means for detecting correspondences of pixels among the multi-viewpoint images, interpolated image generation means for generating pixels constituting an interpolated image on the basis of the detected correspondences, and construction means for constructing the interpolated image on the basis of the pixels generated by said interpolated image generation means.
Claim 50
Dependent Claim
42 words
The apparatus according to claim 48, wherein said correction means corrects a distortion, in an up-and-down direction of an image on the basis of a distance from a viewpoint to an object for the multi-viewpoint image, and the amount of the deviation.
Claim 51
Dependent Claim
42 words
The apparatus according to claim 48, wherein said correction means corrects a distortion, in a right-and-left direction of an image on the basis of a distance from a viewpoint to an object for the multi-viewpoint image, and the amount of the deviation.
Claim 52
Dependent Claim
37 words
The apparatus according to claim 48, further comprising correction means for performing correction processing of images of the multi-viewpoint images, and wherein said detection means detects the correspondences from the multi-viewpoint images corrected by said correction means.
Claim 53
Dependent Claim
19 words
The apparatus according to claim 48, further comprising output means for outputting the image constructed by said construction means.
Claim 54
Dependent Claim
38 words
The apparatus according to claim 53, wherein said generation means generates stereoscopic images which have viewpoint positions deviated by a distance corresponding to a binocular parallax, and said output means displays the stereoscopic images on a stereoscopic display.
Claim 55
Dependent Claim
96 words
An image processing apparatus comprising: detection means for detecting correspondences of pixels among multi-viewpoint images; first generation means for generating pixels constituting an interpolated image on the basis of the detected correspondences; estimation means for estimating correspondences of non-processed pixels, from which correspondences cannot be detected by said detection means, among the multi-viewpoint images; second generation means for further generating pixels constituting the interpolated image on the basis of the correspondences estimated by said estimation means; and construction means for constructing the interpolated image on the basis of the pixels generated by said first generation means.
Claim 56
Dependent Claim
107 words
An image processing apparatus comprising: detection means for detecting correspondences of pixels among multi-viewpoint images; generation means for generating pixels constituting an interpolated image on the basis of the detected correspondences; and construction means for constructing the interpolated image on the basis of the pixels generated by said generation means, wherein said detection means ignores a correspondence between a pixel of interest and a corresponding pixel when the corresponding pixel which has the correspondence with the pixel of interest also has a correspondence with another pixel, and the number of correspondences for the pixel of interest is larger than the number of correspondences for the corresponding pixel.
Claim 57
Dependent Claim
16 words
The apparatus according to claim 29, further comprising an image database for storing the multi-viewpoint images.
Claim 58
Dependent Claim
17 words
The apparatus according to claim 29, further comprising a plurality of cameras for inputting the multi-viewpoint images.
Claim 59
Dependent Claim
16 words
The apparatus according to claim 48, further comprising an image database for storing the multi-viewpoint images.
Claim 60
Dependent Claim
17 words
The apparatus according to claim 48, further comprising a plurality of cameras for inputting the multi-viewpoint images.
Claim 61
Independent Claim
32 words
An apparatus for generating an interpolated image corresponding to a desired viewpoint from input multi-viewpoint images, comprising means for combining corresponding lines of the multi-viewpoint images to generate an epipolar plane image.
Claim 62
Dependent Claim
33 words
An apparatus according to claim 61 further comprising means for detecting straight lines from the epipolar plane image and generating interpolated pixels at intersecting points between interpolated lines and the detected straight lines.
Claim 63
Dependent Claim
33 words
An apparatus according to claim 61 or 62, further comprising means for assigning priority level to pixels of interest, in accordance with the number of straight lines detected for the pixel of interest.
Claim 64
Independent Claim
14 words
A method or apparatus having the features of any combination of the preceding claims.
References Cited (Backward Citations)
No citation records indexed.
USPTO Annuity & Maintenance Windows
Maintenance schedule is calculated for granted US utility patents.