EP0701369A1
Expired / Lapsed
EP
⏱️ 0.0 years remaining
Image processing method and apparatus
Assignee: Canon Inc
•
Filing: 1995-09-08
•
Granted: —
•
Expires: 2015-09-08
Abstract
Abstract An image sensing apparatus according to an embodiment of the present invention includes an image input unit (100) for inputting an image, a detection unit (200) for detecting a state of movement of the image input unit (100) in an image input operation, a storage unit (310, 9000) for storing a plurality of images input by the image input unit (100) and movement information corresponding to the state of movement detected by the detection unit (200), and an image generating unit (320) for generating an image from an arbitrary viewpoint position on the basis of the plurality of images and the movement information stored in the storage unit.
Inventors
- 👤 Katsumi Iijima
- 👤 Jun Tokumitsu
- 👤 Masakazu Matsugu
- 👤 Kotaro Yano
- 👤 Sunao Kurahashi
- 👤 Toshiaki Kondo
- 👤 Katsuhiko Mori
- 👤 Motohiro Ishikawa
Bibliographic Details
Application Number
—
Jurisdiction
EP
Data Source
google_patents
Last Synchronized
2026-11-09 19:57
CPC Classifications
G; G06; G06T; G06T7/00; G06T7/50; G06T7/55; G06T7/579; G06T7/20; H; H04; H04N; H04N13/00; H04N13/10; H04N13/106; H04N13/111; H04N13/117; H04N13/20; H04N13/204; H04N13/239; H04N13/282; H04N5/00; H04N5/222; H04N5/262; H04N5/2628; H04N13/189; H04N13/207; H04N13/221; H04N13/243; H04N13/30; H04N2013/0074; H04N2013/0081; H04N2013/0088; H04N5/14; H04N5/144; H04N5/145
Claims Analysis
Total Claims: 61
Claim 1
Independent Claim
94 words
An image sensing apparatus characterized by comprising: image input means (100) for inputting an image; detection means (200) for detecting a state of movement of said image input means in an image input operation; storage means (310) for storing a plurality of images input by said image input means and movement information corresponding to the state of movement detected by said detection means; and image generating means (320) for generating an image from an arbitrary viewpoint position on the basis of the plurality of images and the movement information stored in said storage means.
Claim 2
Dependent Claim
33 words
The apparatus according to claim 1, wherein said image input means outputs a plurality of images obtained by sensing the image from relatively different positions, and stores the images in said storage means.
Claim 3
Dependent Claim
24 words
The apparatus according to claim 1, wherein said image input means includes a multi-eye image sensing system having a plurality of image sensing systems.
Claim 4
Dependent Claim
17 words
The apparatus according to claim 1, wherein said image input means includes a single-eye image sensing system.
Claim 5
Dependent Claim
32 words
The apparatus according to claim 1, wherein said image generating means generates an image from the arbitrary viewpoint position by interpolating the plurality of images on the basis of the movement information.
Claim 6
Dependent Claim
20 words
The apparatus according to claim 5, further comprising viewpoint position input means for inputting coordinates of the arbitrary viewpoint position.
Claim 7
Dependent Claim
27 words
The apparatus according to claim 6, wherein said viewpoint position input means inputs a permutation of coordinates of arbitrary viewpoint positions in correspondence with a reproduction order.
Claim 8
Dependent Claim
55 words
The apparatus according to claim 6, wherein said viewpoint position input means further comprises display means for displaying a relationship between a range of images sensed by said image input means and time points of image sensing, and inputs coordinates of a viewpoint position on the basis of the relationship displayed on said display means.
Claim 9
Dependent Claim
55 words
The apparatus according to claim 7, wherein said viewpoint position input means further comprises display means for displaying a relationship between a range of images sensed by said image input means and time points of image sensing, and inputs coordinates of a viewpoint position on the basis of the relationship displayed on said display means.
Claim 10
Dependent Claim
29 words
The apparatus according to any one of claim 6, wherein said viewpoint position input means also inputs a viewpoint direction at the coordinates of the input arbitrary viewpoint position.
Claim 11
Dependent Claim
29 words
The apparatus according to any one of claim 7, wherein said viewpoint position input means also inputs a viewpoint direction at the coordinates of the input arbitrary viewpoint position.
Claim 12
Dependent Claim
121 words
The apparatus according to any one of claim 6, wherein said image generating means comprises: selection means for selecting a plurality of images sensed at viewpoint positions near a viewpoint position input by said viewpoint position input means; corresponding point detection means for dividing the plurality of selected images into predetermined blocks, and detecting corresponding points on the basis of block matching; calculation means for calculating coordinates including a distance to an object to be sensed on the basis of the detected corresponding points; and image generating means for generating an interpolated image corresponding to a viewpoint position input by said viewpoint position input means on the basis of the corresponding points, the coordinates of the object, and the movement information.
Claim 13
Dependent Claim
121 words
The apparatus according to any one of claim 7, wherein said image generating means comprises: selection means for selecting a plurality of images sensed at viewpoint positions near a viewpoint position input by said viewpoint position input means; corresponding point detection means for dividing the plurality of selected images into predetermined blocks, and detecting corresponding points on the basis of block matching; calculation means for calculating coordinates including a distance to an object to be sensed on the basis of the detected corresponding points; and image generating means for generating an interpolated image corresponding to a viewpoint position input by said viewpoint position input means on the basis of the corresponding points, the coordinates of the object, and the movement information.
Claim 14
Dependent Claim
97 words
The apparatus according to claim 1, wherein said corresponding point detection means comprises: first corresponding point detection means for detecting, by block matching, corresponding points between a plurality of images which are sensed from different viewpoints at the same time; and second corresponding point determination means for performing second corresponding point detection between corresponding points detected by said first detection means at first and second times upon determining that an evaluation function including a predetermined feature amount of the corresponding points at the first and second times and the movement information is smaller than a predetermined threshold.
Claim 15
Dependent Claim
13 words
The apparatus according to claim 14, wherein the feature amount is a density.
Claim 16
Dependent Claim
14 words
The apparatus according to claim 14, wherein the feature amount includes basic color components.
Claim 17
Dependent Claim
50 words
The apparatus according to claim 14, wherein the evaluation function is a function of a distance between a feature amount of corresponding points at the first time and a feature amount of positions obtained by correcting positions of corresponding points at the second time in accordance with the movement information.
Claim 18
Dependent Claim
23 words
The apparatus according to claim 14, wherein the threshold is set to be larger with an increase in amount of the movement information.
Claim 19
Dependent Claim
24 words
The apparatus according to claim 14, wherein the threshold is set to be larger as the object is located closer to said input means.
Claim 20
Dependent Claim
43 words
The apparatus according to claim 1, wherein said image generating means further comprises rotational transformation means for performing rotational transformation of an image generated by said image generating means on the basis of a viewpoint direction input by said viewpoint position input means.
Claim 21
Dependent Claim
68 words
The apparatus according to claim 20, wherein said image generating means further comprises: back-and-forth direction moving amount input means for inputting a moving amount of a viewpoint in a back-and-forth direction; and back-and-forth movement transformation means for performing back-and-forth movement transformation of an image having undergone rotational transformation in said rotational transformation means on the basis of the input moving amount of the viewpoint in the back-and-forth direction.
Claim 22
Dependent Claim
65 words
The apparatus according to claim 12, wherein said image generating means further comprises: back-and-forth direction moving amount input means for inputting a moving amount of a viewpoint in a back-and-forth direction; and back-and-forth movement transformation means for performing back-and-forth movement transformation of an image generated by said image generating means on the basis of the input moving amount of the viewpoint in the back-and-forth direction.
Claim 23
Dependent Claim
65 words
The apparatus according to claim 13, wherein said image generating means further comprises: back-and-forth direction moving amount input means for inputting a moving amount of a viewpoint in a back-and-forth direction; and back-and-forth movement transformation means for performing back-and-forth movement transformation of an image generated by said image generating means on the basis of the input moving amount of the viewpoint in the back-and-forth direction.
Claim 24
Dependent Claim
214 words
The apparatus according to claim 3, wherein said corresponding point detection means comprises: stereo image corresponding point detection means for detecting, by block matching, corresponding points between stereo images which are sensed from different viewpoints at the same time; left image corresponding point detection means for detecting corresponding points on a left image of the stereo images detected by said stereo image corresponding point detection means at a first time, and corresponding points on a left image of the stereo image at a second time upon determining that a value of an evaluation information including a predetermined feature amount of corresponding points on the left image of the stereo images at the first and second times and the movement information is smaller than a predetermined threshold; and right image corresponding point detection means for detecting corresponding points on a right image of the stereo images detected by said stereo image corresponding point detection means at the first time, and corresponding points on a right image of the stereo image at the second time upon determining that a value of an evaluation information including a predetermined feature amount of corresponding points on the right image of the stereo images at the first and second times and the movement information is smaller than a predetermined threshold.
Claim 25
Dependent Claim
187 words
The apparatus according to claim 3, wherein said calculation means calculates a first coordinate value including a distance to the object on the basis of corresponding points between the stereo images detected by said corresponding point detection means at the first time, calculates a second coordinate value including a distance to the object on the basis of corresponding points between the stereo images detected by said corresponding point detection means at the second time, calculates a third coordinate value including a distance to the object on the basis of corresponding points on the left image detected by said corresponding point detection means, calculates a fourth coordinate value including a distance to the object on the basis of corresponding points on the right image detected by said corresponding point detection means, and sets the first to fourth coordinate values as coordinate values of the object if the movement information is larger than a baseline length of said image input means, and the first and second coordinate values as coordinate values of the object if the movement information is smaller than the baseline length of said image input means.
Claim 26
Dependent Claim
25 words
The apparatus according to claim 1, wherein the movement information is a movement vector in an image input operation performed by said image input means.
Claim 27
Dependent Claim
73 words
The apparatus according to claim 1, wherein said apparatus comprises reproduction means for displaying an image generated by said image generating means on a monitor, and said image generating means interpolates an image corresponding to a viewpoint position and a viewpoint direction of a user within a screen of the monitor on the basis of pieces of image information stored in said storage means, and outputs the interpolated image to said reproduction means.
Claim 28
Dependent Claim
25 words
The apparatus according to claim 1, further comprising reproduction order setting means for setting an order in which images are reproduced by said reproduction means.
Claim 29
Dependent Claim
52 words
The apparatus according to claim 26, wherein said detection means comprises: corresponding point input means for inputting corresponding points on a plurality of images input by said input means; and representative movement vector generating means for generating a representative movement vector from each movement vector based on each of the corresponding points.
Claim 30
Dependent Claim
54 words
The apparatus according to claim 26, wherein said detection means comprises: corresponding point detection means for detecting corresponding points on a plurality of images from said input means by block matching; and representative movement vector generating means for generating a representative movement vector from each movement vector based on each of the corresponding points.
Claim 31
Dependent Claim
95 words
An image sensing method characterized by comprising: image input step for inputting an image using image input means ; detection step for detecting a state of movement of said image input means in an image input operation; storage step for storing a plurality of images input at said image input step and movement information corresponding to the state of movement detected by said detection step; and image generating step for generating an image from an arbitrary viewpoint position on the basis of the plurality of images and the movement information stored in said storage step.
Claim 32
Dependent Claim
29 words
The method according to claim 31, wherein said image input step includes a step for outputting a plurality of images obtained by sensing the image from relatively different positions.
Claim 33
Dependent Claim
26 words
The method according to claim 31, wherein said image input step makes use of a multi-eye image sensing system having a plurality of image sensing systems.
Claim 34
Dependent Claim
24 words
The method according to claim 31, wherein said image input step make use of a single-eye image sensing system having one image sensing system.
Claim 35
Dependent Claim
34 words
The method according to claim 31, wherein at said image generating step an image from the arbitrary viewpoint position by interpolating the plurality of images on the basis of the movement information is generated.
Claim 36
Dependent Claim
20 words
The method according to claim 35, further comprising viewpoint position input step for inputting coordinates of the arbitrary viewpoint position.
Claim 37
Dependent Claim
29 words
The method according to claim 35, wherein at said viewpoint position input step a permutation of coordinates of arbitrary viewpoint positions in correspondence with a reproduction order is input.
Claim 38
Dependent Claim
55 words
The method according to claim 36, wherein said viewpoint position input step further comprises display step for displaying a relationship between a range of images sensed by said image input step and time points of image sensing, and inputting coordinates of a viewpoint position on the basis of the relationship displayed on said display step.
Claim 39
Dependent Claim
55 words
The method according to claim 37, wherein said viewpoint position input step further comprises display step for displaying a relationship between a range of images sensed by said image input step and time points of image sensing, and inputting coordinates of a viewpoint position on the basis of the relationship displayed on said display step.
Claim 40
Independent Claim
31 words
The method according to any one of claims 36, wherein at said viewpoint position input step a viewpoint direction at the coordinates of the input arbitrary viewpoint position is also input.
Claim 41
Independent Claim
31 words
The method according to any one of claims 37, wherein at said viewpoint position input step a viewpoint direction at the coordinates of the input arbitrary viewpoint position is also input.
Claim 42
Independent Claim
121 words
The method according to any one of claims 36, wherein said image generating step comprises: selection step for selecting a plurality of images sensed at viewpoint positions near a viewpoint position input by said viewpoint position input step; corresponding point detection step for dividing the plurality of selected images into predetermined blocks, and detecting corresponding points on the basis of block matching; calculation step for calculating coordinates including a distance to an object to be sensed on the basis of the detected corresponding points; and image generating step for generating an interpolated image corresponding to a viewpoint position input by said viewpoint position input step on the basis of the corresponding points, the coordinates of the object, and the movement information.
Claim 43
Independent Claim
121 words
The method according to any one of claims 37, wherein said image generating step comprises: selection step for selecting a plurality of images sensed at viewpoint positions near a viewpoint position input by said viewpoint position input step; corresponding point detection step for dividing the plurality of selected images into predetermined blocks, and detecting corresponding points on the basis of block matching; calculation step for calculating coordinates including a distance to an object to be sensed on the basis of the detected corresponding points; and image generating step for generating an interpolated image corresponding to a viewpoint position input by said viewpoint position input step on the basis of the corresponding points, the coordinates of the object, and the movement information.
Claim 44
Dependent Claim
97 words
The method according to claim 31, wherein said corresponding point detection step comprises: first corresponding point detection step for detecting, by block matching, corresponding points between a plurality of images which are sensed from different viewpoints at the same time; and second corresponding point determination step for performing second corresponding point detection between corresponding points detected by said first detection step at first and second times upon determining that an evaluation function including a predetermined feature amount of the corresponding points at the first and second times and the movement information is smaller than a predetermined threshold.
Claim 45
Dependent Claim
13 words
The method according to claim 44, wherein the feature amount is a density.
Claim 46
Dependent Claim
14 words
The method according to claim 44, wherein the feature amount includes basic color components.
Claim 47
Dependent Claim
50 words
The method according to claim 44, wherein the evaluation function is a function of a distance between a feature amount of corresponding points at the first time and a feature amount of positions obtained by correcting positions of corresponding points at the second time in accordance with the movement information.
Claim 48
Dependent Claim
23 words
The method according to claim 44, wherein the threshold is set to be larger with an increase in amount of the movement information.
Claim 49
Dependent Claim
24 words
The method according to claim 44, wherein the threshold is set to be larger as the object is located closer to said input step.
Claim 50
Dependent Claim
43 words
The method according to claim 31, wherein said image generating step further comprises rotational transformation step for performing rotational transformation of an image generated by said image generating step on the basis of a viewpoint direction input by said viewpoint position input step.
Claim 51
Dependent Claim
68 words
The method according to claim 50, wherein said image generating step further comprises: back-and-forth direction moving amount input step for inputting a moving amount of a viewpoint in a back-and-forth direction; and back-and-forth movement transformation step for performing back-and-forth movement transformation of an image having undergone rotational transformation in said rotational transformation step on the basis of the input moving amount of the viewpoint in the back-and-forth direction.
Claim 52
Dependent Claim
64 words
The method according to claim 42, wherein said image generating step further comprises: back-and-forth direction moving amount input of inputting a moving amount of a viewpoint in a back-and-forth direction; and back-and-forth movement transformation step for performing back-and-forth movement transformation of an image generated by said image generating step on the basis of the input moving amount of the viewpoint in the back-and-forth direction.
Claim 53
Dependent Claim
64 words
The method according to claim 43, wherein said image generating step further comprises: back-and-forth direction moving amount input of inputting a moving amount of a viewpoint in a back-and-forth direction; and back-and-forth movement transformation step for performing back-and-forth movement transformation of an image generated by said image generating step on the basis of the input moving amount of the viewpoint in the back-and-forth direction.
Claim 54
Dependent Claim
214 words
The method according to claim 33, wherein said corresponding point detection step comprises: stereo image corresponding point detection step for detecting, by block matching, corresponding points between stereo images which are sensed from different viewpoints at the same time; left image corresponding point detection step for detecting corresponding points on a left image of the stereo images detected by said stereo image corresponding point detection step at a first time, and corresponding points on a left image of the stereo image at a second time upon determining that a value of an evaluation information including a predetermined feature amount of corresponding points on the left image of the stereo images at the first and second times and the movement information is smaller than a predetermined threshold; and right image corresponding point detection step for detecting corresponding points on a right image of the stereo images detected by said stereo image corresponding point detection step at the first time, and corresponding points on a right image of the stereo image at the second time upon determining that a value of an evaluation information including a predetermined feature amount of corresponding points on the right image of the stereo images at the first and second times and the movement information is smaller than a predetermined threshold.
Claim 55
Dependent Claim
193 words
The method according to claim 33, wherein at said calculation step a first coordinate value including a distance to the object on the basis of corresponding points between the stereo images detected by said corresponding point detection step at the first time is calculated; a second coordinate value including a distance to the object on the basis of corresponding points between the stereo images detected by said corresponding point detection step at the second time is calculated; a third coordinate value including a distance to the object on the basis of corresponding points on the left image detected by said corresponding point detection step is calculated, a fourth coordinate value including a distance to the object on the basis of corresponding points on the right image detected by said corresponding point detection step is calculated; and the first to fourth coordinate values as coordinate values of the object is set if the movement information is larger than a baseline length of said image input step, and the first and second coordinate values as coordinate values of the object if the movement information is smaller than the baseline length of said image input step.
Claim 56
Dependent Claim
25 words
The method according to claim 31, wherein the movement information is a movement vector in an image input operation performed by said image input step.
Claim 57
Dependent Claim
76 words
The method according to claim 31, wherein said method comprises reproduction step for displaying an image generated by said image generating step on a monitor, and at said image generating step an image corresponding to a viewpoint position and a viewpoint direction of a user within a screen of the monitor on the basis of pieces of image information stored in said storage step is interpolated, and the interpolated image is outputted to said reproduction step.
Claim 58
Dependent Claim
25 words
The method according to claim 31, further comprising reproduction order setting step for setting an order in which images are reproduced by said reproduction step.
Claim 59
Dependent Claim
52 words
The method according to claim 56, wherein said detection step comprises: corresponding point input step for inputting corresponding points on a plurality of images input by said input step; and representative movement vector generating step for generating a representative movement vector from each movement vector based on each of the corresponding points.
Claim 60
Dependent Claim
54 words
The method according to claim 56, wherein said detection step comprises: corresponding point detection step for detecting corresponding points on a plurality of images from said input step by block matching; and representative movement vector generating step for generating a representative movement vector from each movement vector based on each of the corresponding points.
Claim 61
Dependent Claim
19 words
A camera in which three-dimensional images can be obtained by combining movement data with a plurality of different images.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
Maintenance schedule is calculated for granted US utility patents.