Methods and apparatus for motion estimation in motion picture processing

Claim 3

The method of claim 1 wherein the step of obtaining a continuous signal representation further includes generating a continuous signal representation from a linear combination of predefined basis functions which are differentiable and linearly independent.

Claim 4

The method of claim 2 wherein said continuous signal representation is a set of three dimensional polynominals.

Claim 5

The method of claim 1 wherein the method further includes repeating the processing steps over other regions of the motion picture.

Claim 6

The method of claim 4 wherein the method further includes improving an initial velocity field estimate over a region by weighted averaging with velocity estimates obtained from neighboring regions.

Claim 7

The method of claim 5 wherein the weigths are derived from the eigenvalues computed during eigenvalue/eigenvector decomposition.

Claim 8

The method of claim 1 wherein the method further includes detecting an inaccurate velocity field estimate by computing the average squared directional gradient residual from said velocity field estimate and comparing said computed residual with a predefined threshold value.

Claim 9

The method of claim 1 wherein the method further includes repeating the processing steps over successive frames of said motion picture.

Claim 10

The method of claim 1 wherein the region is selected by down-sampling a portion of image data from a first and a second frame of said motion picture.

Claim 11

The method of claim 9 wherein the successively smaller down-sampling factors are used to refine the estimates of the velocity field.

Claim 12

The method of claim 1 wherein the method further includes employing the velocity vector obtained by processing said image to filter noise from said picture frames.

Claim 13

The method of claim 1 wherein the method further includes employing the velocity vector obtained by processing said image to synthesize intermediate picture frames.

Claim 14

The method of claim 1 wherein the method further includes employing the velocity vector obtained by processing said image to obtain a coded image for transmission.

Claim 15

Apparatus for estimating a velocity field within a motion picture defined by a plurality of picture frames, the apparatus comprising: sampling means for selecting a set of samples of image intensity values from at least a first and a second frame by mapping an initial velocity vector onto a region of interest in said first and second frames and selecting from said region in each frame a set of samples of said image values, signal estimating means connected to the sampling means for estimating from said samples a set of signal coefficients which define a continuous signal representation; and velocity estimating means connected to the signal estimating means for computing directional gradients from said continuous signal representation and minimizing an average value of a squared directional gradient residual by eigenvalue/eigenvector decomposition to obtain a velocity estimate over said region of interest.

Claim 16

Apparatus for estimating a velocity field within a motion picture defined by a plurality of picture frames, the apparatus comprising: sampling means for selecting a set of samples of image intensity values from at least a first and a second frame by mapping an initial velocity vector onto a region of interest in said first and second frames and selecting from said region in each frame a set of samples of said image values, signal estimating means connected to the sampling means for estimating from said samples a set of signal coefficients which define a continuous signal representation; and velocity estimating means connected to the signal estimating means for computing directional gradients from said continuous signal representation and minimizing an average value of a squared directional gradient residual by eigenvalue/eigenvector decomposition to obtain a velocity estimate over said region of interest.

Claim 17

The apparatus of claim 14 wherein the sampling means further includes a first and a second frame memory and means for down-sampling image values stored in said first and second memories.

Claim 18

The apparatus of claim 15 wherein the means for down-sampling the image values further includes means for varying the down-sampling rate.

Claim 19

The apparatus of claim 14 wherein the signal estimating means generates from said samples a continuous signal representation by linear combination of predefined basis functions which are differentiable and linearly independent.

Claim 20

The apparatus of claim 14 wherein the signal estimating means further includes a matrix multiplier in which the set of image samples are multiplied by a matrix of basis functions.

Claim 21

The apparatus of claim 18 wherein a set of signal coefficients, S i , are derived from a set of image samples, I, by matrix multiplication such that: ##EQU20##

Claim 22

The apparatus of claim 18 wherein said basis functions are three dimensional polynomials.

Claim 23

The apparatus of claim 20 wherein said basis functions are a 3×3 matrix, the entries of which are defined by the equations: ##EQU21##

Claim 24

The apparatus of claim 14 wherein the velocity estimating means further includes a quadratic evaluator which derives from said signal coefficients, a covariance matrix and a correlation vector, and a velocity computer which uses said covariance matrix and correlating vector to minimize the average value of a squared directional gradient residual and, thereby, estimate a velocity vector for the region of interest.

Claim 25

The apparatus of claim 14 wherein the apparatus further includes means for updating the initial velocity vector with said revised velocity estimate.

Claim 26

The apparatus of claim 14 wherein said apparatus further includes interpolation means for generating from said velocity estimate a synthetic frame between said first and second frames.

Claim 27

The apparatus of claim 14 wherein said apparatus further includes means for sampling a third frame and means for deriving another velocity estimate over the region of interest between said second and third frames; and means for enhancing the image values of said second frame by generating synthetic versions of said second frame from said velocity estimates.

Claim 28

The apparatus of claim 14 wherein the apparatus further includes a picture coding means comprising frame predicting means for generating a predicted next frame from said first and second frames, means for comparing said predicted next frame with an actual next frame; and means for transmitting a signal comprising differences between the predicted frame and the actual frame.

Claim 29

An image synthesizing system for generating between-frame, motion picture images, the system comprising: sampling means for selecting a set of samples of image intensity values from at least a first and a second frame by mapping an initial velocity vector onto a region of interest in each of said frames; signal estimating means for estimating from said samples a set of signal coefficients which define a continuous signal representation of changes in image values from said first frame to said second frame; velocity estimating means for computing directional gradients from said continuous signal representation and minimizing an average value of a squared directional gradient by eigenvalue/eigenvector decomposition to obtain a velocity field estimate over said region from the first frame to the second frame; and interpolating means for generating from said velocity field estimate a synthetic between-frame image.

Claim 30

An image synthesizing system for generating between-frame, motion picture images, the system comprising: sampling means for selecting a set of samples of image intensity values from at least a first and a second frame by mapping an initial velocity vector onto a region of interest in each of said frames; signal estimating means for estimating from said samples a set of signal coefficients which define a continuous signal representation of changes in image values from said first frame to said second frame; velocity estimating means for computing directional gradients from said continuous signal representation and minimizing an average value of a squared directional gradient by eigenvalue/eigenvector decomposition to obtain a velocity field estimate over said region from the first frame to the second frame; and interpolating means for generating from said velocity field estimate a synthetic between-frame image.

Claim 31

An image enhancing system for enhancing a frame of a motion picture, the image enhancing system comprising: sampling means for selecting a set of samples of image intensity values from a frame of interest and at least one preceding and one succeeding frame by mapping an initial velocity vector onto a region of interest in each of said frames; signal estimation means for estimating from said samples a first set of signal coefficients which define a continuous signal representation of changes in the image values from the preceding frame to the frame of interest, and a second set of signal coefficients which define a continuous signal representation of changes in the image values from the frame of interest to the succeeding frame; velocity estimating means for estimating a first velocity field over said region from the first frame to the second frame and for estimating a second velocity field over said region from the second frame to the third frame from said continuous signal representation by eigenvalue/eigenvector decomposition; interpolation means for generating from said velocity estimates a first and second synthetic version of the frame of interest; and processing means for processing the frame of interest and the synthetic versions thereof to obtain an enhanced set of image values for the frame.

Claim 32

An image enhancing system for enhancing a frame of a motion picture, the image enhancing system comprising: sampling means for selecting a set of samples of image intensity values from a frame of interest and at least one preceding and one succeeding frame by mapping an initial velocity vector onto a region of interest in each of said frames; signal estimation means for estimating from said samples a first set of signal coefficients which define a continuous signal representation of changes in the image values from the preceding frame to the frame of interest, and a second set of signal coefficients which define a continuous signal representation of changes in the image values from the frame of interest to the succeeding frame; velocity estimating means for estimating a first velocity field over said region from the first frame to the second frame and for estimating a second velocity field over said region from the second frame to the third frame from said continuous signal representation by eigenvalue/eigenvector decomposition; interpolation means for generating from said velocity estimates a first and second synthetic version of the frame of interest; and processing means for processing the frame of interest and the synthetic versions thereof to obtain an enhanced set of image values for the frame.

Claim 33

The apparatus of claim 28 wherein the processing means is an averaging means.

Claim 34

The apparatus of claim 28 wherein the processing means is a means for selecting the median value from the frame of interest and the synthetic versions thereof.

Claim 35

A picture coding system for transmitting image data, the picture coding system comprising: sampling means for selecting a set of samples of image intensity values from at least a first and a second frame by mapping an initial velocity vector onto a region of interest in each of said frames; signal estimating means for estimating from said samples a set of signal coefficients which define a continuous signal representation of changes in image values from said first frame to said second frame; velocity estimating means for computing directional gradients from said continuous signal representation and minimizing an average value of a squared directional gradient by eigenvalue/eigenvector decomposition to obtain a velocity field estimate over said region from the first frame to the second frame; frame prediction means for generating or predicted next frame from said velocity field estimate and the image values of said first and second frames; comparison means for comparing the predicted next frame with an actual next frame to detect differences in image value; and transmitter means for transmitting a signal comprising differences between the predicted frame and the actual frame.

Claim 36

A picture coding system for transmitting image data, the picture coding system comprising: sampling means for selecting a set of samples of image intensity values from at least a first and a second frame by mapping an initial velocity vector onto a region of interest in each of said frames; signal estimating means for estimating from said samples a set of signal coefficients which define a continuous signal representation of changes in image values from said first frame to said second frame; velocity estimating means for computing directional gradients from said continuous signal representation and minimizing an average value of a squared directional gradient by eigenvalue/eigenvector decomposition to obtain a velocity field estimate over said region from the first frame to the second frame; frame prediction means for generating or predicted next frame from said velocity field estimate and the image values of said first and second frames; comparison means for comparing the predicted next frame with an actual next frame to detect differences in image value; and transmitter means for transmitting a signal comprising differences between the predicted frame and the actual frame.