US5978030A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for encoding a video signal using feature point based motion estimation
Assignee: Quarterhill Inc
•
Filing: 1995-05-05
•
Granted: 1999-11-02
•
Expires: 2015-05-05
Abstract
Abstract An apparatus for encoding a digital video signal to reduce a transmission rate of the digital video signal, which comprises a feature point based motion compensation circuit for selecting a set of feature points from the reconstructed reference frame to detect a set of motion vectors between a current frame and a original reference frame corresponding to the set of feature points by using a feature point based motion estimation, and for generating a second predicted frame based on the set of motion vectors and the reconstructed reference frame. The feature point based motion estimation employs a convergence process in which a displacement of each of the feature points are given to a motion vector thereof and the six triangles of each of the hexagon are affine-transformed independently using the displacements of their vertex feature points. If the displacements provide a better PSNR, the motion vector of the subject feature point is sequentially updated. Therefore, the inventive convergence process is very efficient in the matching process to determine the predicted image as close as possible to the original image having zooming, rotation or scaling of objects.
Inventors
- 👤 Hae-Mook Jung
- 👤 Min-Sub Lee
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-04 09:58
CPC Classifications
H; H04; H04N; H04N19/00; H04N19/50; H04N19/503; H04N19/51; H04N19/537; H04N19/54; G; G06; G06T; G06T3/00; G06T3/02; H04N19/10; H04N19/134; H04N19/136; H04N19/137; H04N19/139; H04N19/169; H04N19/17; H04N19/172
Claims Analysis
Total Claims: 23
Claim 1
Independent Claim
439 words
A method for detecting a set of motion vectors between a current frame and a reference frame of video signals by employing a feature point based motion estimation approach, wherein the reference frame includes a reconstructed reference frame and an original reference frame, which comprises the steps of: (a) selecting a set of feature points from pixels contained in the reconstructed reference frame wherein the set of feature points forms a polygonal grid having a plurality of overlapping polygons; (b) determining a set of quasi-feature points on the current frame based on the set of feature points; (c) assigning a set of initial motion vectors for the set of quasi-feature points, wherein each of the initial motion vectors is set to (0,0); (d) appointing one of the quasi-feature points as a subject quasi-feature point, wherein the subject quasi-feature point has N number of neighboring quasi-feature points which form a subject current polygon defined by line segments connecting the subject quasi-feature point and said N number of neighboring quasi-feature points, N being a positive integer; (e) sequentially adding the initial motion vector of the subject quasi-feature point to M number of candidate motion vectors with to produce M number of updated initial motion vectors, M being a positive integer, wherein said M number of candidate motion vectors cover a predetermined region in the subject current polygon and the initial motion vectors of said neighboring feature points are fixed; (f) determining a predicted position on the original reference frame for each pixel contained in the subject current polygon based on each of the M number of updated initial motion vectors for the subject quasi-feature point and said N number of the initial motion vectors of the neighboring quasi-feature points; (g) providing a predicted pixel value for said each pixel based on the predicted position from the original reference frame to form M number of predictive subject current polygons; (h) calculating the difference between the current polygon and each of the predicted subject current polygons to produce M number of peak signal to noise ratios(PSNR's) (i) selecting one of the updated motion vectors as a selected updated motion vector, which entails a predicted subject current polygon having a maximum PSNR, to update the initial motion vector of the subject quasi-feature point with the selected updated motion vector; (j) repeating the steps (d) to (i) until all of the initial motion vectors are updated; (k) repeating the step (j) until said repeating is carried out for a predetermined number of times; and (n) establishing the set of initial vectors as the set of motion vectors, to thereby determine the set of motion vectors.
Claim 2
Independent Claim
439 words
A method for detecting a set of motion vectors between a current frame and a reference frame of video signals by employing a feature point based motion estimation approach, wherein the reference frame includes a reconstructed reference frame and an original reference frame, which comprises the steps of: (a) selecting a set of feature points from pixels contained in the reconstructed reference frame wherein the set of feature points forms a polygonal grid having a plurality of overlapping polygons; (b) determining a set of quasi-feature points on the current frame based on the set of feature points; (c) assigning a set of initial motion vectors for the set of quasi-feature points, wherein each of the initial motion vectors is set to (0,0); (d) appointing one of the quasi-feature points as a subject quasi-feature point, wherein the subject quasi-feature point has N number of neighboring quasi-feature points which form a subject current polygon defined by line segments connecting the subject quasi-feature point and said N number of neighboring quasi-feature points, N being a positive integer; (e) sequentially adding the initial motion vector of the subject quasi-feature point to M number of candidate motion vectors with to produce M number of updated initial motion vectors, M being a positive integer, wherein said M number of candidate motion vectors cover a predetermined region in the subject current polygon and the initial motion vectors of said neighboring feature points are fixed; (f) determining a predicted position on the original reference frame for each pixel contained in the subject current polygon based on each of the M number of updated initial motion vectors for the subject quasi-feature point and said N number of the initial motion vectors of the neighboring quasi-feature points; (g) providing a predicted pixel value for said each pixel based on the predicted position from the original reference frame to form M number of predictive subject current polygons; (h) calculating the difference between the current polygon and each of the predicted subject current polygons to produce M number of peak signal to noise ratios(PSNR's) (i) selecting one of the updated motion vectors as a selected updated motion vector, which entails a predicted subject current polygon having a maximum PSNR, to update the initial motion vector of the subject quasi-feature point with the selected updated motion vector; (j) repeating the steps (d) to (i) until all of the initial motion vectors are updated; (k) repeating the step (j) until said repeating is carried out for a predetermined number of times; and (n) establishing the set of initial vectors as the set of motion vectors, to thereby determine the set of motion vectors.
Claim 3
Dependent Claim
152 words
The method as recited in claim 1, wherein the step (a) includes the steps of: (a1) detecting an edge image of the reconstructed reference frame, wherein the edge image eg(x,y) is defined as follows ##EQU4## where p(x,y) represents the reference frame; |∇p(x,y) | denotes an output from a known Sobel operator; and Te is a predetermined threshold; (a2) establishing a polygonal grid on the edge image wherein the polygonal grid includes a number of grid points to form the plurality of overlapping polygons; (a3) assigning a non-overlapping search range for each of the grid points; and (a4) determining the set of feature points wherein the set of feature points includes a grid point overlapping an edge point, said edge point being located in the non-overlapping search range and having a maximum summation value of its surrounding pixel points, and said grid point having no edge point contained in its non-overlapping search range.
Claim 4
Dependent Claim
39 words
The method as recited in claim 1, wherein the set of feature points includes an edge point nearest to the polygonal grid when more than one edge point having the equal maximum summation value appear in the search range.
Claim 5
Dependent Claim
17 words
The method as recited in claim 3, wherein the polygon is a hexagon and N is 6.
Claim 6
Dependent Claim
43 words
The method as recited in claim 4, wherein the subject current hexagon includes six triangles defined by line segments connecting the subject quasi-feature point; and its neighboring quasi-feature points and the steps (f) and (g) are performed by using a known affine transform.
Claim 7
Dependent Claim
28 words
The method as recited in claim 5, wherein the number of surrounding pixel points is 8; the predetermined repeating number is 5; and the predetermined threshold is 6.
Claim 8
Dependent Claim
22 words
The method as recited in claim 6, wherein the predetermined region is in a range from -7 to 7, horizontally and vertically.
Claim 9
Dependent Claim
38 words
The method as recited in claim 7, wherein the feature point D i is defined as follows ##EQU5## wherein EG(x,y) is a value of the edge point contained in the search region and i is a positive integer.
Claim 10
Independent Claim
504 words
An apparatus, for use in a video encoding system, for detecting a set of motion vectors between a current frame and a reference frame of video signals by employing a feature point based motion estimation, wherein the reference frame includes a reconstructed reference frame and an original reference frame, which comprises: first selection means for selecting a set of pixels from the reconstructed reference frame as a set of feature points, wherein the set of feature points forms a polygonal grid having a plurality of overlapping polygons; means for determining a set of quasi-feature points on the current frame corresponding to the set of feature points; memory means for storing a set of initial motion vectors for the set of quasi-feature points, wherein each of the initial motion vectors is set to (0,0); second selection means for selecting L number of subject quasi-feature points from the set of quasi-feature points, wherein each of the subject quasi-feature points has N number of neighboring quasi-feature points which form a non-overlapping subject current polygon defined by line segments connecting the subject quasi-feature point and said N number of neighboring quasi-feature points, said L and N being positive integers; adder means for adding the initial motion vector of each of the subject quasi-feature points to M number of candidate motion vectors to generate M number of updated initial motion vectors for each of the subject quasi-feature points, M being a positive integer, wherein said M number of candidate motion vectors cover a predetermined region in each of the non-overlapping subject current polygons and the initial motion vectors of the neighboring feature points for each of the subject quasi-feature points are fixed; means for determining a predicted position on the original reference frame for each pixel contained in each of the non-overlapping subject current polygons based on each of the updated initial motion vectors and the initial motion vectors of the corresponding neighboring quasi-feature points; means for obtaining a predicted pixel value from the original reference frame based on the predicted position to thereby form M number of predicted subject current polygons for each of the non-overlapping subject current polygons; means for calculating the differences between each of the non-overlapping subject current polygons and the corresponding M number of predicted subject current polygons to produce M number of peak signal to noise ratios(PSNR's) for each of the non-overlapping subject current polygons; third selection means for selecting one of the updated initial vectors, for each of the subject quasi-feature points, as a selected updated initial motion vector which entails the predicted subject current polygon having a maximum PSNR to produce L number of selected updated initial motion vectors; means or updating the initial motion vector for each of the subject quasi-feature points stored in the memory means with the corresponding selected updated initial motion vector; and means for retrieving the set of initial motion vectors from the memory means as the set of motion vectors when all of the initial motion vectors are updated by a predetermined number of times.
Claim 11
Independent Claim
504 words
An apparatus, for use in a video encoding system, for detecting a set of motion vectors between a current frame and a reference frame of video signals by employing a feature point based motion estimation, wherein the reference frame includes a reconstructed reference frame and an original reference frame, which comprises: first selection means for selecting a set of pixels from the reconstructed reference frame as a set of feature points, wherein the set of feature points forms a polygonal grid having a plurality of overlapping polygons; means for determining a set of quasi-feature points on the current frame corresponding to the set of feature points; memory means for storing a set of initial motion vectors for the set of quasi-feature points, wherein each of the initial motion vectors is set to (0,0); second selection means for selecting L number of subject quasi-feature points from the set of quasi-feature points, wherein each of the subject quasi-feature points has N number of neighboring quasi-feature points which form a non-overlapping subject current polygon defined by line segments connecting the subject quasi-feature point and said N number of neighboring quasi-feature points, said L and N being positive integers; adder means for adding the initial motion vector of each of the subject quasi-feature points to M number of candidate motion vectors to generate M number of updated initial motion vectors for each of the subject quasi-feature points, M being a positive integer, wherein said M number of candidate motion vectors cover a predetermined region in each of the non-overlapping subject current polygons and the initial motion vectors of the neighboring feature points for each of the subject quasi-feature points are fixed; means for determining a predicted position on the original reference frame for each pixel contained in each of the non-overlapping subject current polygons based on each of the updated initial motion vectors and the initial motion vectors of the corresponding neighboring quasi-feature points; means for obtaining a predicted pixel value from the original reference frame based on the predicted position to thereby form M number of predicted subject current polygons for each of the non-overlapping subject current polygons; means for calculating the differences between each of the non-overlapping subject current polygons and the corresponding M number of predicted subject current polygons to produce M number of peak signal to noise ratios(PSNR's) for each of the non-overlapping subject current polygons; third selection means for selecting one of the updated initial vectors, for each of the subject quasi-feature points, as a selected updated initial motion vector which entails the predicted subject current polygon having a maximum PSNR to produce L number of selected updated initial motion vectors; means or updating the initial motion vector for each of the subject quasi-feature points stored in the memory means with the corresponding selected updated initial motion vector; and means for retrieving the set of initial motion vectors from the memory means as the set of motion vectors when all of the initial motion vectors are updated by a predetermined number of times.
Claim 12
Dependent Claim
151 words
The apparatus as recited in claim 9, wherein the first selection means includes: means for detecting an edge image of the reconstructed reference frame, wherein the edge image eg(x,y) is defined as follows ##EQU6## where p(x,y) represents the reference frame; |∇p(x,y) | denotes an output from a known Sobel operator; and Te is a predetermined threshold; means for providing a polygonal grid on the edge image wherein the polygonal grid includes a number of grid points to form the plurality of overlapping polygons; means for establishing a non-overlapping search range for each of the grid points; and means for determining the set of feature points wherein the set of feature points includes a grid point overlapping an edge point, said edge point being located in the search rangeand having a maximum summation value of its surrounding pixel points, said grid point having no edge point contained in its non-overlapping searching range.
Claim 13
Dependent Claim
39 words
The method as recited in claim 10, wherein the set of feature points includes an edge point nearest to the polygonal grid when more than one edge point having the equal maximum summation value appear in the search range.
Claim 14
Dependent Claim
17 words
The apparatus as recited in claim 11, wherein the polygon is a hexagon and N is 6.
Claim 15
Dependent Claim
41 words
The apparatus as recited in claim 12, wherein the subject current hexagon includes six triangles defined by line segments connecting the subject quasi-feature point and its neighboring quasi-feature points; and means for determining the predicted position includes a known affine transformer.
Claim 16
Dependent Claim
28 words
The apparatus as recited in claim 13, wherein the number of surrounding pixel points is 8; the predetermined repeating number is 5; and the predetermined threshold is 6.
Claim 17
Dependent Claim
22 words
The apparatus as recited in claim 14, wherein the predetermined region is in a range from -7 to 7, horizontally and vertically.
Claim 18
Independent Claim
652 words
An apparatus for encoding a digital video signal to reduce the transmission rate of the digital video signal, said digital video signal having a plurality of frames including a current frame and a reference frame, which comprises: first memory means for storing a reconstructed reference frame of the digital video signal; second memory means for storing an original reference frame of the digital video signal; first motion compensation means for detecting a number of motion vectors between the current frame and the reconstructed reference frame by using a block based motion estimation and for generating a first predicted current frame based on the number of motion vectors and the reconstructed reference frame; first selection means for selecting a set of pixels from the reconstructed reference frame as a set of feature points, wherein the set of feature points forms a polygonal grid having a plurality of overlapping polygons; means for determining a set of quasi-feature points on the current frame corresponding to the set of feature points; memory means for storing a set of initial motion vectors for the set of quasi-feature points, wherein each of the initial motion vectors is set to (0,0); second selection means for selecting L number of subject quasi-feature points from the set of quasi-feature points, wherein each of the subject quasi-feature points has N number of neighboring quasi-feature points which form a non-overlapping subject current polygon defined by line segments connecting the subject quasi-feature point and said N number of neighboring quasi-feature points, said L and N being positive integers; adder means for adding the initial motion vector of each of the subject quasi-feature points to M number of candidate motion vectors to generate M number of updated initial motion vectors for each of the subject quasi-feature points, M being a positive integer, wherein said M number of candidate motion vectors cover a predetermined region in each of the non-overlapping subject current polygons and the initial motion vectors of the neighboring feature points for each of the subject quasi-feature points are fixed; means for determining a predicted position on the original reference frame for each pixel contained in each of the non-overlapping subject current polygons based on each of the updated initial motion vectors and the initial motion vectors of the corresponding neighboring quasi-feature points; means for obtaining a predicted pixel value from the original reference frame based on the predicted position to thereby form M number of predicted subject current polygons for each of the non-overlapping subject current polygons; means for calculating the differences between each of the non-overlapping subject current polygons and the corresponding M number of predicted subject current polygons to produce M number of peak signal to noise ratios(PSNR's) for each of the non-overlapping subject current polygons; third selection means for selecting one of the updated initial vectors, for each of the subject quasi-feature points, as a selected updated initial motion vector which entails the predicted subject current polygon having a maximum PSNR to produce L number of selected updated initial motion vectors; means for updating the initial motion vector for each of the subject quasi-feature points stored in the memory means with the corresponding selected updated initial motion vector; means for retrieving the set of initial motion vectors from the memory means as the set of motion vectors when all of the initial motion vectors are updated by a predetermined number of times; and means for selectively providing the number of motion vectors and the first predicted current frame or the set of motion vectors and the second predicted current frame as selected motion vectors and the predicted current frame; means for transform-coding an error signal representing the difference between the selectively provided predicted current frame and the current frame to produce a transform coded error signal; and means for statistically coding the transform coded error signal and the selected motion vectors to produce an encoded video signal to be transmitted.
Claim 19
Independent Claim
652 words
An apparatus for encoding a digital video signal to reduce the transmission rate of the digital video signal, said digital video signal having a plurality of frames including a current frame and a reference frame, which comprises: first memory means for storing a reconstructed reference frame of the digital video signal; second memory means for storing an original reference frame of the digital video signal; first motion compensation means for detecting a number of motion vectors between the current frame and the reconstructed reference frame by using a block based motion estimation and for generating a first predicted current frame based on the number of motion vectors and the reconstructed reference frame; first selection means for selecting a set of pixels from the reconstructed reference frame as a set of feature points, wherein the set of feature points forms a polygonal grid having a plurality of overlapping polygons; means for determining a set of quasi-feature points on the current frame corresponding to the set of feature points; memory means for storing a set of initial motion vectors for the set of quasi-feature points, wherein each of the initial motion vectors is set to (0,0); second selection means for selecting L number of subject quasi-feature points from the set of quasi-feature points, wherein each of the subject quasi-feature points has N number of neighboring quasi-feature points which form a non-overlapping subject current polygon defined by line segments connecting the subject quasi-feature point and said N number of neighboring quasi-feature points, said L and N being positive integers; adder means for adding the initial motion vector of each of the subject quasi-feature points to M number of candidate motion vectors to generate M number of updated initial motion vectors for each of the subject quasi-feature points, M being a positive integer, wherein said M number of candidate motion vectors cover a predetermined region in each of the non-overlapping subject current polygons and the initial motion vectors of the neighboring feature points for each of the subject quasi-feature points are fixed; means for determining a predicted position on the original reference frame for each pixel contained in each of the non-overlapping subject current polygons based on each of the updated initial motion vectors and the initial motion vectors of the corresponding neighboring quasi-feature points; means for obtaining a predicted pixel value from the original reference frame based on the predicted position to thereby form M number of predicted subject current polygons for each of the non-overlapping subject current polygons; means for calculating the differences between each of the non-overlapping subject current polygons and the corresponding M number of predicted subject current polygons to produce M number of peak signal to noise ratios(PSNR's) for each of the non-overlapping subject current polygons; third selection means for selecting one of the updated initial vectors, for each of the subject quasi-feature points, as a selected updated initial motion vector which entails the predicted subject current polygon having a maximum PSNR to produce L number of selected updated initial motion vectors; means for updating the initial motion vector for each of the subject quasi-feature points stored in the memory means with the corresponding selected updated initial motion vector; means for retrieving the set of initial motion vectors from the memory means as the set of motion vectors when all of the initial motion vectors are updated by a predetermined number of times; and means for selectively providing the number of motion vectors and the first predicted current frame or the set of motion vectors and the second predicted current frame as selected motion vectors and the predicted current frame; means for transform-coding an error signal representing the difference between the selectively provided predicted current frame and the current frame to produce a transform coded error signal; and means for statistically coding the transform coded error signal and the selected motion vectors to produce an encoded video signal to be transmitted.
Claim 20
Dependent Claim
153 words
The apparatus as recited in claim 16, wherein the first selection means includes: means for detecting an edge image of the reconstructed reference frame, wherein the edge image eg(x,y) is defined as follows ##EQU7## where p(x,y) represents the reference frame; |∇p(x,y) | denotes an output from a known Sobel operator; and Te is a predetermined threshold; means for providing a polygonal grid on the edge image wherein the polygonal grid includes a number of grid points to form the plurality of overlapping polygons; means for setting a non-overlapping search range for each of the grid points; and means for determining the set of feature points wherein the set of feature points includes a grid point overlapping an edge point, said edge point being located in the search range and having a maximum summation value of its surrounding pixel points, and said grid point having no edge point contained in its non-overlapping search range.
Claim 21
Dependent Claim
49 words
The apparatus as recited in claim 17, wherein the polygon is a hexagon, N is 6, the subject current hexagon includes six triangles defined by line segments connecting the subject quasi-feature point and its neighboring quasi-feature points; and means for determining the predicted position includes a known affine transformer.
Claim 22
Dependent Claim
28 words
The apparatus as recited in claim 18, wherein the number of surrounding pixel points is 8, the predetermined repeating number is 5; and the predetermined threshold is 6.
Claim 23
Dependent Claim
22 words
The apparatus as recited in claim 19, wherein the predetermined region is in a range from -7 to 7, horizontally and vertically.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2002-11-01 | 2003-05-03 | 2003-11-02 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2006-11-01 | 2007-05-03 | 2007-11-02 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2010-11-01 | 2011-05-03 | 2011-11-02 | $7700.00 | Window Closed / Lapsed |