US7760956B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
System and method for producing a page using frames of a video stream
Assignee: Hewlett Packard Development Co LP
•
Filing: 2005-05-12
•
Granted: 2010-07-20
•
Expires: 2025-05-12
Abstract
Abstract A method and system for automatically producing a page using frames of a video stream. A plurality of video frames from a video stream or clip are read, and multiple frames are extracted from the video stream, based on the content of each frame. Frames which neighbor the extracted frames in the video stream are also extracted, with the number of neighboring frames to be extracted being controlled by parameters provided to the page production system. The resolution of the images in the extracted video frames is enhanced, based on the information in each extracted frame and each respective extracted neighboring frame. The enhanced images are automatically cropped to enhance the important features in each image, and the cropped images are composed into one or more pages that are representative of the content of the video stream.
Inventors
- 👤 Xiaofan Lin
- 👤 Tong Zhang
- 👤 C. Brian Atkins
- 👤 Gary L. Vondran, Jr.
- 👤 Mei Chen
- 👤 Charles A. Untulis
- 👤 Stephen Philip Cheatle
- 👤 Dominic Lee
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 22:52
CPC Classifications
G; G06; G06V; G06V20/00; G06V20/40
Claims Analysis
Total Claims: 28
Claim 1
Independent Claim
145 words
A computer-implemented method of processing a sequence of video frames, comprising: extracting key frames from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores; resolution-enhancing image content of each of the keyframes, based on estimated motion between the keyframe and one or more of the video frames that neighbor the keyframe in the sequence; generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes; and determining a layout of the sequence of cropped images on one or more pages; and producing a graphical representation of the layout.
Claim 2
Dependent Claim
145 words
A computer-implemented method of processing a sequence of video frames, comprising: extracting key frames from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores; resolution-enhancing image content of each of the keyframes, based on estimated motion between the keyframe and one or more of the video frames that neighbor the keyframe in the sequence; generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes; and determining a layout of the sequence of cropped images on one or more pages; and producing a graphical representation of the layout.
Claim 3
Dependent Claim
47 words
The method according to claim 1 , wherein the identifying comprises determining a respective color histogram for each of the video frames and selecting ones of the video frames as respective ones of the candidate frames based on comparisons between the respective ones of the color histograms.
Claim 4
Dependent Claim
30 words
The method according to claim 1 , wherein the extracting comprises for each of the keyframes extracting the one or more video frames that neighbor the keyframe from the sequence.
Claim 5
Dependent Claim
130 words
The method according to claim 1 , wherein the resolution-enhancing comprises: computing a motion map for each pairing of an extracted frame and a respective extracted neighboring frame in a sequence of base images, wherein each computed motion map comprising a set of motion vectors mapping extracted frame image pixels to respective extracted neighboring frame image pixels; assigning respective regions of a target image to motion classes based on the computed motion maps, the target image having a target resolution level and the base images having a base resolution level equal to or lower than the target resolution level; and computing pixel values for the target image based on corresponding pixel value contributions from the base images selected in accordance with the motion classes assigned to the target image regions.
Claim 6
Dependent Claim
39 words
The method according to claim 1 , wherein the cropping comprises identifying a respective region of interest in respective ones of the keyframes and cropping the keyframes such that the respective cropped images contain the respective regions of interest.
Claim 7
Dependent Claim
51 words
The method according to claim 5 , wherein the cropping comprises: identifying the regions of interest based on respective saliency images that are derived from the keyframes; and determining one or more crop boundaries for each of the regions of interest based on application of one or more image composition rules.
Claim 8
Dependent Claim
20 words
The method according to claim 1 , wherein the cropping comprises removing black top and bottom margins of the keyframes.
Claim 9
Dependent Claim
29 words
The method according to claim 1 , wherein the cropping comprises cropping each of the keyframes to respective ones of the cropped images that comprise identified regions of interest.
Claim 10
Dependent Claim
21 words
The method according to claim 1 , wherein the layout comprises images corresponding to cropped and uncropped ones of the keyframes.
Claim 11
Dependent Claim
66 words
The method according to claim 1 , wherein the determining comprises for each of the pages; establishing a respective candidate tree representing a recursive partitioning of the page into regions containing respective ones of the cropped images; and in response to a determination that the candidate tree is a feasible candidate tree arranging the respective cropped images within the page in accordance with the candidate tree.
Claim 12
Dependent Claim
30 words
The method according to claim 1 , wherein the determining comprises assigning the cropped frame images to respective ones of the pages based on the temporal sequence of the keyframes.
Claim 13
Dependent Claim
17 words
The method according to claim 1 , wherein the producing comprises creating a poster from the layout.
Claim 14
Dependent Claim
19 words
The method according to claim 1 , wherein the producing comprises creating a DVD cover page from the layout.
Claim 15
Dependent Claim
19 words
The method according to claim 1 , wherein the producing comprises creating a video index page from the layout.
Claim 16
Dependent Claim
17 words
The method according to claim 1 , wherein the producing comprises creating a storybook from the layout.
Claim 17
Dependent Claim
17 words
The method according to claim 1 , wherein the producing comprises creating a postcard from the layout.
Claim 18
Dependent Claim
17 words
The method according to claim 1 , wherein the producing comprises creating a calendar from the layout.
Claim 19
Dependent Claim
18 words
The method according to claim 1 , wherein the producing comprises displaying the layout on a display monitor.
Claim 20
Dependent Claim
23 words
The method according to claim 1 , wherein the cropping comprises cropping the keyframes in accordance with a user-selected cropping control parameter value.
Claim 21
Dependent Claim
14 words
The method of claim 1 , wherein the cropping is performed before the resolution-enhancing.
Claim 22
Dependent Claim
14 words
The method of claim 1 , wherein the resolution-enhancing is performed before the cropping.
Claim 23
Dependent Claim
112 words
A computer-implemented method of processing a sequence of video, comprising: extracting keyframes from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores; generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes; determining a layout of the sequence of cropped images on one or more pages; and producing a graphical representation of the layout.
Claim 24
Dependent Claim
112 words
A computer-implemented method of processing a sequence of video, comprising: extracting keyframes from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores; generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes; determining a layout of the sequence of cropped images on one or more pages; and producing a graphical representation of the layout.
Claim 25
Dependent Claim
180 words
Apparatus for processing a page sequence of video frames, comprising: a computer-readable medium storing computer-readable instructions; and a data processing unit coupled to the memory, operable to execute the instructions, and based at least in part on the execution of the instructions operable to perform operations comprising: extracting keyframes from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores; resolution-enhancing image content of each of the keyframes based on estimated motion between the keyframe and one or more of the video frames that neighbor the keyframe in the sequence; generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes; and determining a layout of the sequence of cropped images on one or more panes; and producing a graphical representation of the layout.
Claim 26
Dependent Claim
180 words
Apparatus for processing a page sequence of video frames, comprising: a computer-readable medium storing computer-readable instructions; and a data processing unit coupled to the memory, operable to execute the instructions, and based at least in part on the execution of the instructions operable to perform operations comprising: extracting keyframes from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores; resolution-enhancing image content of each of the keyframes based on estimated motion between the keyframe and one or more of the video frames that neighbor the keyframe in the sequence; generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes; and determining a layout of the sequence of cropped images on one or more panes; and producing a graphical representation of the layout.
Claim 27
Dependent Claim
164 words
At least one computer readable medium encoded with computer instructions adapted to be executed by a computer to perform a method of processing a sequence of video frames, the method comprising: extracting keyframes from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores; resolution-enhancing image content of each of the keyframes based on estimated motion between the keyframe and one or more of the video frames that neighbor the keyframe in the sequence; generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes and determining a layout of the sequence of cropped images on one or more pages; and producing a graphical representation of the layout.
Claim 28
Dependent Claim
164 words
At least one computer readable medium encoded with computer instructions adapted to be executed by a computer to perform a method of processing a sequence of video frames, the method comprising: extracting keyframes from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores; resolution-enhancing image content of each of the keyframes based on estimated motion between the keyframe and one or more of the video frames that neighbor the keyframe in the sequence; generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes and determining a layout of the sequence of cropped images on one or more pages; and producing a graphical representation of the layout.
References Cited (Backward Citations)
US3956758A
Cited
US4317620A
Cited
US4384336A
Cited
GB2124055B
Cited
US4423936A
Cited
US4541704A
Cited
US4605970A
Cited
US4724330A
Cited
US5053875A
Cited
US5345284A
Cited
US5136686A
Cited
EP0456414A2
Cited
US5227889A
Cited
US5091654A
Cited
US5486893A
Cited
US5499366A
Cited
US5555362A
Cited
US5557684A
Cited
US5511148A
Cited
US5384615A
Cited
US5517242A
Cited
US5485554A
Cited
US5500711A
Cited
EP0690413A3
Cited
US5666186A
Cited
US6393163B1
Cited
US5900909A
Cited
US5973733A
Cited
US5712995A
Cited
US5719594A
Cited
US5729254A
Cited
US5760786A
Cited
US5956738A
Cited
US6628805B1
Cited
US6067112A
Cited
US5920315A
Cited
GB2350251B
Cited
US5978519A
Cited
EP0824246B1
Cited
US6798834B1
Cited
US6727909B1
Cited
US6249613B1
Cited
US6137544A
Cited
US6125229A
Cited
US6008809A
Cited
US6385245B1
Cited
US6686970B1
Cited
US6301586B1
Cited
EP0912047B1
Cited
US6289110B1
Cited
US6448956B1
Cited
EP0924923A2
Cited
US20020051208A1
Cited
US6380954B1
Cited
US6587599B1
Cited
EP0949802A3
Cited
EP0959299B1
Cited
US6563602B1
Cited
US6570612B1
Cited
JP2000227960A
Cited
US6304682B1
Cited
US6476873B1
Cited
US6269118B1
Cited
US6298145B1
Cited
US6415306B2
Cited
US6307560B1
Cited
JP2000244800A
Cited
US6535639B1
Cited
US7281199B1
Cited
US6636650B1
Cited
US6728407B1
Cited
US6636648B2
Cited
US6813395B1
Cited
US7124360B1
Cited
US7093263B1
Cited
JP2001148843A
Cited
US6466618B1
Cited
US7016540B1
Cited
WO2001039019A2
Cited
US6549643B1
Cited
US6671405B1
Cited
EP1117251A1
Cited
US6556721B1
Cited
US20010033296A1
Cited
US7149968B1
Cited
US6771801B1
Cited
EP1132835A1
Cited
US6701306B1
Cited
EP1158464A1
Cited
US20010048763A1
Cited
US20020118761A1
Cited
US20020028071A1
Cited
US20040125877A1
Cited
US20040128317A1
Cited
US20020070982A1
Cited
EP1182582A3
Cited
EP1186992A2
Cited
US6711587B1
Cited
WO2002037939A2
Cited
JP2002142092A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2013-07-19 | 2014-01-18 | 2014-07-20 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2017-07-19 | 2018-01-18 | 2018-07-20 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2021-07-19 | 2022-01-18 | 2022-07-20 | $7700.00 | Window Closed / Lapsed |