Patent DE206450C successfully refreshed from Google Patents.
Patent US1372635A successfully refreshed from Google Patents.
Patent US3567373A successfully refreshed from Google Patents.
US8687941B2
Active / In Force
US
⏱️ 4.0 years remaining
Automatic static video summarization
Assignee: International Business Machines Corp
•
Filing: 2010-10-29
•
Granted: 2014-04-01
•
Expires: 2030-10-29
Abstract
Abstract Techniques are disclosed for automatic static summarization of videos. For example, a method of creating a static summary of a video comprises the following steps. Shots in the video are detected, wherein the detected shots are frames of the video having a correlation. The detected shots are clustered into clusters based on similarity. The clusters of shots are ranked. At least a portion of the shots are selected based on cluster ranking for inclusion in the static summary. The static summary is generated by combining thumbnail images of the selected shots. Prior to the ranking step, the method may further comprise detecting a presence of slides in any of the shots, and the ranking of a given shot is based in part on whether the shot is a slide. By way of example, such static summaries can be shared in emails and in calendar applications.
Inventors
- 👤 Ahmet Emir Dirik
- 👤 Jennifer Lai
- 👤 Mercan Topkara
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:02
CPC Classifications
H; H04; H04N; H04N5/00; H04N5/14; H04N5/147; G; G06; G06F; G06F16/00; G06F16/70; G06F16/73; G06F16/738; G06F16/739; G11; G11B; G11B27/00; G11B27/10; G11B27/19; G11B27/28; G11B27/34; H04N21/00; H04N21/20; H04N21/23; H04N21/234; H04N21/23418; H04N21/2343; H04N21/234345; H04N21/234363; H04N21/80; H04N21/85; H04N21/854; H04N21/8549; H04N9/00; H04N9/79; H04N9/80; H04N9/82; H04N9/8205; H04N9/8227
Claims Analysis
Total Claims: 25
Claim 1
Independent Claim
170 words
A method of creating a static summary of a video, comprising: detecting shots in the video, wherein the detected shots are frames of the video having a correlation; clustering the detected shots into clusters based on similarity; ranking the clusters of shots; selecting at least a portion of the shots based on cluster ranking for inclusion in the static summary; and generating the static summary by combining thumbnail images of the selected shots, wherein prior to the ranking step, the method further comprises detecting a presence of slides in any of the shots, and ranking of a given shot is based in part on whether the shot is a slide, wherein the ranking step further comprises ranking the shots that contain slides among each other, wherein the slides are assigned to higher or lower ranks based on an amount of text in the slides, wherein ranking of shots of the video that contain slides further comprises ranking based on a slide content activity metric to detect visually busy slide frames.
Claim 2
Independent Claim
170 words
A method of creating a static summary of a video, comprising: detecting shots in the video, wherein the detected shots are frames of the video having a correlation; clustering the detected shots into clusters based on similarity; ranking the clusters of shots; selecting at least a portion of the shots based on cluster ranking for inclusion in the static summary; and generating the static summary by combining thumbnail images of the selected shots, wherein prior to the ranking step, the method further comprises detecting a presence of slides in any of the shots, and ranking of a given shot is based in part on whether the shot is a slide, wherein the ranking step further comprises ranking the shots that contain slides among each other, wherein the slides are assigned to higher or lower ranks based on an amount of text in the slides, wherein ranking of shots of the video that contain slides further comprises ranking based on a slide content activity metric to detect visually busy slide frames.
Claim 3
Dependent Claim
67 words
The method of claim 1 , wherein the detection of shots in the video that do not contain slides further comprises: determining color and intensity histograms for frames in the video; computing a correlation between the color and intensity histograms of successive frames; comparing histogram correlation results with shot detection threshold values; and identifying a frame as a shot based on shot detection threshold value comparison results.
Claim 4
Dependent Claim
46 words
The method of claim 1 , wherein the detection of slides in the video further comprises: computing frame correlations for successive frames; and tagging successive frames as a slide when the correlation between the successive frames is at least greater than a slide transition threshold value.
Claim 5
Dependent Claim
25 words
The method of claim 3 , wherein prior to frame correlation computing, frames in the video are filtered using a two dimensional high pass filter.
Claim 6
Dependent Claim
30 words
The method of claim 3 , wherein, when one frame in a given shot is tagged as a slide, all other frames in the given shot are tagged as slides.
Claim 7
Dependent Claim
18 words
The method of claim 1 , wherein a frame comprising a screen-share image is detected as a slide.
Claim 8
Dependent Claim
63 words
The method of claim 1 , wherein clustering the detected shots into clusters based on similarity further comprises: sorting shots based on shot duration; finding shots similar to the longest non-clustered shot, wherein similarity is based on a correlation coefficient function; and assigning the longest non-clustered shot and the shots found to be similar to the longest non-clustered shot to a given cluster.
Claim 9
Dependent Claim
17 words
The method of claim 7 , further comprising removing shots with unrelated content from the given cluster.
Claim 10
Dependent Claim
76 words
The method of claim 8 , wherein the removal of unrelated shots from the given cluster further comprises: selecting a given number of random shots from the given cluster; comparing each random shot with other shots; generating alternative clusters based on the random shot comparisons; computing an average similarity for each of the given cluster and the alternative clusters; and selecting the cluster with the maximum average similarity as the cluster used in the ranking step.
Claim 11
Dependent Claim
18 words
The method of claim 7 , wherein prior to clustering, a representative frame is assigned to each shot.
Claim 12
Dependent Claim
26 words
The method of claim 10 , wherein the middle of the shot is selected as the representative frame when the shot does not contain any slides.
Claim 13
Dependent Claim
25 words
The method of claim 10 , wherein the first frame of the shot is selected as the representative frame when the shot contains a slide.
Claim 14
Dependent Claim
36 words
The method of claim 1 , wherein the ranking further comprises: ranking shots that do not contain slides; and for any shots in which slides were detected, performing optical character recognition on elements in the slides.
Claim 15
Dependent Claim
27 words
The method of claim 13 , wherein the ranking of shots of the video that do not contain slides further comprises ranking based on duration and diversity.
Claim 16
Dependent Claim
25 words
The method of claim 13 , wherein ranking of shots of the video that contain slides further comprises ranking based on readability and human comprehension.
Claim 17
Dependent Claim
26 words
The method of claim 13 , wherein ranking of shots of the video that contain slides further comprises ranking based on an optical character recognition metric.
Claim 18
Dependent Claim
25 words
The method of claim 13 , wherein ranking of shots of the video that contain slides further comprises ranking based on a slide duration metric.
Claim 19
Dependent Claim
20 words
The method of claim 1 , further comprising generating textual information corresponding to the thumbnail images of the static summary.
Claim 20
Dependent Claim
12 words
The method of claim 18 , wherein the textual information is searchable.
Claim 21
Dependent Claim
35 words
The method of claim 18 , wherein the textual information comprises a uniform resource locator which is selectable to playback some or all of the video corresponding to the thumbnail images of the static summary.
Claim 22
Dependent Claim
70 words
The method of claim 1 , wherein when it is determined that the video is attached to an email message or the email message contains a link to the video, further comprising the step of including the generated static summary in the email message in place of the video, wherein some or all of the video is selectable for playback using one or more hyperlinks associated with the static summary.
Claim 23
Dependent Claim
69 words
The method of claim 1 , wherein when the video is a recording of a meeting, further comprising identifying one or more calendar entries corresponding to the meeting in a database of a calendar application program, and including the generated static summary in the one or more calendar entries, wherein some or all of the video is selectable for playback using one or more hyperlinks associated with the summary.
Claim 24
Dependent Claim
203 words
An article of manufacture for creating a static summary of a video, the article of manufacture comprising a non-transitory computer readable storage medium having tangibly embodied thereon computer readable program code which, when executed, causes a computer to: detect shots in the video, wherein the detected shots are frames of the video having a correlation; cluster the detected shots into clusters based on similarity; rank the clusters of shots; select at least a portion of the shots based on cluster ranking for inclusion in the static summary; and generate the static summary by combining thumbnail images of the selected shots, wherein prior to ranking the clusters of shots, the computer readable program code, when executed, further causes the computer to detect a presence of slides in any of the shots, and ranking of a given shot is based in part on whether the shot is a slide, wherein the shots that contain slides are ranked among each other, wherein the slides are assigned to higher or lower ranks based on an amount of text in the slides, wherein ranking of shots of the video that contain slides further comprises ranking based on a slide content activity metric to detect visually busy slide frames.
Claim 25
Dependent Claim
203 words
An article of manufacture for creating a static summary of a video, the article of manufacture comprising a non-transitory computer readable storage medium having tangibly embodied thereon computer readable program code which, when executed, causes a computer to: detect shots in the video, wherein the detected shots are frames of the video having a correlation; cluster the detected shots into clusters based on similarity; rank the clusters of shots; select at least a portion of the shots based on cluster ranking for inclusion in the static summary; and generate the static summary by combining thumbnail images of the selected shots, wherein prior to ranking the clusters of shots, the computer readable program code, when executed, further causes the computer to detect a presence of slides in any of the shots, and ranking of a given shot is based in part on whether the shot is a slide, wherein the shots that contain slides are ranked among each other, wherein the slides are assigned to higher or lower ranks based on an amount of text in the slides, wherein ranking of shots of the video that contain slides further comprises ranking based on a slide content activity metric to detect visually busy slide frames.
References Cited (Backward Citations)
US6956573B1
Cited
US6646655B1
Cited
US7246314B2
Cited
US6535639B1
Cited
US6690725B1
Cited
US7151852B2
Cited
US20020069218A1
Cited
US20080320509A1
Cited
US20020097277A1
Cited
US7593618B2
Cited
US20030081937A1
Cited
US7203620B2
Cited
US20080109848A1
Cited
US20090287685A1
Cited
US7149974B2
Cited
US20030236792A1
Cited
US7143352B2
Cited
EP1456779B1
Cited
US20040139042A1
Cited
US7212666B2
Cited
US20070046669A1
Cited
US20050151735A1
Cited
US7530021B2
Cited
CN1581163A
Cited
US20060047816A1
Cited
US20090177469A1
Cited
US7466858B2
Cited
US7679518B1
Cited
US20070112926A1
Cited
US20070168864A1
Cited
US7613365B2
Cited
US20100039296A1
Cited
US7509347B2
Cited
US20080027921A1
Cited
US20080077869A1
Cited
US20080082557A1
Cited
US20090006608A1
Cited
US20090049129A1
Cited
US20110282706A1
Cited
US20090080853A1
Cited
US8200063B2
Cited
US20090119246A1
Cited
US20090125365A1
Cited
US20090234721A1
Cited
US20090177774A1
Cited
US20090271438A1
Cited
US20100114627A1
Cited
US20100153160A1
Cited
US20100191741A1
Cited
US20100299763A1
Cited
US20100312726A1
Cited
US20110022967A1
Cited
US20110078635A1
Cited
US20110137988A1
Cited
US20130132138A1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2017-03-31 | 2017-09-30 | 2018-04-01 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2021-03-31 | 2021-09-30 | 2022-04-01 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2025-03-31 | 2025-09-30 | 2026-04-01 | $7700.00 | Window Closed / Lapsed |