US5805733A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and system for detecting scenes and summarizing video sequences
Assignee: Apple Inc
•
Filing: 1994-12-12
•
Granted: 1998-09-08
•
Expires: 2014-12-12
Abstract
Abstract A method and system summarizes scenes in a video sequence by detecting scene changes, and then comparing scenes in a moving window to determine their similarity. Similar scenes are consolidated and represented by a representative frame, a number of which are displayed to a user. Scene changes are detected by comparing average color histograms for each scene, motion compensated pixel differences or motion compensated edge maps, or a combination of these methods. Scenes in the video sequence are selected for summarizing according to their normalized time duration. Of the selected scenes, similar or related scenes are determined by comparing the average color histograms of each pair of scenes in a moving window, using a standard population error measure, such as a Chi-squared test. For each set of related scenes, a representative frame is taken, either as the medial frame from the entire time duration of the related scenes or as the first frame of the medial scene in the set. The representative frames are displayed to the user, wherein set of related scenes are retrived by selecting the representative frame for the set. A movie bar, a visual representation of a rectangular prism, is displayed to show the relative length and positioning of the related scenes.
Inventors
- 👤 Katherine Wang
- 👤 James Normile
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-27 12:42
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/70; G06F16/73; G06F16/738; G06F16/739; G06F16/74; G06F16/745; G06F16/78; G06F16/783; G06F16/7847; G06F16/785; G06V; G06V20/00; G06V20/40
Claims Analysis
Total Claims: 28
Claim 1
Independent Claim
125 words
A method of summarizing a temporally ordered plurality of scenes in a video sequence including a plurality of frames, comprising the steps of: detecting at least one scene in the plurality of frames, each scene including at least one related frame by: determining for each frame at least one global measurement; and determining a scene change between a pair of successive frames from a difference between the global measurements of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames; determining at least one set of related scenes; determining a representative frame for each set of related scenes; and displaying each representative frame.
Claim 2
Independent Claim
125 words
A method of summarizing a temporally ordered plurality of scenes in a video sequence including a plurality of frames, comprising the steps of: detecting at least one scene in the plurality of frames, each scene including at least one related frame by: determining for each frame at least one global measurement; and determining a scene change between a pair of successive frames from a difference between the global measurements of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames; determining at least one set of related scenes; determining a representative frame for each set of related scenes; and displaying each representative frame.
Claim 3
Dependent Claim
35 words
The method of claim 1, wherein the determination of the scene change is performed using a neural network receiving the difference between the global measurements, the Chi-squared value, and the normalized motion compensated pixel difference.
Claim 4
Dependent Claim
79 words
The method of claim 1, wherein the step of detecting at least one scene further comprises the steps of: for each frame of the video sequence, beginning with a first frame, determining a motion compensated pixel luminance difference between a current frame and a next frame; and responsive to the motion compensated pixel luminance difference being greater than a threshold value, associating the first frame with all frames up to and including the current frame to form a scene.
Claim 5
Dependent Claim
30 words
The method of claim 3, further comprising the step of: equalizing a color histogram of each frame of the video sequence prior to determining the motion compensated pixel luminance difference.
Claim 6
Dependent Claim
79 words
The method of claim 1, wherein the step of detecting at least one scene further comprises the steps of: for each frame of the video sequence, beginning with a first frame, determining a motion compensated edge map difference between a current frame and a next frame; and responsive to the motion compensated edge map difference being greater than a threshold value, associating the first frame with all frames up to and including the current frame to form a scene.
Claim 7
Dependent Claim
83 words
The method of claim 1, wherein the step of detecting at least one scene further comprises the steps of: for each frame of the video sequence, beginning with a first frame, determining a standard error of differences between a color histogram for a current frame and a color histogram of a next frame; and responsive to the standard error being greater than a threshold value, associating the first frame with all frames up to and including the current frame to form a scene.
Claim 8
Dependent Claim
15 words
The method of claim 9, wherein the standard error is measured using a Chi-squared test.
Claim 9
Dependent Claim
68 words
The method of claim 1, wherein the step determining at least one set of related scenes further comprises the steps of: determining for each scene a summary measure derived from all frames in the scene; comparing the summary measure of each scene with the summary measures of selected other scenes using a standard error measure; and identifying at least one set of scenes having approximately equal summary measures.
Claim 10
Dependent Claim
20 words
The method of claim 8 wherein the summary measure is an average color histogram for all frames in the scene.
Claim 11
Dependent Claim
28 words
The method of claim 8, further comprising the step of: linking scenes in each group of scenes into a consolidated scene for display as a single continuous scene.
Claim 12
Dependent Claim
34 words
The method of claim 8, wherein the representative frame for each set of related scenes is a frame temporally medial to a first frame and a last frame in the set of related scenes.
Claim 13
Dependent Claim
27 words
The method of claim 8, wherein the representative frame for each set of related scenes is selected from a longest scene in the set of related scenes.
Claim 14
Independent Claim
169 words
A method of removing redundant scenes in a plurality of frames, each scene including at least one frame, the method comprising the steps of: detecting each scene in the plurality of frames, each scene including at least one related frame by: determining for each frame at least one global measurement; and determining a scene change between a pair of successive frames from a difference between the global measurements of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames; determining for each scene, an average color histogram for all frames included the scene; and comparing the average color histogram of each scene with the average color histogram of each of a selected number of subsequent scenes to identify scenes within the selected number of scenes having substantially similar average color histograms, each set of scenes having substantially similar average color histograms forming a set of related scenes.
Claim 15
Independent Claim
169 words
A method of removing redundant scenes in a plurality of frames, each scene including at least one frame, the method comprising the steps of: detecting each scene in the plurality of frames, each scene including at least one related frame by: determining for each frame at least one global measurement; and determining a scene change between a pair of successive frames from a difference between the global measurements of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames; determining for each scene, an average color histogram for all frames included the scene; and comparing the average color histogram of each scene with the average color histogram of each of a selected number of subsequent scenes to identify scenes within the selected number of scenes having substantially similar average color histograms, each set of scenes having substantially similar average color histograms forming a set of related scenes.
Claim 16
Dependent Claim
28 words
The method of claim 13 wherein the step of comparing is performing by determining a Chi squared value for the average color histograms of each pair of scenes.
Claim 17
Independent Claim
149 words
A method of removing redundant scenes in a plurality of frames, each scene including at least one frame, the method comprising the steps of: detecting each scene in the plurality of frames, each scene including at least one related frame by: determining for each frame at least one global measurement; and determining a scene change between a pair of successive frames from a difference between the global measurements of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames; determining for each scene a summary measure derived from all frames in the scene; and comparing the summary measure of each scene with the summary measure of selected subsequent scenes using a standard error measure to identify at least one group of scenes having substantially similar summary measures.
Claim 18
Independent Claim
149 words
A method of removing redundant scenes in a plurality of frames, each scene including at least one frame, the method comprising the steps of: detecting each scene in the plurality of frames, each scene including at least one related frame by: determining for each frame at least one global measurement; and determining a scene change between a pair of successive frames from a difference between the global measurements of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames; determining for each scene a summary measure derived from all frames in the scene; and comparing the summary measure of each scene with the summary measure of selected subsequent scenes using a standard error measure to identify at least one group of scenes having substantially similar summary measures.
Claim 19
Dependent Claim
14 words
The method of claim 15 wherein the summary measure is an average color histogram.
Claim 20
Dependent Claim
18 words
The method of claim 15, wherein the selected other scenes consists of a predetermined number of subsequent scenes.
Claim 21
Dependent Claim
21 words
The method of claim 15, further comprising the step of: linking scenes in each group of scenes into a consolidated scene.
Claim 22
Dependent Claim
33 words
The method of claim 15, wherein the representative frame for each set of related scenes a frame temporally medial to a first frame and a last frame in the set of related scenes.
Claim 23
Dependent Claim
43 words
The method of claim 1, further comprising the steps of: displaying a plurality of slices of frames as a rectangular prism; and highlighting in the plurality of slices, the slices of frames either a first frame or a last frame of each scene.
Claim 24
Dependent Claim
25 words
The method of claim 20, further comprising the step of retrieving in response to user selection of any frame, the scene including the selected frame.
Claim 25
Dependent Claim
69 words
The method of claim 20, further comprising the steps of: determining at least one scene having a time duration in excess of a threshold; displaying for the scene, and within the rectangular prism at the location of the scene, a compressed representation of all frames in the scene; and responsive to a user selection of the compressed representation, displaying a plurality of slices for all frames in the scene.
Claim 26
Independent Claim
188 words
A system for summarizing a temporally ordered plurality of scenes in a video sequence including a plurality of frames, comprising: an input device for receiving an input video sequence; a scene change detector coupled to the input device, the scene change detector detecting at least one scene in the plurality of frames, each scene including at least one related frame, the scene change detector detecting a scene change between a pair of successive frames from a difference between a global measurement of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames; a related scene detector, coupled to the scene change detector to receive a set of scenes, and determining at least one set of related scenes, the related scene detector determining a representative frame for each set of related scenes; a display for displaying the representative frame, and selected scenes; and a user interface display driver coupled to the related scene detector to receive the representative frames, and displaying each representative frame on the display.
Claim 27
Independent Claim
188 words
A system for summarizing a temporally ordered plurality of scenes in a video sequence including a plurality of frames, comprising: an input device for receiving an input video sequence; a scene change detector coupled to the input device, the scene change detector detecting at least one scene in the plurality of frames, each scene including at least one related frame, the scene change detector detecting a scene change between a pair of successive frames from a difference between a global measurement of each frame in the pair of frames, a Chi-squared value on a color histogram of each frame in the pair of frames, and a normalized motion compensated pixel difference between the frames; a related scene detector, coupled to the scene change detector to receive a set of scenes, and determining at least one set of related scenes, the related scene detector determining a representative frame for each set of related scenes; a display for displaying the representative frame, and selected scenes; and a user interface display driver coupled to the related scene detector to receive the representative frames, and displaying each representative frame on the display.
Claim 28
Dependent Claim
50 words
The system of claim 23, further comprising: a movie bar generator coupled to receive the scenes from the scene change detector, and the related scenes from the related scene detector, the movie bar generator creating a slice of each frames in each scene, and displaying the slices as rectangular prism.
References Cited (Backward Citations)
US4270143A
Cited
US4931868A
Cited
US5049991A
Cited
US5537528A
Cited
US5404174A
Cited
US5459517A
Cited
US5521841A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2001-09-07 | 2002-03-09 | 2002-09-08 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2005-09-07 | 2006-03-09 | 2006-09-08 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2009-09-07 | 2010-03-09 | 2010-09-08 | $7700.00 | Window Closed / Lapsed |