US6956904B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Summarizing videos using motion activity descriptors correlated with audio features
Assignee: Mitsubishi Electric Research Laboratories Inc
•
Filing: 2002-01-15
•
Granted: 2005-10-18
•
Expires: 2022-01-15
Abstract
Abstract A method for summarizing a video first detects audio peaks in a sub-sampled audio signal of the video. Then, motion activity in the video is extracted and filtered. The filtered motion activity is quantized to a continuous stream of digital pulses, one pulse for each frame. If the motion activity is greater than a predetermined threshold the pulse is one, otherwise the pulse is zero. Each quantized pulse is tested with respect to the timing of rising and falling edges. If the pulse meets the condition of the test, then the pulse is selected as a candidate pulse related to an interesting event in the video, otherwise the pulse is discarded. The candidate pulses are correlated, time-wise to the audio peaks, and patterns between the pulses and peaks are examined. The correlation patterns segment the video into uninteresting and interesting portions, which can then be summarized.
Inventors
- 👤 Romain Cabasson
- 👤 Kadir A. Peker
- 👤 Ajay Divakaran
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 11:07
CPC Classifications
G; G06; G06V; G06V20/00; G06V20/40; G06V20/46; G06F; G06F16/00; G06F16/70; G06F16/73; G06F16/738; G06F16/739; G06F16/78; G06F16/783; G06F16/7834; G06F16/7847; G06F16/786
Claims Analysis
Total Claims: 19
Claim 1
Independent Claim
55 words
A method for summarizing a compressed video, comprising: detecting audio peaks in an audio signal of the video; quantizing motion activity in the video as a continuous stream of pulses; and correlating the audio peaks with the stream of quantized pulses to identify uninteresting events and interesting events in the video to summarize the video.
Claim 2
Independent Claim
55 words
A method for summarizing a compressed video, comprising: detecting audio peaks in an audio signal of the video; quantizing motion activity in the video as a continuous stream of pulses; and correlating the audio peaks with the stream of quantized pulses to identify uninteresting events and interesting events in the video to summarize the video.
Claim 3
Dependent Claim
35 words
The method of claim 1 further comprising: discarding frames of the video associated with the uninteresting events; and concatenating frames of the video associated with the interesting events to form a summary of the video.
Claim 4
Dependent Claim
39 words
The method of claim 1 further comprising: sub-sampling the audio signal of the video down to a volume contour; and applying a sliding window to the volume contour to detect a local maximum corresponding to a particular audio peak.
Claim 5
Dependent Claim
26 words
The method of claim 3 where the local maximum is detected when (localMax−localMin)>(globalMax−globalMin)/3, using a local minimum, and predetermined global maximum and a predetermined global minimum.
Claim 6
Dependent Claim
24 words
The method of claim 3 wherein the sliding window has a duration of one minute, and slides forward in time in half minute steps.
Claim 7
Dependent Claim
68 words
The method of claim 1 further comprising: extracting the motion activity from each P-frame in the video; applying a moving average filter and a moving median filter to the extracted motion activity to generated smoothed motion activity; and setting the smoothed motion activity for each P-frame to one if greater than a predetermined threshold, and zero otherwise to quantize the motion activity as the continuous stream of pulses.
Claim 8
Dependent Claim
23 words
The method of claim 1 further comprising: measuring an average of magnitudes of motion vectors of each P-frame to extract the motion activity.
Claim 9
Dependent Claim
19 words
The method of claim 6 wherein the predetermined threshold is half a mean motion activity of the compressed video.
Claim 10
Dependent Claim
62 words
The method of claim 6 further comprising: testing each pulse to determine whether the quantized motion activity is at one for at least a first predetermined length of time before falling to zero and remains at zero for a second predetermined length of time; and selecting the test pulse as a candidate pulse associated with a particular interesting event in the video.
Claim 11
Dependent Claim
31 words
The method of claim 9 further comprising: discarding pulses failing the test from the continuous stream of pulses; and transforming each candidate pulse to have a third predetermined length of time.
Claim 12
Dependent Claim
28 words
The method of claim 10 further comprising: merging the transformed pulses, time-wise, with the detected audio peaks to obtain a set of time-correlated transformed pulses and audio peaks.
Claim 13
Dependent Claim
59 words
The method of claim 11 further comprising: testing if a rising edge of a particular transformed pulse is less than ten seconds after a particular time-correlated audio peak; and designating an entire duration starting from the particular audio peak and ending at a first falling edge after the particular audio peak is as a particular interesting event if true.
Claim 14
Dependent Claim
53 words
The method of claim 11 further comprising: testing if a falling edge of a particular transformed pulse is less than two seconds before a particular audio peak; and designating an entire duration starting from an immediately preceding rising edge and ending at the particular audio peak as a particular interesting event if true.
Claim 15
Independent Claim
61 words
A system for summarizing a compressed video, comprising: means for detecting audio peaks in an audio signal of the video; means for quantizing motion activity in the video as a continuous stream of pulses; and means for correlating the audio peaks with the stream of quantized pulses to identify uninteresting events and interesting events in the video to summarize the video.
Claim 16
Independent Claim
61 words
A system for summarizing a compressed video, comprising: means for detecting audio peaks in an audio signal of the video; means for quantizing motion activity in the video as a continuous stream of pulses; and means for correlating the audio peaks with the stream of quantized pulses to identify uninteresting events and interesting events in the video to summarize the video.
Claim 17
Dependent Claim
39 words
The system of claim 14 further comprising: means for discarding frames of the video associated with the uninteresting events; and means for concatenating frames of the video associated with the interesting events to form a summary of the video.
Claim 18
Dependent Claim
74 words
The system of claim 14 further comprising: means for extracting the motion activity from each P-frame in the video; means for applying a moving average filter and a moving median filter to the extracted motion activity to generated smoothed motion activity; and means for setting the smoothed motion activity for each P-frame to one if greater than a predetermined threshold, and zero otherwise to quantize the motion activity as the continuous stream of pulses.
Claim 19
Dependent Claim
66 words
The system of claim 16 further comprising: means for testing each pulse to determine whether the quantized motion activity is at one for at least a first predetermined length of time before falling to zero and remains at zero for a second predetermined length of time; and means for selecting the test pulse as a candidate pulse associated with a particular interesting event in the video.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2008-10-17 | 2009-04-18 | 2009-10-18 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2012-10-17 | 2013-04-18 | 2013-10-18 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2016-10-17 | 2017-04-18 | 2017-10-18 | $7700.00 | Window Closed / Lapsed |