US20160034786A1
Active / In Force
US
⏱️ 7.7 years remaining
Computerized machine learning of interesting video sections
Assignee: Microsoft Technology Licensing LLC
•
Filing: 2014-07-29
•
Granted: —
•
Expires: 2034-07-29
Abstract
Abstract This disclosure describes techniques for training models from video data and applying the learned models to identify desirable video data. Video data may be labeled to indicate a semantic category and/or a score indicative of desirability. The video data may be processed to extract low and high level features. A classifier and a scoring model may be trained based on the extracted features. The classifier may estimate a probability that the video data belongs to at least one of the categories in a set of semantic categories. The scoring model may determine a desirability score for the video data. New video data may be processed to extract low and high level features, and feature values may be determined based on the extracted features. The learned classifier and scoring model may be applied to the feature values to determine a desirability score associated with the new video data.
Inventors
- 👤 Nitin Suri
- 👤 Xian-Sheng Hua
- 👤 Tzong-Jhy Wang
- 👤 William D. Sproule
- 👤 Andrew S. Ivory
- 👤 Jin Li
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 09:54
CPC Classifications
G06K9/6256; G; G06; G06T; G06T7/00; G06T7/20; G06F; G06F18/00; G06F18/20; G06F18/21; G06F18/214; G06K9/46; G06K9/66; G06T7/0022; G06V; G06V10/00; G06V10/70; G06V10/77; G06V10/774; G06V20/00; G06V20/40; G06V20/46; G06T2207/00; G06T2207/10; G06T2207/10016
Claims Analysis
Total Claims: 23
Claim 1
Independent Claim
142 words
1 . A method comprising: receiving, by one or more computing devices, video data; extracting, by at least one of the one or more computing devices, a plurality of features from the video data; determining, by at least one of the one or more computing devices, a first set of feature values associated with the plurality of features, the first set of feature values for training a classifier and a scoring model; determining, by at least one of the one or more computing devices, a second set of feature values based on applying the classifier to the video data; training, by at least one of the one or more computing devices, the scoring model based on the first set of feature values and the second set of feature values, wherein the scoring model determines a desirability score associated with the video data.
Claim 2
Dependent Claim
142 words
1 . A method comprising: receiving, by one or more computing devices, video data; extracting, by at least one of the one or more computing devices, a plurality of features from the video data; determining, by at least one of the one or more computing devices, a first set of feature values associated with the plurality of features, the first set of feature values for training a classifier and a scoring model; determining, by at least one of the one or more computing devices, a second set of feature values based on applying the classifier to the video data; training, by at least one of the one or more computing devices, the scoring model based on the first set of feature values and the second set of feature values, wherein the scoring model determines a desirability score associated with the video data.
Claim 3
Dependent Claim
21 words
2 . The method of claim 1 , wherein the plurality of features includes low level features and high level features.
Claim 4
Dependent Claim
84 words
3 . The method of claim 2 , wherein the first set of feature values represents: one or more low level feature values associated with the low level features; one or more high level feature values associated with the high level features; and a plurality of derivative feature values, wherein individual derivative feature values of the plurality of derivative feature values are derived from at least some of the one or more low level feature values or one or more high level feature values.
Claim 5
Dependent Claim
35 words
4 . The method of claim 1 , wherein the second set of feature values represents probabilities that the video data belongs to at least one semantic category of a predefined set of semantic categories.
Claim 6
Dependent Claim
23 words
5 . The method of claim 1 , wherein the video data comprises a video collection, a video segment, or a video frame.
Claim 7
Dependent Claim
54 words
6 . The method of claim 1 , wherein the first set of feature values represents at least one of video collection level feature values, video file level feature values, video segment level feature values, or video frame level feature values and the second set of feature values represents video file level feature values.
Claim 8
Dependent Claim
130 words
7 . A system comprising: memory; one or more processors; and one or more modules stored in the memory and executable by the one or more processors, the one or more modules including: an extracting module configured for extracting features from video data and determining a first set of feature values based on the extracted features; and a ranking module configured for determining a desirability score for the video data, the ranking module including: a classifying module configured for applying a classifier to the first set of feature values to determine a second set of feature values; and a scoring module configured for applying a scoring model to the first set of feature values and the second set of feature values to determine a desirability score for the video data.
Claim 9
Dependent Claim
130 words
7 . A system comprising: memory; one or more processors; and one or more modules stored in the memory and executable by the one or more processors, the one or more modules including: an extracting module configured for extracting features from video data and determining a first set of feature values based on the extracted features; and a ranking module configured for determining a desirability score for the video data, the ranking module including: a classifying module configured for applying a classifier to the first set of feature values to determine a second set of feature values; and a scoring module configured for applying a scoring model to the first set of feature values and the second set of feature values to determine a desirability score for the video data.
Claim 10
Dependent Claim
39 words
8 . The system of claim 7 , wherein the features include at least one of: exposure quality; saturation quality; hue variety; stability; face detection; face recognition; face tracking; saliency analysis; audio power analysis; speech detection; and motion analysis.
Claim 11
Dependent Claim
53 words
9 . The system of claim 7 , wherein the first set of feature values represents: one or more feature values associated with the features; and a plurality of derivative feature values, wherein individual derivative feature values of the plurality of derivative feature values are derived from the one or more feature values.
Claim 12
Dependent Claim
35 words
10 . The system of claim 7 , wherein the second set of feature values represents probabilities that the video data belongs to at least one semantic category of a predefined set of semantic categories
Claim 13
Dependent Claim
56 words
11 . The system of claim 7 , wherein the one or more modules further include a segmenting module configured for: identifying video segments in the video data based at least in part on the video segments having desirability scores above a predetermined threshold; and detecting boundaries in the video data associated with the video segments.
Claim 14
Dependent Claim
72 words
12 . The system of claim 11 , wherein the one or more modules further include a post-processing module configured for: ranking the video data based at least in part on the desirability scores for the video data; or creating a highlight video based at least in part on the detecting the boundaries in the video data and adding transitions between the video segments having the desirability scores above the predetermined threshold.
Claim 15
Dependent Claim
140 words
13 . One or more computer-readable storage media encoded with instructions that, when executed by a processor, perform acts comprising: receiving video data including a plurality of video frames; extracting a plurality of features from individual video frames of the plurality of video frames to determine a first set of feature values associated with the individual video frames; applying a classifier to the first set of feature values to determine a second set of feature values associated with the individual video frames, wherein the second set of feature values represents probabilities that the individual video frames belong to at least one semantic category of a predefined set of semantic categories; and applying a scoring model to the first set of feature values and the second set of feature values to determine a desirability score associated with the individual video frames.
Claim 16
Dependent Claim
140 words
13 . One or more computer-readable storage media encoded with instructions that, when executed by a processor, perform acts comprising: receiving video data including a plurality of video frames; extracting a plurality of features from individual video frames of the plurality of video frames to determine a first set of feature values associated with the individual video frames; applying a classifier to the first set of feature values to determine a second set of feature values associated with the individual video frames, wherein the second set of feature values represents probabilities that the individual video frames belong to at least one semantic category of a predefined set of semantic categories; and applying a scoring model to the first set of feature values and the second set of feature values to determine a desirability score associated with the individual video frames.
Claim 17
Dependent Claim
39 words
14 . The computer-readable storage media of claim 13 , wherein the first set of feature values represents feature values associated with low level features, high level features, and derivatives of the low level features and high level features.
Claim 18
Dependent Claim
47 words
15 . The computer-readable storage media of claim 13 , wherein the video data comprises video files and the acts further comprise determining a desirability score for individual video files based on an average desirability score associated with individual video frames associated with the individual video file.
Claim 19
Dependent Claim
33 words
16 . The computer-readable storage media of claim 15 , wherein the acts further comprise ranking the individual video files based at least in part on the desirability scores for the video files.
Claim 20
Dependent Claim
57 words
17 . The computer-readable storage media of claim 13 , wherein the acts further comprise: identifying video segments having desirability scores above a predetermined threshold, wherein the video segments include two or more of the individual video frames having desirability scores above the predetermined threshold; and detecting boundaries in the video data associated with the video segments.
Claim 21
Dependent Claim
100 words
18 . The computer-readable storage media of claim 17 , wherein the detecting boundaries associated with the video segments comprises: analyzing the individual video frames for motion data; detecting a camera motion in the motion data including at least one of: pan to left; pan to right; pan to top; pan to bottom; zoom in; or zoom out; and identifying the boundaries associated with the video segments based at least in part on a change in camera motion between a first individual video frame of the individual video frames and a second individual video frame of the individual video frames.
Claim 22
Dependent Claim
81 words
19 . The computer-readable storage media of claim 17 , wherein the detecting boundaries associated with the video segments comprises: analyzing the individual video frames for motion data; determining object motion intensities for the individual video frames based on the motion data; and detecting the boundaries when an object motion intensity of a first video frame of the individual video frames and an object motion intensity of a second video frame of the individual video frames differ by a predetermined threshold.
Claim 23
Dependent Claim
61 words
20 . The computer-readable storage media of claim 17 , wherein the acts further comprise creating a new video file including the video segments having desirability scores above a predetermined threshold by manipulating the video data based on the detecting of the boundaries in the video data and adding transitions between the video segments having desirability scores above the predetermined threshold.
References Cited (Backward Citations)
US6771268B1
Cited
US7383504B1
Cited
US20050008198A1
Cited
US20030091235A1
Cited
US20040010430A1
Cited
US20040201609A1
Cited
US20050198067A1
Cited
US7707008B1
Cited
US8185543B1
Cited
US7933338B1
Cited
US8134596B2
Cited
US20100045799A1
Cited
US20070101269A1
Cited
US20070183497A1
Cited
US8560406B1
Cited
US8379154B2
Cited
US20070263128A1
Cited
US20070289432A1
Cited
US20080002771A1
Cited
US20080184120A1
Cited
US20080195981A1
Cited
US20090079871A1
Cited
US8200063B2
Cited
US8311344B2
Cited
US20090208106A1
Cited
US20090292685A1
Cited
US9141859B2
Cited
US20100142803A1
Cited
US20120095817A1
Cited
US8730397B1
Cited
US8452778B1
Cited
US8489589B2
Cited
US9025866B2
Cited
US8565536B2
Cited
US20110243450A1
Cited
US8687941B2
Cited
US20120233159A1
Cited
US20120301014A1
Cited
US20140135966A1
Cited
US20130051756A1
Cited
US20130142418A1
Cited
US20140270350A1
Cited
US20150082349A1
Cited
US20150296228A1
Cited
US20160034748A1
Cited
US20160117749A1
Cited
USPTO Annuity & Maintenance Windows
Maintenance schedule is calculated for granted US utility patents.