§
Luxuse
Patent Intelligence
Sign In
← Back to Patent Explorer
US9646227B2 Active / In Force US ⏱️ 7.7 years remaining

Computerized machine learning of interesting video sections

Assignee: Microsoft Technology Licensing LLC
•
Filing: 2014-07-29
•
Granted: 2017-05-09
•
Expires: 2034-07-29
📄 Original PDF 🌐 Google Patents 🔄 Sync Now

Abstract

Abstract This disclosure describes techniques for training models from video data and applying the learned models to identify desirable video data. Video data may be labeled to indicate a semantic category and/or a score indicative of desirability. The video data may be processed to extract low and high level features. A classifier and a scoring model may be trained based on the extracted features. The classifier may estimate a probability that the video data belongs to at least one of the categories in a set of semantic categories. The scoring model may determine a desirability score for the video data. New video data may be processed to extract low and high level features, and feature values may be determined based on the extracted features. The learned classifier and scoring model may be applied to the feature values to determine a desirability score associated with the new video data.

Inventors

  • 👤 Nitin Suri
  • 👤 Xian-Sheng Hua
  • 👤 Tzong-Jhy Wang
  • 👤 William D. Sproule
  • 👤 Andrew S. Ivory
  • 👤 Jin Li

Bibliographic Details

Application Number —
Jurisdiction US
Data Source google_patents
Last Synchronized 2026-11-09 09:53
CPC Classifications
G06K9/6256; G; G06; G06T; G06T7/00; G06T7/20; G06F; G06F18/00; G06F18/20; G06F18/21; G06F18/214; G06K9/00744; G06V; G06V10/00; G06V10/70; G06V10/77; G06V10/774; G06V20/00; G06V20/40; G06V20/46; G06T2207/00; G06T2207/10; G06T2207/10016

Claims Analysis

Total Claims: 20
Claim 1
Independent Claim 289 words
A method comprising: receiving, by one or more computing devices, video data; extracting, by at least one of the one or more computing devices, a plurality of features from the video data; determining, by at least one of the one or more computing devices, a first set of feature values associated with the plurality of features, the first set of feature values for training a classifier and a scoring model; determining, by at least one of the one or more computing devices, a second set of feature values based on applying the classifier to the video data; training, by at least one of the one or more computing devices, the scoring model based on the first set of feature values and the second set of feature values; using the scoring model to determine a plurality of desirability scores associated with the video data, wherein an individual desirability score indicative of video quality is associated with an individual video frame in the video data; identifying video frames in the video data that have a desirability score above a predetermined threshold desirability score; analyzing the video data to determine, in association with the video frames, changes in camera motion and changes in object motion; and locating, based at least in part on the changes in camera motion and the changes in object motion, boundaries in the video data to produce one or more video segments, wherein: an individual video segment includes at least one video frame that has the desirability score above the predetermined threshold desirability score, and the locating the boundaries in the video data comprises determining that object motion intensity of a first video frame and object motion intensity of a second video frame differ by a predetermined threshold.
Claim 2
Independent Claim 289 words
A method comprising: receiving, by one or more computing devices, video data; extracting, by at least one of the one or more computing devices, a plurality of features from the video data; determining, by at least one of the one or more computing devices, a first set of feature values associated with the plurality of features, the first set of feature values for training a classifier and a scoring model; determining, by at least one of the one or more computing devices, a second set of feature values based on applying the classifier to the video data; training, by at least one of the one or more computing devices, the scoring model based on the first set of feature values and the second set of feature values; using the scoring model to determine a plurality of desirability scores associated with the video data, wherein an individual desirability score indicative of video quality is associated with an individual video frame in the video data; identifying video frames in the video data that have a desirability score above a predetermined threshold desirability score; analyzing the video data to determine, in association with the video frames, changes in camera motion and changes in object motion; and locating, based at least in part on the changes in camera motion and the changes in object motion, boundaries in the video data to produce one or more video segments, wherein: an individual video segment includes at least one video frame that has the desirability score above the predetermined threshold desirability score, and the locating the boundaries in the video data comprises determining that object motion intensity of a first video frame and object motion intensity of a second video frame differ by a predetermined threshold.
Claim 3
Dependent Claim 67 words
The method of claim 1 , wherein: the plurality of features includes low level features and high level features; the first set of feature values includes at least a low level feature value and a first high level feature value; and the second set of feature values includes at least one second high level feature value that is not included in the first set of feature values.
Claim 4
Dependent Claim 47 words
The method of claim 2 , wherein the first set of feature values represents a plurality of derivative feature values, wherein individual derivative feature values of the plurality of derivative feature values are derived from at least some low level feature values or high level feature values.
Claim 5
Dependent Claim 33 words
The method of claim 1 , wherein the second set of feature values represents probabilities that the video data belongs to at least one semantic category of a predefined set of semantic categories.
Claim 6
Dependent Claim 14 words
The method of claim 1 , wherein the video data comprises a video collection.
Claim 7
Dependent Claim 52 words
The method of claim 1 , wherein the first set of feature values represents at least one of video collection level feature values, video file level feature values, video segment level feature values, or video frame level feature values and the second set of feature values represents video file level feature values.
Claim 8
Independent Claim 306 words
A system comprising: memory; one or more processors; and one or more modules stored in the memory and executable by the one or more processors, the one or more modules including: an extracting module configured to extract features from video data and determine a first set of feature values based on the extracted features; a classifying module configured to apply a classifier to the first set of feature values to determine a second set of feature values and to use the second set of feature values to determine a probability that the video data belongs to at least one semantic category of a predefined set of semantic categories; a scoring module configured to apply a scoring model, based at least in part on the at least one semantic category, to the first set of feature values and the second set of feature values to determine a plurality of desirability scores for the video data, wherein an individual desirability score indicative of video quality is associated with an individual video frame in the video data; and a segmenting module configured to: identify video frames in the video data that have a desirability score above a predetermined threshold desirability score; analyze the video data to determine, in association with the video frames, changes in camera motion and changes in object motion; and locate, based at least in part on the changes in camera motion and the changes in object motion, boundaries in the video data to produce one or more video segments, wherein: an individual video segment includes at least one video frame that has the desirability score above the predetermined threshold desirability score, and the locating the boundaries in the video data comprises determining that object motion intensity of a first video frame and object motion intensity of a second video frame differ by a predetermined threshold.
Claim 9
Independent Claim 306 words
A system comprising: memory; one or more processors; and one or more modules stored in the memory and executable by the one or more processors, the one or more modules including: an extracting module configured to extract features from video data and determine a first set of feature values based on the extracted features; a classifying module configured to apply a classifier to the first set of feature values to determine a second set of feature values and to use the second set of feature values to determine a probability that the video data belongs to at least one semantic category of a predefined set of semantic categories; a scoring module configured to apply a scoring model, based at least in part on the at least one semantic category, to the first set of feature values and the second set of feature values to determine a plurality of desirability scores for the video data, wherein an individual desirability score indicative of video quality is associated with an individual video frame in the video data; and a segmenting module configured to: identify video frames in the video data that have a desirability score above a predetermined threshold desirability score; analyze the video data to determine, in association with the video frames, changes in camera motion and changes in object motion; and locate, based at least in part on the changes in camera motion and the changes in object motion, boundaries in the video data to produce one or more video segments, wherein: an individual video segment includes at least one video frame that has the desirability score above the predetermined threshold desirability score, and the locating the boundaries in the video data comprises determining that object motion intensity of a first video frame and object motion intensity of a second video frame differ by a predetermined threshold.
Claim 10
Dependent Claim 37 words
The system of claim 7 , wherein the features include at least one of: exposure quality; saturation quality; hue variety; stability; face detection; face recognition; face tracking; saliency analysis; audio power analysis; speech detection; or motion analysis.
Claim 11
Dependent Claim 56 words
The system of claim 7 , wherein the one or more modules further include a post-processing module configured to: rank the one or more video segments based at least in part on desirability scores of video frames included in an individual video segment; and create a highlight video based at least in part on the ranking.
Claim 12
Dependent Claim 272 words
One or more computer-readable storage media encoded with instructions that, when executed by a processor, perform acts comprising: receiving video data including a plurality of video frames; extracting a plurality of features from individual video frames of the plurality of video frames to determine a first set of feature values associated with the individual video frames; applying a classifier to the first set of feature values to determine a second set of feature values associated with the individual video frames; using the second set of feature values to determine individual probabilities that the individual video frames belong to at least one semantic category of a predefined set of semantic categories; applying a scoring model to the first set of feature values and the second set of feature values to determine desirability scores associated with the individual video frames; identifying a subset of the plurality of video frames in the video data that have a desirability score above a predetermined threshold desirability score; analyzing the video data to determine, in association with the subset of video frames, changes in camera motion and changes in object motion; and locating, based at least in part on the changes in camera motion and the changes in object motion, boundaries in the video data to produce one or more video segments, wherein: an individual video segment includes at least one video frame that has the desirability score above the predetermined threshold desirability score, and the locating the boundaries in the video data comprises determining that object motion intensity of a first video frame and object motion intensity of a second video frame differ by a predetermined threshold.
Claim 13
Dependent Claim 272 words
One or more computer-readable storage media encoded with instructions that, when executed by a processor, perform acts comprising: receiving video data including a plurality of video frames; extracting a plurality of features from individual video frames of the plurality of video frames to determine a first set of feature values associated with the individual video frames; applying a classifier to the first set of feature values to determine a second set of feature values associated with the individual video frames; using the second set of feature values to determine individual probabilities that the individual video frames belong to at least one semantic category of a predefined set of semantic categories; applying a scoring model to the first set of feature values and the second set of feature values to determine desirability scores associated with the individual video frames; identifying a subset of the plurality of video frames in the video data that have a desirability score above a predetermined threshold desirability score; analyzing the video data to determine, in association with the subset of video frames, changes in camera motion and changes in object motion; and locating, based at least in part on the changes in camera motion and the changes in object motion, boundaries in the video data to produce one or more video segments, wherein: an individual video segment includes at least one video frame that has the desirability score above the predetermined threshold desirability score, and the locating the boundaries in the video data comprises determining that object motion intensity of a first video frame and object motion intensity of a second video frame differ by a predetermined threshold.
Claim 14
Dependent Claim 37 words
The computer-readable storage media of claim 10 , wherein the first set of feature values represents feature values associated with low level features, high level features, and derivatives of the low level features and high level features.
Claim 15
Dependent Claim 45 words
The computer-readable storage media of claim 10 , wherein the video data comprises video files and the acts further comprise determining a desirability score for individual video files based on an average desirability score associated with individual video frames associated with the individual video file.
Claim 16
Dependent Claim 31 words
The computer-readable storage media of claim 12 , wherein the acts further comprise ranking the individual video files based at least in part on the desirability scores for the video files.
Claim 17
Dependent Claim 34 words
The computer-readable storage media of claim 10 , wherein the camera motion comprises at least one of: pan to left; pan to right; pan to top; pan to bottom; zoom in; or zoom out.
Claim 18
Dependent Claim 59 words
The computer-readable storage media of claim 10 , wherein the acts further comprise creating a new video file including the video segments having desirability scores above a predetermined threshold by manipulating the video data based on the detecting of the boundaries in the video data and adding transitions between the video segments having desirability scores above the predetermined threshold.
Claim 19
Dependent Claim 65 words
The system of claim 7 , wherein: the features include low level features and high level features; the first set of feature values includes at least a low level feature value and a first high level feature value; and the second set of feature values includes at least one second high level feature value that is not included in the first set of feature values.
Claim 20
Dependent Claim 69 words
The computer-readable storage media of claim 10 , wherein: the plural of features includes low level features and high level features; the first set of feature values includes at least a low level feature value and a first high level feature value; and the second set of feature values includes at least one second high level feature value that is not included in the first set of feature values.

References Cited (Backward Citations)

US5854902A Cited
US5917958A Cited
WO2000016243A1 Cited
US6771268B1 Cited
US7383504B1 Cited
US7386170B2 Cited
US20050008198A1 Cited
US20030091235A1 Cited
US20040010430A1 Cited
US20040202377A1 Cited
US20040201609A1 Cited
US20040228503A1 Cited
US20050084152A1 Cited
US20050289482A1 Cited
US20050198067A1 Cited
US7707008B1 Cited
US7933338B1 Cited
US8185543B1 Cited
US8134596B2 Cited
US20100045799A1 Cited
US20070019863A1 Cited
US7755619B2 Cited
US20070101269A1 Cited
US20090169168A1 Cited
US20070183497A1 Cited
US7616816B2 Cited
US8560406B1 Cited
US8379154B2 Cited
US20070263128A1 Cited
US20070289432A1 Cited
US20080002771A1 Cited
US20080019661A1 Cited
US7881505B2 Cited
US8661029B1 Cited
US7916894B1 Cited
US20080184120A1 Cited
US20080195981A1 Cited
US8189880B2 Cited
US20090016576A1 Cited
US20090079871A1 Cited
US8200063B2 Cited
US20090080853A1 Cited
US8170280B2 Cited
US8705810B2 Cited
US20090169065A1 Cited
US20090208106A1 Cited
US8311344B2 Cited
US20090292685A1 Cited
WO2010006334A1 Cited
US9245190B2 Cited
US20100014717A1 Cited
US20100054705A1 Cited
US9141859B2 Cited
US20100142803A1 Cited
US20100306193A1 Cited
US8595781B2 Cited
US20120095817A1 Cited
US8730397B1 Cited
US8605956B2 Cited
US8452778B1 Cited
US20120230545A1 Cited
US8489589B2 Cited
US20110211736A1 Cited
US20110243450A1 Cited
US9025866B2 Cited
US8565536B2 Cited
US9161007B2 Cited
US20110249953A1 Cited
US20110263946A1 Cited
US8619150B2 Cited
US20110292288A1 Cited
US20110292232A1 Cited
US8824747B2 Cited
US8687941B2 Cited
US8494231B2 Cited
US20120233159A1 Cited
WO2012138828A2 Cited
US20120294540A1 Cited
US20120301014A1 Cited
US20140135966A1 Cited
US20130051756A1 Cited
WO2013063736A1 Cited
US20130114942A1 Cited
US20130142418A1 Cited
US20130148003A1 Cited
US20130262462A1 Cited
US20150142744A1 Cited
US20130322765A1 Cited
US20140201126A1 Cited
US20140270350A1 Cited
US20140369422A1 Cited
US20150082349A1 Cited
US20150110349A1 Cited
US20150296228A1 Cited
US20160034748A1 Cited
US20160034786A1 Cited
US20160117749A1 Cited

USPTO Annuity & Maintenance Windows

Annuity Stage Window Opens Payment Deadline Surcharge Period Until Estimated Fee Status
3.5 Years (Year 4) 2020-05-08 2020-11-07 2021-05-09 $2000.00 Window Closed / Lapsed
7.5 Years (Year 8) 2024-05-08 2024-11-07 2025-05-09 $3760.00 Window Closed / Lapsed
11.5 Years (Year 12) 2028-05-08 2028-11-07 2029-05-09 $7700.00 Pending Window

Assigned Categories & Tags

No categories assigned to this patent yet.

Assign Category Tag