Patent US8311344B2 successfully refreshed from Google Patents.
US8311344B2
Active / In Force
US
⏱️ 2.3 years remaining
Systems and methods for semantically classifying shots in video
Assignee: DigitalSmiths Inc
•
Filing: 2009-02-17
•
Granted: 2012-11-13
•
Expires: 2029-02-17
Abstract
Abstract The present disclosure relates to systems and methods for classifying videos based on video content. For a given video file including a plurality of frames, a subset of frames is extracted for processing. Frames that are too dark, blurry, or otherwise poor classification candidates are discarded from the subset. Generally, material classification scores that describe type of material content likely included in each frame are calculated for the remaining frames in the subset. The material classification scores are used to generate material arrangement vectors that represent the spatial arrangement of material content in each frame. The material arrangement vectors are subsequently classified to generate a scene classification score vector for each frame. The scene classification results are averaged (or otherwise processed) across all frames in the subset to associate the video file with one or more predefined scene categories related to overall types of scene content of the video file.
Inventors
- 👤 Heather Dunlop
- 👤 Matthew G. BERRY
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:14
CPC Classifications
G; G06; G06V; G06V20/00; G06V20/40; G06V20/10
Claims Analysis
Total Claims: 50
Claim 1
Independent Claim
307 words
A system for classifying videos based on video content, comprising: a processor; one or more software modules including one or more classifiers; the one or more software modules operable on the processor to perform the steps of: receiving a video file, the video file including a plurality of frames; extracting a subset of frames from the video file; if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset; determining whether each frame in the extracted subset includes content associated with a general content category; for each frame in the extracted subset that includes content associated with the general content category: dividing the frame into one or more segments based on image content in each segment; generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content; assigning the material classification score vector for each segment to each respective pixel in the segment; generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel; generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category; determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and associating the video file with the one or more predefined scene categories based on the representative scene classification score vector.
Claim 2
Independent Claim
307 words
A system for classifying videos based on video content, comprising: a processor; one or more software modules including one or more classifiers; the one or more software modules operable on the processor to perform the steps of: receiving a video file, the video file including a plurality of frames; extracting a subset of frames from the video file; if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset; determining whether each frame in the extracted subset includes content associated with a general content category; for each frame in the extracted subset that includes content associated with the general content category: dividing the frame into one or more segments based on image content in each segment; generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content; assigning the material classification score vector for each segment to each respective pixel in the segment; generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel; generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category; determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and associating the video file with the one or more predefined scene categories based on the representative scene classification score vector.
Claim 3
Dependent Claim
44 words
The system of claim 1 , wherein the one or more software modules are selected from the group comprising: an intensity classification module, an indoor/outdoor classification module, an outdoor classification module, a segmentation module, a material arrangement module, and a video file classification module.
Claim 4
Independent Claim
319 words
A system for classifying videos based on video content, comprising: a processor; one or more software modules including one or more classifiers; a computer program product that includes a computer-readable medium that is usable by the processor, the medium having stored thereon a sequence of instructions associated with the one or more software modules that when executed by the processor causes the execution of the steps of: receiving a video file, the video file including a plurality of frames; extracting a subset of frames from the video file; if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset; determining whether each frame in the extracted subset includes content associated with a general content category; for each frame in the extracted subset that includes content associated with the general content category, dividing the frame into one or more segments based on image content in each segment; generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content; assigning the material classification score vector for each segment to each respective pixel in the segment; generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel; classifying the material arrangement vector via the one or more scene classifiers to generate the scene classification score vector for the frame; determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and associating the video file with the one or more predefined scene categories based on the representative scene classification score vector.
Claim 5
Independent Claim
319 words
A system for classifying videos based on video content, comprising: a processor; one or more software modules including one or more classifiers; a computer program product that includes a computer-readable medium that is usable by the processor, the medium having stored thereon a sequence of instructions associated with the one or more software modules that when executed by the processor causes the execution of the steps of: receiving a video file, the video file including a plurality of frames; extracting a subset of frames from the video file; if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset; determining whether each frame in the extracted subset includes content associated with a general content category; for each frame in the extracted subset that includes content associated with the general content category, dividing the frame into one or more segments based on image content in each segment; generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content; assigning the material classification score vector for each segment to each respective pixel in the segment; generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel; classifying the material arrangement vector via the one or more scene classifiers to generate the scene classification score vector for the frame; determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and associating the video file with the one or more predefined scene categories based on the representative scene classification score vector.
Claim 6
Dependent Claim
22 words
The system of claim 3 , further comprising the step of combining adjacent segments based on similar content properties of the segments.
Claim 7
Dependent Claim
34 words
The system of claim 3 , wherein the one or more predefined material content types are selected from the group comprising: building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.
Claim 8
Dependent Claim
21 words
The system of claim 3 , wherein the material arrangement vector represents the spatial arrangement of material content in the frame.
Claim 9
Dependent Claim
19 words
The system of claim 1 , wherein a dark frame comprises a frame shot in low or no light.
Claim 10
Dependent Claim
27 words
The system of claim 1 , further comprising the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame.
Claim 11
Dependent Claim
103 words
The system of claim 8 , wherein the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame further comprises the steps of: for each frame, dividing the frame into a plurality of grid cells; calculating an intensity value for each pixel in each cell; calculating an average intensity value across all pixels in each cell; concatenating the average intensity values for each cell in the frame to form an intensity feature vector for the frame; and classifying the intensity feature vector via an intensity classifier to determine if the frame comprises a dark frame.
Claim 12
Dependent Claim
84 words
The system of claim 1 , wherein the step of determining whether each frame in the extracted subset includes content associated with the general content category further comprises the steps of: extracting a plurality of features from each frame; generating a feature vector for each frame, wherein each feature vector includes the extracted features for the respective frame; and classifying the feature vector for each frame via a general category classifier to determine whether each frame includes content associated with the general content category.
Claim 13
Dependent Claim
14 words
The system of claim 1 , wherein the general content category comprises outdoor content.
Claim 14
Dependent Claim
35 words
The system of claim 1 , wherein the one or more predefined scene categories are selected from the group comprising: coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.
Claim 15
Dependent Claim
33 words
The system of claim 1 , wherein each of the one or more scene classification scores represents the probability that a frame includes content associated with each of the respective predefined scene categories.
Claim 16
Dependent Claim
32 words
The system of claim 1 , wherein each of the one or more scene classification scores comprises a value greater than or equal to zero and less than or equal to one.
Claim 17
Dependent Claim
24 words
The system of claim 1 , wherein the representative scene classification score vector comprises a statistical property of the generated scene classification score vectors.
Claim 18
Dependent Claim
21 words
The system of claim 15 , wherein the statistical property is selected from the group comprising: average, median, maximum, and minimum.
Claim 19
Independent Claim
262 words
A system for classifying videos based on video content, comprising: a processor; one or more software modules including one or more classifiers; the one or more software modules operable on the processor to perform the steps of: receiving a video file, the video file including a plurality of frames; extracting a subset of frames from the video file; if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset; determining whether each frame in the extracted subset includes content associated with a general content category; for each frame in the extracted subset that includes content associated with the general content category: generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category; determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; associating the video file with the one or more predefined scene categories based on the representative scene classification score vector; identifying representative scene classification scores in the representative scene classification score vector that exceed a predetermined threshold value; and for each representative scene classification score that exceeds the threshold value, associating the video file with the one or more predefined scene categories associated with the classification scores that exceeded the threshold value.
Claim 20
Independent Claim
262 words
A system for classifying videos based on video content, comprising: a processor; one or more software modules including one or more classifiers; the one or more software modules operable on the processor to perform the steps of: receiving a video file, the video file including a plurality of frames; extracting a subset of frames from the video file; if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset; determining whether each frame in the extracted subset includes content associated with a general content category; for each frame in the extracted subset that includes content associated with the general content category: generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category; determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; associating the video file with the one or more predefined scene categories based on the representative scene classification score vector; identifying representative scene classification scores in the representative scene classification score vector that exceed a predetermined threshold value; and for each representative scene classification score that exceeds the threshold value, associating the video file with the one or more predefined scene categories associated with the classification scores that exceeded the threshold value.
Claim 21
Dependent Claim
15 words
The system of claim 1 , wherein the video file comprises a shot of video.
Claim 22
Dependent Claim
27 words
The system of claim 1 , further comprising the step of generating a report including the one or more predefined scene categories associated with the video file.
Claim 23
Dependent Claim
28 words
The system of claim 1 , wherein the one or more scene categories associated with the video file are used for indexing and retrieval of the video file.
Claim 24
Independent Claim
306 words
A method for classifying videos based on video content, comprising the steps of: receiving a video file, the video file including a plurality of frames; extracting a subset of frames from the video file; if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset; determining whether each frame in the extracted subset includes content associated with a general content category; for each frame in the extracted subset that includes content associated with the general content category: dividing the frame into one or more segments based on image content in each segment; generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content; assigning the material classification score vector for each segment to each respective pixel in the segment; generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel; classifying the material arrangement vector via the one or more scene classifiers to generate the scene classification score vector for the frame; generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category; determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and labeling the video file according to the one or more predefined scene categories based on the representative scene classification score vector.
Claim 25
Independent Claim
306 words
A method for classifying videos based on video content, comprising the steps of: receiving a video file, the video file including a plurality of frames; extracting a subset of frames from the video file; if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset; determining whether each frame in the extracted subset includes content associated with a general content category; for each frame in the extracted subset that includes content associated with the general content category: dividing the frame into one or more segments based on image content in each segment; generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content; assigning the material classification score vector for each segment to each respective pixel in the segment; generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel; classifying the material arrangement vector via the one or more scene classifiers to generate the scene classification score vector for the frame; generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category; determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; and labeling the video file according to the one or more predefined scene categories based on the representative scene classification score vector.
Claim 26
Dependent Claim
22 words
The method of claim 21 , further comprising the step of combining adjacent segments based on similar content properties of the segments.
Claim 27
Dependent Claim
34 words
The method of claim 21 , wherein the one or more predefined material content types are selected from the group comprising: building, grass, person, road/sidewalk, rock, sand/gravel/soil, sky/clouds, snow/ice, trees/plants, vehicle, water, and miscellaneous.
Claim 28
Dependent Claim
21 words
The method of claim 21 , wherein the material arrangement vector represents the spatial arrangement of material content in the frame.
Claim 29
Dependent Claim
19 words
The method of claim 21 , wherein a dark frame comprises a frame shot in low or no light.
Claim 30
Dependent Claim
27 words
The method of claim 21 , further comprising the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame.
Claim 31
Dependent Claim
103 words
The method of claim 26 , wherein the step of determining whether any of the frames in the extracted subset of frames comprises a dark frame further comprises the steps of: for each frame, dividing the frame into a plurality of grid cells; calculating an intensity value for each pixel in each cell; calculating an average intensity value across all pixels in each cell; concatenating the average intensity values for each cell in the frame to form an intensity feature vector for the frame; and classifying the intensity feature vector via an intensity classifier to determine if the frame comprises a dark frame.
Claim 32
Dependent Claim
84 words
The method of claim 21 , wherein the step of determining whether each frame in the extracted subset includes content associated with the general content category further comprises the steps of: extracting a plurality of features from each frame; generating a feature vector for each frame, wherein each feature vector includes the extracted features for the respective frame; and classifying the feature vector for each frame via a general category classifier to determine whether each frame includes content associated with the general content category.
Claim 33
Dependent Claim
14 words
The method of claim 21 , wherein the general content category comprises outdoor content.
Claim 34
Dependent Claim
35 words
The method of claim 21 , wherein the one or more predefined scene categories are selected from the group comprising: coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.
Claim 35
Dependent Claim
33 words
The method of claim 21 , wherein each of the one or more scene classification scores represents the probability that a frame includes content associated with each of the respective predefined scene categories.
Claim 36
Dependent Claim
32 words
The method of claim 21 , wherein each of the one or more scene classification scores comprises a value greater than or equal to zero and less than or equal to one.
Claim 37
Dependent Claim
24 words
The method of claim 21 , wherein the representative scene classification score vector comprises a statistical property of the generated scene classification score vectors.
Claim 38
Dependent Claim
21 words
The method of claim 33 , wherein the statistical property is selected from the group comprising: average, median, maximum, and minimum.
Claim 39
Independent Claim
236 words
A method for classifying videos based on video content, comprising: receiving a video file, the video file including a plurality of frames; extracting a subset of frames from the video file; if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset; determining whether each frame in the extracted subset includes content associated with a general content category; for each frame in the extracted subset that includes content associated with the general content category: generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category; determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; labeling the video file according to the one or more predefined scene categories based on the representative scene classification score vector; identifying representative scene classification scores in the representative scene classification score vector that exceed a predetermined threshold value; and for each representative scene classification score that exceeds the threshold value, associating the video file with the one or more predefined scene categories associated with the classification scores that exceeded the threshold value.
Claim 40
Independent Claim
236 words
A method for classifying videos based on video content, comprising: receiving a video file, the video file including a plurality of frames; extracting a subset of frames from the video file; if one or more frames in the extracted subset of frames comprises a dark frame, discarding the one or more dark frames from the subset; determining whether each frame in the extracted subset includes content associated with a general content category; for each frame in the extracted subset that includes content associated with the general content category: generating a scene classification score vector for the frame via one or more scene classifiers, the scene classification score vector including one or more scene classification scores associated with one or more predefined scene categories within the general content category; determining a representative scene classification score vector for the video file based on the generated scene classification score vectors for each extracted frame in the subset that includes content associated with the general content category; labeling the video file according to the one or more predefined scene categories based on the representative scene classification score vector; identifying representative scene classification scores in the representative scene classification score vector that exceed a predetermined threshold value; and for each representative scene classification score that exceeds the threshold value, associating the video file with the one or more predefined scene categories associated with the classification scores that exceeded the threshold value.
Claim 41
Dependent Claim
15 words
The method of claim 35 , wherein the video file comprises a shot of video.
Claim 42
Dependent Claim
20 words
The method of claim 35 , further comprising the step of generating a report based on the labeled video file.
Claim 43
Dependent Claim
24 words
The method of claim 35 , wherein the labeled one or more scene categories are used for indexing and retrieval of the video file.
Claim 44
Independent Claim
193 words
A method for labeling videos based on video content, comprising the steps of: receiving a video file, wherein the video file includes a plurality of frames; extracting a set of frames from the plurality of frames in the video file; for each frame in the extracted set of frames, calculating a probability that the frame includes content associated with a predefined scene category; dividing the frame into one or more segments based on image content in each segment; generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content; assigning the material classification score vector for each segment to each respective pixel in the segment; generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel; determining a representative probability for the set of frames based on the calculated probabilities for each frame; and if the representative probability exceeds a predetermined threshold, associating the scene category with the video file.
Claim 45
Independent Claim
193 words
A method for labeling videos based on video content, comprising the steps of: receiving a video file, wherein the video file includes a plurality of frames; extracting a set of frames from the plurality of frames in the video file; for each frame in the extracted set of frames, calculating a probability that the frame includes content associated with a predefined scene category; dividing the frame into one or more segments based on image content in each segment; generating a material classification score vector for each segment, each material classification score vector including one or more material values associated with one or more predefined material content types, wherein each material value represents a probability that the respective segment includes that type of material content; assigning the material classification score vector for each segment to each respective pixel in the segment; generating a material arrangement vector for the frame based on the material classification score vectors assigned to each pixel; determining a representative probability for the set of frames based on the calculated probabilities for each frame; and if the representative probability exceeds a predetermined threshold, associating the scene category with the video file.
Claim 46
Dependent Claim
21 words
The method of claim 39 , wherein the representative probability comprises a statistical property of the calculated probabilities for each frame.
Claim 47
Dependent Claim
21 words
The method of claim 40 , wherein the statistical property is selected from the group comprising: average, median, maximum, and minimum.
Claim 48
Dependent Claim
26 words
The method of claim 39 , further comprising the step of indexing the video file according to the associated scene category for search and retrieval purposes.
Claim 49
Dependent Claim
31 words
The method of claim 39 , wherein the scene category is selected from the list comprising: coast/beach, desert, forest, grassland, highway, indoor, lake/river, mountainous, open water, outdoor, sky, snow, and urban.
Claim 50
Dependent Claim
18 words
The method of claim 39 , wherein the calculated probability for each frame is calculated via a classifier.
References Cited (Backward Citations)
US5267351A
Cited
US6249280B1
Cited
US5537528A
Cited
US5630121A
Cited
US5650941A
Cited
US5655117A
Cited
US6154601A
Cited
US5903892A
Cited
US6192183B1
Cited
US5832495A
Cited
US5956729A
Cited
US6285361B1
Cited
US6172675B1
Cited
US6032156A
Cited
US6195458B1
Cited
US6134380A
Cited
US6327420B1
Cited
US6330004B1
Cited
US6292620B1
Cited
US6895407B2
Cited
US20020089646A1
Cited
US20020108112A1
Cited
US20040019524A1
Cited
US20060036948A1
Cited
US20070286489A1
Cited
US20080162561A1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2015-11-13 | 2016-05-14 | 2016-11-13 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2019-11-13 | 2020-05-14 | 2020-11-13 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2023-11-13 | 2024-05-14 | 2024-11-13 | $7700.00 | Window Closed / Lapsed |