Patent US7246314B2 successfully refreshed from Google Patents.
US7246314B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Methods and apparatuses for interactive similarity searching, retrieval and browsing of video
Assignee: Fujifilm Business Innovation Corp
•
Filing: 2004-06-03
•
Granted: 2007-07-17
•
Expires: 2024-06-03
Abstract
Abstract Methods for interactive selecting video queries consisting of training images from a video for a video similarity search and for displaying the results of the similarity search are disclosed. The user selects a time interval in the video as a query definition of training images for training an image class statistical model. Time intervals can be as short as one frame or consist of disjoint segments or shots. A statistical model of the image class defined by the training images is calculated on-the-fly from feature vectors extracted from transforms of the training images. For each frame in the video, a feature vector is extracted from the transform of the frame, and a similarity measure is calculated using the feature vector and the image class statistical model. The similarity measure is derived from the likelihood of a Gaussian model producing the frame. The similarity is then presented graphically, which allows the time structure of the video to be visualized and browsed. Similarity can be rapidly calculated for other video files as well, which enables content-based retrieval by example. A content-aware video browser featuring interactive similarity measurement is presented. A method for selecting training segments involves mouse click-and-drag operations over a time bar representing the duration of the video; similarity results are displayed as shades in the time bar. Another method involves selecting periodic frames of the video as endpoints for the training segment.
Inventors
- 👤 Jonathan T. Foote
- 👤 Andreas Girgensohn
- 👤 Lynn Wilcox
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 11:54
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/70; G06F16/78; G06F16/783; G06F16/7847; G06F16/7864; G06F16/73; G06F16/732; G06F16/7328; G06F16/738; G06V; G06V20/00; G06V20/40; G06V20/48
Claims Analysis
Total Claims: 42
Claim 1
Independent Claim
104 words
A method of performing a similarity search of a video, the method comprising the steps of interactively defining by a user on a video browser a training video segment from the video, the training video segment comprising one or more video segments; obtaining a reduced feature vectors for each video frames of the video; training a statistical model using reduced feature vectors from the training video segment; computing a similarity score for each video frame using the frame's reduced feature vector and the statistical model; and displaying one or more similarity scores and one or more frames of the video to the video browser.
Claim 2
Independent Claim
104 words
A method of performing a similarity search of a video, the method comprising the steps of interactively defining by a user on a video browser a training video segment from the video, the training video segment comprising one or more video segments; obtaining a reduced feature vectors for each video frames of the video; training a statistical model using reduced feature vectors from the training video segment; computing a similarity score for each video frame using the frame's reduced feature vector and the statistical model; and displaying one or more similarity scores and one or more frames of the video to the video browser.
Claim 3
Dependent Claim
22 words
A method as in claim 1 , further comprising: segmenting the video into similar and non-similar segments based upon the similarity scores.
Claim 4
Dependent Claim
32 words
A method as in claim 2 , wherein segmenting the video into similar and non-similar segments based upon the similarity scores comprises comparing the similarity scores to an interactively defined similarity threshold.
Claim 5
Dependent Claim
35 words
A method as in claim 1 , wherein obtaining a reduced feature vectors for each video frames comprises retrieving each of the reduced feature vectors from a precomputed feature vector database corresponding to the video.
Claim 6
Dependent Claim
40 words
A method as in claim 4 , wherein training a statistical model comprises retrieving each of the reduced feature vectors from a pretrained statistical model using the training video segment's reduced feature vectors retrieved from the precomputed feature vector database.
Claim 7
Dependent Claim
34 words
A method as in claim 1 , wherein obtaining a reduced feature vectors for each video frames comprises transforming frames of the video using one of a Discrete Cosine Transform and a Hadamard transform.
Claim 8
Dependent Claim
34 words
A method as in claim 1 , wherein obtaining a reduced feature vector for each video frame includes obtaining features representing chromatic components of the frame and features representing luminance components of the frame.
Claim 9
Dependent Claim
29 words
A method as in claim 7 , wherein obtaining a reduced feature vector for each video frame includes obtaining fewer features representing chromatic components than features representing luminance components.
Claim 10
Dependent Claim
41 words
A method as in claim 1 , wherein obtaining a reduced feature vector for each video frame includes obtaining features representing red components of the frame, features representing green components of the frame, and features representing blue components of the frame.
Claim 11
Dependent Claim
25 words
A method as in claim 1 , wherein training a statistical model comprises training a Gaussian model. vectors retrieved from the precomputed feature vector database.
Claim 12
Dependent Claim
48 words
A method as in claim 1 , wherein displaying comprises displaying a timeline bar of similarity scores, the similarity scores depicted as bars within the timeline bar such that the darker the similarity score bar, the more similar the corresponding video frame is to the training video segment.
Claim 13
Dependent Claim
34 words
A method as in claim 1 , wherein displaying comprises providing interactive indexing to allow the user to jump to similar video frames in the video by skipping nonsimilar video frames in the video.
Claim 14
Dependent Claim
41 words
A method as in claim 1 , wherein obtaining a reduced feature vector for each video frame comprises reducing a number of less significant coefficients in each feature vector by using one of truncation, principal component analysis, and linear discriminant analysis.
Claim 15
Dependent Claim
112 words
A machine readable medium having instructions stored thereon that when executed by a processor are configured to cause the processor to perform the steps of: interactively defining by a user on a video browser a training video segment from the video, the training video segment comprising one or more video segments; obtaining reduced feature vectors for each video frames of the video; training a statistical model using reduced feature vectors from the training video segment; computing a similarity score for each video frame using the frame's reduced feature vector and the statistical model; and displaying one or more similarity scores and one or more frames of the video to the video browser.
Claim 16
Dependent Claim
112 words
A machine readable medium having instructions stored thereon that when executed by a processor are configured to cause the processor to perform the steps of: interactively defining by a user on a video browser a training video segment from the video, the training video segment comprising one or more video segments; obtaining reduced feature vectors for each video frames of the video; training a statistical model using reduced feature vectors from the training video segment; computing a similarity score for each video frame using the frame's reduced feature vector and the statistical model; and displaying one or more similarity scores and one or more frames of the video to the video browser.
Claim 17
Dependent Claim
39 words
A machine readable medium as in claim 14 , further comprising instructions that when executed by the processor cause the processor to perform the step of: segmenting the video into similar and non-similar segments based upon the similarity scores.
Claim 18
Dependent Claim
34 words
A machine readable medium as in claim 15 , wherein segmenting the video into similar and non-similar segments based upon the similarity scores comprises comparing the similarity scores to an interactively defined similarity threshold.
Claim 19
Dependent Claim
45 words
A machine readable medium as in claim 14 , wherein obtaining a reduced feature vectors for each video frames comprises retrieving each of the reduced feature vectors comprises retrieving each of the reduced feature vectors from a precomputed feature vector database corresponding to the video.
Claim 20
Dependent Claim
42 words
A machine readable medium as in claim 17 , wherein training a statistical model comprises retrieving each of the reduced feature vectors from a pretrained statistical model using the training video segment's reduced feature vectors retrieved from the precomputed feature vector database.
Claim 21
Dependent Claim
36 words
A machine readable medium as in claim 14 , wherein obtaining a reduced feature vectors for each video frames comprises transforming frames of the video using one of a Discrete Cosine Transform and a Hadamard transform.
Claim 22
Dependent Claim
36 words
A machine readable medium as in claim 14 , wherein obtaining a reduced feature vector for each video frame includes obtaining features representing chromatic components of the frame and features representing luminance components of the frame.
Claim 23
Dependent Claim
31 words
A machine readable medium as in claim 20 , wherein obtaining a reduced feature vector for each video frame includes obtaining fewer features representing chromatic components than features representing luminance components.
Claim 24
Dependent Claim
43 words
A machine readable medium as in claim 14 , wherein obtaining a reduced feature vector for each video frame includes obtaining features representing red components of the frame, features representing green components of the frame, and features representing blue components of the frame.
Claim 25
Dependent Claim
19 words
A machine readable medium as in claim 14 , wherein training a statistical model comprises training a Gaussian model.
Claim 26
Dependent Claim
50 words
A machine readable medium as in claim 14 , wherein displaying comprises displaying a timeline bar of similarity scores, the similarity scores depicted as bars within the timeline bar such that the darker the similarity score bar, the more similar the corresponding video frame is to the training video segment.
Claim 27
Dependent Claim
36 words
A machine readable medium as in claim 14 , wherein displaying comprises providing interactive indexing to allow the user to jump to similar video frames in the video by skipping non-similar video frames in the video.
Claim 28
Dependent Claim
43 words
A machine readable medium as in claim 14 , wherein obtaining a reduced feature vector for each video frame comprises reducing a number of less significant coefficients in each feature vector by using one of truncation, principal component analysis, and linear discriminant analysis.
Claim 29
Independent Claim
117 words
A system for performing a similarity search of a video, the system comprising: a video browser that enables a user to interactively define a query video training segment from the video, the training video segment comprising one or more video segments; and a model generation component configured to: obtain a reduced feature vectors from each video frames of the video; and train a statistical model using reduced feature vectors from the query video training segment; and a score computation component configured to: compute a similarity score for each video frame using the frame's reduced feature vector and the statistical model; and display the similarity scores and one or more frames of the video to the video browser.
Claim 30
Independent Claim
117 words
A system for performing a similarity search of a video, the system comprising: a video browser that enables a user to interactively define a query video training segment from the video, the training video segment comprising one or more video segments; and a model generation component configured to: obtain a reduced feature vectors from each video frames of the video; and train a statistical model using reduced feature vectors from the query video training segment; and a score computation component configured to: compute a similarity score for each video frame using the frame's reduced feature vector and the statistical model; and display the similarity scores and one or more frames of the video to the video browser.
Claim 31
Dependent Claim
29 words
A system as in claim 27 , wherein the score computation component is further configured to: segment the video into similar and non-similar segments based upon the similarity scores.
Claim 32
Dependent Claim
37 words
A system as in claim 28 , wherein, when the video is segmented into similar and non-similar segments based upon the similarity scores, a comparison is performed of the similarity scores to an interactively defined similarity threshold.
Claim 33
Dependent Claim
40 words
A system as in claim 27 , wherein the model generation component is further configured to: retrieves each of the reduced feature vectors from a precomputed feature vector database corresponding to the video, in order to obtain reduced feature vectors.
Claim 34
Dependent Claim
42 words
A system as in claim 30 , wherein the trained statistical model comprises a retrieval of each of the reduced feature vectors from a pretrained statistical model using the training video segment's reduced feature vectors retrieved from the precomputed feature vector database.
Claim 35
Dependent Claim
39 words
A system as in claim 27 , wherein the model generation component is further configured to: transforms frames of the video using one of a Discrete Cosine Transform and a Hadamard transform, in order to obtain reduced feature vectors.
Claim 36
Dependent Claim
39 words
A system as in claim 27 , wherein the model generation component is further configured to: obtain features representing chromatic components of the frame and features representing luminance components of the frame, in order to obtain reduced feature vectors.
Claim 37
Dependent Claim
34 words
A system as in claim 33 , wherein the model generation component is further configured to: obtain fewer features representing chromatic components than features representing luminance components, in order to obtain reduced feature vectors.
Claim 38
Dependent Claim
46 words
A system as in claim 27 , wherein the model generation component is further configured to: obtain features representing red components of the frame, features representing green components of the frame, and features representing blue components of the frame, in order to obtain reduced feature vectors.
Claim 39
Dependent Claim
17 words
A system as in claim 27 , wherein a trained statistical model comprises a trained Gaussian model.
Claim 40
Dependent Claim
52 words
A system as in claim 27 , wherein the display of similarity scores comprises displaying a timeline bar of similarity scores, the similarity scores depicted as bars within the timeline bar such that die darker the similarity score bar, the more similar the corresponding video frame is to the training video segment.
Claim 41
Dependent Claim
38 words
A system as in claim 27 , wherein the display of similarity scores comprises providing interactive indexing to allow the user to jump to similar video frames in the video by skipping non-similar video frames in the video.
Claim 42
Dependent Claim
43 words
A system as in claim 27 , wherein an obtained reduced feature vector for each video flame comprises a reduction of a number of less significant coefficients in each feature vector by using one of truncation, principal component analysis, and linear discriminant analysis.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2010-07-16 | 2011-01-15 | 2011-07-17 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2014-07-16 | 2015-01-15 | 2015-07-17 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2018-07-16 | 2019-01-15 | 2019-07-17 | $7700.00 | Window Closed / Lapsed |