US8488682B2
Active / In Force
US
⏱️ 1.2 years remaining
System and method for extracting text captions from video and generating video summaries
Assignee: Columbia University
•
Filing: 2007-12-19
•
Granted: 2013-07-16
•
Expires: 2027-12-19
Abstract
Abstract Caption boxes which are embedded in video content can be located and the text within the caption boxes decoded. Real time processing is enhanced by locating caption box regions in the compressed video domain and performing pixel based processing operations within the region of the video frame in which a caption box is located. The captions boxes are further refined by identifying word regions within the caption boxes and then applying character and word recognition processing to the identified word regions. Domain based models are used to improve text recognition results. The extracted caption box text can be used to detect events of interest in the video content and a semantic model applied to extract a segment of video of the event of interest.
Inventors
- 👤 Shih-Fu Chang
- 👤 Dongqing Zhang
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-23 23:11
CPC Classifications
G; G11; G11B; G11B27/00; G11B27/02; G11B27/031; G06; G06F; G06F16/00; G06F16/70; G06F16/73; G06F16/738; G06F16/739; G06F16/78; G06F16/783; G06F16/7844; G06F16/7847; G06F16/7857; G06F16/786; G06T; G06T7/00; G06T7/70; G06T7/73; G06T7/75; G06V; G06V20/00; G06V20/60; G06V20/62; G06V20/635; G11B27/034; G11B27/10; G11B27/19; G11B27/28; H; H04; H04N; H04N5/00; H04N5/76; H04N5/91; H04N5/93; H04N7/00; H04N7/025; H04N7/08; H04N7/087; H04N7/088; H04N7/0882; G06V30/00; G06V30/10; H04N21/00; H04N21/40; H04N21/43; H04N21/434; H04N21/47; H04N21/488; H04N21/4884
Claims Analysis
Total Claims: 18
Claim 1
Independent Claim
104 words
A method of decoding a caption box in video content comprising: determining at least one expected location of a caption box in a frame of the video content; determining at least one caption box mask within the expected location; identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location; for at least a portion of the caption frames, identifying word regions within the confines of the expected location; for each word region, identifying text characters within the region; and processing the identified text characters.
Claim 2
Independent Claim
104 words
A method of decoding a caption box in video content comprising: determining at least one expected location of a caption box in a frame of the video content; determining at least one caption box mask within the expected location; identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location; for at least a portion of the caption frames, identifying word regions within the confines of the expected location; for each word region, identifying text characters within the region; and processing the identified text characters.
Claim 3
Dependent Claim
76 words
The method of decoding a caption box according to claim 1 further comprising: comparing the text characters in the word region against a domain specific model to enhance word recognition, determining a word region type for at least one of the identified word regions; wherein the video content is of a sporting event and wherein the word region types are selected from the group consisting of data points that indicate the current state of the event.
Claim 4
Dependent Claim
18 words
The method of decoding a caption box according to claim 2 wherein the data point is a period.
Claim 5
Dependent Claim
18 words
The method of decoding a caption box according to claim 2 wherein the data point is a quarter.
Claim 6
Dependent Claim
18 words
The method of decoding a caption box according to claim 2 wherein the data point is field position.
Claim 7
Dependent Claim
22 words
The method of decoding a caption box according to claim 2 wherein the data point is the number of shots on goal.
Claim 8
Dependent Claim
118 words
A non-transitory computer-readable storage medium storing a program for causing a computer to implement a method of decoding a caption box in video content comprising: determining at least one expected location of a caption box in a frame of the video content; determining at least one caption box mask within the expected location; identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location; for at least a portion of the caption frames, identifying word regions within the confines of the expected location; for each word region, identifying text characters within the region; and processing the identified text characters.
Claim 9
Dependent Claim
118 words
A non-transitory computer-readable storage medium storing a program for causing a computer to implement a method of decoding a caption box in video content comprising: determining at least one expected location of a caption box in a frame of the video content; determining at least one caption box mask within the expected location; identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location; for at least a portion of the caption frames, identifying word regions within the confines of the expected location; for each word region, identifying text characters within the region; and processing the identified text characters.
Claim 10
Dependent Claim
32 words
The non-transitory computer-readable storage medium of claim 7 wherein the method includes the step of comparing the text characters in the word region against a domain specific model to enhance word recognition.
Claim 11
Independent Claim
186 words
A system for decoding a caption box in video content comprising: location means for determining at least one expected location of a caption box in a frame of the video content; determining means, coupled to the location means and receiving the at least one expected location therefrom, for determining at least one caption box mask within the expected location; frame identifying means, coupled to the determining means and location means and receiving the at least one caption box mask and the at least one expected location therefrom, for identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location; word region identifying means, coupled to the frame identifying means and receiving the identified caption frames therefrom, for at least a portion of the caption frames, identifying word regions within the confines of the expected location; text character means, coupled to the word region identifying means and receiving the word regions therefrom, for each word region, identifying text characters within the region; and processing the identified text characters.
Claim 12
Independent Claim
186 words
A system for decoding a caption box in video content comprising: location means for determining at least one expected location of a caption box in a frame of the video content; determining means, coupled to the location means and receiving the at least one expected location therefrom, for determining at least one caption box mask within the expected location; frame identifying means, coupled to the determining means and location means and receiving the at least one caption box mask and the at least one expected location therefrom, for identifying frames in the video content as caption frames if the current frame exhibits substantial correlation to the at least one caption box mask within the expected caption box location; word region identifying means, coupled to the frame identifying means and receiving the identified caption frames therefrom, for at least a portion of the caption frames, identifying word regions within the confines of the expected location; text character means, coupled to the word region identifying means and receiving the word regions therefrom, for each word region, identifying text characters within the region; and processing the identified text characters.
Claim 13
Dependent Claim
42 words
The system of claim 9 wherein the system includes a comparing means, coupled to the text character means and receiving the text characters therefrom, for comparing the text characters in the word region against a domain specific model to enhance word recognition.
Claim 14
Dependent Claim
52 words
The system of claim 9 wherein the location means includes a features means for evaluating motion features of the video frame in the compressed domain, evaluating texture features of the video frame in the compressed domain, and identifying regions having low motion features and high texture features as candidate caption box regions.
Claim 15
Dependent Claim
52 words
The system of claim 10 wherein the location means includes a features means for evaluating motion features of the video frame in the compressed domain, evaluating texture features of the video frame in the compressed domain, and identifying regions having low motion features and high texture features as candidate caption box regions.
Claim 16
Dependent Claim
77 words
The system of claim 9 wherein the system includes a removal means, coupled to the frame identifying means, the determining means and the location means and receiving the identified caption frames, the at least one caption box mask and the at least one expected location therefrom, for evaluating the identified caption frames, within the caption box location, for changes in content and removing caption frames from word region processing which do not exhibit a change in content.
Claim 17
Dependent Claim
48 words
The system of claim 9 wherein the system includes an interval means, coupled to the frame identifying means and receiving the identified caption frames therefrom, for selecting a subset of caption frames based on a predetermined time interval and sending that subset to the word region identifying means.
Claim 18
Dependent Claim
55 words
The system of claim 9 wherein the system includes a number means, coupled to the frame identifying means and receiving the identified caption frames therefrom, for determining a subset of caption frames by selecting caption frames based on a predetermined number of intervening caption frames and sending that subset to the word region identifying means.
References Cited (Backward Citations)
US4649380A
Cited
US4712248A
Cited
US5579444A
Cited
US5664177A
Cited
US5546571A
Cited
US5144685A
Cited
US6058186A
Cited
US5208857A
Cited
US5204706A
Cited
US5191645A
Cited
US5724484A
Cited
US5461679A
Cited
US5557728A
Cited
US5546572A
Cited
US5701510A
Cited
US20060200260A1
Cited
US5428774A
Cited
US5623690A
Cited
US5262856A
Cited
EP0579319B1
Cited
US5809160A
Cited
EP0587329A2
Cited
US5708805A
Cited
US5579471A
Cited
US5751286A
Cited
US5615112A
Cited
US5630121A
Cited
US5408274A
Cited
US5555354A
Cited
US5694945A
Cited
US5794178A
Cited
US5742283A
Cited
US5768578A
Cited
US5555378A
Cited
US5668897A
Cited
US5606655A
Cited
US5465353A
Cited
US5488664A
Cited
US5617119A
Cited
US5493677A
Cited
US5613032A
Cited
US5694334A
Cited
US5642477A
Cited
US5802361A
Cited
US5566089A
Cited
US6246804B1
Cited
US5655117A
Cited
US5805804A
Cited
US5805733A
Cited
US6047374A
Cited
US5530759A
Cited
US5821945A
Cited
US5794242A
Cited
US5572260A
Cited
US6718047B2
Cited
US5684715A
Cited
US5721815A
Cited
US5713021A
Cited
US5758076A
Cited
US5822524A
Cited
US6122403A
Cited
US6356309B1
Cited
US5734893A
Cited
US5969755A
Cited
US5664018A
Cited
US5949885A
Cited
US5987459A
Cited
US5884298A
Cited
US5893095A
Cited
US5767922A
Cited
US5852435A
Cited
US5696964A
Cited
US5870754A
Cited
US5887061A
Cited
US6243419B1
Cited
US6154755A
Cited
US5943422A
Cited
US6282299B1
Cited
US6031914A
Cited
US5848155A
Cited
US5809139A
Cited
US5873080A
Cited
US5734752A
Cited
US6075875A
Cited
US5852823A
Cited
US6526099B1
Cited
US5825892A
Cited
US5915027A
Cited
US6172675B1
Cited
US7733956B1
Cited
US6078664A
Cited
US6453053B1
Cited
US6058205A
Cited
US5790703A
Cited
US6115717A
Cited
US6157746A
Cited
US5930783A
Cited
US6466940B1
Cited
US6778223B2
Cited
US6079566A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2016-07-15 | 2017-01-14 | 2017-07-16 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2020-07-15 | 2021-01-14 | 2021-07-16 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2024-07-15 | 2025-01-14 | 2025-07-16 | $7700.00 | Window Closed / Lapsed |