US8494231B2
Active / In Force
US
⏱️ 4.0 years remaining
Face recognition in video content
Assignee: Microsoft Technology Licensing LLC
•
Filing: 2010-11-01
•
Granted: 2013-07-23
•
Expires: 2030-11-01
Abstract
Abstract The subject disclosure relates to face recognition in video. Face detection data in frames of input data are used to generate face galleries, which are labeled and used in recognizing faces throughout the video. Metadata that associates the video frame and the face are generated and maintained for subsequent identification. Faces other than those found by face detection may be found by face tracking, in which facial landmarks found by the face detection are used to track a face over previous and/or subsequent video frames. Once generated, the maintained metadata may be accessed to efficiently determine the identity of a person corresponding to a viewer-selected face.
Inventors
- 👤 Florin O. Folta
- 👤 Yaming He
- 👤 King Wei Hor
- 👤 Minesh G. Shilotri
- 👤 Stacey Spears
- 👤 Chuang Gu
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:00
CPC Classifications
G; G06; G06V; G06V40/00; G06V40/10; G06V40/16; G06V40/172; G06V40/173; G06V40/179
Claims Analysis
Total Claims: 20
Claim 1
Independent Claim
122 words
In a computing environment, a method performed at least in part on at least one processor, comprising, receiving face detection data corresponding to a face detected in an input video frame, building face galleries, including grouping faces detected in input video frames into candidate groups based upon similarity data, filtering at least some faces from a candidate group, adding remaining faces to one of the face galleries, and labeling each face gallery with the face identification data, matching the face detection data against face identification data maintained in a face gallery among a plurality of face galleries to recognize the face in the input video frame, and generating metadata that associates the video frame and the face with the face identification data.
Claim 2
Dependent Claim
122 words
In a computing environment, a method performed at least in part on at least one processor, comprising, receiving face detection data corresponding to a face detected in an input video frame, building face galleries, including grouping faces detected in input video frames into candidate groups based upon similarity data, filtering at least some faces from a candidate group, adding remaining faces to one of the face galleries, and labeling each face gallery with the face identification data, matching the face detection data against face identification data maintained in a face gallery among a plurality of face galleries to recognize the face in the input video frame, and generating metadata that associates the video frame and the face with the face identification data.
Claim 3
Dependent Claim
42 words
The method of claim 1 further comprising, tracking the face that is detected in the input video frame over one or more subsequent frames, and generating metadata that associates each of the one or more subsequent frames with the face identification data.
Claim 4
Dependent Claim
42 words
The method of claim 1 further comprising, tracking the face that is detected in the input video frame over one or more previous frames, and generating metadata that associates each of the one or more previous frames with the face identification data.
Claim 5
Dependent Claim
22 words
The method of claim 1 further comprising, dividing a candidate group into at least two candidate groups based upon the similarity data.
Claim 6
Dependent Claim
23 words
The method of claim 1 further comprising, combining two or more candidate groups into a single candidate group based upon the similarity data.
Claim 7
Dependent Claim
36 words
The method of claim 1 wherein filtering at least some of the faces from the candidate group comprises discarding a candidate group if any face in that group appears to not be of the same person.
Claim 8
Dependent Claim
53 words
The method of claim 1 further comprising, receiving a request to identify a viewer-selected face, the request associated with a video frame number, accessing the metadata to determine whether face identification data exists for that viewer-selected face, and if so, returning information corresponding to the face identification data in response to the request.
Claim 9
Dependent Claim
141 words
In a computing environment, a system comprising, a face recognition pipeline that recognizes faces from input video, including a face grouping module configured to group faces into groups by similarity based upon face detection data provided by a face detection module data, the grouping module further configured to provide face galleries corresponding to the groups, including information that identifies each person associated with a face in a face gallery, the face recognition pipeline further comprising a face recognition mechanism that matches faces in the input video with faces in the face galleries to output information corresponding to recognized faces in the input video, wherein the input video comprises a full set of episodes, wherein the face grouping module groups faces using a lesser subset of the episodes, and wherein the face recognition mechanism matches faces for the full set of episodes.
Claim 10
Dependent Claim
141 words
In a computing environment, a system comprising, a face recognition pipeline that recognizes faces from input video, including a face grouping module configured to group faces into groups by similarity based upon face detection data provided by a face detection module data, the grouping module further configured to provide face galleries corresponding to the groups, including information that identifies each person associated with a face in a face gallery, the face recognition pipeline further comprising a face recognition mechanism that matches faces in the input video with faces in the face galleries to output information corresponding to recognized faces in the input video, wherein the input video comprises a full set of episodes, wherein the face grouping module groups faces using a lesser subset of the episodes, and wherein the face recognition mechanism matches faces for the full set of episodes.
Claim 11
Dependent Claim
35 words
The system of claim 8 further comprising a face tracking module configured to track a face in one or more frames adjacent a frame for which the face detection module data provided face detection data.
Claim 12
Dependent Claim
29 words
The system of claim 9 wherein the face tracking module is configured to estimate and track facial landmarks in the one or more adjacent frames to track the face.
Claim 13
Dependent Claim
31 words
The system of claim 9 wherein the face recognition mechanism matches faces tracked by the face tracking module to output at least some of the information corresponding to the recognized faces.
Claim 14
Dependent Claim
38 words
The system of claim 8 wherein the information corresponding to recognized faces in the input video comprises metadata from which a person in a show or movie at a given frame and location in that frame is identifiable.
Claim 15
Dependent Claim
32 words
The system of claim 12 further comprising a mechanism configured to access the metadata to identify a person given a show or movie, a given frame and a location in that frame.
Claim 16
Dependent Claim
33 words
The system of claim 8 wherein the face detection module provides the data to the face grouping module for a sampling of less than the set of available frames of the input video.
Claim 17
Dependent Claim
116 words
One or more computer-readable storage media having computer-executable instructions, which when executed perform steps, comprising: receiving face detection data corresponding to a face detected in an input video frame, wherein the face detection data corresponds to similarity data; tracking the face in one or more adjacent video frames based on at least some of the face detection data to acquire a tracked face; and utilizing the similarity data to determine whether the tracked face matches a threshold level of similarity; and maintaining the tracked face in a face gallery in a single candidate group, wherein the tracked face from each video frame among the one or more adjacent video frames is maintained in the face gallery.
Claim 18
Dependent Claim
116 words
One or more computer-readable storage media having computer-executable instructions, which when executed perform steps, comprising: receiving face detection data corresponding to a face detected in an input video frame, wherein the face detection data corresponds to similarity data; tracking the face in one or more adjacent video frames based on at least some of the face detection data to acquire a tracked face; and utilizing the similarity data to determine whether the tracked face matches a threshold level of similarity; and maintaining the tracked face in a face gallery in a single candidate group, wherein the tracked face from each video frame among the one or more adjacent video frames is maintained in the face gallery.
Claim 19
Dependent Claim
62 words
The one or more computer-readable storage media of claim 15 wherein upon detecting an undetected face in the one or more adjacent video frames, a reverse order tracking of the one or more adjacent video frames is conducted, wherein the reverse order tracking comprises analyzing all video frames that appear prior to the video frame in which the undetected face is located.
Claim 20
Dependent Claim
32 words
The one or more computer-readable storage media of claim 15 having further computer-executable instructions comprising, generating metadata that associates face identification data with the input video frame and each adjacent video frame.
References Cited (Backward Citations)
US20060008152A1
Cited
US20060187305A1
Cited
US20040264780A1
Cited
WO2005096213A1
Cited
US20080267461A1
Cited
US20090141949A1
Cited
US20100310134A1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2016-07-22 | 2017-01-21 | 2017-07-23 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2020-07-22 | 2021-01-21 | 2021-07-23 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2024-07-22 | 2025-01-21 | 2025-07-23 | $7700.00 | Window Closed / Lapsed |