US7756334B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method, medium, and apparatus for person-based photo clustering in digital photo album, and person-based digital photo albuming method, medium, and apparatus
Assignee: Samsung Electronics Co Ltd
•
Filing: 2005-10-12
•
Granted: 2010-07-13
•
Expires: 2025-10-12
Abstract
Abstract A person-based digital photo clustering method, medium, and apparatus and a person-based digital photo albuming method, medium, and apparatus. The clustering method may include detecting a face area from photo data, extracting predetermined content-based face feature value information from the detected face area, obtaining person-based photo clustering hint information, and performing person-based clustering by applying photo clustering hint information to the extracted content-based face feature value information. The person-based digital photo albuming method may include generating photo description information for identifying at least a photo, generating albuming tool description information supporting person-based photo clustering and including at least a parameter for person-based photo clustering, performing photo albuming including person-based clustering by using at least photo description information and albuming tool description information, generating predetermined photo group description information from photo albumed data, and generating predetermined photo album information by using photo description information and photo group description information.
Inventors
- 👤 Sangkyun Kim
- 👤 Jiyeun Kim
- 👤 Youngsu Moon
- 👤 Yongman Ro
- 👤 Seungji Yang
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 09:47
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/50; G06F16/55; G06T; G06T7/00; G06F16/58; G06F16/583; G06F16/5838; G06F16/51; G06F16/5854; G06F16/5862; G06V; G06V20/00; G06V20/30; G06V40/00; G06V40/10; G06V40/16; G06V40/172
Claims Analysis
Total Claims: 62
Claim 1
Independent Claim
114 words
A person-based digital photo clustering method, comprising: detecting a face area from photo data; extracting content-based face feature value information from the detected face area; obtaining person-based photo clustering hint information; and performing, through controlling at least one processor, person-based clustering by applying the person-based photo clustering hint information to the extracted content-based face feature value information, wherein the performing of person-based clustering comprises: performing clustering based on a situation in which a photo is taken, based on the extracted content-based face feature value information, to obtain a situation-based cluster; and performing person-based clustering in the situation-based cluster, by applying the person-based photo clustering hint information to the extracted content-based face feature value information.
Claim 2
Dependent Claim
114 words
A person-based digital photo clustering method, comprising: detecting a face area from photo data; extracting content-based face feature value information from the detected face area; obtaining person-based photo clustering hint information; and performing, through controlling at least one processor, person-based clustering by applying the person-based photo clustering hint information to the extracted content-based face feature value information, wherein the performing of person-based clustering comprises: performing clustering based on a situation in which a photo is taken, based on the extracted content-based face feature value information, to obtain a situation-based cluster; and performing person-based clustering in the situation-based cluster, by applying the person-based photo clustering hint information to the extracted content-based face feature value information.
Claim 3
Dependent Claim
58 words
The method of claim 1 , wherein the detecting of the face area further comprises normalizing a size of the face area by using detected face area information, to generate a normalized face area, and in the extracting of the content-based face feature value information, the content-based face feature value information is extracted from the normalized face area.
Claim 4
Dependent Claim
75 words
The method of claim 1 , wherein the extracting of the content-based face feature value information from the detected face area includes extracting content-based feature value information including at least one of: an item expressing at least one of color, texture, and shape feature values extracted by using a visual descriptor of a moving picture experts group (MPEG)-7; and an item expressing an audio feature value extracted by using an audio descriptor of the MPEG-7.
Claim 5
Dependent Claim
89 words
The method of claim 1 , wherein the person-based photo clustering hint information further comprises at least one of: a personal appearance hint indicating appearance feature information of a person; a situation hint indicating information on a result of the clustering based on situation information including a place or background of a photo; an audio hint indicating audio information of a user stored together with the photo file; and a camera hint indicating at least one of camera information, photographing information, and photo information included in a photo file.
Claim 6
Dependent Claim
28 words
The method of claim 4 , wherein the personal appearance hint comprises at least one of clothes information, skin tone information, and hair style information of the person.
Claim 7
Dependent Claim
26 words
The method of claim 4 , wherein the situation hint comprises a situation ID expressing an identifier of a situation to which the photo is clustered.
Claim 8
Dependent Claim
98 words
The method of claim 4 , wherein the camera hint comprises at least one of: a taken time expressing information on a time when the photo was taken; a focal length expressing information on a focal length when the photo was taken; an exposure time expressing information on an exposure time when the photo was taken; a zoom info expressing information on a camera zoom when the photo was taken; a flash info expressing information on a flash when the photo was taken; and a GPS info expressing global positioning system (GPS) information when the photo was taken.
Claim 9
Dependent Claim
30 words
The method of claim 4 , wherein the audio hint comprises a speech info expressing speech information extracted from audio information stored together with photo data and recognized keyword information.
Claim 10
Dependent Claim
18 words
The method of claim 1 further comprising: indexing each of multiple person-based clusters if person-based clustering is finished.
Claim 11
Dependent Claim
40 words
The method of claim 9 , wherein in the indexing of each of the multiple person-based clusters, a person-based cluster is compared with predetermined indexed personal photos stored in a database and is stored according to a closest corresponding index.
Claim 12
Dependent Claim
73 words
The method of claim 9 , wherein the predetermined indexed personal photos stored in the database have respective person information, and the person information is formed with person list information comprising at least one of an identifier of the person, the name of the person, importance of the person, relations of the person with other people in a group of the person, and information on a higher-level group to which the person belongs.
Claim 13
Dependent Claim
78 words
The method of claim 11 , wherein the predetermined indexed personal photos stored in the database further comprise information on the group to which the person belong, and the information of the person is formed with person group list information comprising at least one of an identifier of the group, the name of the group, importance of the group, relations of the group with other person groups, and information on a higher-level group to which the group belongs.
Claim 14
Dependent Claim
140 words
A person-based digital photo clustering apparatus comprising: a face area detection unit to detect a face area from photo data; a face feature value extraction unit to extract content-based face feature value information from the detected face area; a clustering hint information obtaining unit to obtain person-based photo clustering hint information; and a clustering unit to perform person-based clustering by applying the person-based photo clustering hint information to the extracted content-based face feature value information, wherein the clustering unit comprises: a situation-based clustering unit to perform clustering based on a situation in which a photo is taken, based on the extracted content-based face feature value information, to obtain a situation-based cluster; and a person-based clustering unit to perform person-based clustering in the situation-based cluster, by applying the person-based photo clustering hint information to the extracted content-based face feature value information.
Claim 15
Dependent Claim
140 words
A person-based digital photo clustering apparatus comprising: a face area detection unit to detect a face area from photo data; a face feature value extraction unit to extract content-based face feature value information from the detected face area; a clustering hint information obtaining unit to obtain person-based photo clustering hint information; and a clustering unit to perform person-based clustering by applying the person-based photo clustering hint information to the extracted content-based face feature value information, wherein the clustering unit comprises: a situation-based clustering unit to perform clustering based on a situation in which a photo is taken, based on the extracted content-based face feature value information, to obtain a situation-based cluster; and a person-based clustering unit to perform person-based clustering in the situation-based cluster, by applying the person-based photo clustering hint information to the extracted content-based face feature value information.
Claim 16
Dependent Claim
52 words
The apparatus of claim 13 , further comprising: a face area normalization unit to normalize a size of the face area by using detected face area information to generate a normalized face area, wherein the face feature value extraction unit extracts the content-based face feature value information from the normalized face area.
Claim 17
Dependent Claim
65 words
The apparatus of claim 13 , wherein extracting of the content-based face feature value information includes extracting content-based feature value information including at least one of: at least one of an item expressing color, texture, and shape feature values extracted by using a visual descriptor of an MPEG-7; and an item expressing an audio feature value extracted by using an audio descriptor of the MPEG-7.
Claim 18
Dependent Claim
119 words
The apparatus of claim 13 , wherein the person-based clustering hint information obtaining unit further comprises at least one of: a personal appearance hint extraction unit extracting a personal appearance hint indicating appearance feature information of a person; a situation hint extraction unit extracting a situation hint indicating information on a result of the clustering based on situation information including a place or background of a photo; an audio hint extraction unit extracting an audio hint indicating audio information of a user stored together with the photo file; and a camera hint extraction unit extracting at least one of a camera hint indicating at least one of camera information, photographing information, and photo information included in a photo file.
Claim 19
Dependent Claim
24 words
The apparatus of claim 13 , further comprising: a person-based indexing unit to index each of multiple person-based clusters if person-based clustering is finished.
Claim 20
Dependent Claim
61 words
The apparatus of claim 17 , further comprising: a group preference hint obtaining unit to obtain a group preference value, wherein by referring to the group preference value obtained by the group preference hint obtaining unit, the person-based indexing unit compares a person-based cluster with predetermined indexed personal photos stored in a database and stored according to a closest corresponding index.
Claim 21
Dependent Claim
129 words
The apparatus of claim 18 , wherein the predetermined indexed personal photos stored in the database have respective person information, and information on a group to which other people and the person belong, and the person information is formed with person list information comprising at least one of an identifier of the person, the name of the person, importance of the person, relations of the person with other people, and information on a higher-level group to which the person belongs, and the person group information is formed with person group list information comprising at least one of an identifier of the group, the name of the group, importance of the group, relations of the group with other groups, and information on a higher-level group to which the group belongs.
Claim 22
Dependent Claim
201 words
A person-based digital photo albuming method, comprising: generating photo description information including a photo identifier for identifying at least a photo; generating albuming tool description information supporting person-based photo clustering and including at least a parameter for person-based photo clustering, wherein the parameter for person-based photo clustering at least includes a person-based photo clustering hint item for improvement of performance of person-based clustering; performing, through controlling at least one processor, photo albuming including the person-based clustering by using at least the photo description information and the albuming tool description information, and wherein the performing of the photo albuming including person-based clustering comprises person-based clustering of digital photo data; generating predetermined photo group description information from photo albumed data from the performing of the photo albuming; and generating predetermined photo album information by using the photo description information and the predetermined photo group description information, wherein the performing of person-based clustering comprises: performing clustering based on a situation in which a photo is taken, based on the extracted content-based face feature value information, to obtain a situation-based cluster; and performing person-based clustering in the situation-based cluster, by applying the person-based photo clustering hint item to the extracted content-based face feature value information.
Claim 23
Dependent Claim
201 words
A person-based digital photo albuming method, comprising: generating photo description information including a photo identifier for identifying at least a photo; generating albuming tool description information supporting person-based photo clustering and including at least a parameter for person-based photo clustering, wherein the parameter for person-based photo clustering at least includes a person-based photo clustering hint item for improvement of performance of person-based clustering; performing, through controlling at least one processor, photo albuming including the person-based clustering by using at least the photo description information and the albuming tool description information, and wherein the performing of the photo albuming including person-based clustering comprises person-based clustering of digital photo data; generating predetermined photo group description information from photo albumed data from the performing of the photo albuming; and generating predetermined photo album information by using the photo description information and the predetermined photo group description information, wherein the performing of person-based clustering comprises: performing clustering based on a situation in which a photo is taken, based on the extracted content-based face feature value information, to obtain a situation-based cluster; and performing person-based clustering in the situation-based cluster, by applying the person-based photo clustering hint item to the extracted content-based face feature value information.
Claim 24
Dependent Claim
28 words
The method of claim 20 , further comprising: receiving an input of photo data from an internal memory device of a digital camera or a portable memory device.
Claim 25
Dependent Claim
60 words
The method of claim 20 , wherein in the generating of the photo description information, existence of EXIF information, such as camera information and photographing information, is confirmed in a photo file of an input photo, and if the EXIF information exists in the photo file, the EXIF information is extracted and expressed according to a predetermined photo description scheme.
Claim 26
Dependent Claim
78 words
The method of claim 22 , wherein the photo description information comprises at least a photo identifier, information on a photographer having taken the photo, photo file information, camera information, photographing information and a content-based feature value, with the content-based feature value including at least one of a visual descriptor generated by using pixel information of the photo, including at least one of color, texture, and shape feature values, and an audio descriptor including a speech feature value.
Claim 27
Dependent Claim
35 words
The method of claim 20 , wherein in the generating of the albuming tool description information, the parameter for person-based photo clustering further comprises: a person-based group list item describing an indexed person-based group list.
Claim 28
Dependent Claim
44 words
The method of claim 24 , wherein the person-based group list item comprises: group list information indicating a list of groups after person-based clustering and indexing are performed; and person list information indicating a list of persons after person-based clustering and indexing are performed.
Claim 29
Dependent Claim
55 words
The method of claim 25 , wherein individual group information on each of multiple groups forming the group list information comprises at least one of an identifier of a group, a name of the group, importance of the group, relationships of the group, and an identifier of a higher-level group to which the group belongs.
Claim 30
Dependent Claim
31 words
The method of claim 26 , wherein the importance of the group is determined by a frequency that a user watches photos included in the group or people in the group.
Claim 31
Dependent Claim
54 words
The method of claim 25 , wherein individual person information forming the person list information comprises at least one of an identifier of the person, a name of the person, relationships of the person with other people, an importance of the person, and an identifier of an upper-level group to which the person belongs.
Claim 32
Dependent Claim
25 words
The method of claim 28 , wherein the importance of the person is determined by a frequency that the user watches photos including the person.
Claim 33
Dependent Claim
109 words
The method of claim 24 , wherein the person-based photo clustering hint item further comprises at least one of: a personal appearance hint indicating appearance feature information of a person; a situation hint indicating information on a result of the situation-clustering based on situation information including a place or background of the photo; a group preference hint indicating information on a group list, selected by the user, which photos to be clustered belong, an audio hint indicating audio information of a user stored together with the photo file; and a camera hint indicating at least one of camera information, photographing information, and photo information included in a photo file.
Claim 34
Dependent Claim
31 words
The method of claim 30 , wherein the situation hint comprises an identifier of a situation to which the photo belongs as the result of the situation-based clustering of the photo.
Claim 35
Dependent Claim
36 words
The method of claim 30 , wherein the personal appearance hint comprises at least one of information on clothes worn by the person, skin tone information of the person, and hair style information of the person.
Claim 36
Dependent Claim
71 words
The method of claim 30 , wherein the camera hint comprises at least one of information on a time when the photo was taken, information on a focal length when the photo was taken, information on an exposure time when the photo was taken, information on a camera zoom when the photo was taken, information on a flash when the photo was taken, and GPS information when the photo was taken.
Claim 37
Dependent Claim
31 words
The method of claim 30 , wherein the group preference hint comprises a group preference value numerically expressing groups selected by the user and a degrees of importance of the groups.
Claim 38
Dependent Claim
31 words
The method of claim 30 , wherein the audio hint comprises a speech info expressing speech information extracted from audio information stored together with the photo file and recognized keyword information.
Claim 39
Dependent Claim
66 words
The method of claim 20 , wherein the person-based photo clustering comprises: detecting a face area from photo data; extracting the content-based face feature value information from the detected face area; obtaining predetermined person-based photo clustering hint information as the person-based photo clustering hint item; and performing person-based clustering by applying the predetermined person-based photo clustering hint information to the extracted content-based face feature value information.
Claim 40
Dependent Claim
58 words
The method of claim 36 , wherein the person-based photo clustering further comprises normalizing a size of the face area by using information of the detected face area, to generate a normalized face area, and in the extracting of the content-based face feature value information, the content-based face feature value information is extracted from the normalized face area.
Claim 41
Dependent Claim
19 words
The method of claim 36 , further comprising: indexing each of multiple person-based clusters if person-based clustering is finished.
Claim 42
Dependent Claim
41 words
The method of claim 38 , wherein in the indexing of each of the multiple person-based clusters, a person-based cluster is compared with predetermined indexed personal photos stored in a database and is stored with according to a closest corresponding index.
Claim 43
Dependent Claim
67 words
The method of claim 39 , wherein the predetermined indexed personal photos stored in the database have respective person information, and the person information is formed with person list information comprising at least one of an identifier of the person, a name of the person, importance of the person, relations of the person with other people, and information on a higher-level group to which the person belongs.
Claim 44
Dependent Claim
80 words
The method of claim 40 , wherein the predetermined indexed personal photos stored in the database further comprise information on a group to which the person and the other people belong, and the person group information is formed with person group list information comprising at least one of an identifier of the group, the name of the group, importance of the group, relationship of the group with other groups, and information on a higher-level group to which the group belongs.
Claim 45
Dependent Claim
236 words
A person-based digital photo albuming apparatus, comprising: a photo description information generation unit to generate photo description information including a photo identifier for identifying at least a photo; an albuming tool description information generation unit to generate albuming tool description information supporting person-based photo clustering and including at least a parameter for person-based photo clustering, wherein the parameter for person-based photo clustering of the albuming tool description information generation unit comprises a person-based photo clustering hint item for improvement of the performance of person-based clustering; an albuming tool to perform photo albuming including the person-based clustering by using at least the photo description information and the albuming tool description information, wherein the albuming tool comprises a person-based photo clustering tool performing person-based clustering of digital photo data; photo group information generation unit to generate predetermined photo group description information based on an output of the albuming tool; and photo album information generation unit to generate predetermined photo album information by using the photo description information and the predetermined photo group description information, wherein the albuming tool comprises: a situation-based clustering unit to perform clustering based on a situation in which a photo is taken, based on extracted content-based face feature value information, to obtain a situation-based cluster; and a person-based clustering unit to perform person-based clustering in the situation-based cluster, by applying the person-based photo clustering hint item to the extracted content-based face feature value information.
Claim 46
Dependent Claim
236 words
A person-based digital photo albuming apparatus, comprising: a photo description information generation unit to generate photo description information including a photo identifier for identifying at least a photo; an albuming tool description information generation unit to generate albuming tool description information supporting person-based photo clustering and including at least a parameter for person-based photo clustering, wherein the parameter for person-based photo clustering of the albuming tool description information generation unit comprises a person-based photo clustering hint item for improvement of the performance of person-based clustering; an albuming tool to perform photo albuming including the person-based clustering by using at least the photo description information and the albuming tool description information, wherein the albuming tool comprises a person-based photo clustering tool performing person-based clustering of digital photo data; photo group information generation unit to generate predetermined photo group description information based on an output of the albuming tool; and photo album information generation unit to generate predetermined photo album information by using the photo description information and the predetermined photo group description information, wherein the albuming tool comprises: a situation-based clustering unit to perform clustering based on a situation in which a photo is taken, based on extracted content-based face feature value information, to obtain a situation-based cluster; and a person-based clustering unit to perform person-based clustering in the situation-based cluster, by applying the person-based photo clustering hint item to the extracted content-based face feature value information.
Claim 47
Dependent Claim
33 words
The apparatus of claim 42 , further comprising: a photo input unit to receive an input of photo data from an internal memory device of a digital camera or a portable memory device.
Claim 48
Dependent Claim
57 words
The apparatus of claim 42 , wherein the photo description information generation unit confirms an existence of EXIF information, such as camera information and photographing information, in a photo file of an input photo, and if the EXIF information exists in the photo file, extracts the EXIF information and expresses according to a predetermined photo description scheme.
Claim 49
Dependent Claim
78 words
The apparatus of claim 44 , wherein the photo description information comprises at least a photo identifier, information on a photographer having taken the photo, photo file information, camera information, photographing information and a content-based feature value, with the content-based feature value including at least one of a visual descriptor generated by using pixel information of the photo, including at least one of color, texture, and shape feature values, and an audio descriptor including a speech feature value.
Claim 50
Dependent Claim
34 words
The apparatus of claim 42 , wherein the parameter for person-based photo clustering of the albuming tool description information generation unit further comprises: a person-based group list item describing an indexed person-based group list.
Claim 51
Dependent Claim
44 words
The apparatus of claim 46 , wherein the person-based group list item comprises: group list information indicating a list of groups after person-based clustering and indexing are performed; and person list information indicating a list of persons after person-based clustering and indexing are performed.
Claim 52
Dependent Claim
55 words
The apparatus of claim 47 , wherein individual group information on each of multiple groups forming the group list information comprises at least one of an identifier of a group, a name of the group, importance of the group, relationships of the group, and an identifier of a higher-level group to which the group belongs.
Claim 53
Dependent Claim
54 words
The apparatus of claim 47 , wherein individual person information forming the person list information comprises at least one of an identifier of the person, a name of the person, relationships of the person with other people, an importance of the person, and an identifier of an upper-level group to which the person belongs.
Claim 54
Dependent Claim
109 words
The apparatus of claim 46 , wherein the person-based photo clustering hint item further comprises at least one of: a personal appearance hint indicating appearance feature information of a person; a situation hint indicating information on a result of the situation-clustering based on situation information including a place or background of the photo; a group preference hint indicating information on a group list, selected by the user, which photos to be clustered belong; an audio hint indicating audio information of a user stored together with the photo file; and a camera hint indicating at least one of camera information, photographing information, and photo information included in a photo file.
Claim 55
Dependent Claim
90 words
The apparatus of claim 42 , wherein the person-based photo clustering tool comprises: a face area detection unit to detect a face area from photo data; a face feature value extraction unit to extract content-based face feature value information from the detected face area; a clustering hint information obtaining unit to obtain predetermined person-based photo clustering hint information as the person-based photo clustering hint item; and a clustering unit to perform person-based clustering by applying the predetermined person-based photo clustering hint information to the extracted content-based face feature value information.
Claim 56
Dependent Claim
62 words
The apparatus of claim 51 , wherein the person-based photo clustering tool further comprises: a face area normalizing unit to normalize a size of the detected face area by using information of the detected face area, to generate a normalized face area, and the face feature value extraction unit to extract the content-based face feature value information from the normalized face area.
Claim 57
Dependent Claim
24 words
The apparatus of claim 52 , further comprising: a person-based indexing unit to index each of multiple person-based clusters if person-based clustering is finished.
Claim 58
Dependent Claim
67 words
The apparatus of claim 53 , wherein the person-based indexing unit further comprises: a group preference hint obtaining unit to obtain a group preference value, and by referring to the group preference value, obtained by the group preference hint obtaining unit, the person-based indexing unit compares a person-based cluster with predetermined indexed personal photos stored in a database and is stored according to a closest corresponding index.
Claim 59
Dependent Claim
130 words
The apparatus of claim 54 , wherein the predetermined indexed personal photos stored in the database have respective person information, and information on a group to which the person and other people belong, and the person information is formed with person list information comprising at least one of an identifier of the person, a name of the person, importance of the person, relationships of the person with the other people, and information on a higher-level group to which the person belongs, and the person group information is formed with person group list information comprising at least one of an identifier of the group, a name of the group, importance of the group, relationships of the group with other groups, and information on a higher-level group to which the group belongs.
Claim 60
Dependent Claim
23 words
A computer readable recording medium comprising computer readable code to control at least one processor to implement the method of claim 1 .
Claim 61
Dependent Claim
23 words
A computer readable recording medium comprising computer readable code to control at lest one processor to implement the method of claim 20 .
Claim 62
Dependent Claim
141 words
The method of claim 1 , further comprising: generating photo description information including a photo identifier for identifying at least a photo; generating albuming tool description information supporting the person-based clustering and including at least a parameter for the person-based clustering, wherein the parameter for the person-based clustering includes a person-based group list item, describing an indexed person-based group list, and the person-based photo clustering hint item for improvement of performance of person-based clustering; performing, using at least one processing device, photo albuming including the person-based clustering of digital photo data by using at least the photo description information and the albuming tool description information; generating predetermined photo group description information from photo albumed data from the performing of the photo albuming; and generating predetermined photo album information by using the photo description information and the predetermined photo group description information.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2013-07-12 | 2014-01-11 | 2014-07-13 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2017-07-12 | 2018-01-11 | 2018-07-13 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2021-07-12 | 2022-01-11 | 2022-07-13 | $7700.00 | Window Closed / Lapsed |