Patent US6167369A successfully refreshed from Google Patents.
US6167369A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Automatic language identification using both N-gram and word information
Assignee: Google LLC
•
Filing: 1998-12-23
•
Granted: 2000-12-26
•
Expires: 2018-12-23
Abstract
Abstract The predominant language of a sample text is automatically identified using probability data that include N-gram probability data for at least one language and word probability data for at least one language. The N-gram probability data of a language indicate, for each N-gram, the probability that it occurs if the language is predominant. Similarly, the word probability data of a language indicate, for each word, the probability that it occurs if the language is predominant. The probability data are used to automatically obtain sample probability data for at least two languages. The sample probability data include N-gram probability information for at least one language and word probability information for at least one language. The sample probability data are used to automatically obtain language identifying data identifying the language whose sample probability data indicate the highest probability. The N-grams can be trigrams, while the words can be short words of no more than five characters. Some languages can have both trigram and word probabilities, while some can have only trigram probabilities.
Inventors
- 👤 Bruno M. Schulze
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 18:45
CPC Classifications
G; G06; G06F; G06F40/00; G06F40/20; G06F40/263; G06F40/279; G06F40/284
Claims Analysis
Total Claims: 17
Claim 1
Independent Claim
292 words
A method of automatically identifying a natural language that is likely to be the predominant language of a sample text, the method including: (A) using text data defining the sample text and probability data for each of a set of natural languages to automatically obtain, for each of at least two languages in the set, sample probability data indicating a probability that the sample text would occur in the language; and (B) using the sample probability data to automatically obtain language identifying data; the language identifying data identifying the language whose sample probability data indicate the highest probability; the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data; the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text; the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text; in which (A) includes: (A1) automatically obtaining sample probability data for each of a third subset of the languages; for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data; for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data.
Claim 2
Independent Claim
292 words
A method of automatically identifying a natural language that is likely to be the predominant language of a sample text, the method including: (A) using text data defining the sample text and probability data for each of a set of natural languages to automatically obtain, for each of at least two languages in the set, sample probability data indicating a probability that the sample text would occur in the language; and (B) using the sample probability data to automatically obtain language identifying data; the language identifying data identifying the language whose sample probability data indicate the highest probability; the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data; the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text; the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text; in which (A) includes: (A1) automatically obtaining sample probability data for each of a third subset of the languages; for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data; for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data.
Claim 3
Dependent Claim
21 words
The method of claim 1 in which the third subset includes every language that is in the first or second subsets.
Claim 4
Dependent Claim
17 words
The method of claim 1 in which each N-gram in the set of N-grams is a trigram.
Claim 5
Dependent Claim
19 words
The method of claim 1 in which each word in the set of words includes five characters or less.
Claim 6
Dependent Claim
59 words
The method of claim 1 in which at least one language in the third subset is in both the first and second subsets and in which the sample probability data obtained in (A1) for each language in both the first and second subsets includes information both from the language's N-gram probability data and from the language's word probability data.
Claim 7
Dependent Claim
21 words
The method of claim 5 in which every language in the third subset is in both the first and second subsets.
Claim 8
Dependent Claim
28 words
The method of claim 5 in which the third subset further includes at least one language that is in the first subset but not in the second subset.
Claim 9
Dependent Claim
83 words
The method of claim 5 in which, for each language in the third subset that is in both the first and second subsets, the N-gram probability data and the word probability data include, for each N-gram in the set of N-grams or for each word in the set of words, a probability value and in which (A1) includes: (A1a) using the probability values for the N-grams and words that occur in the sample text to obtain the sample probability data for the language.
Claim 10
Dependent Claim
139 words
The method of claim 8 in which each probability value is logarithmic and in which (A1a) includes, for each language in the third subset that is in both the first and second subsets: adding probability values for the N-grams that occur in the sample text to obtain a total N-gram probability value and adding probability values for words that occur in the sample text to obtain a total word probability value, the probability value for each N-gram and word that occurs in the sample text but is not in the set of N-grams or in the set of words being a constant probability value indicating a low probability; and combining the total N-gram probability value and the total word probability value to obtain a sample probability value; the sample probability value being the sample probability data for the language.
Claim 11
Dependent Claim
25 words
The method of claim 9 in which the act of combining the total probability values comprises adding the total probability values and dividing by two.
Claim 12
Independent Claim
301 words
A system for automatically identifying a natural language that is likely to be the predominant language of a sample text, the system comprising: text data defining the sample text; probability data for each of a set of natural languages; the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data; the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text; the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text; and a processor connected for accessing the text data and the probability data; the processor operating to: use the text data and the probability data to automatically obtain, for each of a third subset that includes at least two of the languages in the set, sample probability data indicating a probability that the sample text would occur in the language; for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data; for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data; and use the sample probability data to automatically obtain language identifying data; the language identifying data identifying the language whose sample probability data indicate the highest probability.
Claim 13
Independent Claim
301 words
A system for automatically identifying a natural language that is likely to be the predominant language of a sample text, the system comprising: text data defining the sample text; probability data for each of a set of natural languages; the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data; the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text; the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text; and a processor connected for accessing the text data and the probability data; the processor operating to: use the text data and the probability data to automatically obtain, for each of a third subset that includes at least two of the languages in the set, sample probability data indicating a probability that the sample text would occur in the language; for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data; for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data; and use the sample probability data to automatically obtain language identifying data; the language identifying data identifying the language whose sample probability data indicate the highest probability.
Claim 14
Dependent Claim
356 words
An article of manufacture for use in a system for automatically identifying a natural language that is likely to be the predominant language of a sample text; the system including: text data defining the sample text; a storage medium access device; and a processor connected for receiving data accessed on a storage medium by the storage medium access device and for accessing the text data; the article of manufacture comprising: a storage medium; probability data for each of a set of natural languages, stored by the storage medium; the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data; the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text; the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text; and instruction data stored by the storage medium; the instruction data indicating instructions the processor can execute; the processor, in executing the instructions: using the text data and the probability data to automatically obtain, for each of a third subset that includes at least two of the languages in the set, sample probability data indicating a probability that the sample text would occur in the language; for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data; for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data; and using the sample probability data to automatically obtain language identifying data; the language identifying data identifying the language whose sample probability data indicate the highest probability.
Claim 15
Dependent Claim
356 words
An article of manufacture for use in a system for automatically identifying a natural language that is likely to be the predominant language of a sample text; the system including: text data defining the sample text; a storage medium access device; and a processor connected for receiving data accessed on a storage medium by the storage medium access device and for accessing the text data; the article of manufacture comprising: a storage medium; probability data for each of a set of natural languages, stored by the storage medium; the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data; the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text; the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text; and instruction data stored by the storage medium; the instruction data indicating instructions the processor can execute; the processor, in executing the instructions: using the text data and the probability data to automatically obtain, for each of a third subset that includes at least two of the languages in the set, sample probability data indicating a probability that the sample text would occur in the language; for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data; for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data; and using the sample probability data to automatically obtain language identifying data; the language identifying data identifying the language whose sample probability data indicate the highest probability.
Claim 16
Independent Claim
363 words
A method of operating a first machine to transfer data to a second machine over a network, the second machine including a memory and a processor connected for accessing the memory and for receiving text data defining a sample text; the method comprising: establishing a connection between the first and second machines over the network; and operating the first machine to transfer probability data for each of a set of natural languages and instruction data to the memory of the second machine; the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data; the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text; the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text; the instruction data indicating instructions the processor can execute; the processor, in executing the instructions, automatically identifying a natural language that is likely to be the predominant language of the sample text; the processor operating to: use the text data and the probability data to automatically obtain, for each of a third subset that includes at least two of the languages in the set, sample probability data indicating a probability that the sample text would occur in the language; for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data; for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data; and use the sample probability data to automatically obtain language identifying data; the language identifying data identifying the language whose sample probability data indicate the highest probability.
Claim 17
Independent Claim
363 words
A method of operating a first machine to transfer data to a second machine over a network, the second machine including a memory and a processor connected for accessing the memory and for receiving text data defining a sample text; the method comprising: establishing a connection between the first and second machines over the network; and operating the first machine to transfer probability data for each of a set of natural languages and instruction data to the memory of the second machine; the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data; the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text; the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text; the instruction data indicating instructions the processor can execute; the processor, in executing the instructions, automatically identifying a natural language that is likely to be the predominant language of the sample text; the processor operating to: use the text data and the probability data to automatically obtain, for each of a third subset that includes at least two of the languages in the set, sample probability data indicating a probability that the sample text would occur in the language; for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data; for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data; and use the sample probability data to automatically obtain language identifying data; the language identifying data identifying the language whose sample probability data indicate the highest probability.
References Cited (Backward Citations)
US4610025A
Cited
US4829580A
Cited
US4773009A
Cited
US4930077A
Cited
US5062143A
Cited
US5182708A
Cited
US5251131A
Cited
US5392419A
Cited
US5371807A
Cited
US5418951A
Cited
US5377280A
Cited
US5548507A
Cited
US5913185A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2003-12-26 | 2004-06-26 | 2004-12-26 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2007-12-26 | 2008-06-26 | 2008-12-26 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2011-12-26 | 2012-06-26 | 2012-12-26 | $7700.00 | Window Closed / Lapsed |