Patent EP0672988A2 successfully refreshed from Google Patents.
EP0672988A2
Expired / Lapsed
EP
⏱️ 0.0 years remaining
Language identification process using coded language words
Assignee: International Business Machines Corp
•
Filing: 1995-03-14
•
Granted: —
•
Expires: 2015-03-14
Abstract
Abstract Provides a process which identifies the language or genre of a stored or transmitted document. The process uses a plurality of Word Frequency Tables (WFTs) respectively associated with languages/genre of interest. Each WFT contains a relatively few of the most common words of one of the languages of interest. Each word code in a WFT has an associated normalized frequency of occurrence value (NFO); use of NFOs increases the language/genre detection ability of the process. A plurality of respective accumulators are associated with the plurality of WFTs. All accumulators are set to zero before identification processing starts. The language/genre identification process receives a sequence of words from an inputted document, and compares each received word to all of the words in all WFTs. Whenever a received word is found in any WFT, the process adds the word's associated NFO to a current total in the associated accumulator. In this manner, totals in all accumulators build up into language discriminating values after a number of words are read from the document. Processing stops when either the end of the document is reached or when a predetermined number of words are received; and then the language/genre associated with the accumulator containing the largest total is the identified language.
Inventors
- 👤 Michael John Martino
- 👤 Robert Charles Paulsen, Jr.
Bibliographic Details
Application Number
—
Jurisdiction
EP
Data Source
google_patents
Last Synchronized
2026-11-08 05:30
CPC Classifications
G; G06; G06F; G06F40/00; G06F40/20; G06F40/205; G06F40/216; G06F40/263
Claims Analysis
Total Claims: 8
Claim 1
Independent Claim
218 words
A machine process for identifying a human language or genre used in a computer coded document from text in the document, comprising the steps of reading a sequence of coded words from the document, comparing each coded word obtained by the reading step to words in a plurality of Word Frequency Tables (WFTs) respectively associated with languages of interest, each WFT containing most frequently used words (also called common words) statistisized from a respective language, and each word in a WFT having an associated frequency field for containing a frequency of occurrence value for the associated word, associating a respective Language Word Accumulator (WFA) with each WFT, and resetting each LA to a predetermined value prior to reading a document, outputting a frequency of occurrence value stored in the frequency field associated with each common word comparing-equal to a read word, inputting the outputted frequency of occurrence value to the WFA associated with the WFT for which the comparing-equal occurred, adding the frequency of occurrence value to a current sum contained in the WFA to generate an accumulated amount, detecting the WFA having a largest accumulated amount of all WFAs when the reading step ends, and identifying the language of the document being read as the language associated with the WFA detected to have the largest accumulated value.
Claim 2
Dependent Claim
103 words
A machine process for identifying a human language or genre used in a computer coded document from text in the document as defined in claim 1, further comprising the steps of finding the largest frequency of occurrence value for any word in each WFT, normalizing the frequency of occurrence value associated with each word in each WFT by dividing the frequency of occurrence value by the largest frequency of occurrence value found in the WFT to generate a normalized-frequency of occurrence (NFO) for the word, and replacing each frequency of occurrence value with the NFO determined for the associated word in the WFT.
Claim 3
Dependent Claim
93 words
A machine process for identifying a human language or genre used in a computer coded document from text in the document as defined in claim 2, further comprising the steps of inserting special words in any WFT which are not found in the other WFTs, and replacing the NFO value associated with the special word with a value larger than the largest NFO value in the WFT to give the special word greater weight in an accumulated sum for a document containing the special word than for the common words in the document.
Claim 4
Dependent Claim
97 words
A machine process for identifying a human language or genre used in a computer coded document from text in the document as defined in claim 1, further comprising the steps of inserting special words in any WFT which are not found in the other WFTs, and replacing the frequency of occurrence value associated with the special word with a value larger than the largest frequency of occurrence value in the WFT to give the special word greater weight in an accumulated sum for a document containing the special word than for the common words in the document.
Claim 5
Dependent Claim
114 words
A machine process for identifying a human language or genre used in a computer coded document from text in the document as defined in claim 3, further comprising the steps of finding special words in each WFT by comparing each common word in the WFT with words in each of the other WFTs, detecting a special word whenever any common word does not have any equally coded common word in any other WFT, and placing a larger value in the frequency field of the special word than any NFO value found in the frequency field for any common word for which the detecting step detects an equally coded common word in any other WFT.
Claim 6
Dependent Claim
107 words
A machine process for identifying a human language or genre used in a computer coded document from text in the document as defined in claim 1 or claim 2, further comprising the steps of scanning a plurality of documents in each language or genre of interest for which a WFT is to be generated, counting each of a plurality of words in the documents being scanned by the scanning step to establish a frequency of occurrence value for each word, and retaining the words having a frequency of occurrence value exceeding a threshold value in the WFT for the language or genre of the documents being scanned.
Claim 7
Dependent Claim
91 words
A machine process for identifying a human language or genre used in a computer coded document from text in the document as defined in claim 1 or claim 2, further comprising the steps of setting a threshold count for a minimum number of words which must be read from a coded document before a language or genre identification can be made by the method, counting the words as the reading step is reading a document, and performing or validating the detecting and identifying steps only if the threshold count is exceeded.
Claim 8
Dependent Claim
117 words
A machine process for identifying a human language or genre used in a computer coded document from text in the document as defined in claim 1 or claim 2, further comprising the steps of setting a threshold count for a minimum number of words which must be read from a coded document before a language or genre identification can be made by the method, monitoring the difference (or factor) between the largest-valued and next-largest-valued WFA, counting the words as the reading step is reading a document, identifying the language of the document when the difference (or factor) exceeds a predetermined value, and performing or validating the detecting and identifying steps only if the threshold count is exceeded.
References Cited (Backward Citations)
No citation records indexed.
USPTO Annuity & Maintenance Windows
Maintenance schedule is calculated for granted US utility patents.