US5913185A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Determining a natural language shift in a computer document
Assignee: International Business Machines Corp
•
Filing: 1996-12-20
•
Granted: 1999-06-15
•
Expires: 2016-12-20
Abstract
Abstract Language shift points in a computer document written in a plurality of natural languages are determined. An interval is defined on and moved through a text document in a computer memory, the interval contains a portion of the text in the document. As the interval is moved through the document for each position of the interval, a probability that the text in the interval is written in each of a plurality of candidate languages is determined for the position. For the first position of the interval, generally the beginning of the document, a first candidate language is classified as the current language if it has the highest probability of all the candidate languages within the interval. A language shift point in the document is identified where the relative probability of a second candidate language is higher than the current language at a new position of the interval. At this point, the second candidate language is classified as the current language in the document after the language shift point. The process continues to identify other language shift points in the document.
Inventors
- 👤 Michael John Martino
- 👤 Robert Charles Paulsen, Jr.
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 20:37
CPC Classifications
G; G06; G06F; G06F40/00; G06F40/20; G06F40/263; G06F40/205; G06F40/216
Claims Analysis
Total Claims: 31
Claim 1
Independent Claim
170 words
A method for detecting language shift points in a computer document written in a plurality of natural languages, comprising the steps of: moving an interval through a text document in a computer memory, the interval containing a plurality of words in the document; for each position of the interval, determining a probability that text in the interval is written in each of a plurality of candidate languages according to a respective number of matches of words in the interval with words in each of a plurality of word lists of a few common words selected from each respective candidate language; for a first position of the interval, classifying a first candidate language having the highest probability as the current language within the interval; finding a language shift point in the document where the probability that a second candidate language is higher than the current language for a new position of the interval; and classifying the second candidate language as the current language in the document after the language shift point.
Claim 2
Independent Claim
170 words
A method for detecting language shift points in a computer document written in a plurality of natural languages, comprising the steps of: moving an interval through a text document in a computer memory, the interval containing a plurality of words in the document; for each position of the interval, determining a probability that text in the interval is written in each of a plurality of candidate languages according to a respective number of matches of words in the interval with words in each of a plurality of word lists of a few common words selected from each respective candidate language; for a first position of the interval, classifying a first candidate language having the highest probability as the current language within the interval; finding a language shift point in the document where the probability that a second candidate language is higher than the current language for a new position of the interval; and classifying the second candidate language as the current language in the document after the language shift point.
Claim 3
Dependent Claim
77 words
The method as recited in claim 1 wherein the determining step comprises the steps of: comparing words in the interval with word lists comprised of common words in each of the candidate languages; counting a number of words in the interval which match the common words in each respective word list; and determining the relative probability of each candidate language based on a cumulative count of words in the interval which match for each respective candidate language.
Claim 4
Dependent Claim
22 words
The method as recited in claim 2 wherein the count for each word is weighted by a factor in the cumulative count.
Claim 5
Dependent Claim
28 words
The method as recited in claim 2 wherein the common words selected for the word lists are chosen so that strong aliases between candidate languages are not included.
Claim 6
Dependent Claim
65 words
The method as recited in claim 4 wherein the candidate languages to be compared to the document are selected before the language shift determination from all the candidates languages for which word lists are stored, and wherein the strong aliases between the selected candidate languages are removed from master word lists for the selected candidate languages composed of common words in each selected candidate language.
Claim 7
Dependent Claim
16 words
The method as recited in claim 2 wherein partial words are used in the comparing step.
Claim 8
Dependent Claim
43 words
The method as recited in claim 1 further comprising the steps of: searching the document for syntax cues proximate to the found language shift point; and responsive to finding syntax cues, shifting the language shift point if indicated by the found syntax cues.
Claim 9
Dependent Claim
50 words
The method as recited in claim 7, wherein the method operates in batch mode, so that a plurality of language shift points are found in a first pass through the document and syntax cues proximate to the language shift points are searched for in a second pass through the document.
Claim 10
Dependent Claim
60 words
The method as recited in claim 1 further comprising the steps of: performing a language related operation on the document according to a first dictionary for the first candidate language prior to the language change point; and performing the language related operation on the document according to a second dictionary for the second candidate language after the language change point.
Claim 11
Independent Claim
162 words
A system including processor and memory for detecting language shift points in a computer document written in a plurality of natural languages, comprising: interval defining means for moving an interval through a computer document in the memory, the interval containing a plurality of words in the document; language comparing means for determining for each position of the interval a probability that text in the interval is written in each of a plurality of candidate languages according to a respective number of matches of words in the interval with words in each of a plurality of word lists of a few common words selected from each respective candidate language; language determining means for determining for each position of the interval the candidate language having the highest relative probability within the interval; language shift determining means for finding language shift points in the document according to the positions of the interval where the candidate language having the highest relative probability changes from prior positions.
Claim 12
Independent Claim
162 words
A system including processor and memory for detecting language shift points in a computer document written in a plurality of natural languages, comprising: interval defining means for moving an interval through a computer document in the memory, the interval containing a plurality of words in the document; language comparing means for determining for each position of the interval a probability that text in the interval is written in each of a plurality of candidate languages according to a respective number of matches of words in the interval with words in each of a plurality of word lists of a few common words selected from each respective candidate language; language determining means for determining for each position of the interval the candidate language having the highest relative probability within the interval; language shift determining means for finding language shift points in the document according to the positions of the interval where the candidate language having the highest relative probability changes from prior positions.
Claim 13
Dependent Claim
79 words
The system as recited in claim 10 wherein the language comparing means comprises: means for comparing words in the interval with word lists comprised of common words in each of the candidate languages; means for determining a number of words in the interval which match the common words in each respective word list; and means for determining the probability of each candidate language based on the number of words in the interval which match for each respective candidate language.
Claim 14
Dependent Claim
27 words
The system as recited in claim 11 wherein the common words in the word lists are chosen do not include strong aliases between candidate languages being compared.
Claim 15
Dependent Claim
57 words
The system as recited in claim 12 wherein the candidate languages to be compared to the document are selected from all the candidate languages for which word lists are stored, and wherein the strong aliases between the selected candidate languages are removed from master word lists for the selected candidate languages before language shift determination takes place.
Claim 16
Dependent Claim
43 words
The system as recited in claim 11 wherein the words in the word lists are stored as bits in N×N bit tables, where each bit represents a given letter pair at a particular place in a word in one of the candidate languages.
Claim 17
Dependent Claim
29 words
The system as recited in claim 14 wherein the bit tables are arranged in tableaus, each tableau for storing words of a particular length for a given candidate language.
Claim 18
Dependent Claim
14 words
The system as recited in claim 11 wherein the interval contains non adjacent words.
Claim 19
Dependent Claim
23 words
The system as recited in claim 10 further comprising dictionaries for performing a language related operation on the document for the candidate languages.
Claim 20
Dependent Claim
134 words
A computer program product in a computer readable medium for detecting language shift points in a computer document written in a plurality of natural languages, comprising: means for moving an interval through a computer document in a computer memory, the interval containing a plurality of words in the document; means for determining a probability that text in the interval is written in each of a plurality of candidate languages within the interval according to a respective number of matches of words in the interval with words in each of a plurality of word lists of a few common words selected from each respective candidate language; and means for finding language shift points in the document according to changes in the relative probabilities of the candidate language as the interval changes positions in the document.
Claim 21
Dependent Claim
134 words
A computer program product in a computer readable medium for detecting language shift points in a computer document written in a plurality of natural languages, comprising: means for moving an interval through a computer document in a computer memory, the interval containing a plurality of words in the document; means for determining a probability that text in the interval is written in each of a plurality of candidate languages within the interval according to a respective number of matches of words in the interval with words in each of a plurality of word lists of a few common words selected from each respective candidate language; and means for finding language shift points in the document according to changes in the relative probabilities of the candidate language as the interval changes positions in the document.
Claim 22
Dependent Claim
75 words
The product as recited in claim 18 further comprising: means for comparing words in the interval with word lists comprised of common words in each of the candidate languages; means for determining a number of words in the interval which match the common words in each respective word list; and means for determining the probability of each candidate language based on the number of words in the interval which match for each respective candidate language.
Claim 23
Dependent Claim
27 words
The product as recited in claim 19 wherein the common words in the word lists are chosen do not include strong aliases between candidate languages being compared.
Claim 24
Dependent Claim
69 words
The product as recited in claim 20 further comprising: means for selecting the candidate languages to be compared to the document from a plurality of candidate languages for which word lists are stored; a master word list for each candidate language; and means for removing the strong aliases between the selected candidate languages from the master word lists for the selected candidate languages before language shift determination takes place.
Claim 25
Dependent Claim
23 words
The product as recited in claim 18 further comprising dictionaries for performing a language related operation on the document for the candidate languages.
Claim 26
Dependent Claim
43 words
The product as recited in claim 18 further comprising: means for searching the document for syntax cues proximate to the language shift points; and means responsive to finding syntax cues for shifting the language shift point if indicated by the found syntax cues.
Claim 27
Independent Claim
124 words
A method for detecting language shift points in a computer document written in a plurality of natural languages, comprising the steps of: selecting successive pluralities of words located at successive locations in the document; for each plurality of words, recognizing no words or one or more words, but not all words, as being members of each of a respective candidate language; for each plurality of words, classifying the plurality of words as being written in a respective language according to a greatest number of words in the respective plurality of words being recognized as members of the respective language; and finding language shift points in the document according to where the respective languages of successive pluralities of words change according to the classifying step.
Claim 28
Independent Claim
124 words
A method for detecting language shift points in a computer document written in a plurality of natural languages, comprising the steps of: selecting successive pluralities of words located at successive locations in the document; for each plurality of words, recognizing no words or one or more words, but not all words, as being members of each of a respective candidate language; for each plurality of words, classifying the plurality of words as being written in a respective language according to a greatest number of words in the respective plurality of words being recognized as members of the respective language; and finding language shift points in the document according to where the respective languages of successive pluralities of words change according to the classifying step.
Claim 29
Dependent Claim
31 words
The method as recited in claim 24, wherein each successive plurality of words shares a predetermined number of words with a predecessor plurality of words and a successor plurality of words.
Claim 30
Independent Claim
123 words
A method for detecting language shift points in a computer document written in a plurality of natural languages, comprising the steps of: selecting successive pluralities of words located at successive locations in the document, wherein each plurality of words includes a first number of words of which a second number of words is shared by a next plurality of words; for each plurality of words, classifying the plurality of words as being written in a respective language according to a greatest number of words in the respective plurality of words being recognized as members of the respective language; and finding language shift points in the document according to where the respective languages of successive pluralities of words change according to the classifying step.
Claim 31
Independent Claim
123 words
A method for detecting language shift points in a computer document written in a plurality of natural languages, comprising the steps of: selecting successive pluralities of words located at successive locations in the document, wherein each plurality of words includes a first number of words of which a second number of words is shared by a next plurality of words; for each plurality of words, classifying the plurality of words as being written in a respective language according to a greatest number of words in the respective plurality of words being recognized as members of the respective language; and finding language shift points in the document according to where the respective languages of successive pluralities of words change according to the classifying step.
References Cited (Backward Citations)
US3988715A
Cited
US4610025A
Cited
US4829580A
Cited
US4773009A
Cited
US4890230A
Cited
US5371673A
Cited
US4930077A
Cited
WO1991002347A1
Cited
US5062143A
Cited
US5182708A
Cited
US5251131A
Cited
US5392419A
Cited
US5371807A
Cited
US5608624A
Cited
US5418951A
Cited
US5548507A
Cited
US5715446A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2002-06-14 | 2002-12-14 | 2003-06-15 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2006-06-14 | 2006-12-14 | 2007-06-15 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2010-06-14 | 2010-12-14 | 2011-06-15 | $7700.00 | Window Closed / Lapsed |