Method and apparatus for automatic language determination of European script documents

Claim 2

An automatic language determining apparatus for determining a language of a text portion of document having a known script-type, comprising: input means for inputting a digital data signal representative of the text portion of the document, the text portion being in an unknown language; word token generating means for converting the digital data signal to a plurality of word tokens, each word token comprising at least one of a limited number of abstract-coded character classes, each abstract-coded character class representing a group of characters of the known script-type; feature determining means for determining at least one word token occurrence value of word tokens occurring within the plurality of word tokens and corresponding to at least one predetermined word token; and language determining means for determining the language of the text portion of the document based on the at least one word token occurrence value.

Claim 3

The automatic language determining means of claim 1, wherein the at least one predetermined word token comprises at least one abstract-coded character class and is selected from a set of word tokens comprising at least the word tokens "AAx", "Ax" and "Aix".

Claim 4

An automatic language determining apparatus for determining a language of a text portion of a document having a known script-type, comprising: input means for inputting a digital data signal representative of the text portion of the document, the text portion being in an unknown language; word token generating means for converting the digital data signal to a plurality of word tokens, each word token comprising at least one of a limited number of abstract-coded character classes, each abstract-coded character class representing a group of characters of the known script-type of the document; feature determining means for determining at least one word token occurrence value of word tokens occurring within the plurality of word tokens and corresponding to at least one predetermined word token; and language determining means for determining the language of the document based on the at least one word token occurrence value, said language determining means comprising: means for determining frequency-of-occurrence-rates for word tokens within the text portion for each at least one predetermined word token from the word token occurrence value; means for converting the determined frequency rates to a point in a coordinate space; and means for determining a closest one of a plurality of predetermined regions within the coordinate space to the point, each predetermined region having a corresponding language, the language corresponding to the closest region being determined as the language of the text portion.

Claim 5

An automatic language determining apparatus for determining a language of a text portion of a document having a known script-type, comprising: input means for inputting a digital data signal representative of the text portion of the document, the text portion being in an unknown language; word token generating means for converting the digital data signal to a plurality of word tokens, each word token comprising at least one of a limited number of abstract-coded character classes, each abstract-coded character class representing a group of characters of the known script-type of the document; feature determining means for determining at least one word token occurrence value of word tokens occurring within the plurality of word tokens and corresponding to at least one predetermined word token; and language determining means for determining the language of the document based on the at least one word token occurrence value, said language determining means comprising: means for determining frequency-of-occurrence-rates for word tokens within the text portion for each at least one predetermined word token from the word token occurrence value; means for converting the determined frequency rates to a point in a coordinate space; and means for determining a closest one of a plurality of predetermined regions within the coordinate space to the point, each predetermined region having a corresponding language, the language corresponding to the closest region being determined as the language of the text portion.

Claim 6

The automatic language determining apparatus of claim 3, wherein each predetermined region corresponds to a different one of a plurality of languages.

Claim 7

The automatic language determining apparatus of claim 4, wherein the plurality of languages comprise English, French and German.

Claim 10

The method of automatic language determination of claim 6, wherein the language determining step comprises the steps of: dividing the coordinate space into at least one predetermined region, each predetermined region having a corresponding language, determining which at least one predetermined region is closest to the point; and selecting the corresponding language of the closest predetermined region as the language of the text portion.

Claim 11

The method of claim 7, wherein each predetermined region comprises a plurality of points, each of the plurality of points corresponding to a document in the corresponding language, and wherein the step of determining the closest predetermined region comprises the steps of: determining a centroid of each predetermined region based on locations of the plurality of points; determining, for each predetermined region, a centroid distance between the point and the centroid; and selecting the predetermined region having a smallest centroid distance as the closest predetermined region.