Method and apparatus for improved tokenization of natural language text

Claim 2

A computerized tokenizer for identifying a token formed of a string of lexical characters found in a stream of digitized natural language text, the computerized tokenizer comprising: parsing means for extracting lexical and non-lexical characters from the stream of digitized text, identifying means coupled with said parsing means for identifying a set of tokens, each token being formed of a string of parsed lexical characters bounded by non-lexical characters, and filtering means coupled with said identifying means for selecting a candidate token from said set of tokens, said candidate token being suitable for linguistic processing beyond the identification of tokens.

Claim 3

A tokenizer according to claim 1, wherein said filtering means further comprises an associative processing element for associating with said candidate token a tag identifying additional linguistic processing for said candidate token.

Claim 4

A tokenizer according to claim 2, wherein said associative processing element further includes a group processing element for associating with a plurality of tokens, as a function of said candidate token, a plurality of tags identifying additional linguistic processing for said plurality of tokens.

Claim 5

A tokenizer according to claim 2, further comprising a modifying processor for modifying said candidate token as a function of said tag associated with said candidate token.

Claim 6

A tokenizer according to claim 4, wherein said modifying processor includes splitting means for splitting said candidate token into multiple tokens.

Claim 7

A tokenizer according to claim 4, wherein said modifying processor includes stripping means for stripping a character from said candidate token.

Claim 8

A tokenizer according to claim 4, w herein said modifying processor includes ignoring means for ignoring a non-lexical character surrounding said candidate token.

Claim 9

A tokenizer according to claim 4, wherein said modifying processor includes merging means for merging said candidate token with another token in the stream of text.

Claim 10

A tokenizer according to claim 1, wherein said filtering selects said candidate token from said set of tokens during a single scan of the parsed stream of text.

Claim 11

A tokenizer according to claim 1, wherein said filtering means further comprises a character analyzer for selecting said candidate token from said set of tokens, said character analyzer including comparing means for comparing a selected character in the parsed stream of text with entries in a character table, and associating means for associating a first tag with a first token located proximal to said selected character, when said selected character has an equivalent entry in the character table.

Claim 12

A tokenizer according to claim 10, comprising lexical processing means for comparing a selected lexical character with entries in the character table and for associating said first tag with a token including said selected lexical character, when said selected lexical character has an equivalent entry in the character table.

Claim 13

A tokenizer according to claim 10, comprising non-lexical processing means for comparing a selected non-lexical character with entries in the character table and for associating said first tag with a token preceding said selected non-lexical character, when said selected non-lexical character has an equivalent entry in the character table.

Claim 14

A tokenizer according to claim 10, wherein said character table includes entries representative of a plurality of languages such that said tokenizer operates in the plurality of languages.

Claim 15

A tokenizer according to claim 13, wherein the plurality of languages is selected from the group consisting of English, French, Catalan, Spanish, Italian, Portuguese, German, Danish, Norwegian, Swedish, Dutch, Finish, Russian, and Czech.

Claim 16

A tokenizer according to claim 1, wherein said filtering means further comprises a contextual processor for selecting said candidate token from said set of tokens as a function of a contextual analysis of the lexical and non-lexical characters surrounding a selected character in the parsed stream of text.

Claim 17

A tokenizer according to claim 15, wherein said contextual processor includes a set of rules applicable in a plurality of languages such that said tokenizer operates in the plurality of languages.

Claim 18

A tokenizer according to claim 16, wherein the plurality of languages is selected from the group consisting of English, French, Catalan, Spanish, Italian, Portuguese, German, Danish, Norwegian, Swedish, Dutch, Finish, Russian, and Czech.

Claim 19

A tokenizer according to claim 1, further comprising a memory element for storing and retrieving the digitized stream of natural language text and for storing and retrieving a data structure that includes parameters for each token.

Claim 20

A tokenizer according to claim 18, wherein said parameters include an input stream flag identifying the location of a digitized stream of natural language text in said memory element.

Claim 21

A tokenizer according to claim 18, wherein said parameters include a flag identifying the number of lexical characters and non-lexical characters forming a token.

Claim 22

A tokenizer according to claim 18, wherein said parameters include an output flag identifying the location of an output signal generated by said tokenizer.

Claim 23

A tokenizer according to claim 18, wherein said parameters include a flag identifying the number of lexical characters forming a token.

Claim 24

A tokenizer according to claim 18, wherein said parameters include the lexical and non-lexical attributes of a token.

Claim 25

A tokenizer according to claim 23, wherein said non-lexical attributes are selected from the group consisting of: contains white space, single new line, and multiple new line.

Claim 26

A computerized data processing method for identifying a token formed of a string of lexical characters found in a stream of digitized natural language text, said method comprising the steps of extracting lexical and non-lexical characters from the stream of text, identifying a set of tokens, each token being formed of a string of extracted lexical characters bounded by extracted non-lexical characters, and using a filter to select a candidate token from said set of tokens, said candidate token being suitable for linguistic processing beyond the identification of tokens.

Claim 27

A computerized data processing method for identifying a token formed of a string of lexical characters found in a stream of digitized natural language text, said method comprising the steps of extracting lexical and non-lexical characters from the stream of text, identifying a set of tokens, each token being formed of a string of extracted lexical characters bounded by extracted non-lexical characters, and using a filter to select a candidate token from said set of tokens, said candidate token being suitable for linguistic processing beyond the identification of tokens.

Claim 28

A computerized data processing method according to claim 25, wherein said extracting step further comprises the step of ignoring stoplist characters in the stream of text.

Claim 29

A computerized data processing method according to claim 25, wherein said identifying step further comprises the step of identifying a beginning of a first token as a function of the attributes associated with the first token.

Claim 30

A computerized data processing method according to claim 25, wherein said identifying step further comprises the step of identifying an end of a first token as a function of a pattern formed by a plurality of characters in the stream of digitized text.

Claim 31

A computerized data processing method according to claim 25, wherein said candidate token is selected from said set of tokens during a single scan of the parsed stream of text.

Claim 32

A computerized data processing method according to claim 25, wherein said selecting step further comprises the steps of comparing a selected character in the parsed stream of text with entries in a character table, and associating a first tag with a first token located proximal to said selected character, when said selected character has an equivalent entry in the character table.

Claim 33

A computerized data processing method according to claim 30, further comprising the steps of comparing a selected lexical character with entries in the character table, and associating said first tag with a token including said selected lexical character, when said selected lexical character has an equivalent entry in the character table.

Claim 34

A computerized data processing method according to claim 30, further comprising the steps of comparing a selected non-lexical character with entries in the character table, and associating said first tag with a token preceding said selected non-lexical character, when said selected non-lexical character has an equivalent entry in the character table.

Claim 35

A computerized data processing method according to claim 30, further comprising the step of forming a character table having entries representative of a plurality of languages.

Claim 36

A computerized data processing method according to claim 25, further comprising the steps of selecting said candidate token from said set of tokens as a function of a contextual analysis of the lexical and non-lexical characters surrounding a selected character in the parsed stream of text.

Claim 37

A computerized data processing method according to claim 25, further comprising the step of associating with said candidate token a tag identifying additional linguistic processing for said candidate token.

Claim 38

A computerized data processing method according to claim 35, further comprising the step of associating with a plurality of tokens, as a function of said candidate token, a plurality of tags identifying additional linguistic processing for said plurality of tokens.

Claim 39

A computerized data processing method according to claim 35, further comprising the step of modifying said candidate token as a function of said tag associated with said candidate token.

Claim 40

A computerized data processing method according to claim 37, further comprising the step of splitting said candidate token into multiple tokens.

Claim 41

A computerized data processing method according to claim 37, further comprising the step of stripping a character from said candidate token.

Claim 42

A computerized data processing method according to claim 37, further comprising the step of ignoring a non-lexical character surrounding said candidate token.

Claim 43

A computerized data processing method according to claim 37, further comprising the step of merging said candidate token with another token in the stream of text.

Claim 44

A computerized data processing method according to claim 35, further comprising the steps of storing in a first location of a memory element attributes of said candidate token, said attributes identifying the additional linguistic processing suitable for said candidate token, and causing the tag to point to the first location.

Claim 45

A computerized data processing method according to claim 42, further comprising the step of storing in the first location attributes selected from the group consisting of lexical attributes and non-lexical attributes.

Claim 46

A computerized data processing method according to claim 43, further comprising the step of selecting the non-lexical attributes from the group consisting of: contains white space, single new line, and multiple new line.

Claim 47

A computerized tokenizer for identifying a token formed of a string of lexical characters found in a stream of digitized natural language text, the computerized tokenizer comprising: parsing means for extracting lexical and non-lexical characters from the stream of digitized text, identifying means coupled with said parsing means for identifying a set of tokens, each token being formed of a string of parsed lexical characters bounded by non-lexical characters, filtering means coupled with said identifying means for selecting a candidate token from said set of tokens, said candidate token being suitable for additional linguistic processing, and a memory element for storing and retrieving the digitized stream of natural language text and for storing and retrieving a data structure that includes parameters for each token, wherein said parameters include the lexical and non-lexical attributes of a token, wherein said lexical attributes are selected from the group consisting of internal character attributes, special processing attributes, end of sentence attributes, and noun phrase attributes.

Claim 48

A computerized tokenizer for identifying a token formed of a string of lexical characters found in a stream of digitized natural language text, the computerized tokenizer comprising: parsing means for extracting lexical and non-lexical characters from the stream of digitized text, identifying means coupled with said parsing means for identifying a set of tokens, each token being formed of a string of parsed lexical characters bounded by non-lexical characters, filtering means coupled with said identifying means for selecting a candidate token from said set of tokens, said candidate token being suitable for additional linguistic processing, and a memory element for storing and retrieving the digitized stream of natural language text and for storing and retrieving a data structure that includes parameters for each token, wherein said parameters include the lexical and non-lexical attributes of a token, wherein said lexical attributes are selected from the group consisting of internal character attributes, special processing attributes, end of sentence attributes, and noun phrase attributes.

Claim 49

A tokenizer according to claim 45, wherein said internal character attributes are selected from the group consisting of leading apostrophe, internal apostrophe, trailing apostrophe, leading hyphen, internal hyphen, trailing hyphen, internal slash, and internal parentheses.

Claim 50

A tokenizer according to claim 45, wherein said special processing attributes are selected from the group consisting of number flags, possible pre-clitic, possible post-clitic, and unicode error.

Claim 51

A tokenizer according to claim 45, wherein said end of sentence attributes are selected from the group consisting of probable sentence termination, attached end of word period, stripped end of word period, capcode high, capcode low, and definite non sentence termination.

Claim 52

A tokenizer according to claim 45, wherein said noun phrase attributes are selected from the group consisting of probable sentence termination, attached end of word period, stripped end of word period, capcode high, capcode low, definite non sentence termination, pre noun phrase break, and post noun phrase break.

Claim 53

A computerized data processing method for identifying a token formed of a string of lexical characters found in a stream of digitized natural language text, said method comprising the steps of: extracting lexical and non-lexical characters from the stream of text, identifying a set of tokens, each token being formed of a string of extracted lexical characters bounded by extracted non-lexical characters, selecting a candidate token from said set of tokens, said candidate token being suitable for additional linguistic processing, associating with said candidate token a tag identifying additional linguistic processing for said candidate token, storing in a first location of a memory element attributes of said candidate token, said attributes identifying the additional linguistic processing suitable for said candidate token, causing the tag to point to the first location, and selecting the lexical attributes from the group consisting of internal character attributes, special processing attributes, end of sentence attributes, and noun phrase attributes.

Claim 54

A computerized data processing method for identifying a token formed of a string of lexical characters found in a stream of digitized natural language text, said method comprising the steps of: extracting lexical and non-lexical characters from the stream of text, identifying a set of tokens, each token being formed of a string of extracted lexical characters bounded by extracted non-lexical characters, selecting a candidate token from said set of tokens, said candidate token being suitable for additional linguistic processing, associating with said candidate token a tag identifying additional linguistic processing for said candidate token, storing in a first location of a memory element attributes of said candidate token, said attributes identifying the additional linguistic processing suitable for said candidate token, causing the tag to point to the first location, and selecting the lexical attributes from the group consisting of internal character attributes, special processing attributes, end of sentence attributes, and noun phrase attributes.