Method for language-independent text tokenization using a character categorization

Claim 2

A computer method for word isolation in a digital data processor, comprising the steps of: inputting a first input signal representative of a linguistic expression consisting of characters forming words and delimiters; inputting a second input signal representative of categories of each possible input character in the first input signal; comparing the said first input signal character-by-character against corresponding categories of said second input signal to locate token boundaries in said first input signal; generating an indication of the location of each word and each delimiter in said first input signal, in response to locating said token boundaries.

Claim 3

The method for word isolation of claim 1, wherein said generating step further comprises matching said first input signal against a set of prefixes or suffixes to indicate additional locations of words and delimiters.

Claim 6

The apparatus of claim 1, wherein said means for generating further includes means for matching said first input signal against a set of prefixes or suffixes to indicate additional locations of words and delimiters.

Claim 7

A computer method for isolating words from an input stream of characters forming natural language text, comprising the steps of: inputting a character classification table which describes each character in a character code as being either a delimiter character, an alpha/numeric character or a conditional delimiter character which assumes the function of a delimiter character when it occurs in predefined character contexts in said input stream of characters; building a string of alpha/numeric characters to form a word from said input stream of characters; inputting three consecutive characters from said input stream as a previous character, a current character and a next character; accessing said character classification table to determine if said current character is a delimiter character, an alpha/numeric character or a conditional delimiter character; appending said current character to said string when said character classification table identifies it as an alpha/numeric character; signaling that said string is a complete word when said character classification table identifies said current character as a delimiter character; analyzing said previous character, said current character and said next character to determine if said current character assumes the function of a delimiter character when said character classification table identifies said current character as a conditional delimiter character.

Claim 8

A computer method for isolating words from an input stream of characters forming natural language text, comprising the steps of: inputting a character classification table which describes each character in a character code as being either a delimiter character, an alpha/numeric character or a conditional delimiter character which assumes the function of a delimiter character when it occurs in predefined character contexts in said input stream of characters; building a string of alpha/numeric characters to form a word from said input stream of characters; inputting three consecutive characters from said input stream as a previous character, a current character and a next character; accessing said character classification table to determine if said current character is a delimiter character, an alpha/numeric character or a conditional delimiter character; appending said current character to said string when said character classification table identifies it as an alpha/numeric character; signaling that said string is a complete word when said character classification table identifies said current character as a delimiter character; analyzing said previous character, said current character and said next character to determine if said current character assumes the function of a delimiter character when said character classification table identifies said current character as a conditional delimiter character.

Claim 9

The computer method of claim 5, wherein said analyzing step further comprises: appending said current character to said string when said previous character is an alphabetic character and said next character is an alphabetic character.

Claim 10

The computer method of claim 5, wherein said analyzing step further comprises: appending said current character to said string when said previous character is a numeric character and said next character is a numeric character.

Claim 11

The computer method of claim 5, wherein said analyzing step further comprises: appending said current character to said string when said previous character is not an alphabetic character and not a numeric character and said next character is a numeric character.

Claim 12

The computer method of claim 5, wherein said analyzing step further comprises: appending said current character to said string when said previous character is not an alphabetic character and not a numeric character and said current character is the same as said previous character.

Claim 13

The computer method of claim 5, which further comprises: inputting a second character classification table which describes each character in a second character code as being either a delimiter character, an alpha/numeric character or a conditional delimiter character which assumes the function of a delimiter character when it occurs in predefined character contexts in said input stream of characters; selecting either the first said character code or said second character code for said input character stream; selectively accessing in said accessing step, either the first said character classification table or said second character classification table, in response to selecting either said first character code or said second character code.

Claim 14

The computer method of claim 10, wherein said first said classification table characterizes a first natural language and said second classification table characterizes a second natural language.

Claim 15

The computer method of claim 10, wherein said first said classification table characterizes a first type data processing code and said second classification table characterizes a second type data processing code.

Claim 16

The computer method of claim 5, which further comprises: inputting a second character classification table which describes each character in said character code as being either a delimiter character, an alpha/numeric character or a conditional delimiter character which assumes the function of a delimiter character when it occurs in predefined character contexts in said input stream of characters, the first said classification table characterizing a first natural language and said second classification table characterizing a second natural language; selectively accessing in said accessing step, either the first said character classification table or said second character classification table, in response to selecting either said first natural language or said second natural language for said input character stream.

Claim 17

The computer method of claim 5, which further comprises the step of comparing said characters in said input stream with a predefined set of prefixes and suffixes to indicate additional locations of words in said input stream.

Claim 18

The computer method of claim 5, wherein said input stream is from typing characters on a keyboard.

Claim 19

The computer method of claim 5, wherein said input stream is from reading a text file stored on a disk drive.

Claim 20

The computer method of claim 5, wherein said input stream is from recognition of text read from a document being scanned by an optical character recognition machine.

Claim 21

The computer method of claim 5, wherein said input stream is from receiving text files from a communications link.

Claim 22

The computer method of claim 5, which further comprises the steps of: outputting said complete word identified in said signaling step, as a search term and inputting it into an index for dictionary accessing.

Claim 23

The computer method of claim 5, which further comprises the steps of: outputting said complete word identified in said signaling step, as a search term and inputting it into an index for database accessing.

Claim 24

The computer method of claim 5, which further comprises the steps of: outputting said complete word identified in said signaling step, as a word in a first natural language and inputting it into a language translator for translation into a word in a second natural language.

Claim 25

The computer method of claim 5, applied to an input stream containing chemical literature terms.

Claim 26

The computer method of claim 5, applied to an input stream containing mathematical literature terms.