US6003027A
Expired / Lapsed
US
⏱️ 0.0 years remaining
System and method for determining confidence levels for the results of a categorization system
Assignee: International Business Machines Corp
•
Filing: 1997-11-21
•
Granted: 1999-12-14
•
Expires: 2017-11-21
Abstract
Abstract After a categorization process has been run, the scores of the top-two ranking categories along with the size or number of features in the object being categorized, are passed to a confidence assignment process. This determines a value for the confidence in the top category based on the evidence afforded by the input parameters. The magnitude of this confidence value will determine whether the system can accept the automatic categorization results, or whether human involvement is required. This invention also describes the process of determining the optimal value of an internal scaling parameter in the confidence assignment process. The construction of a threshold table based on this parameter is also described. The threshold table matches confidence values against error levels. For a given error rate the previously assigned confidence determines whether the categorization results can be accepted without need for human intervention. This invention maximizes the number of objects that can be automatically processed, for a given error rate.
Inventors
- 👤 John Martin Prager
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-29 23:12
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/30; G06F16/35; Y; Y10; Y10S; Y10S707/00; Y10S707/99931; Y10S707/99933; Y10S707/99935; Y10S707/99936
Claims Analysis
Total Claims: 23
Claim 1
Independent Claim
151 words
A computer system for determining a confidence level of a category of one or more documents, comprising: a memory and a central processing unit (CPU); a categorization process executed by the CPU and generating a results table, one or more documents being categorized into one or more categories, each category corresponding to a score in said results table and each document having a size and one or more features; and a confidence process executed by the CPU on said results table to generate a threshold table, said confidence process determining a confidence level for said each category by taking a product of a first credibility that a highest scoring category of said categories is correct with a second credibility that a next highest scoring category of said categories is incorrect and normalizing the product by a third credibility that at least one of the highest and second-highest scoring categories is incorrect.
Claim 2
Dependent Claim
151 words
A computer system for determining a confidence level of a category of one or more documents, comprising: a memory and a central processing unit (CPU); a categorization process executed by the CPU and generating a results table, one or more documents being categorized into one or more categories, each category corresponding to a score in said results table and each document having a size and one or more features; and a confidence process executed by the CPU on said results table to generate a threshold table, said confidence process determining a confidence level for said each category by taking a product of a first credibility that a highest scoring category of said categories is correct with a second credibility that a next highest scoring category of said categories is incorrect and normalizing the product by a third credibility that at least one of the highest and second-highest scoring categories is incorrect.
Claim 3
Dependent Claim
37 words
A system, as in claim 1, where the first and second credibilities are weighted by a raising each of the first and second credibilities to a power that is a function of the size of the document.
Claim 4
Dependent Claim
24 words
A system, as in claim 2, where the function is log(1+N/k), where N is the size of the document and k is a constant.
Claim 5
Dependent Claim
16 words
A system, as in claim 3, where k is greater than 1 but less than N.
Claim 6
Dependent Claim
31 words
A system, as in claim 3, where an optimum value of k is empirically determined for one or more of the following: a set of documents and a set of features.
Claim 7
Dependent Claim
18 words
A system, as in claim 3, where N is measured by the number of features of the document.
Claim 8
Dependent Claim
47 words
A system, as in claim 6, where the document contains text and one or more of the features is the frequency of occurence of any one of the following: an individual word, an individual phrase, and an N-gram that is a sequence of one or more characters.
Claim 9
Dependent Claim
39 words
A system, as in claim 6, where the document contains audio and one or more of the features is the frequency of occurence of any one of the following: a pitch, an amplitude, and a duration of a note.
Claim 10
Dependent Claim
37 words
A system, as in claim 6, where the document contains still images and one or more of the features is the frequency of occurence of any one of the following: a color, a texture, and a shape.
Claim 11
Dependent Claim
31 words
A system, as in claim 1, where one of the confidence levels is correlated to a correctness level of a set of one or more correctness levels of the categorization process.
Claim 12
Dependent Claim
21 words
A system, as in claim 10, where each of the confidence levels is determined at an optimal empirical value of k.
Claim 13
Dependent Claim
43 words
A system, as in claim 1, where the documents can be any one or more of the following types: electronic mail, scientific articles, news stories, business information, sports information, requests to a service organization, a claim, a letter, and transcription of telephone calls.
Claim 14
Dependent Claim
28 words
A system, as in claim 12, where the documents are tagged with one or more of the types as determined by the category with the highest confidence level.
Claim 15
Dependent Claim
24 words
A system, as in claim 12, where the documents are routed to a location as determined by the category with the highest confidence level.
Claim 16
Independent Claim
86 words
A method of verifying categorization of one or more documents, the method comprising the steps of: taking a product of the credibility that a highest scoring document category is correct with a second credibility that a next highest scoring document category is incorrect; normalizing the product by the credibility that at least one of the highest and second-highest scoring document categories is incorrect to create a confidence score; and assigning the document to the highest scoring document category if the confidence score is above a threshold.
Claim 17
Independent Claim
86 words
A method of verifying categorization of one or more documents, the method comprising the steps of: taking a product of the credibility that a highest scoring document category is correct with a second credibility that a next highest scoring document category is incorrect; normalizing the product by the credibility that at least one of the highest and second-highest scoring document categories is incorrect to create a confidence score; and assigning the document to the highest scoring document category if the confidence score is above a threshold.
Claim 18
Dependent Claim
66 words
A method as in claim 15, further comprising before the step of taking the product of credibilities, the step of: generating intermediate values for the highest and next highest scoring categories, the intermediate value being generated according to (1-S) log (1+N/k) and wherein, S is the score of a corresponding category, N is determined by a corresponding document being scored and k is a scaling constant.
Claim 19
Dependent Claim
34 words
A method as in claim 15, wherein scaling constant k is initially set to the square root of an average number of categorized features and adjusted to increase the number of correctly categorized documents.
Claim 20
Independent Claim
89 words
A system for verifying categorization of one or more documents, the system comprising: means for taking a product of the credibility that a highest scoring document category is correct with a second credibility that a next highest scoring document category is incorrect; means for normalizing the product by the credibility that at least one of the highest and second-highest document scoring categories is incorrect to create a confidence score; and means for assigning the document in the highest scoring document category if the confidence score is above a threshold.
Claim 21
Independent Claim
89 words
A system for verifying categorization of one or more documents, the system comprising: means for taking a product of the credibility that a highest scoring document category is correct with a second credibility that a next highest scoring document category is incorrect; means for normalizing the product by the credibility that at least one of the highest and second-highest document scoring categories is incorrect to create a confidence score; and means for assigning the document in the highest scoring document category if the confidence score is above a threshold.
Claim 22
Dependent Claim
56 words
A system as in claim 18, further comprising: means for generating intermediate values for the highest and next highest scoring categories, the intermediate value being generated according to (1-S) log (1+N/k) and wherein, S is the score of a corresponding category, N is determined by a corresponding document being scored and k is a scaling constant.
Claim 23
Dependent Claim
39 words
A method as in claim 19, wherein scaling constant k is initially set to the square root of an average number of categorized features and further comprising: means for adjusting k to increase the number of correctly categorized documents.
References Cited (Backward Citations)
US4414629A
Cited
US5680511A
Cited
US5640553A
Cited
US5659742A
Cited
US5717914A
Cited
US5742816A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2002-12-13 | 2003-06-14 | 2003-12-14 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2006-12-13 | 2007-06-14 | 2007-12-14 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2010-12-13 | 2011-06-14 | 2011-12-14 | $7700.00 | Window Closed / Lapsed |