US5956419A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Unsupervised training of character templates using unsegmented samples
Assignee: Xerox Corp
•
Filing: 1995-04-28
•
Granted: 1999-09-21
•
Expires: 2015-04-28
Abstract
Abstract A method for operating a machine to perform unsupervised training of a set of character templates uses as the source of training samples an image source of character images, called glyphs, that need not be manually or automatically segmented or isolated prior to training. A recognition operation performed on the image source of character images produces a labeled glyph position data structure that includes, for each glyph in the image source, a glyph image position in the image source associating an estimated image location of the glyph in the image source with a character label paired with the glyph image position that indicates the character in the character set being trained. The labeled glyph position data and the image source are then used to determine sample image regions in the image source; each sample image region is large enough to contain at least a single glyph but need not be restricted in size to only contain a single glyph. The template construction process using unsegmented samples is mathematically modeled as an optimization problem that optimizes a function that represents the set of character templates being trained as an ideal image to be reconstructed to match the input image. The method produces all of the character templates substantially contemporaneously by using a novel pixel scoring technique that implements an approximation of a maximum likelihood criterion subject to a constraint on the templates produced which holds that foreground pixels in adjacently positioned character images have substantially nonoverlapping foreground pixels. The character templates produced may be binary templates or arrays of probability values.
Inventors
- 👤 Gary E. Kopec
- 👤 Philip Andrew Chou
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-08 14:51
CPC Classifications
G; G06; G06V; G06V10/00; G06V10/70; G06V10/74; G06V10/75; G06F; G06F18/00; G06F18/20; G06F18/21; G06F18/214
Claims Analysis
Total Claims: 18
Claim 1
Independent Claim
559 words
A method of operating a machine to perform unsupervised training of a set of character templates, the machine including a processor and a memory device for storing data, the data including instruction data the processor executes to operate the machine, the processor being coupled to the memory device for accessing the data, the method comprising the steps of: A) receiving and storing an image glyph source including a plurality of image pixel positions defining an image, the image glyph source including a plurality of glyphs, each glyph being an image instance of a respective one of a plurality of characters in a character set, the set of character templates being trained representing respective ones of the plurality of characters in the character set; B) performing a recognition operation on the image glyph source, the recognition operation producing a plurality of labeled glyph position data items, each labeled glyph position data item indicating one of the plurality of image pixel positions in the image glyph source and a respectively paired glyph label paired with the image pixel position, each image pixel position associating an image glyph source location with a glyph occurring in the image glyph source, each respectively paired glyph label identifying the glyph associated with the image pixel position as a respective one of the plurality of characters in the character set; C) determining a sample image region included in the image glyph source for each labeled glyph position data item, the sample image region including the image pixel position indicating the image glyph source location of a glyph and being identified as a training data sample for the character template indicated by the respectively paired glyph label, each sample image region including a plurality of sample pixel positions in the image glyph source each sample pixel position indicating a sample pixel value; D) for each respective character template to be trained producing a template image region including a plurality of template pixel positions for storing the respective character template; and E) producing the set of character templates using the template image regions and the sample image regions by the sub-steps of: (a) producing an image definition data structure for defining and storing an ideal image, the ideal image being represented as a function of the set of character templates being trained, and being a reconstruction of the image glyph source formed by positioning respective ones of the character templates in an image plane at image pixel positions identified as image glyph source locations of glyphs occurring in the image glyph source, each respective one of the character templates positioned in the ideal image being identified by the glyph label paired with the image glyph source location; (b) computing pixel scores for template pixel positions in template image regions using selected ones of the sample pixel positions in selected ones of the sample image regions included in the image source of glyphs; and (c) sequentially assigning a pixel value to selected template pixel positions in selected template image regions, the selected template pixel positions being selected on the basis of the pixel scores optimizing the function representing the ideal image such that, when all template pixel positions have been assigned pixel values, the pixel value assigned to each selected template pixel position optimizes a matching score measuring a match between the image glyph source and the ideal image.
Claim 2
Independent Claim
559 words
A method of operating a machine to perform unsupervised training of a set of character templates, the machine including a processor and a memory device for storing data, the data including instruction data the processor executes to operate the machine, the processor being coupled to the memory device for accessing the data, the method comprising the steps of: A) receiving and storing an image glyph source including a plurality of image pixel positions defining an image, the image glyph source including a plurality of glyphs, each glyph being an image instance of a respective one of a plurality of characters in a character set, the set of character templates being trained representing respective ones of the plurality of characters in the character set; B) performing a recognition operation on the image glyph source, the recognition operation producing a plurality of labeled glyph position data items, each labeled glyph position data item indicating one of the plurality of image pixel positions in the image glyph source and a respectively paired glyph label paired with the image pixel position, each image pixel position associating an image glyph source location with a glyph occurring in the image glyph source, each respectively paired glyph label identifying the glyph associated with the image pixel position as a respective one of the plurality of characters in the character set; C) determining a sample image region included in the image glyph source for each labeled glyph position data item, the sample image region including the image pixel position indicating the image glyph source location of a glyph and being identified as a training data sample for the character template indicated by the respectively paired glyph label, each sample image region including a plurality of sample pixel positions in the image glyph source each sample pixel position indicating a sample pixel value; D) for each respective character template to be trained producing a template image region including a plurality of template pixel positions for storing the respective character template; and E) producing the set of character templates using the template image regions and the sample image regions by the sub-steps of: (a) producing an image definition data structure for defining and storing an ideal image, the ideal image being represented as a function of the set of character templates being trained, and being a reconstruction of the image glyph source formed by positioning respective ones of the character templates in an image plane at image pixel positions identified as image glyph source locations of glyphs occurring in the image glyph source, each respective one of the character templates positioned in the ideal image being identified by the glyph label paired with the image glyph source location; (b) computing pixel scores for template pixel positions in template image regions using selected ones of the sample pixel positions in selected ones of the sample image regions included in the image source of glyphs; and (c) sequentially assigning a pixel value to selected template pixel positions in selected template image regions, the selected template pixel positions being selected on the basis of the pixel scores optimizing the function representing the ideal image such that, when all template pixel positions have been assigned pixel values, the pixel value assigned to each selected template pixel position optimizes a matching score measuring a match between the image glyph source and the ideal image.
Claim 3
Dependent Claim
330 words
The method of claim 1 of operating a machine to perform unsupervised training of a set of character templates wherein glyphs occurring in the image glyph source are positioned according to a characteristic image positioning property such that, when a second glyph is positioned in the image glyph source with a glyph image origin position thereof displaced from a glyph image origin position of a preceding first glyph by a set width thereof, a first bounding box entirely containing the first glyph overlaps in the image glyph source with a second bounding box entirely containing the second glyph, and the first and second glyphs have substantially nonoverlapping foreground pixel positions; wherein each sample image region has size dimensions large enough to include at least two glyphs positioned in the image glyph source according to the characteristic image positioning property; wherein each character template in the set of character templates being trained is defined relative to a template origin position; the function representing the ideal image requiring that respective ones of the character templates forming the ideal image in the image plane be positioned according to the characteristic image positioning property, such that, when a second character template is positioned in the ideal image with the template origin position thereof displaced from the template origin position of a preceding first character template by a character set width thereof, a first bounding box entirely containing the first character template is permitted to overlap in the image with a second bounding box entirely containing the second character template as long as the first and second character templates have substantially nonoverlapping foreground pixel positions; and wherein the selected template pixel positions selected on the basis of the pixel scores optimizing the function representing the ideal image are selected further on the basis of the pixel scores optimizing the matching score between the image glyph source and the ideal image when the ideal image has character templates positioned according to the characteristic image positioning property.
Claim 4
Dependent Claim
107 words
The method of claim 1 of operating a machine to perform unsupervised training of the set of character templates wherein each character template stored in a respective template image region has a foreground probability parameter associated therewith indicating a probability value that the foreground pixel values indicated by the template pixel positions in the template image region are valid foreground pixel values; and the substep (b) of computing the pixel scores for the template pixel positions in the template image regions further includes computing the pixel scores for a respective one of the template image regions using the probability value associated with the respective template image region.
Claim 5
Dependent Claim
28 words
The method of claim 3 of operating a machine to perform unsupervised training of the set of character templates wherein the foreground probability value varies by character template.
Claim 6
Dependent Claim
211 words
The method of operating a machine to train a set of character templates of claim 3 wherein the set of character templates being trained includes a "space" character template defined as a plurality of foreground template pixel positions each indicating a background pixel value; the template image regions produced for storing respective character templates including a template image region, hereafter referred to as a "space" template image region for storing the "space" character template; the "space" character template being produced using the "space" template image region and using at least one sample image region identified as a training sample for a "space" character in the character set; wherein the white foreground pixels of the "space" character are stored as black foreground pixels in the "space" template image region representing the "space" character, and the black background pixels of the "space"character are stored as white background pixels in the "space" template image region; and wherein the foreground probability value indicates a low probability value indicating that the black foreground pixel values indicated by the template pixel positions in the "space" template image region are less likely to be valid black foreground pixel values, such that an operation using the "space" character template interprets the black foreground pixel values as white foreground pixels.
Claim 7
Dependent Claim
59 words
The method of claim 1 of operating a machine to perform unsupervised training of a set of character templates wherein the pixel value assigned to a selected template pixel position in a selected template image region is a probability value indicating the probability that a pixel color assigned to the selected template pixel position is a valid pixel color.
Claim 8
Independent Claim
639 words
A method of operating a machine to perform unsupervised training of a set of character templates, the machine including a processor and a memory device for storing data, the data including instruction data the processor executes, the method comprising the steps of: A) receiving and storing in the memory device an image glyph source including a plurality of image pixel positions defining an image and including a plurality of glyphs therein, each glyph being an image instance of a respective one of a plurality of characters in a character set, the set of character templates being trained representing respective ones of the plurality of characters in the character set; B) receiving and storing in the memory device an image source model modeling as a grammar a spatial image structure of a set of images each including a plurality of glyphs, the image glyph source being one of the set of images modeled by the image source model, the image source model including spatial positioning data modeling spatial positioning of the plurality of glyphs occurring in the image glyph source, the image source model further including mapping data mapping a respective one of the glyphs occurring in the image glyph source to a glyph label indicating the character in the character set represented by the glyph; C) performing a recognition operation on the image glyph source using the image source model, the recognition operation using the spatial positioning data to determine an image pixel position in the image glyph source associating an image glyph source location with a glyph occurring in the image glyph source; the recognition operation further using the mapping data to pair a respectively paired glyph label with the image pixel position, each respectively paired glyph label identifying the glyph associated with the image pixel position as a respective one of the plurality of characters in the character set, the recognition operation producing a plurality of labeled glyph position data items each indicating an image pixel position of a glyph in the image glyph source and a respectively paired glyph label; D) determining a sample image region included in the image glyph source for each labeled glyph position data item, the sample image region including the image pixel position indicating the image glyph source location of a glyph and being identified as a training data sample for the character template indicated by the respectively paired glyph label each sample image region including a plurality of sample pixel positions in the image glyph source, each indicating a sample pixel value; E) for each respective character template to be trained producing a template image region including a plurality of template pixel positions for storing the respective character template; and F) producing the set of character templates using the template image regions and the sample image regions by the sub-steps of: (a) computing template pixel scores for respective ones of the template pixel positions in the template image regions using the sample pixel values indicated by the sample pixel positions included in the sample image regions; (b) assigning a foreground pixel value to a template pixel position, referred to as an assigned template pixel position, in one of the template image regions; the assigned template pixel position being selected on the basis of the template pixel scores; (c) modifying the sample pixel values of the sample pixel positions used in computing the template pixel score for the assigned template pixel position to indicate modified sample pixel values that, when used in computing a subsequent template pixel score for an unassigned template pixel position, reduce the chance that a foreground pixel value will be assigned to the unassigned template pixel position on the basis of the subsequent template pixel score; and (d) repeating substeps (a), (b) and (c) until a stopping condition indicates that the set of character templates is complete.
Claim 9
Independent Claim
639 words
A method of operating a machine to perform unsupervised training of a set of character templates, the machine including a processor and a memory device for storing data, the data including instruction data the processor executes, the method comprising the steps of: A) receiving and storing in the memory device an image glyph source including a plurality of image pixel positions defining an image and including a plurality of glyphs therein, each glyph being an image instance of a respective one of a plurality of characters in a character set, the set of character templates being trained representing respective ones of the plurality of characters in the character set; B) receiving and storing in the memory device an image source model modeling as a grammar a spatial image structure of a set of images each including a plurality of glyphs, the image glyph source being one of the set of images modeled by the image source model, the image source model including spatial positioning data modeling spatial positioning of the plurality of glyphs occurring in the image glyph source, the image source model further including mapping data mapping a respective one of the glyphs occurring in the image glyph source to a glyph label indicating the character in the character set represented by the glyph; C) performing a recognition operation on the image glyph source using the image source model, the recognition operation using the spatial positioning data to determine an image pixel position in the image glyph source associating an image glyph source location with a glyph occurring in the image glyph source; the recognition operation further using the mapping data to pair a respectively paired glyph label with the image pixel position, each respectively paired glyph label identifying the glyph associated with the image pixel position as a respective one of the plurality of characters in the character set, the recognition operation producing a plurality of labeled glyph position data items each indicating an image pixel position of a glyph in the image glyph source and a respectively paired glyph label; D) determining a sample image region included in the image glyph source for each labeled glyph position data item, the sample image region including the image pixel position indicating the image glyph source location of a glyph and being identified as a training data sample for the character template indicated by the respectively paired glyph label each sample image region including a plurality of sample pixel positions in the image glyph source, each indicating a sample pixel value; E) for each respective character template to be trained producing a template image region including a plurality of template pixel positions for storing the respective character template; and F) producing the set of character templates using the template image regions and the sample image regions by the sub-steps of: (a) computing template pixel scores for respective ones of the template pixel positions in the template image regions using the sample pixel values indicated by the sample pixel positions included in the sample image regions; (b) assigning a foreground pixel value to a template pixel position, referred to as an assigned template pixel position, in one of the template image regions; the assigned template pixel position being selected on the basis of the template pixel scores; (c) modifying the sample pixel values of the sample pixel positions used in computing the template pixel score for the assigned template pixel position to indicate modified sample pixel values that, when used in computing a subsequent template pixel score for an unassigned template pixel position, reduce the chance that a foreground pixel value will be assigned to the unassigned template pixel position on the basis of the subsequent template pixel score; and (d) repeating substeps (a), (b) and (c) until a stopping condition indicates that the set of character templates is complete.
Claim 10
Dependent Claim
179 words
The method of claim 7 of operating a machine to perform unsupervised training of the set of character templates wherein each template pixel score computed for a respective template pixel position is a function of a count of foreground pixel values indicated by the sample pixel positions included in the sample image regions used for computing the template pixel score; wherein substep (b) of assigning a foreground pixel value to a template pixel position further includes selecting the template pixel position on the basis of a maximum template pixel score; and wherein substep (c) of modifying the sample pixel values of the sample pixel positions used in computing the template pixel score for an assigned template pixel position includes modifying the sample pixel values to indicate background pixel values, such that a subsequent template pixel score computed for an unassigned template pixel position using the modified sample pixel values will be reduced, thereby reducing the chance that a foreground pixel value will be assigned to the unassigned template pixel position on the basis of the subsequent template pixel score.
Claim 11
Dependent Claim
403 words
The method of operating a machine to train a set of character templates of claim 7 wherein each template pixel position in each template image region is identified relative to a local template region origin position defined according to a local template coordinate system; one of the template pixel positions included in each template image region being designated as a template origin position having a displacement from the local template region origin position in the local template coordinate system; wherein a plurality of sample image regions identified by respectively paired glyph labels as training data samples for a respective one of the character templates is included in a glyph sample collection data structure, referred to as a glyph sample collection, for the respective character template; wherein each sample pixel position in each sample image region in each glyph sample collection is identified relative to a local sample region origin position according to a local sample coordinate system; the image pixel position in each sample image region that indicates the image glyph source location of a glyph in the image glyph source being positioned in the sample image region at a displacement from the local sample region origin position equal to the displacement of the template origin position from the local template region origin position in the template image region representing the character template identified by the glyph sample collection, such that the sample image regions included in a glyph sample collection are effectively aligned at their respective image glyph source locations, being hereafter referred to as aligned sample image regions; each sample pixel position in a first one of the aligned sample image regions being respectively paired with a sample pixel position at the same displacement from the local sample region origin position in a second one of the aligned sample image regions, and being respectively paired with a template pixel position at the same displacement from the local template region origin position in the template image region representing the character template identified by the glyph sample collection; and wherein the substep (a) of computing the template pixel scores for respective ones of the template pixel positions in the template image regions further includes computing each template pixel score for a respective template pixel position using respectively paired sample pixel positions in the aligned sample image regions included in the glyph sample collection identifying training data samples for the respective template image region.
Claim 12
Dependent Claim
109 words
The method of operating a machine to train a set of character templates of claim 7 wherein each character template stored in a respective template image region has a foreground probability parameter associated therewith indicating a probability value that the foreground pixel values indicated by the template pixel positions in the template image region are valid foreground pixel values; and the substep (a) of computing the template pixel scores for respective ones of the template pixel positions in the template image regions further includes computing the template pixel scores for a respective one of the template image regions using the probability value associated with the respective template image region.
Claim 13
Dependent Claim
214 words
The method of claim 10 of operating a machine to perform unsupervised training of the set of character templates wherein the set of character templates being trained includes a "space" character template defined as a plurality of foreground template pixel positions each indicating a background pixel value; the template image regions produced for storing respective character templates including a template image region, hereafter referred to as a "space" template image region for storing the "space" character template; the "space" character template being produced using the "space" template image region and using at least one sample image region identified as a training sample for a "space" character in the character set; wherein the white foreground pixels of the "space" character are stored as black foreground pixels in the "space" template image region representing the "space" character, and the black background pixels of the "space" character are stored as white background pixels in the "space" template image region; and wherein the foreground probability value indicates a low probability value indicating that the black foreground pixel values indicated by the template pixel positions in the "space" template image region are less likely to be valid black foreground pixel values, such that an operation using the "space"character template interprets the black foreground pixel values as white foreground pixels.
Claim 14
Dependent Claim
107 words
The method of claim 7 of operating a machine to perform unsupervised training of the set of character templates wherein each template pixel position included in a respective template image region has a foreground probability parameter associated therewith indicating a probability value that the foreground pixel value indicated by the template pixel position is a valid foreground pixel value; and wherein the substep (a) of computing the template pixel scores for respective ones of the template pixel positions in the template image regions further includes computing each template pixel score for a respective template pixel position using the probability value associated with the respective template pixel position.
Claim 15
Dependent Claim
41 words
The method of operating a machine to train a set of character templates of claim 12 wherein the foreground probability parameter associated with each character template indicates a probability value that varies by character template in the set of character templates.
Claim 16
Dependent Claim
188 words
The method of claim 7 of operating a machine to perform unsupervised training of the set of character templates wherein the image source model models the spatial image structure of the set of images as a stochastic finite state transition network; the image glyph source being modeled as at least one path through the finite state transition network that indicates an ideal image consistent with the spatial image structure of the image glyph source; the at least one path indicating path data items associated therewith and accessible by the processor; the path data items indicating the labeled glyph position data items including image pixel positions and glyph labels paired therewith of respective ones of the plurality of glyphs included in the image glyph source; wherein step C) further comprises the sub-steps of: a) performing a decoding operation using finite state transition network, the decoding operation producing at least one complete path indicating an ideal image consistent with the spatial image structure of the image glyph source; and b) obtaining the labeled glyph position data items using the path data items associated with the at least one complete path.
Claim 17
Dependent Claim
596 words
A machine for use in training a set of bitmapped character templates for use in a recognition operation; the machine comprising: A) a signal source for providing image definition data defining an image; B) image input circuitry connected for receiving the image definition data defining the image from the signal source; C) a processor connected for receiving the image definition data defining the image from the image input circuitry; and D) memory for storing data, the data including instructions the processor can execute; the processor being further coupled the memory; wherein the instructions comprise the steps of: 1) receiving from the image input circuitry an image glyph source including a plurality of image pixel positions defining an image, including a plurality of glyphs therein, each glyph being an image instance of a respective one of a plurality of characters in a character set; the set of character templates being trained representing respective ones of the plurality of characters in the character set; 2) performing a recognition operation on the image glyph source; the recognition operation producing a plurality of labeled glyph position data items, each labeled glyph position data item indicating one of the plurality of image pixel positions in the image glyph source and a respectively paired glyph label paired with the image pixel position, each image pixel position associating an image glyph source location with a glyph occurring in the image glyph source, each respectively paired glyph label identifying the glyph associated with the image pixel position as a respective one of the plurality of characters in the character set; 3) determining a sample image region included in the image glyph source for each labeled glyph position data item; the sample image region including the image pixel position indicating the image glyph source location of a glyph and being identified as a training data sample for the character template indicated by the respectively paired glyph label; each sample image region including a plurality of sample pixel positions in the image glyph source, each indicating a sample pixel value; and for each respective character template to be trained further producing a template image region including a plurality of template pixel positions for storing the respective character template; and 4) producing the set of character templates using the template image regions and the sample image regions by the sub-steps of: (a) producing an image definition data structure for defining and storing an ideal image; the ideal image being represented as a function of the set of character templates being trained, and being a reconstruction of the image glyph source formed by positioning respective ones of the character templates in an image plane at image pixel positions identified as image glyph source locations of glyphs occurring in the image glyph source; each respective one of the character templates positioned in the ideal image being identified by the glyph label paired with the image glyph source location; (b) computing pixel scores for template pixel positions in template image regions using selected ones of the sample pixel positions in selected ones of the sample image regions included in the image source of glyphs; and (c) sequentially assigning a pixel value to selected template pixel positions in selected template image regions; the selected template pixel positions being selected on the basis of the pixel scores optimizing the function representing the ideal image such that, when all template pixel positions have been assigned pixel values, the pixel value assigned to each selected template pixel position optimizes a matching score measuring a match between the image glyph source and the ideal image.
Claim 18
Dependent Claim
596 words
A machine for use in training a set of bitmapped character templates for use in a recognition operation; the machine comprising: A) a signal source for providing image definition data defining an image; B) image input circuitry connected for receiving the image definition data defining the image from the signal source; C) a processor connected for receiving the image definition data defining the image from the image input circuitry; and D) memory for storing data, the data including instructions the processor can execute; the processor being further coupled the memory; wherein the instructions comprise the steps of: 1) receiving from the image input circuitry an image glyph source including a plurality of image pixel positions defining an image, including a plurality of glyphs therein, each glyph being an image instance of a respective one of a plurality of characters in a character set; the set of character templates being trained representing respective ones of the plurality of characters in the character set; 2) performing a recognition operation on the image glyph source; the recognition operation producing a plurality of labeled glyph position data items, each labeled glyph position data item indicating one of the plurality of image pixel positions in the image glyph source and a respectively paired glyph label paired with the image pixel position, each image pixel position associating an image glyph source location with a glyph occurring in the image glyph source, each respectively paired glyph label identifying the glyph associated with the image pixel position as a respective one of the plurality of characters in the character set; 3) determining a sample image region included in the image glyph source for each labeled glyph position data item; the sample image region including the image pixel position indicating the image glyph source location of a glyph and being identified as a training data sample for the character template indicated by the respectively paired glyph label; each sample image region including a plurality of sample pixel positions in the image glyph source, each indicating a sample pixel value; and for each respective character template to be trained further producing a template image region including a plurality of template pixel positions for storing the respective character template; and 4) producing the set of character templates using the template image regions and the sample image regions by the sub-steps of: (a) producing an image definition data structure for defining and storing an ideal image; the ideal image being represented as a function of the set of character templates being trained, and being a reconstruction of the image glyph source formed by positioning respective ones of the character templates in an image plane at image pixel positions identified as image glyph source locations of glyphs occurring in the image glyph source; each respective one of the character templates positioned in the ideal image being identified by the glyph label paired with the image glyph source location; (b) computing pixel scores for template pixel positions in template image regions using selected ones of the sample pixel positions in selected ones of the sample image regions included in the image source of glyphs; and (c) sequentially assigning a pixel value to selected template pixel positions in selected template image regions; the selected template pixel positions being selected on the basis of the pixel scores optimizing the function representing the ideal image such that, when all template pixel positions have been assigned pixel values, the pixel value assigned to each selected template pixel position optimizes a matching score measuring a match between the image glyph source and the ideal image.
References Cited (Backward Citations)
US3233219A
Cited
US4769716A
Cited
US5020112A
Cited
US5321773A
Cited
US5526444A
Cited
US5303313A
Cited
US5361356A
Cited
US5706364A
Cited
US5828771A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2002-09-20 | 2003-03-22 | 2003-09-21 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2006-09-20 | 2007-03-22 | 2007-09-21 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2010-09-20 | 2011-03-22 | 2011-09-21 | $7700.00 | Window Closed / Lapsed |