Patent US4293054A successfully refreshed from Google Patents.
Patent US6046740A successfully refreshed from Google Patents.
US6046740A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Application testing with virtual object recognition
Assignee: Borland Software Corp
•
Filing: 1998-02-06
•
Granted: 2000-04-04
•
Expires: 2018-02-06
Abstract
Abstract A method of recognizing graphical objects by subjecting the graphical information gathered through "spying" to a series of rules by which the object becomes understood or recognized as an instance of a standard logical object. Before the rules are applied, graphical objects are first interpreted as primitives including groups of text, lines and images. In order to recognize a graphical object as a logical object, the graphical information is subjected to the rules in an iterative process whereby an understanding of the object is continually refined. As the rules are applied, the results are evaluated to determine whether the graphical object can be "mapped" to a standard logical object such as a textfield or listbox. Once the object is understood as a logical element with which the user is accustomed, it is possible to interact with the object and obtain data from the object as if it were a standard object with a published interface. By subjecting the graphical data to a series of rules designed specifically to recognize tables, the boundaries and the internal structure of rows and columns will be understood. Once the graphical data is recognized as a table, the data which it contains in rows can then be accessed. By classifying an object as an instance of a known object, assumptions can be made about the object so that it can be navigated or validated by sending events or messages.
Inventors
- 👤 David C. Laroche
- 👤 Timothy A. Anderson
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-07 19:41
CPC Classifications
G; G06; G06F; G06F9/00; G06F9/06; G06F9/44; G06F9/451; G06V; G06V30/00; G06V30/40; G06V30/42; G06V30/422; Y; Y10; Y10S; Y10S715/00; Y10S715/961; Y10S715/965; Y10S715/966; Y10S715/967
Claims Analysis
Total Claims: 57
Claim 1
Independent Claim
122 words
A method for recognizing graphical objects comprising the following steps: obtaining a current display state from a graphical user interface operating system; classifying displayed windows as canvas windows or control windows; classifying control windows which are separate windows as native controls, otherwise classifying control windows which are part of a canvas window as rendered controls; transforming a real window hierarchy which is used to create the displayed windows into a perceived window hierarchy corresponding to the visual display of the displayed windows; identifying graphical objects in the perceived window hierarchy as graphical object primitives; applying a series of rules to the graphical object primitives to characterize the graphical object primitives as logical objects based on visual characteristics of said graphical object primitives.
Claim 2
Independent Claim
122 words
A method for recognizing graphical objects comprising the following steps: obtaining a current display state from a graphical user interface operating system; classifying displayed windows as canvas windows or control windows; classifying control windows which are separate windows as native controls, otherwise classifying control windows which are part of a canvas window as rendered controls; transforming a real window hierarchy which is used to create the displayed windows into a perceived window hierarchy corresponding to the visual display of the displayed windows; identifying graphical objects in the perceived window hierarchy as graphical object primitives; applying a series of rules to the graphical object primitives to characterize the graphical object primitives as logical objects based on visual characteristics of said graphical object primitives.
Claim 3
Dependent Claim
21 words
The method of claim 1, further comprising the step of updating the perceived window hierarchy as windows are created or destroyed.
Claim 4
Dependent Claim
26 words
The method of claim 1, wherein the perceived window hierarchy includes container windows and wherein each of said container windows is capable of including child windows.
Claim 5
Dependent Claim
37 words
The method of claim 1, further comprising the step of synchronizing access to data structures by blocking execution of a process thread if it requires access to a data structure already in use in another process thread.
Claim 6
Dependent Claim
91 words
The method of claim 1, further comprising the following steps: grouping the graphical primitive objects in a window using an object grouping process thread in order to create virtual objects; preventing an application under test from drawing while said object grouping is being performed, said preventing step comprising at least one of the following steps: marking said object grouping process thread as time critical; allowing said object grouping process thread to obtain a read lock on objects being grouped; and determining whether the application under test is in a ready condition.
Claim 7
Dependent Claim
37 words
The method of claim 5, further comprising the step of adding at least one new graphical primitive object to at least one window corresponding to at least one of text and line drawing operations in said window.
Claim 8
Dependent Claim
24 words
The method of claim 6, wherein said adding step comprises the step of recording the text and clipping rectangle for said text drawing operation.
Claim 9
Dependent Claim
25 words
The method of claim 5, further comprising the step of updating information concerning said graphical primitive objects to reflect modifications to said graphical primitive objects.
Claim 10
Dependent Claim
89 words
The method of claim 5, further comprising the step of updating information concerning said graphical primitive objects to reflect movement of said graphical primitive objects, said movement including at least one of the following types of movement: copying an off-screen bitmap into a container window in said virtual window hierarchy to thereby simulate the drawing of objects corresponding to said bitmap directly in said container window; moving a portion of said display and updating origin information concerning said moved portion; and moving a content window within its parent window.
Claim 11
Dependent Claim
25 words
The method of claim 5, wherein said graphical primitive objects include lines and text strings and said virtual objects include text fields and composite objects.
Claim 12
Dependent Claim
11 words
The method of claim 10, wherein said composite objects include tables.
Claim 13
Dependent Claim
67 words
The method of claim 5, wherein said object grouping process thread resides in between said application under test and said graphical user interface operating system such that requests from said application under test to said graphical user interface operating system are intercepted and analyzed by said object grouping process thread before being passed on from said object grouping process thread to said graphical user interface operating system.
Claim 14
Dependent Claim
59 words
The method of claim 5, further comprising the step of adding at least one new graphical primitive object to at least one window said new graphical primitive object including a hyperlink, and said adding step further comprising the steps of positioning a mouse cursor in the vicinity of said hyperlink and extracting status window information corresponding to said hyperlink.
Claim 15
Dependent Claim
24 words
The method of claim 5, wherein said grouping step includes the use of a set of parameterized rules applied to the graphical primitive objects.
Claim 16
Dependent Claim
14 words
The method of claim 14, wherein said rules are applied in a predetermined order.
Claim 17
Dependent Claim
26 words
The method of claim 14, wherein said rules are applied iteratively such that the object grouping is continually refined by the iterative application of said rules.
Claim 18
Dependent Claim
18 words
The method of claim 14, wherein a predetermined tolerance parameter is used in the application of said rules.
Claim 19
Dependent Claim
17 words
The method of claim 5, wherein said grouping step comprises the step of creating rectangles from lines.
Claim 20
Dependent Claim
92 words
The method of claim 18, wherein said lines include horizontal and vertical lines, and said step of creating rectangles further comprises the following steps: sorting said lines by vertical position; selecting a first vertical line; selecting a first horizontal line having a left end which matches a top end of said selected first vertical line; and selecting a second vertical line having a top end which matches a right end of said selected first horizontal line, and having a bottom end which matches a bottom end of said selected first vertical line.
Claim 21
Dependent Claim
43 words
The method of claim 19, wherein said step of creating rectangles further comprises the following step: selecting a second horizontal line which extends from the bottom end of said selected first vertical line to the bottom end of said selected second vertical line.
Claim 22
Dependent Claim
27 words
The method of claim 20, wherein said selected lines have colors, said colors being used to assign one of a raised and lowered attribute to said rectangle.
Claim 23
Dependent Claim
25 words
The method of claim 20, wherein said step of creating rectangles is repeated and a list is assembled of the rectangles constructed from individual lines.
Claim 24
Dependent Claim
19 words
The method of claim 22, wherein said list of constructed rectangles is merged with a list of drawn rectangles.
Claim 25
Dependent Claim
25 words
The method of claim 19, wherein said first vertical line, said second vertical line and said first horizontal line identify a three-sided page list control.
Claim 26
Dependent Claim
112 words
The method of claim 24, wherein a plurality of page list controls are grouped in accordance with the following steps: the page list controls are sorted by the vertical position of a missing edge in each page list control; the page list controls are sorted from left to right; grouping two page list controls if at least one of the following conditions is met: the two page list controls are bottom aligned; the two page list controls are within a predetermined distance of each other; and each of the page list controls is labelled with a single text object or a plurality of overlapping text objects used to create a three-dimensional effect.
Claim 27
Dependent Claim
20 words
The method of claim 5, wherein said grouping step comprises the step of merging adjacent images into a single object.
Claim 28
Dependent Claim
34 words
The method of claim 26, wherein said merging step merges two images having a same top and bottom edge, and wherein a right edge of one image matches a left edge of another image.
Claim 29
Dependent Claim
33 words
The method of claim 26, wherein said merging step merges two images having matching left and right edges, and wherein a bottom edge of one image matches a top edge of another image.
Claim 30
Dependent Claim
20 words
The method of claim 5, wherein said grouping step comprises the step of merging text objects into a single object.
Claim 31
Dependent Claim
26 words
The method of claim 29, wherein said step of merging text objects is repeated to merge a plurality of text objects into a single text object.
Claim 32
Dependent Claim
80 words
The method of claim 29, wherein said step of merging text objects further comprises the following step: merging at least one of a list number and a bullet with a text string to produce a list object, wherein the text string meets at least one of the following conditions: the text string is positioned within a predetermined distance to the right of said list number or bullet; and the text string is center aligned with said list number or bullet.
Claim 33
Dependent Claim
24 words
The method of claim 31, wherein said list objects are grouped into a list if they are within a predetermined distance of each other.
Claim 34
Dependent Claim
52 words
The method of claim 32, wherein said step of grouping list objects continues until at least one of the following conditions is met: a non-list object is encountered; a horizontal line is encountered; and a list object is encountered which begins to the left of the first list object in the group.
Claim 35
Dependent Claim
82 words
The method of claim 31, wherein said step of merging text objects includes the step of merging a text object ending with a colon with an adjacent text object, if at least one of the following conditions is met: the text object is below the text object ending with the colon; the text object is within a predetermined distance of the text object ending with a colon; and the text object is left aligned with the text object ending with a colon.
Claim 36
Dependent Claim
23 words
The method of claim 29, wherein said step of merging text objects further comprises the following step: merging two objects which overlap vertically.
Claim 37
Dependent Claim
74 words
The method of claim 29, wherein said step of merging text objects further comprises the following step: merging two text objects which are stacked, wherein a bottom of one text object is in the vicinity of a top of another text object, if the two text objects satisfy at least one of the following conditions: the text objects are left aligned; the text objects are right aligned; and the text objects are center aligned.
Claim 38
Dependent Claim
47 words
The method of claim 36, wherein the step of merging two text objects is prevented if at least one of the following conditions is met: a line is positioned between said objects; said objects end with a colon; and only one object is in bold face type.
Claim 39
Independent Claim
203 words
A method for recognizing graphical objects and grouping said graphical objects into a table having at least one row and at least one column, said method comprising the following steps: identifying graphical objects as graphical object primitives; grouping the graphical primitive objects in a window using an object grouping process thread in order to create virtual objects, said grouping step comprising the step of creating rectangles from lines; constructing a list of cells in said window, wherein said cells comprise rectangles; sorting said cells by their vertical position, and then left to right for cells with the same vertical position; dividing said cells into rows based on the vertical positions of said cells, the cells on a given row having substantially similar vertical positions, said dividing step including the step of identifying a row break when the top of a cell being considered is below the bottom of a cell in the row having the highest bottom position; and creating links for at least one cell in the table, said links including a below link which points to the next cell in the same column, and a right link which points to the cell which is the next cell in the same row.
Claim 40
Independent Claim
203 words
A method for recognizing graphical objects and grouping said graphical objects into a table having at least one row and at least one column, said method comprising the following steps: identifying graphical objects as graphical object primitives; grouping the graphical primitive objects in a window using an object grouping process thread in order to create virtual objects, said grouping step comprising the step of creating rectangles from lines; constructing a list of cells in said window, wherein said cells comprise rectangles; sorting said cells by their vertical position, and then left to right for cells with the same vertical position; dividing said cells into rows based on the vertical positions of said cells, the cells on a given row having substantially similar vertical positions, said dividing step including the step of identifying a row break when the top of a cell being considered is below the bottom of a cell in the row having the highest bottom position; and creating links for at least one cell in the table, said links including a below link which points to the next cell in the same column, and a right link which points to the cell which is the next cell in the same row.
Claim 41
Dependent Claim
33 words
The method of claim 38, further comprising the step of associating a scroll bar in said window with said table, wherein said associated scroll bar is located at an edge of the table.
Claim 42
Dependent Claim
82 words
The method of claim 38, wherein the step of dividing cells into rows includes the additional steps of locating a horizontal center line which maximizes the number of cells in a row, and adjusting the cells in a row to have substantially the same top position corresponding to the top of the leftmost cell in the row, and adjusting the cells in a row to have substantially the same bottom position corresponding to the bottom-most position of the cells in the row.
Claim 43
Dependent Claim
32 words
The method of claim 38, further comprising the step of identifying gaps corresponding to missing cells in said table by examining the below links for at least one row of said table.
Claim 44
Dependent Claim
36 words
The method of claim 41, further comprising the steps of inserting an empty cell corresponding to each said identified gap, and adjusting the links of the remaining cells to reflect the insertion of each empty cell.
Claim 45
Dependent Claim
72 words
The method of claim 42, wherein said inserting step includes the insertion of an empty cell in a gap corresponding to the left-most position of the top row of said table, said left-most gap being identified by a below link of the first cell in a row which is a target of a right link and a source of the right link and the target of the below link are top aligned.
Claim 46
Dependent Claim
17 words
The method of claim 38, further comprising the step of identifying a starting cell for said table.
Claim 47
Dependent Claim
36 words
The method of claim 38, wherein said cells are arranged in columns wherein each cell in a column is substantially left aligned with other cells in the column and located on different rows of said table.
Claim 48
Dependent Claim
59 words
The method of claim 38, further comprising the following steps: identifying a spanning cell which is left aligned with a first cell on another row and which is right aligned with a second cell on the same other row; and splitting said spanning cell into two or more individual cells each containing the same information as the spanning cell.
Claim 49
Dependent Claim
28 words
The method of claim 38, further comprising the step of merging a first cell on a first row which substantially overlaps a second cell on a second row.
Claim 50
Dependent Claim
63 words
The method of claim 38, wherein the step of dividing cells into rows includes the step of identifying an end of a first row of said table and advancing to a second row of said table when at least the following condition is met: each cell in the first row has a below link to a cell which is within a predetermined distance.
Claim 51
Dependent Claim
86 words
The method of claim 48, further comprising the step of identifying the first row as a table heading if at least one of the following conditions is met: the first row contains text which is all in bold face type and the second row does not contain all bold face type text; and the first row contains text that is all of a first color and the second row contains text which is all of a second color and said first and second colors are different.
Claim 52
Dependent Claim
29 words
The method of claim 48, wherein an indication of a cell table is reported if the number of rows in the table exceeds a predetermined minimum number of rows.
Claim 53
Dependent Claim
34 words
The method of claim 48, wherein a column object is constructed corresponding to each column of the table, and wherein the cells of a column object are identified as children of the column object.
Claim 54
Dependent Claim
79 words
The method of claim 48, further comprising the following steps: identifying a table heading by examining text objects that are within a predetermined distance of the top of said table and substantially centered about said table; identifying a scroll bar associated with said table by selecting a scroll bar having a bottom which is substantially aligned with the bottom of said table, and a top which is substantially aligned with the heading or the first row of the table.
Claim 55
Dependent Claim
12 words
The method of claim 38, wherein said table is a cell table.
Claim 56
Dependent Claim
12 words
The method of claim 38, wherein said table is a text table.
Claim 57
Dependent Claim
15 words
The method of claim 54, wherein the elements in each row are substantially center aligned.
References Cited (Backward Citations)
US5018219A
Cited
US5020112A
Cited
US5233611A
Cited
US5379375A
Cited
US5329596A
Cited
US5781720A
Cited
US5596714A
Cited
US5787194A
Cited
US5886694A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2003-04-04 | 2003-10-04 | 2004-04-04 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2007-04-04 | 2007-10-04 | 2008-04-04 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2011-04-04 | 2011-10-04 | 2012-04-04 | $7700.00 | Window Closed / Lapsed |