Augmented reality maintenance system employing manipulator arm with archive and comparison device

Claim 2

A remote maintenance system for inspection and identifying differences in structures in a predetermined region over time comprising: a) a manipulator arm having 1. a utility package, for inspecting said structures according to selected acquisition settings, and for creating a present multi-dimensional sensor signal indicating physical properties of said structures, and 2. actuators capable of maneuvering the manipulator arm responsive to user-selected input; b) environmental model storage device capable of retaining computer graphic information regarding said structures; c) a position and attitude (P&A) sensing unit for measuring a position and orientation of the manipulator arm; d) environment modeler for receiving the computer graphic information defining said structures and producing a conventional computer generated model of said structures and storing the model in the environment model storage device; e) environment renderer coupled to the environment model storage device for generating an image of said structures from the environment model as viewed from a user-selected viewpoint; f) manipulator arm renderer coupled to the P&A sensing unit for generating a conventional computer generated image of the manipulator arm at a position and orientation corresponding to that of the manipulator arm, generated from a prestored model of the manipulator arm as viewed from a user-selected viewpoint; g) monitor for displaying images provided to it; h) video mixer coupled to the environment renderer and the manipulator arm renderer for displaying the images on the monitor, thereby allowing an operator to visualize the position of the manipulator arm relative to its environment; and i) an archive and comparison (A&C) device coupled to the video mixer and the P&A sensing unit and utility package, operating to: i. archive the present multi-dimensional sensor signal from the utility package along with its acquisition settings, ii. retrieve a previously archived multi-dimensional sensor signal measuring the same structure and its physical properties, and its associated acquisition settings, iii. rectify the present and archived multi-dimensional sensor signals to have the same acquisition settings, iv. compare the rectified signals to identify and characterize differences, and v. display differences and archived signals on the monitor.

Claim 3

The remote maintenance system of claim 1 further comprising offset computation device coupled to the P&A sensing unit which provides the viewpoint, being a predetermined offset from a user-selected location and orientation of the manipulator arm, as measured by the P&A sensing unit, the viewpoint being employed by the environment renderer and the manipulator arm renderer in image rendering.

Claim 4

A remote maintenance system for inspection and repair of structures in a predetermined region comprising: a) a manipulator arm having 1. a utility package, for inspecting said structures according to selected acquisition settings, and for creating a present multi-dimensional sensor signal indicating physical properties of said structures; 2. actuators capable of maneuvering the manipulator arm responsive to user-selected input; b) environmental model storage device capable of retaining computer graphic information regarding said structures; c) a position and attitude (P&A) sensing unit for measuring a position and orientation of the manipulator arm; d) environment modeler for receiving the computer graphic information defining said structures and producing a 3D computer model of said structures and storing the model in the environment model storage device; e) environment renderer coupled to the environment model storage device for generating an image of said structures from the environment model as viewed from a user-selected viewpoint; f) manipulator arm renderer coupled to the P&A sensing unit for generating a computer generated image of the manipulator arm at a position and orientation corresponding to that of the manipulator arm, generated from a prestored 3D model of the manipulator arm as viewed from a user-selected viewpoint; g) monitor for displaying images provided to it; h) video mixer coupled to the environment renderer and the manipulator aim renderer for displaying the images on the monitor, thereby allowing an operator to visualize the position of the manipulator arm relative to its environment; i) sensor data storage unit for archiving a plurality of utility package signals and corresponding acquisition settings; j) retrieval computer coupled to the sensor data storage unit operating to: i. receive user-defined criteria indicating archived multi-dimensional sensor signals to be selected, ii. select an archived multi-dimensional sensor signal and its acquisition settings from the sensor data storage unit which fit criteria set by the operator, iii. receive the present multi-dimensional sensor signal and acquisition settings from the manipulator arm utility package, and 4.

Claim 5

A remote maintenance system for inspection and repair of structures in a predetermined region comprising: a) a manipulator arm having 1. a utility package, for inspecting said structures according to selected acquisition settings, and for creating a present multi-dimensional sensor signal indicating physical properties of said structures; 2. actuators capable of maneuvering the manipulator arm responsive to user-selected input; b) environmental model storage device capable of retaining computer graphic information regarding said structures; c) a position and attitude (P&A) sensing unit for measuring a position and orientation of the manipulator arm; d) environment modeler for receiving the computer graphic information defining said structures and producing a 3D computer model of said structures and storing the model in the environment model storage device; e) environment renderer coupled to the environment model storage device for generating an image of said structures from the environment model as viewed from a user-selected viewpoint; f) manipulator arm renderer coupled to the P&A sensing unit for generating a computer generated image of the manipulator arm at a position and orientation corresponding to that of the manipulator arm, generated from a prestored 3D model of the manipulator arm as viewed from a user-selected viewpoint; g) monitor for displaying images provided to it; h) video mixer coupled to the environment renderer and the manipulator aim renderer for displaying the images on the monitor, thereby allowing an operator to visualize the position of the manipulator arm relative to its environment; i) sensor data storage unit for archiving a plurality of utility package signals and corresponding acquisition settings; j) retrieval computer coupled to the sensor data storage unit operating to: i. receive user-defined criteria indicating archived multi-dimensional sensor signals to be selected, ii. select an archived multi-dimensional sensor signal and its acquisition settings from the sensor data storage unit which fit criteria set by the operator, iii. receive the present multi-dimensional sensor signal and acquisition settings from the manipulator arm utility package, and 4.

Claim 6

recitify the archived multi-dimensional sensor signal according to the present multi-dimensional sensor signal acquisition settings to result in a rectified signal; and k) signal processing and visualization (SP&V) unit coupled to the retrieval computer operating to: i. compare the rectified signal to the present multi-dimensional sensor signal to identify and characterize differences, and

Claim 7

recitify the archived multi-dimensional sensor signal according to the present multi-dimensional sensor signal acquisition settings to result in a rectified signal; and k) signal processing and visualization (SP&V) unit coupled to the retrieval computer operating to: i. compare the rectified signal to the present multi-dimensional sensor signal to identify and characterize differences, and

Claim 8

ii. producing a visual image of the differences to a user for analysis. 4. A remote maintenance system for inspection and repair of structures in a predetermined region comprising: a) a manipulator arm having 1. a utility package, for inspecting said structures according to selected acquisition settings, and for creating a present multi-dimensional sensor signal indicating physical properties of said structures; 2. actuators capable of maneuvering the manipulator arm responsive to user-selected input; b) environmental model storage device capable of retaining computer graphic information regarding said structures; c) a position and attitude (P&A) sensing unit for measuring a position and orientation of the manipulator arm; d) environment modeler for receiving the computer graphic information defining said structures and producing a 3D computer model of said structures and storing the model in the environment model storage device; e) environment renderer coupled to the environment model storage device for generating an image of said structures from the environment model as viewed from a user-selected viewpoint; f) manipulator arm renderer coupled to the P&A sensing unit for generating a conventional computer generated image of the manipulator arm at a position and orientation corresponding to that of the manipulator arm, generated from a prestored 3D computer model of the manipulator arm as viewed from a user-selected viewpoint; g) monitor for displaying images provided to it; h) video mixer coupled to the environment renderer and the manipulator arm renderer for displaying the images on the monitor, thereby allowing an operator to visualize the position of the manipulator arm relative to its environment; i) sensor data storage unit for archiving a plurality of multi-dimensional sensor signals and corresponding acquisition settings; j) retrieval computer coupled to the sensor data storage unit operating to: i. receive user-selected criteria defining archived utility package signals to be selected, ii. select an archived multi-dimensional sensor signal and its acquisition settings from the sensor data storage unit which fit criteria set by the operator, iii. select a first and second archived multi-dimensional sensor signal and corresponding acquisition settings, and iv. rectify the first archived multi-dimensional sensor signal according to the second archived multi-dimensional sensor signal acquisition settings to result in a rectified signal; and k) signal processing and visualization (SP&V) unit coupled to the retrieval computer operating to: i. compare the rectified signal to the second archived multi-dimensional sensor signal to identify and characterize differences, and ii. produce a visual image of the differences to the operator for analysis.

Claim 9

ii. producing a visual image of the differences to a user for analysis. 4. A remote maintenance system for inspection and repair of structures in a predetermined region comprising: a) a manipulator arm having 1. a utility package, for inspecting said structures according to selected acquisition settings, and for creating a present multi-dimensional sensor signal indicating physical properties of said structures; 2. actuators capable of maneuvering the manipulator arm responsive to user-selected input; b) environmental model storage device capable of retaining computer graphic information regarding said structures; c) a position and attitude (P&A) sensing unit for measuring a position and orientation of the manipulator arm; d) environment modeler for receiving the computer graphic information defining said structures and producing a 3D computer model of said structures and storing the model in the environment model storage device; e) environment renderer coupled to the environment model storage device for generating an image of said structures from the environment model as viewed from a user-selected viewpoint; f) manipulator arm renderer coupled to the P&A sensing unit for generating a conventional computer generated image of the manipulator arm at a position and orientation corresponding to that of the manipulator arm, generated from a prestored 3D computer model of the manipulator arm as viewed from a user-selected viewpoint; g) monitor for displaying images provided to it; h) video mixer coupled to the environment renderer and the manipulator arm renderer for displaying the images on the monitor, thereby allowing an operator to visualize the position of the manipulator arm relative to its environment; i) sensor data storage unit for archiving a plurality of multi-dimensional sensor signals and corresponding acquisition settings; j) retrieval computer coupled to the sensor data storage unit operating to: i. receive user-selected criteria defining archived utility package signals to be selected, ii. select an archived multi-dimensional sensor signal and its acquisition settings from the sensor data storage unit which fit criteria set by the operator, iii. select a first and second archived multi-dimensional sensor signal and corresponding acquisition settings, and iv. rectify the first archived multi-dimensional sensor signal according to the second archived multi-dimensional sensor signal acquisition settings to result in a rectified signal; and k) signal processing and visualization (SP&V) unit coupled to the retrieval computer operating to: i. compare the rectified signal to the second archived multi-dimensional sensor signal to identify and characterize differences, and ii. produce a visual image of the differences to the operator for analysis.

Claim 10

The remote maintenance system of claim 4 wherein the utility package further comprises a repair device connected to the manipulator arm for effecting repair of a selected structure.