Apparatus and method for tracking handwriting from visual input

Claim 3

The system as in claim 1 wherein said camera monitors the visual input of a user writing on a piece of paper.

Claim 4

A system as in claim 1 wherein said camera monitors a visual input of a user writing on a writable board.

Claim 5

A system as in claim 1 wherein said image processor includes a handwriting recognition unit, and further comprising an output unit, coupled to said image processor, which displays handwriting-recognized information.

Claim 6

A system as in claim 1 wherein said tracking element processes said image signals by: 1) determining the portion that represents a tip of the writing implement in said image signal; and 2) following motion of said portion to determine a path defined thereby.

Claim 7

A system as in claim 5 wherein said tracking element determines a motion of a centroid of said tip, and wherein said tracking element also corrects for an offset between a centroid of the tip and an actual position of the tip.

Claim 8

A system as in claim 5 wherein said processing element detects the tip by accepting a manual input indicating the position of the tip.

Claim 9

A system as in claim 5 further comprising a template memory storing at least one image template indicating an image portion of a specific writing implement pen tip, and wherein said tracking element correlates said template with said image signal to determine a position of a tip matching said template within said image signal.

Claim 10

A system as in claim 8 wherein said template memory stores a plurality of said image templates, and said tracking element searches said image signal to determine any of said plurality of said image templates in said image signal.

Claim 11

A system as in claim 5 wherein said image element detects said tip by detecting an area in the image signal which has maximum motion.

Claim 12

A system as in claim 5 wherein said tracking element follows motion by correlating a kernel image of the tip to positions of matching features in said image signal.

Claim 13

A system as in claim 11 wherein said tracking element carries out said correlating by determining a kernel indicative of a shape of said tip; multiplying said kernel by a plurality of portions of the image signal; and taking a maximum of said multiplications by said portions as being a correct position.

Claim 14

A system as in claim 11 wherein said correlating monitors positions in the image signal which are within a predetermined distance of a last position in the image signal.

Claim 15

A system as in claim 11 wherein said tracking element also carries out a prediction operation to predict a most likely position of the tip, said correlating carried out in said most likely position of the tip.

Claim 16

A system as in claim 11, further comprising a pyramid filter which filters said image signal to produce a reduced-resolution image signal; wherein said correlating is carried out between said reduced-resolution image signal and said kernel.

Claim 17

A system as in claim 1, further comprising a memory for storing frames of information from said camera, said frames obtained at spaced time increments.

Claim 18

A system as in claim 16, wherein said tracking element determines a first condition of said writing implement relative to said writing surface, and does not store information in said memory while said first condition is determined to exist.

Claim 19

A system as in claim 17, wherein said first condition is said writing implement being spaced from said writing surface.

Claim 20

A writing processing system, comprising: a first device which obtains video images of a part of a writing implement relative to a writing surface; a processor, receiving said video images from said first device, and processing said video images to determine a position of the writing implement therein over multiple frames defined by said video images, to produce a tracking signal indicative of position and/or movement of the writing implement over said multiple frames, said processor recognizing different kinds of writing implements by storing a plurality of different template images, said different template images including different shapes of writing implement tips therein, determining a template image indicating a tip of a writing implement within a frame of the video image, and tracking a position of said template image within said frame, and said tracking signal indicating position and/or movement of said any of said different kinds of writing implements including a writing implement which is dark relative to its background; and an output unit, which produces an output signal indicative of said tracking signal.

Claim 21

A writing processing system, comprising: a first device which obtains video images of a part of a writing implement relative to a writing surface; a processor, receiving said video images from said first device, and processing said video images to determine a position of the writing implement therein over multiple frames defined by said video images, to produce a tracking signal indicative of position and/or movement of the writing implement over said multiple frames, said processor recognizing different kinds of writing implements by storing a plurality of different template images, said different template images including different shapes of writing implement tips therein, determining a template image indicating a tip of a writing implement within a frame of the video image, and tracking a position of said template image within said frame, and said tracking signal indicating position and/or movement of said any of said different kinds of writing implements including a writing implement which is dark relative to its background; and an output unit, which produces an output signal indicative of said tracking signal.

Claim 22

The system as in claim 19 wherein said first device monitors the visual input of a user writing on a piece of paper.

Claim 23

A system as in claim 19 wherein said first device monitors a visual input of a user writing on a writable board.

Claim 24

A system as in claim 19 wherein said output unit includes a handwriting recognizer, which recognizes handwriting based on said tracking signal and which displays recognized information.

Claim 25

A system as in claim 19 wherein said processor processes said video image by: 1) determining an image of a tip of the writing implement in said video images; and 2) following motion of said tip to determine a path defined thereby.

Claim 26

A system as in claim 23 wherein said processor follows a motion of a centroid near the tip, and wherein said processing element also corrects for an offset between a centroid of the tip and an actual position of the tip.

Claim 27

A system as in claim 23 wherein said processor detects the tip by accepting a manual input indicating the position of the tip.

Claim 28

A system as in claim 23 further comprising a template memory storing at least one image template indicating a image portion of a specific writing implement tip, and wherein said processing element correlates said template with said video image to determine a position of a tip matching said template within said video images.

Claim 29

A system as in claim 26 wherein said template memory stores a plurality of said templates, and said processing element searches said video images to determine any of said plurality of said templates in said video images.

Claim 30

A system as in claim 23 wherein said processing element detects said tip by detecting an area in the video images which has maximum motion.

Claim 31

A system as in claim 23 wherein said processing element follows motion by correlating a kernel image of the tip to positions of matching features in the video image.

Claim 32

A system as in claim 29 wherein said processing element carries out said correlation by determining a kernel indicative of a shape of said pen tip, multiplying said kernel by a plurality of portions in the video image; and taking a maximum of said multiplications by said portion as being a correct position.

Claim 33

A system as in claim 29 wherein said correlating monitors positions in the video image which are within a predetermined distance of a last position in the video image.

Claim 34

A system as in claim 29 wherein said processor also carries out a prediction operation to predict a most likely position of the tip, said correlation carried out in said most likely position of the tip.

Claim 35

A system as in claim 19 wherein said processor operates by determining an area of the video image which has maximum motion, determining a centroid of said area of the image which has the maximum motion, and determining a distance between a position of a tip of the writing implement and the centroid.

Claim 36

A system as in claim 33 wherein said maximum motion is obtained by using a low pass filter to blur the image and find the maximum motion position.

Claim 37

A system as in claim 26 wherein said processing element finds an area of maximum motion of the image if no portion of the image matches a template.

Claim 38

A system as in claim 19 wherein said processing element determines whether the writing implement is touching the writing surface, and said output indicative of movement of the writing implement is produced only when said writing implement is touching the writing surface.

Claim 39

A system as in claim 36 further comprising a second camera, located in a different position spaced from the image, to triangulate x, y and z positions; said x and y positions used for outputting positions of movement, and said z position used for determining whether the writing implement is touching the surface.

Claim 40

A system as in claim 32 wherein said prediction is carried out using a Kalman filter.

Claim 41

A system as in claim 19, wherein said processing element determines a first condition of said writing implement relative to said writing surface, and does not store information in said memory while said first condition is determined.

Claim 42

A method as in claim 39, wherein said desired point in said video image is a position of a tip of the writing implement, and said center point is a centroid of the kernel.

Claim 43

A method as in claim 40, wherein said tracing comprises obtaining a video image of a writing implement relative to a writing surface at a first time; determining a portion of said image representing a kernel to be tracked in position; obtaining a position of said kernel in said video image at said first time; obtaining a video image of the writing implement relative to the writing surface at a second time; obtaining a position of said kernel in said video image at said second time; determining a distance between a desired point in said video image to be tracked, and a center point of said kernel; and correcting said positions using said distance, and determining a path between said first and second positions.