US7536029B2 — Apparatus and method performing audio-video senso…

Claim 1

Independent Claim 241 words

An apparatus for tracking and identifying objects using received sounds and video, comprising: an audio likelihood module which determines corresponding audio likelihoods for each of a plurality of the sounds received from corresponding different directions based on a signal subspace and noise subspace approach, with a spatial covariance matrix that is updated only when target audio is absent, considering together a respective audio source vector, measurement noise vector, and a transform function matrix including predefined steering vectors representing attenuation and delay reflecting propagation of audio at respective directions to at least two audio sensors, each audio likelihood indicating a likelihood the sound is an object to be tracked; a video likelihood module which determines video likelihoods for each of a plurality of images disposed in corresponding different directions in the video, each video likelihood indicating a likelihood that the image in the video is an object to be tracked; and an identification and tracking module which: determines correspondences between the audio likelihoods and the video likelihoods, if a correspondence is determined to exist between one of the audio likelihoods and one of the video likelihoods, identifies and tracks a corresponding one of the objects using each determined pair of audio and video likelihoods, and if a correspondence does not exist between a corresponding one of the audio likelihoods and a corresponding one of the video likelihoods, identifies a source of the sound or image as not being an object to tracked.

Claim 2

Independent Claim 241 words

An apparatus for tracking and identifying objects using received sounds and video, comprising: an audio likelihood module which determines corresponding audio likelihoods for each of a plurality of the sounds received from corresponding different directions based on a signal subspace and noise subspace approach, with a spatial covariance matrix that is updated only when target audio is absent, considering together a respective audio source vector, measurement noise vector, and a transform function matrix including predefined steering vectors representing attenuation and delay reflecting propagation of audio at respective directions to at least two audio sensors, each audio likelihood indicating a likelihood the sound is an object to be tracked; a video likelihood module which determines video likelihoods for each of a plurality of images disposed in corresponding different directions in the video, each video likelihood indicating a likelihood that the image in the video is an object to be tracked; and an identification and tracking module which: determines correspondences between the audio likelihoods and the video likelihoods, if a correspondence is determined to exist between one of the audio likelihoods and one of the video likelihoods, identifies and tracks a corresponding one of the objects using each determined pair of audio and video likelihoods, and if a correspondence does not exist between a corresponding one of the audio likelihoods and a corresponding one of the video likelihoods, identifies a source of the sound or image as not being an object to tracked.

Claim 3

Dependent Claim 40 words

The apparatus of claim 1 , wherein, when the identification and tracking module determines a correspondence between multiple pairs of audio and video likelihoods, the identification and tracking module identifies and individually tracks objects corresponding to each of the pairs.

Claim 4

Dependent Claim 21 words

The apparatus of claim 2 , wherein the identification and tracking module identifies and tracks a location of each determined pair.

Claim 5

Dependent Claim 36 words

The apparatus of claim 1 , wherein, for each image in the received video, the video likelihood module compares the image against a pre-selected image profile in order to determine the video likelihood for the image.

Claim 6

Dependent Claim 45 words

The apparatus of claim 4 , wherein the pre-selected image profile comprises a color of an object to be tracked, and the video likelihood module compares a color of portions of the image in order to identify features indicative of an object to be tracked.

Claim 7

Dependent Claim 53 words

The apparatus of claim 4 , wherein the pre-selected image profile comprises a shape of an object to be tracked, and the video likelihood module detects an outer edge of each image and compares the outer edge of each image against the shape to identify features indicative of an object to be tracked.

Claim 8

Dependent Claim 45 words

The apparatus of claim 6 , wherein the pre-selected image profile further comprises poses for the object to be tracked, and the video likelihood module further compares each outer edge against each of the poses to identify features indicative of the object to be tracked.

Claim 9

Dependent Claim 45 words

The apparatus of claim 7 , wherein the pre-selected image profile comprises a color of an object to be tracked, and the video likelihood module compares a color of portions of the image in order to identify features indicative of an object to be tracked.

Claim 10

Dependent Claim 46 words

The apparatus of claim 8 , wherein the video likelihood module uses the identified images to normalize each outer edge in order to be closer to a size of the poses and the shape in order to identify features indicative of an object to be tracked.

Claim 11

Dependent Claim 35 words

The apparatus of claim 9 , wherein the video likelihood identifies an outer edge as not being an object to be tracked if the outer edge does not correspond to the shape and the poses.

Claim 12

Dependent Claim 31 words

The apparatus of claim 9 , wherein the video likelihood identifies an outer edge as not being an object to be tracked if the outer edge does not include the color.

Claim 13

Dependent Claim 67 words

The apparatus of claim 1 , wherein: a first one of the objects is disposed in a first direction, a second one of the objects is disposed in a second direction, and based on the correspondences between the audio and video likelihoods, the identification and tracking module identifies the first object as being in the first direction and the second object as being in the second direction.

Claim 14

Dependent Claim 26 words

The apparatus of claim 12 , wherein the identification and tracking module tracks the first object as the first object moves relative to the second object.

Claim 15

Dependent Claim 57 words

The apparatus of claim 13 , wherein: the video likelihood module receives the video included the images from a camera, and the identification and tracking module tracks and identifies the first object as the first object moves relative to the second object such that the first object crosses the second object from a perspective of the camera.

Claim 16

Dependent Claim 51 words

The apparatus of claim 1 , further comprising a beam-former which, for each identified object, determines a location of the identified object, and separates from the received sounds audio corresponding to a location of each identified object so as to output audio channels corresponding uniquely to each of the identified objects.

Claim 17

Dependent Claim 60 words

The apparatus of claim 15 , wherein: the apparatus receives the sounds using a microphone array outputting a first number of received audio channels, each received audio channel includes an element of the sounds, the beam-former outputs a second number of the audio channels other than the first number, and the second number corresponds to the number of identified objects.

Claim 18

Dependent Claim 30 words

The apparatus of claim 16 , further comprising a recording apparatus which records each beam formed audio channel for each identified object as separate audio tracks associated with each object.

Claim 19

Dependent Claim 55 words

The apparatus of claim 15 , wherein: each output channel includes audible periods in which speech is detected and silent periods between corresponding audible periods in which speech is not detected, and the apparatus further comprises a speech interval detector which detects, for each output channel, a start and stop time for each audible period.

Claim 20

Dependent Claim 81 words

The apparatus of claim 18 , wherein the speech interval detector further: detects a proximity between adjacent audible periods, if the proximity is less than a predetermined amount, determines that the adjacent audible periods are one continuous audible period and connects the adjacent audible periods to form the continuous audible period, and if the proximity is more than the predetermined amount, determines that the adjacent audible periods are separated by the silent period and does not connect the adjacent audible periods.

Claim 21

Dependent Claim 69 words

The apparatus of claim 18 , wherein the speech interval detector further: detects a length of each audible period, if the length is less than a predetermined amount, determines that the audible period is a silent period and erases the audible period, and if the length is more than the predetermined amount, determines that the audible period is not a silent period and does not erase the audible period.

Claim 22

Dependent Claim 32 words

The apparatus of claim 18 , wherein the speech interval detector further: for each audible period, outputs the detected speech, and for each silent period, deletes the sound from the audio channel.

Claim 23

Dependent Claim 42 words

The apparatus of claim 15 , further comprising a post processor which, for each of plural audio channels received from the beam-former, detects audio portions related to cross channel interference caused by the remaining audio channels and removes the cross channel interference.

Claim 24

Dependent Claim 20 words

The apparatus of claim 1 , further comprising a controller which controls a robotic element according to the identified object.

Claim 25

Dependent Claim 25 words

The apparatus of claim 23 , wherein the robotic element comprises at least one motor used to move the apparatus according to the identified object.

Claim 26

Dependent Claim 33 words

The apparatus of claim 23 , wherein the robotic element comprises at least one motor used to remotely move an element connected to the apparatus through an interface according to the identified object.

Claim 27

Dependent Claim 23 words

The apparatus of claim 1 , further comprising an omnidirectional camera which outputs a 360° panoramic view image to the video likelihood module.

Claim 28

Dependent Claim 36 words

The apparatus of claim 1 , further comprising at least one limited field of view camera which outputs an image to the video likelihood module which has a field of view that is less than 360°.

Claim 29

Dependent Claim 67 words

The apparatus of claim 1 , wherein: the audio likelihood module further detects, for each received sound, an audio direction from which a corresponding sound is received, the video likelihood module further detects, for each image, a video direction from which the image is observed, and the identification and tracking module further determines the correspondences based upon a correspondence between the audio directions and the video directions.

Claim 30

Dependent Claim 23 words

The apparatus of claim 1 , wherein the video received by the video likelihood module is an infrared video received from a pyrosensor.

Claim 31

Independent Claim 244 words

A method of tracking and identifying objects using at least one computer receiving audio and video data, the method comprising: for each of a plurality of sounds received from corresponding different directions, determining in the at least one computer corresponding audio likelihoods based on a signal subspace and noise subspace approach, with a spatial covariance matrix that is updated only when target audio is absent, considering together a respective audio source vector, measurement noise vector, and a transform function matrix including predefined steering vectors representing attenuation and delay reflecting propagation of audio at respective directions to at least two audio sensors, each audio likelihood indicating a likelihood the sound is an object to be tracked; for each of a plurality of images disposed in corresponding different directions in a video, determining in the at least one computer video likelihoods, each video likelihood indicating a likelihood that the image in the video is an object to be tracked; if a correspondence is determined to exist between one of the audio likelihoods and one of the video likelihoods, identifying and tracking in the at least one computer a corresponding one of the objects using each determined pair of audio and video likelihoods, and if a correspondence does not exist between a corresponding one of the audio likelihoods and a corresponding one of the video likelihoods, identifying in the at least one computer a source of the sound or image as not being an object to tracked.

Claim 32

Independent Claim 244 words

A method of tracking and identifying objects using at least one computer receiving audio and video data, the method comprising: for each of a plurality of sounds received from corresponding different directions, determining in the at least one computer corresponding audio likelihoods based on a signal subspace and noise subspace approach, with a spatial covariance matrix that is updated only when target audio is absent, considering together a respective audio source vector, measurement noise vector, and a transform function matrix including predefined steering vectors representing attenuation and delay reflecting propagation of audio at respective directions to at least two audio sensors, each audio likelihood indicating a likelihood the sound is an object to be tracked; for each of a plurality of images disposed in corresponding different directions in a video, determining in the at least one computer video likelihoods, each video likelihood indicating a likelihood that the image in the video is an object to be tracked; if a correspondence is determined to exist between one of the audio likelihoods and one of the video likelihoods, identifying and tracking in the at least one computer a corresponding one of the objects using each determined pair of audio and video likelihoods, and if a correspondence does not exist between a corresponding one of the audio likelihoods and a corresponding one of the video likelihoods, identifying in the at least one computer a source of the sound or image as not being an object to tracked.

Claim 33

Dependent Claim 35 words

The method of claim 30 , further comprising determining correspondences between multiple pairs of audio and video likelihoods, wherein the identifying and tracking comprises individually tracking the objects corresponding to each of the determined pairs.

Claim 34

Dependent Claim 34 words

The method of claim 30 , wherein the determining the video likelihoods comprises comparing each of the images against a pre-selected image profile in order to determine the video likelihoods for the corresponding images.

Claim 35

Dependent Claim 19 words

The method of claim 32 , further comprising, for each determined pair, identifying a location of each determined pair.

Claim 36

Dependent Claim 47 words

The method of claim 33 , wherein the pre-selected image profile comprises a color of an object to be tracked, and the determining the video likelihoods comprises comparing a color of portions of the image in order to identify features indicative of an object to be tracked.

Claim 37

Dependent Claim 55 words

The method of claim 33 , wherein the pre-selected image profile comprises a shape of an object to be tracked, and the determining the video likelihoods comprises detecting an outer edge of each image and comparing the outer edge of each image against the shape to identify features indicative of an object to be tracked.

Claim 38

Dependent Claim 46 words

The method of claim 35 , wherein the pre-selected image profile further comprises poses for the object to be tracked, and the determining the video likelihoods comprises comparing each outer edge against each of the poses to identify features indicative of the object to be tracked.

Claim 39

Dependent Claim 47 words

The method of claim 36 , wherein the pre-selected image profile comprises a color of an object to be tracked, and the determining the video likelihoods comprises comparing a color of portions of the image in order to identify features indicative of an object to be tracked.

Claim 40

Dependent Claim 48 words

The method of claim 37 , wherein the determining the video likelihoods comprises using the identified images to normalize each outer edge in order to be closer to a size of the poses and the shape in order to identify features indicative of an object to be tracked.

Claim 41

Dependent Claim 43 words

The method of claim 38 , wherein the determining the video likelihoods comprises determining that an outer edge of the image does not correspond to an object to be tracked if the outer edge does not correspond to the shape and the poses.

Claim 42

Dependent Claim 39 words

The method of claim 38 , wherein the determining the video likelihoods comprises determining that an outer edge of the image does not correspond to an object to be tracked if the outer edge does not include the color.

Claim 43

Dependent Claim 122 words

The method of claim 30 , wherein: a first one of the objects is disposed in a first direction, a second one of the objects is disposed in a second direction, and the method further comprises: determining that a correspondence exists between a first pair of the audio likelihoods and one of the video likelihoods and that another correspondence exists between a second pair of the one of the audio likelihoods and one of the video likelihoods, and based on the correspondences between first and second determined pairs of the audio and video likelihoods, identifying the first object as being in the first direction using the first pair and the second object as being in the second direction using the second pair.

Claim 44

Dependent Claim 24 words

The method of claim 41 , wherein method further comprising tracking the first object as the first object moves relative to the second object.

Claim 45

Dependent Claim 58 words

The method of claim 42 , wherein: the at least one computer receives the video included the images from a camera, and the tracking and identifying comprises tracking and identifying the first object as the first object moves relative to the second object such that the first object crosses the second object from a perspective of the camera.

Claim 46

Dependent Claim 52 words

The method of claim 30 , further comprising a performing beam-forming by, for each identified object, determining a location of the identified object, and separating from the received sounds audio corresponding to a location of each identified object so as to output audio channels corresponding uniquely to each of the identified objects.

Claim 47

Dependent Claim 64 words

The method of claim 44 , wherein: the at least one computer receives the sounds using a microphone array outputting a first number of received audio channels, each received audio channel includes an element of the sounds, the beam-forming comprising outputting a second number of the audio channels other than the first number, and the second number corresponds to the number of identified objects.

Claim 48

Dependent Claim 28 words

The method of claim 45 , further comprising recording each of the beam formed audio channels for each identified object as separate audio tracks associated with each object.

Claim 49

Dependent Claim 54 words

The method of claim 44 , wherein: each output channel includes audible periods in which speech is detected and silent periods between corresponding audible periods in which speech is not detected, and the method further comprises detecting a speech interval by, for each output channel, a start and stop time for each audible period.

Claim 50

Dependent Claim 83 words

The method of claim 47 , wherein the detecting the speech interval further comprises: detecting a proximity between adjacent audible periods, if the proximity is less than a predetermined amount, determining that the adjacent audible periods are one continuous audible period and connecting the adjacent audible periods to form the continuous audible period, and if the proximity is more than the predetermined amount, determining that the adjacent audible periods are separated by the silent period and does not connect the adjacent audible periods.

Claim 51

Dependent Claim 70 words

The method of claim 47 , wherein the detecting the speech interval further comprises; detecting a length of each audible period, if the length is less than a predetermined amount, determining that the audible period, is a silent period and erasing the audible period, and if the length is more than the predetermined amount, determining that the audible period is not a silent period and not erasing the audible period.

Claim 52

Dependent Claim 34 words

The method of claim 47 , wherein the detecting the speech interval further comprises: for each audible period, outputting the detected speech, and for each silent period, deleting the sound from the audio channel.

Claim 53

Dependent Claim 45 words

The method of claim 44 , further comprising a post processing the beam formed audio channels by, for each of plural beam formed audio channels, detecting audio portions related to cross channel interference caused by the remaining audio channels and removing the cross channel interference.

Claim 54

Dependent Claim 17 words

The method of claim 30 , further comprising controlling a robotic element according to the identified object.

Claim 55

Dependent Claim 32 words

The method of claim 52 , wherein the robotic element comprises at least one motor, and the method further comprises controlling the motor to move an apparatus according to the identified object.

Claim 56

Dependent Claim 36 words

The method of claim 53 , wherein the robotic element comprises at least one motor used to remotely move an element connected to the at least one computer through an interface according to the identified object.

Claim 57

Dependent Claim 69 words

The method of claim 30 , wherein: the determining the audio likelihood further comprises detecting, for each received sound, an audio direction from which a corresponding sound is received, the determining the video likelihood further comprises detecting, for each image, a video direction from which the image is observed, and the method further comprises determining the correspondences based upon a correspondence between the audio directions and the video directions.

Claim 58

Dependent Claim 22 words

A computer readable medium structure encoded with processing instructions for performing the method of claim 30 using the at least one computer.

Claim 59

Dependent Claim 22 words

A computer readable medium structure encoded with processing instructions for performing the method of claim 37 using the at least one computer.

Claim 60

Dependent Claim 22 words

A computer readable medium structure encoded with processing instructions for performing the method of claim 55 using the at least one computer.

Claim 61

Dependent Claim 57 words

The apparatus of claim 1 , wherein the apparatus is configured such that the video likelihood module determines video likelihoods for each of the plurality of images disposed in the corresponding different directions in the video by determining and considering a corresponding moving direction of at least one corresponding visual sensor relative to the corresponding different directions.

Claim 62

Dependent Claim 49 words

The method of claim 31 , wherein the video likelihoods are determined for each of the plurality of images disposed in the corresponding different directions in the video by determining and considering a corresponding moving direction of at least one corresponding visual sensor relative to the corresponding different directions.

Annuity Stage	Window Opens	Payment Deadline	Surcharge Period Until	Estimated Fee	Status
3.5 Years (Year 4)	2012-05-18	2012-11-17	2013-05-19	$2000.00	Window Closed / Lapsed
7.5 Years (Year 8)	2016-05-18	2016-11-17	2017-05-19	$3760.00	Window Closed / Lapsed
11.5 Years (Year 12)	2020-05-18	2020-11-17	2021-05-19	$7700.00	Window Closed / Lapsed

Apparatus and method performing audio-video sensor fusion for object localization, tracking, and separation

Abstract

Inventors

Bibliographic Details

Claims Analysis

References Cited (Backward Citations)

USPTO Annuity & Maintenance Windows

Assigned Categories & Tags

Assign Category Tag