Video content understanding through real time video motion analysis

Claim 1

A method of estimating camera motion between a pair of successive images, comprising: determining motion vectors from the pair of successive images, wherein the determining comprises segmenting one of the pair of successive images into a plurality of blocks of pixels, selecting for motion vector determination only ones of the blocks having respective levels of feature content that satisfy a criteria, determining luminance values corresponding to respective ones of the selected blocks, matching the selected blocks to blocks of pixels in the other of the pair of successive images based on the luminance values, and determining motion vectors associated with the matched blocks in the pair of successive images; and estimating an affine model of the camera motion based on the motion vectors, wherein the estimated affine model determines a set of camera motion parameters.

Claim 2

A method of estimating camera motion between a pair of successive images, comprising: determining motion vectors from the pair of successive images, wherein the determining comprises segmenting one of the pair of successive images into a plurality of blocks of pixels, selecting for motion vector determination only ones of the blocks having respective levels of feature content that satisfy a criteria, determining luminance values corresponding to respective ones of the selected blocks, matching the selected blocks to blocks of pixels in the other of the pair of successive images based on the luminance values, and determining motion vectors associated with the matched blocks in the pair of successive images; and estimating an affine model of the camera motion based on the motion vectors, wherein the estimated affine model determines a set of camera motion parameters.

Claim 13

A method of estimating camera motion between a pair of successive images, comprising the steps of: determining motion vectors from the pair of successive images by the steps of: segmenting one of the pair of successive images into a plurality of blocks of pixels; selecting blocks corresponding to at least one feature in the image; determining luminance values corresponding to respective selected blocks; matching the selected blocks to blocks of pixels in the other of the pair of successive images based on the luminance values; and determining motion vectors associated with matched blocks in the pair of successive images; and estimating an affine model of the camera motion based on the motion vectors, wherein the estimated affine model determines a set of camera motion parameters, wherein the affine model is based on the camera's focal length and rotation angles around three orthogonal axes.

Claim 14

A method of estimating camera motion between a pair of successive images, comprising the steps of: determining motion vectors from the pair of successive images by the steps of: segmenting one of the pair of successive images into a plurality of blocks of pixels; selecting blocks corresponding to at least one feature in the image; determining luminance values corresponding to respective selected blocks; matching the selected blocks to blocks of pixels in the other of the pair of successive images based on the luminance values; and determining motion vectors associated with matched blocks in the pair of successive images; and estimating an affine model of the camera motion based on the motion vectors, wherein the estimated affine model determines a set of camera motion parameters, wherein the affine model is based on the camera's focal length and rotation angles around three orthogonal axes.

Claim 16

A method of transforming camera motion parameters into semantic meanings, comprising: determining motion vectors from pairs of successive images; determining respective sets of camera motion parameters by estimating affine models based on the motion vectors; and generating semantic meanings from the sets of camera motion parameters, wherein the generating comprises quantizing the sets of camera motion parameters into a plurality of quantized magnitude classes and quantized direction classes, and producing at least one sentence from a sequence of the images, wherein the producing comprises merging into the sentence successive ones of the images having respective sets of camera motion parameters that are quantized into substantially similar magnitude and direction classes.

Claim 17

A method of transforming camera motion parameters into semantic meanings, comprising: determining motion vectors from pairs of successive images; determining respective sets of camera motion parameters by estimating affine models based on the motion vectors; and generating semantic meanings from the sets of camera motion parameters, wherein the generating comprises quantizing the sets of camera motion parameters into a plurality of quantized magnitude classes and quantized direction classes, and producing at least one sentence from a sequence of the images, wherein the producing comprises merging into the sentence successive ones of the images having respective sets of camera motion parameters that are quantized into substantially similar magnitude and direction classes.

Claim 22

A method of detecting an object in images captured by a camera, comprising the steps of: determining motion vectors from pairs of successive images; estimating affine models based on the motion vectors; computing motion compensation residual errors for respective blocks of pixels in the pairs of successive images from the motion vectors and the affine model estimates; filtering the motion compensation residual errors by applying a set of weight values, wherein the filtering step includes generating adjusted residual errors by spatial position filtering by assigning larger weight values to blocks located near a central portion of an image and zero weight values to blocks on an image boundary, and motion magnitude filtering by assigning larger weight values to blocks located near the central portion of the image and non-zero weight values to blocks on the image boundary; and detecting an object based on the residual errors.

Claim 23

A method of detecting an object in images captured by a camera, comprising the steps of: determining motion vectors from pairs of successive images; estimating affine models based on the motion vectors; computing motion compensation residual errors for respective blocks of pixels in the pairs of successive images from the motion vectors and the affine model estimates; filtering the motion compensation residual errors by applying a set of weight values, wherein the filtering step includes generating adjusted residual errors by spatial position filtering by assigning larger weight values to blocks located near a central portion of an image and zero weight values to blocks on an image boundary, and motion magnitude filtering by assigning larger weight values to blocks located near the central portion of the image and non-zero weight values to blocks on the image boundary; and detecting an object based on the residual errors.

Claim 28

A method of detecting an object in images captured by a camera, comprising the steps of: determining motion vectors from pairs of successive images; estimating affine models based on the motion vectors; computing residual errors for respective blocks of pixels in the pairs of successive images from the motion vectors and the affine model estimates; detecting an object based on the residual errors; tracking a detected object by representing the detected object as a quantized color histogram, normalizing the quantized color histogram to a probability shape, and weighting the normalized quantized color histogram, wherein smaller weight values are assigned to pixels farther from the center of the detected object.

Claim 29

A method of detecting an object in images captured by a camera, comprising the steps of: determining motion vectors from pairs of successive images; estimating affine models based on the motion vectors; computing residual errors for respective blocks of pixels in the pairs of successive images from the motion vectors and the affine model estimates; detecting an object based on the residual errors; tracking a detected object by representing the detected object as a quantized color histogram, normalizing the quantized color histogram to a probability shape, and weighting the normalized quantized color histogram, wherein smaller weight values are assigned to pixels farther from the center of the detected object.

Claim 32

An apparatus for video motion understanding, comprising: a data preparation device, comprising a decoder that decodes an incoming video signal and produces an image sequence, a down-sampler that receives the image sequence and produces a sequence of lower-resolution images, and a device that extracts luminance information from the down-sampled image sequence; a camera motion estimator that includes an optical flow analyzer that uses the extracted luminance information for optical flow analysis to generate motion vectors from blocks of pixels in pairs of successive images, and an affine model estimator that generates respective sets of camera motion parameters of affine models of camera motion based on the motion vectors; and an object detector that determines motion compensation residual errors for ones of the blocks based on the sets of camera motion parameters, selects ones of the blocks based on the determined motion compensation residual errors, and filters the selected blocks, wherein the object detector generates a bounding box that identifies a size and position of an object in the image.

Claim 33

An apparatus for video motion understanding, comprising: a data preparation device, comprising a decoder that decodes an incoming video signal and produces an image sequence, a down-sampler that receives the image sequence and produces a sequence of lower-resolution images, and a device that extracts luminance information from the down-sampled image sequence; a camera motion estimator that includes an optical flow analyzer that uses the extracted luminance information for optical flow analysis to generate motion vectors from blocks of pixels in pairs of successive images, and an affine model estimator that generates respective sets of camera motion parameters of affine models of camera motion based on the motion vectors; and an object detector that determines motion compensation residual errors for ones of the blocks based on the sets of camera motion parameters, selects ones of the blocks based on the determined motion compensation residual errors, and filters the selected blocks, wherein the object detector generates a bounding box that identifies a size and position of an object in the image.

Claim 34

An apparatus for video motion understanding, comprising: a camera motion estimator that includes an optical flow analyzer that uses luminance images for optical flow analysis to generate motion vectors from blocks of pixels in pairs of successive images, and an affine model estimator that respective sets of camera motion parameters of affine models of camera motion based on the motion vectors; and an object detector that determines motion compensation residual errors for ones of the blocks based on the affine models of camera motion, selects ones of the blocks based on the determined motion compensation residual errors, and filters the selected blocks, wherein the object detector generates a bounding box that identifies a size and position of an object in the image; and a semantic meaning parser that generates semantic meanings from the affine models of camera motion, wherein in generating the semantic meanings the semantic meaning parser performs operations comprising quantizing the sets of camera motion parameters into a plurality of quantized magnitude classes and quantized direction classes, and producing at least one sentence from a sequence of the images, wherein the producing comprises merging into the sentence successive ones of the images having respective sets of camera motion parameters that are quantized into substantially similar magnitude and direction classes.

Claim 35

An apparatus for video motion understanding, comprising: a camera motion estimator that includes an optical flow analyzer that uses luminance images for optical flow analysis to generate motion vectors from blocks of pixels in pairs of successive images, and an affine model estimator that respective sets of camera motion parameters of affine models of camera motion based on the motion vectors; and an object detector that determines motion compensation residual errors for ones of the blocks based on the affine models of camera motion, selects ones of the blocks based on the determined motion compensation residual errors, and filters the selected blocks, wherein the object detector generates a bounding box that identifies a size and position of an object in the image; and a semantic meaning parser that generates semantic meanings from the affine models of camera motion, wherein in generating the semantic meanings the semantic meaning parser performs operations comprising quantizing the sets of camera motion parameters into a plurality of quantized magnitude classes and quantized direction classes, and producing at least one sentence from a sequence of the images, wherein the producing comprises merging into the sentence successive ones of the images having respective sets of camera motion parameters that are quantized into substantially similar magnitude and direction classes.

Claim 36

An apparatus for video motion understanding, comprising: a camera motion estimator that includes an optical flow analyzer that uses luminance images for optical flow analysis to generate motion vectors from blocks of pixels in pairs of successive images, and an affine model estimator that generates affine model estimates based on the motion vectors; an object detector that determines residual errors of blocks based on the affine model estimates, selects blocks based on the residual errors, and filters selected blocks, wherein the object detector generates a bounding box that identifies a size and position of an object; a semantic meaning parser that generates at least one sentence based on combinations of the affine model estimates; and a moving object tracker that includes a target model and probability calculator and a coefficient maximizer that determine target models and target candidate probability density functions based on the bounding box and generate an indication of a tracked moving object.

Claim 37

An apparatus for video motion understanding, comprising: a camera motion estimator that includes an optical flow analyzer that uses luminance images for optical flow analysis to generate motion vectors from blocks of pixels in pairs of successive images, and an affine model estimator that generates affine model estimates based on the motion vectors; an object detector that determines residual errors of blocks based on the affine model estimates, selects blocks based on the residual errors, and filters selected blocks, wherein the object detector generates a bounding box that identifies a size and position of an object; a semantic meaning parser that generates at least one sentence based on combinations of the affine model estimates; and a moving object tracker that includes a target model and probability calculator and a coefficient maximizer that determine target models and target candidate probability density functions based on the bounding box and generate an indication of a tracked moving object.