US6990681B2
Expired / Lapsed
US
⏱️ 0.0 years remaining
Enhancing broadcast of an event with synthetic scene using a depth map
Assignee: Sony Corp
•
Filing: 2002-03-29
•
Granted: 2006-01-24
•
Expires: 2022-03-29
Abstract
Abstract A broadcast of an event is enhanced with synthetic scenes generated from audio visual and supplemental data received in the broadcast. A synthetic scene is integrated into the broadcast in accordance with a depth map that contains depth information for the synthetic scene. The supplemental data may be sensing data from various sensors placed at the event, position and orientation data of particular objects at the event, or environmental data on conditions at the event. The supplemental data may also be camera tracking data from a camera that is used to generate a virtual camera and viewpoints for the synthetic scene. The present invention describes systems, clients, servers, methods, and computer-readable media of varying scope. In addition to the aspects of the present invention described in this summary, further aspects of the invention will become apparent by reference to the drawings and by reading the detailed description that follows.
Inventors
- 👤 Sidney Wang
- 👤 Richter A. Rafey
- 👤 Hubert Le Van Gong
- 👤 Peter Broadwell
- 👤 Simon Gibbs
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 10:03
CPC Classifications
H; H04; H04N; H04N21/00; H04N21/40; H04N21/43; H04N21/44; H04N21/44012; H04N21/20; H04N21/21; H04N21/218; H04N21/2187; H04N21/23; H04N21/234; H04N21/23412; H04N21/2343; H04N21/234318; H04N21/47; H04N21/472; H04N21/47205; H04N21/80; H04N21/81; H04N21/8126; H04N21/8146; H04N5/00; H04N5/222; H04N5/2224; H04N5/2226; A; A63; A63F; A63F2300/00; A63F2300/60; A63F2300/69; A63F2300/695; A63F2300/80; A63F2300/8017
Claims Analysis
Total Claims: 34
Claim 1
Independent Claim
145 words
A method for enhancing a broadcast of an event, comprising: generating a synthetic scene based on audio visual data and supplemental data received in the broadcast; generating a depth map to store depth information for the synthetic scene; and integrating the synthetic scene into the broadcast using the depth map, wherein generating the depth map comprises: establishing the virtual camera using camera tracking data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to a corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.
Claim 2
Independent Claim
145 words
A method for enhancing a broadcast of an event, comprising: generating a synthetic scene based on audio visual data and supplemental data received in the broadcast; generating a depth map to store depth information for the synthetic scene; and integrating the synthetic scene into the broadcast using the depth map, wherein generating the depth map comprises: establishing the virtual camera using camera tracking data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to a corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.
Claim 3
Dependent Claim
35 words
The method of claim 1 , wherein the supplemental data comprises sensing data from a plurality of sensors placed at strategic points to render realistic synthetic movement of a synthetic object in the synthetic scene.
Claim 4
Dependent Claim
42 words
The method of claim 1 , wherein the supplemental data comprises position and orientation data of an object from a global positioning system receiver, the position and orientation data indicating the position and orientation of the object in a three dimensional space.
Claim 5
Dependent Claim
24 words
The method of claim 3 , wherein the global positioning system receiver further provides environmental conditions of the event to synchronize with the broadcast.
Claim 6
Dependent Claim
47 words
The method of claim 1 , wherein the supplemental data comprises camera tracking data from a tracked camera located at the event, the camera tracking data including positions of the tracked camera in a three dimensional space to generate a virtual camera corresponding to the tracked camera.
Claim 7
Dependent Claim
27 words
The method of claim 5 , further comprising registering the camera tracking data with the broadcast to render a virtual version of an object in the broadcast.
Claim 8
Dependent Claim
30 words
The method of claim 1 , further comprising a signal processor for processing the audio visual data and the supplemental data to generate the synthetic scene using the depth map.
Claim 9
Dependent Claim
24 words
The method of claim 7 , further comprising selecting a desired synthetic camera view in response to receiving an input through a user interface.
Claim 10
Dependent Claim
21 words
The method of claim 1 , further comprising repositioning the virtual camera in response to position changes of the tracked camera.
Claim 11
Dependent Claim
37 words
The method of claim 1 , further comprising: reconstructing a virtual view of a model, using the position information and camera tracking data; and extracting the depth information from the virtual view to generate the depth map.
Claim 12
Independent Claim
176 words
A method for enhancing a broadcast of an event, comprising: collecting, at a broadcast server, audio visual data and supplemental data from the event; transmitting the audio visual data and the supplemented data to a broadcast client over a network; generating, at the broadcast client, a synthetic scene based on the audio visual data and the supplemental data; generating a depth map to store depth information for the synthetic scene; and integrating the synthetic scene into the broadcast using the depth map, wherein the generating the depth map comprises: establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.
Claim 13
Independent Claim
176 words
A method for enhancing a broadcast of an event, comprising: collecting, at a broadcast server, audio visual data and supplemental data from the event; transmitting the audio visual data and the supplemented data to a broadcast client over a network; generating, at the broadcast client, a synthetic scene based on the audio visual data and the supplemental data; generating a depth map to store depth information for the synthetic scene; and integrating the synthetic scene into the broadcast using the depth map, wherein the generating the depth map comprises: establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.
Claim 14
Dependent Claim
50 words
The method of claim 11 , further comprising: encoding, at the broadcast server, the audio visual data and the supplemental data as motion picture expert group compatible data; and decoding, at the broadcast client, the motion picture expert group compatible data to retrieve the audio visual data and supplemental data.
Claim 15
Dependent Claim
44 words
The method of claim 11 , wherein the supplemental data comprises: sensing data from a plurality of sensors placed at the strategic points of the event; position and orientation data from a global positioning system receiver; and camera tracking data from a tracked camera.
Claim 16
Dependent Claim
24 words
The method of claim 11 , further comprising selecting a desired synthetic camera view in response to receiving an input at the broadcast client.
Claim 17
Independent Claim
174 words
A system for enhancing a broadcast of an event, comprising: a video signal unit coupled to provide audio visual data from the event; a supplemental data unit coupled to provide supplemental data from the event; a depth map coupled to provide depth information; and a processing unit configured to process the audio visual data and the supplemental data to generate a synthetic scene and further, the processing unit configured to integrate the synthetic scene into the broadcast using the depth map, wherein generating the depth map comprises: establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.
Claim 18
Independent Claim
174 words
A system for enhancing a broadcast of an event, comprising: a video signal unit coupled to provide audio visual data from the event; a supplemental data unit coupled to provide supplemental data from the event; a depth map coupled to provide depth information; and a processing unit configured to process the audio visual data and the supplemental data to generate a synthetic scene and further, the processing unit configured to integrate the synthetic scene into the broadcast using the depth map, wherein generating the depth map comprises: establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.
Claim 19
Dependent Claim
21 words
The system of claim 15 , further comprising a viewer control unit to receive a user selection of the synthetic scene.
Claim 20
Dependent Claim
109 words
The system of claim 15 , further comprising: a plurality of sensors placed at strategic points of a synthetic object to provide sensing data to render realistic synthetic movement of the synthetic object in the synthetic scene; a global positioning system receiver to provide position and orientation data of the synthetic object, the position and orientation data indicating the position and orientation of the synthetic object in a three dimensional space; and a tracked camera located at the event to provide camera track data, the camera track data including positions of the tracked camera in a three dimensional space to generate a virtual camera corresponding to the tracked camera.
Claim 21
Independent Claim
193 words
A system for enhancing a broadcast of an event, comprising: a broadcast server configured to receive audio visual (A/V) data and supplemental data; and a broadcast client configured to receive the audio visual data and the supplemental data transmitted from the broadcast server over a network, the broadcast client communicating with the broadcast server over the network, wherein the broadcast client: generates a synthetic scene based on the audio visual data and the supplemental data; generates a depth map to store depth information for the synthetic scene, wherein generating the depth map comprises: establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; and extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera; and integrates the synthetic scene into the broadcast using the depth map.
Claim 22
Independent Claim
193 words
A system for enhancing a broadcast of an event, comprising: a broadcast server configured to receive audio visual (A/V) data and supplemental data; and a broadcast client configured to receive the audio visual data and the supplemental data transmitted from the broadcast server over a network, the broadcast client communicating with the broadcast server over the network, wherein the broadcast client: generates a synthetic scene based on the audio visual data and the supplemental data; generates a depth map to store depth information for the synthetic scene, wherein generating the depth map comprises: establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; and extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera; and integrates the synthetic scene into the broadcast using the depth map.
Claim 23
Dependent Claim
43 words
The system of claim 18 , further comprising: a display device coupled to the broadcast client to display the integrated broadcast; and a viewer control unit coupled to the broadcast client to allow a user to select the synthetic scene to be displayed.
Claim 24
Dependent Claim
50 words
The system of claim 18 , wherein the broadcast server encodes the audio visual data and the supplemental data to be motion picture expert group compatible data, and wherein the broadcast client decodes the motion picture expert group compatible data to retrieve the audio visual data and the supplemental data.
Claim 25
Dependent Claim
52 words
The system of claim 18 , wherein the supplemental data comprises: sensing data from a plurality of sensors placed at the strategic points of a synthetic object at the event; position and orientation data of the synthetic object from a global positioning system receiver; and camera tracking data from a tracked camera.
Claim 26
Dependent Claim
157 words
A machine-readable medium having executable code to cause a machine to perform a method for enhancing a broadcast of an event, the method comprising: generating a synthetic scene based on audio visual data and supplemental data received in the broadcast; generating a depth map to store depth information for the synthetic scene; and integrating the synthetic scene into the broadcast using the depth map, wherein generating the depth map comprises: establishing the virtual camera tracking data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to a corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.
Claim 27
Dependent Claim
157 words
A machine-readable medium having executable code to cause a machine to perform a method for enhancing a broadcast of an event, the method comprising: generating a synthetic scene based on audio visual data and supplemental data received in the broadcast; generating a depth map to store depth information for the synthetic scene; and integrating the synthetic scene into the broadcast using the depth map, wherein generating the depth map comprises: establishing the virtual camera tracking data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to a corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.
Claim 28
Dependent Claim
28 words
The machine-readable medium of claim 22 , wherein the method further comprises selecting a desired synthetic camera view in response to receiving an input through a user interface.
Claim 29
Dependent Claim
41 words
The machine-readable medium of claim 22 , wherein the method further comprises: reconstructing a virtual view of a model, using the position information and camera tracking data; and extracting the depth information from the virtual view to generate the depth map.
Claim 30
Dependent Claim
190 words
A machine-readable medium having executable code to cause a machine to perform a method for enhancing a broadcast of an event, the method comprising: collecting, at a broadcast server, audio visual data and supplemental data from the event; transmitting the audio visual data and the supplemented data to a broadcast client over a network; generating, at the broadcast client, a synthetic scene based on the audio visual data and the supplemental data; generating a depth map to store depth information for the synthetic scene; and integrating the synthetic scene into the broadcast using the depth map, wherein the generating the depth map comprises: establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.
Claim 31
Dependent Claim
190 words
A machine-readable medium having executable code to cause a machine to perform a method for enhancing a broadcast of an event, the method comprising: collecting, at a broadcast server, audio visual data and supplemental data from the event; transmitting the audio visual data and the supplemented data to a broadcast client over a network; generating, at the broadcast client, a synthetic scene based on the audio visual data and the supplemental data; generating a depth map to store depth information for the synthetic scene; and integrating the synthetic scene into the broadcast using the depth map, wherein the generating the depth map comprises: establishing the virtual camera using camera data of a tracked camera which defines a viewpoint for the synthetic scene; setting a field of view of the virtual camera to corresponding field of view of the tracked camera; positioning a synthetic tracked object in the synthetic scene according to position information of the tracked object; extracting depth information of the synthetic tracked object to generate the depth map; and refining the depth map by distorting grid coordinates of the depth map based on characteristics of the tracked camera.
Claim 32
Dependent Claim
54 words
The machine-readable medium of claim 25 , wherein the method further comprises: encoding, at the broadcast server, the audio visual data and the supplemental data as motion picture expert group compatible data; and decoding, at the broadcast client, the motion picture expert group compatible data to retrieve the audio visual data and supplemental data.
Claim 33
Dependent Claim
45 words
The machine-readable medium of claim 25 , wherein the supplemental data comprises: sensing data from a plurality of sensors placed at the strategic points of the event; position and orientation data from a global positioning system receiver; and camera tracking data from a tracked camera.
Claim 34
Dependent Claim
28 words
The machine-readable medium of claim 25 , wherein the method further comprises selecting a desired synthetic camera view in response to receiving an input at the broadcast client.
References Cited (Backward Citations)
US4716458A
Cited
US6031545A
Cited
US5742521A
Cited
US5966132A
Cited
US5600368A
Cited
US5745126A
Cited
US5729471A
Cited
US6193610B1
Cited
US6151009A
Cited
US6080063A
Cited
US6147716A
Cited
US20010003715A1
Cited
US6593936B1
Cited
US20020069265A1
Cited
US6359585B1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2009-01-23 | 2009-07-25 | 2010-01-24 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2013-01-23 | 2013-07-25 | 2014-01-24 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2017-01-23 | 2017-07-25 | 2018-01-24 | $7700.00 | Window Closed / Lapsed |