Patent US5959684A successfully refreshed from Google Patents.
US5959684A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Method and apparatus for audio-video synchronizing
Assignee: Sony Corp
•
Filing: 1997-07-28
•
Granted: 1999-09-28
•
Expires: 2017-07-28
Abstract
Abstract A method and apparatus for synchronizing the playback of audio and video frames from a program source. The method associates an audio presentation time stamp ("PTS") value with an output audio frame. Selected ones of audio and video data packets include respective audio and video PTS values representing desired playback times of the respective audio and data associated therewith. The selected ones of the audio data packets further include audio frame numbers representing a number of output frames of audio to be played back between the selected ones of the audio data packets. The method comprises the steps of first storing the audio and video PTS values in respective audio and video PTS tables during an audio demultiplexing process. In addition, the audio frame numbers are stored in frame counters in association with respective PTS values during the demultiplexing process. Thereafter, the process sequentially decodes the audio and video input data to produce respective frames of audio and video which are presented to the user. With the presentation of each audio and video frame, the respective audio and video frame counters are selectively decremented. Upon detecting one of the audio frame counters having a zero value, the audio PTS value for that zero value audio frame counter is retrieved. Thereafter, the playback of the audio and video frames is selectively modified so that frames of audio and video are played back in synchronization.
Inventors
- 👤 Yew-Koon Tan
- 👤 Gong-san Yu
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-11-09 13:38
CPC Classifications
H; H04; H04N; H04N21/00; H04N21/40; H04N21/43; H04N21/434; H04N21/4341; H04N19/00; H04N19/42; H04N19/44; H04N21/4302; H04N21/4307; H04N21/43072
Claims Analysis
Total Claims: 36
Claim 1
Independent Claim
312 words
A method for associating an audio presentation time stamp ("PTS") value with an output frame of audio being a part of a sequence of frames of audio derived by demultiplexing and decoding audio input data in respective audio data packets, the output frames of audio being played back with output frames of video being a part of a sequence of frames of video are derived by demultiplexing and decoding video data in respective video data packets, selected ones of the audio and video data packets include respective audio and video PTS values representing desired playback times of the respective audio and video data associated therewith, the selected ones of the audio data packets further including audio frame numbers representing a number of output frames of audio to be played back between the selected ones of the audio data packets, the method comprising the steps of: storing the audio and video PTS values contained in the selected ones of the respective audio and video data packets during respective audio and video demultiplexing processes; storing audio frame numbers in respective audio frame counters during the audio demultiplexing process, each of the audio frame counters being associated with one of the stored audio PTS values; sequentially decoding the audio and video data in the selected ones of the respective audio and video data packets to produce the frames of audio and video, respectively; providing a simultaneous playback of the frames of audio and video to the user; selectively decrementing the audio frame counters in response to presenting each of the frames of audio; detecting one of the audio frame counters having a zero value; and if so, retrieving the audio PTS value corresponding to the one of the audio frame counters; and selectively modifying the playback of the frames of audio and video to synchronize the presentation of the audio and video to the user.
Claim 2
Independent Claim
312 words
A method for associating an audio presentation time stamp ("PTS") value with an output frame of audio being a part of a sequence of frames of audio derived by demultiplexing and decoding audio input data in respective audio data packets, the output frames of audio being played back with output frames of video being a part of a sequence of frames of video are derived by demultiplexing and decoding video data in respective video data packets, selected ones of the audio and video data packets include respective audio and video PTS values representing desired playback times of the respective audio and video data associated therewith, the selected ones of the audio data packets further including audio frame numbers representing a number of output frames of audio to be played back between the selected ones of the audio data packets, the method comprising the steps of: storing the audio and video PTS values contained in the selected ones of the respective audio and video data packets during respective audio and video demultiplexing processes; storing audio frame numbers in respective audio frame counters during the audio demultiplexing process, each of the audio frame counters being associated with one of the stored audio PTS values; sequentially decoding the audio and video data in the selected ones of the respective audio and video data packets to produce the frames of audio and video, respectively; providing a simultaneous playback of the frames of audio and video to the user; selectively decrementing the audio frame counters in response to presenting each of the frames of audio; detecting one of the audio frame counters having a zero value; and if so, retrieving the audio PTS value corresponding to the one of the audio frame counters; and selectively modifying the playback of the frames of audio and video to synchronize the presentation of the audio and video to the user.
Claim 3
Dependent Claim
88 words
The method of claim 1 further comprising after the step of retrieving the audio PTS value, the steps of: providing a audio clock extension for a system time counter approximately equal to a difference between an audio PTS value associated with the one of the audio frame counters and a current value of the system time counter; and adjusting a current state of the system time counter by the audio clock extension, thereby bringing the system time counter in synchronization with the playback of the frames of audio.
Claim 4
Dependent Claim
147 words
The method of claim 2 further comprising the steps of: storing at the start of a video decoding process for the selected ones of the video data packets, a number of video frames in respective video frame counters, the number of video frames representing a time duration approximately equal to a duration of the video decoding process; decrementing all of the video frame counters in response to presenting each of the frames of video; detecting one of the video frame counters having a zero value; and if so, retrieving the video PTS value corresponding to the one of the video frame counters; and determining a video PTS value associated with the one of the video frame counters to be approximately equal to the current state of the system time counter plus the audio clock extension, thereby determining that the frames of audio and video are in synchronization.
Claim 5
Dependent Claim
77 words
The method of claim 3 further comprising the steps of: determining in response to the one of the video frame counters having a zero value that the frames of video are currently being presented to the user too fast and out of synchronization with the frames of audio currently being presented to the user; and repeating a frame of video to bring the frames of video into closer synchronization with the presentation of the frames of audio.
Claim 6
Dependent Claim
63 words
The method of claim 4 further comprising the step of determining a video PTS value associated with the one of the video frame counters to be greater than the current state of the system time counter plus the audio clock extension, thereby indicating that the frames of video are being presented too fast with respect to the presentation of the frames of audio.
Claim 7
Dependent Claim
77 words
The method of claim 5 further comprising the steps of: determining in response to the one of the video frame counters having a zero value that the frames of video are currently being presented to the user too slowly and out of synchronization with the frames of audio currently being presented to the user; and skipping a frame of video to bring the frames of video into closer synchronization with the presentation of the frames of audio.
Claim 8
Dependent Claim
63 words
The method of claim 6 further comprising the step of determining a video PTS value associated with the one of the video frame counters to be less than the current state of the system time counter plus the audio clock extension, thereby indicating that the frames of video are being presented too slowly with respect to the presentation of the frames of audio.
Claim 9
Dependent Claim
61 words
The method of claim 7 further comprising the step of determining a video PTS value associated with the one of the video frame counters to be less than the current state of the system time counter plus the audio clock extension minus a limit factor, the limit factor being determined as a function of the delays in decoding the audio data.
Claim 10
Dependent Claim
35 words
The method of claim 8 further comprising the step of detecting the selection of an audio master synchronization determining a synchronization of the playback of frames of video to the playback of frames of audio.
Claim 11
Dependent Claim
171 words
The method of claim 1 further comprising the steps of: storing at the start of a video decoding process for the selected ones of the video data packets, a number of video frames in respective video frame counters, the number of video frames representing a time duration approximately equal to a duration of the video decoding process; decrementing all of the video frame counters in response to presenting each of the frames of video; detecting one of the video frame counters having a zero value; and if so, retrieving the video PTS value corresponding to the one of the video frame counters; providing a video clock extension for the system time counter approximately equal to a difference between a video PTS value associated with the one of the video frame counters and a current value of the system time counter; and adjusting a current state of the system time counter by the video clock extension, thereby bringing the system time counter in synchronization with the playback of the frames of video.
Claim 12
Dependent Claim
77 words
The method of claim 10 further comprising the steps of: determining in response to the one of the audio frame counters having a zero value that the frames of audio are currently being presented to the user too fast and out of synchronization with the frames of video currently being presented to the user; and repeating a frame of audio to bring the frames of audio into closer synchronization with the presentation of the frames of video.
Claim 13
Dependent Claim
63 words
The method of claim 11 further comprising the step of determining an audio PTS value associated with the one of the audio frame counters to be greater than the current state of the system time counter plus the video clock extension, thereby indicating that the frames of audio are being presented too fast with respect to the presentation of the frames of video.
Claim 14
Dependent Claim
58 words
The method of claim 12 further comprising the step of determining an audio PTS value associated with the one of the audio frame counters to be greater than the current state of the system time counter plus the video clock extension plus a limit factor, the limit factor being determined as a function of the video frame length.
Claim 15
Dependent Claim
77 words
The method of claim 13 further comprising the steps of: determining in response to the one of the audio frame counters having a zero value that the frames of audio are currently being presented to the user too slowly and out of synchronization with the frames of video currently being presented to the user; and skipping a frame of audio to bring the frames of audio into closer synchronization with the presentation of the frames of video.
Claim 16
Dependent Claim
63 words
The method of claim 14 further comprising the step of determining an audio PTS value associated with the one of the audio frame counters to be less than the current state of the system time counter plus the video clock extension, thereby indicating that the frames of audio are being presented too slowly with respect to the presentation of the frames of video.
Claim 17
Dependent Claim
58 words
The method of claim 15 further comprising the step of determining an audio PTS value associated with the one of the audio frame counters to be less than the current state of the system time counter plus the video clock extension minus a limit factor the limit factor being determined as a function of the video frame length.
Claim 18
Dependent Claim
24 words
The method of claim 16 where in the limit factor is approximately equal to approximately one-half of a duration of a shortest video frame.
Claim 19
Dependent Claim
35 words
The method of claim 17 further comprising the step of detecting the selection of a video master synchronization determining a synchronization of the playback of frames of audio to the playback of frames of video.
Claim 20
Dependent Claim
58 words
The method of claim 11 further comprising the step of determining an audio PTS value associated with the one of the audio frame counters to be greater than the current state of the system time counter, thereby indicating that the frames of audio are being presented too fast with respect to the presentation of the frames of video.
Claim 21
Dependent Claim
53 words
The method of claim 19 further comprising the step of determining an audio PTS value associated with the one of the audio frame counters to be greater than the current state of the system time counter plus a limit factor, the limit factor being determined as a function of the video frame length.
Claim 22
Dependent Claim
77 words
The method of claim 20 further comprising the steps of: determining in response to the one of the audio frame counters having a zero value that the frames of audio are currently being presented to the user too slowly and out of synchronization with the frames of video currently being presented to the user; and skipping a frame of audio to bring the frames of audio into closer synchronization with the presentation of the frames of video.
Claim 23
Dependent Claim
58 words
The method of claim 21 further comprising the step of determining an audio PTS value associated with the one of the audio frame counters to be less than the current state of the system time counter, thereby indicating that the frames of audio are being presented too slowly with respect to the presentation of the frames of video.
Claim 24
Dependent Claim
53 words
The method of claim 22 further comprising the step of determining an audio PTS value associated with the one of the audio frame counters to be less than the current state of the system time counter minus a limit factor, the limit factor being determined as a function of the video frame length.
Claim 25
Dependent Claim
23 words
The method of claim 23 wherein the limit factor is approximately equal to approximately one-half of a duration of a shortest video frame.
Claim 26
Dependent Claim
77 words
The method of claim 24 further comprising the steps of: determining in response to the one of the video frame counters having a zero value that the frames of video are currently being presented to the user too fast and out of synchronization with the frames of audio currently being presented to the user; and repeating a frame of video to bring the frames of video into closer synchronization with the presentation of the frames of audio.
Claim 27
Dependent Claim
58 words
The method of claim 25 further comprising the step of determining a video PTS value associated with the one of the video frame counters to be greater than the current state of the system time counter, thereby indicating that the frames of video are being presented too fast with respect to the presentation of the frames of audio.
Claim 28
Dependent Claim
56 words
The method of claim 26 further comprising the step of determining a video PTS value associated with the one of the video frame counters to be greater than the current state of the system time counter plus a limit factor, the limit factor being determined as a function of the delays in decoding the audio data.
Claim 29
Dependent Claim
77 words
The method of claim 27 further comprising the steps of: determining in response to the one of the video frame counters having a zero value that the frames of video are currently being presented to the user too slowly and out of synchronization with the frames of audio currently being presented to the user; and skipping a frame of video to bring the frames of video into closer synchronization with the presentation of the frames of audio.
Claim 30
Dependent Claim
58 words
The method of claim 28 further comprising the step of determining a video PTS value associated with the one of the video frame counters to be less than the current state of the system time counter, thereby indicating that the frames of video are being presented too slowly with respect to the presentation of the frames of audio.
Claim 31
Dependent Claim
56 words
The method of claim 29 further comprising the step of determining a video PTS value associated with the one of the video frame counters to be less than the current state of the system time counter minus a limit factor, the limit factor being determined as a function of the delays in decoding the audio data.
Claim 32
Dependent Claim
41 words
The method of claim 30 further comprising the step of detecting the selection of a no master synchronization for which no clock extension is determined or used and both the video and audio are independently synchronized to the system time counter.
Claim 33
Independent Claim
318 words
A method for synchronizing a play back of frames of audio and video output data, the frames of audio and video output data being derived from demultiplexing and decoding audio and video data contained in respective audio and video data packets, selected ones of the audio and video data packets containing presentation time stamp PTS values, and the selected ones of the audio data packets containing audio frame numbers, each of the audio frame numbers representing a number of frames of audio to be played back between the selected ones of the audio data packets, the method comprising: sequentially storing during an audio demultiplexing process of the audio data packets containing PTS values, audio data in respective audio FIFO memory locations; sequentially storing PTS values in first memory locations in an audio PTS memory table during the first demultiplexing process; storing in second memory locations in the audio PTS table during the first demultiplexing process, write pointer locations associated with the data written into the video FIFO memory locations for the selected video data packets containing a PTS value; storing audio frame numbers in counter memory locations in the PTS table during the first demultiplexing process; initiating an audio decoding process simultaneously with the a start audio frame interrupt to a CPU; acquiring a read pointer location of the audio FIFO in response to the start audio frame interrupt; finding a write pointer location in the audio PTS table approximately equal to the read pointer location; decoding the audio data being read from the audio FIFO; presenting a successive audio output frame to the user; decrementing all of the counter memory locations in the audio PTS table by one; detecting any of the counter locations having a zero value; and if so, adjusting a current state of a system time counter to the PTS value in the audio PTS table associated with the counter having a zero value.
Claim 34
Independent Claim
318 words
A method for synchronizing a play back of frames of audio and video output data, the frames of audio and video output data being derived from demultiplexing and decoding audio and video data contained in respective audio and video data packets, selected ones of the audio and video data packets containing presentation time stamp PTS values, and the selected ones of the audio data packets containing audio frame numbers, each of the audio frame numbers representing a number of frames of audio to be played back between the selected ones of the audio data packets, the method comprising: sequentially storing during an audio demultiplexing process of the audio data packets containing PTS values, audio data in respective audio FIFO memory locations; sequentially storing PTS values in first memory locations in an audio PTS memory table during the first demultiplexing process; storing in second memory locations in the audio PTS table during the first demultiplexing process, write pointer locations associated with the data written into the video FIFO memory locations for the selected video data packets containing a PTS value; storing audio frame numbers in counter memory locations in the PTS table during the first demultiplexing process; initiating an audio decoding process simultaneously with the a start audio frame interrupt to a CPU; acquiring a read pointer location of the audio FIFO in response to the start audio frame interrupt; finding a write pointer location in the audio PTS table approximately equal to the read pointer location; decoding the audio data being read from the audio FIFO; presenting a successive audio output frame to the user; decrementing all of the counter memory locations in the audio PTS table by one; detecting any of the counter locations having a zero value; and if so, adjusting a current state of a system time counter to the PTS value in the audio PTS table associated with the counter having a zero value.
Claim 35
Dependent Claim
259 words
A digital video processor for synchronizing a play back of output frames of audio and video data derived from demultiplexing and decoding audio and video data contained in audio and video data packets and selected ones of the audio and video data packets containing a presentation time stamp value, the video processor comprising: audio and video FIFO memories for sequentially storing the audio and video data contained the respective audio and video data packets; an audio PTS memory table having first memory locations for sequentially storing the PTS values in the selected ones of the audio data packets, second memory locations for storing values representing write pointer locations associated with the data written into the audio FIFO memory location in the selected audio data packets containing a PTS value, and third memory locations functioning as counter locations for storing audio frame numbers, each representing a number of frames of audio to be played back until an occurrence of a subsequent PTS value; and a video PTS memory table having first memory locations for sequentially storing the PTS values in the selected ones of the video data packets, second memory locations for storing values representing write pointer locations associated with the data written into the video FIFO memory locations for the selected video data packets containing a PTS value, and third memory locations functioning as counter locations for storing a value representing a number of frames of video approximately equal to a delay time required for decoding a frame of video data prior to its play back to a user.
Claim 36
Dependent Claim
259 words
A digital video processor for synchronizing a play back of output frames of audio and video data derived from demultiplexing and decoding audio and video data contained in audio and video data packets and selected ones of the audio and video data packets containing a presentation time stamp value, the video processor comprising: audio and video FIFO memories for sequentially storing the audio and video data contained the respective audio and video data packets; an audio PTS memory table having first memory locations for sequentially storing the PTS values in the selected ones of the audio data packets, second memory locations for storing values representing write pointer locations associated with the data written into the audio FIFO memory location in the selected audio data packets containing a PTS value, and third memory locations functioning as counter locations for storing audio frame numbers, each representing a number of frames of audio to be played back until an occurrence of a subsequent PTS value; and a video PTS memory table having first memory locations for sequentially storing the PTS values in the selected ones of the video data packets, second memory locations for storing values representing write pointer locations associated with the data written into the video FIFO memory locations for the selected video data packets containing a PTS value, and third memory locations functioning as counter locations for storing a value representing a number of frames of video approximately equal to a delay time required for decoding a frame of video data prior to its play back to a user.
References Cited (Backward Citations)
US5396497A
Cited
US5668601A
Cited
US5642171A
Cited
US5598352A
Cited
US5815634A
Cited
US5841472A
Cited
US5771075A
Cited
US5661728A
Cited
US5812201A
Cited
US5808722A
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2002-09-27 | 2003-03-29 | 2003-09-28 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2006-09-27 | 2007-03-29 | 2007-09-28 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2010-09-27 | 2011-03-29 | 2011-09-28 | $7700.00 | Window Closed / Lapsed |