US5737487A
Expired / Lapsed
US
⏱️ 0.0 years remaining
Speaker adaptation based on lateral tying for large-vocabulary continuous speech recognition
Assignee: Apple Inc
•
Filing: 1996-02-13
•
Granted: 1998-04-07
•
Expires: 2016-02-13
Abstract
Abstract A system and method for performing speaker adaptation in a speech recognition system which includes a set of reference models corresponding to speech data from a plurality of speakers. The speech data is represented by a plurality of acoustic models and corresponding sub-events, and each sub-event includes one or more observations of speech data. A degree of lateral tying is computed between each pair of sub-events, wherein the degree of tying indicates the degree to which a first observation in a first sub-event contributes to the remaining sub-events. When adaptation data from a new speaker becomes available, a new observation from adaptation data is assigned to one of the sub-events. Each of the sub-events is then populated with the observations contained in the assigned sub-event based on the degree of lateral tying that was computed between each pair of sub-events. The reference models corresponding to the populated sub-events are then adapted to account for speech pattern idiosyncrasies of the new speaker, thereby reducing the error rate of the speech recognition system.
Inventors
- 👤 Jerome R. Bellegarda
- 👤 John W. Butzberger
- 👤 Yen-Lu Chow
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-27 06:57
CPC Classifications
G; G10; G10L; G10L15/00; G10L15/06; G10L15/065
Claims Analysis
Total Claims: 34
Claim 1
Independent Claim
170 words
A method of performing speaker adaptation in a speech recognition system which includes a set of reference models corresponding to speech data from a plurality of speakers, the speech data represented by a plurality of acoustic models and corresponding sub-events, wherein each sub-event includes one or more observations of speech data, the method comprising the steps of: (a) computing a degree of lateral tying between each pair of sub-events, wherein the degree of tying indicates the degree to which a first observation in a first sub-event contributes to the remaining sub-events; (b) assigning a new observation from adaptation data of a new speaker to one of the sub-events; (c) populating each of the sub-events with a transformed version of the observation contained in the assigned sub-event based on the degree of lateral tying computed between each pair of sub-events; (d) adapting the reference models that correspond to the populated sub-events to account for speech pattern idiosyncrasies of the new speaker, thereby reducing the error rate of the speech recognition system.
Claim 2
Independent Claim
170 words
A method of performing speaker adaptation in a speech recognition system which includes a set of reference models corresponding to speech data from a plurality of speakers, the speech data represented by a plurality of acoustic models and corresponding sub-events, wherein each sub-event includes one or more observations of speech data, the method comprising the steps of: (a) computing a degree of lateral tying between each pair of sub-events, wherein the degree of tying indicates the degree to which a first observation in a first sub-event contributes to the remaining sub-events; (b) assigning a new observation from adaptation data of a new speaker to one of the sub-events; (c) populating each of the sub-events with a transformed version of the observation contained in the assigned sub-event based on the degree of lateral tying computed between each pair of sub-events; (d) adapting the reference models that correspond to the populated sub-events to account for speech pattern idiosyncrasies of the new speaker, thereby reducing the error rate of the speech recognition system.
Claim 3
Dependent Claim
36 words
A method as in claim 1 wherein step (a) further includes the step of: (a1) computing a transformation for each pair of sub-events to indicate if a high degree of lateral tying exist between the pair.
Claim 4
Dependent Claim
37 words
A method as in claim 2 wherein step (a) further includes the step of: (a2) for each of the sub-events, identifying a neighborhood of other sub-events that exhibit a high degree of tying with the respective sub-event.
Claim 5
Dependent Claim
29 words
A method as in claim 3 wherein step (c) further includes the step of: (c1) populating only those sub-events that are included in the neighborhood of the assigned sub-event.
Claim 6
Dependent Claim
149 words
A method as in claim 4 wherein step (a1) further includes the steps of: (a1i) representing observations by feature vectors; (a1ii) segregating each feature vector in the sub-events according to which speaker the feature vector originated from; (a1iii) for each speaker s, determining an anchor point of the feature vectors in each of the sub-events i, given by X i (s); (a1iv) defining the degree of tying between sub-events E 1 and E 2 is based on a cross-covariance matrix between E 1 and E 2 by ##EQU6## where the summation is over all the speakers; and (a1v) computing a transformation matrix for every pair of sub-events using the quantity Γ.sub.1 = U.sub.12 V.sub.12.sup.T ( 6) to represent the least squares rotation that must be applied to X 1 (s) in sub-event E 1 in order to obtain an estimate of X 2 (s) in sub-event E 2 .
Claim 7
Dependent Claim
63 words
A method as in claim 5 wherein step (a2) further includes the step of: (a2i) determining which sub-events E j are closely correlated with E i by a distance measure ##EQU7## which is computed for each E j , and wherein λ j has a value that ranges from one to zero, where a value of one represents a maximum degree of tying.
Claim 8
Dependent Claim
49 words
A method as in claim 6 wherein step (a2) further includes the step of: (a2ii) creating a neighborhood for sub-event E i by excluding those sub-events E j from the neighborhood whose value of λ j deviates from the maximum value of λ by more than a pre-determined percentile.
Claim 9
Dependent Claim
61 words
A method as in claim 6 wherein step (a2) further includes the step of: (a2iii) creating a neighborhood for sub-event E i by ranking each E j from the highest value of λ j to the lowest, and by including in the neighborhood only a predetermined number of sub-events E j that have the highest ranked values of λ j .
Claim 10
Dependent Claim
74 words
A method as in claim 8 wherein step (c1) further includes the step of: (c1i) calculating Z.sub.j =(S.sub.j.sup.1/2 λ.sub.j Γ.sub.ij S.sub.i.sup.-1/2)Z.sub.i +(M.sub.j -s.sub.j.sup.1/2 λ.sub.j Γ.sub.ij S.sub.i.sup.-1/2 M.sub.i) wherein Z i denotes a first feature vector from the adaptation data assigned to sub-event E i , which is used to obtain an estimate of an unobserved feature vector .sup. Z j in sub-event E j that is contained in neighborhood of sub-event E i .
Claim 11
Dependent Claim
175 words
A speech recognition system that performs speaker adaptation, the system including a set of reference models corresponding to speech data from a plurality of speakers, the speech data represented by a plurality of acoustic models and corresponding sub-events, wherein each sub-event includes one or more observations of speech data, the speech recognition system comprising: (a) means for computing a degree of lateral tying between each pair of sub-events, wherein the degree of tying indicates the degree to which a first observation in a first sub-event contributes to the remaining sub-events; (b) means for assigning a new observation from adaptation data of a new speaker to one of the sub-events; (c) means for populating each of the sub-events with a transformed version of the observation contained in the assigned sub-event based on the degree of lateral tying computed between each pair of sub-events; (d) means for adapting the reference models that correspond to the populated sub-events to account for speech pattern idiosyncrasies of the new speaker, thereby reducing the error rate of the speech recognition system.
Claim 12
Dependent Claim
175 words
A speech recognition system that performs speaker adaptation, the system including a set of reference models corresponding to speech data from a plurality of speakers, the speech data represented by a plurality of acoustic models and corresponding sub-events, wherein each sub-event includes one or more observations of speech data, the speech recognition system comprising: (a) means for computing a degree of lateral tying between each pair of sub-events, wherein the degree of tying indicates the degree to which a first observation in a first sub-event contributes to the remaining sub-events; (b) means for assigning a new observation from adaptation data of a new speaker to one of the sub-events; (c) means for populating each of the sub-events with a transformed version of the observation contained in the assigned sub-event based on the degree of lateral tying computed between each pair of sub-events; (d) means for adapting the reference models that correspond to the populated sub-events to account for speech pattern idiosyncrasies of the new speaker, thereby reducing the error rate of the speech recognition system.
Claim 13
Dependent Claim
32 words
A system as in claim 10 wherein the means for computing computes a transformation for each pair of sub-events to indicate if a high degree of lateral tying exist between the pair.
Claim 14
Dependent Claim
38 words
A system as in claim 11 wherein the means for computing further includes means for identifying a neighborhood for each of the sub-events that represents other sub-events that exhibit a high degree of tying with the respective sub-event.
Claim 15
Dependent Claim
25 words
A system as in claim 12 wherein the means for populating only populates those sub-events that are included in the neighborhood of the assigned sub-event.
Claim 16
Dependent Claim
154 words
A system as in claim 13 wherein the means for computing further includes: means for representing observations by feature vectors; means for segregating each feature vector in the sub-events according to which speaker the feature vector originated from; means for determining for each speaker s, an anchor point of the feature vectors in each of the sub-events i, given by X i (s); means for defining the degree of tying between sub-events E 1 and E 2 is based on a cross-covariance matrix between E 1 and E 2 by ##EQU8## where the summation is over all the speakers; and (a1v) means for computing a transformation matrix for every pair of sub-events using the quantity Γ.sub.1 = U.sub.12 V.sub.12.sup.T ( 6) to represent the least squares rotation that must be applied to X 1 (s) in sub-event E 1 in order to obtain an estimate of X 2 (s) in sub-event E 2 .
Claim 17
Dependent Claim
62 words
A system as in claim 14 wherein means for computing further includes means for determining which sub-events E j are closely correlated with E i by a distance measure ##EQU9## which is computed for each E j , and wherein λ j has a value that ranges from one to zero, where a value of one represents a maximum degree of tying.
Claim 18
Dependent Claim
45 words
A system as in claim 15 wherein the means for computing creates a neighborhood for sub-event E i by excluding those sub-events E j from the neighborhood whose value of λ j deviates from the maximum value of λ by more than a pre-determined percentile.
Claim 19
Dependent Claim
57 words
A system as in claim 16 wherein the means for computing creates a neighborhood for sub-event E i by ranking each E j from the highest value of λ j to the lowest, and by including in the neighborhood only a predetermined number of sub-events E j that have the highest ranked values of λ j .
Claim 20
Dependent Claim
72 words
A system as in claim 17 wherein the means for populating includes means for calculating Z.sub.j =(S.sub.j.sup.1/2 λ.sub.j Γ.sub.ij S.sub.i.sup.-1/2)Z.sub.i +(M.sub.j -s.sub.j.sup.1/2 λ.sub.j Γ.sub.ij S.sub.i.sup.-1/2 M.sub.i) wherein Z i denotes a first feature vector from the adaptation data assigned to sub-event E i , which is used to obtain an estimate of an unobserved feature vector Z j in sub-event E j that is contained in neighborhood of sub-event E i .
Claim 21
Dependent Claim
172 words
A computer-readable medium containing program instructions for performing speaker adaptation in a speech recognition system which includes a set of reference models corresponding to speech data from a plurality of speakers, the speech data represented by a plurality of acoustic models and corresponding sub-events, wherein each sub-event includes one or more observations of speech data, the program instructions for: (a) computing a degree of lateral tying between each pair of sub-events, wherein the degree of tying indicates the degree to which a first observation in a first sub-event contributes to the remaining sub-events; (b) assigning a new observation from adaptation data of a new speaker to one of the sub-events; (c) populating each of the sub-events with a transformed version of the observation contained in the assigned sub-event based on the degree of lateral tying computed between each pair of sub-events; (d) adapting the reference models that correspond to the populated sub-events to account for speech pattern idiosyncrasies of the new speaker, thereby reducing the error rate of the speech recognition system.
Claim 22
Dependent Claim
172 words
A computer-readable medium containing program instructions for performing speaker adaptation in a speech recognition system which includes a set of reference models corresponding to speech data from a plurality of speakers, the speech data represented by a plurality of acoustic models and corresponding sub-events, wherein each sub-event includes one or more observations of speech data, the program instructions for: (a) computing a degree of lateral tying between each pair of sub-events, wherein the degree of tying indicates the degree to which a first observation in a first sub-event contributes to the remaining sub-events; (b) assigning a new observation from adaptation data of a new speaker to one of the sub-events; (c) populating each of the sub-events with a transformed version of the observation contained in the assigned sub-event based on the degree of lateral tying computed between each pair of sub-events; (d) adapting the reference models that correspond to the populated sub-events to account for speech pattern idiosyncrasies of the new speaker, thereby reducing the error rate of the speech recognition system.
Claim 23
Dependent Claim
37 words
A computer-readable medium as in claim 1 wherein instruction (a) further includes the instruction of: (a1) computing a transformation for each pair of sub-events to indicate if a high degree of lateral tying exist between the pair.
Claim 24
Dependent Claim
38 words
A computer-readable medium as in claim 2 wherein instruction (a) further includes the instruction of: (a2) for each of the sub-events, identifying a neighborhood of other sub-events that exhibit a high degree of tying with the respective sub-event.
Claim 25
Dependent Claim
30 words
A computer-readable medium as in claim 3 wherein instruction (c) further includes the instruction of: (c1) populating only those sub-events that are included in the neighborhood of the assigned sub-event.
Claim 26
Dependent Claim
150 words
A computer-readable medium as in claim 4 wherein instruction (a1) further includes the instructions of: (a1i) representing observations by feature vectors; (a1ii) segregating each feature vector in the sub-events according to which speaker the feature vector originated from; (a1iii) for each speaker s, determining an anchor point of the feature vectors in each of the sub-events i, given by X i (s); (a1iv) defining the degree of tying between sub-events E 1 and E 2 is based on a cross-covariance matrix between E 1 and E 2 by ##EQU10## where the summation is over all the speakers; and (a1v) computing a transformation matrix for every pair of sub-events using the quantity Γ.sub.1 = U.sub.12 V.sub.12.sup.T ( 6) to represent the least squares rotation that must be applied to X 1 (s) in sub-event E 1 in order to obtain an estimate of X 2 (s) in sub-event E 2 .
Claim 27
Dependent Claim
64 words
A computer-readable medium as in claim 5 wherein instruction (a2) further includes the instruction of: (a2i) determining which sub-events E j are closely correlated with E i by a distance measure ##EQU11## which is computed for each E j , and wherein λ j has a value that ranges from one to zero, where a value of one represents a maximum degree of tying.
Claim 28
Dependent Claim
50 words
A computer-readable medium as in claim 6 wherein instruction (a2) further includes the instruction of: (a2ii) creating a neighborhood for sub-event E i by excluding those sub-events E j from the neighborhood whose value of λ j deviates from the maximum value of λ by more than a pre-determined percentile.
Claim 29
Dependent Claim
62 words
A computer-readable medium as in claim 6 wherein instruction (a2) further includes the instruction of: (a2iii) creating a neighborhood for sub-event E i by ranking each E j from the highest value of λ j to the lowest, and by including in the neighborhood only a predetermined number of sub-events E j that have the highest ranked values of λ j .
Claim 30
Dependent Claim
74 words
A computer-readable medium as in claim 8 wherein instruction (c1) further includes the instruction of: (c1i) calculating Z.sub.j =(S.sub.j.sup.1/2 λ.sub.j Γ.sub.ij S.sub.i.sup.-1/2)Z.sub.i +(M.sub.j -s.sub.j.sup.1/2 λ.sub.j Γ.sub.ij S.sub.i.sup.-1/2 M.sub.i) wherein Z i denotes a first feature vector from the adaptation data assigned to sub-event E i , which is used to obtain an estimate of an unobserved feature vector Z j in sub-event E j that is contained in neighborhood of sub-event E i .
Claim 31
Independent Claim
200 words
A method of performing speaker adaptation in a speech recognition system which includes a set of reference models corresponding to speech data, the speech data represented by a first and second reference model, the first reference model comprising a first sub-event and second reference model comprising a second sub-event, wherein the first sub-event is well populated with a plurality of feature vectors and the second sub-event is sparsely populated with feature vectors, the method comprising the steps of: (a) computing a transformation between the first and second sub-event to indicate a degree of lateral tying between the first and second sub-event; (b) assigning a new feature vector extracted from adaptation data to the first sub-event; (c) if the computed transformation indicates a degree of lateral tying that surpasses a desired threshold, applying the computed transformation to the feature vectors in the first sub-event to transform the feature vectors into the space of the second sub-event to thereby populate the second sub-event with feature vectors from the first sub-event; and (d) adapting the reference models that correspond to the populated sub-events to account for speech pattern idiosyncrasies of the new speaker, thereby reducing the error rate of the speech recognition system.
Claim 32
Independent Claim
200 words
A method of performing speaker adaptation in a speech recognition system which includes a set of reference models corresponding to speech data, the speech data represented by a first and second reference model, the first reference model comprising a first sub-event and second reference model comprising a second sub-event, wherein the first sub-event is well populated with a plurality of feature vectors and the second sub-event is sparsely populated with feature vectors, the method comprising the steps of: (a) computing a transformation between the first and second sub-event to indicate a degree of lateral tying between the first and second sub-event; (b) assigning a new feature vector extracted from adaptation data to the first sub-event; (c) if the computed transformation indicates a degree of lateral tying that surpasses a desired threshold, applying the computed transformation to the feature vectors in the first sub-event to transform the feature vectors into the space of the second sub-event to thereby populate the second sub-event with feature vectors from the first sub-event; and (d) adapting the reference models that correspond to the populated sub-events to account for speech pattern idiosyncrasies of the new speaker, thereby reducing the error rate of the speech recognition system.
Claim 33
Dependent Claim
145 words
A method as in claim 28 wherein step (c) further includes the steps of: (c1) segregating each feature vector in the sub-events according to which speaker the feature vector originated from; (c2) for each speaker s, determining an anchor point of the feature vectors in each of the sub-events i, given by X i (s); (c3) defining the degree of tying between sub-events E i and E 2 is based on a cross-covariance matrix between E 1 and E 2 by ##EQU12## where the summation is over all the speakers; and (c4) computing a transformation matrix between sub-events E 1 and E 2 using the quantity Γ.sub.1 = U.sub.12 V.sub.12.sup.T ( 6) to represent the least squares rotation that must be applied to X 1 (s) in sub-event E 1 in order to obtain an estimate of X 2 (s) in sub-event E 2 .
Claim 34
Dependent Claim
76 words
A method as in claim 29 wherein step (b) further includes the steps of: (b1) acquiring the speech from the new user in the form of input speech signals; (b2) transforming the input speech signals into a physical representation; (b3) extracting acoustic features from the physical representation and generating a series of feature vectors therefrom; and (B4) determining what allophone each feature vector represents and assigning each feature vector to the sub-event representing the corresponding allophone.
References Cited (Backward Citations)
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2001-04-06 | 2001-10-06 | 2002-04-07 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2005-04-06 | 2005-10-06 | 2006-04-07 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2009-04-06 | 2009-10-06 | 2010-04-07 | $7700.00 | Window Closed / Lapsed |