US6934748B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Automated on-line experimentation to measure users behavior to treatment for a set of content elements
Assignee: Accenture Global Services Ltd
•
Filing: 2000-08-25
•
Granted: 2005-08-23
•
Expires: 2020-08-25
Abstract
Abstract An automated system for experimentation includes an experiment engine which can define an experiment relating to various treatments for a set of content elements. The experiment engine conducts the experiment over a data network. An observation module collects observation data relating to user behavior for each treatment. A scripting/scheduling engine coordinates the operation of the experiment engine and the observation module.
Inventors
- 👤 Jordan J. Louviere
- 👤 Hikaru Phillips
- 👤 Jason P. Bennett
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-27 08:55
CPC Classifications
G; G06; G06Q; G06Q30/00; G06Q30/02
Claims Analysis
Total Claims: 55
Claim 1
Independent Claim
121 words
An automated computer-implemented method comprising: defining an experiment to gauge user reaction to various treatments for a set of content elements, conducting the experiment over a data network; and collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises: statistically sampling to specify a plurality of control groups, each control group comprising at least one user; specifying a particular treatment to be delivered to the at least one user in each control group; receiving identical requests for content from the respective at least one user in each of the control groups; and in response to the identical requests, delivering to the at least one user in each control group a different treatment.
Claim 2
Dependent Claim
121 words
An automated computer-implemented method comprising: defining an experiment to gauge user reaction to various treatments for a set of content elements, conducting the experiment over a data network; and collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises: statistically sampling to specify a plurality of control groups, each control group comprising at least one user; specifying a particular treatment to be delivered to the at least one user in each control group; receiving identical requests for content from the respective at least one user in each of the control groups; and in response to the identical requests, delivering to the at least one user in each control group a different treatment.
Claim 3
Dependent Claim
29 words
The method of claim 1 wherein defining an experiment comprises: identifying desired objectives for user behavior; and identifying which treatments may influence user behavior related to the desired objectives.
Claim 4
Dependent Claim
29 words
The method of claim 2 further comprising: defining at least one control variable for the various treatments; and assigning a respective level for the control variable for each treatment.
Claim 5
Dependent Claim
46 words
The method of claim 1 wherein conducting the experiment comprises: selecting at least one treatment for delivery to users; grouping users into a plurality of segments, each segment comprising users with similar behavioral characteristics; and specifying a particular segment of users to receive the selected treatment.
Claim 6
Dependent Claim
16 words
The method of claim 1 further comprising observing site-related behavior of users receiving the various treatments.
Claim 7
Dependent Claim
20 words
The method of claim 5 wherein collecting observation data comprises recording observed behavior in each control group during the experiment.
Claim 8
Dependent Claim
12 words
The method of claim 1 wherein the data network comprises the internet.
Claim 9
Dependent Claim
17 words
The method of claim 1 further comprising identifying elements of content which potentially influence behavior of users.
Claim 10
Dependent Claim
67 words
The method of claim 1 wherein the experiment is delivered over the data network to respective control groups of users according to the experiment and wherein the method further comprises generating a model for predicting anticipated behavior of users based upon the observed behavior; and generating a prediction for delivering one of the treatments to users according to the model in order to optimize a desired outcome.
Claim 11
Dependent Claim
20 words
The method of claim 9 wherein defining an experiment comprises identifying elements of content which potentially influence behavior of users.
Claim 12
Dependent Claim
147 words
An automated system comprising: a content system operable to store content, the content comprising a set of content elements; and a communication management system in communication with the content system and operable to define an experiment relating to various treatments for the set of content elements, conduct the experiment over a data network, and collect over the data network observation data relating to user behavior for each treatment, wherein the system is configured for statistically sampling to specify a plurality of control groups, each control group comprising at least one user; specifying a particular treatment to be delivered to the at least one user in each control group; receiving identical requests for content from the respective at least one user in each of the control groups; and in response to the identical requests, delivering to the at least one user in each control group a different treatment.
Claim 13
Dependent Claim
147 words
An automated system comprising: a content system operable to store content, the content comprising a set of content elements; and a communication management system in communication with the content system and operable to define an experiment relating to various treatments for the set of content elements, conduct the experiment over a data network, and collect over the data network observation data relating to user behavior for each treatment, wherein the system is configured for statistically sampling to specify a plurality of control groups, each control group comprising at least one user; specifying a particular treatment to be delivered to the at least one user in each control group; receiving identical requests for content from the respective at least one user in each of the control groups; and in response to the identical requests, delivering to the at least one user in each control group a different treatment.
Claim 14
Dependent Claim
21 words
The system of claim 11 wherein the communication management system is operable to identify content elements which may influence user behavior.
Claim 15
Dependent Claim
25 words
The system of claim 11 wherein the communication management system is operable to generate a set of experiment rules for allocating treatments during the experiment.
Claim 16
Dependent Claim
21 words
The system of claim 11 wherein the content system is operable to allocate treatments to users according to the experiment rules.
Claim 17
Dependent Claim
28 words
The system of claim 11 wherein the communication management system is operable to group users into a plurality of segments, each segment comprising users with similar behavioral characteristics.
Claim 18
Dependent Claim
12 words
The system of claim 11 wherein said data network comprises the internet.
Claim 19
Dependent Claim
36 words
The system of claim 11 wherein the communication management system is operable to define at least one control variable for the various treatments and to assign a respective level for the control variables for each treatment.
Claim 20
Dependent Claim
20 words
The system of claim 11 wherein the content system and the communication management system are connected to the data network.
Claim 21
Dependent Claim
23 words
The system of claim 11 wherein the content system comprises an allocator module operable to support an interface with the communication management system.
Claim 22
Dependent Claim
24 words
The system of claim 11 wherein the communication management system comprises an allocator interface object operable to support an interface with the content system.
Claim 23
Dependent Claim
25 words
The system of claim 11 further comprising a content provider interface operable to support an interface between the communication management system and a manager user.
Claim 24
Dependent Claim
28 words
The system of claim 11 wherein the communication management system is operable to generate a set of prediction rules for allocating treatments to users with specific behavioral characteristics.
Claim 25
Dependent Claim
21 words
The system of claim 22 wherein the content system is operable to allocate treatments to users according to the prediction rules.
Claim 26
Dependent Claim
14 words
The system of claim 11 wherein the communication management system implements a personalization process.
Claim 27
Dependent Claim
31 words
The system of claim 11 wherein the designed experiment reduces the number of combinations included in the experiment by measuring and estimating only those effects that are of a predetermined interest.
Claim 28
Dependent Claim
90 words
A computer-implemented method for on-line experimentation comprising: defining a plurality of treatments for a set of content elements; receiving requests for content from users over a data network; statistically sampling to create at least one control group of users; allocating over the data network a first treatment to each user in the control group; allocating over the data network a second treatment to each user not in the control group; and collecting observation data for observed behavior of users in the control group and users not in the control group.
Claim 29
Dependent Claim
90 words
A computer-implemented method for on-line experimentation comprising: defining a plurality of treatments for a set of content elements; receiving requests for content from users over a data network; statistically sampling to create at least one control group of users; allocating over the data network a first treatment to each user in the control group; allocating over the data network a second treatment to each user not in the control group; and collecting observation data for observed behavior of users in the control group and users not in the control group.
Claim 30
Dependent Claim
16 words
The method of claim 26 further comprising identifying elements of content which may influence user behavior.
Claim 31
Dependent Claim
31 words
The method of claim 26 wherein providing a website comprises providing a web page at which the set of content elements is available in the form of any of the treatments.
Claim 32
Dependent Claim
12 words
The method of claim 26 wherein the data network comprises the internet.
Claim 33
Dependent Claim
151 words
An online experimentation system configured to automatically create various treatments for a set of content elements according to systematic rules of experimentation that determine a number of alternative treatments using various combinations of the content elements from the set, and to allocate the treatments to users to test user behavior in response to the various treatments, the system being further configured to capture data relating to the observed behavior of users relative to the various treatments and to perform one or more methods comprising: (1) analyze the data to derive a model in form of a contingency table; (2) process the data for pathologies, including at least one of missing data, and structural dependencies, and processing the data to be model ready; (3) make inferences about some variables that influence user behavior based on random utility theory; and (4) modeling user behavior based on Bayesian Markov Chain Monte Carlo estimation procedure.
Claim 34
Dependent Claim
151 words
An online experimentation system configured to automatically create various treatments for a set of content elements according to systematic rules of experimentation that determine a number of alternative treatments using various combinations of the content elements from the set, and to allocate the treatments to users to test user behavior in response to the various treatments, the system being further configured to capture data relating to the observed behavior of users relative to the various treatments and to perform one or more methods comprising: (1) analyze the data to derive a model in form of a contingency table; (2) process the data for pathologies, including at least one of missing data, and structural dependencies, and processing the data to be model ready; (3) make inferences about some variables that influence user behavior based on random utility theory; and (4) modeling user behavior based on Bayesian Markov Chain Monte Carlo estimation procedure.
Claim 35
Dependent Claim
126 words
A machine readable medium including a set of instructions for: defining an experiment to gauge user reaction to various treatments for a set of content elements; conducting the experiment over a data network; and collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises: statistically sampling to specify a plurality of control groups, each control group comprising at least one user; specifying a particular treatment to be delivered to the at least one user in each control group; receiving identical requests for content from the respective at least one user in each of the control groups; and in response to the identical requests, delivering to the at least one user in each control group a different treatment.
Claim 36
Dependent Claim
126 words
A machine readable medium including a set of instructions for: defining an experiment to gauge user reaction to various treatments for a set of content elements; conducting the experiment over a data network; and collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises: statistically sampling to specify a plurality of control groups, each control group comprising at least one user; specifying a particular treatment to be delivered to the at least one user in each control group; receiving identical requests for content from the respective at least one user in each of the control groups; and in response to the identical requests, delivering to the at least one user in each control group a different treatment.
Claim 37
Dependent Claim
132 words
A content page comprising a set of content elements generated by a computer implemented system for: defining an experiment to gauge user reaction to various treatments for a set of content elements; conducting the experiment over a data network; and collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises: statistically sampling to specify a plurality of control groups, each control group comprising at least one user; specifying a particular treatment to be delivered to the at least one user in each control group; receiving identical requests for content from the respective at least one user in each of the control groups; and in response to the identical requests, delivering to the at least one user in each control group a different treatment.
Claim 38
Dependent Claim
132 words
A content page comprising a set of content elements generated by a computer implemented system for: defining an experiment to gauge user reaction to various treatments for a set of content elements; conducting the experiment over a data network; and collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises: statistically sampling to specify a plurality of control groups, each control group comprising at least one user; specifying a particular treatment to be delivered to the at least one user in each control group; receiving identical requests for content from the respective at least one user in each of the control groups; and in response to the identical requests, delivering to the at least one user in each control group a different treatment.
Claim 39
Dependent Claim
97 words
An automated system for experimentation, the system being configured to: generate and allocate a set of treatments from a predefined set of control elements, according to experiment designs, the system capable of determining a design by selecting from design criteria comprising: the specific effects allowed; the number of factors and factor levels included; and the amount of information produced in relation to the objective outcome; and collect observation data of user behavior to the set of treatments and determine the effect of each control element on user behavior according to choice models, behavioral models, or contingency tables.
Claim 40
Dependent Claim
97 words
An automated system for experimentation, the system being configured to: generate and allocate a set of treatments from a predefined set of control elements, according to experiment designs, the system capable of determining a design by selecting from design criteria comprising: the specific effects allowed; the number of factors and factor levels included; and the amount of information produced in relation to the objective outcome; and collect observation data of user behavior to the set of treatments and determine the effect of each control element on user behavior according to choice models, behavioral models, or contingency tables.
Claim 41
Dependent Claim
31 words
The system of claim 33 wherein the experiment engine is operable to implement statistical sampling procedures to deliver over the data network the various treatments to respective control groups of users.
Claim 42
Dependent Claim
12 words
The system of claim 33 wherein the data network comprises the internet.
Claim 43
Dependent Claim
43 words
The system of claim 33 wherein the system is further configured to determine and allocate a treatment from the set of content elements to increase the probability of achieving a desired objective based on the determination of the effects of the content elements.
Claim 44
Dependent Claim
139 words
A computer implemented method comprising: automatically creating various treatments for a set of content elements according to systematic rules of experimentation that determine a number of alternative treatments using various combinations of the content elements from the set; allocating the treatments to users to test user behavior in response to the various treatments; capturing data relating to the observed behavior of users relative to the various treatments; performing one or more of: (1) analyzing the data to derive a model in form of a contingency table; (2) processing the data for pathologies, including at least one of missing data, and structural dependencies, and processing the data to be model ready; (3) making inferences about some variables that influence user behavior based on random utility theory; and (4) modeling user behavior based on Bayesian Markov Chain Monte Carlo estimation procedure.
Claim 45
Dependent Claim
139 words
A computer implemented method comprising: automatically creating various treatments for a set of content elements according to systematic rules of experimentation that determine a number of alternative treatments using various combinations of the content elements from the set; allocating the treatments to users to test user behavior in response to the various treatments; capturing data relating to the observed behavior of users relative to the various treatments; performing one or more of: (1) analyzing the data to derive a model in form of a contingency table; (2) processing the data for pathologies, including at least one of missing data, and structural dependencies, and processing the data to be model ready; (3) making inferences about some variables that influence user behavior based on random utility theory; and (4) modeling user behavior based on Bayesian Markov Chain Monte Carlo estimation procedure.
Claim 46
Dependent Claim
92 words
A computer implemented method, comprising: generating and allocating a set of treatments from a predefined set of content elements, according to experiment designs, the system capable of determining a design by selecting from design criteria comprising: the specific effects allowed; the number of factors and factor levels included; and the amount of information produced in relation to the objective outcome; and collecting observation data of user behavior for the set of treatments and determining the effect of each content element on user behavior according to choice models, behavioral models, or contingency tables.
Claim 47
Dependent Claim
92 words
A computer implemented method, comprising: generating and allocating a set of treatments from a predefined set of content elements, according to experiment designs, the system capable of determining a design by selecting from design criteria comprising: the specific effects allowed; the number of factors and factor levels included; and the amount of information produced in relation to the objective outcome; and collecting observation data of user behavior for the set of treatments and determining the effect of each content element on user behavior according to choice models, behavioral models, or contingency tables.
Claim 48
Dependent Claim
38 words
The method of claim 38 further comprising determining and allocating a treatment from the set of content elements to increase the probability of achieving a desired objective based on the determination of the effects of the content elements.
Claim 49
Dependent Claim
148 words
A machine readable medium storing a set of instructions for performing the steps comprising: automatically creating various treatments for a set of content elements according to systematic rules of experimentation that determine a number of alternative treatments using various combinations of the content elements from the set; allocating the treatments to users to test user behavior in response to the various treatments capturing data relating to the observed behavior of users relative to the various treatments; performing one or more of: (1) analyzing the data to derive a model in form of a contingency table; (2) processing the data for pathologies, including at least one of missing data, and structural dependencies, and processing the data to be model ready; (3) making inferences about some variables that influence user behavior based on random utility theory; and (4) modeling user behavior based on Bayesian Markov Chain Monte Carlo estimation procedure.
Claim 50
Dependent Claim
148 words
A machine readable medium storing a set of instructions for performing the steps comprising: automatically creating various treatments for a set of content elements according to systematic rules of experimentation that determine a number of alternative treatments using various combinations of the content elements from the set; allocating the treatments to users to test user behavior in response to the various treatments capturing data relating to the observed behavior of users relative to the various treatments; performing one or more of: (1) analyzing the data to derive a model in form of a contingency table; (2) processing the data for pathologies, including at least one of missing data, and structural dependencies, and processing the data to be model ready; (3) making inferences about some variables that influence user behavior based on random utility theory; and (4) modeling user behavior based on Bayesian Markov Chain Monte Carlo estimation procedure.
Claim 51
Dependent Claim
101 words
A machine readable medium storing a set of instructions for performing the steps comprising: generating and allocating a set of treatments from a predefined set of content elements, according to experiment designs, the system capable of determining a design by selecting from design criteria comprising: the specific effects allowed; the number of factors and factor levels included; and the amount of information produced in relation to the objective outcome; and collecting observation data of user behavior for the set of treatments and determining the effect of each content element on user behavior according to choice models, behavioral models, or contingency tables.
Claim 52
Dependent Claim
101 words
A machine readable medium storing a set of instructions for performing the steps comprising: generating and allocating a set of treatments from a predefined set of content elements, according to experiment designs, the system capable of determining a design by selecting from design criteria comprising: the specific effects allowed; the number of factors and factor levels included; and the amount of information produced in relation to the objective outcome; and collecting observation data of user behavior for the set of treatments and determining the effect of each content element on user behavior according to choice models, behavioral models, or contingency tables.
Claim 53
Dependent Claim
42 words
The machine readable medium of claim 41 further comprising instructions for determining and allocating a treatment from the set of content elements to increase the probability of achieving a desired objective based on the determination of the effects of the content elements.
Claim 54
Dependent Claim
121 words
A computer implemented method, comprising: defining an experiment to gauge user reaction to various treatments for a set of content elements; conducting the experiment over a data network; and collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises: statistically sampling to specify a plurality of control groups, each control group comprising at least one user; specifying a particular treatment to be delivered to the at least one user in each control group; receiving identical requests for content from the respective at least one user in each of the control groups; and in response to the identical requests, delivering to the at least one user in each control group a different treatment.
Claim 55
Dependent Claim
121 words
A computer implemented method, comprising: defining an experiment to gauge user reaction to various treatments for a set of content elements; conducting the experiment over a data network; and collecting over the data network observation data relating to user behavior for each treatment wherein conducting the experiment comprises: statistically sampling to specify a plurality of control groups, each control group comprising at least one user; specifying a particular treatment to be delivered to the at least one user in each control group; receiving identical requests for content from the respective at least one user in each of the control groups; and in response to the identical requests, delivering to the at least one user in each control group a different treatment.
References Cited (Backward Citations)
US5124911A
Cited
US6020883A
Cited
US6029195A
Cited
US6460036B1
Cited
US5918014A
Cited
US5872850A
Cited
US5809242A
Cited
US5848396A
Cited
US5968125A
Cited
US6430539B1
Cited
WO2001014952A2
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2008-08-22 | 2009-02-21 | 2009-08-23 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2012-08-22 | 2013-02-21 | 2013-08-23 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2016-08-22 | 2017-02-21 | 2017-08-23 | $7700.00 | Window Closed / Lapsed |