US20070156382A1 - Systems and methods for designing experiments - Google Patents

Systems and methods for designing experiments Download PDF

Info

Publication number
US20070156382A1
US20070156382A1 US11/321,340 US32134005A US2007156382A1 US 20070156382 A1 US20070156382 A1 US 20070156382A1 US 32134005 A US32134005 A US 32134005A US 2007156382 A1 US2007156382 A1 US 2007156382A1
Authority
US
United States
Prior art keywords
experiment
content
system
designing
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US11/321,340
Inventor
James Graham
Craig Carlson
Brian Brooks
David Engler
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
3M Innovative Properties Co
Original Assignee
3M Innovative Properties Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 3M Innovative Properties Co filed Critical 3M Innovative Properties Co
Priority to US11/321,340 priority Critical patent/US20070156382A1/en
Assigned to 3M INNOVATIVE PROPERTIES COMPANY reassignment 3M INNOVATIVE PROPERTIES COMPANY ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BROOKS, BRIAN E., CARLSON, CRAIG M., ENGLER, DAVID A., GRAHAM II, JAMES L.
Publication of US20070156382A1 publication Critical patent/US20070156382A1/en
Application status is Abandoned legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/20Handling natural language data
    • G06F17/21Text processing
    • G06F17/24Editing, e.g. insert/delete
    • G06F17/248Templates
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object or an image, setting a parameter value or selecting a range
    • G06F3/04842Selection of a displayed object
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00Computer systems using knowledge-based models
    • G06N5/04Inference methods or devices
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06QDATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06QDATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce, e.g. shopping or e-commerce
    • G06Q30/02Marketing, e.g. market research and analysis, surveying, promotions, advertising, buyer profiling, customer management or rewards; Price estimation or determination
    • G06Q30/0241Advertisement
    • G06Q30/0242Determination of advertisement effectiveness
    • G06Q30/0245Surveys
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06QDATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce, e.g. shopping or e-commerce
    • G06Q30/02Marketing, e.g. market research and analysis, surveying, promotions, advertising, buyer profiling, customer management or rewards; Price estimation or determination
    • G06Q30/0241Advertisement
    • G06Q30/0242Determination of advertisement effectiveness
    • G06Q30/0246Traffic

Abstract

Methods and systems for designing an experiment using a computer to determine whether the experiment is a true experiment are described. These approaches allow a user who is unsophisticated in the complexities of true experimental design to design and deploy an experiment that produces substantially confound-free results and can be used to determine and quantify any causal relationship between independent and dependent variables. The computer may select one or more independent and/or dependent variables of the experiment or may assist the user in selection of independent and/or dependent variables. Formation of control and treatment groups, randomization and/or blocking to reduce the effects of confounding variables may be performed by the computer with or without input from the user.

Description

    FIELD OF THE INVENTION
  • The present invention relates to methods and systems for designing true experiments.
  • BACKGROUND
  • Experiments are typically conducted to determine empirically if there are relationships between two or more variables. An experiment typically begins with the formation of one or more hypotheses positing that there is a relationship between one or more independent variables and one or more dependent variables. For example, a researcher at a pharmaceutical company might formulate a hypothesis that the amount of a new drug that patients take will be related to the blood pressure of patients. Independent variables are the variables defined or manipulated by the experimenter during an experiment (e.g., the amount and/or frequency of a drug administered to patients). Dependent variables are the variables posited to depend on the value of the independent variable (e.g., the blood pressure of patients). The experimenter then conducts an experiment to determine if there is indeed a relationship between the independent and dependent variables (e.g., if the amount of a drug patients receive is related to the blood pressure of patients).
  • Confounding variables (things that vary systematically with the levels of the independent variable) may also influence the dependent variable. These confounding variables are not of primary interest in the experiment, yet can influence the dependent variables. Some examples of confounding variables include regression to the mean, order effects, floor-effects, ceiling effects, Hawthorne effects, and demand characteristics. Confounding variables make it impossible to know which factor (variable) caused any observed change in the dependent variable(s). And thus, the existence of confounding variables that are not properly controlled during the experiment renders it impossible to make statistical inferences about causal relationships between the independent and dependent variables. Various types of experiments may be distinguished by the manner and degree to which they are able to reduce or eliminate the effects of confounding variables. The term “true experiment” denotes an experiment in which:
  • 1. There are at least two levels of an independent variable.
  • 2. Samples are randomly assigned to levels of the independent variable. That is, each sample in the experiment is equally likely to be assigned to levels of the independent variable.
  • 3. There is some method of controlling for or eliminating confounds.
  • Experiments that lack any of the above three characteristics are not true experiments, and are often referred to as quasi-experiments or correlational designs. Only true experiments allow statistical inferences to be drawn regarding the causal relationships between independent and dependent variables. Quasi-experiments and correlational designs may allow relationships between independent and dependent variables to be established, but it is not possible to determine whether those relationships are causal. Various types of experimental designs (including true experiments) have been described, for example, in Campbell, D. T., & Stanley, J. C. (1963) Experimental and quasi-experimental designs for research, Chicago: Rand McNally. Data produced by a true experiment are substantially unaffected by confounding variables. However, the complexity of designing of a true experiment that appropriately controls or eliminates confounding variables may be significant.
  • It is also desirable to design experiments that have a sufficient degree of internal and external validity. Internal validity refers to the confidence that the independent variables caused any observed difference in the dependent variables. External validity refers to the confidence that the observed relationship between the independent and dependent variable in the experiment will apply to settings or situations outside of the settings of the experiment. Designing a true experiment having sufficient internal and external validity may be daunting for investigators who have only a limited knowledge of the statistical and experimental design principles. Systems and methods that provide investigators with a simplified approach to designing true experiments are desirable.
  • SUMMARY OF THE INVENTION
  • The present invention is directed to systems and methods for designing experiments. One embodiment of the invention involves a method for designing an experiment using a computer to determine whether the experiment is a true experiment.
  • According to various approaches, the computer may select one or more independent variables of the experiment and/or may select one or more dependent variables. The computer may automatically form one or more control groups of the experiment and/or one or more treatment groups of the experiment, including automatically randomizing the treatment or control groups. In some implementations, the computer may apply techniques (for example, blocking and counterbalancing) to reduce effects of one or more confounding variables.
  • According to other approaches, the computer may assist the user in various steps involving the experiment. For example, the computer may assist the user in selection of one or more independent variables and/or one or more dependent variables. The computer may assist the user in forming at least one of a control group and a treatment group. The computer may assist the user in randomizing samples to control and treatment groups and may alternatively or additionally apply techniques to reduce effects of one or more confounding variables of the experiment.
  • Another aspect of the invention involves performing the experiment. The computer may automatically run the experiment or may perform various functions to assist the user in running the experiment.
  • Another aspect of the invention is directed to analyzing results of the true experiment. Some implementations allow for the analysis to be performed automatically by the computer. In other implementations, the computer assists the user in analyzing the results of the experiment.
  • Another embodiment of the invention is directed to system for experimental design, the system including a design processor configured to determine whether an experiment is a true experiment. In some implementations, the design processor may be configured to select at least one of an independent variable and a dependent variable of the experiment. The design processor may be configured to form at least one of a control group and a treatment group of the experiment, including performing randomizing and blocking.
  • In some implementations, the system may assist the user in various functions associated with the experiment. In these implementations, the system includes a user interface configured to accept input from a user. The design processor may be configured to assist the user in selection of at least one of an independent variable and a dependent variable of the experiment using the user input. The design processor may be configured to assist the user in selection of at least one of a control group and a treatment group of the experiment based on the user input and may also assist the user in randomizing the groups and in applying blocking to reduce effects of one or more confounding variables of the experiment based on the user input.
  • According to another aspect of the invention, the system may include a deployment unit configured to run the experiment and/or an analysis unit configured to analyze results of the experiment. Deployment and/or analysis of the experiment may be performed automatically by the system or using input from a user.
  • In one implementation, the experiment comprises a digital signage experiment. In another implementation, the experiment involves an advertisement.
  • The above summary of the present invention is not intended to describe each embodiment or every implementation of the present invention. Advantages and attainments, together with a more complete understanding of the invention, will become apparent and appreciated by referring to the following detailed description and claims taken in conjunction with the accompanying drawings.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 illustrates elements needed for a true experiment;
  • FIG. 2A is a block diagram of a digital signage system that may incorporate the capability for designing true experiments in accordance with embodiments of the invention;
  • FIG. 2B is a block diagram of a system for designing true experiments in accordance with embodiments of the invention;
  • FIG. 3 is a flowchart illustrating a method that includes the design of a true experiment in accordance with embodiments of the invention;
  • FIGS. 4A-4C are flowcharts of a method incorporating designing experiments for digital signage implementations in accordance with embodiments of the invention;
  • FIG. 5 illustrates an exemplary layout for a digital signage display, including a weather/news panel, store logo, text crawl and area for video advertisements that may be implemented in accordance with embodiments of the invention;
  • FIG. 6 conceptually illustrates the functionality of a semi-automatic digital signage system in accordance with embodiments of the invention;
  • FIG. 7 illustrates the process flow of creating and deploying content using the components and functionality of a digital signage system in accordance with embodiments of the invention;
  • FIG. 8 is a flowchart illustrating an exemplary implementation of a digital signage system for a sporting goods retailer in accordance with an embodiment of the invention; and
  • FIG. 9 is a flowchart illustrating a method of determining if an experimental design eliminates confounds from the experiment in accordance with embodiments of the invention.
  • While the invention is amenable to various modifications and alternative forms, specifics thereof have been shown by way of example in the drawings and will be described in detail. It is to be understood, however, that the intention is not to limit the invention to the particular embodiments described. On the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the scope of the invention as defined by the appended claims.
  • DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS
  • In the following description of the illustrated embodiments, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration, various embodiments in which the invention may be practiced. It is to be understood that the embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.
  • The present invention is directed to methods and systems that use a computer to determine whether the design of an experiment is a true experiment. The elements needed for a true experiment are illustrated in FIG. 1. A true experiment includes development of a hypothesis or objective. Dependent and independent variables are identified, and at least two levels of an independent variable are used. Samples are randomly assigned to levels of the independent variable. There is some kind of method for controlling for or eliminating confounding variables. If all of these elements are appropriately applied, the experiment produces results that can be used to make statistical inferences about the relationship between the dependent and independent variables. Methods and systems described herein allow a user who is unsophisticated in the complexities of true experimental design to design and deploy an experiment that produces substantially confound-free results and can be used to determine and quantify any causal relationship between independent and dependent variables.
  • A true experiment has at least two levels of an independent variable. As described herein, some embodiments of the invention provide methods and systems that assist users in choosing independent variables for the experiment and in balancing between internal and external validity. For example, with respect to threats to internal validity, the methods and systems of the present invention assist the user through the process of identifying threats to internal validity, and may suggest and/or automate methods of controlling these threats, such as through counterbalancing and/or blocking. Some embodiments herein assist the user and/or automate the process of assigning samples randomly to groups so that each sample in an experiment is equally likely to be assigned to levels of the independent variable. In some configurations, the randomization, counterbalancing and/or blocking may be automatically performed. The system may select, or may assist the user in selecting, independent variables (or levels of independent variables) and dependent variables based factors associated with internal and/or external validity.
  • In yet other embodiments, the methods and systems of the present invention may be used to evaluate previously designed or conducted experiments. In these embodiments, based on input from the user regarding how an experiment was previously designed or conducted, the system determines if the experiment was indeed a true experiment (as opposed to a quasi-experiment or correlational study) and/or identifies the existence of confounds in the experiment. In some implementations, the approaches of the present invention may be used to determine the internal and/or external validity of an experimental design.
  • In some embodiments, the computer may operate in a semi-automatic mode, wherein the user is led by the computer through one or more interactive sessions to design, deploy, and/or analyze data acquired from a true experiment. In other embodiments, the computer is programmed to operate fully automatically without user interaction. In a fully automatic mode, a computer-based system may perform one or more of designing the experiment, deploying the experiment, acquiring data produced by the experiment, analyzing the data, determining internal validity of the experiment, determining external validity of the experiment, and/or modifying or implementing one or more processes based on the analysis. In yet other embodiments, the system may perform one or more of the steps described above semi-automatically and may perform another one or more of the steps fully automatically. The computer-based approaches to experimental design are described herein based on a computerized signage information system. The present invention is not limited, however, to the fields of communications systems or signage. The approaches of the present invention may be applied to the design of a true experiment regardless of the field of interest. For example, the methods and systems described herein may be applied to the design of experiments for any number of subject areas, including, but not limited to, biology, chemistry, linguistics, medicine, cognitive sciences, social sciences, education, economics, and/or other scientific fields. The examples are described in the context of a digital signage information system to allow the reader to develop an understanding of the principles of the invention which generally span all fields of scientific endeavor.
  • FIG. 2A is a block diagram of a digital signage system (DSS) that may incorporate the capability for designing true experiments in accordance with embodiments of the invention. The block diagram of FIG. 2A illustrates one configuration of a DSS divided into functional blocks. Those skilled in the art will appreciate that the DSS may be alternatively illustrated using different function blocks and that various components of the DSS may be implemented as hardware, software, firmware, or any combination of hardware, software and firmware.
  • The DSS illustrated in FIG. 2A is a computerized system configured to present informational content via audio, visual, and/or other media formats. The DSS may include functionality to automatically or semi-automatically generate playlists, which provide a list of the information content to be presented, and schedules, which define an order for the presentation of the content. In a semi-automatic mode, a user may access a DSS control processor 205 via an interactive user interface 210. Assisted by the DSS control processor 205, the user may identify content to be presented and generate playlists and schedules that control the timing and order of presentations on one or more DSS players 215. Each player 215 presents content to recipients according to a playlist and schedule developed for the player. The informational content may comprise graphics, text, video clips, still images, audio clips, web pages, and/or any combination of video and/or audio content, for example.
  • In some implementations, after a playlist and schedule are developed, the DSS control processor 205 determines the content required for the playlist, downloads the content from a content server, and transfers the content along with the playlist and schedule to a player controller 220 that distributes content to the players 215. Although FIG. 2A shows only one player controller 220, multiple player controllers may be coupled to a single DSS control processor 205. Each player controller 220 may control a single player 215 or multiple players 215. The content and/or the playlists and schedules may be transferred from the DSS control processor 205 to the one or more player controllers 220 in a compressed format with appropriate addressing providing information identifying the player 215 for which the content/playlist/schedule is intended. In some applications, the players 215 may be distributed in stores and the content presented on the players 215 may be advertisements.
  • In other implementations, the DSS control processor 205 may transfer only the playlists and schedules to the player controller 220. If the content is not resident on the player controller 220, the player controller 220 may access content storage 225 to acquire the content to be presented. In some scenarios, one or more of the various components of the DSS system, including the content storage 225, may be accessible via a network connection, such as an intranet or Internet connection. The player controller 220—may assemble the desired content, or otherwise facilitate display of the desired content on the players according to the playlist and schedule. The playlists, schedules, and/or content presented on the players 215 can be modified periodically or as desired by the user through the player controller 220, or through the DSS control processor 205, for example.
  • In some implementations, the DSS control processor 205 facilitates the development and/or formatting of a program of content to be played on a player. For example, the DSS control processor 205 may facilitate formatting of an audiovisual program through the use of a template. The template includes formatting constraints and/or rules that are applied in the development of an audiovisual program to be presented. For example, the template may include rules associated with the portions of the screen used for certain types of content, what type of content can be played in each segment, and in what sequence, font size, and/or other constraints or rules applicable to the display of the program. A separate set of rules and/or constraints may be desirable for each display configuration. In some embodiments, formatting a program for different displays may be performed automatically by the DSS control processor 205.
  • In some embodiments, the DSS may create templates, generate content, select content, assemble programs, and/or format programs to be displayed based on information acquired through research and experimentation in the area of cognitive sciences. Cognitive science seeks to understand the mechanisms of human perception. The disciplines of cognitive and vision sciences have generated a vast knowledge base regarding how human perceptual systems process information, the mechanisms that underlie attention, how the human brain stores and represents information in memory, and the cognitive basis of language and problem solving. Application of the cognitive sciences to content design, layout, formatting, and/or content presentation yields information that is easily processed by human perceptual systems, is easy to understand, and is easily stored in human memory. Knowledge acquired from the cognitive sciences and stored in a cognitive sciences database 230 may be used automatically or semi-automatically to inform one or more processes of the DSS including creation of templates, content design, selection of content, distribution of content, assembly of programs, and/or formatting of programs for display. The cognitive sciences database 230 used in conjunction with the programming of the DSS yields advertisements or other digital signage programs that are enhanced by the teachings of cognitive science, while relieving the system user from needing specific training in the field.
  • In development of a digital signage program, e.g., ad campaign or the like, the DSS control processor 205 may guide a user through various processes that are enhanced using knowledge acquired through the cognitive sciences. For example, information stored in the cognitive sciences database 230 may be applied to the choice of templates to produce an optimal program layout and/or to the selection of content, such as whether content elements should be graphical, text, involve movement, color, size, and/or to the implementation of other aspects of program development.
  • The DSS may include the capability for designing alternative versions of a digital signage program to accommodate diverse display types and viewing conditions. Display technology is diverse and there are large differences in the types of displays used to present content on a digital signage network. For example, the size, shape, brightness, and viewing conditions will vary greatly across a digital signage network (e.g., some displays will be small, flexible and non-rectilinear, whereas others will be standard large format Liquid Crystal Display (LCD) and plasma displays). The variation in display types and viewing conditions means that any single version of a piece of content will not be optimal for all the displays across a network. In order to overcome this problem, it may be necessary to generate versions of each piece of content for each display type and viewing environment, and to selectively distribute these versions of content to their corresponding screens in the network. However, it is not realistic to expect content designers to have such detailed knowledge of the display types and viewing conditions across a large DSS network. Furthermore, even if such content designers had such detailed knowledge, it would be time-consuming to manually create versions of content for each display and to manually schedule the content to play on each corresponding display at the appropriate time.
  • The DSS may include a data acquisition unit 235 for collecting data used to improve the effectiveness of deployed content. The data acquisition unit 235 allows distribution factors that underlie the effectiveness of digital signage networks to be continuously gathered in real-time during deployment of content. The information acquired can facilitate continuous improvement in content effectiveness of the DSS as well as improvement of individual versions of content pieces. Real-time data may be used to learn what sensor or sales events should trigger the display of specific types of content, for example.
  • Individual pieces of content in any content program each have a specific goal (e.g., to sell a specific product). It is usually the case that there is variability in the value of each goal to the user of the digital signage network. For example, there may be variability in the profit margin and inventory level for each product which factor into the value of the goal for the product. The value of achieving each goal continuously changes during the time a digital signage program is deployed. For example, the inventory level of a product may change, thus affecting the goal for sales of the product.
  • Enhancing the effectiveness of a DSS as a whole, involves 1) accurate prediction of the impact of deploying a digital signage program on the goal associated with the digital signage program, and 2) continuously changing the distribution patterns (timing, frequency, and location) of individual pieces of content as the value of each individual goal corresponding to the pieces of content change. In many cases, it is unfeasible for users of the DSS to predict the impact of deploying content and to manually change content distribution patterns based on continuously changing values of goals associated with each piece of content. The DSS provides the functionality to predict the impact of digital signage programs and to alter the distribution of content based on the predictions.
  • As previously stated, content is displayed on the players 215 with the goal of affecting human behavior (e.g., to impact purchasing behavior). However, prior digital signage systems are unable to demonstrate a cause-and-effect relationship between signage content and human behavior or to measure the strength of the cause and effect relationship. This difficulty arises because the methods by which content is delivered across current digital signage networks does not support the determination of whether any measured change in human behavior was caused by signage content or the result of some confounding factors (e.g., change in weather, change in general demand for the product, change in price of the product). The only way to decisively determine cause-and-effect relationships between signage content and human behavior is to conduct a true experiment during which signage content is systematically manipulated using complex experimental designs, and the effects of those manipulations on human behavior are carefully measured. Manually conducting such experiments is time consuming and requires significant knowledge and training in the scientific method of how to design true experiments. The users of digital signage systems may not have sufficient training to understand how to design a true experiment to acquire confound-free results.
  • The DSS may include components that provide the capability to design, deploy, and/or analyze data acquired from true experiments. As previously discussed, the components providing this functionality may be incorporated into a DSS or may be implemented by other types of systems. Components that may be used in the design, deployment, and/or analysis of true experiments, regardless of the particular type of system in which they are implemented, are set forth separately in the block diagram of FIG. 2B. A system according to the present invention may include one or more of the features, structures, methods, or combinations thereof described herein. For example, a system may be implemented to include one or more of the advantageous features and/or processes illustrated in FIGS. 2A or 2B. It is intended that such a system need not include all of the features described herein, but may be implemented to include selected features that provide for useful structures and/or functionality.
  • FIG. 2B illustrates an experiment design system (EDS) including experiment design processor that is configured to ensure the design of a true experiment. As previously discussed, the experiment design processor 240 may be configured to operate fully automatically or semi-automatically with user interaction. In semi-automatic mode, the experiment design processor 240 may lead a user through various interactive sessions conducted via the user interface 210 to design a true experiment. In such a process, the experiment design processor 240 ensures the design of a true experiment that produces confound-free data. Thus, a user is able to rely on the programming of the experiment design processor 240 and is not required to have knowledge or experience in designing true experiments. The EDS may comprise only an experiment design processor 240, or may include additional elements such as an experiment deployment unit 245, a data acquisition unit 235, and data analysis unit 250.
  • The experiment design processor 240 may, automatically or semi-automatically, develop an objective or hypothesis for the experiment, identify independent and dependent variables of the experiment, form control and treatment groups applying appropriate randomization, counterbalancing and/or blocking. In the context of a DSS, for example, the experimental objective may be to evaluate the effectiveness of a content element in an ad campaign promoting sales of a certain product. The independent variable(s) may be associated with some aspect of the display of the content element. The dependent variable(s) may be associated with an increase in sales of the product.
  • The experiment design processor 240 may form appropriate treatment and control groups including the selection of various venues of the DSS system where the experimental content and control content is to be displayed. Presentation of the experimental content, including content format, schedule, presentation location, and/or other factors that may produce confounds into the experimental process, are controlled by the experiment design processor 240. The experiment design processor 240 may ensure adequate randomization, counterbalancing, and blocking of the control and treatment groups to achieve experimental results that are confound-free. Design of the experiment in the context of the DSS system may involve, for example, generating appropriate playlists and schedules for the presentation of content to be tested via the experiment, and may also involve generating playlists and schedules for presentation of control content.
  • The EDS may further include an experiment deployment unit 245. The experiment deployment unit 245 is configured to facilitate deployment of the experiment. In the context of the exemplary DSS system, the experiment deployment unit 245 formats the experimental content and the control group content for various player configurations and facilitates the transfer of the experimental content and the control content to the player controller 220 for presentation on players 215 as specified by the playlists and schedules.
  • The data acquisition unit 235 may be configured to collect experimental data from the control and treatment groups. The data acquisition unit 235 may perform or facilitate acquisition of data associated with the experiment via any means. For example, in the context of the exemplary DSS, the data acquisition unit 235 may be coupled to various sensor or data acquisition devices 262, 264, 266 that gather information including product movement, product sales, customer actions or reactions, and/or other information. Sensors 262 may be used to detect, for example, if a customer picks up the product, or if a customer is in the vicinity of the display when the content is displayed. Sales may be determined based on information acquired by a point of sales (POS) system 264. Other devices 266 that measure the dependent variable may also be used. Changes in inventory levels of a product may be available via an inventory control system. Customer reactions may be acquired via questionnaires. If the conducted experiment is a true experiment, the data acquired by the data acquisition unit 235 is substantially confound-free.
  • The data acquisition unit 235 may be coupled to a data analysis module 250 that is configured to analyze the experimental data collected by the data acquisition unit 235. The data analysis module 250 may determine and/or quantify cause and effect relationships between the independent and dependent variables of the experiment. For the illustrated DSS, the results of the analysis may be used to determine if the content is effective at influencing product sales.
  • The results of the analysis may be additionally or alternatively used to implement or modify various processes. For example, if the content was effective at influencing product sales, an advertisement campaign may be developed incorporating the content. A value may be assigned to the content by a content valuation process 272 based on the effectiveness of increasing sales. An advertiser using the content may be invoiced by a billing unit 274 according the value of the content. The data analysis module 250 may also provide information to inventory control 276. Additionally, the data analysis module 250 may provide information to a prediction unit 278 that generates a prediction of sales when the advertising campaign is deployed. The prediction unit 278 may additionally or alternatively predict the product inventory needed to support the sales generated by the advertisement campaign.
  • The flowchart illustrated in FIG. 3 provides an overview of a method that may be implemented by the DSS (FIG. 2A) and/or the EDS (FIG. 2B) in accordance with embodiments of the invention. The method includes design 310 and performance 320 of a true experiment. Data produced by the experiment are collected 330 and analyzed 340. One or more processes may be modified or implemented 350 based on the data analysis.
  • The flowcharts of FIGS. 4A-4C provide a more specific example of these processes in the context of digital signage in accordance with embodiments of the invention. In this example, the objective of the experiment is to determine the effect of a video advertisement for a DSS incorporating Content A on sales of Product X. For example, Content A may be an advertisement featuring a video of an actor, athlete, or other famous person. The content to be tested (Content A) is identified 402 and control content 404 is selected. A video advertisement incorporating Content A is produced 406 using template rules stored in local memory. The template rules may also be applied to produce video content to be used for the control group. The template rules may be used to provide a structure for arranging the layout of content on the display. In some cases, the template rules are based on information derived from the cognitive sciences. An exemplary layout for a digital signage display including a weather/news panel, store logo, text crawl, and area for video advertisements is illustrated in FIG. 5.
  • Playlists and schedules are specified 412, 414 for the video advertisement incorporating Content A and the control content. Venues for the advertisement and the control content are selected 416, 418. For example, the venues of the advertisement and control content may be restaurants, stores, shopping malls, or other locations. Development 412, 414 of the playlists and schedules, and selection 416, 418 of venues is performed using appropriate randomization and blocking to exclude or reduce confounding variables in the experimental results.
  • The advertisement and control content may be deployed to a number of venues, each venue having a particular set of viewing characteristics. For example, venues may vary with respect to display size, display shape, viewing distance, ambient lighting, noise level, and other viewing conditions. The advertisement is adjusted 422 to conform to the attributes of each display on which the advertisement it deployed. Similar adjustments are performed 424 for the control content. The advertisement and the control content are shown 426, 428 according to their respective playlists and schedules.
  • Data may be collected at each venue before, during and after display of the advertisement and/or the control content. The data may be collected via sensors, point of sale terminals, inventory control systems, and/or other input devices. For example, viewer presence in the vicinity of the display during presentation of the advertisement may be detected. The number of times the advertisement and control content was displayed 432, 434 or viewed 436, 438 may be detected. Viewer motion, eye movements, and/or interaction with Product X may be sensed 442. The volume and timing of sales of Product X may be determined 444 from point of sales terminals. Viewer responses to the advertisement may be acquired via questionnaires. For example, the questionnaires may be used to determine if the viewers reported a generally positive or generally negative reaction to the advertisement. Before and after questionnaires may be used to determine if the advertisement changed the consumer's level of familiarity with Product X. Changes in inventory levels of product X may be determined.
  • The collected data may be analyzed 446 to determine causal relationships between display of the advertisement containing Content A and sales of Product X. Based on the analysis, a value may be assigned 448 to Content A. If Content A is successful at increasing sales, an advertisement campaign may be generated 452 incorporating Content A. The return on investment (ROI) for the advertising campaign may be determined 454. The business providing the advertisement campaign or the systems for presenting the advertisement campaign may bill 456 their customers according to the value of Content A or predicted ROI as determined by the experiment. The sellers of Product X may predict inventory requirements 458 for Product X during an advertisement campaign incorporating Content A based on the results of the experiment. The system may iteratively modify 462 one or more processes based on the experimental results.
  • FIG. 5 illustrates an exemplary layout for a digital signage display that may be controlled by the DSS of the present invention. The digital signage display may be configured to include a number of areas such as a weather/news panel, a store logo graphic, text crawl, and area for video advertisements.
  • FIG. 6 conceptually illustrates the functionality of a semi-automatic DSS, such as the system illustrated in FIG. 2A, in accordance with embodiments of the invention. The DSS may functionally be broadly grouped into four areas. The first functional area illustrated in FIG. 6 provides for the application of cognitive and vision sciences 610 to digital signage. Programming tools are provided that allow content designers without advanced training in the visual and cognitive sciences to apply principles from these disciplines during the content creation process, in order to improve content effectiveness. The system prompts the user to input both the goal and the intended message (the critical information) for each piece of content. The user is assisted in the identification of key attributes across the digital signage network that have implications for content design. The system guides the user through the process of applying the cognitive and visual sciences to design content based on the goals and key digital signage network attributes. For example, the system would help users choose the templates (i.e., best layout) and the elements (e.g., whether elements should be graphical, text; involve movement, color, size, etc.) to display on the signs.
  • Another functional component of the DSS provides content effectiveness measurement 620. The programming of the DSS allows the user with little or no training or skills in conducting experiments to generate complex experimental designs. The experimental designs may be used to investigate the content design and distribution factors that underlie effective digital signage networks, and to measure the impact of content on human behavior. Users are assisted in identifying the independent variables likely to be critical to content effectiveness and the dependent variables corresponding to the independent variables. An appropriate experimental design is generated by the system, including identifying appropriate control and experimental conditions, appropriate blocking, counterbalancing, and randomization, to determine the strength of any causal relationship among and between those independent and dependent variables. The experiment is performed, data are collected via sensors and/or other processes, and is the data are analyzed. Results of the experiment are may be used by various other components of the system and/or may be reported to users.
  • The use of true experiments provides complex and rigorous methods to deliver content that allows the collection of very clear (confound-free) data. This is in contrast with the approach of using quasi-experiments which require extremely complex analysis methods (i.e., behavioral analytics) to analyze and use data that are fraught with confounds.
  • The DSS provides automated content design 630 that automatically generates new templates and applies transformations to existing elements. New templates and elements may be generated to improve the content effectiveness, and/or to create appropriate content to fulfill the needs of the experimental designs previously described. The tools provided by the DSS are capable of generating unique versions of pieces of content for each player in the system. The DSS system may prompt users to provide input or may use information supplied from other components regarding the network attributes and factors that underlie content effectiveness. Knowledge from the cognitive and visions sciences may be used to extrapolate, fill in, and otherwise explore the information space for the particular pieces of content the system aims to enhance. The functionality of the DSS provides the ability to generate completely new content that is not simply a reconfiguration of deployed templates or elements associated with deployed versions of content. That is, the DSS does not simply rely on the hybridization/blending of deployed templates and elements that data suggest are effective, although the system is capable of hybridization/blending.
  • The DSS system includes the functionality to distribute 640 different content pieces across a network of displays to enhance the system level (i.e., superordinate-level) goals. For example, content pieces may be distributed system-wide to coordinate the sales of different items, or to respond to different inventory levels (sales rates, profit margins) at different geographic locations.
  • The functional components 610-640 illustrated in FIG. 6 are individually useful. However, when the components 610-640 are combined into a unified system, a number of other key benefits arise out of the combined system. Application of cognitive and vision sciences 610 allows users with little or no background in the cognitive and visual sciences to apply these disciplines in order to create more effective content. This functionality can be used in either a single or multi-screen environment. On a system-wide level, application of cognitive and vision sciences provides input and constraints for the automated content design system in order to tailor content on a screen-by screen basis. For example, if the average viewing distance is known for each network sign, then the component for applying the cognitive and visual sciences will determine the ideal font size for each sign, and this information will be used by the automated content design component to generate text with those font-size parameters. The system may suggest the key parameters that should be manipulated during the experiment process, and may provide the upper and lower bounds of those parameters.
  • Content effectiveness measurement 620 can operate in either a single or multi-screen environment to generate experimental designs and analyze data regarding the impact of content on any measurable human behavior. Content effectiveness measurement 620 can determine causal relationships between signage content and human behavior. In one example, it is possible to determine the precise financial value of content (and thus of the digital signage system) for any human behavior that can itself be assigned a precise value. The human behavior having the most obvious known value is purchasing behavior. However, through the system's ability to sense other human behaviors, users could assign dollar values to a wide variety of actions, such as eye movements, picking up products, reduction in wayfinding times, etc.
  • In another example, market researchers could test their hypotheses regarding what feature sets in products are most valuable by generating content describing different feature sets of the same product. By determining what content pieces are the most effective, it would be possible to make inferences regarding what feature sets are the most valuable to consumers.
  • On a system level, content effectiveness measurement 620 provides input to the automated content design component regarding the effectiveness of design parameters, which allows the automated content design component to continuously improve the effectiveness of deployed content. Further, continuously updated input can be provided to allow the content distribution component to predict the impact that specific content distribution patterns will have on a given goal-state.
  • Content distribution functionality 640 provides for continuous changes in the relative frequency with which individual pieces of content are presented across the network in order to attain or maximize a goal of the digital signage network. Changing the relative frequency involves increasing or decreasing the number of times individual pieces of content are shown on individual signs. All other content distribution factors, such as the versions of pieces of content that are shown on specific screens remain the same. For example, the system can decrease the frequency of presenting pieces of content corresponding to products that have lower inventory levels and increase the frequency of presenting content corresponding to products with higher inventory levels.
  • System wide, the content distribution component 640 can receive input from the content effectiveness measurement component 620, and leverage that input to strategically distribute content on a screen-by-screen basis based on the predictions that arise out of the cause and effect information gathered by the content effectiveness measurement component.
  • FIG. 7 illustrates the process flow of creating and deploying content using the components and functionality of the DSS described above. During the first cycle, or initialization, the process uses data from outside the system to optimize the system goals. During subsequent cycles, the process may rely on data acquired by the system itself to modify and/or enhance the system goals. To elaborate, during the first cycle, the process illustrated in FIG. 7 uses prior knowledge from the cognitive and vision sciences to optimize goals; and subsequent cycles use cognitive and vision sciences and also results from real-time experimental data to optimize system goals. Thus, during initialization, the process uses a priori sources of data. During subsequent cycles no user-interaction is required. During the subsequent cycles, both a-priori and a-postiori data are used.
  • The process walks the user through a series of tools and scripts, and creates 710 a number of alternative templates that specify how categories of content elements might appear on the screen (e.g., the location, size, and orientation of elements such as text, graphics and videos). The tools and scripts suggest recommended templates by drawing on three sets of information: a) principles from the cognitive and visual sciences regarding effective display of information, b) the goals for the content (e.g., way-finding, advertising), and c) the known attributes of the digital signage network (e.g., size and shape of the different displays, different viewing distances, and viewer demographics across the network). For example, the tools and scripts might help a user determine whether an element should be represented graphically or via text. The tools and scripts might also help a user determine which of a large number of pre-defined templates are appropriate given the viewing conditions across the network, goals for the content, and if available, metrics regarding the types of templates that have been effective from previous campaigns.
  • The process walks the user through a series of tools and scripts to generate 720 the particular content elements that will later be placed within the templates created at block 710. The individual content elements can include specific text messages, static images, animations, movie clips, sound bites, etc. Each element could have many variants, and software helps the user determine which elements of content can be combined within a template, the rules for how those elements can be combined, and the parameters on which the content elements can be manipulated during the content creation process. For example, it may be legal to change the color of a font during deployment, but not the color of the face of a famous person used in the template.
  • The software tools and scripts facilitate content generation by drawing on three sets of information: a) data regarding the types of content elements that were effective in previous campaigns, b) principles from the cognitive and visual sciences, and c) the known attributes of the digital signage network. After the content is created, in this example, user interaction is no longer necessary.
  • Content creation is enhanced 730. The process may involve various constraints to combine elements and templates to create a number of versions of content. The first time through this process, the constraints will be based on: a) the factors previously used for used in creation of templates and content elements above, b) pre-programmed guidelines for how to combine elements and templates, c) goals for the piece of content being deployed, and d) the parameters of experimental design. On subsequent passes through this block, the process will also use effectiveness data to alter existing or create novel templates (through interpolation) and elements before creating new versions of content. Because each display in a network may have different attributes (e.g., different lighting levels, noise levels, shape, size, and mean viewing distances), a unique version of content may be created for each display in the network.
  • The content is distributed 740 across the digital signage network. Content distribution involves the determination of what, where, and when individual pieces of content are displayed in order to: a) allow cause-and-effect relationships between content and viewer behavior to be determined, b) enhance the system-level goals of the active signage network, and thus the overall network return on investment, c) allow accurate measurement of the effectiveness of specific templates and content elements.
  • The content distribution process allows versions of content to be distributed using appropriate blocking and counterbalancing procedures. Further, appropriate baseline control conditions are used for known attributes of the signs across the network, and versions of content are properly randomized for unknown factors. These algorithms determine the appropriate experimental design given the signage network attributes, e.g., the number of attributes, and the relations among the attributes. This functionality coordinates the playback requirements, such as the frequency and timing of playback and location of playback of individual pieces of content across the system.
  • Using sensor data, point of sale, inventory data, and/or other data in conjunction with the experimental procedures used to distribute content, the impact of content is calculated and analyzed 750. To describe this step in terms of perceptual experiments, the effect-size of the content elements and templates is calculated. Effect size refers to the amount of variability in the data that any defined variable can explain. The process analyzes and predicts what content would be effective for a given attribute across the network. Also, co-occurrences of sensor data, content presentation, and movement towards the goal are detected. Therefore, it is possible to learn that some detected event, when paired with content, increases the direction towards the goal. These co-occurrences then become new digital signage network attributes. Content may be distributed in order to take advantage of co-occurrences of sensor data, content presentation, and movement towards the goal state.
  • The analysis performed at block 750 forms the basis of reporting return on investment (ROI), future content creation, and future content deployment. Inferential statistics may be conducted on the pre-identified dependent variables. From these inferential statistics, the system can calculate effect sizes and confidence in cause and effect relationships, including the effects of content elements, templates, and deployment.
  • FIG. 8 is a flowchart illustrating an exemplary implementation of the DSS system in accordance with an embodiment of the invention. The implementation involves a sporting goods retailer with 200 stores. The retailer desires to advertise four overstocked products and four products that are not overstocked but that have higher profit margins than the overstocked products. The super ordinate-level goal of the campaign is to maximize gross profit while eliminating excessive inventory of the overstocked items. That is, once the excessive inventory is eliminated, the goal will simply reduce to maintaining a balanced inventory at each store location.
  • Using cognitive and visual-science driven software, the signage manager of the retailer creates 810 a number of different templates that will be used to develop content for each of the eight product lines. These templates include layout of messages, color schemes, and/or other variables that make up the program. These templates can be used for each of the eight product lines, and are not specific to a single product. Additionally, pre-existing or stock templates are available for use during this phase.
  • After creating the base templates for this campaign, the signage manager creates 820 individual content elements that are needed to populate the templates. The individual elements are specific to the product lines being promoted, and include product branding and messages for given products. As in the template creation process, creation of individual elements is guided by software wizards using cognitive and visual-science driven software.
  • The templates are automatically populated 830 with the individual content elements to generate a number of different content packages for each of the eight products that the signage network is promoting. Potentially hundreds of differing versions of each content piece are created for each product line by merging elements with templates to accommodate varying signage attributes such as screen size or viewing distance.
  • Using pre-existing or learned knowledge about the signage network, content is distributed 840 by using algorithms that enable collection of success metrics for individual pieces of content. The content is distributed across the network in a way that ensures proper counterbalancing, blocking, and confound-free measurement can be made. Additionally, the deployment algorithm ensures that relevant content is sent to the appropriate signs in the network, considering network attributes, viewer demographics, and viewing conditions among others.
  • Point of sale and sensor data allow the impact of the various content packages to be monitored 850 and analyzed to determine what templates and content elements, and their combinations, are most effective for each screen on the network. From this information, cause and effect, as well as return on investment can be analyzed, enabling value-based billing. This example may determine whether across all 200 stores, the signage system itself was responsible for X% increase in profits and Y% decrease in excessive inventory. Exploratory data analysis generates new possible network attributes. For example, there is a spike in sales when customers pick up product X and when content Y is concurrently shown. On the next iteration, this new network attribute will be tested experimentally, not just measured from a correlation study. For example, the system may determine whether content pieces presented on X type screens is most effective using Y-type templates, and that the most effective content elements have XYZ properties.
  • Based on the effectiveness data, the system automatically generates 860 new templates, new content elements, and new combinations thereof. Again, using signage network attributes (both old and new), the software deploys these new pieces of content across the network.
  • During the remainder of the campaign, the processes described in blocks 830 through 860 are repeated, for example, without user interaction. The signage network manager is able to monitor the impact that the content has on sales at any given point during the campaign while the system automatically attempts to achieve the campaign goals.
  • Upon completion ofthis campaign, templates and elements that were manually or automatically generated during the campaign are available for future campaigns as well. Furthermore, the knowledge that was gained regarding the types of templates and elements that are effective for particular displays, demographics, or other factors, is used create and distribute content more effectively across the network during future campaigns.
  • Determination of whether an experiment is a true experiment can be performed proactively or retroactively with respect to running the experiment. According to some embodiments, a computer may be used to determine if an experiment that is yet to be performed is a true experiment. According to other embodiments, a computer may be used to determine if an experiment that was previously performed is a true experiment. According to the approach illustrated in FIG. 9, the computer determines, based on information provided by the user, whether an experimental design eliminates or controls confounds. In this example, the user enters 910 information about the experiment, including the independent and dependent variables of the experiment. The computer identifies 920 situations that may produce confounds in the experiment. The user selects 930 the confound-producing situations identified by the computer that are present in the context of the experiment. The computer prompts 940 the user to identify steps taken to eliminate or control the identified confounds. The computer determines 950 if the combination of steps is sufficient to eliminate confounds in the experiment.
  • The foregoing description of the various embodiments of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. For example, embodiments of the present invention may be implemented in a wide variety of applications. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.

Claims (34)

1. A method comprising designing an experiment using a computer to determine whether the experiment is a true experiment.
2. The method of claim 1, wherein designing the experiment comprises designing the experiment using the computer to select one or more independent variables of the experiment.
3. The method of claim 1, wherein designing the experiment comprises designing the experiment using the computer to assist a user in selection of one or more independent variables of the experiment.
4. The method of claim 1, wherein designing the experiment comprises designing the experiment using the computer to select one or more dependent variables of the experiment.
5. The method of claim 1, wherein designing the experiment comprises designing the experiment using the computer to assist a user in selection of one or more dependent variables of the experiment.
6. The method of claim 1, wherein designing the experiment comprises designing the experiment using the computer to form at least one of a control group and a treatment group of the experiment.
7. The method of claim 1, wherein designing the experiment comprises designing the experiment using the computer to randomize at least one of a control group and a treatment group of the experiment.
8. The method of claim 1, wherein designing the experiment comprises designing the experiment using the computer to assist a user in forming at least one of a control group and a treatment group of the experiment.
9. The method of claim 1, wherein designing the experiment comprises designing the experiment using the computer to assist a user in randomizing at least one of a control group and a treatment group of the experiment.
10. The method of claim 1, wherein designing the experiment comprises designing the experiment using the computer to apply blocking to reduce effects of one or more confounding variables of the experiment.
11. The method of claim 1, wherein designing the experiment comprises designing the experiment using the computer to assist a user in applying blocking to reduce effects of one or more confounding variables of the experiment.
12. The method of claim 1, further comprising performing the experiment.
13. The method of claim 12, wherein running the experiment comprises using the computer to automatically run the experiment.
14. The method of claim 12, wherein running the experiment comprises using the computer to assist a user in running the experiment.
15. The method of claim 1, further comprising analyzing results of the true experiment.
16. The method of claim 15, wherein analyzing the results of the experiment comprises automatically analyzing the results via the computer.
17. The method of claim 15, wherein analyzing the results of the experiment comprises using the computer to assist a user in analyzing the results of the experiment.
18. A system for designing true experiments comprising a design processor configured to determine whether an experiment is a true experiment.
19. The system of claim 18, wherein the design processor is configured to select at least one of an independent variable and a dependent variable of the experiment.
20. The system of claim 18, wherein the design processor is configured to form at least one of a control group and a treatment group of the experiment.
21. The system of claim 18, wherein the design processor is configured to apply blocking to reduce effects of one or more confounding variables of the experiment.
22. The system of claim 18, further comprising a user interface configured to accept input from a user, wherein the design processor is configured to use the input to design the experiment.
23. The system of claim 22, wherein the design processor is configured to assist the user in selection of at least one of an independent variable and a dependent variable of the experiment using the user input.
24. The system of claim 22, wherein the design processor is configured to assist the user in selection of at least one of a control group and a treatment group of the experiment based on the user input.
25. The system of claim 22, wherein the design processor is configured to assist the user in applying blocking to reduce effects of one or more confounding variables of the experiment based on the user input.
26. The system of claim 18, further comprising a deployment unit configured to run the experiment.
27. The system of claim 18, further comprising an analysis unit configured to analyze results of the experiment.
28. The system of claim 27, wherein the analysis unit is configured to analyze results of the experiment without input from a user.
29. The system of claim 27, wherein the analysis unit is configured to analyze results of the experiment using input from a user.
30. The system of claim 18, wherein the experiment comprises a digital signage experiment.
31. The system of claim 18, wherein the experiment involves an advertisement.
32. A system, comprising:
means for designing an experiment using a computer to determine whether the experiment is a true experiment; and
means for selecting one or more of a dependent and an independent variable of the experiment.
33. The system of claim 32, further comprising means for forming one or more of a control group and a treatment group of the experiment.
34. The system of claim 32, further comprising means for applying blocking to reduce effects of one or more confounding variables of the experiment.
US11/321,340 2005-12-29 2005-12-29 Systems and methods for designing experiments Abandoned US20070156382A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/321,340 US20070156382A1 (en) 2005-12-29 2005-12-29 Systems and methods for designing experiments

Applications Claiming Priority (37)

Application Number Priority Date Filing Date Title
US11/321,340 US20070156382A1 (en) 2005-12-29 2005-12-29 Systems and methods for designing experiments
CA002634877A CA2634877A1 (en) 2005-12-29 2006-12-14 Systems and methods for designing experiments
CN2006800531432A CN101379496B (en) 2005-12-29 2006-12-14 Estimation method for designing experiments
JP2008548590A JP2009522651A (en) 2005-12-29 2006-12-14 System and method for planning the experiment
RU2008126454/08A RU2409859C2 (en) 2005-12-29 2006-12-14 Systems and methods for designing experiments
EP06845659A EP1969504A4 (en) 2005-12-29 2006-12-14 Systems and methods for designing experiments
BRPI0621157-7A BRPI0621157A2 (en) 2005-12-29 2006-12-14 method and system for designing experiments
PCT/US2006/048110 WO2007078897A2 (en) 2005-12-29 2006-12-14 Systems and methods for designing experiments
AU2006333040A AU2006333040C1 (en) 2005-12-29 2006-12-14 Systems and methods for designing experiments
KR1020087017406A KR101397745B1 (en) 2005-12-29 2006-12-14 Systems and methods for designing experiments
CA002636017A CA2636017A1 (en) 2005-12-29 2006-12-29 Expert system for designing experiments
JP2008548783A JP2009522658A (en) 2005-12-29 2006-12-29 The development and distribution of content using cognitive science database
KR1020147002645A KR20140033218A (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
RU2008126540/09A RU2008126540A (en) 2005-12-29 2006-12-29 Development and distribution of content using a database of scientific thinking
CA002635789A CA2635789A1 (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
CN2006800530730A CN101379485B (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
BRPI0621159-3A BRPI0621159A2 (en) 2005-12-29 2006-12-29 computer implemented method and system
PCT/US2006/049657 WO2007079254A2 (en) 2005-12-29 2006-12-29 Expert system for designing experiments
KR1020087017761A KR101336500B1 (en) 2005-12-29 2006-12-29 Expert system for designing experiments
RU2008126539/09A RU2008126539A (en) 2005-12-29 2006-12-29 An expert system for the design of experiments
BRPI0621158-5A BRPI0621158A2 (en) 2005-12-29 2006-12-29 computer implemented method, system and media which can be read by computer
EP13182401.3A EP2722797A1 (en) 2005-12-29 2006-12-29 Expert system for designing experiments
AU2006332660A AU2006332660B2 (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
AU2006332658A AU2006332658B2 (en) 2005-12-29 2006-12-29 Expert system for designing experiments
EP20060848390 EP1969488A4 (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
CNA2006800530637A CN101379495A (en) 2005-12-29 2006-12-29 Expert system for designing experiments
US12/159,107 US8594990B2 (en) 2005-12-29 2006-12-29 Expert system for designing experiments
PCT/US2006/049662 WO2007079256A2 (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
US12/159,106 US20090158179A1 (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
EP06848389A EP1969505A4 (en) 2005-12-29 2006-12-29 Expert system for designing experiments
JP2008548780A JP5249045B2 (en) 2005-12-29 2006-12-29 Expert system for planning the experiment
KR1020087018278A KR20080081193A (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
US12/565,312 US20100017288A1 (en) 2005-12-29 2009-09-23 Systems and methods for designing experiments
JP2012264984A JP5563649B2 (en) 2005-12-29 2012-12-04 System and method for planning the experiment
JP2013255621A JP2014059911A (en) 2005-12-29 2013-12-11 Content development and distribution using cognitive science database
US14/254,014 US20140229418A1 (en) 2005-12-29 2014-04-16 Systems and methods for designing experiments
US14/261,929 US10007657B2 (en) 2005-12-29 2014-04-25 Content development and distribution using cognitive sciences database

Related Child Applications (5)

Application Number Title Priority Date Filing Date
US12159106 Continuation-In-Part 2003-12-29
PCT/US2006/049662 Continuation-In-Part WO2007079256A2 (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
US12/159,107 Continuation-In-Part US8594990B2 (en) 2005-12-29 2006-12-29 Expert system for designing experiments
US15910608A Continuation-In-Part 2008-10-17 2008-10-17
US12/565,312 Continuation US20100017288A1 (en) 2005-12-29 2009-09-23 Systems and methods for designing experiments

Publications (1)

Publication Number Publication Date
US20070156382A1 true US20070156382A1 (en) 2007-07-05

Family

ID=38225630

Family Applications (6)

Application Number Title Priority Date Filing Date
US11/321,340 Abandoned US20070156382A1 (en) 2005-12-29 2005-12-29 Systems and methods for designing experiments
US12/159,106 Abandoned US20090158179A1 (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
US12/159,107 Active 2028-06-12 US8594990B2 (en) 2005-12-29 2006-12-29 Expert system for designing experiments
US12/565,312 Abandoned US20100017288A1 (en) 2005-12-29 2009-09-23 Systems and methods for designing experiments
US14/254,014 Abandoned US20140229418A1 (en) 2005-12-29 2014-04-16 Systems and methods for designing experiments
US14/261,929 Active 2024-08-01 US10007657B2 (en) 2005-12-29 2014-04-25 Content development and distribution using cognitive sciences database

Family Applications After (5)

Application Number Title Priority Date Filing Date
US12/159,106 Abandoned US20090158179A1 (en) 2005-12-29 2006-12-29 Content development and distribution using cognitive sciences database
US12/159,107 Active 2028-06-12 US8594990B2 (en) 2005-12-29 2006-12-29 Expert system for designing experiments
US12/565,312 Abandoned US20100017288A1 (en) 2005-12-29 2009-09-23 Systems and methods for designing experiments
US14/254,014 Abandoned US20140229418A1 (en) 2005-12-29 2014-04-16 Systems and methods for designing experiments
US14/261,929 Active 2024-08-01 US10007657B2 (en) 2005-12-29 2014-04-25 Content development and distribution using cognitive sciences database

Country Status (10)

Country Link
US (6) US20070156382A1 (en)
EP (4) EP1969504A4 (en)
JP (5) JP2009522651A (en)
KR (4) KR101397745B1 (en)
CN (3) CN101379496B (en)
AU (3) AU2006333040C1 (en)
BR (3) BRPI0621157A2 (en)
CA (3) CA2634877A1 (en)
RU (3) RU2409859C2 (en)
WO (3) WO2007078897A2 (en)

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080306848A1 (en) * 2007-06-05 2008-12-11 Bank Of America Corporation Lead Generation Platform
US20090012847A1 (en) * 2007-07-03 2009-01-08 3M Innovative Properties Company System and method for assessing effectiveness of communication content
US20090012927A1 (en) * 2007-07-03 2009-01-08 3M Innovative Properties Company System and method for assigning pieces of content to time-slots samples for measuring effects of the assigned content
WO2009036002A1 (en) * 2007-09-13 2009-03-19 Motorola, Inc. Method and system for presenting advertisements
US20090158179A1 (en) * 2005-12-29 2009-06-18 Brooks Brian E Content development and distribution using cognitive sciences database
US20090248491A1 (en) * 2008-03-27 2009-10-01 Mark Hemphill System and method for ranking participating venues in a network according to advertisers needs for targeted advertising
US20090299912A1 (en) * 2008-05-30 2009-12-03 Strategyn, Inc. Commercial investment analysis
US20100082691A1 (en) * 2008-09-19 2010-04-01 Strategyn, Inc. Universal customer based information and ontology platform for business information and innovation management
US20100174671A1 (en) * 2009-01-07 2010-07-08 Brooks Brian E System and method for concurrently conducting cause-and-effect experiments on content effectiveness and adjusting content distribution to optimize business objectives
US20110145230A1 (en) * 2009-05-18 2011-06-16 Strategyn, Inc. Needs-based mapping and processing engine
US20110191282A1 (en) * 2010-01-29 2011-08-04 Google Inc. Evaluating Statistical Significance Of Test Statistics Using Placebo Actions
US20110218837A1 (en) * 2010-03-03 2011-09-08 Strategyn, Inc. Facilitating growth investment decisions
US20140108931A1 (en) * 2008-07-03 2014-04-17 Ebay Inc. System and methods for the segmentation of media
US20140379622A1 (en) * 2012-01-16 2014-12-25 3M Innovative Properties Company Systems and methods for rule-based content optimization
US20150081596A1 (en) * 2013-09-16 2015-03-19 Moresteam Development Llc Computer-implemented system and method of performing a design of experiment, generating a product thereof, generating designed experiment options, and a graphical user interface
WO2015116610A1 (en) * 2014-01-29 2015-08-06 3M Innovative Properties Company Conducting multivariate experiments
CN104851063A (en) * 2015-05-14 2015-08-19 上海知汇云信息技术股份有限公司 Experiment evaluation system and process management method thereof
US20160353166A1 (en) * 2014-01-29 2016-12-01 3M Innovative Properties Company Systems and methods for spacial and temporal experimentation on content effectiveness
US9947018B2 (en) 2007-07-03 2018-04-17 3M Innovative Properties Company System and method for generating time-slot samples to which content may be assigned for measuring effects of the assigned content
US10049645B2 (en) 2013-12-20 2018-08-14 Koninklijke Philips N.V. Control device and method for optimizing a presentation style of specific content for a specific user
US10282391B2 (en) 2008-07-03 2019-05-07 Ebay Inc. Position editing tool of collage multi-media

Families Citing this family (40)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100131333A1 (en) * 2007-08-06 2010-05-27 Keith Ball Method and system for product services analysis and optimization
JP4934861B2 (en) * 2008-01-28 2012-05-23 Necソフト株式会社 Display system, a display method, display effect measurement system, and a display effect measurement method.
US8775113B2 (en) * 2008-01-29 2014-07-08 Apple Inc. Automated portable media device testing system
US8478111B2 (en) 2008-10-03 2013-07-02 3M Innovative Properties Company Systems and methods for optimizing a scene
CN102227753B (en) 2008-10-03 2018-01-19 3M创新有限公司 For robustness assessment system and method
AU2009298428B2 (en) 2008-10-03 2013-03-14 3M Innovative Properties Company Systems and methods for optimizing a scene
KR20100040236A (en) * 2008-10-09 2010-04-19 삼성전자주식회사 Two dimensional image to three dimensional image converter and conversion method using visual attention analysis
US8808195B2 (en) * 2009-01-15 2014-08-19 Po-He Tseng Eye-tracking method and system for screening human diseases
JP5399834B2 (en) * 2009-09-15 2014-01-29 正生 中村 Significance evaluation program and a recording medium
US8554619B2 (en) * 2009-12-31 2013-10-08 Google Inc. Multi-campaign content allocation
US9174775B2 (en) 2010-06-03 2015-11-03 Elix, Llc Actuator-driven dispenser
CN102271262B (en) * 2010-06-04 2015-05-13 三星电子株式会社 Multithread-based video processing method for 3D (Three-Dimensional) display
JP2012014597A (en) * 2010-07-02 2012-01-19 Nippon Telegr & Teleph Corp <Ntt> Attentional area display system, attentional area display method and attentional area display program
US9323418B2 (en) * 2011-04-29 2016-04-26 The United States Of America As Represented By Secretary Of The Navy Method for analyzing GUI design affordances
US8195799B1 (en) 2011-10-26 2012-06-05 SHTC Holdings LLC Smart test article optimizer
US8612375B2 (en) 2011-11-11 2013-12-17 Wetpaint.Com, Inc. Dynamic playbook: experimentation platform for social networks
US10354213B1 (en) * 2011-12-27 2019-07-16 Applied Predictive Technologies, Inc. Natural experiment finder system and method
US9208661B2 (en) * 2012-01-06 2015-12-08 Panasonic Corporation Of North America Context dependent application/event activation for people with various cognitive ability levels
AU2013209987A1 (en) 2012-01-16 2014-07-24 3M Innovative Properties Company Systems and methods for rule-based content generation
US9197512B2 (en) * 2012-02-17 2015-11-24 Wetpaint.Com, Inc. Optimizing content production based upon distribution channel metrics
US8769400B1 (en) * 2012-03-26 2014-07-01 Google Inc. Accelerating view transitions
US8990364B2 (en) * 2012-04-26 2015-03-24 Qualcomm Innovation Center, Inc. Architecture for content awareness and control among peer-to-peer communication devices
CN103488646B (en) * 2012-06-13 2017-11-14 腾讯科技(深圳)有限公司 For browsing the web method and apparatus of the mobile terminal browser low light
JP2014052399A (en) * 2012-09-05 2014-03-20 Toshiba Tec Corp Information distribution server and program
US9265458B2 (en) 2012-12-04 2016-02-23 Sync-Think, Inc. Application of smooth pursuit cognitive testing paradigms to clinical drug development
WO2014099380A1 (en) * 2012-12-21 2014-06-26 3M Innovative Properties Company Systems and methods for rule-based animated content generation
WO2014099571A1 (en) * 2012-12-21 2014-06-26 3M Innovative Properties Company Systems and methods for rule-based animated content optimization
WO2014123337A1 (en) * 2013-02-06 2014-08-14 Samsung Electronics Co., Ltd. System and method for providing object via which service is used
US9380976B2 (en) 2013-03-11 2016-07-05 Sync-Think, Inc. Optical neuroinformatics
CN103745056A (en) * 2014-01-06 2014-04-23 中国人民解放军第二军医大学 Modular calculation method of medical service forces
CN104882145A (en) * 2014-02-28 2015-09-02 杜比实验室特许公司 Audio object clustering by utilizing temporal variations of audio objects
US20150348278A1 (en) * 2014-05-30 2015-12-03 Apple Inc. Dynamic font engine
JP6459231B2 (en) * 2014-06-11 2019-01-30 富士ゼロックス株式会社 Template management device and program
ZA201407620B (en) * 2014-08-26 2015-11-25 Musigma Business Solutions Pvt Ltd Systems and methods for creating and evaluating experiments
CN104301556B (en) * 2014-09-26 2017-12-29 小米科技有限责任公司 Telephone interaction method, apparatus and system for
US10354192B2 (en) * 2014-11-19 2019-07-16 The Government Of The United States Of America, As Represented By The Secretary Of The Navy Recommender system for exploratory data analysis
US20160357173A1 (en) * 2015-06-08 2016-12-08 Evidation Health Evidence Generation and Data Interpretation Platform
US9990414B2 (en) 2015-06-15 2018-06-05 International Business Machines Corporation Cognitive architecture with content provider managed corpus
JP2018531367A (en) * 2015-06-30 2018-10-25 エメラルド クラウド ラボ、インコーポレイテッド Experimental data exploration and visualization of laboratory
CN105446744A (en) * 2015-12-15 2016-03-30 北京中网易企秀科技有限公司 Internet based display material system and display material making method

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030083822A2 (en) * 2001-05-15 2003-05-01 Psychogenics, Inc. Systems and methods for monitoring behavior informatics
US20040088726A1 (en) * 2002-11-01 2004-05-06 Yu-Fei Ma Systems and methods for generating a comprehensive user attention model
US20040210471A1 (en) * 2003-04-17 2004-10-21 Targetrx,Inc. Method and system for analyzing the effectiveness of marketing strategies
US6840905B2 (en) * 2001-06-06 2005-01-11 International Business Machines Corporation System and method of automating multidimensional scaling for psychophysics
US20050039206A1 (en) * 2003-08-06 2005-02-17 Opdycke Thomas C. System and method for delivering and optimizing media programming in public spaces
US20050159921A1 (en) * 1999-08-26 2005-07-21 Louviere Jordan J. On-line experimentation

Family Cites Families (118)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4973952A (en) * 1987-09-21 1990-11-27 Information Resources, Inc. Shopping cart display system
US5309174A (en) * 1987-10-13 1994-05-03 Motorola, Inc. Electronic display system
US5253331A (en) * 1991-07-03 1993-10-12 General Motors Corporation Expert system for statistical design of experiments
US5474385A (en) * 1992-03-02 1995-12-12 Ta Instruments, Inc. Method and apparatus for parsed dynamic differential analysis
JPH06131168A (en) * 1992-10-20 1994-05-13 Fuji Xerox Co Ltd User interface preparation supporting device
JP3234059B2 (en) * 1993-08-11 2001-12-04 株式会社東芝 Screen design evaluation device
KR950016161A (en) 1993-11-17 1995-06-17 김광호 Missed message display method of the PBX system
US5513342A (en) * 1993-12-28 1996-04-30 International Business Machines Corporation Display window layout system that automatically accommodates changes in display resolution, font size and national language
JPH07200662A (en) * 1994-01-07 1995-08-04 Hitachi Ltd Experiment plan production support system having design knowledge base
JP2947704B2 (en) * 1994-03-04 1999-09-13 株式会社ピーエフユー How to create and display screen creating device for a display screen
US7848948B2 (en) * 1996-10-25 2010-12-07 Ipf, Inc. Internet-based product brand marketing communication network configured to allow members of a product brand management team to communicate directly with consumers browsing HTML-encoded pages at an electronic commerce (EC) enabled web-site along the fabric of the world wide web (WWW), using programable multi-mode virtual kiosks (MMVKS) driven by server-side components and managed by product brand management team members
NZ315428A (en) * 1995-07-25 2000-02-28 Horus Therapeutics Inc Computer assisted methods for diagnosing diseases
JP2697646B2 (en) * 1994-12-15 1998-01-14 日本電気株式会社 Multimedia editing apparatus
JPH08241191A (en) * 1995-03-02 1996-09-17 Matsushita Electric Ind Co Ltd Gui automatic evaluation device
JP2994996B2 (en) * 1995-12-01 1999-12-27 松下電器産業株式会社 Multimedia editing apparatus
JPH1049508A (en) * 1995-12-27 1998-02-20 Toshiba Corp Data processing system, system construction device and method, and medium for recording system construction program
JPH09212349A (en) 1996-01-31 1997-08-15 Mitsubishi Electric Corp Contents generation support system
US5812134A (en) * 1996-03-28 1998-09-22 Critical Thought, Inc. User interface navigational system & method for interactive representation of information contained within a database
US6581048B1 (en) * 1996-06-04 2003-06-17 Paul J. Werbos 3-brain architecture for an intelligent decision and control system
JP4615629B2 (en) * 1996-07-12 2011-01-19 クリニカル ディシジョン サポート,エルエルシー Including access to the network, medical diagnosis and treatment advice system using computer
JPH1091412A (en) * 1996-07-25 1998-04-10 Toshiba Corp Device for selecting display parts and its method
JP2998652B2 (en) * 1996-09-20 2000-01-11 日本電気株式会社 Gui screen layout design system
JPH10154070A (en) * 1996-11-26 1998-06-09 Toshiba Corp User interface design device and method
US6023714A (en) * 1997-04-24 2000-02-08 Microsoft Corporation Method and system for dynamically adapting the layout of a document to an output device
JPH10301761A (en) * 1997-04-28 1998-11-13 Matsushita Electric Ind Co Ltd Preparation method for user interface and management device therefor
JPH1185439A (en) * 1997-09-05 1999-03-30 Toshiba Corp User interface control device and method therefor
JPH1185729A (en) * 1997-09-08 1999-03-30 Matsushita Electric Ind Co Ltd Multimedia contents preparing device
US6374271B1 (en) * 1997-09-26 2002-04-16 Fuji Xerox Co., Ltd. Hypermedia document authoring using a goals outline and a presentation outline
US6529940B1 (en) * 1998-05-28 2003-03-04 David R. Humble Method and system for in-store marketing
RU2134457C1 (en) 1998-08-04 1999-08-10 Григорьев Михаил Николаевич Information spreading system
JP2000090079A (en) * 1998-09-08 2000-03-31 Toshiba Corp Device and method for preparing contents and computer readable recording medium recording program
US6448983B1 (en) * 1999-01-14 2002-09-10 General Electric Company Method for selection of a design of experiment
US6604092B1 (en) * 1999-02-26 2003-08-05 Lisa E. Stewart Expert system utilizing a knowledge base and design of experiment (DOE) techniques
BR0005552A (en) * 1999-03-26 2001-01-30 Koninkl Philips Electronics Nv Process to develop a population of preferred ads, computer program to develop a preferred population of ads, and ad evolutionary system
AUPQ246899A0 (en) * 1999-08-26 1999-09-16 Memetrics An automated communications management agent
US7089322B1 (en) * 1999-10-28 2006-08-08 Motient Communications Inc. System and method of aggregating data from a plurality of data generating machines
US6317700B1 (en) * 1999-12-22 2001-11-13 Curtis A. Bagne Computational method and system to perform empirical induction
US7130808B1 (en) * 1999-12-29 2006-10-31 The Product Engine, Inc. Method, algorithm, and computer program for optimizing the performance of messages including advertisements in an interactive measurable medium
CA2400085C (en) * 2000-02-17 2008-02-19 British Telecommunications Public Limited Company Visual attention system
JP2001236443A (en) * 2000-02-24 2001-08-31 Nec Corp Method for providing advertisement information and method and system for grasping its advertisement effect
US8799208B2 (en) * 2000-03-07 2014-08-05 E-Rewards, Inc. Method and system for evaluating, reporting, and improving on-line promotion effectiveness
JP2001256322A (en) * 2000-03-10 2001-09-21 Cross Head Kk System and method for on-line contents production, display, and value payment, and information storage medium stroed with tool for contents production
AU5385701A (en) * 2000-03-14 2001-09-24 Buzzpad Inc Method and apparatus for forming linked multi-user groups of shared software applications
JP2001265904A (en) * 2000-03-17 2001-09-28 Dentsu Inc System and method for examining advertisement representation
JP4066613B2 (en) 2000-04-05 2008-03-26 富士ゼロックス株式会社 Display control device, an image forming apparatus, and a display control program
US7216092B1 (en) * 2000-04-14 2007-05-08 Deluxe Corporation Intelligent personalization system and method
US6604094B1 (en) * 2000-05-25 2003-08-05 Symbionautics Corporation Simulating human intelligence in computers using natural language dialog
AU6540501A (en) 2000-05-26 2001-12-11 Naylor Inc Computer-implemented method and apparatus for creating advertising media
JP2002123288A (en) * 2000-10-12 2002-04-26 Canon Inc Portable telephone, user interface device, contents display method, and storage medium
US20040015399A1 (en) * 2000-10-12 2004-01-22 Maggio Frank S. Method and system for verifying exposure to message content delivered via outdoor media or in a concentrated format
AU1351102A (en) * 2000-10-30 2002-05-15 Elias Arts Corp System and method for performing content experience management
JP2002175290A (en) * 2000-12-07 2002-06-21 Fuji Xerox Co Ltd Document design evaluation system, document design evaluation method and recording medium
US7406434B1 (en) * 2000-12-15 2008-07-29 Carl Meyer System and method for improving the performance of electronic media advertising campaigns through multi-attribute analysis and optimization
JP4665312B2 (en) * 2000-12-26 2011-04-06 日本電気株式会社 Readability rating system of Web content, Web content coloring support system, its method, recording medium
US20040015401A1 (en) * 2001-02-08 2004-01-22 In Lee Systems and methods for distributing template-based multimedia presentations over a network
JP3881179B2 (en) * 2001-02-14 2007-02-14 三菱電機株式会社 User interface design apparatus
US20020129367A1 (en) * 2001-03-02 2002-09-12 Koninklijke Philips Electronics N.V. Method and apparatus for personalized presentation of television/internet contents
US20020154833A1 (en) * 2001-03-08 2002-10-24 Christof Koch Computation of intrinsic perceptual saliency in visual environments, and applications
JP2002330415A (en) * 2001-05-02 2002-11-15 Star Collaboration Kk Contents producing device, method, computer program, recording medium
US7219309B2 (en) 2001-05-02 2007-05-15 Bitstream Inc. Innovations for the display of web pages
JP2003044737A (en) * 2001-07-26 2003-02-14 E Stage Inc Wide area recognition system for company information
US20040205116A1 (en) * 2001-08-09 2004-10-14 Greg Pulier Computer-based multimedia creation, management, and deployment platform
TW523697B (en) * 2001-08-29 2003-03-11 Synq Technology Inc Automatic advertisement transaction system and method therefor
KR100868852B1 (en) 2001-09-29 2008-11-14 주식회사 케이티 Visiting hours measurement method in web log analysis system
US7136871B2 (en) * 2001-11-21 2006-11-14 Microsoft Corporation Methods and systems for selectively displaying advertisements
US20030220830A1 (en) * 2002-04-04 2003-11-27 David Myr Method and system for maximizing sales profits by automatic display promotion optimization
US7283981B2 (en) * 2002-04-30 2007-10-16 Deluxe Corporation Browser-based system for ordering financial products
US20040181435A9 (en) * 2002-06-14 2004-09-16 Reinsurance Group Of America Corporation Computerized system and method of performing insurability analysis
JP2004070677A (en) * 2002-08-07 2004-03-04 Inter-Ment Kk Web site evaluation providing system and method, and program for providing web site evaluation
JP4183456B2 (en) * 2002-08-09 2008-11-19 アイシン精機株式会社 Vehicle testing provides system
JP4356916B2 (en) * 2002-08-23 2009-11-04 富士通株式会社 Usability education support system and methods
US7309315B2 (en) * 2002-09-06 2007-12-18 Epoch Innovations, Ltd. Apparatus, method and computer program product to facilitate ordinary visual perception via an early perceptual-motor extraction of relational information from a light stimuli array to trigger an overall visual-sensory motor integration in a subject
JP2004102868A (en) * 2002-09-12 2004-04-02 Fuji Xerox Co Ltd Usability evaluation support apparatus
JP2004110548A (en) * 2002-09-19 2004-04-08 Fuji Xerox Co Ltd Usability evaluation support apparatus and method
JP2004178276A (en) * 2002-11-27 2004-06-24 Toshiba Lighting & Technology Corp Sales floor plan supporting system
JP2004184845A (en) * 2002-12-05 2004-07-02 Sanyo Electric Co Ltd Apparatus and system for video display
US7130461B2 (en) * 2002-12-18 2006-10-31 Xerox Corporation Systems and method for automatically choosing visual characteristics to highlight a target against a background
US20050062314A1 (en) * 2003-09-19 2005-03-24 Searfoss Timothy K. Trailer cover system
JP2004234646A (en) * 2003-01-06 2004-08-19 Hiroshi Sato Content relevant information providing device, content relevant information providing method, content relevant information providing system, portable terminal and information processing system
US7260261B2 (en) * 2003-02-20 2007-08-21 Microsoft Corporation Systems and methods for enhanced image adaptation
JP2004310384A (en) * 2003-04-04 2004-11-04 Haruyuki Inada Web page advertisement system
KR20030040263A (en) 2003-04-07 2003-05-22 주식회사 드림나우 The Collecting Method and System of an Internet Homepage User's Activity
JP3875974B2 (en) * 2003-04-11 2007-01-31 西川ゴム工業株式会社 Sound insulation sheet
US7694000B2 (en) * 2003-04-22 2010-04-06 International Business Machines Corporation Context sensitive portlets
US20050047647A1 (en) * 2003-06-10 2005-03-03 Ueli Rutishauser System and method for attentional selection
US20040260767A1 (en) * 2003-06-19 2004-12-23 David Kedem Dynamic web advertisement and content display system
JP4314907B2 (en) * 2003-07-18 2009-08-19 株式会社日立製作所 Advertising display system and method
GB2404270A (en) 2003-07-24 2005-01-26 Hewlett Packard Development Co Document composition
JP2005056080A (en) * 2003-08-01 2005-03-03 Inter-Ment Kk Web site analyzing system and method, and program for analyzing web site
US7269517B2 (en) * 2003-09-05 2007-09-11 Rosetta Inpharmatics Llc Computer systems and methods for analyzing experiment design
US20050096950A1 (en) * 2003-10-29 2005-05-05 Caplan Scott M. Method and apparatus for creating and evaluating strategies
JP4396430B2 (en) * 2003-11-25 2010-01-13 セイコーエプソン株式会社 Visual guidance information generating system and visual guidance information generating program, as well as visual guidance information generating process
US20050171843A1 (en) * 2004-02-03 2005-08-04 Robert Brazell Systems and methods for optimizing advertising
US20050195221A1 (en) * 2004-03-04 2005-09-08 Adam Berger System and method for facilitating the presentation of content via device displays
CA2567345A1 (en) * 2004-05-18 2005-12-08 Advertising.Com Systems and methods of achieving optimal advertising
US7949561B2 (en) * 2004-08-20 2011-05-24 Marketing Evolution Method for determining advertising effectiveness
GB2417808A (en) * 2004-09-02 2006-03-08 Hewlett Packard Development Co Document creation system
WO2006024108A1 (en) 2004-09-02 2006-03-09 University Of Technology, Sydney Method, system and computer program product for measuring and tracking brand equity
WO2006031888A2 (en) * 2004-09-13 2006-03-23 Brigham Young University Methods and systems for conducting internet marketing experiments
GB2422739B (en) * 2005-01-31 2010-07-14 Hewlett Packard Development Co Image processing method and apparatus
WO2006086525A1 (en) * 2005-02-08 2006-08-17 Pasadero, Inc. Research protocol toolkit
US20060200759A1 (en) * 2005-03-04 2006-09-07 Microsoft Corporation Techniques for generating the layout of visual content
US7594189B1 (en) * 2005-04-21 2009-09-22 Amazon Technologies, Inc. Systems and methods for statistically selecting content items to be used in a dynamically-generated display
US20070011050A1 (en) * 2005-05-20 2007-01-11 Steven Klopf Digital advertising system
US7903099B2 (en) * 2005-06-20 2011-03-08 Google Inc. Allocating advertising space in a network of displays
US20070100698A1 (en) * 2005-07-08 2007-05-03 Onestop Media Group Adaptive advertisements and adaptive advertising distribution system
US20070038516A1 (en) * 2005-08-13 2007-02-15 Jeff Apple Systems, methods, and computer program products for enabling an advertiser to measure user viewing of and response to an advertisement
US20070073562A1 (en) * 2005-09-28 2007-03-29 Sabre Inc. System, method, and computer program product for providing travel information using information obtained from other travelers
US20070136133A1 (en) * 2005-12-12 2007-06-14 Yvonne Li Digital Signage Transaction and Delivery Methods
US20070156382A1 (en) * 2005-12-29 2007-07-05 Graham James L Ii Systems and methods for designing experiments
US20090030780A1 (en) * 2006-01-03 2009-01-29 Ds-Iq, Inc. Measuring effectiveness of marketing campaigns presented on media devices in public places using audience exposure data
US7861150B2 (en) * 2006-11-07 2010-12-28 Microsoft Corporation Timing aspects of media content rendering
US20080230604A1 (en) * 2007-03-22 2008-09-25 Robert Jason Fong System for optimizing on-premises advertisements
US20080306804A1 (en) * 2007-06-06 2008-12-11 Opdycke Thomas C Systems for scheduling marketing campaigns in public places in order to enable measurement and optimization of audience response
US9947018B2 (en) * 2007-07-03 2018-04-17 3M Innovative Properties Company System and method for generating time-slot samples to which content may be assigned for measuring effects of the assigned content
WO2009006542A2 (en) 2007-07-03 2009-01-08 3M Innovative Properties Company System and method for assessing effectiveness of communication content
KR20100039875A (en) 2007-07-03 2010-04-16 쓰리엠 이노베이티브 프로퍼티즈 컴파니 System and method for assigning pieces of content to time-slots samples for measuring effects of the assigned content
KR20110110786A (en) 2009-01-07 2011-10-07 쓰리엠 이노베이티브 프로퍼티즈 컴파니 System and method for concurrently conducting cause-and-effect experiments on content effectiveness and adjusting content distribution to optimize business objectives

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050159921A1 (en) * 1999-08-26 2005-07-21 Louviere Jordan J. On-line experimentation
US20030083822A2 (en) * 2001-05-15 2003-05-01 Psychogenics, Inc. Systems and methods for monitoring behavior informatics
US6840905B2 (en) * 2001-06-06 2005-01-11 International Business Machines Corporation System and method of automating multidimensional scaling for psychophysics
US20040088726A1 (en) * 2002-11-01 2004-05-06 Yu-Fei Ma Systems and methods for generating a comprehensive user attention model
US20040210471A1 (en) * 2003-04-17 2004-10-21 Targetrx,Inc. Method and system for analyzing the effectiveness of marketing strategies
US20050039206A1 (en) * 2003-08-06 2005-02-17 Opdycke Thomas C. System and method for delivering and optimizing media programming in public spaces

Cited By (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090281896A1 (en) * 2005-12-29 2009-11-12 Brooks Brian E Expert system for designing experiments
US20100017288A1 (en) * 2005-12-29 2010-01-21 3M Innovative Properties Company Systems and methods for designing experiments
US10007657B2 (en) 2005-12-29 2018-06-26 3M Innovative Properties Company Content development and distribution using cognitive sciences database
US8594990B2 (en) 2005-12-29 2013-11-26 3M Innovative Properties Company Expert system for designing experiments
US20090158179A1 (en) * 2005-12-29 2009-06-18 Brooks Brian E Content development and distribution using cognitive sciences database
US20080306848A1 (en) * 2007-06-05 2008-12-11 Bank Of America Corporation Lead Generation Platform
US8392350B2 (en) 2007-07-03 2013-03-05 3M Innovative Properties Company System and method for assigning pieces of content to time-slots samples for measuring effects of the assigned content
US20090012927A1 (en) * 2007-07-03 2009-01-08 3M Innovative Properties Company System and method for assigning pieces of content to time-slots samples for measuring effects of the assigned content
US20090012847A1 (en) * 2007-07-03 2009-01-08 3M Innovative Properties Company System and method for assessing effectiveness of communication content
US9947018B2 (en) 2007-07-03 2018-04-17 3M Innovative Properties Company System and method for generating time-slot samples to which content may be assigned for measuring effects of the assigned content
US9542693B2 (en) 2007-07-03 2017-01-10 3M Innovative Properties Company System and method for assigning pieces of content to time-slots samples for measuring effects of the assigned content
US8589332B2 (en) 2007-07-03 2013-11-19 3M Innovative Properties Company System and method for assigning pieces of content to time-slots samples for measuring effects of the assigned content
WO2009036002A1 (en) * 2007-09-13 2009-03-19 Motorola, Inc. Method and system for presenting advertisements
US20110191153A1 (en) * 2007-09-13 2011-08-04 Motorola, Inc. Method and system for presenting advertisements
US20090248491A1 (en) * 2008-03-27 2009-10-01 Mark Hemphill System and method for ranking participating venues in a network according to advertisers needs for targeted advertising
US8655704B2 (en) * 2008-05-30 2014-02-18 Strategyn Holdings, Llc Commercial investment analysis
US20120317054A1 (en) * 2008-05-30 2012-12-13 Haynes Iii James M Commercial investment analysis
US20090299912A1 (en) * 2008-05-30 2009-12-03 Strategyn, Inc. Commercial investment analysis
US20150081594A1 (en) * 2008-05-30 2015-03-19 Strategyn Holdings, Llc Commercial investment analysis
US8214244B2 (en) * 2008-05-30 2012-07-03 Strategyn, Inc. Commercial investment analysis
US8543442B2 (en) 2008-05-30 2013-09-24 Strategyn Holdings, Llc Commercial investment analysis
US8924244B2 (en) 2008-05-30 2014-12-30 Strategyn Holdings, Llc Commercial investment analysis
US10157170B2 (en) * 2008-07-03 2018-12-18 Ebay, Inc. System and methods for the segmentation of media
US20140108931A1 (en) * 2008-07-03 2014-04-17 Ebay Inc. System and methods for the segmentation of media
US10282391B2 (en) 2008-07-03 2019-05-07 Ebay Inc. Position editing tool of collage multi-media
US20100082691A1 (en) * 2008-09-19 2010-04-01 Strategyn, Inc. Universal customer based information and ontology platform for business information and innovation management
US8494894B2 (en) 2008-09-19 2013-07-23 Strategyn Holdings, Llc Universal customer based information and ontology platform for business information and innovation management
US8458103B2 (en) 2009-01-07 2013-06-04 3M Innovative Properties Company System and method for concurrently conducting cause-and-effect experiments on content effectiveness and adjusting content distribution to optimize business objectives
US20100174671A1 (en) * 2009-01-07 2010-07-08 Brooks Brian E System and method for concurrently conducting cause-and-effect experiments on content effectiveness and adjusting content distribution to optimize business objectives
US9519916B2 (en) 2009-01-07 2016-12-13 3M Innovative Properties Company System and method for concurrently conducting cause-and-effect experiments on content effectiveness and adjusting content distribution to optimize business objectives
US8666977B2 (en) 2009-05-18 2014-03-04 Strategyn Holdings, Llc Needs-based mapping and processing engine
US20110145230A1 (en) * 2009-05-18 2011-06-16 Strategyn, Inc. Needs-based mapping and processing engine
US9135633B2 (en) 2009-05-18 2015-09-15 Strategyn Holdings, Llc Needs-based mapping and processing engine
US8346710B2 (en) 2010-01-29 2013-01-01 Google Inc. Evaluating statistical significance of test statistics using placebo actions
US20110191282A1 (en) * 2010-01-29 2011-08-04 Google Inc. Evaluating Statistical Significance Of Test Statistics Using Placebo Actions
US20110218837A1 (en) * 2010-03-03 2011-09-08 Strategyn, Inc. Facilitating growth investment decisions
US8583469B2 (en) 2010-03-03 2013-11-12 Strategyn Holdings, Llc Facilitating growth investment decisions
US10043131B2 (en) * 2012-01-16 2018-08-07 3M Innovative Properties Company Systems and methods for rule-based content optimization
US20140379622A1 (en) * 2012-01-16 2014-12-25 3M Innovative Properties Company Systems and methods for rule-based content optimization
US9965721B2 (en) * 2013-09-16 2018-05-08 Moresteam Development, Llc Computer-implemented system and method of performing a design of experiment, generating a product thereof, generating designed experiment options, and a graphical user interface
US20150081596A1 (en) * 2013-09-16 2015-03-19 Moresteam Development Llc Computer-implemented system and method of performing a design of experiment, generating a product thereof, generating designed experiment options, and a graphical user interface
US10049645B2 (en) 2013-12-20 2018-08-14 Koninklijke Philips N.V. Control device and method for optimizing a presentation style of specific content for a specific user
US10142685B2 (en) * 2014-01-29 2018-11-27 3M Innovative Properties Company Systems and methods for spacial and temporal experimentation on content effectiveness
WO2015116610A1 (en) * 2014-01-29 2015-08-06 3M Innovative Properties Company Conducting multivariate experiments
US20160353166A1 (en) * 2014-01-29 2016-12-01 3M Innovative Properties Company Systems and methods for spacial and temporal experimentation on content effectiveness
CN104851063A (en) * 2015-05-14 2015-08-19 上海知汇云信息技术股份有限公司 Experiment evaluation system and process management method thereof

Also Published As

Publication number Publication date
RU2008126540A (en) 2010-02-10
RU2008126539A (en) 2010-02-10
BRPI0621159A2 (en) 2011-11-29
CN101379496B (en) 2012-09-05
BRPI0621158A2 (en) 2011-11-29
EP1969488A4 (en) 2011-04-20
EP1969505A2 (en) 2008-09-17
CN101379485B (en) 2011-09-07
KR20080089424A (en) 2008-10-06
EP1969504A4 (en) 2010-11-10
US20100017288A1 (en) 2010-01-21
US20090158179A1 (en) 2009-06-18
US20140237331A1 (en) 2014-08-21
AU2006332658B2 (en) 2013-03-14
AU2006333040B2 (en) 2012-04-05
AU2006333040C1 (en) 2012-10-04
JP2009522657A (en) 2009-06-11
US20140229418A1 (en) 2014-08-14
KR20140033218A (en) 2014-03-17
JP2009522658A (en) 2009-06-11
WO2007079254A2 (en) 2007-07-12
EP1969488A2 (en) 2008-09-17
CN101379485A (en) 2009-03-04
BRPI0621157A2 (en) 2011-11-29
EP1969505A4 (en) 2011-04-06
JP2013069329A (en) 2013-04-18
KR20080081193A (en) 2008-09-08
AU2006332658A1 (en) 2007-07-12
US10007657B2 (en) 2018-06-26
JP5249045B2 (en) 2013-07-31
CN101379496A (en) 2009-03-04
WO2007078897A2 (en) 2007-07-12
KR101397745B1 (en) 2014-05-20
CA2636017A1 (en) 2007-07-12
JP2009522651A (en) 2009-06-11
WO2007079256A3 (en) 2007-08-30
CN101379495A (en) 2009-03-04
CA2634877A1 (en) 2007-07-12
JP5563649B2 (en) 2014-07-30
AU2006333040A1 (en) 2007-07-12
RU2409859C2 (en) 2011-01-20
EP2722797A1 (en) 2014-04-23
JP2014059911A (en) 2014-04-03
AU2006332660A1 (en) 2007-07-12
US8594990B2 (en) 2013-11-26
KR20080089402A (en) 2008-10-06
EP1969504A1 (en) 2008-09-17
CA2635789A1 (en) 2007-07-12
WO2007079256A2 (en) 2007-07-12
WO2007079254A3 (en) 2007-11-08
KR101336500B1 (en) 2013-12-03
RU2008126454A (en) 2010-02-10
US20090281896A1 (en) 2009-11-12
AU2006332660B2 (en) 2012-01-12

Similar Documents

Publication Publication Date Title
Pieters et al. Visual attention to repeated print advertising: A test of scanpath theory
Wedel et al. Eye tracking for visual marketing
US9077463B2 (en) Characterizing dynamic regions of digital media data
Maguire et al. User requirements analysis
Pu et al. Evaluating recommender systems from the user’s perspective: survey of the state of the art
US8370207B2 (en) Virtual reality system including smart objects
Hartmann et al. Towards a theory of user judgment of aesthetics and user interface quality
US6269173B1 (en) Instant response broadcast board system and method
US6934748B1 (en) Automated on-line experimentation to measure users behavior to treatment for a set of content elements
US20100295774A1 (en) Method for Automatic Mapping of Eye Tracker Data to Hypermedia Content
US9489178B2 (en) Method of website optimisation
US20050159921A1 (en) On-line experimentation
US20050289005A1 (en) Systems and methods of achieving optimal advertising
JP6132836B2 (en) Ad selections that through the viewer feedback
US20020089532A1 (en) Graphical user interface and web site evaluation tool for customizing web sites
US5504675A (en) Method and apparatus for automatic selection and presentation of sales promotion programs
CN101669131B (en) Cross channel optimization systems and methods
JP2004502222A (en) Method and apparatus combining the content of the information presented to the audience
Hasan Perceived irritation in online shopping: The impact of website design characteristics
JP5249045B2 (en) Expert system for planning the experiment
EP1417618A1 (en) Providing marketing decision support
WO2009011820A2 (en) System and method for determining relative preferences for marketing, financial, internet, and other commercial applications
US7707171B2 (en) System and method for response clustering
WO2013163742A1 (en) System and method for providing an application development and distribution social platform
US8602791B2 (en) Generation of test stimuli in visual media

Legal Events

Date Code Title Description
AS Assignment

Owner name: 3M INNOVATIVE PROPERTIES COMPANY, MINNESOTA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GRAHAM II, JAMES L.;CARLSON, CRAIG M.;BROOKS, BRIAN E.;AND OTHERS;REEL/FRAME:017301/0065

Effective date: 20060221

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION