JP2023519816A - Automatically improve data quality - Google Patents

Abstract

receiving a request for a digital component from a user device, determining an attribute of the user based on one or more of the information provided by the user or included in the request, and developing a behavioral model corresponding to the attribute; Identifies and dynamically alters the presentation of the item shown by the digital component based on the identified behavioral model to provide an under-represented segment of the user population in the database that contains information about the item by the user. includes a dynamically altered presentation of the item, solicits feedback from the user regarding the item, generates a digital component including a feedback mechanism, and based on the feedback obtained, prompts the user modify the presentation of the item when delivered to other users with attributes of

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is the subject of U.S. application Ser. It claims the benefit of Application No. 63/010,399.

This paper describes data collection and model generation that continuously reduces the bias introduced by using data points from unevenly distributed or unrepresentative populations to explore the design space. Concerning providing process.

In general, one inventive aspect of the subject matter described herein can be implemented in a method for improving the robustness of data sets executed by one or more data processing apparatus, the method comprising: and based on one or more of the information provided by the user or contained within the request for the digital component, one of the users or determining a plurality of attributes; identifying a behavioral model corresponding to the one or more attributes of the user; and based on the identified behavioral model corresponding to the one or more attributes of the user, In response to dynamically altering the presentation of the item represented by the digital component and determining that the user has particular attributes corresponding to an underrepresented segment of the user population, determining that the item has particular attributes corresponding to an underrepresented segment of the user population in a database containing information about the item; generating a digital component that includes a presentation of the item, solicits feedback from the user regarding the item, and includes a feedback mechanism that allows the user to submit feedback regarding the item; updating the database to include the feedback; and, based at least in part on the feedback obtained from the user, presenting the item when delivered to other users having one or more attributes of the user. and modifying.

These and other embodiments may each optionally include one or more of the following features.

In some implementations, the method includes, in response to receiving feedback from a user having a particular attribute corresponding to an underrepresented segment of the user population, one or more attributes of the user. Indicating the feedback information; and storing the indicated feedback information in the indicated searchable database.

In some implementations, determining that the user has a particular attribute corresponding to an underrepresented segment of the user population in the database includes identifying an underrepresented segment of the user population. using statistical analysis to

In some implementations, based at least in part on feedback obtained from the user, modifying the presentation of the item when delivered to other users having one or more attributes of the user comprises: Selecting a particular feedback mechanism included by the digital component.

In some implementations, dynamically altering the presentation of the item shown by the digital component based on the identified behavioral model corresponding to one or more attributes of the user includes: using machine learning or artificial intelligence techniques to identify feedback to be solicited from.

In some implementations, determining the one or more attributes of the user is based on information provided by the user and based on identified behavioral models corresponding to the one or more attributes of the user. , dynamically altering the presentation of the item shown by the digital component includes updating the identified behavioral model based on information provided by the user.

In some implementations, determining the one or more attributes of the user includes determining the identified behavior corresponding to the one or more attributes of the user based on information contained within the request for the digital component. Based on the model, dynamically altering the presentation of the item indicated by the digital component includes updating the identified behavioral model based on information contained within the request to the digital component.

In some implementations, the method includes selecting a format in which feedback is solicited for a digital component that solicits feedback from users regarding the item, and accommodating underrepresented segments of the user population. and verifying the information requested by the digital component based on certain attributes of the digital component.

In some implementations, the digital component displays two different product design shapes for the suit and footwear, and the feedback obtained from the user about the item determines the two shapes for the footwear that the user thinks will look better on the suit. Includes a selection of one of four different product design shapes.

In some implementations, the digital component represents two different product design shapes for the car, the digital component specifies a particular subjective product design shape descriptor, and the feedback mechanism is A slider, the feedback obtained from the user on that item allows the user to choose between two different product design shapes for the car, which the user believes can be better described by a particular subjective product design shape descriptor. include.

In some implementations, the digital component displays three or more different product design shapes for three or more different item types, and feedback obtained from users regarding the items is visually coherent. and the method also includes using feedback obtained from the user in separate models.

In some implementations, the digital component requests a user to create a new product design, the feedback mechanism receives user input indicative of the new product design, and the method also generates a product having the new product design. to the user.

In some implementations, the digital component requests a user to modify an existing product design to produce a customized product design, and the feedback mechanism determines one or more aspects of the existing product design. and the method also includes providing the product with the customized product design to the user.

In some implementations, the digital component displays two different software attributes and the feedback obtained from the user regarding the item includes a selection of one of the two different software attributes that the user prefers; The method also includes providing the user with the software package having the selected one of the two different software attributes that the user prefers.

Other embodiments of this aspect include computer programs encoded on corresponding systems, apparatus, and computer storage devices and configured to perform the actions of the method.

Particular embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages. To date, no method exists to automatically and systematically reduce bias in datasets that are determined to be unrepresentative of the population or lacking data in certain segments in certain circumstances, and its shortcomings are , is addressed by the techniques, devices, and systems discussed herein.

In some implementations of this new system, underrepresented segments of the population about which data is collected are identified and tasks are generated for delivery to users within these segments. These tasks solicit responses from the user on specific topics or areas so that they complement existing datasets. For example, the system determines that there are less than a threshold amount of responses regarding preferred choices between two products from users within a particular age range, and then chooses to choose between those two products. A task may be generated for delivery to users within this age range to request the user. These responses are received and processed by the system in addition to existing data sets, thereby improving data quality. The system continuously monitors the dataset and provides a solution that automatically maintains data quality even when the dataset is updated.

This new system automatically improves data quality and has been used by many to improve model results when no solution existed to maintain data quality as more data was collected. have access to a complex data processing infrastructure that cleans, processes, and maintains a comprehensive set of labeled, searchable data that can be used across different contexts. If the model uses a data set that is incomplete or unrepresentative, the model may produce results that are not representative of the actual behavior of the population. The new system automatically supplements the dataset from which the model draws, improving the robustness of the data and the accuracy of the model's results. By automatically identifying population segments for which insufficient or unrepresentative amounts of data exist, the system reduces bias in datasets used as inputs in various models. These more robust datasets improve the reliability and accuracy of data-dependent model results. The improved datasets can be displayed and used by different parties, including content providers or product manufacturers who may not have the infrastructure or access to the displayed, searchable datasets maintained by the system. .

In addition to improving the dataset, the system can automatically explore the design space. A design space is a conceptual representation of a design's values or attributes, sometimes referred to as a continuous shape. A design space may be associated with a particular product or service, may be multi-dimensional, and represent possible design values for key parameters. In some implementations, the design space may be extended to map design parameters to semantic values. For example, the system may use mappings of semantic attributes and geometric features of products to create models, which are single continuous shapes representing a universe of possible attribute values. exist in space. A design may be based on one or more perceptual features. For example, the design may be visual, auditory, tactile, scent-based, or taste-based. For example, a design for a car seat may include visual and tactile features.

This new system automates the design space by identifying segments of the design space with little or no data and generating tasks for the identified segments that will be delivered to users. can be explored. By collecting data on these segments, the system allows previously unexplored designs to be taken into account. For example, the system may automatically generate designs with parameter values that have not been submitted to the user for feedback. Based on the feedback received, the system can continuously update existing designs and generate new designs for which user feedback is solicited prior to the prototyping, manufacturing and distribution stages. This feedback may, for example, be provided as an input to a model that outputs predictions of user preferences. Based on the output of data analysis and behavioral models, the system guides product designs and focuses content providers, product designers, and manufacturers on designs most likely to be appreciated by their target segment of consumers. can determine the direction that allows In other words, the improved update process reduces the number of feedback cycles required to finalize a product design, thereby reducing the time required for the complete design cycle from initial data collection to design finalization. can reduce computational resources.

Leveraging this method enables rapid design development of products that more accurately and faithfully meet consumer needs and expectations across various demographics. In addition, the improved update process provides efficiencies in design systems by improving data quality and reducing the number of feedback cycles required to collect user data for input into behavioral models. do. The system allows product designers, developers, and manufacturers to receive diverse feedback on a global scale immediately, and adapt user preferences according to demographic data prior to the resource-intensive manufacturing and shipping stages. provide a means for determining This method may allow design and manufacturing to be personalized to an individual user or segment of users. For example, this approach can be used by manufacturers to examine individual preferences to gauge acceptance statistics for the larger market.

The techniques described herein enable the system to use fewer resources and perform fewer operations to produce high quality data related to a particular product or service. By automatically detecting and reducing bias in datasets, the system enables models using the data to provide more accurate and reliable results. Additionally, the system reduces the amount of resources required to complete a new product or service design by enabling targeted exploration of the design space and continuous improvement of data quality.

The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will be apparent from the description, drawings, and claims.

1 is a block diagram of an exemplary environment for data quality improvement and design space exploration; FIG. 4 is an exemplary data flow of the data quality improvement process; FIG. 3 illustrates the model training process; FIG. 3 illustrates the model training process; 4 is an exemplary data flow of the data quality improvement process; 4 is an exemplary data flow of the design space exploration process; Figure 2 is a specific example data flow in which the system uses behavioral models to generate tasks for users; Figure 2 is a specific example data flow in which the system uses behavioral models to generate tasks for users; 4 is a specific example data flow in which the system incorporates user feedback into the design cycle; 4 is a specific example data flow in which the system incorporates user feedback into the design cycle; A specific example data flow in which the system implements user feedback to customize existing designs and products. A specific example data flow in which the system implements user feedback to customize existing designs and products. A specific example of how the system implements user feedback within the design cycle. A specific example of how the system implements user feedback within the design cycle. A specific example of how the system implements user feedback within the design cycle. A specific example of how the system implements user feedback within the design cycle. 4 is a flowchart of an exemplary process for data quality improvement; 4 is a flowchart of an exemplary process for automatic design space search; 1 is a block diagram of an exemplary computing system; FIG.

The same reference numbers and names in the various drawings identify the same elements.

This specification describes methods, systems, and devices for improving data quality, reducing inherent bias, and enabling automatic intelligent design space exploration. A model is only a representative population as accurate as the input data provided to the model. The proposed system improves the quality of data that can be provided to various systems for modeling and product development. User feedback data and behavioral data may be collected in a variety of ways.

In some implementations, the system generates tasks that are delivered to users. Each task may be a question, task, or other form of request for input from the user. User input can be displayed and used as part of a comprehensive database that can be used in different contexts. For example, the user may be presented with a task to complete in which the user must select a single logo design from among multiple logo designs that looks most appealing. The user's selections can then be labeled with demographic information and stored as part of the labeled searchable database. Based on analyzing the indicated data set, the system identifies data from specific user demographics or data pertaining to specific product segments that are missing, inadequate, or representative of known populations. Decide not and automatically generate tasks or questions to collect more data and reduce inherent biases in non-representative datasets. A complementary set of labeled data can be provided as input to the various models. For example, an indicated set of data can be provided to a behavioral model to predict whether a particular design will be attractive to a particular segment of users. For example, various models can be used to predict user reaction and reception of a particular design.

The system also enables automatic and intelligent exploration of specific design spaces. For example, the system determines that data for a particular area of the design space is missing, inadequate, or not representative of a known population based on analyzing the indicated data set; Tasks or questions can be automatically generated to collect more data. The system may explore the design space by using artificial intelligence and machine learning models to generate product designs based on unexplored areas of the design space. These artificially generated designs can then be presented to the user along with a task requesting feedback.

FIG. 1 is a block diagram of an exemplary environment 100 for data quality improvement and design space exploration. Exemplary environment 100 includes a network 102 such as a local area network (LAN), wide area network (WAN), the Internet, or combinations thereof. Network 102 connects electronic document servers 104 (“Electronic Doc Servers”), user devices 106, and digital component distribution system 110 (also referred to as DCDS 110). Exemplary environment 100 may include many different electronic document servers 104 and user devices 106 .

User device 106 is an electronic device capable of requesting and receiving resources (eg, electronic documents) over network 102 . Exemplary user devices 106 may include personal computers, mobile communication devices, and other devices capable of transmitting and receiving data over network 102 . User device 106 typically includes a user application, such as a web browser, to facilitate sending and receiving data over network 102, although native applications executed by user device 106 also use network 102. It can also facilitate the transmission and reception of data via.

One or more third parties 140 include content providers, product designers, product manufacturers, and other parties involved in the design, development, manufacture, marketing, or distribution of products or services.

An electronic document is data that presents a set of content on user device 106 . Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (eg, "apps"), such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents 105 (“Electronic Docs”) can be provided to user devices 106 by electronic document servers 104 . For example, electronic document server 104 may include a server that hosts a publisher website. In this example, a user device 106 can initiate a request for a given publisher web page, and an electronic document server 104 hosting a given publisher web page can access the given web page on the user device 106. Requests may be responded to by sending machine hypertext markup language (HTML) code that initiates presentation of the page.

Electronic documents may contain a variety of content. For example, electronic document 105 may contain static content (eg, text or other designated content) that is within the electronic document itself and/or does not change over time. Electronic documents can also contain dynamic content that can change over time or from request to request. For example, a publisher of a given electronic document may maintain data sources used to populate portions of the electronic document. In this example, a given electronic document instructs user device 106 to request content from a data source when the given electronic document is processed (eg, rendered or executed) by user device 106 . may contain tags or scripts that User device 106 incorporates content obtained from the data source into a given electronic document to create a composite electronic document containing content obtained from the data source.

In some situations, a given electronic document may contain digital content tags or digital content scripts that reference DCDS 110 . In these situations, the digital content tags or digital content scripts are executed by user device 106 when a given electronic document is processed by user device 106 . Execution of the digital content tag or digital content script configures the user device 106 to generate a request 108 for digital content that is sent over the network 102 to the DCDS 110 . For example, a digital content tag or digital content script can enable user device 106 to generate a packetized data request that includes header and payload data. Request 108 may include the name (or network location) of the server from which the digital content is being requested, the name (or network location) of the requesting device (eg, user device 106), and/or the name (or network location) of the requesting device (e.g., user device 106) provided by DCDS 110 in response to the request. may include data such as information that can be used to select digital content to be delivered. Request 108 is sent by user device 106 over network 102 (eg, a telecommunications network) to the server of DCDS 110 .

Request 108 may include data specifying characteristics of locations where electronic documents and digital content may be presented. For example, data specifying references (e.g., URLs) to electronic documents (e.g., web pages) on which digital content is presented; available locations of electronic documents available for presenting digital content (e.g., digital content slots), the sizes of available locations, the locations of available locations within the presentation of the electronic document, and/or media types eligible for presentation at the locations may be provided to DCDS 110 . Similarly, data specifying keywords specified for the selection of electronic documents ("document keywords") or entities (e.g., people, places, or things) referenced by an electronic document may also be included in the request 108 (e.g., payload data) and may be provided to the DCDS 110 to facilitate identification of digital content items eligible for presentation with the electronic document.

The request 108 may also include user-provided information, geographic information indicating the state or region in which the request was submitted, or other information that provides context for the environment in which the digital content will be displayed (e.g., mobile or tablet device, etc.). , the type of device on which the digital content is displayed), and other information related data. User-provided information may include demographic data about the user of user device 106 . For example, demographic information may include age, gender, geographic location, education level, marital status, household income, occupation, hobbies, social media data, and whether the user owns a particular item, among other characteristics. obtain.

In situations where the systems discussed herein collect or may make use of personal information about a user, the user will not be permitted to use the program or functionality to identify personal information (e.g., the user's social networks, social behavior or activities, occupational , user preferences, or information about the user's current location), or whether and/or how to receive content from content servers that may be more relevant to the user. You may be given the opportunity to control what you receive. Additionally, some data may be anonymized in one or more ways before being stored or used so that personally identifiable information is removed. For example, a user's identity may be anonymized so that no personally identifiable information can be determined about the user, or the user's geographic location may be altered so that the user's specific location cannot be determined. May be generalized (to city, zip code, or state level, etc.) when retrieved. Therefore, the user can control how information is collected about the user and used by the content server.

User device 106, such as information identifying the model of user device 106, the configuration of user device 106, or the size (eg, physical size or resolution) of the electronic display (eg, touch screen or desktop monitor) on which the electronic document is presented. Data specifying the characteristics of may also be provided in the request 108 . The request 108 can be sent over a packetized network, for example, and the request 108 itself can be formatted as packetized data with header and payload data. The header may specify the destination of the packet and the payload data may contain any of the information discussed above.

DCDS 110, in response to receiving request 108 and/or using information contained within request 108, selects digital content to be presented with a given electronic document. In some implementations, DCDS 110 is an interconnected distributed computing system (or environment) that includes, for example, a set of servers and multiple computing devices that identify and deliver digital content in response to requests 108. Implemented. A set of computing devices work together to identify a set of digital content eligible to be presented in an electronic document from among the millions of available corpora of digital content. The millions of available digital content can be indexed in a digital component database 112, for example. Each digital content index entry may reference corresponding digital content and/or may include distribution parameters (eg, selection criteria) that regulate distribution of the corresponding digital content.

In some implementations, digital components from digital component database 112 may include content provided by third parties 140 . For example, the digital component database 112 may receive photographs of public intersections from third parties 140 that use machine learning and/or artificial intelligence to navigate public roads. In another example, the digital component database 112 may receive from a third party 140 that provides services to cyclists specific questions to which the third party 140 desires responses from the user.

Identification of eligible digital content can be segmented into multiple tasks, which are then assigned among computing devices in a set of multiple computing devices. For example, different computing devices in the set of multiple computing devices each analyze a different portion of the digital component database 112 to identify various digital content having delivery parameters that match the information included in the request 108. can be done.

DCDS 110 aggregates the results received from a set of multiple computing devices and uses information associated with the aggregated results to determine one of the digital content to be provided in response to request 108. Select one or more instances. DCDS 110 then generates response data 114 (e.g., digital data representing the response) that enables user device 106 to incorporate the selected set of digital content into a given electronic document and distributes the data over network 102 . , resulting in the selected set of digital content and the content of the electronic document being presented together on the display of the user device 106 . Digital content delivered by DCDS 110 and represented by response data 114 may include, for example, digital content that solicits input from a user. This input may be analyzed, labeled, and stored as part of a comprehensive database, such as indicator database 130 . An indication database 130 stores the indicated data that has been analyzed and classified. The indication database 130 is searchable and can store data about users, including demographic information, user response data, and other user characteristics. For example, the indication database 130 may store a user's anonymized demographic information and associate it with the user's responses to questions previously posed to the user. Input from the user is sent to data quality processor 120 as response data 116 from user device 106 .

The data quality processor 120 generates such digital content soliciting user input, receives and processes user input data, and creates and modifies design spaces and designs. Data quality processor 120 includes task processor 122 , data processor 124 , and model generator 126 . Task processor 122 generates tasks to be delivered to users. Data processor 124 analyzes and displays the input received in response to the task. The model generator 126 generates and modifies design spaces and designs based on the indicated data and input from third parties such as content providers, product designers, and manufacturers, among other parties. For ease of explanation, task processor 122, data processor 124, and model generator 126 are shown as separate components of data quality processor 120 in FIG. Data quality processor 120 may be implemented as a single system on computer-readable media, which may be non-transitory. In some implementations, one or more of task processor 122, data processor 124, and model generator 126 may be implemented as integrated components of a single system.

The task processor 122 creates digital content that solicits input or tasks from the user. Task processor 122 is in communication with DCDS 110 , electronic document server 104 , and third party 140 . Data quality processor 120 receives data from tasks issued directly by task processor 122 or from tasks issued by a third party, such as third party 140, that provides data quality processor 120 with access to its data sources. can be collected. Tasks can range from an activity that asks the user to draw a picture to a question that simply asks the user to select an answer to a single-click activity that asks the user to give permission to the system to access the user's data. may include content with varying levels of interaction. In some implementations, a task may include a question that requires the user to enter an answer. For example, a task presented to a user may include the question "Which do you prefer, chocolate ice cream or vanilla ice cream?" Either choice is possible. In some implementations, tasks may include activities that require more involvement on the part of the user. For example, a task presented to the user may require selection of one or more portions of an image of a road intersection containing bicycles, and the user clicks or possibly indicates the appropriate portion. be able to. In some implementations, a task may include an authentication protocol challenge, such as CAPTCHA or reCAPTCHA.

In addition to generating tasks to be serviced to users, task processor 122 may modify tasks. For example, the task processor 122 may modify a task previously serviced to one or more users, modify the task to collect different data, ask more targeted questions, or if Depending on the task, the instructions for the task can be changed. Task processor 122 and its outputs are described in more detail below.

Data processor 124 receives and processes data, identifies missing, inaccurate, underrepresented, or non-representative data, and automatically determines data quality improvement solutions. Data processor 124 analyzes a particular data set and, based on design guidelines and other inputs, determines whether existing data meets quality thresholds. Data processor 124 processes response data received from user devices and existing response data. For example, the data processor 124 may determine the ratio of responses received from consumers in the age 45-54 demographic versus responses received from consumers in other age groups, and compare the existing ratio to camping backpacks. Existing user response data stored within the indication database 130 for specific camping backpack designs is representative from consumers of this age by comparing to expected or real-world proportions for the target market. can determine whether it contains a significant number of responses. Data processor 124 can also determine whether design values in the design space have been searched or whether sufficient data exists for those values. For example, data processor 124 determines whether a particular size trackpad on a laptop has received a sufficient number of user responses by comparing the existing number of user responses to a threshold user response amount. obtain.

Data processor 124 can also receive and extract data collected from the digital content distribution process. For example, data analyzer 124 may receive request data 108 and response data 114 to determine the population and characteristics of users represented by the cookies indicated in request data 108 and response data 114 . Data analyzer 124 may store demographic and other characteristic data in a database, such as indicator database 130 . In some implementations, the data analyzer 124 can retrieve data from the indication database 130 that has been analyzed and labeled by other systems. Data analyzer 124 may, for example, retrieve data from display database 130 that indicates demographic data for users who provided request data 108 and received response data 114 . Data processor 124 may, for example, segment the data based on demographic information. Data processor 124 and its outputs are described in more detail below.

In addition to the above discussion, the user may also request that the systems, programs, or features described herein include user information (e.g., user's social networks, social actions or activities, occupation, user's preferences, or user's current allows the user to make choices regarding both whether and when the collection of information about location) can be enabled and whether content or communications from the server is sent to the user Control can be given. Additionally, some data may be treated in one or more ways before being stored or used such that personally identifiable information is removed. For example, a user's identity may be treated such that personally identifiable information cannot be determined about the user, or the user's geographic location may be obtained such that the user's specific location cannot be determined. It can be generalized in some cases (to city, zip code, or state level, etc.). Thus, the user can control what information is collected about the user, how that information is used, and what information is provided to the user.

Model generator 126 creates, modifies, and maintains design spaces and models. Model generator 126 may generate design spaces and/or models based on data provided by data processor 124 or retrieved from indication database 130 . For example, model generator 126 may generate behavioral models that predict user preferences for particular designs based on data provided by data processor 124 . Behavioral models map subjective factors to continuous semantic shapes according to demographics, creating a design space that can be used to optimize designs within specific constraints.

The outputs of such models can range from specific designs to expected user responses to specific designs. The model generator 126 and its outputs are described in further detail below.

The techniques described below allow the system to continuously and automatically improve data quality and explore the design space.

FIG. 2A shows an exemplary data flow 200 of the data quality improvement process in the exemplary environment of FIG. The operations of dataflow 200 may be performed by various components of system 100 . For example, the operations of dataflow 200 may be performed by data quality processor 120 in communication with DCDS 110 , user device 106 , third party 140 , and indication database 130 .

Flow begins at step A, where data processor 124 of data quality processor 120 analyzes and segments a set of data related to a population of users. In some implementations, the set of data is existing data retrieved from the indication database 130 . For example, a set of data may include a set of user inputs in response to a reCAPTCHA asking the user to select from a grid of photographs all rectangles representing portions of a vehicle type, such as a bicycle. In another example, the set of data may include a set of user inputs in response to a question asking the user how much they typically spend on dog food each month. User input may be associated with user characteristic data of the user who provided the input. For example, user characteristic data may include demographic data and browsing history, among other data that system 100 has access to and that system 100 has permission to use. In another example, a set of data indicates how likely a user is to purchase a particular handbag product design across a range of handbag designs mapped to subjective descriptors such as "practical" or "fashionable." It may include a set of user inputs in response to questions that request the user to rate.

Data processor 124 segments the data based on various parameters, including user characteristic data. For example, data processor 124 may compile sets of data based on user age, user location, and/or user interests, among other user-characteristic data that system 100 has access to and is authorized to use. Can be divided into segments. In another example, data processor 124 may segment data based on characteristics of the data itself. For example, the data processor 124 partitions the data set based on the value of certain subjective factors of the product design, such as how "stylish" the handbag product design is perceived based on the user-provided feedback. be able to.

Flow continues to step B, where data processor 124 identifies segments of the data set that are underperforming according to one or more metrics. In some implementations, this metric is provided by a third party 140, such as a product designer. For example, this metric may be the number of user responses and the target demographic of the responding users. In some implementations, this metric is automatically determined. For example, this metric may be a threshold difference in population proportion of responding users, where a data set with a population proportion difference greater than the threshold difference is from which a response was solicited. It may be considered not representative of the actual population. In one example, data processor 124 may identify a bicycle detection segment of the set of vehicle detections as having insufficient granularity.

Flow continues to step C, where task processor 122 dynamically alters the task presented to the user based on the identified segments. In some implementations, the task processor 122 dynamically modifies existing tasks that have been previously generated and/or presented to the user. In some implementations, task processor 122 generates entirely new tasks that are presented to the user. Task processor 122 can perform this alteration in real-time in response to segment identification. For example, data quality processor 120 can continuously monitor the quality of the dataset and update its metrics based on new and updated information received.

In one example, the task processor 122 asked the user to select all rectangles containing a portion of the bicycle by modifying the grid system to use smaller rectangles to provide a better resolution. Automatically modify one aspect of a previously delivered task. In step B, the data processor 124 automatically determines that a higher granularity was required, and the task processor 122 can use this information to divide the photograph of the intersection with the bicycle into smaller rectangles. can.

The data quality processor 120 automatically modifies or generates new tasks based on analysis of existing data sets and performs additional actions to help generate the tasks. For example, data quality processor 120 may determine that the modified task includes providing the user with photographic data of an intersection with a bicycle in view. Data processor 122 may receive intersection-specific photographic data from third party 140 . Data processor 122 can also automatically source data to be provided as part of a task. For example, the data processor 122 may retrieve photographic data from, for example, the marking database 130 marked as a public intersection with at least one bicycle in view. The data processor 122 can then perform data cleansing operations, which, among other operations, scrub the data for personally identifiable information, clean the data, and make the data usable. including adjusting the data so that For example, the data processor 122 may, among other things, filter out images that do not contain bicycles in view, adjust lighting, and create more dynamic ranges in the image, thereby allowing street traffic to be trained at public intersections. You can adjust the live stream of photos from your camera. The data processor 122 is capable of performing complex data manipulation operations that, among other operations, remove objects that occlude other objects and Including improving focus.

Task processor 122 may also determine one or more delivery parameters that must be met prior to delivery of the task based on the identified segments. Delivery parameters may include user characteristics that a user must have in order to be served the task. For example, distribution parameters may include a particular demographic of women ages 18-24 living on the West Coast of the United States.

The task processor 122 may modify the task to, for example, sample existing or new areas of the image, video, or audio object space in the indication database 130 . The task processor 122 may also test removal or addition of brand information, for example, to assess user reaction or user bias regarding the brand.

Flow continues to step D, where task processor 122 sends the dynamically modified or generated task to DCDS 110 for delivery to the user. For example, task processor 122 may transmit task data indicating the tasks to be presented to the user and the content to be presented to the user as part of the task. Task processor 122 may include delivery parameters that must be met in order for a task to be delivered to a particular user. For example, task processor 122 may include demographic data of target users to whom tasks may be presented.

Flow continues to step E, where DCDS 110 receives request 108 for content from user device 106 . Requests 108 are sent to DCDS 110 by user devices 106 when client devices interact with digital content. For example, if a user of user device 106 clicks a link to download a shopping application, that link may cause user device 106 to send request 108 to DCDS 110 . Request 108 may include interaction tracking data from client device 106 . For example, request 108 may include tracking data such as interaction instructions, digital content with which user device 106 interacted, and an identifier that uniquely identifies user device 106 . In some implementations, the request 108 includes an indication of the provider of the digital content and the location of the destination server hosting the requested resource.

Flow continues to step F, where DCDS 110 sends reply data 114 to user device 106 . As explained above, response data 114 may indicate tasks to be delivered to users that meet certain delivery parameters in addition to the requested electronic document. Response data 114 is sent by DCDS 110 to user device 106 in response to receipt of request 108 by DCDS 110 and a determination that the delivery parameters are satisfied based on the received delivery parameters and the user data indicated in request 108 . be. For example, DCDS 110 may determine, based on receipt of request data 108, that user of user device 106 is a 22-year-old female living in Oregon, so that the delivery parameters may be satisfied. DCDS 110 can then send the requested electronic document and dynamically modified task in response data 114 to user device 106 .

Flow continues to step G, where DCDS 110 receives response data 116 from user device 106 . Response data 116 is sent by user device 106 to DCDS 110 in response to completion of the task performed in response data 114 by the user of user device 106 . Response data 116 includes user information, such as demographic data, device data, information about user responses, and includes user inputs in response to the tasks provided in response data 114 . For example, the response data 116 may include the user's selection of a rectangle containing a portion of a bicycle, the amount of time the user spent making that selection, the user's mouse movement pattern, and the user's mouse movement pattern, all of which the user has granted access to the system 100 . may include anonymized demographic data, device data, and browsing history for The response data may include user-provided semantic descriptors about the task, product, or design. Semantic descriptors may include any descriptor that provides semantic information about an object, such as a product or design. Semantic descriptors may be generated by humans or artificial intelligence, and may take the form of words (eg, keywords or key phrases), sentences, symbols, or other descriptors that convey semantic information. In addition, semantic descriptors can be used for other purposes, such as interaction with presented information (e.g. pictures or icons), evaluation elements (e.g. product evaluation tools), or submission of free-form feedback on subject matter. Objects may be assigned based on actions. DCDS 110 can provide this data to data processor 124 for analysis and display.

Flow continues to step H, where data processor 124 analyzes response data 116 from user device 106 . Data processor 124 can analyze response data 116, classify the data, and mark the data with the user's information so that the data is searchable. For example, data processor 124 may display the user-selected rectangle with the user's demographic information, the amount of time the user spent making that selection, and the accuracy of the user's selection compared to a ground truth set of rectangles. .

Flow continues to step I, where data quality processor 120 provides the analyzed data to indication database 130 . Data processor 124 may provide the labeled data for storage in indicator database 130 so that the data is retrievable.

System 100 can continuously perform process 200 such that data quality processor 120 automatically and continuously monitors the amount of data contained within a particular data set. Accordingly, system 100 maintains and improves data quality across comprehensive database 130 so that the accuracy and completeness of model output results is continually improved. As the system 100 constantly updates the indication database 130, the system retrieves content, such as instances of images, audio, or video, from which the system 100 retrieves content based on user response information, such as task results and user information. provide a searchable database that

The system reduces bias in the delivery of user feedback by selectively soliciting additional feedback from under-represented or under-represented user demographics.

Figures 2B and 2C show the model training process. Server 250 maintains baseline model 252 and task repository 254 . Baseline model 252 is a model that serves as a baseline behavioral model and can be updated. Task repository 254 maintains a set of tasks that can be delivered to users.

Devices A 260a, B 260b, . . . N 260n (collectively referred to as devices 260) each include models A, B, . Each locally maintained model 262 may be updated and improved based on tasks and model updates provided by server 250 .

Devices 260 each receive input from users A 270a, B 270b, . For example, the device 260 can provide each of the users 270 with tasks such as those described above. Tasks may be provided from task repository 254, for example. Users 270 can each provide semantic mappings 272a, 272b, . . . 272n (collectively semantic mappings 272) to device 260 in response to a task. Based on the responses provided by user 270 , device 260 can update model 262 .

In FIG. 2C, model training module 256 generates updated baseline model 258 based on information from responses provided by user 270, and updates base model 252 to replace or update baseline model 252. A line model 258 can be provided to the server 250 .

FIG. 3 shows an exemplary design space 300. As shown in FIG. Design space 300 is a visual representation of a conceptual universe of possible design values. Design space 300 may be generated by a system such as system 100 shown in FIG. For example, model generator 126 of data quality processor 120 can generate design space 300 based on user response data from indication database 130 .

Design space 300 may be multi-dimensional. In this particular example, design space 300 includes two dimensions and was generated as a result of user-submitted data in response to questions requesting the user to evaluate various bag designs. In other examples, design space 300 may include two or more dimensions and may be represented as a three or more dimensional model. Dimensions include shape, color, texture, size, or relative distance to another object, among others.

Design space 300 is used to visually represent all possible designs for a particular product or service. Products can include non-durables and durables. For example, products may include clothing, cosmetics, food, furniture, and automobiles, among others.

Design space 300 can be constrained by a variety of constraints, from whether a design can be physically created, to the manufacturability of the design to the constraints required by the designer. Design space 300 may be constrained by data collected from users, such as the target audience for the product. For example, if the target population of consumers for a particular mobile device does not carry that mobile device if it weighs more than 3 pounds (1.36 kg), design space 300 may be limited in its weight dimension. In some implementations, design space 300 may be automatically bounded by data and decisions made by system 100 . For example, data processor 124 may determine, based on user response data from indication database 130, that none of the surveyed users are interested in laptops with screens less than 8 inches (20.32 cm). , and data processor 124 can provide this constraint on design space 300 to model generator 126 .

By automatically generating and modifying the design space 300, this new system reduces the amount of time and resources used to reach the final design. The system can place the emphasis of design exploration on areas most likely to yield good results according to metrics specified by the involved third parties. For example, the data quality processor 120 can automatically focus the design search for bag designs into areas where consumers aged 25 to 34 are most likely to purchase, the target demographics of third party bag designers and manufacturers 140. can be placed on target. The data quality processor 120 focuses requests on user response data for ages 25 to 34 by automatically generating bag shapes and designs that fall within the design space 300 that are most likely to be of interest to the user. It can be placed on bag designs that have been shown to be of greatest interest to the user under study.

In this particular example, the design space 300 maps semantic attributes to geometric features of the design that fall within its spatial extent. For example, design space 300 maps the semantic attributes “realistic” and “fashionable” to specific shapes and forms of bag designs, including bag designs 302 and 304 . These semantic attributes are subjective factors that represent specific qualitative design targets and demographics. Based on user response data 306 and 308 from indication database 130, data quality processor 120 determined that users were most interested in bags that were a mixture of "practical" and "fashionable." In this particular example, the data processor 124 of the data quality processor 120 analyzes the user responses from the labeling database 130 and determines that there is the highest interest in bags that are more "practical" and at least somewhat "fashionable." Decided. Using this determination, the model generator 126 can restrict the design space 300 to focus on bag designs that exceed a threshold "fashionable" quantity and some other threshold "practical" quantity. can. Model generator 126 can, in some implementations, automatically generate designs that meet these design criteria without further input from the designer. In some implementations, the model generator 126 can generate bag designs that meet "fashionable" and "practical" thresholds that have not been generated before. For example, model generator 126 can generate bag designs 312 and 314 without input from third party bag designer 140, and designs 312 and 314 can be transformed into new, previously unknown bag designs. can be.

Model generator 126 is input without additional information, for example, by using artificial intelligence and machine learning techniques to transform certain geometric features of existing designs to create new designs. A new design can be generated based on the initial set of designs. In some implementations, model generator 126 can generate an entirely new design that does not use an existing design as a starting point. The data quality processor 120 can be integrated with design programs, including 3D modeling programs and computer aided design (CAD) programs for generating new designs or modifying existing designs.

Model generator 126 may use statistical and/or machine learning models that accept user-provided information as input. Machine learning models include decision trees, generative adversarial network-based models, deep learning models, linear regression models, logistic regression models, neural networks, classifiers, support vector machines, inductive logic programming, (e.g. bagging, boosting, random Any of a variety of models can be used, such as ensembles of models (using techniques such as forests), genetic algorithms, Bayesian networks, deep learning, association rules, inductive logic, clustering, maximum entropy classification, It can be trained using various techniques, such as learning classification. In some examples, the machine learning model may use supervised learning. In some examples, the machine learning model may use unsupervised learning.

In some implementations, a probabilistic model may be used that defines a probabilistic ranking of attributes given a product design or shape or a probabilistic ranking of product designs or shapes given one or more attributes.

The model generator 126 allows the designer to explore new areas of the design space that have not been explored before, and analyzes the user response data to increase the amount of time and likelihood that the design will be appreciated by the user. By automatically generating new designs on a basis, it reduces resource-intensive feedback cycles that need to be performed specifically for a particular product. The designer can then select a particular new design, which can then be researched and tested. In some implementations, the probability ranking may be used to define a set of candidates, which may then be tested in future tasks to refine user preferences. The data collection and design generation process are described in more detail below. Model generator 126 allows system 100 to characterize user behavioral requirements for creating designs according to market demographics.

Model generator 126 generates various types of models, including generic models that are used for all users, and customized models that can be used for specific subsets of users that share a set of features. can be used and the model can be dynamically adjusted based on user information received for a particular user 102 or based on detected activity. For example, model generator 126 can use a base network for users and then adjust the model for each user.

In some implementations, model generator 126 may use machine learning to determine the objective function for a particular user based on subjective feedback from the user. Objective functions can be simple and implement common modifications to the product design. As system 100 collects user response data, system 100 anonymizes the data and provides the data to a central database, which stores and analyzes the collected data to improve common behavioral models. , allowing the system 100 to provide a more personalized strategy for each user 102 .

System 100 may utilize, for example, general profiles for users such as specific age, location, interests, and the like. The system 100 can generalize support configurations across users expected to have similar interests. In some implementations, the system 100 accepts input from the user of profile information such as the user's age, location, interests, among other parameters.

The system 100 may, for example, utilize a range-individualized "shoe size" model. For example, the system 100 may use general profiles, such as about people within a certain age range, about people in New York, about people who like motorcycles, and so on. For example, if multiple users indicate similar preferences, the system 100 can determine if there are matching products. If a matching product exists, system 100 can return that product to the user. If no matching product exists, system 100 can modify the design of the existing product or generate a new design. In some implementations, the system 100 performs aggregate personalization or clustering by mapping particular users to existing clusters or segments of users, or creates new clusters or segments of users. can be formed. The system 100 may also be used to identify purchasing trends within products or particular users. In some implementations, the system 100 performs aggregate configuration clustering by mapping users to existing customer segments or products, or mapping user preferences to existing feature sets. can do.

Additionally, each model can be individualized. For example, each model can be created from the generic model by modifying the model parameters for each user based on characteristics determined from the collected data. Each model can be different for a particular user over long and short time periods. For example, the system 100 can track how interested a user is in a particular design element and adjust the behavioral model when it is determined that the user has lost interest. In some implementations, each model is created from models that have been personalized using a generic profile, and may be further modified for each user. For example, models can be created by modifying model parameters based on characteristics about each user determined from collected data.

In some implementations, models may be individualized without using a base model. For example, user response data may be input to model generator 126 and provided to a product designer, manufacturer, or design program to be mapped to product configurations without adjustment. In one example, the model generator 126 allows users to purchase particular items immediately or set up alerts when particular items are available.

FIG. 4 shows an exemplary data flow 400 for the data space exploration process in the exemplary environment of FIG. The operations of dataflow 400 may be performed by various components of system 100 . For example, the operations of dataflow 400 may be performed by data quality processor 120 in communication with DCDS 110 , user device 106 , third party 140 , and indication database 130 .

Flow begins at step A, where data processor 124 of data quality processor 120 analyzes and segments a set of data related to a population of users. In some implementations, the set of data is existing data retrieved from the indication database 130 . For example, the data set may include a set of user responses to a question that asks the user to rate two different bag designs with respect to two semantic descriptors on a sliding scale. User input may be associated with user characteristic data of the user who provided the input. For example, user characteristic data may include demographic data, browsing history, among other data to which system 100 has access and permission.

Flow continues to step B, where data processor 124 identifies segments of the data set that are underperforming according to one or more metrics. Details of this step can be found above in the description of FIG.

Flow continues to step C, where model generator 126 generates the design space. As explained above with respect to FIG. 3, a design space is a visual representation of a universe of possible designs. In some implementations, model generator 126 simply updates an existing design space provided by third party 140 . For example, model generator 126 can receive a design space from third party bag designer 140 and update the design space based on the set of data.

Model generator 126 can also generate behavioral models that predict user reception of a particular design. For example, the model generator 126 may utilize artificial intelligence and/or machine learning techniques to determine which design spaces and which design spaces are most likely to be highly rated by a particular user population in evaluating a particular metric. A behavioral model can be generated that outputs partial ratings, does not have a sufficient number of data points, or does not represent a representative proportion of users in a particular demographic segment.

Flow continues to step D, where data processor 124 uses the behavioral model to determine one or more segments of the design space to target. For example, data processor 124 may use the output of the behavioral model generated in step C by model generator 126 to identify a design that is very "fashionable" but also very "practical" for that design. Determine that you do not have a threshold number of user responses or that no such design has been generated.

Flow continues to step E, where task processor 122 dynamically alters the task presented to the user based on the determined segment or segments. As described above with respect to FIG. 2A, the task processor 122 modifies existing tasks that have been previously generated and/or presented to the user, or generates entirely new tasks that are presented to the user. be able to. For example, the task processor 122 can generate new bag designs that are both very "fashionable" and very "practical" to be presented to the user for feedback.

In some implementations, model generator 126 may display two or more product designs to allow the user to visualize differences or variations between two or more product instances resulting from the models. can.

In some implementations, model generator 126 may be embedded with a computer-aided generative design program through which the design of a product or service package can be improved, modified, or altered.

Flow continues to step F, where task processor 122 sends the dynamically modified or generated task to DCDS 110 for delivery to the user. Details of this step can be found above in the description of FIG. 2A for step D.

Flow continues to step G, where DCDS 110 receives request 108 for content from user device 106 . Details of this step can be found above in the description of FIG.

Flow continues to step H, where DCDS 110 sends reply data 114 to user device 106 . Details of this step can be found above in the description of FIG.

Flow continues to step I, where DCDS 110 receives response data 116 from user device 106 . Details of this step can be found above in the description of FIG.

Flow continues to step J, where data processor 124 analyzes response data 116 from user device 106 . Details of this step can be found above in the description of FIG.

Flow continues to step K, where model generator 126 updates the design space and/or behavioral model based on the analyzed response data from user device 106 . The model generator 126 can narrow or expand the design space based on user feedback. For example, the model generator 126 may discard portions of the design determined to have a threshold number or percentage of user responses and less than a threshold amount of positive responses. . Model generator 126 can update the behavioral model to reflect the updated data set. For example, model generator 126 may input analyzed response data as input for training a behavioral model that predicts user reception of a particular bag design.

Data quality processor 120 may also provide the analyzed data to indication database 130 . Model and design space updates may be performed concurrently with the data quality processor 120 sending the analyzed data to the indication database 130 . In some implementations, these portions of step K may be performed asynchronously.

System 100 continuously executes process 400 as data quality processor 120 automatically and intelligently explores the design space. Thus, system 100 enables efficient design development and reduces the time and resources required to generate and finalize new designs for products or services.

The system described above with respect to FIGS. 1 and 3-4 automatically modifies tasks to provide data quality improvements. In some implementations, the data quality processor 120 evaluates data quality, for example, lack of consistent response to one or more types of designs, to different image locations, or to images with particular characteristics, among other factors. Tasks can be modified based on data that indicates particular characteristics of their segments. In some implementations, the data quality processor 120 may, for example, consider factors such as an unusually short amount of time to complete a task or lack of consistent response from a segment of users, among other factors. Tasks can be modified based on data indicative of particular characteristics. The system 100 automatically creates product designs based on consumer feedback, obtains further feedback on these designs, and innovates to align with consumer preferences and needs without designing and maintaining focus groups. design. Design improvements can occur at a faster pace and with more representative data. Additionally, additional feedback can be used as input to a network that trains a behavioral model to improve the model's classification of, for example, what constitutes a positive or negative example.

Figures 5A and 5B show the data flow in which the system uses behavioral models to generate tasks for users.

FIG. 5A shows the data flow 500 when the system has user content for personalizing the tasks the user receives in the exemplary environment of FIG. The operations of dataflow 500 may be performed by various components of system 100 . For example, the operations of dataflow 400 may be performed by data quality processor 120 in communication with DCDS 110 , user device 106 , third party 140 , and indication database 130 .

Flow 500 begins with an individual uploading (502) one or more design primitives and/or semantic descriptors, such as keywords or ways to express user preferences, such as user clicks or text input. . Uploaded data represent human preferences, semantic descriptors, or ratings that can be mapped to a given design using visual or auditory elements or features of the product or service being designed can provide information that enables For example, a market researcher may provide the system 100 with a product design for backpacks and a set of keywords related to backpacks such as "sporty", "functional", "practical", and "professional". can be done. Design primitives may relate to products or services and may include designs for physical and digital products. In another example, a task can present a design, description, or characteristics to determine the value or price of a product or service. These designs and/or keywords can be stored, for example, in a database of product designs and keywords. A person can also upload an experiment plan that determines which classes or instances of designs should be shown with which semantic descriptors. The experimental plan may include statistical measures or other guidance criteria regarding how, when, or where given design types and semantic descriptors are indicated. In some implementations, designs and/or keywords may be stored in digital component database 112 and/or labeling database 130 .

Flow 500 continues with the system selecting a format for the content to be provided to the user (504). For example, DCDS 110 of system 100 can select layouts for content to be provided to users. A layout may include, for example, the types of user interface elements available and the types of information provided. In one example, DCDS 110 may select a layout for content that includes sliders and radio buttons for tasks that will be provided to users when delivered. In another example, the format layout may include rewards provided to users when they receive answers.

Flow 500 continues with the system pre-processing the data and verifying the correctness of the task to be provided to the user (506). For example, task processor 122 and data processor 124 of data quality processor 120 can pre-process the data and verify the correctness of the task to be provided to the user based on the determined layout format ( 504).

Flow 500 continues by storing pre-processed data and verified tasks to be provided to the user (508). At flow 500, the system 100 has user consent to provide personalized content and thus pre-processed data and verified tasks are modified and/or personalized based on user information. obtain.

Flow 500 continues with a user of the website or application interacting with the served content (510). For example, the DCDS 110 selects preprocessed data and validated tasks to be provided to a particular user of a website or application, as described above with respect to FIG. User input can be received from user interaction with the content.

Flow 500 continues by storing the user's response (512). For example, DCDS 110 may receive user responses that include user information, such as user demographic data after the response data has been analyzed and processed by data processor 124 of data quality processor 120 . The analyzed response data may be labeled by data processor 124 and stored in indicator database 130 .

Flow 500 continues by building one or more behavioral models based on the user response data (514). Examples of behavioral models are the use of linear or nonlinear functional approximation (e.g., deep-learnable convolutional neural networks) to map designs, design features, design variants or graph-based design representations to semantic descriptors or assessments. including. A reversible model that allows mapping of inputs to outputs and vice versa may be used, or multiple models for mapping inputs to outputs and vice versa may be used. A behavioral model may include demographic information stored with a given user's response. An example would be the use of task data to create a model that allows viewing apparel design feedback for women aged 20-30 living in London. For example, a behavioral model may have demographics as inputs, or may be essentially probabilities that allow the computer to tailor the model response to a given demographic parameter set. For example, model generator 126 of data quality processor 120 may generate a behavioral model based on user response data, as described above with respect to FIGS. 1-4. A model or model-based analysis may include information or input from other sources such as online surveys, pricing and conversion data, sales attribution models, theme clinics, and focus group feedback. A model or model-based analysis can also use pricing information secured from task feedback.

Flow 500 continues by analyzing and identifying new ideas and/or concepts based on one or more behavioral models (516). For example, model generator 126 of data quality processor 120 can analyze and identify new design ideas and/or new design concepts based on the output of behavioral models.

Flow 500 continues by exploring and/or optimizing the design to generate new designs and/or keywords or semantic descriptors or preferences (518). For example, model generator 126 of data quality processor 120 can use the design space to explore and/or optimize designs to generate new designs or modify existing designs. Model generator 126 can also generate new keywords associated with the design. For example, model generator 126 may generate new keywords that are trending or popular in current marketing collateral and have been shown to be popular with users based on the generated behavioral models. can. A model may be used to generate a design from a semantic descriptor or evaluation, and may be used to generate a semantic descriptor or evaluation from a given design. Models can be used to explore or optimize the model space to identify new designs or semantic descriptors for inclusion in future tasks. An example can be seen in Figure 3, where the model space is shown with a design located on two axes related to keywords (stylish and practical). In this example, an optimization of the cost function weighting 60% practical against 40% stylish may select design 312 . Examples include the use of behavioral models, including pricing feedback secured from tasks, where optimization can take into account design trade-offs, costs, and pricing. These new designs and/or keywords are provided to the database of product designs and keywords mentioned at (502). In some implementations, designs and/or keywords may be stored in digital component database 112 and/or labeling database 130 .

FIG. 5B shows data flow 550 when the system does not have user content to personalize the tasks the user receives in the exemplary environment of FIG. The operations of dataflow 550 may be performed by various components of system 100 . For example, the operations of dataflow 550 may be performed by data quality processor 120 in communication with DCDS 110 , user device 106 , third party 140 , and indication database 130 .

Flow 550 follows flow 500 closely. Flow 550 begins with the customer uploading (552) one or more design primitives and/or keywords. Details of this step can be found above in the description of FIG. 5A with respect to (502).

Flow 550 continues with the system selecting a format for the content to be provided to the user (554). Details of this step can be found above in the description of FIG. 5A with respect to (504).

Flow 550 continues with the system pre-processing the data and verifying the correctness of the task to be provided to the user (556). Details of this step can be found above in the description of FIG. 5A with respect to (506).

Flow 550 continues by storing pre-processed data and verified tasks to be provided to the user (558). In flow 550, the system 100 does not have user consent to provide personalized content, so neither preprocessed data nor verified tasks are modified or personalized based on user information. not.

Flow 550 continues with a user of the website or application interacting with the served content (560). Details of this step can be found above in the description of FIG. 5A with respect to (510).

Flow 550 continues by storing the user's response (562). For example, DCDS 110 may receive user responses that may include user information, such as user demographic data after the response data has been analyzed and processed by data processor 124 of data quality processor 120 . DCDS 110 may remove user information if it is included. In some implementations, no user information is provided because the system 100 does not have the user's consent to access the user's information. The analyzed response data may be labeled by data processor 124 and stored in indicator database 130 .

Flow 550 continues with building one or more behavioral models (564) based on the user response data. Details of this step can be found above in the description of FIG. 5A with respect to (514).

Flow 550 continues with analyzing and identifying new ideas and/or concepts based on one or more behavioral models (566). Details of this step can be found above in the description of FIG. 5A with respect to (516).

Flow 550 continues with exploring and/or optimizing the design (568) to generate new designs and/or semantic descriptors. Details of this step can be found above in the description of FIG. 5A with respect to (518).

Figures 6A and 6B show the data flow when the system incorporates user feedback into the design cycle. Figures 6A and 6B are based on the data flow shown in Figures 5A and 5B. In some implementations, system 100 has user consent to personalize content. In some implementations, system 100 does not have user consent to personalize content.

FIG. 6A shows the data flow 600 when the system incorporates user feedback into the design cycle, subject to designer input in the exemplary environment of FIG. The operations of dataflow 600 may be performed by various components of system 100 . For example, the operations of dataflow 600 may be performed by data quality processor 120 in communication with DCDS 110 , user device 106 , third party 140 , and indication database 130 .

Flow 600 closely follows flow 500 or 550. Flow 600 begins with a person choosing a task format and uploading 602 design primitives and semantic descriptors. Details of this step can be found above in the description of FIG. 5A for (502) and FIG. 5B for (552). For example, users can upload design primitives, or design shapes, and associated keywords to the database. This database stores the initial product design shapes and keywords, as well as any updates based on user responses viewing the task. For example, the database can store updates based on step 618 .

Flow 600 continues with the system selecting a format for the content to be provided to the user (604). Details of this step can be found above in the description of FIG. 5A for (504) and FIG. 5B for (554). The format of primitives and semantic descriptors in storage can be selected by a person or automatically configured by the system.

Flow 600 continues with the system pre-processing the data and verifying the correctness of the task to be provided to the user (606). Details of this step can be found above in the description of FIG. 5A for (506) and FIG. 5B for (556). The data can be pre-processed, for example, to identify a set of keywords that are viable for a particular design shape. Additionally, the data can be verified for suitability or correctness for a given format or desired user audience. In some implementations, tasks may relate to specific countries, geographies, or demographic characteristics. For example, interpreting French writing may not be a suitable task for a large segment of non-French speaking users in the United States and Europe. Tasks can be stored and served with content delivered over the Internet or through applications.

Flow 600 continues by storing pre-processed data and verified tasks to be provided to the user (608). Details of this step can be found above in the description of FIG. 5A for (508) and FIG. 5B for (558).

Flow 600 continues with a user of the website or application interacting with the served content (610). Details of this step can be found above in the description of FIG. 5A for (510) and FIG. 5B for (560). For example, users interact with content and requests are made for tasks. The user can then interact with the task and provide responses. For example, a user can select the most appropriate keywords or labels for a photo or image of a product design.

Flow 600 continues by storing the user's response (612). Details of this step can be found above in the description of FIG. 5A for (512) and FIG. 5B for (562). In some implementations, only responses are recorded. In some implementations, responses and user stratum data may be stored.

Flow 600 continues by building one or more behavioral models based on the user response data (614). Details of this step can be found above in the description of FIG. 5A for (514) and FIG. 5B for (564). For example, a model may map keywords and semantic descriptors to specific design shapes or geometries. In another example, the model maps product features to preferred ensembles of other features, or maps products to preferred ensembles of other products. These models can also map design features or decisions to customer preferences and values.

Flow 600 continues by analyzing and identifying new ideas and/or concepts based on one or more behavioral models (616). Details of this step can be found above in the description of FIG. 5A for (516) and FIG. 5B for (566). For example, models are analyzed and identified to generate data and results that can be used to help determine new tasks.

Flow 600 continues with searching and/or optimizing the design to generate new designs and/or keywords (618). Details of this step can be found above in the description of FIG. 5A for (518) and FIG. 5B for (568). For example, the system can use the data to explore, optimize, or modify task content, formatting, or messaging in response to analysis. Any learnings, modifications, updates regarding models, analyses, design decisions and optimizations are stored in the database.

Flow 600 includes loading a behavioral model into a CAD-based design tool and making 620 product design decisions based on the behavioral model. In some implementations, a human designer can provide input to a CAD-based design tool and/or provide product design decisions. For example, a human designer may implicitly have in mind the desired semantically described result, the result of a CAD-based design tool and the automatically generated product design done by the model generator 126. Decisions can be shaped. CAD-based design tools can also receive constraints and specifications provided by human designers, product manufacturers, or customers, for example. In some implementations, models that capture user feedback or user behavior may be made available, directly or indirectly, for use in computer-aided design decisions and manufacturing production. These models can be used to analyze or modify the design.

Flow 600 continues with fabrication of modified design geometry (622). For example, the model generator 126 can use the behavioral model and the design space for the product to produce modified design geometry. In some implementations, the model can map keyword descriptors to actual 3D product designs, and designers can use the keyword descriptors and user feedback to create new or revised designs. can. A design may be represented as an explicit shape, as a data structure representing a shape or variations on a given shape, or as a shape graph. In one example, a designer uses a model to create or modify a design based on other products that the design can be sold or bundled with. This loop can be repeated until product requirements or release criteria are met. If these are satisfied, the design is provided as a specification, manufacturing instructions, recipe, or method that guides or directs the manufacturing process. The resulting product may be sent to the original user who provided the feedback, or provided to a warehouse or store for distribution to customers. In some implementations, this approach may be provided in a manual manner in which the human being making the decision is informed. In some implementations, this approach can be used to automatically modify designs according to constraints, specifications, or cost functions that lead the design to a better state. In some implementations, the approach may use a hybrid, where some steps are performed by humans and some steps are automated.

Flow 600 continues by determining whether the modified design geometry meets release criteria (624). For example, model generator 126 can determine whether a design shape is suitable for release or manufacturing based on release criteria from, for example, product designers, manufacturers, carriers, and the like.

Flow 600 may optionally include manufacturing and shipping the product with the modified design shape (626). For example, the design process may be integrated with the product manufacturing and distribution workflow. In some implementations, the product may be distributed to warehouses and stores, as well as the user who provided the initial response, such as other users within the same group, cluster, location, or user segment, among other shared groups. can be delivered directly to users with preferences for

FIG. 6B shows data flow 650 when the system incorporates user feedback into the design cycle to automatically generate new designs in the exemplary environment of FIG. The operations of dataflow 650 may be performed by various components of system 100 . For example, the operations of dataflow 650 may be performed by data quality processor 120 in communication with DCDS 110 , user device 106 , third party 140 , and indication database 130 .

A key feature of Flow 650 is that generative design can be used to augment the vision of a human designer, and in some implementations replace the human designer entirely, based on desired results. obtain.

Flow 650 closely follows flow 500, 550, or 600. Flow 650 begins with a customer uploading 652 one or more design primitives and/or keywords. Details of this step can be found above in the description of FIG. 5A for (502), FIG. 5B for (552), and FIG. 6A for (602).

Flow 650 continues with the system selecting a format for the content to be provided to the user (654). Details of this step can be found above in the description of FIG. 5A for (504), FIG. 5B for (554), and FIG. 6A for (604).

Flow 650 continues with the system pre-processing the data and verifying the correctness of the task to be provided to the user (656). Details of this step can be found above in the description of FIG. 5A for (506), FIG. 5B for (556), and FIG. 6A for (606).

Flow 650 continues by storing pre-processed data and verified tasks to be provided to the user (658). Details of this step can be found above in the description of FIG. 5A for (508), FIG. 5B for (558), and FIG. 6A for (608).

Flow 650 continues with a user of the website or application interacting with the served content (660). Details of this step can be found above in the description of FIG. 5A for (510), FIG. 5B for (560), and FIG. 6A for (610).

Flow 650 continues by storing the user's response (662). Details of this step can be found above in the description of FIG. 5A for (512), FIG. 5B for (562), and FIG. 6A for (612).

Flow 650 continues with building one or more behavioral models (664) based on the user response data. Details of this step can be found above in the description of FIG. 5A for (514), FIG. 5B for (564), and FIG. 6A for (614).

Flow 650 continues by analyzing and identifying new ideas and/or concepts based on one or more behavioral models (666). Details of this step can be found above in the description of FIG. 5A for (516), FIG. 5B for (566), and FIG. 6A for (616).

Flow 650 continues with exploring and/or optimizing the design (668) to generate new designs and/or semantic descriptors. Details of this step can be found above in the description of FIG. 5A for (518), FIG. 5B for (568), and FIG. 6A for (618).

Flow 650 includes loading the behavioral model into a CAD-based design tool and making 670 product design decisions based on the behavioral model. In some implementations, a generative design process can be used to automatically create or modify designs according to desired criteria or cost functions. This generative design process can provide input to CAD-based design tools and/or provide product design decisions. For example, a generative design system can determine or receive an appropriate cost function based on desired results. CAD-based design tools can also receive constraints and specifications provided by human designers, product manufacturers, or customers, for example. For example, artificial intelligence methods can be used in conjunction with iterative generative design to search a design space and produce designs that meet specifications as well as behavioral model results.

Flow 650 continues with fabrication of modified design geometry (672). Details of this step can be found above in the description of FIG. 6A with respect to (622).

Flow 650 continues by determining whether the modified design geometry meets release criteria (674). Details of this step can be found above in the description of FIG. 6A with respect to (624).

Flow 650 may optionally include manufacturing and shipping the product with the modified design shape (676). Details of this step can be found above in the description of FIG. 6A with respect to (626).

In one example, a task may include showing a user two different products, such as shoes, and asking the user which pair of shoes would look better with the apparel shown. Apparel may be, for example, a suit. Apparel may include multiple different clothing items and may require the user to select an ensemble.

In one example, a task may include how a user configures or modifies a design, and may solicit feedback on how to create a new product or improve an existing product. . The improved product may then be provided to the user in new tasks, and the user may iteratively improve the product design.

In one example, the task presents the user with two different product designs, such as designs for cars, and asks the user how well a given attribute, such as the subjective product design descriptor (or adjective) "compact", describes each design. providing a feedback mechanism, such as a slider or button, to provide an assessment of how the

A computer-aided design process can be automatically driven to generate and modify designs according to standards and specifications. A cost function may be used to trade off design criteria or factors within constraints and specifications. Behavioral models can be used with cost functions to maximize user preferences within constraints, specifications, or manufacturing rules. For example, generative design methods can be used to iterate over a design space guided by behavioral models and cost functions or decision criteria. AI and enhancements to use generative design to iterate over many design decisions that are allowed or within specifications or constraints but may represent different customer preferences, such as body style or running shoe color combinations Including learning use. Generative algorithm approaches include one or more of evolutionary algorithms, variational ordinal encoders, and generative adversarial networks. These methods can utilize cloud computing and iterate through a large number of design iterations to optimize individual customers, segments of customers, or multiple segments of customers. Examples of algorithms that may be used include evolutionary algorithms, including genetic algorithms that evolve a given design or create hybrids of multiple designs. In some implementations, the system can use a generative adversarial network that inputs multiple existing designs to create entirely new designs. These methods can be iterated to produce new designs that the task presents to the user to provide feedback. The entire process can be automated to iterate, optimize and produce new designs. In some implementations, the system can be incorporated into automated manufacturing flows using robotics or 3D printing.
In some implementations, the system can be used to personalize products to users or user segments based on the user's provided preferences.

Variational autoencoders and associated generative adversarial networks can then be used in conjunction with task generation behavioral models to develop new designs that are fed into the system for additional user feedback. The system can be iterated and scored until a stopping criterion defined by a cost function is reached.

Variational autoencoders can be used with libraries of designs represented as images, shapes, shape-related data structures, shape-related graphs, or polygonal data. Variational autoencoders can be used to create a set of latent factors that effectively represent a reduced set of features that describe a given design. Once the variational autoencoder is trained, the design is encoded into its latent factors and then decoded into its reconstructed design. In one example, an existing library of design representations can be used to build an initial mapping to accelerate convergence. A behavioral model developed using task-based feedback can be used to map a given reconstructed design representation to a user characterization or classification. In some implementations, the user classification may be a scoring of bad (unfavorable) versus good (favorable). User/human-based characterizations can be used in conjunction with cost functions to create mathematical expressions or scores that can be used by optimizers to create or improve designs. The output of the optimizer is the design type associated latent factor description. Optimization can even use genetic mutations or crossovers to create new latent factors. New latent factors can be passed through a decoder/generator to create new design representations that can be inserted into new tasks. Designs, latent factors, and keywords all provide spatial and distance metrics of designs that can be used to relate or cluster one design to another in creating new tasks that are presented to the user. Form. The system can automatically iterate around the design space until cost-based scoring converges, where either the introduced latent factor meets the stopping criteria or the correction is very small. . At that point, the design is considered complete and ready for production.

In another implementation, creating or optimizing the design includes creating a mathematical representation of a design space in which all car or apparel designs are represented by design features defined by each dimension of the space. obtain. An N-dimensional feature space is represented by N-space. In many cases, the space can be transformed to a lower dimensional space. In other cases, a distance metric can be used to cluster or segment design instances into classes. The distance from every design instance or class to another design instance or class can be calculated and stored. A behavioral model represents a mapping between a design instance and a set of keywords or semantic descriptions by the task evaluator/user. Discover and explore new parts of the design space by exploring this mapping to find designs that are similar to, and therefore far from (optimized) or different from (search) the most promising designs evaluated. can do. This system can be used with distance-based clustering methods, such as K-nearest neighbor filtering or collaborative filtering, to define design instances for future tasks presented to the user.

Figures 7A and 7B show the data flow when the system implements user feedback to customize existing designs and products.

FIG. 7A shows dataflow 700 when the system implements user feedback to modify existing designs and products in the exemplary environment of FIG. The operations of dataflow 700 may be performed by various components of system 100 . For example, the operations of dataflow 700 may be performed by data quality processor 120 in communication with DCDS 110 , user device 106 , third party 140 , and indication database 130 .

Dataflow 700 can be combined with various manufacturing processes, including 3D printing or automated manufacturing.

Flow 700 closely follows flow 500, 550, 600, or 650. Flow 700 begins with a customer uploading 702 one or more design primitives and/or keywords. Details of this step can be found above in the description of FIG. 5A for (502), FIG. 5B for (552), FIG. 6A for (602), or FIG. 6B for (652).

Flow 700 continues with the system selecting a format for the content to be provided to the user (704). Details of this step can be found above in the description of FIG. 5A for (504), FIG. 5B for (554), FIG. 6A for (604), or FIG. 6B for (654).

Flow 700 continues with the system pre-processing the data and verifying the correctness of the task to be provided to the user (706). Details of this step can be found above in the description of FIG. 5A for (506), FIG. 5B for (556), FIG. 6A for (606), or FIG. 6B for (656).

Flow 700 continues by storing preprocessed data and verified tasks to be provided to the user (708). Details of this step can be found above in the description of FIG. 5A for (508), FIG. 5B for (558), FIG. 6A for (608), or FIG. 6B for (658).

Flow 700 continues with a user of the website or application interacting with the served content (710). Details of this step can be found above in the description of FIG. 5A for (510), FIG. 5B for (560), FIG. 6A for (610), or FIG. 6B for (660).

Flow 700 continues by storing the user's response (712). Details of this step can be found above in the description of FIG. 5A for (512), FIG. 5B for (562), FIG. 6A for (612), or FIG. 6B for (662).

Flow 700 continues by building one or more behavioral models based on the user response data (714). Details of this step can be found above in the description of FIG. 5A for (514), FIG. 5B for (564), FIG. 6A for (614), or FIG. 6B for (664).

Flow 700 continues by analyzing and identifying new ideas and/or concepts based on one or more behavioral models (716). Details of this step can be found above in the description of FIG. 5A for (516), FIG. 5B for (566), FIG. 6A for (616), or FIG. 6B for (666).

Flow 700 continues by exploring and/or optimizing the design to generate new designs and/or semantic descriptors (718). Details of this step can be found above in the description of FIG. 5A for (518), FIG. 5B for (568), FIG. 6A for (618), or FIG. 6B for (668).

Flow 700 includes using the behavioral model to identify an existing product or ensemble of products whose characteristics most closely follow the design identified by the behavioral model (720). In some implementations, model generator 126 may access a database or other searchable structure that stores existing product designs and features, with or without existing keyword descriptions.

Flow 700 continues with fabricating the modified design geometry (722). Details of this step can be found above in the description of FIG. 6A for (622) and FIG. 6B for (672).

Flow 700 continues by determining whether the modified design shape meets release criteria (724). Details of this step can be found above in the description of FIG. 6A for (624) and FIG. 6B for (674).

Flow 700 may optionally include manufacturing and shipping the product with the modified design shape (726). Details of this step can be found above in the description of FIG. 6A for (626) and FIG. 6B for (676).

In one example, the task includes displaying an ensemble of product designs. For example, a product design ensemble may include different product designs for different item types, such as room furnishings (e.g., designs for chairs, tables, beds, accessories, art, etc.) that are both visually appealing to users. Ask the user which of the different product designs and different item types they think would look good to choose. In some implementations, user feedback can be used in subsequent tasks to test existing designs and form new marketing strategies, such as cross-selling and up-selling strategies.

In another example, a task includes how a user designs or configures a product such as a shoe or a car and how a user receives the product or how a manufacturer builds and ships the product to the user. For example, a user can provide input through a feedback mechanism to indicate new designs or designs constructed using a predetermined set of feature values.

In another example, tasks include how a user modifies an existing product design and has a manufacturer provide a custom-tailored product to the user (eg, using a 3D printer). For example, users can provide input through a feedback mechanism to indicate modifications to existing designs.

FIG. 7B shows data flow 750 when the system implements user feedback to customize software design and products in the exemplary environment of FIG. The operations of dataflow 750 may be performed by various components of system 100 . For example, the operations of dataflow 750 may be performed by data quality processor 120 in communication with DCDS 110 , user device 106 , third party 140 , and indication database 130 .

Dataflow 750 uses semantic behavioral models to configure software products to be tailored, for example, to specific users, user segments, or groups. In one example, dataflow 750 is used to create a microgame with characters, weapons, environments, and/or situations, among other features, selected or customized based on user information.

Flow 750 closely follows flow 500, 550, 600, 650, or 700. Flow 750 begins with a customer uploading 752 one or more design primitives and/or semantic descriptors. Details of this step can be found above in the description of FIG. 5A for (502), FIG. 5B for (552), FIG. 6A for (602), FIG. 6B for (652), or FIG. can.

Flow 750 continues with the system selecting a format for the content to be provided to the user (754). Details of this step can be found above in the description of FIG. 5A for (504), FIG. 5B for (554), FIG. 6A for (604), FIG. 6B for (654), or FIG. can.

Flow 750 continues with the system pre-processing the data and verifying the correctness of the task to be provided to the user (756). Details of this step can be found above in the description of FIG. 5A for (506), FIG. 5B for (556), FIG. 6A for (606), FIG. 6B for (656), or FIG. can.

Flow 750 continues by storing preprocessed data and verified tasks to be provided to the user (758). Details of this step can be found above in the description of FIG. 5A for (508), FIG. 5B for (558), FIG. 6A for (608), FIG. 6B for (658), or FIG. can.

Flow 750 continues with a user of the website or application interacting with the served content (760). Details of this step can be found above in the description of FIG. 5A for (510), FIG. 5B for (560), FIG. 6A for (610), FIG. 6B for (660), or FIG. can.

Flow 750 continues by storing the user's response (762). Details of this step can be found above in the description of FIG. 5A for (512), FIG. 5B for (562), FIG. 6A for (612), FIG. 6B for (662), or FIG. can.

Flow 750 continues with building one or more behavioral models based on the user response data (764). Details of this step can be found above in the description of FIG. 5A for (514), FIG. 5B for (564), FIG. 6A for (614), FIG. 6B for (664), or FIG. can.

Flow 750 continues with analyzing and identifying new ideas and/or concepts based on one or more behavioral models (766). Details of this step can be found above in the description of FIG. 5A for (516), FIG. 5B for (566), FIG. 6A for (616), FIG. 6B for (666), or FIG. can.

Flow 750 continues with exploring and/or optimizing the design to generate new designs and/or semantic descriptors (768). Details of this step can be found above in the description of FIG. 5A for (518), FIG. 5B for (568), FIG. 6A for (618), FIG. can.

Flow 750 includes using the behavioral model to configure (770) a software or digital product that will be personalized or tailored to the user's preferences. In some implementations, software products are personalized using configurators. In some implementations, the software product may be a microgame. In some implementations, a software product may be embedded within a web page or application.

For example, if the user information indicates that a particular user enjoys the Star Wars franchise and the software product is a microgame, then the data quality processor 120, and specifically the model generator 126, will receive the data from the user and Lucasfilm. With permission from , you may modify the microgame to include characters, audio, items, etc. from the Star Wars franchise.

In another example, a game developer may use an existing or new game that may have many levels, characters, weapons, and allow users to play small portions online or through downloadable forms through tasks. Allows you to select sections that can be configured with user feedback to enable.

Flow 750 continues by determining whether the revised game design meets release criteria (772). Details of this step can be found above in the description of FIG. 6A for (624), FIG. 6B for (674), or FIG. 7A for (724).

Flow 750 optionally distributes or provides (774) a download option with the modified game design to the user who has interacted with the system and provided feedback, and distributes the modified game design to other users. Including actually delivering (775). Details of this step can be found above in the description of FIG. 6A for (626), FIG. 6B for (676), or FIG. 7A for (726).

In one example, the task includes presenting the user with two different software game characteristics or attributes for the game application, such as window size, and asking which one the user is more interested in. In some examples, the method may include offering pre-configured software with selected characteristics or attributes for the user to download. In another example, the method may include offering software preconfigured with selected characteristics or attributes for the user to play the game online. In another example, one or more people to design new software, such as an app or game, which may have multiple forms with characters, settings, and backgrounds tailored to one or more user demographics. A behavioral model created through task feedback from users. In another example, multiple users may use tasks to collaboratively design game features to be inserted into multiplayer gaming environments to which they may have access.

Flow 750 optionally distributes or provides (776) a download option with the modified game design to users other than the user who interacted with the system and provided feedback, and distributes the modified game design to others. , including actually delivering 777 to users of Implementation details are the same as for (774).

FIG. 7C is an example of how the system implements user feedback into the design cycle as described with respect to FIG. 7B. The system can receive products such as software titles (780). For example, the system may receive games or applications. The system may identify product attributes or assets. For example, the system may determine game attributes, which include characters, weapons, scenes, environments, keywords, and the like. The system may identify game assets including thumbnails, demo versions, multiple configurations, usage/gameplay videos, reviews, descriptions, keywords, hyper-casual versions, and the like. The user may be presented with game attributes (782). For example, a user may be presented with Character A, Character B, Weapon A, and Weapon B. Users can make their own selections and preferences for playing games. The user may be presented with game assets (783). For example, the user may be presented with Thumbnail 1 and Thumbnail 2 and two sliders for requesting user input. The first slider asks "Which game is this sign 'Roleplaying'?" and the user can drag the slider to the side that represents the answer. The second slider asks "Which game is this sign 'puzzle'?"

FIG. 7D is a specific example of how the system implements user feedback within the design cycle as described with respect to FIG. 7B. The user can interact with the task to select settings or configure preferences (784). The user may be presented with task-related UI elements (785). A game or application may be constructed using the flow outlined by (786)-(788). For example, flows may include settings, configurations, or game state descriptions such as characters/skins, inventories/loadouts, settings/environments, missions, etc. (786). These settings and configurations can be encrypted to control usage and restrict licensing to subgames. The engine can then accept the input and use the settings, configurations, or game state descriptions to create a usable and executable environment (787). The game or application may then be a structured/personalized experience (788). A configured game or application may be provided to the user through a task-related UI element (789). The user can then receive or download the configured game or application (790). The user may be the same user who interacted with the task at 784, or it may be a different user.

FIG. 7E is an example of how the system implements user feedback into the design cycle, as described with respect to FIG. 7B. The user can interact 791 with the task to select settings or configure preferences. The user may be presented with task-related UI elements (792). A game or application can be constructed using the flow outlined by (793)-(795). For example, flows may include settings, configurations, or game state descriptions such as characters/skins, inventories/loadouts, settings/environments, missions, etc. (793). These settings and configurations can be encrypted to control usage and restrict licensing to subgames. The engine can then map user settings, configurations, or game state preferences from the library to preconfigured or precompiled games/apps (794). A game or application can then be a structured/personalized experience (795). A configured game or application may be provided to the user through a task-related UI element (796). The user can then receive or download 797 the configured game or application. The user may be the same user who interacted with the task at (791), or it may be a different user.

FIG. 7F is a specific example in which the system uses autoencoders to implement user feedback into design feedback. As explained above, autoencoders can be used with libraries of designs to create a set of latent factors that effectively represent a reduced set of features that describe a given design. The system may access a library (A) of design representations including shapes, images, graphs, data structures, and the like. The system can then generate a reconstructed design (B) using a set of latent factors that effectively represent the set of features that describe the design. The system can automatically create a vehicle design from the original data, from the developed latent factors, or from the improved design features (C). The system can receive results from behavioral models (D) created through user-defined tasks. The system can use behavioral models in the form of semantic characterizations along with user classifications of car designs (eg, sport, family-focused, compact) (E). The system uses a cost function-based scoring method (F) and performs optimization (G) to create new feature vectors for automatically creating design objects. If the optimization converges on a certain set of criteria, the method is complete (H).

FIG. 8 is a flowchart illustrating an exemplary process 800 for data quality improvement. In some implementations, process 800 may be performed by one or more systems. For example, process 800 may be implemented by data quality processor 120, DCDS 110, user device 106, and third party 140 of FIGS. In some implementations, the process 800 may be implemented as instructions stored on a computer-readable medium, which may be non-transitory, and when the instructions are executed by one or more servers, one One or more servers may perform the operations of process 800 .

Process 800 begins by receiving 802 a request from a user device for a digital component for presentation at the user device. For example, system 100 can receive request 108 from user device 106 .

The process 800 continues by determining 804 one or more attributes of the user based on one or more of the information provided by the user or included in the request for the digital component. As discussed above with respect to FIG. 2A, data processor 124 may determine one or more attributes of the user based on information provided by the user or included in the request, such as the user profile. can be done. For example, information provided within a user profile may include the user's age, gender, interests, and location, among other characteristics. The information contained within the request 108 may include anonymized information or information that cannot be used to identify the user, such as the website from which the request 108 was generated and the user navigated to. You may include the destination website you were visiting. In one example, data processor 124 may determine that the user is a male over the age of 66.

The process 800 continues by identifying (806) a behavioral model corresponding to one or more attributes of the user. For example, model generator 126 can identify behavioral models that predict user behavior based on one or more attributes of the user. In one example, model generator 126 may identify a behavioral model for men age 66 and older.

The process 800 continues by dynamically altering 808 the presentation of the item shown by the digital component based on the identified behavioral model corresponding to one or more attributes of the user. For example, task processor 122 and/or data processor 124 may trigger the presentation of items indicated by the digital component, e.g., task questions, based on the identified behavioral model corresponding to one or more attributes of the user. can be modified accordingly. In one example, the task processor 122 may modify the task questions regarding the mug designs presented in the task based on a behavioral model for men age 66 and older.

In some implementations, task processor 122 selects the format in which feedback is requested for digital components requesting feedback from users regarding items. For example, task processor 122 selects a format for user responses to a particular mug design.

In some implementations, the data quality processor 120 uses one of the users, including using machine learning or artificial intelligence techniques, to identify feedback to be requested from the user regarding the item. or dynamically altering the presentation of the item shown by the digital component based on an identified behavioral model that corresponds to multiple attributes. For example, data quality processor 120 may use the output of model generator 126 to identify feedback that will be solicited from users regarding the mag design.

In some implementations, task processor 122 validates the information requested by the digital component based on certain attributes corresponding to underrepresented segments of the user population. For example, task processor 122 verifies information requested by tasks to be delivered based on user age attributes.

The process 800 continues by determining 810 that the user has certain attributes corresponding to underrepresented segments of the user population in the database that contain information about the item. For example, data processor 124 may determine that the user segment of males age 66 and older is an underrepresented segment based on the threshold number of responses. In some implementations, data processor 124 uses statistical analysis to identify underrepresented segments of the user population. Data processor 124 may then determine that the user of user device 106 has an age attribute that corresponds to an underrepresented segment of the user population in the database containing information about mug designs.

In response to a determination that a user has a particular attribute that corresponds to an underrepresented segment of the user population, process 800 responds to the request with a dynamically altered presentation of the item. It continues by generating 812 a digital component that includes a feedback mechanism that includes and solicits feedback from the user regarding the item and allows the user to submit feedback regarding the item. For example, the task processor 122 solicits feedback from the user of the user device 106 regarding the mug design and determines whether the item has been modified, including a feedback mechanism such as a poll function that allows the user to submit feedback regarding the item. can generate modified presentations or modified tasks. In some implementations, the DCDS 110 generates or selects digital components to be delivered to the user device 106, as described above with respect to FIGS. 1-7.

The process 800 continues by updating 814 the database to include the feedback obtained from the user regarding the item. For example, data processor 124 may update a database containing information about the item with feedback from the user of user device 106 .

In some implementations, in response to receiving feedback from users with particular attributes corresponding to underrepresented segments of the user population, data quality processor 120 performs one or more The feedback information is labeled with attributes, and the labeled feedback information is stored in a labeled searchable database, such as labeled database 130 .

The process 800 continues by modifying (816) the presentation of the item when delivered to other users having one or more attributes of the user based at least in part on the feedback obtained from the user. . For example, data quality processor 120 may modify the presentation of the item when delivered to other users sharing one or more attributes of the user based on feedback obtained from the user. This allows the data quality processor 120 to adapt its tasks and product designs based on attributes of users interacting with the system.

In some implementations, the data quality processor 120 determines the quality of the item when delivered to other users having one or more attributes of the user by selecting a particular feedback mechanism included by the digital component. You can modify the presentation.

FIG. 9 is a flowchart illustrating an exemplary process 900 for automatic design space search. In some implementations, process 900 may be performed by one or more systems. For example, process 900 may be implemented by data quality processor 120, DCDS 110, user device 106, and third party 140 of FIGS. In some implementations, the process 900 may be implemented as instructions stored on a computer-readable medium, which may be non-transitory, and when the instructions are executed by one or more servers, one One or more servers may perform the operations of process 900 .

Process 900 begins by receiving 902 a request from a user device for a digital component for presentation at the user device. For example, as described above, system 100, and specifically DCDS 110, may receive request 108 for digital components for presentation on user device 106. FIG.

The process 900 continues by receiving 904 a dataset of user-provided information regarding the particular product design. For example, data quality processor 120 may receive user-provided responses regarding a particular product design, such as a handbag design. In some implementations, the product design is related to a specific product. Product design may relate to services or software products. For example, a particular product design may be a user interface design for a software application.

The process 900 continues by generating 906 a visual representation mapping the design coefficients to a continuous shape representing the potential product design geometry based on the data set of user-provided information. As described above with respect to FIGS. 3-4, model generator 126 can generate a design space that maps subjective coefficients to continuous shapes representing potential product designs. For example, model generator 126 may generate a design space that maps semantic coefficients, such as descriptors, to continuous shapes that represent the universe of possible designs.

The design space is based on a data set of user-provided information to generate a visual representation that maps design coefficients to continuous shapes representing potential product design geometries. It can be reversible to include generating a representation.

The process 900 continues by segmenting 908 the visual representation into multiple segments based on the design factor values. As described above with respect to FIGS. 3-4, the model generator 126 may segment the design space into groups based on the values of the subjectivity coefficients.

In some implementations, segmenting the visual representation into a plurality of segments based on the design factor values includes design factor values such that each segment of the visual representation shares a design factor value within a defined range of values. It involves dividing the visual representation into multiple segments based on numerical values. For example, data processor 124 may segment the design space into multiple segments based on design factor value ranges, such as subjective assessments of how easily handbag designs are perceived.

The process 900 continues by selecting 910 segments of the visual representation that contain less than the threshold amount of data points. As described above with respect to FIGS. 3-4, data processor 124 can identify segments of data according to a metric such as a threshold number of data points. For example, data processor 124 may identify segments of bags that are both very "stylish" and very "utilitarian" as having less than a threshold amount of data points or other metrics.

Process 900 continues by selecting 912 a digital component for which information is requested from the user. For example, task processor 122 or DCDS 110 can select the digital components to solicit information from the user. Task processor 122 may select or generate a task for a user requesting information from the user.

In some implementations, task processor 122 selects the format in which feedback is requested for the digital component requesting information from the user. For example, task processor 122 selects a format for user responses to handbag designs.

In some implementations, selecting the format in which the information is requested includes selecting a particular feedback mechanism for which the dynamically modified digital component will be provided.

In some implementations, task processor 122 validates the information requested by the digital component based on certain attributes corresponding to underrepresented segments of the user population. For example, task processor 122 verifies information requested by tasks to be delivered based on user age attributes.

The processor 900 dynamically alters the presentation of the digital component requesting information from the user regarding segments of the visual representation containing less than the threshold amount of data points based on the selected segment of the visual representation ( 914). For example, as described above with respect to FIGS. 3-4, task processor 122 can dynamically modify existing tasks or create new tasks. In one example, task processor 122 may modify an existing task to present the user with a new product design that has not been previously generated or presented to the user.

In some implementations, dynamically altering the presentation of the content item includes using machine learning or artificial intelligence techniques to specify information to be requested by the digital component. For example, data quality processor 120 may use machine learning models generated by model generator 126 to determine and specify the information that will be required by the task.

In some implementations, dynamically altering the presentation of the digital component is based on the user's interest in a particular product design based on a request for the digital component to be presented at the user device. Based on determining that the user of the user device is within the first cluster of users interested in a particular product design, user interface elements of the digital component are selected based on determining that they are within the first cluster. including identifying and altering user interface elements of the presentation of the digital component. For example, data processor 124 determines that the user of user device 106 is within a cluster of interested users, identifies user interface elements for the task, and customizes the task for users already interested in handbag designs. User interface elements can be modified for

In some implementations, dynamically altering the presentation of the digital component is based on the user's interest in a particular product design based on a request for the digital component to be presented at the user device. determining to be within a first cluster, wherein the request for the digital component for presentation at the user device indicates one or more attributes of the user based on information provided by the user; and identifying a user interface element of the digital component based on determining that the user of the user device is within a first cluster of users interested in a particular product design; and the user of the presentation of the digital component. and modifying the interface elements.

In some implementations, process 900 includes building behavioral models that predict user reception of potential product design geometries based on feedback information. For example, model generator 126 may generate behavioral models that predict user reception of handbag product designs. Modification of design factors for a particular product design is based at least in part on a behavioral model.

Process 900 continues by distributing (916) the dynamically altered digital component for presentation at the user device. For example, DCDS 110 may deliver the task and any requested content as response 114 to user device 106 .

The process 900 continues by obtaining 918 feedback information from the user device via a feedback mechanism regarding the segment of the visual representation containing less than the threshold amount of data points. For example, DCDS 110 can receive response data 116 from user device 106 and provide it to data processor 124 . As described above with respect to FIGS. 3-4, the data quality processor 120 can receive feedback information from the user regarding particular segments of the visual representation. For example, data processor 124 and DCDS 110 may receive feedback information from user device 106 regarding segments of the design space that have less than a threshold amount of data points.

In some implementations, the request for the digital component for presentation at the user device indicates demographic information for the user of the user device. Process 900 further includes determining that the user of the user device is within a first group of users, such as a group of users who are female in California, based on the request for the digital component to be presented at the user device. obtain.

In some implementations, process 900 includes receiving from a second user device a request for a digital component for presentation at the second user device that indicates user demographic information of the user of the second user device. It can contain more. System 100 (eg, data processor 124) then selects a first user group in which the user of the second user device is the same as the user of the user device based on the request for the digital component to present at the second user device. may include determining that it is within the For example, system 100 (eg, data processor 124) may determine that the user of the second user device is a California female. In response to determining that the user of the second user device is within the same first group of users as the user of the user device, system 100 may provide a modified product design rather than the specific product design. can. For example, due to the similarity between the user of the first user device and the user of the second user device, task processor 122 may select the modified handbag design for the user of the second user device instead of the original handbag design. can be provided to

The process 900 continues by modifying 920 design factors for the particular product design based at least in part on the feedback information obtained from the user to create the modified product design. For example, the model generator 126 may modify the design factors of the handbag design based at least in part on feedback information from the user for creating the modified handbag design.

As described above with respect to FIGS. 3-4, the model generator 126 can update the design space and/or the behavioral model. For example, the model generator 126 may update the dataset by providing feedback information as input to the training system for the behavioral model or design generator.

In some implementations, based on the modified product design, process 900 calculates from the plurality of existing product designs the closest existing product design that has the largest number of design factor values in common with the modified product design. Including identifying the design. For example, data quality processor 120 may identify existing products that most closely follow the revised design. The system 100 can, for example, modify existing products and their manufacturing methods instead of creating entirely new products. In some implementations, the system 100 can provide modified product designs to embedded manufacturing systems. For example, data quality processor 120 may provide revised product designs to a 3D printing system or automated manufacturing system for immediate fabrication.

FIG. 10 is a block diagram of an exemplary computer system 1000 that can be used to perform the operations described above. System 1000 includes processor 1010 , memory 1020 , storage devices 1030 and input/output devices 1040 . Each of components 1010, 1020, 1030, and 1040 may be interconnected using system bus 1050, for example. Processor 1010 is capable of processing instructions for execution within system 1000 . In one implementation, processor 1010 is a single-threaded processor. In another implementation, processor 1010 is a multithreaded processor. Processor 1010 can process instructions stored in memory 1020 or on storage device 1030 .

Memory 1020 stores information within system 1000 . In one implementation, memory 1020 is a computer-readable medium. In one implementation, memory 1020 is a volatile memory unit. In another implementation, memory 1020 is a non-volatile memory unit.

Storage device 1030 may provide mass storage for system 1000 . In one implementation, storage device 1030 is a computer-readable medium. In various different implementations, storage device 1030 is, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (eg, cloud storage device), or some other mass storage device. can include

Input/output devices 1040 provide input/output operations for system 1000 . In one implementation, input/output device 1040 may include one or more network interface devices such as Ethernet cards, serial communication devices such as RS-232 ports, and/or wireless interface devices such as 802.11 cards. . In another implementation, input/output devices may include driver devices configured to receive input data and send output data to other input/output devices, such as keyboards, printers, and display devices 1060 . However, other implementations may also be used, such as mobile computing devices, mobile communications devices, set-top box television client devices, and the like.

Although an exemplary processing system is illustrated in FIG. 10, implementations of the subject matter and functional operations described herein may be implemented in other types of digital electronic circuits or in the structures and It can be implemented in computer software, firmware, or hardware, including structural equivalents thereof, or in a combination of one or more thereof.

An electronic document (referred to simply as a document for brevity) does not necessarily correspond to a file. A document may be stored in part of a file that holds other documents, in a single file dedicated to that document, or in multiple collaborative files.

Embodiments of the subject matter and operations described herein may be in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents. can be implemented in one or more combinations of Embodiments of the subject matter described herein may be one or more encoded on one or more computer storage media for execution by or for controlling operation of a data processing apparatus. A plurality of computer programs may be implemented as one or more modules of computer program instructions. Alternatively or additionally, the program instructions may be transmitted to an artificially generated propagated signal, e.g. , may be encoded on a machine-generated electrical, optical, or electromagnetic signal. The computer storage medium may be or be contained within a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more thereof. Moreover, although a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be or be contained within one or more separate physical components or media (eg, multiple CDs, discs, or other storage devices).

The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

The term "data processor" means any kind of apparatus, device, and for processing data including, by way of example, programmable processors, computers, systems-on-chips, or any number or combination of the above. Including machinery. The device may include dedicated logic circuitry, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits). The apparatus includes, in addition to hardware, code that creates an execution environment for the computer program in question, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or among them. can also include code that constitutes a combination of one or more of Devices and execution environments can implement a variety of different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages. , as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be either in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), or in a single file dedicated to that program. or in multiple collaboration files (eg, files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described herein are performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. can be The processes and logic flows may also be performed by dedicated logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and the apparatus may be implemented as such.

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors. Generally, a processor receives instructions and data from read-only memory, random-access memory, or both. The essential elements of a computer are a processor, for performing actions according to instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes or receives data from or stores one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data. , or both. However, a computer need not have such devices. Additionally, a computer may be used in another device such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver, or portable storage, to name just a few. It can be embedded within a device (eg, a Universal Serial Bus (USB) flash drive). Suitable devices for storing computer program instructions and data include, by way of example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices, magnetic disks such as internal or removable disks, magneto-optical disks and CDs. - includes all forms of non-volatile memory, non-volatile media, and non-volatile memory devices, including ROM discs and DVD-ROM discs. The processor and memory may be augmented by, or embedded within, dedicated logic circuitry.

To provide interaction with a user, embodiments of the subject matter described herein include a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to a user, and a can be implemented on a computer that has a keyboard and pointing device, such as a mouse or trackball, that allows a user to provide input to the computer. Other types of devices may also be used to provide interaction with the user, e.g., the feedback provided to the user may be any form of sensory feedback, e.g., visual, auditory, or tactile feedback. Input from the user may be received in any form, including acoustic, speech, or tactile input. In addition, a computer can send documents to and receive documents from a device used by a user to, for example, render web pages in response to requests received from a web browser on the user's client device. You can interact with the user by sending to their web browser.

Embodiments of the subject matter described herein may include back-end components, such as data servers, or middleware components, such as application servers, or front-end components, such as user includes a client computer having a graphical user interface or web browser through which an implementation of the subject matter described herein can be interacted, or one or more such backend components, middleware components, or It can be implemented in a computing system that includes any combination of front-end components. The components of the system may be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include local area networks (“LAN”) and wide area networks (“WAN”), internetworks (eg, the Internet), and peer-to-peer networks (eg, ad-hoc peer-to-peer networks).

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, the server sends data (eg, HTML pages) to the client device (eg, to display data to and receive user input from a user interacting with the client device). Data generated at the client device (eg, results of user interactions) may be received from the client device at the server.

Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, rather than the specific features of the particular embodiment of the particular invention. should be interpreted as an explanation. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may have been described above as operating in some combination, and may even have been originally claimed as such, one or more features from the claimed combination may may optionally be omitted from the combination, and claimed combinations may relate to subcombinations or variations of subcombinations.

Similarly, although operations are shown in the figures in a particular order, this does not mean that such operations are performed in the specific order shown, or in any order shown, or all illustrated to achieve a desired result. should not be understood to require that the actions of In some situations, multitasking and parallel processing can be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems generally It should be understood that they may be incorporated together in a single software product or packaged in multiple software products.

Particular embodiments of the present subject matter have been described above. Other embodiments are within the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. Additionally, the processes illustrated in the accompanying figures do not necessarily require the particular order or order shown to achieve desirable results. Multitasking and parallel processing may be advantageous in some implementations.

100 environment, system
102 network
104 Electronic Document Server, Electric Doc Server
105 Electronic Doc
106 user devices
108 request, request data
110 Digital Component Distribution System, DCDS
112 Digital Component Database
114 reply data, reply
116 Response data
120 data quality processor
122 task processor
124 data processor
126 Model Generator
130 Labeling Database, Comprehensive Database
140 third party, third party bag designer
200 data flow, process
250 servers
252 baseline model
254 Task Repository
256 model training modules
258 Updated baseline model
260 devices
260a Device A
260b Device B
260n device N
262 models
262a Model A
262b Model B
262n Model N
270 users
270a User
270b User
270n users
272 Semantic Mapping
272a Semantic mapping
272b Semantic mapping
272n semantic mapping
300 design space
302 bag design
304 bag design
306 User response data
308 User response data
312 bag design, design
314 bag design, design
400 data flow, process
500 data flow, flow
600 data flow, flow
700 data flow, flow
800 processes
900 processes
1000 computer system, system
1010 processor, components
1020 memory, components
1030 storage devices, components
1040 Input/Output Devices, Components
1050 system bus

Claims (21)

A method of improving robustness of a data set executed by one or more data processing apparatus, comprising:
receiving a request from a user device for a digital component for display on the user device;
determining one or more attributes of the user based on one or more of information provided by the user or included in the request for the digital component;
identifying a behavioral model corresponding to said one or more attributes of said user;
dynamically altering the provision of items represented by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user;
determining that the user has a particular attribute corresponding to an underrepresented segment of the user population in a database containing information about the item;
In response to determining that the user has the specified attribute corresponding to the underrepresented segment of the user population;
Feedback comprising the dynamically modified provision of the item, in response to the request, soliciting feedback from the user regarding the item, and enabling the user to submit feedback regarding the item. generating a digital component that includes the mechanism;
updating the database to include the feedback obtained from the user regarding the item;
and modifying the provision of the item when delivered to other users having the one or more attributes of the user based at least in part on the feedback obtained from the user. ,Method. transmitting the feedback information in the one or more attributes of the user in response to receiving the feedback from the user having the particular attribute corresponding to the underrepresented segment of the user population; marking;
2. The method of claim 1, further comprising: storing the indicated feedback information in an indicated searchable database. The step of determining that the user has a particular attribute corresponding to an underrepresented segment of the user population in a database is for identifying the underrepresented segment of the user population. 3. The method of claim 1 or 2, comprising using statistical analysis. said modifying said provision of said item when delivered to other users having said one or more attributes of said user based at least in part on said feedback obtained from said user; 4. A method according to any preceding claim, comprising selecting a particular feedback mechanism included by said digital component. The step of dynamically altering a provision of an item represented by the digital component based on the identified behavioral model responsive to the one or more attributes of the user includes: 5. The method of any one of claims 1-4, comprising using machine learning or artificial intelligence techniques to identify feedback to be solicited from. wherein said determining one or more attributes of said user is based on information provided by said user;
provided by the user to dynamically modify the provision of the item represented by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user. 6. A method according to any one of claims 1 to 5, comprising updating said identified behavioral model based on said information obtained. wherein said determining one or more attributes of said user is based on information contained within said request for said digital component;
dynamically altering the provision of items represented by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user; 7. The method of any one of claims 1-6, comprising updating the identified behavioral model based on the information contained in the request. for said digital component requesting feedback from said user regarding said item, selecting a format in which said feedback is solicited;
and verifying the information requested by the digital component based on the specified attributes corresponding to the underrepresented segments of the user population. The method according to item 1. The digital component displays two different product design shapes for a suit and footwear, and the feedback obtained from the user regarding the item indicates the two different footwear that the user believes will look better on the suit. 9. The method of any one of claims 1-8, comprising selecting one of the product design shapes. said digital component displaying two different product design shapes for a car,
the digital component specifies a particular subjective product design shape descriptor;
the feedback mechanism is a slider,
A selection of one of said two different product design shapes relating to automobiles, wherein said feedback obtained from said user regarding said item, said user believes can be better described by said particular subjective product design shape descriptor. 10. A method according to any one of claims 1 to 9, comprising said digital component displaying three or more different product design shapes for three or more different item types;
wherein the feedback obtained from the user regarding the item includes a selection of two or more different product design shapes that the user believes are visually harmonious;
11. The method of any one of claims 1-10, wherein the method further comprises using the feedback obtained from the user in a separate model. the digital component requests the user to create a new product design;
the feedback mechanism receives user input indicative of a new product design;
12. The method of any one of claims 1-11, wherein the method further comprises providing the user with a product having the new product design. the digital component requests the user to modify an existing product design to produce a customized product design;
the feedback mechanism receives user input to modify one or more aspects of the existing product design;
13. The method of any one of claims 1-12, wherein the method further comprises providing the user with a product having the customized product design. said digital component displaying two different software attributes,
wherein the feedback obtained from the user regarding the item comprises a selection of one of the two different software attributes that the user prefers;
14. A method according to any preceding claim, wherein the method comprises providing the user with a software package having the selected one of the two different software attributes that the user prefers. the method of. a system,
one or more processors;
and one or more memory elements containing instructions that, when executed, cause the one or more processors to perform an operation, the operation comprising:
receiving a request from a user device for a digital component for display on the user device;
determining one or more attributes of the user based on one or more of information provided by the user or included in the request for the digital component;
identifying a behavioral model corresponding to the one or more attributes of the user;
dynamically altering the provision of items represented by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user;
determining that the user has a particular attribute corresponding to an underrepresented segment of the user population in a database containing information about the item;
In response to determining that the user has the specified attribute corresponding to the underrepresented segment of the user population;
Feedback comprising the dynamically modified provision of the item, in response to the request, soliciting feedback from the user regarding the item, and enabling the user to submit feedback regarding the item. generating a digital component that includes the mechanism;
updating the database to include the feedback obtained from the user regarding the item;
and modifying the provision of the item when delivered to other users having the one or more attributes of the user based at least in part on the feedback obtained from the user. ,system. the operation is
transmitting the feedback information in the one or more attributes of the user in response to receiving the feedback from the user having the particular attribute corresponding to the underrepresented segment of the user population; to mark and
16. The system of claim 15, further comprising: storing the indicated feedback information in an indicated searchable database. said determining that said user has a particular attribute corresponding to an underrepresented segment of a user population in a database for identifying said underrepresented segment of said user population; 17. A system according to claim 15 or 16, comprising using statistical analysis. said modifying said provision of said item when delivered to other users having said one or more attributes of said user based at least in part on said feedback obtained from said user; 18. The system of any one of claims 15-17, comprising selecting a particular feedback mechanism included by the digital component. Based on the identified behavioral model corresponding to the one or more attributes of the user, the dynamically altering the provision of the item represented by the digital component causes the user to 19. The system of any one of claims 15-18, comprising using machine learning or artificial intelligence techniques to identify feedback to be solicited from. A computer storage medium encoded with instructions that, when executed by a distributed computing system, cause the distributed computing system to perform an action, the action comprising:
receiving a request from a user device for a digital component for display on the user device;
determining one or more attributes of the user based on one or more of information provided by the user or included in the request for the digital component;
identifying a behavioral model corresponding to the one or more attributes of the user;
dynamically altering the provision of items represented by the digital component based on the identified behavioral model corresponding to the one or more attributes of the user;
determining that the user has a particular attribute corresponding to an underrepresented segment of the user population in a database containing information about the item;
In response to determining that the user has the specified attribute corresponding to the underrepresented segment of the user population;
Feedback comprising the dynamically modified provision of the item, in response to the request, soliciting feedback from the user regarding the item, and enabling the user to submit feedback regarding the item. generating a digital component that includes the mechanism;
updating the database to include the feedback obtained from the user regarding the item;
and modifying the provision of the item when delivered to other users having the one or more attributes of the user based at least in part on the feedback obtained from the user. , computer storage media. A computer storage medium comprising instructions that, when executed by a distributed computing system, cause the distributed computing system to perform the method of any one of claims 2-14.

Images