KR20070007964A - System and method for rating documents comprising an image - Google Patents

Abstract

A system and method for rating a document comprising an image are provided. A document is received for distribution. Rating information associated with the document is received from one or more rating entities. At least one of said one or more rating entities comprises a processor to determine rating information associated with the image. The document is approved for distribution based on the rating information. ® KIPO & WIPO 2007

Description

SYSTEM AND METHOD FOR RATING DOCUMENTS COMPRISING AN IMAGE}

The present invention relates to a system and method for evaluating documents such as image advertisements.

With the advent of the Internet, seemingly diverse text and graphic content such as advertisements can be provided to people of various ages, preferences, and sensitivities. Some images and other content may be more appropriate for some individuals and groups than others. For example, violent or adult images are generally considered inappropriate for children. It is desirable to provide appropriate content to the audience because it generally maximizes benefits and minimizes the burden on both the content provider and the user. For example, advertisements that are appropriate and relevant to the user are more likely to bring sales than those that are bothersome and irrelevant.

Thus, advertisements and other content often target specific users who may be interested in that content. For example, an advertisement directed at men may be displayed during a football game nationally televised because the game attracts a large number of male viewers. Similarly, advertisements for flights may be displayed on an Internet search engine site when a user sends a query that includes the word "airplane."

However, providing advertisements and other content based on user related content does not guarantee that the text, images and other content of the document are appropriate for a particular user. For example, although the content of a web site is related by topic, beer advertisements may be inappropriate on a web site to recover alcoholics.

One way to tailor the appropriate image and other content to the user is to manually review the content before distributing it to the user. However, there are problems associated with manual review. For example, one major drawback of manual evaluation systems and methods is the time and cost involved in reviewing content.

These and other disadvantages exist in existing systems and methods.

Accordingly, various embodiments of the present invention may relate to systems and methods for evaluating and / or approving documents such as image advertisements. In one exemplary embodiment, a method of evaluating a document including an image is provided. Documents for distribution are received. Evaluation information associated with the document is received from one or more evaluation entities. At least one of the one or more evaluation entities includes a processor for determining evaluation information associated with the image. The document for distribution is approved based on the evaluation information.

Another example embodiment includes a computer-readable medium encoded with computer program code for evaluating an electronic document. Wherein the program code is further configured to: receive evaluation information associated with the document from one or more evaluation entities, wherein at least one of the one or more evaluation entities includes a processor to determine evaluation information associated with the image; To determine whether to approve the document.

In another exemplary embodiment, a system for evaluating a document is provided. The input device receives evaluation information associated with a document from one or more evaluation entities, wherein at least one of the one or more evaluation entities includes an image processor. The image processor determines evaluation information associated with the image. The processor determines whether to approve the document based on the evaluation information.

In another exemplary embodiment, a system for evaluating a document is provided. The evaluation receiving means receives evaluation information associated with the document from one or more evaluation entities, and at least one of the one or more evaluation entities comprises an image processor. The image processing means determines evaluation information associated with the image. The judging means judges whether to approve the document based on the evaluation information.

In another exemplary embodiment, a method of evaluating a document is provided. The signal is sent to an entity such as a server. The entity is configured to interpret the signal as being associated with a reference to a document. This entity is also configured to determine whether a document containing an image should be delivered. The entity is configured to make this determination based in part on evaluation information received from a plurality of evaluators for one or more documents and in part based on whether the one or more documents are related to the criteria. At least one of the one or more documents includes an image. The plurality of evaluators includes an image processor configured to process an image and evaluate a document. A document containing an image is received in response to the signal.

In another exemplary embodiment, a method of evaluating a document is provided. One or more documents are received from a system such as a server system, wherein at least one of the one or more documents includes an image, and the system includes an image processor that processes the image and evaluates the document. Evaluation information is provided to the server system for use by the system in determining whether to deliver one or more documents in response to a request for a document, and the determining step is based on the provided evaluation.

In another example embodiment, a method of evaluating a particular document comprising an image is provided. The document is distributed to one or more evaluation entities, wherein at least one of the one or more evaluation entities includes a processor for determining evaluation information by processing at least one of a file storing the image and optical data of the image. Evaluation information associated with the particular document is received from the one or more evaluation entities. The particular document is associated with one or more specific criteria. It is determined whether to approve the specific document based on the evaluation information. A request for a document relating to the criteria is received. A document for distribution is selected from a database containing the specific document, and the selecting step is based on the criteria and the evaluation information.

In another exemplary embodiment, a method of evaluating a document is provided. A first content rating associated with a first advertisement that includes an image is received, the first content rating being received from an image processor configured to determine the first content rating by processing the image. One or more second content ratings associated with the first advertisement are received from one or more raters, wherein at least one of the one or more raters is an end user. A comprehensive evaluation of the first advertisement is determined based on the evaluation information. A request for electronic delivery of one or more advertisements is received, which request is associated with a concept. An advertisement is selected from a database comprising the first advertisement and one or more second advertisements, each of the one or more second advertisements comprising an image and a comprehensive rating. The selected advertisement is delivered electronically based on the request.

In another exemplary embodiment, a method of approving an electronic document is provided. An electronic document for distribution is processed, and the electronic document includes a feedback mechanism that allows a recipient to evaluate the electronic document. A plurality of requests for the document are received and each request is received from one of the plurality of end users. The electronic document is delivered to the plurality of end users based on the corresponding plurality of requests. Based on delivering the electronic document to a predetermined number of end users of the plurality of end users, the electronic document for distribution is approved without receiving feedback above a threshold from the end users.

Other embodiments are also within the scope of the present invention.

1 is a diagram illustrating a system for evaluating an image, according to an exemplary embodiment.

2 illustrates a networked environment for the operation of a system for evaluating an image, in accordance with an embodiment of the present invention.

3 is a flowchart illustrating an exemplary method of evaluating an image, in accordance with an embodiment of the present invention.

4 is a diagram illustrating an exemplary document according to an embodiment of the present invention.

5 is a diagram illustrating an exemplary image according to an embodiment of the present invention.

6 is a diagram illustrating an exemplary document evaluation in accordance with one embodiment of the present invention.

7 is a diagram illustrating an exemplary content evaluation request according to an embodiment of the present invention.

8 is a diagram illustrating an exemplary content evaluation according to an embodiment of the present invention.

Embodiments described herein solve many problems in existing systems and methods. One problem facing Internet content providers is evaluating a large number of documents (such as images and / or advertisements provided through their services) to determine whether each document is appropriate for a wide range of different users. Embodiments described herein overcome these and other problems by obtaining evaluation information from various evaluation entities, including humans and / or computer processors. The processor may automatically determine the assessment information, but a person may manually review the document to provide the assessment information. As used herein, the term “automatically” refers to the operation being performed by any machine-executable process, for example, a process that does not require human intervention or input. Each evaluator may, for example, review a small number of documents, but each evaluator may, for example, review a small number of documents. In sum, a plurality of assessments can be collected and a reliable aggregated assessment can be determined for each document.

Another problem faced by Internet content providers is the selection of appropriate content each time a document is requested by different users with different sensitivities and preferences. Embodiments described herein overcome these problems by determining the generally applicable aggregated content rating for each electronic document. Rather than reevaluating a document each time a document can be provided to a user, the content provider can only review or process the aggregated content rating of the document to match the appropriate content with the user or provider. Each time a document is distributed, the suitability of the document for a particular user can be determined efficiently with reference to content ratings and user suitability data.

Another problem is that content providers may, in some circumstances, have more obligations (e.g., legal obligations) to proper content distribution when the content provider knows the content that is offensive. In an example embodiment, the content provider relies on evaluation (s) by others and thus does not have to assume much obligation. Rather, the server delegates at least partly a review of its content to evaluators (eg, third parties such as partner advertisers or others). Because content providers take proactive steps to ensure that content is appropriate for end users, content providers can be thought of as meeting general attention obligations in preventing inappropriate distribution of content that is annoying.

As used herein, the terms “document” and “electronic document” refer to one or more advertisements, content pages (eg, web pages), search results, emails, applications, IM messages, audio content or files, video content. Or files, other files, other data or applications that may reside on one or several (eg, network) computer systems, or other definable concepts or content. While "image" or "image advertisement" is often used herein as an exemplary document, it will be appreciated that any document may be used.

This application is related to US patent application (agent document number 64557.000020) entitled “System and Methods for Rating Electronic Documents” filed March 30, 2004. , Which is hereby incorporated by reference in its entirety. The present application also discloses a U.S. patent application entitled "Method and System For Approving Documents Based On Image Similarity" entitled Agent Document No. 64557.000027. US Patent Application entitled "Method and System For Providing Targeted Documents Based On Concepts Automatically Indentified Therein" Agent document number 64557.000025), both of which are incorporated herein by reference in their entirety.

Overview and system description

According to one exemplary embodiment shown in FIGS. 1 and 2, one or more document sources 12 provide one or more documents to the server 2. Although a single server 2 is shown, multiple servers 2 may be provided and such multiple servers may share data and computational tasks to achieve the efficiency and operation of the functions described herein. You will know well. The server 2 may select one or more documents (such as image advertisements) and one or more evaluators 4 evaluating the documents. The server 2 may deliver one or more documents to one or more selected evaluators 4 or otherwise make the documents available to one or more evaluators 4 for evaluation. For example, one or more document sources 12, providers 8 and end users 10 may also evaluate documents, and to this extent they may also be considered to be "evaluators 4". The evaluator 4 evaluates the document (s) according to one or more criteria, such as sexual content and violent content. The evaluator 4 transmits its evaluation information to the server 2. The server 2 may approve, disapprove, or otherwise evaluate the document based on the received (or inferred) evaluation information about the document. The server 2 may determine a comprehensive evaluation of the document by processing all or part (one or more) of the relevant evaluator evaluations of the document. For example, if a document is evaluated by 10 evaluators, the overall assessment may be based on any of 1-10 evaluators. For example, the overall rating may be based on six intermediate or any other combination of ratings.

One or more documents may be requested from the server 2 or the server 2 may transmit the documents. For example, the server 2 may transmit a document in response to a signal without human intervention. This operation may be based on a number of factors. For example, provider 8 and / or end user 10 may request a document. This request may come in the form of a signal that the server interprets as a request for a document, but the signal does not specifically indicate that the document is being requested. For example, an end user may send a signal to the server requesting a web page, which may also be interpreted as a request for an image advertising document. The server 2 may randomly respond to the signal based on the evaluation of one or more potential documents and / or one or more criteria (eg, whether the document satisfies or is associated with content associated with the signal). It may be determined whether the document of is available. Server 2 may deliver one or more documents to a recipient, such as the requesting entity. The server 2 may identify other criteria (and / or recipients of the document, if different from the requestor) associated with the signal. While the criteria may be based on the content of a signal or request for a document (eg, a search request for a keyword, etc.), the criteria may also be based on a preference associated with the recipient. For example, server 2 may identify susceptibility information and / or suitability information associated with the requestor or receiver. If the recipient is identified with a preference against violent content, the server 2 may determine whether any available document has a rating that indicates low or minimal violent content.

1 illustrates an example system 100 for evaluating an electronic document, in accordance with an embodiment of the present invention. The system comprises a server (2), one or more evaluators (4), one or more providers (8), one or more document sources (12), an optical processor (6), an image file processor (7), one or more end users (10) And one or more databases 39 that operate in connection with the server 2.

System 100 may enable server 2 to request, receive, and / or process content evaluation of an electronic document. The document source 12, the evaluator 4, the provider 8, and the end user 10 can communicate with one or more servers 2 via electronic communications, including Internet communications. Document source 12, evaluator 4, provider 8, and end user 10 may include or access one or more servers 2 to provide functionality associated with an electronic document.

Information that can be transferred between the server 2, the evaluator 4, the provider 8, the document source 12, and the end user 10 is as follows: document information, content evaluation information, sensitivity evaluation information, suitability. And may include one or more of standard information, confidence score information, and / or evaluator or provider information. Document information may include the following: the document itself, any language (s) used in the document, length information, file type information (eg html, doc, zip, etc.), type of document (advertising, educational) Documents), summary information, audio-video content (eg, facial photos, song lyrics, etc.), adult content, other offensive content (eg, potentially offensive use), age-related content, documents The identity of the owner and / or the creator of the document, information about the intended user of the document (region, age range, gender, race, nationality, religion, other demographic information, etc.), and related to the document or server (2), evaluator (4), the provider 8, or any other information relating to the document source 12.

Specifically, the evaluator 4, the provider 8, the document source 12, the end user 10, the optical processor 6, and the image file processor 7 (collectively and individually, the "evaluation entity" Provide document evaluation information for one or more documents to the server 2. For example, the evaluation entity may receive a document, such as an image advertisement, from a server 2 (or provider 8) and then evaluate information about the document (and / or other documents that refer to or are linked to this document). Can be provided to the server 2.

The document source 12 may provide a document to the server 2 or the server 2 may “fetch” or retrieve a document from the document source 12. For example, document source 12 may provide an advertisement to server 2, whereby server 2 may then provide an advertisement to one or more content providers 8, which content provider 8 may provide. May then provide this advertisement to one or more end users 10 (or the server 2 may provide the advertisement directly to the end user 10). The document source 12 may include any content producer or content provider 8, such as an advertisement listing provider or server 2.

The provider 8 may provide a document to one or more end users 10a to 10n. Providers 8 and evaluators 4 may include content providers, search engines, or other entities that make information, services, and / or products available through electronic networks such as the Internet. The provider 8 may be, for example, an ad listing provider, an electronic document provider, a website host, a server 2, any other entity that provides an electronic document to a user or other entity, or any other content provider. It may include one or more of. The provider 8 may be an evaluator 4 or a document provider 12.

Optical processor 6 may include one or more computers or data processors with optical sensors. An optical sensor of the optical processor 6 may be equipped to identify and / or read optical data from an image of the document itself (eg from a photocopy of the image advertisement), which optical sensor (eg , Without human intervention). In other words, the optical processor 6 may “read” and process image information of a document in a manner similar to how the human eye and brain read and process text and images from pages of a newspaper.

In some embodiments, the optical sensor may first "read" the physical implementation of the document and convert the optical image data into a file (eg, .pdf file, .tiff file, or other image file format). The optical sensor may use a laser, scanner, or other optical reader to read data from the physical document. The optical processor 6 may then process the file. For example, the optical processor may use OCR (eg, OCR software or other software) to recognize or identify a pattern in stored optical data. Some types of OCR require the conversion of optically scanned bitmaps of printed or written text characters to character codes such as ASCII.

By processing optical data from the image of the document, various optical scanning techniques can enable the optical processor to identify text and images from the document. For example, optical character recognition (OCR) technology (eg, OCR scanner and software) allows the optical processor 6 to identify text characters in a document. For example, OCR allows the optical processor 6 to extract the text of a newspaper by optically "reading" the newspaper. This extracted text can be saved to a file.

In processing optical data of an image, the optical processor 6 may compare the image (or portion thereof) with other images (or portion thereof), for example an image or portion stored in the content database 40. Specifically, the optical processor 6 may compare an image from one document source 12a with one or more documents from the same document source 12a already stored in the content database 40. For example, the optical processor 6 may determine whether a document is the same (or substantially the same) as another document. A "substantially identical" document may be a document that is determined to have 90% (or 95%, 98% or 99.5%, etc.) identical content to another document. For example, 90% of the text may be the same (90% of the words are the same in the same order), and / or the pattern defining the image of one document is 99% the same as the pattern defining the image within another document. can do. Other measures of "substantially the same" can be envisioned.

One document may be determined to have nearly identical text and images but different formatting, and the optical processor 6 may determine the (relative) equivalence of the two documents. For the (substantially) same document, the optical processor 6 can indicate that the document is (substantially) identical to another document and can identify the other document. If the first document is identified as being substantially the same as the second document already stored in the database, it may receive the same evaluation information as the first document.

Although conventional OCR techniques are generally used to identify text, OCR and OCR-like techniques may allow the optical processor 6 to identify one or more images in a document by optically "reading" the document. . For example, the optical processor may read the image and determine that the image includes a photograph of a film projector, a beer bottle, a person (varies in clothing), or other object. The optical processor 6 may also determine other optical data about the image, such as image color, color scheme, pattern, and other information. By using this technique, the optical processor 6 can determine the amount of the human body (eg, the body of an undressed person) appearing in the image. Thus, the optical processor 6 may determine the amount or presence of text or images of drugs, alcohol, nudity, or other possibly offensive or inappropriate content. Based on this information, the optical processor 6 may evaluate the document (eg an image) according to these and other criteria. Sample evaluation by the optical processor is shown in FIG. 6.

The image file processor 7 can also identify and process text and images on the document. However, the image file processor 7 can do so by processing computer files that store documents or images (e.g., .pdf or .tiff files) rather than optically reading the physical implementation of the document. Thus, the image file processor 7 operates in a similar manner to the optical processor 6 except that it may not have an optical scanning element.

It will be appreciated that a single document may have one or more electronic embodiments and one or more physical embodiments. For example, a single document may take the form of a .pdf file, a .gif file, a .tiff file, a file in another format, printed paper, a photo, another physical entity, or a visual display on a screen or monitor. Thus, a single document may be evaluated from the server 2 in an electronic format (e.g., .pdf format) or in a physical implementation (e.g., a document printed on paper) (optical processor 6 and image file processor). (7), etc.). The optical processor 6 can determine the evaluation information when the physical implementation of the document is used, and the optical processor 6 or the image file processor 7 can be used when the document is in electronic format. The optical processor 6 and / or the image file processor 7 may be connected to the server 2 or may be part of the server 2.

An advantage of using the processors 6, 7 to determine the assessment information is that the assessment information can be determined (eg automatically). Since the server 2 can receive and distribute thousands, millions, and / or billions of different documents, the transaction and management costs of manually reviewing each document may be prohibited and / or expensive. . In order to determine whether a document is approved (e.g., legitimate and appropriate) for general distribution or for distribution to one or more specific providers 8 or end-users 10, the processor 6, 7 is an image. Alternatively, it may provide sufficient prior review of other documents.

Processors 6 and 7 may also examine the links in the document (eg, automatically in response to identifying the links). For example, if an image advertising document contains one or more links (e.g., an image ad displays a URL link on an ad, embeds a hypertext link, or otherwise views the viewer, In which the associated document can be selected by clicking on the link), the processors 6 and 7 can access the link and the linked document. If the processor 6, 7 cannot access a link or linked document (e.g., the processor's web browser fails to load the linked document or there is another error), the processor (e.g., In response to identifying a "broken" link may automatically report a "broken link". For example, the processors 6 and 7 can report this information in their evaluation information.

The evaluator 4 may be a provider 8, a document source 12, an end user 10, an optical processor 6, an image file processor 7, and the server 2 itself. The evaluator 4 may be one or more people, groups, and / or processors. The evaluator 4 and the provider 8 may be partners of the entity associated with the production server 2. The end user 10 can be one or more people, computers, computer networks, or other entities on the network. End user 10 may request content from and receive content from provider 8 and / or server 2. Additional participants may be included based on various applications.

Server 2 may include any server 2, hub, central processor, provider, search engine, or other entity within the network. Database 39 connected to server 2 may include one or more databases 39-54. In addition, the databases 39-54 may include portions of a single database 39. It will be appreciated that the databases 39-54 may or may not be physically separated. The server 2 and its modules 20-36 may store the information and access the information stored in the database (s) 39, 39-54.

The content database 40 may store data related to the document and / or portions of the document, such as images and text. The document may be rated and / or unrated. For example, a document may or may not have a particular type of assessment information associated with it. For example, the document may have evaluation information from the processors 6, 7 but may not have evaluation information from the end user 10 or the provider 12.

The document may or may not be received from the document source 12, the evaluator 4 and the provider 8 or may be stored by the server 2. The content database 40 can identify any type and / or group of documents or otherwise associate the documents with each other. For example, the document may include the following features: subject, content rating, comprehensive content rating, sensitivity score, content type, language, country of origin (eg, country of origin or city), region of target user, document source 12. May be identified and / or associated with other documents according to one or more of the owner of the content, creator of the content, target demographics, or other criteria. For example, a document may contain other harassing content, associated keywords, associated sites (e.g., sites that are explicitly or implicitly linked to the document via embedded hypertext links, etc.), and the status of associated sites (e.g., For example, links in the document are broken and / or point to invalid URLs, physical content (eg, undressed person images), pornography or other indecent content, adult content, drug or alcohol related content, Children can be identified or grouped according to their content, or target age, gender, race, nationality, or religion. Other factors related to matching content with the appropriate user can be considered.

The content rating database 42 may store content ratings (also referred to as "rating information") received from the rater 4 and any other information that may be associated with rating the document. For example, the content rating database 42 may include the number of clicks on the document, the number of times the document was presented, the click rate, the number of times the document reached a transaction (eg, a user clicked on the document and then linked or otherwise associated with it). Information about feedback (e.g., feedback received from end user 10 or provider 12), information about complaint letters submitted for documents, verification information (if specific) Content creators, etc. to assure the adequacy of the document), and third parties or other decisions regarding the content of the document. Each stored content rating may be associated with a particular appraiser 4 and a particular document (eg, an electronic document). Each content rating may include one or more topic ratings (see FIG. 7). For example, content ratings may include ratings for violent content and other ratings for sexual content and drug related content. If the document is linked to one or more other documents (eg, linked to a URL), the evaluation of the content of the linked document may also be included here. These individual assessments may or may not be combined into a single score (eg, a single comprehensive assessment of a document). The content evaluation and / or the comprehensive evaluation of the document may change when new evaluation information about the document is received.

The content rating database 42 may also store general approval status of documents and performance data related to the documents. The approval status can be "approved", "disapproved", "suspended pending further review", or "un-approved". Documents can have approval status for various criteria. Performance data includes, inter alia, the click rate of the document, the number of feedback ratings, the number of feedback ratings, and the transaction rate (the number of times a document has reached the transaction compared to the number of times the document has been provided by the server 2 or selected by the end user). , And transaction volume.

Confidence score database 44 may store confidence scores associated with one or more evaluators 4, document source 12, and content creators (collectively referred to as “trust entities”). A single trust entity may have one or more trust scores associated with it. A single trust entity may have different scores for different types of content evaluation subject areas. For example, a trust entity may have a high confidence score for an age suitability assessment but a low confidence score for a violence assessment. The evaluator is also subject to the following criteria: time (e.g., time of day, date of year), evaluator's geographic location, document recipient profile, size of evaluator entity, evaluator's business area or industry, pre-evaluator evaluation consistency or accuracy , Trust terms and the terms of the agreement between the server 2 (if any), and trust ratings of trust entities by other evaluators 4b-4n. The evaluator 4 may be assigned a low confidence rating when evaluating his own content. This assessment may be automatically assigned in response to identifying that the evaluator is evaluating his own content.

Confidence scores can be normalized or otherwise adjusted for one or more evaluators 4. For example, the trust score may be adjusted to reflect that the trust entity consistently provides a higher or lower rating, eg, a particular type of rating (violence) for a particular type of content (advertising). For example, a trust entity may provide an adult content rating that is at least one standard deviation of the mean higher than the ratings of other raters 4b-4n in 80% of the raters 4a. This may inform the system that, on average, the adult score rating of the evaluator 4a is likely to be higher than the average in future assessments, eg, a particular type of future assessment for a particular type of content.

Other information may be determined, deduced, or deduced from the evaluation behavior of one or more evaluators. For example, if evaluator A provides a higher adult content rating for document XYZ than evaluator B and evaluator B provides a higher adult content rating for document QRS than evaluator C, the system evaluates that evaluator A is better than evaluator C. It can be assumed to provide (or will be) providing a higher adult content rating. As a result, the system can adjust (eg, lower) the confidence score for evaluator A (or B or C) in future assessments, such as assessments involving adult content assessments from both evaluators A and C.

Likewise, instead of (or in addition to) adjusting the assessor's confidence score, the assessment itself may be adjusted in a normalized or otherwise manner, as described above for the confidence score. For example, for a particular trust entity that provides a consistently high violent rating compared to ratings of similarly located evaluators 4b-4g for the same or similar content, that particular trust entity for that type of content Violence assessment can be adjusted downward.

It should be noted that because ratings can be requested from and received from hundreds (or perhaps thousands or millions) of raters (4), it is biased against a large number of users, content types, rating categories and other criteria. Other evaluation behaviors can be tracked and predicted. As will be appreciated by those skilled in the art, normalization and other statistical and / or data processing techniques may be used to process the data. The processed data can be used to adjust assessments, confidence scores, comprehensive assessments, and other calculations and information.

Sensitivity data 46 may store assessor 4, provider 8, end user 10, document source 12, server 2, sensitivity information for a particular document, or other sensitivity information. The sensitivity score may indicate a level of conservatism that is a standard for synthesizing the content evaluation. For example, a high sensitivity score (eg 95%) may indicate that the overall assessment should be very conservatively aggregated. Conservative aggregation means that if an evaluator (4) rated a document as inappropriate for children, even if the "average" rating indicates that the content is appropriate for all ages, the comprehensive rating is also It means that the document is inadequate for children. A sensitivity score can be designated by the provider 8 and / or the end user 10, or any other entity. For example, the provider 8 may assign very high sensitivity to violent and sexual content, and may assign low sensitivity to age-related content. Thus, when the server 2 selects a document for its provider 8, none of the document's evaluators 4 finds any sexual or violent content (or very few evaluators have found sexual or violent content). If the document can include content for users of any age, the server 2 can select the document.

For example, the sensitivity database 46 indicates that Provider A (or a particular end user 10a) has a 55% sensitivity score for sexual content and a 20% sensitivity score for age-related content. It can contain records.

The comprehensive assessment database 48 may store a comprehensive assessment. Each comprehensive assessment may be associated with the evaluator 4 used to determine the overall assessment as well as the document evaluated. A comprehensive assessment can also be associated with one or more sensitivity assessments and confidence scores. Each comprehensive rating may include a content rating across various criteria. The content of the comprehensive assessment is further described below with respect to a rating aggregation module 30. The comprehensive evaluation database 48 may also store the approval status of the document.

The evaluator database 50 may store evaluator (and other trusted entity) information. For example, the evaluator database 50 may store information related to a trust score of a particular trust entity. The evaluator database 50 includes trust such as name, address, nationality, company size, prior content assessment information, industry sector, company reputation information, prior trust score, and other information that may relate to the trust score of the trust entity. Entity information can be stored.

The evaluator database 50 may store information that allows the server 2 to identify a particular end user 10a or provider 8a. For example, the evaluator database 50 may store a static IP address of a user who previously provided evaluation information about the document.

The conformance database 52 stores information about providers 8 and end users 10 that can request documents. For example, suitability database 52 may identify gender, age, name, nationality, and other personal information about end user 10 or other content requestor. This profile will be different from other end users 10 who were 62-year-old men from Spain. Each user 10 and / or provider 8 may have a set of conformance standards associated with them and stored in the conformance database 52.

Other database (s) 54 may include assessor 4, provider 8, end user 10, document source 12, server 2, content assessment, comprehensive assessment, trust score, suitability score, and sensitivity Other information related to the score can be stored.

The server 2 may include one or more modules that process content, ratings, and other data.

The document selection module 20 may select documents to be evaluated, for example by one or more specific evaluators. This module 20 can select documents stored in the document database 40. The document selection module 20 may randomly select a document or select a document having a particular feature. For example, the document selection module 20 may include a document provided by a particular document source 12, a document evaluated by a specific evaluator 4, a document of a particular length or type (eg, a short advertisement or Image ads). The document may be selected according to a subject, associated keywords (eg, keywords commonly associated with suspicious content), financial data, performance data, origin, owner's identity, creator's identity, and / or other information associated with the document. have.

For example, a document that can be selected for review (eg, a manual review) has the following characteristics: high enough physical assessment, high enough number of times that the document has been provided to the provider 8 or the end user 10. Or low, the number of times a document has been selected (eg clicked) by the end user 10 is high enough or low, the rating in one or more categories is high or low enough, the number of times the document has been rated high or low It may include a document having one or more of the following. If the document is associated with another document, any content-related considerations of the associated document may be the basis for the review. For example, if an image ad is explicitly or implicitly linked to a URL of a website, content considerations (eg, "evaluation" of the website) may be the basis for reviewing or disapproving the document. have. For example, the document may be reviewed if the site is flagged as having the potential to provide or show inappropriate content, or if the site is known to provide such content. The evaluator may report this finding (eg, through feedback from an image advertisement) through his content rating.

In addition, a document, such as an image advertisement, may be marked as disapproved or reviewed if the document is linked to a "broken" document. For example, an end user may cause an error in other ways, such as that a link in the document does not effectively guide the end user's browser to the intended site (or any site at all) or fails to load the linked document properly. Can be reported through feedback. The document may also be disapproved or selected for review if the provider 8 complains about the document (or document source 12) and blocks the document from its site.

Evaluated and unrated documents may be selected for review. ("Unrated" documents may include documents that do not have a passive evaluation, and therefore may not include documents that have evaluations from the optical processor 6 and the image file processor 7). As will be appreciated, for example, an advertisement with a high clickthrough rate (CTR) may be selected for evaluation before an advertisement with a low clickthrough rate. Documents with more evaluation information (eg, negative feedback) or contradictory evaluation information may have priority over documents with less evaluation information (or positive evaluation information). The web page from the wine distributor may be selected before the advertisement from the appliance manufacturer.

The evaluator selection module 22 may select evaluators 4a-4n evaluating one or more electronic documents. The evaluator 4 and associated document can be selected randomly or by other methods. For example, the evaluator 4 may be randomly selected from a group of particular types of evaluators, such as partner advertisers or groups of ad listing partners. The evaluator 4 may be selected according to any criteria such as the evaluator's confidence score.

Unrated documents may be selected for evaluation before (or after) the evaluated documents. Some evaluators 4 may send some of the evaluable documents to the server 2. These evaluators 4 may or may not be selected to review the document. In one embodiment, entities that provide evaluable content to the server 2 evaluate the content before providing that content to the server 2.

The confidence score module 24 determines the confidence score for the evaluator 4. The confidence score of the evaluator 4 is a measure of the degree to which the evaluator 4's content rating is taken into account in determining the overall rating. For example, the content rating of the highly trusted evaluator 4 may be given a greater weight in calculating the overall rating than the less trusted evaluator 4. The trust score is based on the following factors related to the evaluator (4): evaluator's name, company, company size, evaluator's location (and nationality), evaluator's language, evaluator's reputation, company industry, prior content assessment, prior content assessment Number and a comparison between the overall rating associated with the prior content rating or the associated content rating by another evaluator.

The sensitivity module 26 may determine a sensitivity score for the provider 8, the end user 10, the evaluator 4, and / or the server 2. These scores may be based on information collected about these entities, such as pre-evaluation data and any information stored in any of the database 39. High sensitivity can indicate a high degree of conservatism in determining comprehensive assessments. For example, if provider 8a complains of receiving inappropriately violent content in the past, sensitivity module 26 may associate a high violence sensitivity score with that provider 8a.

The content rating module 28 may generate content rating requests and pass them to the rater 4. An example content rating request is shown in FIG. 7. The content evaluation request may include a list of requested information about the document to be evaluated. The content evaluation request may be the same or different for each document. Thus, the requested information may be different (or may be the same) for each document, such as a particular type of document (such as a particular type of keyword or industry related image advertisement or document). For example, a request to evaluate a beer advertisement may request detailed information about how the beer / booze is represented, while a request to evaluate a power tool merchant's website may not. The content rating module 28 may also receive and process content ratings received from the rater 4. The content rating request and content rating may be delivered via conventional mail, email, or other methods. For example, the server 2 may send a link to the evaluator 4 a server 2 website containing the document and associated content rating information. The evaluator may then access the document and provide content ratings directly on the server 2 site. The content rating module 28 may then electronically process the rating information.

The assessment synthesis module 30 may determine a comprehensive assessment of the document by processing one or more content assessments of the documents received from the one or more evaluators 4. Evaluation synthesis module 30 may use any synthesis algorithm or method. For example, aggregation may relate to total score, mean, mode, median, step function, 75th (or other) percentile score, or data considered. May be any other measure that exists. The algorithm may be one of the following: content evaluation, sensitivity information, suitability information, confidence score information, document performance data (such as click rate and display frequency), and other information about the evaluator, document, document content, or synthesis; One or more may be considered. For example, in one particular iteration, if any content rating indicated any violence, the rating synthesis module 30 may mark the document as violent. This can happen if a high sensitivity to violence is included in the algorithm. Another algorithm can determine a high violence score only if the overwhelming majority of evaluators 4 evaluate the document as having strong violence. This algorithm can weight different evaluators' ratings differently depending on each evaluator's confidence score, if any. For example, a violent assessment of a highly trusted evaluator may be as important as three times the violent assessment of a less trusted evaluator. This algorithm may also neglect the evaluation received from the processor 6, 7. Other examples are also possible.

The comprehensive assessment may include one or more distinct numerical scores (eg, for different subject areas such as pornography and violence) and / or one or more verbal scores. The verbal score may be a word (eg, writing) analysis of the characteristics of the document. For example, the evaluator may provide a numerical score of 4/10 in the “sexual content” category, and the evaluator may also say that the document “contains a half-naked woman with a sexy provocative look”. . The assessment may also include one or more binary scores (eg / no or flag / no-flag, etc.). For example, if three of the five evaluators flag a document as pornographic in their content rating, the rating synthesis module 30 may flag the document as pornographic. Numeric scores may include one or more measures of total assessments in a particular subject area, and numerical scores may also represent other information about the various assessments aggregated. For example, the score may include the mean in addition to the standard deviation of the mean. The comprehensive assessment may include multidimensional vectors combined with verbal responses.

Evaluation synthesis module 30 may also determine that a particular document may be subject to additional review. For example, if a less trusted evaluator has marked the document as pornographic, the document may be marked for further review (such as manual review by the server 2 or its agent). The assessment synthesis module may update (eg, automatically) the comprehensive assessment at a particular time or when a new assessment is entered into the system.

Evaluation synthesis module 30 may also suspend, disapprove or otherwise change the evaluation or status of a document flagged for review by document selection module 20.

The conformance standards module 32 may identify or determine the conformance standards of document recipients, such as the provider 8 and the end user 10. The conformance standard may be based on provider 8 or user 10 information such as the above information. Compliance standards can be identified by the compliance standards module 32. For example, a fitness standard for a user identified as a 20-year-old woman viewing a pop singer's website may indicate that a small amount of violent and obscene content may be appropriate for that user, but alcohol-related content is not. . Conformance standards can be used to select appropriate documentation for that user.

If the server 2 attempts to select one or more documents that are likely to be delivered to the requesting service provider 8 or the end user 10, the conformance standard module 32 may provide the provider 8 or the end user 10. May attempt to determine whether any information related to the preference of. If there is a conformance standard for the potential content recipient, document providing module 34 may select a document that meets the conformance standard (eg, a document that contains little violent content). Similar but slightly different, suitable content identification methods may be achieved using sensitivity scores, as described herein.

The document providing module 34 may select and present a document in response to a request from the provider 8, the end user 10, or another entity. For example, a children's book retailer can request an advertisement to display on his site. In response, document providing module 34 may identify or determine susceptibility score information and suitability information associated with the children's book retailer and then select a document based on this information (such as an identified suitability standard). For example, if the site is associated with high susceptibility to violent content, document providing module 34 may select content with a comprehensive rating indicating no violent content. The document providing module 34 may also select documents that have been verified not to have violent content that is not intended by a trusted evaluator.

Document selection may be based on suitability and / or sensitivity information as well as a comprehensive assessment (or content assessment) associated with the document. For example, document serving module 34 may determine that provider 8 (or end user 10) has high sensitivity to pornographic content, so that no evaluator finds pornographic content. Or, any evaluator can choose a document for which no sexual content of any kind was found. The document providing module 34 may also determine that the provider prefers a document having a comprehensive rating that indicates a high value in the category of child content, and thus may select a document that contains the actual child content.

By identifying the suitability and / or susceptibility scores for a particular provider 8 and end user 10 (or other entity receiving the document), a document may be selected that conforms to the conformance profile (or standard) associated with that entity. have. Thus, the end user can receive unobtrusive content, and the content provider can ensure the appropriateness of the content provided on that site.

In some embodiments, document providing module 34 may select and provide an "approved" document. The approved document can be any document that has a reasonable overall rating that is high enough or otherwise in any of the various content areas or that has a sufficiently high content rating. For example, in some embodiments, an approved document can be any document with an assessment indicating general approval for all users. In some embodiments, the approved document may be any document that does not have a negative evaluation (or no evaluation). For example, in the case of an advertising image having means for providing evaluation information, such as feedback (see FIG. 4), the approved document may include a document without feedback.

The document selection module 34 may select a document based on the recipient of the document. For example, if the recipient is male, document selection module 34 may select a document associated with the male user.

The document selection module 34 may identify recipient information, for example by examining the "cookies" on the end user's computer. For example, document selection module 34 may identify the preferences of end user 10 based on prior information received from the end user, such as evaluation information for previously provided documents. Information enabled or identified by the document selection module may be stored in the evaluator database 50.

The optical data module 35 may transfer information to and from the optical processor 6. The optical data module 35 may receive optical data and process the data to determine evaluation information (as described above for the optical processor 6 itself), and / or the optical data module 35 may only be optical Evaluation information determined by the processor 6 can be received and processed. Optical data module 35 may convert this data into a format readable by other modules 20-38 as needed.

File data module 36 may transfer information to and from the image file processor 7. Optical data module 35 may receive optical data and process the data to determine evaluation information (as described above for optical processor 6 itself), and / or optical data module 35 may only image The evaluation information determined by the file processor 7 can be received and processed. Optical data module 35 may convert this data into a format readable by other modules 20-38 as needed.

In sum, the optical data module 35 and the file data module 36 may handle the evaluation of documents such as image advertisements or other documents containing images. When server 2 receives a new or modified document from document source 12 (or requests to modify the document), modules 35 and 36 may cause the document to be processed and evaluated as described above. . As such, when each document is received by the server 2 (or generated by the server 2), an evaluation for each document can be automatically obtained. Manual evaluation of the selected document may also be obtained, for example, via document selection module 20, as described above.

The other module (s) 38 may achieve other functions related to evaluating electronic documents.

Example System Network Environment

2 illustrates a networked environment for the operation of a system for evaluating a document, in accordance with an embodiment of the present invention. In this environment, the evaluator 4 and the provider 8 provide the document and the evaluation information to the server 2 and receive the document and the evaluation request information from the server 2 via the network 14, 15 ( For example, it may be connected to the server 2 using a secure https connection. The server 2 may store the document and evaluation information in the database 39. The server 2 may distribute the document through various forums or feeds, including distributing it directly to print media, providing the document on one or more websites affiliated with the server 2, and through the provider 8. You can distribute it. Note that the provider is connected to the server 2 (eg, connected via the network 14 or network 15 depending on the desired security), the content system (eg, having an associated content database), and the server 2 Or a syndication partner of the search engine system operated by the provider (s) 8.

Through these various forums, documents provided to the provider 8 may be included in pages (or other documents) displayed to the end user 10 (often referred to as impressions).

Each of the server 2, the evaluator 4, the provider 8, and the document source 12 are the following systems: web server 2, database server 2, proxy server 2, network balancing mechanism and It may include a computerized system that includes one or more of a variety of software components that allow a system, system to operate on the Internet or other network type system. In addition, the networks 14 and 15 are shown as http networks, but may include other networks, such as private lines, intranets, or any other network. In an exemplary embodiment, the connection between the document source 12, such as an advertising provider, and the server 2 (and other connections, such as between the provider 8 and the server 2) is such that the data may be sent to any hacker or other third party. It may include a secure network connection to ensure that it is not attacked or damaged by. In addition, although two evaluators 4 and two document providers 12 are shown, it will be appreciated that one or more evaluators 4 and one or more document providers 12 may be provided in the network. Similarly, although one database 39 is shown, multiple databases 39 may be provided and such databases 39 may be of any type of network connection, including the distributed database server 2 architecture. It will be appreciated that it can be connected to server 2 via.

Similarly, provider 8a may comprise any number of such systems connected to evaluator 4 or server 2 via any type of network, including http or https networks. The content provider 8 may comprise a system, such as a server 2, that provides a function that allows connection via the Internet or other network protocols. End user 10 can include any user (such as a user connected to the Internet) and can be any type of network, including through Internet service providers, cable companies, and any other method of accessing data over the Internet. It may include a computerized system that enables the connection through any of the. The provider 8 may include any system that distributes content, such as advertisements, to the end user 10.

Example Process

3 is a flowchart illustrating an exemplary method of evaluating an electronic document, in accordance with an embodiment of the present invention. The method described herein may be implemented by the systems described in FIGS. 1 and 2.

At block 300, one or more documents may be received. For example, server 2 may receive one or more advertisements from one or more partner advertisers or other content providers. As another alternative or in addition, the server 2 may generate one or more documents.

In block 310, the server 2 may select one or more documents to deliver to one or more evaluators. For example, server 2 may deliver the document to one or more partner advertisers or content providers, in which case these entities may be evaluators. These evaluators may be the same or different entities as the ones from which the document was received at block 300 (ie, the server 2 may select the original provider of the document to be one of the evaluators at block 310 (block 300). You can choose from)).

The server 2 may select documents and evaluators randomly or based on algorithms and / or one or more criteria. The document may be selected from a plurality of documents, and the evaluator may be selected from a plurality of evaluators. Although this description describes how to evaluate a single electronic document, it will be appreciated that many documents can be evaluated simultaneously using this method. For example, when each additional document is received by the server 2, the documents may be delivered to one or more evaluators (automatically) selected by the system. These documents may be selected by the document selection module 20 as described above.

At block 320, the selected document (or link to the selected document) can be delivered to one or more evaluators. For example, the server 2 may deliver the document to one or more partner advertisers who will evaluate the document. The server 2 may also send an evaluation command to the evaluator 4 describing the various evaluations requested for the document. The evaluation command may provide a link (eg, a feedback link) to guide the evaluator to a site that prompts the evaluator to obtain specific evaluation information. The linked site may also include means for obtaining or viewing the document itself. For example, server 2 may request each evaluator to rate the document according to various criteria, such as target age range, family content, violent content, and other criteria.

The server 2 may also deliver the document (either directly or indirectly through a provider) to an end user who may be able to evaluate the document (or may not). The document itself may have built-in means for evaluation. For example, an image advertisement may have a "feedback" link to provide rating information. An end user receiving an image advertisement (or other document having means for providing rating information) may then evaluate the document. In such a scenario, the evaluator need not be selected at block 310 because the recipient of the document may be a potential evaluator.

At block 330, the evaluator may generate one or more content ratings (also referred to as “rating information”) associated with the document. For example, the partner evaluator may have one or more evaluator staff or agents review the document and evaluate the document in each category designated by the server 2. Content ratings may be numerical scores (eg, numbers between 1 and 5, or other numbers or percentages), codes (such as "V" for violent), or verbal comments (eg, writing responses). Can be.

For example, an end user or other evaluator may choose to provide rating information by clicking on a feedback link. The feedback link can direct the evaluator to a site (or other location in cyberspace) where the evaluator can provide the assessment information. For example, a feedback link may direct the evaluator to a site urging the evaluator to obtain various evaluation information (see FIG. 7). Evaluation information can be entered at the prompt.

For example, it should be understood that the document can be evaluated by the processor, as described above for the optical processor 6, the image file processor 7, and the optical and file data modules 35, 36. If the document is evaluated automatically, all of blocks 300-330 can be done automatically and almost simultaneously. If the server 2 evaluates the document on its own, for example via optical or file data modules 35, 36, then at block 340 the server 2 need not "receive" the evaluation information from the evaluator. .

At block 340, content ratings may be received from an evaluator. For example, content ratings may be received by the server 2 from an end user or from a partner advertiser via a feedback link.

In optional block 350, a susceptibility assessment and an evaluator confidence score may be identified. For example, server 2 may identify a trust score associated with one or more of the evaluators who submitted a content assessment for the document. Confidence scores may be determined and identified as described for the confidence score module 24 of FIG. 1, and sensitivity information may be determined and identified as described for the sensitivity module 26 of FIG. 1.

At block 360, a comprehensive assessment can be determined. The overall rating may be based on one or more of the content rating, sensitivity score, and confidence score. The comprehensive assessment may be determined by any method as described above for the assessment synthesis module 30 (see FIG. 1). Comprehensive assessments may include general approval or disapproval.

At block 370, a request for a document may be received. For example, server 2 may receive a request for a document from provider 8 or end user 10. At block 380, a conformance standard and / or susceptibility score may be determined.

At block 390, a document may be selected. For example, the server 2 receiving a request for a document from a user or provider can select a document for the user or provider. This selection may be based on a conformance standard of the user or provider, one or more comprehensive assessments of the document, and / or sensitivity scores associated with the evaluator, server 2, provider (s), and / or user (s).

At block 400, one or more selected documents may be delivered to the requesting entity. For example, if the provider or user requested a document at block 370, server 2 may optionally deliver the selected document to the provider or user.

Those skilled in the art will appreciate that the described operations may be implemented in one or more computing systems, such as server 2 systems, connected to entities such as providers, evaluators, and end users, with or without human intervention. It will be appreciated that it may be performed by or a combination thereof. In addition, it should be understood that not all of the blocks have to be achieved. For example, in one embodiment, the method may begin at block 340 and end at block 390. In addition, the operation (s) in each block need not be performed in the order shown in FIG. Any order of execution may be envisioned.

Note that a comprehensive content rating can be determined after a user or content provider makes such a request, rather than before requesting a document from the server 2. Because sensitivity, suitability standards, and evaluator confidence scores can change over time, and because new assessments can be added for a particular document, the overall assessment of the document can also change. Obtaining an up-to-date evaluation of a document prior to providing the document to a user or provider may maximize the likelihood that the document is suitable for the user or provider.

Example User Interface and Results

4 shows an exemplary document 1 according to an embodiment of the invention. 4 shows an exemplary web page 3 search result from an internet search engine. The web page 3 itself may be a document. The content on the web page 3 may also be a document. For example, the advertisement 1 on the search results page 3 may also be a document. Other types of documents, such as advertisements, files, programs, and other information, can also be envisioned.

A document can have several types of content. For example, a document may have a function or program capable of dynamically generating words, images, sounds, and other information as well as words, images, sounds, and other information. Each document may have different amounts of different types of content, such as sexual content, violent content, drug or alcohol related content, financial content, adult-related content, child-related content, and other content.

5 shows an exemplary image advertising document 1 according to one embodiment of the invention. The image advertisement 1 may comprise a banner advertisement, another advertisement that can be displayed on a web page, or another graphical advertisement that can be displayed via electronic means. Image advertisement 1 shows a specific image 1A, advertisement text 1B, one or more links 1C, and a mechanism 1D for evaluating document 1. The specific image 1A includes a film projector and images of DVD, VHS, and digital film containers. Text 1B includes an advertising message, eg, a description of the product or service, a suggestion to a potential customer, and / or other advertising text. The link 1C may include a link to another document, such as the advertiser's web page URL. For example, the link 1C may comprise an embedded hypertext link, and the embedded link may be associated with a link 1C displayed in an image of the document 1. In some embodiments, selecting (eg, clicking) a displayed URL or other link while viewing the document 1 (eg, in a web browser) may result in a viewer's mechanism (eg, Web browser) to point to content associated with the link (eg, the advertiser's web page).

Document 1 may explicitly display link 1C. As another alternative, the link 1C may be embedded within a document (eg, programming of the document) or a portion thereof such that the link 1C is invisible. Here, selecting (eg, clicking) document 1, image 1A, text 1B, or another portion of the document may guide the user's document viewing mechanism to the linked document (s). . The document 1 itself, the image 1A, and the text 1C may also include one or more links 1C. For example, an advertisement advertising multiple products may include a link to each product, where an image 1A, icon 1A, or text 1B related to a particular product is selected (e.g., For example, clicking may direct a web browser (or other document viewing mechanism) to a page (or to another document) on the merchant's site associated with the particular product.

Mechanism 1D may include a link to provide evaluation information. For example, the selection of mechanism 1D (eg, clicking on link 1D) can direct the document viewer to an email or web page where the user can provide rating information. For example, a web page may include a prompt for providing evaluation information or may request evaluation information. Other mechanisms 1D may be contemplated herein to provide a communication link between the evaluator 4 and the server 2.

6 illustrates an example document evaluation 600 in accordance with an embodiment of the present invention. The document evaluation 600 may be generated automatically by, for example, the optical processor 6, the image file processor 7, the optical data module 35, or the file data module 36. Document rating 600 may include information related to content, performance, or other characteristics relating to the document. For example, document evaluation 600 may include the following information: document text (e.g., text identified by OCR), document image (s) (e.g., identified by processor 6, 7). Description of the image, content keywords associated with text and / or image (s), quantity of body (eg, body of undressed person) detected in any image (s), potentially inappropriate content (eg For example, volume of inappropriate content such as alcohol and violence), trust score information, feedback information such as amount and content of feedback, performance data of documents such as click rate, general ratings, and any other rating-related Or any of the document-related information (eg, referred to herein). Performance data for the document may be obtained from the content rating database 42.

Other documents (e.g., image ads) may have pictures of women in bikinis holding wine glasses and a related title, "Enjoy WineCo brand wines in the sun." Another document may show a man holding a gun aiming at an icon labeled "debt." Another document might say, "Invest in FundCo, which returned 6.25% last year." Another document can show children singing "It's a Small World After All" dancing under the rainbow. Another document may have an icon that, when clicked, allows the user to virtually hunt a duck by aiming a rifle to a duck sitting in a lake using a cursor as in a video game. Each document, such as an advertisement, also explicitly specifies the URL (eg, in text characters) of the site being accessed when the advertisement (or portion thereof) is selected (eg, clicked through a user device). You can include a sample at the bottom of the ads you list. In addition, this sample (or other portion of the document) may include graphics or text indicating that the document is a source or sponsor.

7 illustrates an example content rating request 700 (“feedback question”) in accordance with one embodiment of the present invention. The content rating request 700 may include one or more requests for specific ratings 701-703 across a range of topics and content types. For example, there may be a request for evaluation of violence 701, teenage adequacy 702, country-related fitness considerations 703, and other information. For example, a content rating request may include a number (eg, a number from 1 to 10) indicating a quantity of sexual content, violent content, adult content, language-related or country-related content, or other type of content or criteria. The evaluator may be asked to provide. This request may also request information about when the document is best for viewing (day, week, time of year). This request may also ask the requestor to explain the reason for the requestor's numerical evaluation or to provide an evaluation expressed in other words.

8 illustrates an example content rating 80 (completely or partially completed feedback question item) in accordance with an embodiment of the present invention. Content assessment 80 may include one or more subject area content assessments 81-83 for documents such as assessments of violence (81), teenage adequacy (82), and country-related suitability considerations (83), and other information. It may include. The content rating may include one or more responses to the content rating request (see FIG. 7). The content rating may refer to a numerical or verbal evaluation across various criteria, such as sexual content, violent content, adult-themed content, culture-related content, or other content criteria.

Indeed, the server 2 may receive a plurality of image advertisements or other documents from the document source. When the server 2 receives each image advertisement, it can evaluate the document via, for example, the processors 6 and 7. This rating can screen image ads for potentially offensive content, such as pornography, offensive words, and potentially offensive linked sites. For example, this rating may only approve ads that are not determined to have such annoying content.

When the provider (or end user) requests a document such as an image advertisement, the server 2 may provide an image advertisement in response to the request. The server 2 may select an approved image advertisement, depending on its content, suitability, the identity of the requester, the performance of the image advertisement, and other factors. For example, if a user's request is associated with one or more keywords, server 2 may provide an image ad associated with one or more identical or similar keywords (eg, closely related keywords in semantic space). have. For example, if a user searches for the word "flight" on a search engine and requests a search results document, the server 2 may provide a search result with an image advertisement related to the word "flight".

Evaluator 4, such as end user 10, may provide feedback via a feedback link that may be part of an image ad. For example, if an ad contains offensive content, such as depicting extreme violence, end users may click on a feedback link to provide feedback indicative of such content (by evaluating the document as having significant violent content). Can be. The server 2 may receive rating information and change the rating of the image advertisement accordingly.

The server 2 may also request review of the document, stop providing the document (eg, to one or more providers or one or more end users, such as children), alter or suspend distribution of the document, or The status of the document can be changed in other ways based on the evaluation and / or performance information of the document. For example, as noted above, the server 2 may stop providing image ads or add additional advertisements if the rating of the advertisement (eg, a comprehensive rating or one or more specific content ratings) indicates inappropriate or offensive content. You can request a review. This may also be the case if a particular provider 8a blocks a document from being distributed through that provider 8a or if the evaluator 4 requests additional manual review (eg via a feedback mechanism). Can happen. Regarding performance information, if the document has a sufficiently low (or high) clickthrough rate, transaction volume or other performance characteristic, the server 2 may request a manual review of the advertisement or otherwise change the state of the advertisement. For example, if the document has a low clickthrough rate, the server 2 may request one or more additional evaluators 4 to review the document. Documents with undesirable performance characteristics (eg, low clickthrough rates) may be disapproved for distribution to end users. In this case, the reason why they may be disapproved is not because of inappropriate content but because they do not properly generate end user 10 interest or revenue.

The server 2 may request a manual review for any reason. For example, the server 2 may request any limited manual review of the document as a matter of course, regardless of the evaluation. The server 2 may also request a manual review depending on the trust score or other factors of the document source 12.

Prior to the document being evaluated (or before there is sufficient evaluation information to determine a sufficiently reliable evaluation of the document), the document may be classified as "unapproved" or "not yet approved." These may be documents received from document source 12 or they may be documents generated by server 2 that have not yet been evaluated (or only evaluated by a processor). For these documents, the content rating database 42 may indicate that the rating of the document is "unapproved", "undecided", "under review" or "preapproved" and so on. The document may still be communicated to the end user 10 (eg, via the provider 8), for example, to obtain evaluation information (eg, feedback) about the document. However, the provision of the document may be limited due to its uncertainty. For example, the document may not be provided to a particular user 10a or provider 8a, such as a child 10 or a non-partner provider 8. After obtaining the evaluation information and / or determining a comprehensive evaluation of the document, the server 2 informs the user 10 based on the user 10, the request, the user information, the provider information, and / or the document-related information. You can deliver the document. For example, an image advertisement containing sexual content may be selected and delivered to an adult user in response to a request to view an adult web site.

The document may include a feedback mechanism 1D. After the document has been viewed multiple times (eg, 5000 times) without receiving any feedback (or a negative amount of negative feedback, such as none), the document is approved or fully approved (eg, "for all users Approved "). Such documents can be delivered to the end user 10 without limitation. In addition, the feedback mechanism 1C may be removed from the document (eg, because it is considered unnecessary) or the content of the document may be modified in other ways.

In order to obtain review (s) (eg, an additional manual review) for documents identified as needing further review, server 2 may select one or more documents and one or more documents to review. The evaluator 4 can be selected. For example, out of 100 documents requiring manual review, the server 2 may select five for manual evaluation and then three evaluators to evaluate each of the five documents. (Also and / or alternatively, the server 2 may send five documents to a plurality of selected evaluators 4 such that each document has three evaluators 4 but each of the evaluators 4 does not have all three documents. The server 2 may transmit an evaluation request with each document. In addition, server 2 may post documents and evaluation requests (e.g., evaluation forms) to a database (e.g., in database 6 and / or on a website), and to evaluator 4 Allows users to access the database and provide content ratings.

For example, one content rating 80 by a woman in bikini and a reviewer of a document with wine glasses is rated 6 out of 10 for sexual content, 4 out of 5 for alcohol content, The document may show an indication that it is suitable for children, and this assessment may include an indication that the document is most suitable for a time between 7 pm and 6 am. Content ratings from a second rater on the same document may include a rating of 7 out of 10 for sexual content, a rating of 3 out of 5 for alcohol content, an indication that the document is suitable for children, and the document is 4 pm It may indicate that the best fit for the time between and 6 am. A content rating from a third rater on the same document is rated 8 out of 10 for sexual content, 3 out of 5 for alcohol content, an indication that the advertisement is not suitable for children, and the document is 11 pm May indicate an indication that it is most appropriate for times between 6 am and 6 am and not for holidays

The server 2 can produce a comprehensive evaluation of this document based on the three evaluations. The server 2 can designate high sensitivity for children's content and medium sensitivity for violent content. The server 2 may calculate a comprehensive rating for the sexual content based on the average (or median) of the three sexual content ratings (in this case, 7 out of 10). The server 2 may calculate a comprehensive assessment of the alcohol content based on the mode of alcohol assessment (most common score), which in this case is 3 out of 5. In addition, the server 2 may neglect the rating of the first one of four out of five when the first rater has a low confidence score, and the weighted average of the three ratings is 3.1 out of 5 (which may be rounded to three). May be). Since the server 2 applies high sensitivity to children's content, the server may determine that the document is not suitable for children because one of the evaluators indicates that the document is not suitable for children. The server 2 may select a time of day rating of the most trusted evaluator. If the evaluator indicates that the most suitable time is from 11 pm to 6 am, the overall evaluation may indicate that the most suitable time is from 11 pm to 6 am. The overall rating may also represent standard deviations for each component of the overall rating or may use any other measure of variation in different ratings from different raters.

If any of the three assessments differs significantly from the overall assessment, the assessment may be marked for further review. For example, because only one evaluator has indicated that the document is not suitable for children, the server 2 (or an agent or other evaluator of the server 2) may need to examine the document to determine whether the unusual evaluation is accurate. You can review it. For example, server 2 may review the evaluator's assessment and / or generate (or request) one or more additional assessments for the document. Depending on the result, server 2 may change the overall rating (eg, by recalculating the overall rating) and / or change the assessor's confidence score in question for one or more content criteria.

A comprehensive evaluation of other documents can also be determined. For example, an advertisement where children sing under the rainbow may be rated as having no sexual content, no alcohol content, no violent content, and suitable for any time of the day (especially 8 am to 6 pm). have.

The end user or provider can then request the document at midnight. For example, a customer may click on a provider's web page, and the provider may request an advertisement to display to the customer on the provider's site. The server 2 may select a document to be delivered to the provider according to the comprehensive content evaluation of various documents. To select the appropriate document, the server 2 may also identify the sensitivity and suitability information of the requesting provider (or user) and may consider other criteria such as request time. For example, the server 2 may identify that the provider has high sensitivity to sexual content. (In another example, server 2 may identify a user's age and gender (eg, a 25 year old male) (eg, using cookies or information stored in a database associated with the user's IP address). The server 2 may, for example, assign sensitivity or suitability criteria for the time of day. For a request at midnight, the server 2 may assign a lower sensitivity to sexual and violent content than halfway through the day, since the child is very unlikely to be browsing the Internet at that time. The relevant time can be viewed from the perspective of the server 2, the provider 8, or the end user 10.

Thus, the server 2 can select a document suitable for a provider who has high sensitivity to sexual content and medium sensitivity to violent content. The server 2 may consider all or part of the documents stored in the database, including bikini / wine documents, singing children, and duck hunting documents, for selection. For example, the server 2 may consider only those documents with a comprehensive rating indicating little or no sexual content and only mild (or less) violence. In the middle of the day, the server 2 may apply high sensitivity to violent content due to the time of day, so only non-violent documents can be considered at that time. However, at midnight, server 2 may consider other documents that may not be appropriate for the user at an earlier time.

The server 2 may then retrieve documents that meet various criteria associated with the document request and related sensitivity and suitability information. While the images children sing may not have any offensive or potentially offensive content, it also has a low clickthrough rate and thus may not be selected. Note that documents with little or no potentially offensive content may or may not be preferred over documents that contain potentially offensive material. Here, a duck hunting document can be selected if his comprehensive evaluation only shows mild violence. If the document is an advertisement with a high clickthrough rate, the document may be more likely to be selected. Other factors of document selection may be considered.

It will be appreciated that the servers, processors, and modules described herein can perform their functions (eg, reading optical information or determining evaluation information) automatically or through an automated system. . As used herein, the term “automatically” refers to a process in which an operation is performed by any machine-executable process, for example, requiring no human intervention or input.

Embodiments of the present invention should not be limited in scope by the specific embodiments described herein. For example, although many of the embodiments disclosed herein are described in connection with advertising, the principles herein are equally applicable to documents and other content. Indeed, various modifications of the embodiments of the invention, in addition to those described herein, will be apparent to those skilled in the art from the foregoing description and the accompanying drawings. Accordingly, such modifications should be considered to be within the scope of the following appended claims. In addition, while some of the embodiments of the invention are described herein in connection with a particular implementation in a particular environment for a particular purpose, those skilled in the art are not limited to these, and the practice of the invention It will be appreciated that the examples may advantageously be implemented in any number of environments for any number of purposes. Accordingly, the claims set forth below should be construed in view of the full scope and spirit of the embodiments of the invention described herein.

Claims (65)

A method of evaluating a document containing an image, Receiving documents for distribution, Receiving evaluation information associated with the document from one or more evaluation entities, at least one of the one or more evaluation entities including a processor to determine evaluation information associated with the image, and Approving the document for distribution based on the evaluation information How to include. The method of claim 1, The document includes at least one of a web page, content that can be used in the web page, and a program. The method of claim 1, The processor includes an optical reader, The processor, Reading optical data from the image by the optical reader, Converting the optical data into computer-readable data, and Processing the computer-readable data How to determine assessment information via The method of claim 3, And the processor associates one or more concepts with the image by processing the computer-readable data. The method of claim 1, Wherein the document comprises an image advertisement. The method of claim 1, The document includes an image advertisement associated with an advertiser, the image advertisement comprising a link to a web site associated with the advertiser. The method of claim 6, In response to the selection of the image advertisement by the user, directing the user to a website associated with the image advertisement. The method of claim 1, The document includes a feedback mechanism that allows recipients to evaluate the document. The method of claim 1, Comparing the image with at least one stored image by the image processor. The method of claim 1, Identifying one or more concepts associated with the document by processing the file storing the image. The method of claim 1, The processor identifying human flesh content in the image. The method of claim 1, The document includes an image ad, The method, Receiving a request for search results, and Providing a search result that includes the image ad How to include more. The method of claim 1, Disapproving the document in response to determining that the document contains inappropriate content. The method of claim 1, The document includes an image advertisement, The method further includes approving the image advertisement based on the performance parameter of the image advertisement. The method of claim 14, The performance parameter is one of a click-through-rate, a transaction volume, a transaction rate, and a number of feedback ratings. The method of claim 1, Distributing the document to the one or more evaluation entities. The method of claim 1, The document includes an image advertisement associated with the concept, The method further includes providing the image advertisement in response to a request for a document associated with the concept. The method of claim 1, Disapproving the document in response to determining that the provider refuses to distribute the document. The method of claim 1, The document contains a link to a website, The method further includes disapproving the document in response to determining that the link does not direct a user's browser to the web site. The method of claim 1, Approving the document if the rating information does not include an indication of inappropriate content. The method of claim 1, The evaluation information includes at least one indication of inappropriate content, The method further comprises requesting evaluation information from one or more additional evaluation entities. The method of claim 1, The document contains a link to a website, The method further includes disapproving the document in response to determining that the web site contains inappropriate content. The method of claim 1, Approving the document is also based on at least one trust score, each trust score being associated with a particular assessment entity of the one or more assessment entities. The method of claim 23, wherein Modifying a confidence score of the particular assessment entity of the one or more assessment entities based on the assessment information received from the particular assessment entity. The method of claim 23, wherein At least one confidence score is based on a geographic location of the associated particular assessment entity of the one or more assessment entities. The method of claim 23, wherein At least one confidence score is based on one or more prior content ratings received from the associated particular rating entity of the one or more rating entities. The method of claim 23, wherein At least one confidence score is based on an industry associated with the associated particular valuation entity of the one or more valuation entities. The method of claim 23, wherein At least one confidence score is based on a rating deviation of a particular rating entity of the one or more rating entities, The evaluation deviation is based on a comparison of (i) evaluation information for one or more documents received from the particular evaluation entity and (ii) evaluation information for the one or more documents received from one or more other evaluation entities. The method of claim 23, wherein Determining a comprehensive assessment based on the assessment information and the at least one confidence score, Approving the document is also based on the comprehensive evaluation. The method of claim 29, At least one confidence score is based on an evaluation deviation of the rating entity, The evaluation deviation is based on a comparison of evaluation information for (i) one or more documents received from the evaluation entity and (ii) comprehensive evaluations for the one or more documents. The method of claim 23, wherein And the confidence score of a particular evaluation entity of the one or more evaluation entities is based on previously received evaluation information from the particular evaluation entity for one or more documents. The method of claim 23, wherein Determining one or more corrected confidence scores for one or more of the one or more evaluation entities, and Determining a corrected global assessment based on the one or more corrected confidence scores. The method of claim 23, wherein At least one of the at least one confidence score is based on at least one of a geographic location of the associated assessment entity and pre-evaluation information received from the associated assessment entity. The method of claim 1, The evaluation information includes the refusal of distribution of the document by the provider, The method further comprises blocking distribution of the document to the provider. The method of claim 1, Delivering the document to one or more evaluators for manual review in response to determining that the processor has not approved the document. The method of claim 1, The evaluation information includes information regarding at least one of sexual content, violent content, adult content, and targeted age content in the document. The method of claim 1, The document is associated with a document source, The method further includes comparing the image with one or more stored images by a processor, The one or more stored images are also associated with the document source. The method of claim 37, The at least one stored image comprises an approved image, The method further includes accepting the image in response to determining that the image is substantially the same as the approved image. The method of claim 1, Receiving a signal related to a reference, and Delivering the document in response to the signal based on the criteria and the evaluation information. The method of claim 39, The signal is a request received from an entity, the entity being associated with a conformance standard, and wherein the step of delivering is based on whether the document meets the conformance standard. The method of claim 40, Determining a comprehensive evaluation based on the evaluation information, and Selecting the electronic document for distribution based on the comprehensive assessment and the conformance standard. The method of claim 39, Identifying at least one confidence score, each confidence score being associated with a particular assessment entity of the one or more assessment entities; and Determining a comprehensive assessment based on the assessment information and the at least one confidence score, wherein approving the document is also based on the comprehensive assessment. The method of claim 39, Wherein said criterion comprises a sensitivity score. The method of claim 39, Receiving new evaluation information for the document, and Processing the new assessment information to determine a corrected comprehensive assessment associated with the document. The method of claim 1, Determining a comprehensive evaluation of the document by synthesizing the evaluation information, wherein the evaluation information is received from at least two evaluation entities, and approving the document is based on the synthesizing. The method of claim 45, Wherein said comprehensive assessment comprises a median of said assessment information. The method of claim 45, The comprehensive evaluation comprises a mode of the evaluation information. The method of claim 45, Wherein said overall assessment comprises a quantity. The method of claim 45, And wherein said comprehensive assessment comprises a mean of said assessment information. The method of claim 1, Associating the document with one or more concepts. The method of claim 1, Associating the document with one or more keywords based on the evaluation information. The method of claim 51, Receiving a request for a document associated with the concept, wherein the request is received from at least one of a provider and an end user; and Distributing the document to the at least one of the provider and the end user Including, Wherein said distributing step is based on an association between said concept and said one or more keywords. The method of claim 1, The document includes a feedback mechanism that allows recipients to evaluate the document, The method, Receiving a negative content rating of the document from at least one recipient, and Delivering the document to at least one human evaluator based on the negative content rating. The method of claim 1, The document includes a feedback mechanism that allows recipients to evaluate the document, The method, Receiving a negative content rating of the document from at least one recipient, and Approving or disapproving the document for distribution based on the negative content rating. The method of claim 1, The document is an image ad, The method, Determining a clickthrough rate of the image ad, and Approving or disapproving the advertisement based on the clickthrough rate. A computer-readable medium encoded with computer program code for evaluating an electronic document, The program code is, Receiving evaluation information associated with the document from one or more evaluation entities, at least one of the one or more evaluation entities including a processor to determine evaluation information associated with the image, and Determining whether to approve the document based on the evaluation information A computer-readable medium effective for carrying out. A system for evaluating documents, An input device for receiving evaluation information associated with a document from one or more evaluation entities, wherein at least one of the one or more evaluation entities comprises an image processor, An image processor for determining evaluation information associated with the image, and A processor for determining whether to approve the document based on the evaluation information System comprising a. A system for evaluating documents, Evaluation receiving means for receiving evaluation information associated with a document from one or more evaluation entities, wherein at least one of the one or more evaluation entities comprises an image processor, Image processing means for determining evaluation information associated with the image, and Determination means for determining whether to approve the document based on the evaluation information System comprising. Transmitting a signal to a server, the server interprets the signal as being associated with a criterion for the document, and wherein the one or more documents Determine whether a document including an image is to be delivered based in part on whether the criterion is associated with the criterion, wherein at least one of the one or more documents includes an image, the plurality of evaluators processing the image and Includes an image processor to evaluate-, and Receiving a document comprising an image in response to the signal How to include. Receiving one or more documents from a server system, at least one of the one or more documents including an image, the server system including an image processor to process the image and evaluate the document; and Providing evaluation information to the server system for use by the server system in determining whether to deliver the one or more documents in response to requests for documents based on the evaluation provided. How to include. A method of evaluating a particular document containing an image, Distributing the document to one or more evaluation entities, wherein at least one of the one or more evaluation entities includes a processor for determining evaluation information by processing at least one of a file storing the image and optical data of the image; , Receiving evaluation information associated with the particular document from the one or more evaluation entities, Associating the particular document with one or more specific criteria, Determining whether to approve the specific document based on the evaluation information, Receiving a request for a document relating to a criterion, and Selecting a document for distribution from a database containing the particular document, wherein the selecting is based on the criteria and the evaluation information How to include. As a way to evaluate a document, Receiving a first content rating associated with a first advertisement comprising an image, wherein the first content rating is received from an image processor that determines the first content rating by processing the image, Receiving at least one second content rating associated with the first advertisement from at least one rater, at least one of the at least one rater being an end user, Determining a comprehensive evaluation of the first advertisement based on the evaluation information, Receiving a request for electronic delivery of one or more advertisements, the request being associated with a concept, Selecting an advertisement from a database comprising the first advertisement and one or more second advertisements, each of the one or more second advertisements comprising an image and a comprehensive rating; and Electronically delivering the selected advertisement based on the request How to include. The method of claim 62, Distributing the first advertisement to the end user, the first advertisement including a feedback link; and Receiving the end user content rating via the feedback link. As a method of approving an electronic document, Processing the electronic document for distribution, the electronic document including a feedback mechanism that allows recipients to evaluate the electronic document; Receiving a plurality of requests for the document, each request being received from one of the plurality of end users; Delivering the electronic document to the plurality of end users based on the corresponding plurality of requests, and Approving the electronic document for distribution without receiving more than a threshold from the end users based on delivering the electronic document to a predetermined number of end users of the plurality of end users. How to include. 65. The method of claim 64, The threshold of feedback is zero feedback.

Images