WO2018119149A1 - Electronic product testing systems - Google Patents

Electronic product testing systems Download PDF

Info

Publication number
WO2018119149A1
WO2018119149A1 PCT/US2017/067732 US2017067732W WO2018119149A1 WO 2018119149 A1 WO2018119149 A1 WO 2018119149A1 US 2017067732 W US2017067732 W US 2017067732W WO 2018119149 A1 WO2018119149 A1 WO 2018119149A1
Authority
WO
WIPO (PCT)
Prior art keywords
product
testing
tester
testers
product testing
Prior art date
Application number
PCT/US2017/067732
Other languages
English (en)
French (fr)
Inventor
Maciej GRYKA
Russell Howard SMITH
Frederick Henry STEVENS-SMITH
Original Assignee
Rainforest Qa, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/784,064 external-priority patent/US11392277B2/en
Application filed by Rainforest Qa, Inc. filed Critical Rainforest Qa, Inc.
Priority to EP17832139.4A priority Critical patent/EP3559887A1/en
Publication of WO2018119149A1 publication Critical patent/WO2018119149A1/en
Priority to IL267368A priority patent/IL267368B2/he

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01R—MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00—Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/28—Testing of electronic circuits, e.g. by signal tracer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01R—MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00—Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01R—MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00—Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/01—Subjecting similar articles in turn to test, e.g. "go/no-go" tests in mass production; Testing objects at points as they pass through a testing station
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01R—MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00—Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/12—Testing dielectric strength or breakdown voltage ; Testing or monitoring effectiveness or level of insulation, e.g. of a cable or of an apparatus, for example using partial discharge measurements; Electrostatic testing
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01R—MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00—Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/12—Testing dielectric strength or breakdown voltage ; Testing or monitoring effectiveness or level of insulation, e.g. of a cable or of an apparatus, for example using partial discharge measurements; Electrostatic testing
    • G01R31/18—Subjecting similar articles in turn to test, e.g. go/no-go tests in mass production
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01R—MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00—Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/28—Testing of electronic circuits, e.g. by signal tracer
    • G01R31/30—Marginal testing, e.g. by varying supply voltage
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01R—MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00—Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/28—Testing of electronic circuits, e.g. by signal tracer
    • G01R31/316—Testing of analog circuits
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01R—MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00—Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/28—Testing of electronic circuits, e.g. by signal tracer
    • G01R31/3167—Testing of combined analog and digital circuits
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01R—MEASURING ELECTRIC VARIABLES; MEASURING MAGNETIC VARIABLES
    • G01R31/00—Arrangements for testing electric properties; Arrangements for locating electric faults; Arrangements for electrical testing characterised by what is being tested not provided for elsewhere
    • G01R31/28—Testing of electronic circuits, e.g. by signal tracer
    • G01R31/317—Testing of digital circuits
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201—Market modelling; Market analysis; Collecting market data
    • G06Q30/0203—Market surveys; Market polls

Definitions

  • This application relates generally to product testing, and more specifically, to systems and methods for providing product testing.
  • An electronic product testing system includes: a testing device configured to generate and to provide a digital image that includes a feature of a product being tested, and to provide product testing instruction for the product; and a tracking device configured for tracking a cursor movement with respect to the digital image, and/or a selection of an item in the digital image, and to generate product testing data based on a result of the tracking; wherein the tracking device is also configured to output the product testing data for storage in a non-transitory medium, the non-transitory medium having a database structure that is configured to store the product testing data in association with the digital image and the product testing instruction.
  • the tracking device is configured to track both the cursor movement and the selection of the item.
  • the product testing system also includes a graphic generator configured to generate a graphic representing the tracked movement.
  • the graphic comprises a line traversing a plurality of cursor positions.
  • the graphic further comprises a selection identifier identifying a position of the selection of the item.
  • the non-transitory medium comprises a database configured to store a file having a plurality of cursor positions as the product testing data, wherein the cursor positions is retrievable from the database for presentation over the digital image.
  • the database is configured to store the cursor positions in association with an identity of a product tester.
  • the item comprises a tab, a button, an icon, or a text, or any combination of the foregoing; and wherein the tracking device is configured to track a selection of the tab, a selection of the button, a selection of the icon, a selection of the text, or any combination of the foregoing.
  • the product testing system also includes a video generator configured to generate a video of the tracked interaction.
  • the testing device is further configured to present a question after a task has been performed in accordance with the product testing instruction.
  • the non-transitory medium is configured to store an answer of the question in a test result file.
  • the product testing instruction comprises first instruction to perform a first task for testing the product.
  • the product testing instruction comprises second instruction to perform a second task for testing the product.
  • the product testing data indicates a first tracked interaction for the first task
  • the tracking device is also configured to generate additional product testing data indicating a second tracked interaction for the second task.
  • the product testing system also includes a testing monitoring module configured to monitor a progress of a product testing for the product.
  • the testing monitoring module is configured to determine a percentage of the product testing completed, a duration spent in the product testing, a number of steps taken in the product testing, time wasted in the product testing, time lapsed before a first link is clicked, or any combination of the foregoing.
  • the product testing system further includes a monitoring module configured to detect an occurrence of an event due to an interaction of a product tester with the digital image.
  • the testing device is configured to provide the digital image and the product testing instruction for allowing a first entity to perform product testing on the product, and also configured to provide the digital image and the product testing instruction for allowing a second entity to perform product testing on the product.
  • the product testing system also includes a test result retriever configured to obtain a first testing result for the product.
  • the test result retriever is also configured to obtain a second testing result for the product.
  • the product testing data comprises an interaction image indicating the tracked interaction, and wherein the product testing system further comprises a report generator configured to generate a report having the interaction image.
  • the product comprises a web page, a web site, a computer application, a mobile device application, or a processor application.
  • the testing device is configured to provide the digital image for display on a hand-held device.
  • the testing device is configured to provide the digital image for display on a computer screen or a laptop screen.
  • a system for providing a product testing includes: a notification generator configured to generate a notification to inform a plurality of product testers that a product is available for product testing, wherein the product is associated with a first party and the notification generator is associated with a second party that is not the same as the first party; a communication interface configured to receive an electronic reply from a device being used by a first product tester of the plurality of product testers through a network, the reply indicating that the first product tester will perform the product testing; a reply counter for updating a reply count in response to the received electronic reply; and a testing interface generator configured to generate a testing interface for use in a testing environment, the testing interface configured to present features of the product and product testing instruction for allowing the first product tester to perform the product testing on the product based on the product testing instruction, the product testing instruction providing a plurality of tasks for testing the product.
  • the communication interface is configured to receive one or more additional electronic replies from one or more additional product testers from the plurality of product testers until a prescribed number of electronic replies has been reached.
  • the prescribed number of electronic replies is three, four, five, six, or seven.
  • the prescribed number of electronic replies is two.
  • the product testing instruction is configured to prescribe one or more actions be performed on a web page or a mobile application.
  • the notification is for display in a web browser or in a field created by a product testing notification application.
  • the notification is for display in a user interface associated with a crowd-testers provider.
  • the electronic reply is generated using a web browser or a product testing notification application utilized by the first product tester.
  • the system further includes: a timer configured to determine an amount of time that has lapsed since a product testing project is assigned to the product tester; a time-comparator configured to compare the amount of time with a prescribed duration; and a session terminator configured to revoke the product testing project from the product tester if the amount of time is equal to, or exceeds, the prescribed duration.
  • the testing interface is configured to present an image of a web page, and wherein the product testing instruction is configured to instruct the first product tester to perform an action on the web page.
  • the testing interface is further configured to present a question for the first product tester to answer after the action has been performed based on the product testing instruction.
  • the system further includes a non-transitory medium for storing an answer of the question in a test result file.
  • the system further includes a tracker configured to track an interaction of the first product tester with the web page.
  • the tracker is configured to track a movement of a cursor operated by the first product tester.
  • the system further includes a graphic generator configured to generate a graphic representing the tracked movement.
  • the graphic comprises a line traversing a plurality of cursor positions.
  • the graphic further comprises a selection identifier identifying a position in the web page at which the first product tester performed a selection.
  • the system further includes an image database configured to store an electronic image in a non-transitory medium, the electronic image comprising the image of the web page and the graphic superimposed with the image of the web page.
  • the image database is configured to store the electronic image in association with an identity of the first product tester and in association with the product being tested.
  • the tracker is configured to track a selection of a tab, a selection of a button, a selection of an icon, a selection of a text, or any combination of the foregoing, performed by the first product tester.
  • the system further includes a video generator configured to generate a video of the tracked interaction.
  • the system further includes a filter configured to select the plurality of product testers from a list of available product testers for providing the notification.
  • the filter is configured to select the plurality of product testers based at least in part on respective scores of the plurality of product testers, respective ages of the plurality of product testers, respective language skills of the plurality of product testers, respective locations of the plurality of product testers, respective age groups of the plurality of product testers, respective industry expertise of the plurality of product testers, or any
  • the system further includes a testing monitoring module configured to monitor a progress of the product testing.
  • the testing monitoring module is configured to determine a percentage of the product testing completed, a duration spent by the first product tester, a number of steps taken by the first product tester, time wasted by the first product tester, time lapsed before a first link is clicked by the first product tester.
  • the testing interface is also configured to present the features of the product and the product testing instruction for allowing a second product tester to perform another product testing on the product based on the product testing instruction
  • the system further includes: a test result retrieval module configured to obtain a first testing result achieved by the first product tester completing the product testing, and to obtain a second testing result achieved by a second product tester completing the product testing; and an analyzer configured to analyze the first testing result and the second testing result to determine whether the first testing result, the second testing result, or both of the first testing result and the second testing result, are reliable or not.
  • a test result retrieval module configured to obtain a first testing result achieved by the first product tester completing the product testing, and to obtain a second testing result achieved by a second product tester completing the product testing
  • an analyzer configured to analyze the first testing result and the second testing result to determine whether the first testing result, the second testing result, or both of the first testing result and the second testing result, are reliable or not.
  • the product testers are not employees of the first party, and are not employee of the second party.
  • the product comprises a web page, a web site, a computer application, a mobile device application, or a processor application.
  • the testing interface is configured for display on a hand-held device.
  • the testing interface is configured for display on a computer screen or a laptop screen.
  • a processor-implemented method for providing a product testing includes: providing a notification to inform a plurality of product testers that a product is available for product testing, wherein the product is associated with a first party and the notification generator is associated with a second party that is not the same as the first party; receiving an electronic reply from a device being used by a first product tester of the plurality of product testers through a network, the reply indicating that the first product tester will perform the product testing; updating a reply count in response to the received electronic reply; and in response to the electronic reply, providing a testing environment that comprises a testing interface for the first product tester, the testing interface presenting features of the product and the product testing instruction for allowing the first product tester to perform the product testing on the product based on the product testing instruction, the product testing instruction providing a plurality of tasks for testing the product.
  • the method further includes receiving one or more additional electronic replies from one or more additional product testers from the plurality of product testers until a prescribed number of electronic replies has been reached.
  • the prescribed number of electronic replies is three, four, five, six, or seven.
  • the prescribed number of electronic replies is two.
  • the product testing instruction is configured to prescribe one or more actions be performed on a web page or a mobile application.
  • the notification is provided for display in a web browser or in a field created by a product testing notification application.
  • the notification is provided for display in a user interface associated with a crowd-testers provider.
  • the electronic reply is generated using a web browser or a product testing notification application utilized by the first product tester.
  • the method further includes: determining, using a timer, an amount of time that has lapsed since a product testing project is assigned to the product tester; comparing, using a time-comparator, the amount of time with a prescribed duration; and revoking the product testing project from the product tester if the amount of time is equal to, or exceeds, the prescribed duration.
  • the testing interface is configured to present an image of a web page, and wherein the product testing instruction is configured to instruct the first product tester to perform an action on the web page.
  • the testing interface is further configured to present a question for the first product tester to answer after the action has been performed based on the product testing instruction.
  • the method further includes storing an answer of the question in a test result file.
  • the method further includes tracking an interaction of the first product tester with the web page.
  • the act of tracking is performed by a tracker that tracks a movement of a cursor operated by the first product tester.
  • the method further includes generating, using a graphic generator, a graphic representing the tracked movement.
  • the graphic comprises a line traversing a plurality of cursor positions.
  • the graphic further comprises a selection identifier identifying a position in the web page at which the first product tester performed a selection.
  • the method further includes storing an electronic image in a non-transitory medium, the electronic image comprising the image of the web page and the graphic superimposed with the image of the web page.
  • the electronic image is stored in the non-transitory medium in association with an identity of the first product tester and in association with the product being tested.
  • the act of tracking comprises tracking a selection of a tab, a selection of a button, a selection of an icon, a selection of a text, or any combination of the foregoing, performed by the first product tester.
  • the method further includes generating, using a video generator, a video of the tracked interaction.
  • the method further includes selecting the plurality of product testers from a list of available product testers for providing the notification.
  • the act of selecting is performed based at least in part on respective scores of the plurality of product testers, respective ages of the plurality of product testers, respective language skills of the plurality of product testers, respective locations of the plurality of product testers, respective age groups of the plurality of product testers, respective industry expertise of the plurality of product testers, or any combination of the foregoing.
  • the method further includes monitoring a progress of the product testing by a testing monitoring module.
  • the testing monitoring module is configured to determine a percentage of the product testing completed, a duration spent by the first product tester, a number of steps taken by the first product tester, time wasted by the first product tester, time lapsed before a first link is clicked by the first product tester.
  • the method further includes providing a second product tester access to another testing environment for allowing the second product tester to perform another product testing on the product based on the product testing instruction.
  • the method further includes: obtaining a first testing result achieved by the first product tester completing the product testing; obtaining a second testing result achieved by a second product tester completing the product testing; and analyzing the first testing result and the second testing result to determine whether the first testing result, the second testing result, or both of the first testing result and the second testing result, are reliable or not.
  • the product testers are not employees of the first party, and are not employee of the second party.
  • the product comprises a web page, a web site, a computer application, a mobile device application, or a processor application.
  • the testing interface is configured for display on a hand-held device.
  • the testing interface is configured for display on a computer screen or a laptop screen.
  • a processor-readable medium includes a set of instructions, wherein an execution of the instructions by a processing unit will cause a method for providing a product testing to be performed, the method comprising: providing a notification to inform a plurality of product testers that a product is available for product testing, wherein the product is associated with a first party and the notification generator is associated with a second party that is not the same as the first party; receiving an electronic reply from a device being used by a first product tester of the plurality of product testers through a network, the reply indicating that the first product tester will perform the product testing; updating a reply count in response to the received electronic reply; and in response to the electronic reply, providing a testing environment that comprises a testing interface for the first product tester, the testing interface presenting features of the product and the product testing instruction for allowing the first product tester to perform the product testing on the product based on the product testing instruction, the product testing instruction providing a plurality of tasks for testing the product.
  • FIG. 1A illustrates a system for providing product testing.
  • FIG. 1 B illustrates an example of the system of FIG. 1 A.
  • FIG. 2A illustrates a method for providing product testing.
  • FIG. 2B illustrates a method of tracking a progress of a product testing session.
  • FIG. 3 illustrates an example of a user interface for providing a notification to a product tester.
  • FIG. 4 illustrates an example of a testing environment having a user interface for allowing a product tester to perform product testing.
  • FIG. 5 illustrates an example of a data structure for linking assigned product testing tasks with product testing questions, answers, and tracker identifiers.
  • FIG. 6 illustrates an example of an image capturing an interaction of a product tester with a product being tested.
  • FIG. 7 illustrates an example of a data structure for creating a graphic associated with an image of a web page being tested.
  • FIG. 8A-8C illustrates an example of a report summarizing a result of a product testing.
  • FIG. 9 illustrates an example of a user interface for allowing a user to create tasks for product testing and to define product testing parameters.
  • FIG. 10A-10E illustrate an example of a report that is provided to an administrator of the system of FIG. 1 A.
  • FIG. 11 is a block diagram of a specialized processing system.
  • FIG. 1A illustrates a system 10 for providing product testing.
  • the system 10 is configured to connect a party 12 that has developed a product with one or more product testers 14.
  • the product may be a web page, a web site, a computer application (e.g., an enterprise application), a mobile (e.g., handheld) device application, an application for a specialized processor (e.g., an image processor, a signal processor, a sound processor, a speech processor, a processor in a vehicle (such as a car, a plane, etc.), or any of other types of processor).
  • the product testers 14 may be individuals residing in different parts of the world.
  • the testers 14 may reside in different parts of a city, different cities, different states, and/or different countries.
  • the system 10 is associated with a party 16 that is different from the party 12.
  • the party 12 (first party) providing the product may be a first company
  • the party 16 (second party) interfacing with the party 12 and the product testers 14 may be a second company that is not the same as the first company.
  • the product testers 14 are not employees of the first party 12, and are not employees of the second party 16.
  • the product testers 14 may be individual contractors. In other embodiments, one or more of the product testers 14 may be employees of the party 16.
  • FIG. 1 B illustrates an example of the system 10 of FIG. 1 A.
  • the system 10 includes one or more input(s) 1002, a notification generator 1004, a communication interface 1008, a reply counter 1040, a comparator 1042, a filter 1050, a product tester management module 1080, a testing interface generator 1 100, a timer 1200, a time-comparator 1202, a tracker 1300, a graphic generator 1302, an image database 1304, a video generator 1310, a video database 1314, a user interface generator 1350, a testing monitoring module 1400, a test result retrieval module 1500, and an analyzer 1502. Details of these components will be described in further detail below. It should be noted that the system 10 is not limited to the example shown, and that the system 10 may have other
  • system 10 may not include one or more of the components shown in FIG. 1 B.
  • system 10 may be implemented using multiple devices, having different respect sets of one or more of the components shown in FIG. 1 B
  • the product testers 14 have respective memberships and accounts with the party 16. For example, in order for a person to qualify as a product tester for the party 16, the person may need to go through a registration process with the system 10 to become a product tester 14.
  • the product tester management module 1080 in the system 10 is configured to manage the various product testers 14.
  • the product tester management module 1080 may be configured to provide a registration interface to receive information regarding a person who desires to be a product tester 14, so that the person is registered as a product tester 14 in the system 10.
  • Each registered product tester 14 will have his / her login ID and password, which allow the product tester 14 to log into the system 10.
  • the product tester management module 1080 may also be configured to provide training on product testing to train the person so that the person can be qualified as a product tester 14. In some embodiments, the product tester management module 1080 also manages the accounts of the various product testers 14. For example, after a product tester 14 completes a product testing project, the product tester management module 1080 may calculate a fee owed to the product tester 14 based on one or more features of the product testing project. [01 1 1 ] In some embodiments, the product tester management module 1080 also manages the accounts of the various product testers 14. For example, after a product tester 14 completes a product testing project, the product tester management module 1080 may calculate a fee owed to the product tester 14 based on one or more features of the product testing project. In one
  • all of the product testers 14 registered with the system 10 have the same pay rate.
  • different product testers 14 may have different scores indicating their level of experiences, reliabilities, etc. In such cases, the product testers 14 may have different respective pay rates based on these metrics.
  • the first party 12 informs the second party 16 that its product needs to be tested.
  • the first party 12 may send a request to the system 10, and may transmit information regarding the product to be tested to the system 10.
  • the request and/or the information regarding the product may be received by the input(s) 1002 of the system 10.
  • the first party 12 may provide a link for the web page, product testing parameters, and product testing tasks.
  • the user interface generator 1350 in the system 10 may provide an user interface for allowing the first party 12 to define product testing parameters and product testing tasks. These information may also be received via the input(s) 1002 of the system 10.
  • the first party 12 may access an user interface provided by the user interface generator 1350 of the system 10 to generate a list of tasks for a product tester 14 to perform, and to create a list of questions corresponding to the respective tasks for the product tester to answer.
  • the user interface generator 1350 is configured to generate an image of the user interface for display on a device being used by the first party 12 when the first party 12 is on-line, and has logged into an account with the second party 16.
  • FIG. 2A illustrates a processor-implemented method 200 performed by the system 10 of FIG. 1 for providing product testing.
  • the method 200 may be described with reference to the system 10 of FIG. 1A and 1 B.
  • the system 10 provides a notification 1005 to inform a plurality of product testers 14 that a product is available for product testing (item 202).
  • the product is associated with the first party 12 and the notification generator 1004 is associated with a second party 16 that is not the same as the first party 14.
  • the notification 1005 provided by the system 10 is for display in a web browser.
  • the product tester 14 may open a web browser on his / her device.
  • the web browser may include an add-on feature, which allows the product tester 14 to receive the notification 1005 from the system 10. In such cases, if the product tester 14 is on-line, the product tester 14 will receive the notification 1005. On the other hand, if the product tester 14 is off-line, then the product tester 14 will not receive the notification 1005.
  • the notification 1005 is "broadcasted" by the system 10 in the sense that multiple product testers 14 will receive the notification 1005 or will have access to such notification 1005.
  • the notification 1005 may be for display in a field created by a product testing notification application.
  • the product tester 14 may install a product testing notification application onto his / her device, which allows the product tester 14 to receive the notification 1005 from the system 10.
  • the product testing notification application may be provided by the second party 16 for installation or downloading onto a device of a product tester 14.
  • the device being used by the product tester 14 may be a laptop, a desktop computer, or a handheld device (e.g. , an iPhone, iPad, iPad mini, tablet, etc.).
  • the notification 1005 may be sent to a job listing agent, or a crowd-testers provider, such as crowdflower or a mechanical turk, etc.
  • the term "notification” may refer to any information for informing a product tester 14, wherein such information may be sent directly to a user interface on a device being used by the product tester 14, or to a job listing agent for allowing a product tester 14 to view.
  • the notification may actively inform the product tester 14, or may passively inform the product tester 14 (in that the notification may be posted at a location at which the product tester 14 may search or discover such notification).
  • FIG. 3 illustrates an example of a user interface 300 for providing a notification to a product tester 14.
  • the user interface 300 includes a field 302 for displaying a notification 304 provided by the system 10, indicating that a product testing project is available.
  • the user interface 300 also includes a feature 306 for allowing a product tester 14 to accept the product testing project.
  • the user interface 300 for providing the notification to the product tester 14 is not limited to the example described previously, and that the user interface 300 may have other configurations in other embodiments.
  • an electronic reply 1010 is received from a device being used by a product tester 14 of the plurality of product testers through a network (item 204).
  • item 204 may be performed by the
  • the reply 1010 indicates that the product tester 14 will perform the product testing.
  • the network transmitting the electronic reply may be any network, such as the Internet, a
  • the electronic reply 1010 may be generated using a web browser utilized by the product tester 14.
  • the web browser on a device utilized by the product tester 14 may include an add-on feature, which allows the product tester 14 to receive the notification 1005 (e.g., a ping, a message, a posting, etc.) from the system 10, like the example shown in FIG. 3.
  • the same add-on feature may also allow the product tester 14 to provide his / her reply (for reception by the system 10) to accept or to sign up for a product testing project.
  • the electronic reply 1010 may be generated using a product testing notification application utilized by the product tester 14.
  • the product tester 14 may install a product testing notification
  • the product testing notification application onto his / her device, which allows the product tester 14 to receive the notification 1005 from the system 10.
  • the product testing notification application may be provided by the second party 16.
  • the same product testing notification application may also allow the product tester 14 to provide his / her reply (for reception by the system 10) to accept or to sign up for a product testing project.
  • a product tester 14 may use crowdflower or a mechanical turk to obtain the notification 1005 regarding the product testing project.
  • the reply 1010 accepting the product testing project will be generated by a device using the crowdflower or mechanical turk.
  • the method 200 further includes updating a reply count in response to the received electronic reply (item 205).
  • item 205 is performed by the reply counter 1040 of the system 10.
  • the method 200 may also include receiving additional electronic reply / replies 1010, keeping track with a number of replies 1010, and comparing the number of replies 1010 with a prescribed number.
  • the communication interface 1008 of the system 10 is configured to receive one or more additional electronic replies 1010 from one or more additional product testers 14 from the plurality of product testers 14 until a prescribed number of electronic replies 1010 has been reached.
  • the reply counter 1040 in the system 10 will increment the reply count by one. In such cases, after the communication interface 1008 received the first electronic reply 1010 from the product tester 14, the product testing project is then assigned only to the product tester 14, and not to any other product testers 14.
  • the communication interface 1008 continues to receive additional reply until the total number of replies 1010 is equal to the prescribed number of electronic replies 1010, as determined by the comparator 1042 in the system 10.
  • the prescribed number of reply 1010 is one reply. In other embodiments, the prescribed number of replies 1010 may be more than one. For example, in some embodiments, the prescribed number of electronic replies 1010 may be three, four, five, six, or seven. In other embodiments, the prescribed number of electronic replies 1010 may two. In further embodiments, the prescribed number of electronic replies 1010 may be more than seven.
  • the system 10 then provides a testing environment that comprises a testing interface for use by the product tester 14 (item 206).
  • the testing interface may be generated by the testing interface generator 1 100 in the system 10.
  • item 206 may be performed by the system 10 providing the testing interface for display on a device being used by the product tester 14 after the product tester 14 has accepted the product testing project.
  • the generated testing interface may be output to a network (e.g., the Internet) for transmission to the device of the product tester 14.
  • item 206 may be performed by the testing interface generator 1 100, which opens up a virtual machine environment for presenting the testing interface for the product tester 14 in response to the product tester 14 accepting to take the product testing project.
  • the testing interface is configured to present features of the product and the product testing instruction for allowing the first product tester 14 to perform the product testing on the product based on the product testing instruction.
  • the product testing instruction provides a plurality of tasks for the product tester 14 to perform for testing the product.
  • the product being tested comprises a web page
  • the product testing instruction is configured to prescribe one or more actions be performed on the web page being tested.
  • FIG. 4 illustrates an example of a testing environment 400 having a testing interface 402 for allowing a product tester 14 to perform product testing.
  • the testing interface 402 includes a window 404 for presenting features of the product being tested.
  • the product comprises a web page.
  • the window 404 may allow the product tester 14 to access the web page via a link.
  • the window 404 presents an image of a web page 410 for allowing the product tester 14 to perform product testing thereon.
  • the web page 410 displayed in the window 404 includes one or more selectable features 41 1 .
  • the testing interface 402 includes a field 420 providing one or more tasks 422 for the product tester 14 to perform on the web page 410.
  • the tasks 422 are parts of a product testing instruction.
  • a task 422 may require the product tester 14 to perform an action on the image of the web page 410.
  • a task 422 may be to click on a tab on the web page 410, to click on a link on the web page 410, to enter a value in a field in the web page 410, to click on a box in the web page 410, etc.
  • the testing interface 402 may be configured to present a question 412 for the product tester 14 to answer after the action has been performed based on the product testing instruction.
  • the question 412 may ask the product tester 14 whether the product tester 14 sees a pop-up after selecting an object in the web page 410 being tested, whether the product tester 14 sees a new image after selecting a certain option in the web page 410, etc.
  • the testing interface 402 also provides only two possible answers 414a, 414b ("Yes" and "No") for allowing the product tester 14 to select. Such technique reduces ambiguity in the answer provided by the product tester 14.
  • the testing interface 402 may provide more than two possible answers 414 for allowing the product tester 14 to select to answer a question baser the product tester 14 performs a task specified in the product testing instruction.
  • the system 10 if the product tester 14 selects the "No" answer to any of the product testing question, the system 10 then terminates the product testing session, and the product testing is considered completed by the product tester 14. In such cases, no further product testing tasks and no further product testing questions will be presented to the product tester 14. This is because if the product being tested has a certain feature that fails, then the product provider may want to know this right away. Also, the failed feature may impact other features of the product. Accordingly, if a feature in the product has failed the product testing, then one or more other features may fail as well. In other embodiments, instead of terminating the product testing session, the system 10 may allow the product tester 14 to continue testing the product. For example, the system 10 may allow the product tester 14 to continue with the remaining tasks in the product testing prescribed by the product testing instruction. Also, the system 10 may allow the product tester 14 to test other features of the product that are not associated with the failed feature.
  • the method 200 further includes storing an answer of the question in a test result file for the product being tested.
  • the answer may be a "Yes" answer or a "No” answer in response to a question provided to the product tester 14 after the product tester 14 has performed a task to test the product.
  • the answer, the question, and the associated task for testing the product may be stored in a unique data structure that associate these items together.
  • FIG. 5 illustrates an example of a data structure 500 for linking product testing tasks 502, respective product testing questions 504, and the corresponding answers 506 to the product testing questions 504.
  • the data structure 500 also includes tracker identifiers 508 that identify files tracking interaction of the product tester 14 while testing the product.
  • Each tracker identifier 508 may identify an image, a video file, or any of other interaction information that captures the interaction of the product tester 14 on the product being tested. These information are advantageous because they allow a user (from party 12 or party 16) to retroactively determine how a product tester 14 came to a conclusion based on the product tester's 14 interaction with the product being tested.
  • the tracker identifier 508 will be described in further detail below.
  • the testing interface 402 is configured for display on a computer screen or a laptop screen. In other embodiments, the testing interface 402 is configured for display on a hand-held device (e.g. , an iPhone, iPad, iPad mini, tablet, etc.).
  • a hand-held device e.g. , an iPhone, iPad, iPad mini, tablet, etc.
  • testing interface 402 is not limited to the example discussed previously, and that the testing interface 402 may have other configurations in other embodiments.
  • the timer 1200 in the system 10 keeps track of the duration that has lapsed since the providing of the testing interface 402.
  • the method 200 may further include determining (by the timer 1200) an amount of time that has lapsed since the testing interface has been provided to the product tester 14.
  • the method 200 may also include comparing the amount of time with a prescribed duration using the time-comparator 1202. If the lapsed time has exceed a the prescribed duration (e.g. , a starting deadline), and the product tester 14 has not begun performing a product testing task, then the system 10 will revoke the product testing project from the product tester 14.
  • a prescribed duration e.g. , a starting deadline
  • the system 10 may terminate the product testing session being used by the product tester 14.
  • the prescribed duration (starting deadline) may be set by the system 10 to be five minutes after a product testing session has begun.
  • the prescribed duration may be set by an administrator of the system 10 to have other values, such as a duration that is more than five minutes, or less than five minutes.
  • a method 230 of tracking a progress of a product testing session may be provided (FIG. 2B). The method 230 may be a part of the method 200 in some embodiments.
  • the method 230 includes determining an amount of time that has lapsed since a beginning of a product testing session using a timer (item 232), comparing the amount of time with a prescribed duration (item 234), and revoking the product testing project from the product tester if the determined time equals to or exceeds the prescribed duration (item 236).
  • the method 200 may also include terminating the product testing session.
  • item 236 may be performed by the testing interface generator 1 100, which closes a virtual machine environment presenting the testing interface for the product tester 14 in response to the lapsed time meeting the prescribed duration.
  • the method 200 further includes tracking an interaction of the product tester 14 with the web page.
  • the product tester 14 may be using his / her device (e.g., laptop, desktop, handheld device, such as a cell phone, etc.) to perform the product testing.
  • the tracker 1300 of the system 10 is configured to track a position of the cursor in the product tester's device as operated by the product tester 14.
  • the tracker 1300 may be configured to track a position of the finger position as applied on the touchscreen.
  • the term “cursor” is not limited to a displayed item operated using a touchpad, a mouse, or a knob, and the term “cursor” may also include non-displayed item that is associated with a movement of a finger as applied on a touchscreen.
  • the testing interface 402 may instruct the product tester 14 to click on a certain item on an image of a web page being tested. While the product tester 14 is manipulating the cursor in the web page, the tracker 1300 of the system 10 records the positions of the cursor as it appears in the web page.
  • the system 10 may also use the graphic generator 1302 configured to generate a graphic representing the tracked movement.
  • the graphic comprises a line traversing a plurality of cursor positions.
  • the graphic may further comprise a selection identifier identifying a position in the image of the web page at which the product tester 14 performed a selection (e.g., a clicking of a button).
  • the selection may be a selection of a tab, a selection of a button, a selection of an icon, a selection of a text, or any combination of the foregoing, performed by the product tester 14.
  • FIG. 6 illustrates an example of an image 600 capturing an interaction of a product tester with a product being tested.
  • the image 600 includes an image of a web page 602 being tested, the task 604 prescribed by the product testing instruction, and the question 606 corresponding to the task 604.
  • the task 604 instructs the product tester 14 to click on the "Archive" button, and the question 606 asks the product tester 14 whether the clicked item moved to the archived section.
  • the image 600 also shows a graphic 608 generated by the graphic generator 1302.
  • the graphic 608 is in a form of a line, which indicates where the product tester 14 has moved the cursor while performing the task 604.
  • the graphic 608 also includes a plurality of circles 610 (example of selection identifiers) identifying a position in the image of the web page at which the product tester 14 has clicked on a button.
  • the method 200 may further include storing the electronic image 600 in a non-transitory medium 1304 (FIG. 1 B).
  • the non- transitory medium 1304 is illustrated as an image database, but in other embodiments, the non-transitory medium 1304 may be any memory or storage device.
  • the image 200 includes the image of the web page 602 and the graphic 608 superimposed with the image of the web page 602.
  • the system 10 is configured to generate positional data regarding the position of the cursor as the product tester 14 interacts with the web page 602 to perform the assigned task.
  • the positional data may include a position (e.g. , X and Y coordinates) of the cursor.
  • the positional data may be later used by the graphic generator to generate the graphic 608. For example, all of the positional data may be plotted to create the line shown in the example of the graphic 608 in FIG. 6. Also, in some combination
  • the system 10 may also generate timing data that are associated with the respective positions of the cursor. For example, cursor position (X1 , Y1 ) may occur at t1 , and cursor position (X2, Y2) may occur at t2. Thus, the system 10 may generate the timing data t1 , t2, etc. , and store such timing data in association with the respective cursor positions in the non-transitory medium.
  • FIG. 7 illustrates an example of a data structure 700 for storing cursor position data in association with timing data.
  • the data structure 700 includes two columns for positional values in the x-coordinate and y-coordinate, respectively.
  • the data structure 700 also includes a column for indicating whether the product tester 14 made a selection (e.g., by clicking a button) at a certain cursor position.
  • the data structure 700 may be used by the graphic generator 1302 to create the graphic 608 that is associated with an image of a web page being tested.
  • the positional values in the data structure 700 may be used to create the lines in the graphic 608, and the select
  • the indicator(s) in the data structure 700 may be used to create the circle(s) 610 representing the position(s) at which the product tester 14 has made a selection.
  • the data structure 700 may be stored in the non-transitory medium 1304.
  • the data structure 700 may be linked with a certain tracker identifier in the data structure 500 of FIG. 5.
  • the data structure 700 with the identifier "23" may contain data for creating the graphic 608 in the image file 342 in the data structure 500.
  • the image file 342 may contain an image of the web page being tested.
  • the data structure 700 may also include timing data indicating the respective times at which the positions of the cursor occurred.
  • the timing data is advantageous because it may allow the graphic 608 to be presented in a form of a video.
  • the video generator 1310 may utilize the data structure 700 to create a video showing a changing of the position of the cursor over time.
  • the video will indicate a tracked interaction of the product tester 14 with the web page being tested.
  • the video provided by the video generator 1310 may include a moving cursor.
  • the video provided by the video generator 1310 may include a growing line representing a collection of positional points by the cursor as the cursor is moved across the web page.
  • the video may be stored in a non-transitory medium 1314 (FIG. 1 B).
  • the non-transitory medium 1314 is illustrated as a video database, but in other embodiments, the non- transitory medium 1314 may be any memory or storage device. Also, in other embodiments, instead of having separate non-transitory media 1304, 1314 as image database and video database, the non-transitory media 1304, 1314 may be integrated as a single database.
  • the graphic 608 and the image of the web page being tested may be provided by the system 10 to a user associated with the first party 12.
  • the user interface provided by the user interface generator 1350 may be configured to output an image and/or a video having the graphic 608 and the image of the web page for viewing by the user of the system 10.
  • image and/or video may be provided as a part of a test result file so the user can see how the product tester 14 came to a certain product testing conclusion based on the interaction with the web page tested.
  • the image and/or the video may be stored in association with the corresponding product testing task defined in the product testing instruction.
  • the image and/or the video is for tracking the interaction of the product tester 14 while the product tester 14 is performing a certain task prescribed in the product testing instruction. Accordingly, each of the task prescribed to be performed by the product tester 14 will have a
  • the system 10 will have generated 28 images and/or 28 videos for the respective 28 tasks.
  • all of the 28 images and/or the 28 videos in the example may be provided, or made available, to the user who is associated with the first party 12 (the provider of the product).
  • only one image and/or one video may be provided, or made available, to the user who is associated with the first party 12.
  • the system 10 may provide only the image and/or only the video associated with the failed task. This may be advantageous because the user may not be concerned with tasks that have "pass" status, but may be more interested in viewing results for the task that results in a "failed" status.
  • the system 10 may store only the image and/or only the video that is associated with a failed task, and may delete other images and videos for other tasks that have "pass" status after the product testing project is completed or after the product testing project is completed for a certain pre-determined duration.
  • FIG. 8A-8C illustrates an example of a report 800 summarizing a result of a product testing.
  • the system 10 is configured to gather information from the finished product testing by one or more product testers 14, and generate the report 800 based on the gathered information.
  • the report 800 is for a certain product that has been tested by the product testers 14.
  • the report 800 includes a list of product testing tasks 802 performed by the product testers 14.
  • the tasks 802 in the report 800 are the same tasks created (using the user interface provided by the user interface generator 1350) by the user associated with the first party 12 that provides the product for product testing.
  • the report 800 also includes failed test identifier(s) 804 and passed test identifier(s) 806 next to different respective tasks.
  • a task with a failed test identifier 804 indicates that the product testing failed when the product tester(s) performed that task.
  • a task with a passed test identifier 806 indicates that the product testing passed when the product tester(s) 14 performed that task.
  • the report 800 also includes an object 808 next to each of the tasks, which when selected by the user, will provide further details about the product testing on that specific task.
  • the report 800 also includes a list of browsers 810 in which the product testing of the product has been performed.
  • the user viewing the report 800 may be interested in viewing details of the product testing for a certain task.
  • the user may select the corresponding object 808 next to the task 802.
  • the system 10 may present the report 820 shown in FIG. 8B for the user.
  • the report 820 is for a certain task performed during the product testing.
  • the report 820 includes a summary 821 of the product testing for that task.
  • the summary 821 indicates that the specific task in question failed when the product testers 14 perform the product testing in 9 out of 31 browsers.
  • the report 820 also shows the corresponding task 822 performed by the product testers 14 when the product testing failed, and the corresponding question 824.
  • the 9 browsers in which the performance of the task failed are shown by the browser icons 826.
  • the report 820 also includes pass/fail indicator 828 next to the different respective browser icons 826.
  • a failed indicator 828 next to a browser icon 826 indicates that the performance of the task failed when the product testers 14 performed the task in the type of browser represented by the browser icon 826.
  • the user viewing the report 820 has selected one of the pass/fail indicators 828 for a certain browser, as represented by the selector graphic 829. Accordingly, the report 820 displays three images 830a-830c that were captured when three respective different product testers 14 performed the task 822 in the browser type selected. From the images 830, the user viewing the report 820 can see what were on the product testers' 14 screen when they performed that specific task that result in a "failed" status.
  • the user viewing the report 820 can select one of the images 830a- 830c to see more detail.
  • the system 10 may provide a report 840 shown in FIG. 8C to the user.
  • the report 840 includes a description of the task 842, and the
  • the report 840 also includes an image 846 that includes an image 847 of the web page tested, and a graphic 848.
  • the graphic 848 includes a line 849 that tracks the position of the cursor when the product tester 14 interacted with the web page, and selection identifiers 850 indicating the positions where the product tester 14 made selections (e.g., clicked a button).
  • the features in the image 846 may be provided by the graphic generator 1302 of the system 10, as similarly discussed with reference to FIG. 6. Thus, image 846 allows the user to retroactively view the interaction of the product tester 14 with the web page while performing the task 842.
  • the system 10 may provide a video showing a movement of the cursor and selection(s) (e.g. , clicking of button) by the product tester 14.
  • the report 840 further includes an identifier of the product tester 14 who performed the product testing task.
  • the identifier may be the actual name of the product tester 14 or other identifier that identifies the product tester 14. Also, in some embodiments, the identity of the product tester 14 may be kept confidential and may not be presented by the system 10 to the user.
  • the report 840 further includes a browser identifier 856 identifying the type of browser in which the task was conducted by the product tester 14, and a pass / fail indicator 854 indicating whether the task in question failed or passed during the product testing.
  • the test result reports 800, 820, 840 may be generated using the test result retrieval module 1500 in the system 10.
  • the test result retrieval module 1500 gather information regarding the product testing, and create the reports 800, 820, 840 for viewing by the user.
  • the reports 800, 820, 840 may be presented through an user interface created by the user interface generator 1350 of the system 10.
  • the user interface generator 1350 may generate and provide an user interface for use by an administrator of the system 10. In such cases, the reports 800, 820, 840 may be provided for viewing by the administrator through the user interface.
  • the user interface generator 1350 may generate and provide an user interface for use by a user associated with the first party 12 providing the product for product testing.
  • the reports 800, 820, 840 may be provided for viewing by the user through the user interface.
  • test result reports 800, 820, 840 are not limited by the examples described previously, and that the system 10 may provide test result report(s) having different configurations in different
  • the reports 800, 820, 840 may have fewer information than those described in the above example.
  • the system 10 may consolidate the product testing results from the multiple product testers 14, and provide a consolidated product testing report to the user.
  • the analyzer 1502 in the system 10 may employ a logical algorithm to determine whether a certain product testing result from a product tester 14 is reliable or not. If the testing result is considered as non-reliable, the analyzer 1502 then disregards that particular product testing result.
  • the analyzer 1502 may compare two product testing results from two different product testers 14. If the two product testing results indicate the same conclusion (e.g. , pass or fail), then the analyzer 1502 may determine the product testing results as reliable. In such cases, the system 10 may simply report to the user that the particular task in question has a pass or fail status without informing the user the number of product testers 14 performed the testing.
  • the analyzer 1502 may pick a result (e.g. , pass or fail) that has the majority vote. For example, if two product testers 14 performing the task provide a "Yes" answer to the corresponding question, and the third product tester 14 provides a "No" answer, then the analyzer 1502 will pick the "Yes” answer and may determine that the testing of the task has a "pass" status.
  • a result e.g. , pass or fail
  • the analyzer 1502 will pick the "No" answer and may determine that the testing of the task has a "fail” status.
  • the system 10 may be configured to provide a user interface for allowing a user to override a product testing result.
  • the user interface providing the report 800/820/840 may also provide a control for allowing a user to change a product testing result. If a product testing result indicates that a particular task has a "pass" status, the user may utilize the control to change the status from "pass" to "failed". Alternatively, if a product testing result indicates that a particular task has a "failed” status, the user may utilize the control to change the status from "failed" to "pass".
  • the analyzer 1502 of the system 10 is not limited to comparing product testing results. In other embodiments, the analyzer 1502 may also perform other analyses to determine different user metrics associated with the product testers 14 and/or different testing metrics relating to the reliability of the test results. For example, in some embodiments, the analyzer 1502 may compare the images (which tracked the actions of the product testers 14) stored in the image database 1304 to determine whether the three images are similar or not. If all three product testers 14 perform the same action prescribed by the task of the product testing instruction, the graphics representing the movements of the cursors in the three images may have some similar features.
  • the graphics representing the movement of the cursors should all include one or more selection identifiers at the link. If one of the graphics indicate that there is no selection of the link, the analyzer 1502 may then determine that the product testing result by the corresponding product tester 14 is not reliable. This may be the case even if this product tester 14 provided the same "Yes” or "No" answer to reach the same conclusion as that of the other two product testers 14.
  • the testing monitoring module 1400 of the system 10 may gather some intelligence regarding the performance of the product testers 14. After the product testing is finished, the analyzer 1502 may then retrieve the information from the testing monitoring module 1400 and perform further analysis based on the information. For example, the testing monitoring module 1400 may determine an amount of time for each product tester 14 to perform a certain task. Thus, after the product tester 14 finishes the product testing project, the testing monitoring module 1400 will have multiple time values for different respective tasks prescribed in the product testing instruction and performed by that product tester 14. The testing monitoring module 1400 also does the same for the other two product testers 14.
  • the analyzer 1502 may then compare the three time values associated with the respective three product testers 14 for a given task, and determine if any of the product testing results for that task from the three product testers 14 is unreliable. For example, if a certain task takes a first product tester 14 to complete in 30 seconds, takes a second product tester 14 to complete in 37 seconds, and a third product tester 14 to complete in 4 seconds, then then analyzer 1502 may determine that the product testing result from the third product tester 14 is unreliable because the other two time values are closer to each other, and the third time value is different from both of the first two time values.
  • the system 10 may insert one or more questions specifically designed to test the reliability of the product testers 14. For example, the system 10 may ask the product tester 14 a simple mathematical question, such as, "what is 14+6?" that is completely unrelated to the nature of the product being tested. If the product tester 14 is actively paying attention while performing the product testing, the product tester 14 is most likely to answer this question correctly. On the other hand, if the product tester 14 is not paying attention while performing the product testing, the product tester 14 may answer this question incorrectly, thereby indicating that the product testing result from this product tester 14 may be unreliable.
  • This technique also has the benefit of detecting certain product tester 14 who may be earless, reckless, or who pretends to be performing the product testing.
  • the testing monitoring module 1400 may be configured to compare an answer provided by the product tester 14 with an expected answer. If the answer provided by the product tester 14 matches the expected answer, then the testing monitoring module 1400 may determine that the product testing result by that product tester 14 is reliable. On the other hand, if the answer provided by the product tester 14 does not match the expected answer, then the testing monitoring module 1400 may determine that the product testing result by that product tester 14 is not reliable.
  • the testing monitoring module 1400 may also perform other functions to monitor a progress of the product testing.
  • the testing monitoring module 1400 may be configured to determine a percentage of the product testing completed, a duration spent by the product tester 14 on the product testing, a number of steps taken by the product tester 14, time wasted by the product tester 14, time lapsed before a link is clicked by the product tester 14, etc.
  • the user interface generator 1350 is configured to provide an interface for a user to generate tasks for product testing and to define product testing parameters.
  • FIG. 9 illustrates an example of a user interface 900 for allowing a user to generate tasks for product testing and to define product testing parameters.
  • the user may be associated with the first party 12 providing the product for product testing, and/or a person associated with the second party 16 providing the system 10.
  • the user interface 900 may be generated and provided by the user interface generator 1350 in the system 10 in some embodiments.
  • a user at the first party 12 may then log into the system 10 (e.g., using user ID and password).
  • the user interface generator 1350 of the system 10 may then provide the user interface 900 for the user.
  • the user interface 900 includes a section 902 for allowing a user to create product testing tasks 904 and corresponding product testing questions 906.
  • Each product testing task 904 prescribes a product tester 14 to perform an action on the product being tested.
  • Each corresponding question 906 asks the product tester 14 to confirm whether a certain event occurs or not based on the action performed by the product tester 14 in accordance with the assigned task. Accordingly, each of the product testing tasks 904 and each of the product testing questions 906 are designed by the user to test specific features in the product.
  • the user may use the user interface 900 to add, delete, edit, and move task(s) 904 and question(s) 906.
  • the user interface 900 also includes a section 907 having a plurality of platform / browser icons 908 for selection by the user creating the product testing tasks.
  • Each of the platform / browser icons 908 represents a type of platform or browser in which the web page is desired to be tested. If the user wants the web page to be tested under 34 different types of platform / browser, for example, the user can select the 34 corresponding platform / browser icons 908 to prescribe such testing parameters.
  • the platform / browser icons 908 may identify Google Chrome, different versions of Chrome, different versions of Mozilla Firefox, different versions of Microsoft Internet Explorer, different iOS for different versions of iPhone, different versions of Safari, different versions of Android mobile browsers, different versions of iPhone 6 mobile browsers, different versions of Microsoft Office, etc.
  • the user interface 900 may allow the user to prescribe how to test an application on a mobile device (e.g., iPhone, iPad, tablet, etc.). For example, the user may prescribe that a product testing be performed on a certain type of mobile device by using the mobile device to launch a browser to go to a website. Alternatively, the user may prescribe that the product testing be performed using mobile-native-device (through an actual mobile device) or mobile-simulator (simulated mobile environment).
  • a mobile device e.g., iPhone, iPad, tablet, etc.
  • the user may prescribe that a product testing be performed on a certain type of mobile device by using the mobile device to launch a browser to go to a website.
  • the user may prescribe that the product testing be performed using mobile-native-device (through an actual mobile device) or mobile-simulator (simulated mobile environment).
  • the user designing the product testing instruction may group multiple related web pages under one product testing project.
  • the product being tested may have multiple web pages, and the user may divide the product into different features for different product testing projects.
  • the user may save these information to generate a product testing instruction file.
  • the product testing instruction file may be saved in a non-transitory medium (e.g. , the product testing instruction database 1600) at the system 10.
  • the notification generator 1004 of the system 10 then provides the notification 1005 to inform the product testers 14 that the product testing project is available, as discussed.
  • the testing interface generator 1 100 of the system 10 When a product tester 14 has accepted the product testing project, the testing interface generator 1 100 of the system 10 then provides a testing interface, along with the tasks and questions defined in the product testing instruction (stored in the database 1600) for the product tester 14 to conduct the product testing. Also, as discussed, in some embodiments, the system 10 may insert one or more questions into the product testing instruction specifically designed to test the reliability of the product testers 14. For example, the system 10 may ask the product tester 14 a simple mathematical question, such as, "what is 14+6?” that is completely unrelated to the nature of the product being tested, and that is designed to determine reliability of the product testing result.
  • the system 10 may be configured to provide an administrative product testing report for use by personnel at the second party 16. Such report provides a user at the second party 16 a detailed view of all of the tasks in a product testing.
  • the report may be generated using the test result retrieval module 1500 (which gathers and compiled all of the relevant
  • FIGS. 10A-10E illustrate an example of a report 1000 that is for viewing by a user at the second party 16.
  • the report 1000 includes data 1002 regarding the product testing performed, including name of client (product provider), product testing identification, type of platform / browser in which the product testing was conducted, etc.
  • the report 1000 also includes the identifications 1004a-1004c of three product testers 14 who performed the product testing.
  • the report 1000 also includes a summary of the product testing organized by tasks. For example, as shown in FIG. 10B, the report 1000 includes a section that provides the product testing task 1010 and its
  • the report 1000 also includes respective results 1014a-1014c associated with the respective product testers 14.
  • Each result 1014 indicates the product tester's 14 answer 1020 to the product testing question, a pass/fail indicator 1022 indicating whether the performance of the task passes or not, a time value 1024 indicating an amount of time it took for the product tester 14 to complete the task 1010, and an image 1026 that tracked the action of the product tester 14 while performing the task 1010.
  • the image 1026 may be generated by the graphic generator 1302 as similarly discussed previously. In some cases, the user may click on the image 1026, and the system 10 may then provide a more detailed view of the image 1026 (FIG. 10C).
  • the image 1026 includes a graphic that indicates the action of the product tester 14 performed during the product testing, as similarly discussd.
  • the report 1000 may also include respective results 1044a-1044c associated with the respective product testers for a different task 1040, which has a corresponding product testing question 1042.
  • the report 1000 may also include the inserted questions 1050 for the respective product testers 14 that tested the reliability of the product testing results.
  • the system 10 may be any type of
  • the user interface providing the report 1000 may also provide a control for allowing a user (e.g. , an administrator, or a personnel associated with the second party 16, etc.) to change a product testing result. If a product testing result indicates that a particular task has a "pass" status, the user may utilize the control to change the status from "pass" to "failed". Alternatively, if a product testing result indicates that a particular task has a "failed” status, the user may utilize the control to change the status from "failed” to "pass”. [0173] The system 10 of FIG. 1 B will now be described in further detail.
  • the system 10 includes one or more input(s) 1002 for receiving a product testing request, information about the product, and product testing instruction.
  • the system 10 also includes a notification generator 1004 configured to generate a notification 1005 to inform a plurality of product testers 14 that a product is available for product testing.
  • the product is associated with the first party 12 developing the product
  • the notification generator 1004 is associated with the second party 16 that is not the same as the first party 12.
  • the notification 1005 is for display in a web browser.
  • the product tester 14 may open a web browser on his / her device.
  • the web browser may include an add-on feature, which allows the product tester 14 to receive the notification 1005 from the system 10.
  • the notification 1005 may be for display in a field created by a product testing notification application.
  • the product tester 14 may install a product testing notification application onto his / her device, which allows the product tester 14 to receive the notification 1005 from the system 10.
  • the notification may be transmitted to crowdflower or a mechanical turk.
  • the system 10 also includes a communication interface 1008 configured to receive an electronic reply 1010 from a device 1020 being used by a first product tester 14 of the plurality of product testers, the reply 1010 indicating that the first product tester 14 will perform the product testing.
  • the communication interface 1008 and the one or more input(s) 1002 may be integrated.
  • the electronic reply 1010 may be generated using a web browser utilized by the product tester 14.
  • the web browser on a device utilized by the product tester 14 may include an add-on feature, which allows the product tester 14 to receive the notification from the system 10.
  • the same add-on feature may also allow the product tester 14 to provide his / her reply (for reception by the system 10) to accept or to sign up for a product testing project.
  • the electronic reply 1010 may be generated using a product testing notification application utilized by the product tester 14.
  • the product tester 14 may install a product testing notification application onto his / her device, which allows the product tester 14 to receive the notification from the system 10.
  • the same product testing notification application may also allow the product tester 14 to provide his / her reply (for reception by the system 10) to accept or to sign up for a product testing project.
  • the electronic reply may be generated by a device that is using crowdflower or a mechanical turk.
  • the system 10 may also include a reply counter 1040 configured to keep track with a number of replies 1010, and a comparator 1042 configured to compare the number of replies 1010 with a prescribed number.
  • the communication interface 1008 is configured to receive one or more additional electronic replies from one or more additional product testers 14 from the plurality of product testers 14 until a prescribed number of electronic replies has been reached.
  • the reply counter 1040 will increment by one.
  • the communication interface 1008 continues to receive additional reply until the total number of replies is equal to the prescribed number of electronic replies, as determined by the comparator 1042.
  • the prescribed number of replies is one reply. In other
  • the prescribed number of replies may be more than one.
  • the prescribed number of electronic replies may be three, four, five, six, or seven. In other embodiments, the prescribed number of electronic replies may two. In further embodiments, the prescribed number of electronic replies may be more than seven.
  • the system 10 may further include a filter 1050 configured to select the plurality of product testers 14 from a list of available product testers 14 for providing the notification.
  • the filter 1050 may be configured to select the plurality of product testers based at least in part on respective scores of the plurality of product testers, respective ages of the plurality of product testers, respective language skills of the plurality of product testers, respective locations of the plurality of product testers, respective age groups of the plurality of product testers, respective industry expertise of the plurality of product testers, or any combination of the foregoing.
  • the filter 1050 may then perform filtering to obtain a list of all product testers 14 who can read Italian.
  • the notification 1005 may be provided by the notification generator to only the product testers 14 who can read Italian.
  • the system 10 further includes a testing interface generator 1 100 configured to generate a testing interface for use in a testing environment.
  • the testing interface may be the testing interface described with reference to FIG. 4. In other embodiments, the testing interface may have other configurations.
  • the testing interface is configured to present features of the product and product testing instruction for allowing the first product tester 14 to perform the product testing on the product based on the product testing instruction.
  • the product testing instruction in the testing interface is configured to provide a plurality of tasks for testing the product. For example, the product testing instruction may prescribe one or more actions be performed by the product tester 14 on a web page being tested.
  • the testing interface is configured for display on a hand-held device (e.g., an iPhone, iPad, iPad mini, tablet, etc.).
  • the testing interface is configured for display on a computer screen or a laptop screen.
  • the product testing instruction may be created by a user through an user interface provided by the user interface generator 1350.
  • the user creating the product testing instruction may be associated with the first party 12 providing the product for testing, or may be associated with the second party 16.
  • the user interface allows the user to create multiple product testing tasks, and multiple corresponding product testing questions, to test a product.
  • the product includes a web page, and the user interface allows the user to select one or more platforms / browsers in which to test the web page.
  • the product testing instruction may be stored in the product testing instruction database 1600.
  • the user interface generator 1350 may be configured to provide a first product tester 14 access to a testing environment for allowing the first product tester 14 to perform product testing on a product based on the product testing instruction.
  • the user interface generator 1350 may also be configured to provide a second product tester 14 access to another testing environment for allowing the second product tester 14 to perform another product testing on the same product based on the same product testing instruction.
  • the user interface generator 1350 may provide different virtual environments for different respective product testers 14. Accordingly, different product testers 14 may perform the product testing on different respective virtual machines that provide the testing interfaces for testing the product.
  • the testing interface is configured to present an image of a web page (e.g., a web page being tested), and the product testing instruction is configured to instruct the first product tester 14 to perform an action on the image of the web page being tested.
  • the testing interface may further be configured to present a question for the first product tester 14 to answer after the action has been performed based on the product testing instruction.
  • the system 10 may further include a non-transitory medium for storing the product tester's answer to the question in a test result file.
  • the system 10 also includes a timer 1200 that keeps track of the duration that has lapsed since the testing interface 402 was provided to the product tester 14.
  • the system 10 also includes a time- comparator 1202, which compares the lapsed time with a prescribed threshold. If the lapsed time has exceeded the prescribed threshold and the product testing is not completed by the product tester 14, then the system 10 will revoke the product testing project from the product tester 14. For example, the system 10 may terminate the product testing session being used by the product tester 14.
  • the system 10 in response to the revoked product testing project from the product tester 14, the system 10 will update the reply counter 1040 (e.g., by subtracting the total count by one) to indicate that a slot for the product testing project is available (due to the product testing project being released back to the pool of product testers 14 for acceptance). If another product tester 14 submits a reply to take the slot for the product testing project, the product testing project (which was not completed by the previous product tester 14) is then assigned to this other product tester 14.
  • the reply counter 1040 e.g., by subtracting the total count by one
  • the system 10 also includes a tracker 1300 configured to track an interaction of the first product tester 14 with the product.
  • the product may be a web page
  • the tracker 1300 is configured to track an interaction of the first product tester 14 with an image of the web page.
  • the tracker 1300 is configured to track a movement of a cursor operated by the first product tester 14.
  • the first product tester 14 may be using his / her device (e.g. , laptop, desktop, handheld device, such as a cell phone, etc.) to perform the product testing.
  • the tracker 1300 is configured to track a position of the cursor in the product tester's device as operated by the first product tester 14.
  • the tracker 1300 may be configured to track a position of the finger position as applied on the touchscreen.
  • the term “cursor” is not limited to a displayed item operated using a touchpad, a mouse, or a knob, and the term “cursor” may also include non-displayed item that is associated with a movement of a finger as applied on a touchscreen.
  • the tracker 1300 is configured to track a selection of a tab, recording a selection of a button, recording a selection of an icon, recording a typed text, or any combination of the foregoing, performed by the first product tester 14.
  • the system 10 further includes a graphic generator 1302 configured to generate a graphic representing the tracked movement by the tracker 1300.
  • the graphic comprises a line traversing a plurality of cursor positions.
  • the graphic may also comprises a selection identifier identifying a position in the image of the web page at which the first product tester 14 performed a selection.
  • the system 10 may also include an image database 1304 configured to store an electronic image in a non-transitory medium, the electronic image comprising the image of the web page and the graphic superimposed with the image of the web page.
  • the image database 1304 is configured to store the electronic image in association with an identity of the first product tester 14 and in association with the product being tested.
  • the image database 1304 may be one or more non-transitory medium located at a same facility or at different facilities.
  • the system 10 may further include a video generator 1310 configured to generate a video of the tracked interaction.
  • the video may show a movement of a cursor as operated by the product tester 14 while testing a web page.
  • the video may also show a selection of an object in the web page by the product tester 14.
  • the video may be a recording of screenshots over time to show an interaction of the product tester 14 with the web page being tested.
  • the video may contain images of a processor-generated graphic (tracking an interaction of the product tester 14) superimposed with image(s) of the web page being tested.
  • the graphic may be a "growing" line that follows a position of a cursor, thereby representing cursor positions over time.
  • the video may also show a selection identifier identifying a position in the image of the web page at which the first product tester 14 performed a selection (e.g. , a clicking operation).
  • the graphic may include an identifier (e.g. , a circle) placed on the "growing line" wherever the product tester 14 performs a clicking operation.
  • the system 10 may also include a video database 1314 configured to store the video in a non-transitory medium.
  • the video database 1314 is configured to store the video in association with an identity of the first product tester 14 and in association with the product being tested.
  • the image database 1314 may be one or more non- transitory medium located at a same facility or at different facilities.
  • the system 10 also includes a testing monitoring module 1400 configured to monitor a progress of the product testing.
  • the testing monitoring module 1400 is configured to determine a percentage of the product testing completed, a duration spent by the first product tester, a number of steps taken by the first product tester, time wasted by the first product tester, time lapsed before a first link is clicked by the first product tester, etc.
  • the system 10 further includes a test result retrieval module 1500 configured to obtain a first testing result achieved by the first product tester 14 completing the product testing.
  • the test result retrieval module 1500 is also configured to obtain additional testing result(s) from one or more additional product tester(s) 14.
  • the test result retrieval module 1500 may be configured to also obtain a second testing result achieved by the second product tester 14 completing the product testing.
  • the system 10 further includes an analyzer 1502 configured to analyze the first testing result and the second testing result to determine whether the first testing result, the second testing result, or both of the first testing result and the second testing result, are reliable or not.
  • the analyzer 1502 may also be configured to use other information obtained by the testing monitoring module 1400, to determine whether one or more of the product testing results are reliable or not.
  • the test result retrieval module 1500 may be configured to generate one or more product testing reports, such as those shown in FIGS. 8A-8C and FIGS. 10A-10E.
  • the product testing reports may be provided to one or more users through one or more user interface(s) provided by the user interface generator 1350.
  • one or more reports may be provided to a user associated with the first party 12 providing the product being tested.
  • one or more reports may be provided to an administrator or another user associated with the second party 16.
  • the system 10 may connect multiple product testers 14 to a certain product testing project.
  • the testing interface module is configured to provide multiple instances of the testing interface to multiple product testers 14.
  • the testing interface e.g. , an instance of it
  • the system 10 may be configured to provide multiple instances of virtual testing environment for multiple product testers 14 to perform product testing.
  • system 10 may connect multiple product testers 14 to multiple product testing projects to test different products created by different parties 12.
  • system 10 may be implemented using a single device, or using a plurality of devices.
  • the entire system 10 may be implemented on a server in some embodiments.
  • different components of the system 10 may be implemented on different machines, or on different sets of machines.
  • the input(s) 1002, the notification generator 1004, the communication interface 1008, the reply counter 1040, the comparator 1042, and the filter 1050 may be implemented using a first set of one or more devices
  • the testing interface generator 1 100 may be implemented using a second set of one or more devices
  • the tracker 1300, the graphic generator 1302, and the video generator 1310 may be implemented using a third set of one or more devices
  • the testing monitoring module 1400 may be implemented using a fourth set of one or more devices
  • the test result retrieval module 1500 and the analyzer 1502 may be implemented using a fifth set of one or more devices.
  • a device may be a processor, a computer, a server, a database, etc.
  • the system and method described herein may be applied to test other products.
  • the product may be any software, any hardware, any physical device, or any computer modeling.
  • a component in the system 10 may be partially or completely implemented at a product tester's 14 device or at a device used by the first party 12 (e.g. , a computer at the provider of the product being tested).
  • all or a part of the testing interface generator 1 100 may be implemented at a product tester's 14 device.
  • a product tester may install an application provided by the second party 16 onto a device (e.g. , a laptop, a desktop computer, a handheld device, etc.) of the product tester 14, which allows the device to display a testing interface in response to the product tester 14 accepting a product testing project.
  • the same application may also be installed by multiple product testers 14.
  • all or a part of the tracker 1300, all or a part of the graphic generator 1302, and/or all or a part of the video generator 1310 may be implemented at a product tester's 14 device.
  • a product tester may install an application provided by the second party 16 onto a device (e.g., a laptop, a desktop computer, a handheld device, etc.) of the product tester 14, which allows the device to track an action of the product tester 14 as the product tester 14 is using the device to perform the product testing, and to generate images and/or videos capturing the action of the product tester 14.
  • the same application may also be installed by multiple product testers 14.
  • the user interface generator 1350 may be implemented at a product provider's (first party's 12) device.
  • a product provider may install an application provided by the second party 16 onto a device (e.g., a laptop, a desktop computer, a handheld device, etc.) of the product provider, which allows the product provider to use its device to create product testing instruction, select product testing parameters, and provide information regarding the product that is desired to be tested.
  • the application also allows the device at the product provider to view product testing results transmitted by the system 10.
  • the same application may also be installed by multiple product providers 12.
  • the system 10 described herein is not limited to one or more devices associated with the second party 16, and may include one or more devices being used by the product testers 14, and/or one or more devices being used by users at the product provider (the first party 12).
  • any of the components of the system 10 may be implemented using hardware, software, or a combination of hardware and software.
  • each of the items below may be
  • a processor-readable medium including a set of instructions for providing one or more features described herein is provided.
  • the instructions in the processor-readable medium when executed by a processing unit, will cause a method for providing a product testing to be performed.
  • the instructions may include: instruction for providing a notification to inform a plurality of product testers that a product is available for product testing, wherein the product is associated with a first party and the notification generator is associated with a second party that is not the same as the first party; instruction for receiving an electronic reply from a device being used by a first product tester of the plurality of product testers through a network, the reply indicating that the first product tester will perform the product testing; instruction for updating a reply count in response to the received electronic reply; and instruction for, in response to the electronic reply, providing a testing environment that comprises a testing interface for the first product tester, the testing interface presenting features of the product and the product testing instruction for allowing the first product tester to perform the product testing on the product based on the product testing instruction, the product testing instruction providing a plurality of tasks for testing the product.
  • the system 10 and the method 200 described herein are advantageous because they allow a product to be tested efficiently under multiple different testing parameters.
  • a software application may be tested under 10 different platforms by multiple product testers 14. Because the product testing may be performed by multiple product testers 14 in parallel, the user at the product provider can obtain product testing results very quickly. Also, because multiple product testers 14 can be assigned to perform each product testing (e.g., testing an application under a certain platform), the product test result being provided to the product provider can be very accurate and reliable. More importantly, the above benefits are provided for the product provider without the need for the product provider searching for qualified product testers. Accordingly, embodiments and features of the system 10 and the method 200 described herein have practical and real world utility, and they create tangible and concrete results that significantly improve the way product testing is performed.
  • FIG. 11 is a block diagram illustrating an embodiment of a specialized processing system 1600 that can be used to implement various embodiments or features described herein.
  • the processing system 1600 may be used to implement the system 10 of FIG. 1A and FIG. 1 B, or any component thereof.
  • the processing system 1600 may be an example of a device being used by a product tester.
  • Processing system 1600 includes a bus 1602 or other communication mechanism for communicating information, and a processor 1604 coupled with the bus 1602 for processing information.
  • the processor system 1600 also includes a main memory 1606, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1602 for storing information and instructions to be executed by the processor 1604.
  • the main memory 1606 also may be used for storing temporary variables or other intermediate
  • the processor system 1600 further includes a read only memory (ROM) 1608 or other static storage device coupled to the bus 1602 for storing static information and instructions for the processor 1604.
  • ROM read only memory
  • a data storage device 1610 such as a magnetic disk or optical disk, is provided and coupled to the bus 1602 for storing information and instructions.
  • the processor system 1600 may be coupled via the bus 1602 to a display 167, such as a cathode ray tube (CRT), for displaying information to a user.
  • a display 167 such as a cathode ray tube (CRT)
  • An input device 1614 is coupled to the bus 1602 for communicating information and command selections to processor 1604.
  • cursor control 1616 is Another type of user input device
  • cursor control 1616 such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 1604 and for controlling cursor movement on display 167.
  • This input device typically has two degrees of freedom in two axes, a first axis (e.g. , x) and a second axis (e.g. , y), that allows the device to specify positions in a plane.
  • the processor system 1600 can be used to perform various functions described herein. According to some embodiments, such use is provided by processor system 1600 in response to processor 1604 executing one or more sequences of one or more instructions contained in the main memory 1606. Those skilled in the art will know how to prepare such instructions based on the functions and methods described herein. Such instructions may be read into the main memory 1606 from another processor- readable medium, such as storage device 1610. Execution of the sequences of instructions contained in the main memory 1606 causes the processor 1604 to perform the process steps described herein. One or more processors in a multi- processing arrangement may also be employed to execute the sequences of instructions contained in the main memory 1606. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the various embodiments described herein. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.
  • processor-readable medium refers to any medium that participates in providing instructions to the processor 1604 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media.
  • Non-volatile media includes, for example, optical or magnetic disks, such as the storage device 1610.
  • a non-volatile medium may be considered an example of non-transitory medium.
  • Volatile media includes dynamic memory, such as the main memory 1606.
  • a volatile medium may be considered an example of non-transitory medium.
  • Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise the bus 1602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
  • processor-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a processor can read.
  • processor-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 1604 for execution.
  • the instructions may initially be carried on a magnetic disk of a remote computer.
  • the remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem.
  • a modem local to the processing system 1600 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal.
  • An infrared detector coupled to the bus 1602 can receive the data carried in the infrared signal and place the data on the bus 1602.
  • the bus 1602 carries the data to the main memory 1606, from which the processor 1604 retrieves and executes the instructions.
  • the instructions received by the main memory 1606 may optionally be stored on the storage device 1610 either before or after execution by the processor 1604.
  • the processing system 1600 also includes a communication interface 1618 coupled to the bus 1602.
  • the communication interface 1618 provides a two-way data communication coupling to a network link 1620 that is connected to a local network 1622.
  • the communication interface 1618 may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line.
  • ISDN integrated services digital network
  • the communication interface 1618 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN.
  • LAN local area network
  • Wireless links may also be implemented.
  • the communication interface 1618 sends and receives electrical, electromagnetic or optical signals that carry data streams representing various types of information.
  • the network link 1620 typically provides data communication through one or more networks to other devices.
  • the network link 1620 may provide a connection through local network 1622 to a host computer 1624 or to equipment 1626 such as a radiation beam source or a switch operatively coupled to a radiation beam source.
  • the data streams transported over the network link 1620 can comprise electrical, electromagnetic or optical signals.
  • the signals through the various networks and the signals on the network link 1620 and through the communication interface 1618, which carry data to and from the processing system 1600, are exemplary forms of carrier waves transporting the information.
  • the processing system 1600 can send messages and receive data, including program code, through the network(s), the network link 1620, and the communication interface 1618.
  • the term "image” is not limited to an image that is displayed, and may also include image that is not displayed, such as image data stored in a non-transitory medium.
  • video is not limited to a video that is displayed, and may also include video that is not displayed, such as video data stored in a non-transitory medium.
  • first, second, third, etc. do not necessarily refer to order of items. Rather, these terms may be used to distinguish different items from each other. For example, “first product tester” does not necessarily mean that the product tester is the first in order. Also, “second product tester” does not necessarily mean that the product tester is the second in order. Rather, these terms are used to identify two different product testers.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Finance (AREA)
  • General Engineering & Computer Science (AREA)
  • Accounting & Taxation (AREA)
  • Strategic Management (AREA)
  • Development Economics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Game Theory and Decision Science (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • General Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Debugging And Monitoring (AREA)
  • Testing Electric Properties And Detecting Electric Faults (AREA)
PCT/US2017/067732 2016-12-20 2017-12-20 Electronic product testing systems WO2018119149A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP17832139.4A EP3559887A1 (en) 2016-12-20 2017-12-20 Electronic product testing systems
IL267368A IL267368B2 (he) 2016-12-20 2019-06-16 מערכות לבדיקת מוצר אלקטרוני

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201615385742A 2016-12-20 2016-12-20
US15/385,742 2016-12-20
US15/784,064 2017-10-13
US15/784,064 US11392277B2 (en) 2017-10-13 2017-10-13 Electronic product testing systems

Publications (1)

Publication Number Publication Date
WO2018119149A1 true WO2018119149A1 (en) 2018-06-28

Family

ID=61003383

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2017/067732 WO2018119149A1 (en) 2016-12-20 2017-12-20 Electronic product testing systems

Country Status (3)

Country Link
EP (1) EP3559887A1 (he)
IL (1) IL267368B2 (he)
WO (1) WO2018119149A1 (he)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020227182A1 (en) * 2019-05-03 2020-11-12 Rainforest Qa, Inc. Systems and methods for evaluating product testing
CN112056760A (zh) * 2020-08-18 2020-12-11 惠州市德赛西威汽车电子股份有限公司 一站式测试系统及方法

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2000026836A2 (en) * 1998-11-02 2000-05-11 Vividence Corporation Full-service research bureau and test center method and apparatus

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7086032B2 (en) * 2003-02-20 2006-08-01 Adobe Systems Incorporated System and method for representation of object animation within presentations of software application programs
US7941525B1 (en) * 2006-04-01 2011-05-10 ClickTale, Ltd. Method and system for monitoring an activity of a user
US20110191676A1 (en) * 2010-01-29 2011-08-04 Microsoft Corporation Cross-Browser Interactivity Recording, Playback, and Editing
US8549482B2 (en) * 2010-12-15 2013-10-01 Hewlett-Packard Development Company, L.P. Displaying subtitles
US8554605B2 (en) * 2011-06-29 2013-10-08 CrowdFlower, Inc. Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks
US20130014084A1 (en) * 2011-07-05 2013-01-10 Microsoft Corporation International Testing Platform
US20140156331A1 (en) * 2012-12-05 2014-06-05 Matthew Cordasco Methods and systems for connecting third party applications to crowdsourcing communities

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2000026836A2 (en) * 1998-11-02 2000-05-11 Vividence Corporation Full-service research bureau and test center method and apparatus

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
RICHARD ATTERER ET AL: "Knowing the user's every move", WWW '06 PROCEEDINGS OF THE 15TH INTERNATIONAL CONFERENCE ON WORLD WIDE WEB, ACM, NEW YORK, NY, USA, 23 May 2006 (2006-05-23), pages 203 - 212, XP058095370, ISBN: 978-1-59593-323-2, DOI: 10.1145/1135777.1135811 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020227182A1 (en) * 2019-05-03 2020-11-12 Rainforest Qa, Inc. Systems and methods for evaluating product testing
US11947447B2 (en) 2019-05-03 2024-04-02 Rainforest Qa, Inc. Systems and methods for evaluating product testing
CN112056760A (zh) * 2020-08-18 2020-12-11 惠州市德赛西威汽车电子股份有限公司 一站式测试系统及方法
CN112056760B (zh) * 2020-08-18 2023-01-06 惠州市德赛西威汽车电子股份有限公司 一站式测试系统及方法

Also Published As

Publication number Publication date
IL267368B2 (he) 2023-05-01
IL267368A (he) 2019-08-29
EP3559887A1 (en) 2019-10-30
IL267368B1 (he) 2023-01-01

Similar Documents

Publication Publication Date Title
US11334473B2 (en) Electronic product testing systems
US20090024971A1 (en) Cursor path vector analysis for detecting click fraud
CN103984626B (zh) 一种生成测试用例脚本的方法及装置
US11907621B2 (en) Electronic product testing systems for providing automated product testing
US9804915B2 (en) Integrated production support
US20090024460A1 (en) Cursor path vector analysis for detecting click fraud
EP3729275B1 (en) Electronic product testing method and medium
CN112199277A (zh) 一种基于浏览器的缺陷复现方法、装置、设备及存储介质
JP2021507436A (ja) 動的Ｗｅｂページの訪問者の行動分析情報提供装置及びこれを用いたＷｅｂサイトの訪問者の行動分析情報提供方法
EP3559887A1 (en) Electronic product testing systems
CN110866205A (zh) 用于存储信息的方法和装置
US11392278B2 (en) Electronic product testing systems
US9501378B2 (en) Client events monitoring
KR102155793B1 (ko) 인공지능 학습데이터 생성을 위한 크라우드소싱 기반 프로젝트의 작업단가 관리 방법 및 장치
US20220237110A1 (en) Electronic product testing systems for providing automated product testing with human-in-the loop component and/or object detection
CN112363885A (zh) 检测应用异常的方法、装置、电子设备及计算机可读介质
US11947447B2 (en) Systems and methods for evaluating product testing
US11797374B2 (en) Systems and methods for recording major incident response information
US9965131B1 (en) System and processes to capture, edit, and publish problem solving techniques
CN117234399A (zh) 行为数据的分析方法、装置、设备及存储介质
WO2016106775A1 (en) Traceable engineering projects

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17832139

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017832139

Country of ref document: EP

Effective date: 20190722