WO2014210542A2 - Système, procédé et produit programme d'ordinateur destinés à transmettre une logique d'essai à des dispositifs utilisateur - Google Patents

Système, procédé et produit programme d'ordinateur destinés à transmettre une logique d'essai à des dispositifs utilisateur Download PDF

Info

Publication number
WO2014210542A2
WO2014210542A2 PCT/US2014/044713 US2014044713W WO2014210542A2 WO 2014210542 A2 WO2014210542 A2 WO 2014210542A2 US 2014044713 W US2014044713 W US 2014044713W WO 2014210542 A2 WO2014210542 A2 WO 2014210542A2
Authority
WO
WIPO (PCT)
Prior art keywords
user
test
video device
user video
logic
Prior art date
Application number
PCT/US2014/044713
Other languages
English (en)
Other versions
WO2014210542A3 (fr
Inventor
William Amidei
Jason Braness
Eric Grab
Michael Kiefer
Abhishek Shivadas
Original Assignee
Sonic Ip, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US13/930,753 external-priority patent/US9648086B2/en
Priority claimed from US13/930,716 external-priority patent/US20150002680A1/en
Priority claimed from US13/930,666 external-priority patent/US20150002679A1/en
Application filed by Sonic Ip, Inc. filed Critical Sonic Ip, Inc.
Publication of WO2014210542A2 publication Critical patent/WO2014210542A2/fr
Publication of WO2014210542A3 publication Critical patent/WO2014210542A3/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N17/00—Diagnosis, testing or measuring for television systems or their details
    • H04N17/04—Diagnosis, testing or measuring for television systems or their details for receivers

Definitions

  • Video content may be consumed on different devices in a variety of environments through any of a variety of services. This has freed the video consumer (hereinafter, the user) to enjoy content on whatever user video device they would like, in a wide range of settings.
  • User video devices may include, for example and without limitation, televisions, set-top boxes, digital video recorders and players, desktop computers, laptops, netbooks, tablet computers, smartphones, and wearable computing devices.
  • Video may be consumed by playback of recorded or downloaded content, by streaming, or through other processes.
  • the video consumption experience is not always seamless and sometimes sacrifices a consistent high quality experience across the range of devices. This inconsistency may be a result of the device's processing power, the integration of the technology on the device, the age of the firmware of the device, or any of a host of other reasons.
  • a user video device typically includes a variety of technology in the form of hardware, software, and/or firmware.
  • this technology may be delivered by a provider in the form of source code and libraries to a video device manufacturer or component manufacturer, e.g., to any party that provides integrated circuits (ICs), original design manufacturing (ODM), or original equipment manufacturing (OEM), or that integrates such technology into chips or user video devices.
  • ICs integrated circuits
  • ODM original design manufacturing
  • OEM original equipment manufacturing
  • the user video device may be submitted for certification.
  • a user video device is run through a number of tests to verify that the technology is integrated correctly, providing a level of assurance of interoperability across certified devices.
  • This model is based on a user video device being certified a single time with a specific configuration of hardware, firmware, and/or software, and considering a limited number of use cases for consuming video.
  • user video devices are now frequently being upgraded, e.g., Android smartphones getting upgraded operating systems, where a user video device may change significantly after it has been certified.
  • digital televisions and DVD players that are regularly receiving new firmware versions.
  • use cases may result in different levels of interoperability and different levels of quality in the user experience. For example, consuming video from local file playback, adaptive streaming, or in a Digital Living Network Alliance (DNLA) setting may all have different results for the user.
  • DNLA Digital Living Network Alliance
  • FIG. 1 is a block diagram illustrating components and operation of an embodiment.
  • FIG. 2 is a flowchart illustrating a query of the type and configuration of a user video device, according to an embodiment.
  • FIG. 3 is a flowchart illustrating processing at a remote source of test logic, according to an embodiment.
  • FIG. 4 is a flowchart illustrating analysis of test results, according to an embodiment.
  • FIG. 5 is a block diagram illustrating software or firmware modules at an application server, according to an embodiment.
  • FIG. 6 is a flowchart illustrating processing at a user video device, according to an embodiment.
  • FIG. 7 is a flowchart illustrating processing at a user video device, according to an alternative embodiment.
  • FIG. 8 is a flowchart illustrating an incentivized approach to obtaining user consent to certification testing, according to an embodiment.
  • FIG. 9 is a block diagram illustrating functional components at a user video device, according to an embodiment.
  • FIG. 10 is a block diagram illustrating software or firmware modules at a user, according to an embodiment.
  • the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears.
  • test logic and any related test input data are sent from a remote source such as an application server to a user video device.
  • the test logic is executed at the user video device, and test results are output.
  • the test results are sent back to the application server.
  • the operation of the application server may be facilitated by a database.
  • a database stores test logic and test input data, and/or stores tests results for subsequent analysis. Additional variations and embodiments of such a system and method are described below.
  • FIG. 1 The overall processing of the systems described herein is illustrated in FIG. 1 , according to an embodiment.
  • One or more content providers 105 will typically deliver content to user video devices 140, 141 ....Content delivery may represent downloading or streaming of content, for example.
  • content delivery is shown taking place via a network 130.
  • Certification testing may be performed when one or more remote source(s), such as application server(s) 1 10, sends test logic to one or more of the user video devices 140, 141
  • the application server(s) 1 10 may be operated by or on behalf of a service provider, component provider, or other party responsible for certification.
  • the test logic represents the certification test, such that execution of the test logic by a user video device represents the performance of the certification test.
  • the test logic may be accompanied by test input data that serves as input during execution of the test logic.
  • the results of the certification testing can then be returned to the application server 1 10 via network 130.
  • the application server may initially retrieve the test logic and any required test input data from one or more database(s) 120.
  • the test logic and test input data may then be sent to one or more of the user video devices.
  • Test results that result from the certification testing may also be stored in database 120 after having been received at the application server 1 10.
  • the test results may be sent by a user video device to database 120 without going through application server 1 10.
  • some or all of the communications between the application server 1 10, database 120, and the user video devices 140, 141 ,... may be protected by some form of information security mechanism(s). Encryption of test logic and test results may be used while this information is in transit. Authentication of these communications may also take place. Any authentication or encryption processing may use symmetric processes (such as the Data Encrytion Standard (DES) or Advanced Encryption Standard (AES)) or may use asymmetric methods, e.g., public key mechanisms such as the Rivest Shamir Adleman (RSA) or Diffie Hellman algorithms.
  • DES Data Encrytion Standard
  • AES Advanced Encryption Standard
  • RSA Rivest Shamir Adleman
  • Diffie Hellman algorithms e.g., Diffie Hellman
  • Certification testing can examine a wide range of processes and configurations at a user video device. For example, test logic may verify audio/video synchronization given a specific audio format. In another example, test logic may check the aspect ratio by having the user video device play a video stream showing a square box. The test logic may then take a screen shot and measure the box to assess whether the box is correctly displayed.
  • the test input data may be specific to a particular certification test.
  • a certification test may use specific test input data; such test input data may be, for example, a white bounding box.
  • Other certification tests may require a stream containing a number of consecutive bi-predictive frames (B-frames), for example.
  • an HTTP live streaming (HLS) manifest file may be sent to the user video device to test HLS playback.
  • the application server may first have to ascertain configuration and type information about the user video device to be tested. This is because the particular certification test(s) to be administered may depend, at least in part, on the type and configuration of the user video device.
  • the relevant device attributes can include (but are not limited to) the particular device being tested (e.g., manufacturer and model), the software and/or firmware configuration (e.g., operating system (OS), the OS version, the applications present and their versions), and hardware attributes (e.g., memory capacity and processor speed).
  • Others examples may include the audio and video codecs supported, subtitles supported, media container supported, streaming protocols supported, security and encryption algorithms supported, secure execution environment supported, and the presence of high- bandwidth digital content protection (HDCP) and other output protections.
  • HDCP high- bandwidth digital content protection
  • FIG. 2 A process at the application server for determining information about a user video device is illustrated in FIG. 2, according to an embodiment.
  • a determination is made as to whether it is necessary to obtain device information.
  • the application server may already know the necessary information about the user video device required for determining the necessary certification test(s). If information about the user video device is necessary, then there is a need to query the user video device in the illustrated embodiment, and processing continues at 220.
  • the application server will send a query to the user video device regarding the device identity and/or configuration.
  • a response is received by the application server, where the response includes at least some of the needed information.
  • a determination is made as to whether additional information is needed.
  • the process can reiterate at 220, where an additional query may be sent by the application server to the user video device.
  • an additional query may be sent by the application server to the user video device.
  • FIG. 2 may take place prior to the sending of test logic to a user video device, in alternative embodiments this processing may take place as part of test logic execution, so that the test logic operation may include the determination of a device's type or configuration.
  • the certification testing may proceed first, and the determination of a user video device's type or configuration may take place afterwards. This latter scenario may take place, for example, if the user video device's type or configuration is not important to the selection of particular certification tests, but may be useful for post-testing analysis of test results. Note that in other cases the processing of FIG. 2 may not be necessary.
  • the application server may have the relevant information about the user video device, as discussed above. In other circumstances, the configuration of a device may not be relevant to the selection of particular certification tests.
  • FIG. 3 Further processing at the application server is illustrated in FIG. 3, according to an embodiment.
  • a determination is made as to whether certification testing is required for a given user video device.
  • the need for testing may be based on a number of factors.
  • a user video device may have been newly fielded, for example, and there may be questions as to its performance or interoperability.
  • a known configuration change may have taken place or a new service may have been introduced, which would raise interoperability questions.
  • a new version of an operating system or application may have been recently released, for example, which may motivate certification testing of affected video devices.
  • certification testing may be performed on a scheduled basis, so that testing is required at certain points in time.
  • certification testing may range from diagnostic testing of functionality to the testing of particular discrete operations. Certification testing may follow a particular logical protocol, and may include initial tests that determine the need for more specific subsequent testing of particular functions.
  • test logic embodying these tests is retrieved by the application server.
  • this test logic may be retrieved from a database in communication with the application server.
  • test input data is also retrieved, where this test input data represents data to be used by an identified certification test when the corresponding test logic is executed by the user video device.
  • the test input data may also be retrieved from the above database, or from a separate location.
  • the test logic and/or test input data may be stored in the application server itself, and an external database would not be needed for storage of these items.
  • test input data represents information used by the test logic.
  • An example would be a video clip constructed to exercise certain video processing operations.
  • Test input data may alternatively take other forms, such as binary data to exercise a particular routine or circuit.
  • test logic and any test input data are sent to the user video device.
  • test logic is executed at the user video device
  • test results are received at the application server from the user video device.
  • test results may be indexed in an embodiment.
  • test results may be annotated or otherwise flagged as being received from a particular user video device type having a particular software, firmware, and/or hardware configuration.
  • the test results are stored. In an embodiment, the test results are stored in accordance with the indexing discussed above. As noted above, this storage may take place in a database separate from the application server.
  • analysis and/or reporting of test results takes place. Analysis may be performed by the application server or by another computing device; moreover, the analysis may be performed by human analysts in conjunction with or instead of automated analysis. The analysis process will be discussed further below. In 380, reporting may take place.
  • Reporting may be directed to the user, to the manufacturer of the user video device, to the supplier of components, software, or firmware of the user video device, to industry groups, to consumer groups, and/or to any other interested party. Reporting may include raw test results or include the output of the analysis process, or both.
  • testing may be directed to a particular user video device, or to a sampling of a population of such devices. The latter would allow statistical analysis of a set of user video devices, for example, in a survey process.
  • a broader testing scenario may be used, such that a given test is divided into discrete portions, where different portions are allocated to different user video devices. This would minimize the burden on any particular user video device. Results of each portion could then be compiled and analyzed.
  • test results are obtained. If the results are stored in a database, the database is accessed to retrieve the test results.
  • the test results are compiled according to any of a variety of parameters or indices, such as (but not limited to) the device type, the configuration, and/or the test input data.
  • the test results as organized at 420 may be used to create inferences, hypotheses, or conclusions about the interoperability of the user video device(s) tested. Moreover, analysis may reveal a need for more testing.
  • the processing of FIG. 4 may be performed by the application server, by a different computing platform, or by human analysts, or by some combination thereof.
  • test results from different user video devices of a specific model can be aggregated, to examine the performance of that particular model. Analysis may be used to determine the model having the best performance or providing the best experience for a particular service. It may also be useful to track the number of certification tests that fail on a specific model. Certification testing may be used to identify the number of user video devices having a particular version of a software package. Such testing may also be used to determine that a specific type of user video device can run a particular number or type of services. Other analytic goals may also be pursued, as would be understood by a person of ordinary skill in the art.
  • FIGs. 2-4 While the processing of FIGs. 2-4 may be applied to a single user video device, these operations may also be applied to several such devices in aggregate, in order to survey the status of a population of user video devices. This may, for example, allow for the statistical study of the interoperability of similarly configured devices.
  • FIGs. 2-4 may be implemented in hardware, software, firmware, or a combination thereof, including discrete and integrated circuit logic, application specific integrated circuit (ASIC) logic, and microcontrollers, and may be implemented as part of a domain-specific integrated circuit package, or a combination of integrated circuit packages. In an embodiment, this processing may be implemented in an application server, as stated above.
  • ASIC application specific integrated circuit
  • the term software refers to a computer program product including at least one computer readable medium having computer program logic stored therein to cause a computer system to perform one or more features and/or combinations of features disclosed herein.
  • the computer readable medium may be transitory or non-transitory.
  • An example of a transitory computer readable medium may be a digital signal transmitted over a radio frequency or over an electrical conductor, through a local or wide area network, or through a network such as the Internet.
  • An example of a non-transitory computer readable medium may be a compact disk, a flash memory, RAM, ROM, or other data storage device.
  • System 500 may represent an application server or a portion thereof.
  • This system includes one or more central processing unit(s) (CPUs) 520 and a body of memory 510 that includes one or more transitory and/or non-transitory computer readable media that store computer program logic 540 and data received or generated during the processing described above.
  • Such data may include but is not limited to test logic, test input data, and test results for example, or configuration and identity information for a user video device.
  • the body of memory 510 may be implemented as one or more devices including a read-only memory (ROM) or random access memory (RAM) device, for example, or a combination thereof.
  • ROM read-only memory
  • RAM random access memory
  • CPU 520 and memory 510 may be in communication using any of several technologies known to one of ordinary skill in the art, such as a bus or a point-to-point interconnect.
  • Computer program logic 540 contained in memory 510 may be read and executed by CPU 520.
  • one or more I/O ports and/or I/O devices, shown collectively as I/O 530, are connected to CPU 520 and memory 510.
  • I/O 530 may include connectivity to a network 130 and/or a database 120, shown in FIG. 1 .
  • computer program logic 540 may include a module 550 responsible for determination of an appropriate certification test to be sent to a user video device.
  • Test determination module 550 may include the functionality discussed above with respect to FIG. 2 (user video device configuration determination) and 310 and 320 of FIG. 3 (identification of required certification tests).
  • Computer program logic 540 may include a module 560 responsible for retrieval of the appropriate test logic and test input data (330 of FIG. 3) from a database, for example; a module 570 is responsible for the sending of the test logic and test input data to the user video device (340 of FIG. 3) in an embodiment.
  • Computer program logic 540 may also include a module 550 responsible for the processing of test results. This would include the receipt of test results from a user video device (350 of FIG. 3), the indexing of the test results (360), and the storage of the indexed test results (370). Computer program logic 540 may also include a module 550 responsible for the analysis and reporting of test results (380 of FIG. 3).
  • the user video device receives and responds to a query from the application server regarding the identity and/or configuration of the user video device. As discussed above, in some embodiments or in certain situations, such a query and response may not be necessary.
  • the user video device receives test input data from the application server.
  • the user video device receives test logic from the application server. As previously described, in some circumstances test input data may not be needed to execute the test logic. Some tests may not require test input data; in other embodiments, test input data may already be resident at the user video device, so that such data will not be needed from the application server. Moreover, if both the test logic and test input data are received from the application server, they may be received in any order in alternative embodiments.
  • the test logic and test input data may be received at the user input device by a certification framework.
  • the certification framework may be internal to the user video device and serves as an interface to the CPU(s), memory, and other video processing logic of the user video device.
  • the certification framework allows execution of the test logic using the test input data and provides access, or "hooks" into the relevant data and processes of the user video device. The certification framework will be discussed further below.
  • test results are output. Test results may be sent to the application server; alternatively, they may be output to the user, to the manufacturer of the user video device, and/or to any other interested party.
  • test logic may execute and in so doing may present an image or video for the user to evaluate.
  • the user may have to answer questions about what is seen, e.g., provide subjective impressions of an image, or state whether the image or video appears to meet certain criteria.
  • the user's response(s) may then be required for continuation of the certification testing, or the response itself may represent the test results.
  • the user video device receives and responds to a query from the application server regarding the identity and/or configuration of the user video device. As discussed above, in some embodiments or in some situations, the query and response may not be necessary.
  • the user video device receives test input data from the application server.
  • the user video device receives test logic from the application server. As previously described, in some circumstances test input data may not be needed to execute the test logic. Moreover, if both the test logic and test input data are received from the application server, they may be received in any order in alternative embodiments.
  • the user video device executes the received test logic.
  • the user may be prompted for input, so that the user video device then receives the user input.
  • the certification testing may be repeated for a number of users of the particular user video device or feature. This would allow development of some statistical confidence in particular test results, given the human input into the certification test, or raise additional questions to be investigated.
  • test results are output. As in the process of FIG. 6, the test results may be sent to the application server; alternatively, they may be output to the user, to the manufacturer of the user video device, and/or to any other interested party.
  • test logic may be triggered by a command from the application server, or may be triggered by events or conditions at the user video device. For example, certain operations in the user video device, or a particular context may trigger execution of the test logic. In alternative embodiments, execution of the test logic may be at the discretion of the user (or of a test technician acting as a user in a certification test lab), or may be executed as processing bandwidth becomes available at the user video device.
  • an incentive may be provided to the user, to encourage participation in certification testing.
  • test input data may be incorporated into a video, such as a movie or music video. If the user consents to allow certification testing of his user video device, he will be allowed access to the video, as a streaming video or as a download for example. In this way, the certification testing can be performed using the provided video as test input and the test results sent to the application server. In exchange for consenting to the certification testing, the user is given access to the video. In an embodiment, subsequent viewings of the video at the user video device represent subsequent opportunities for further certification testing.
  • an offer is presented to the user.
  • the offer may be presented by the party responsible for certification, for example.
  • the offer includes access to a video, in exchange for the user's cooperation in certification testing.
  • the offer may be made through any of several means, such as email, SMS messaging, a social network communication, or any other mechanism known to persons of ordinary skill in the art.
  • the user may convey acceptance to the party responsible for certification through any of the means just mentioned.
  • an acceptance may be received at the application server.
  • a determination is made as to whether an acceptance has been received. If so, then at 830 test logic and the video are sent to the user video device from the application server and certification testing may proceed as described above.
  • test results are sent to the application server.
  • the test results may be indexed and at 860 the indexed test results may be stored at a database.
  • the test results may undergo analysis as discussed above.
  • the test dispatch module at the application server (see ref. 570, FIG. 5) is responsible for providing the video to the user video device.
  • a user video device may have one or more central processor units 910 as shown in FIG. 9, each with one or more read-only memory (ROM) devices 920 and one or more random access memory (RAM) devices 930.
  • Video processing at the user video processor unit may be performed by the CPU(s) 910 and/or video processing logic 940.
  • Video processing logic 940 may include one or more programmable processors or co-processors (such as a graphics processing unit) with associated memory, software or firmware, and/or dedicated video processing circuitry.
  • Test logic once received from the application server and installed at the user video device, is shown as module 950.
  • Test logic 950 may be embodied as software that is stored in a memory device. Though shown as a separate module, in an embodiment, this logic may be stored in RAM 930 or in memory associated with video processing logic 940.
  • Test logic 940 may run on CPU 910 and/or on another processor in video processing logic 940. As test logic 940 executes, it may read data that results from its operations. For example, this data may be found in RAM 930, in video processing logic 940, and/or in CPU 910.
  • the test logic 950 and test input data may be received at the user input device by certification framework 945.
  • the certification framework is internal to the user video device and serves as an interface between test logic 950 and the CPU(s) 910, memory 930, and/or other video processing logic 940 of the user video device.
  • the certification framework allows execution of the test logic 950 using the test input data and provides access, or "hooks" into the relevant data and processes of the user video device. This would allow tapping into data structures in RAM or access to CPU register values for example, or allow monitoring of CPU or graphics processing unit (GPU) performance.
  • the available hooks and/or other features in the certification framework 945 may be reported to the application server prior to the server's identification of the particular certification tests to be run (ref. 320, FIG. 3). For example, these features may be specified to the application server during the query of the user video device shown in FIG. 2.
  • Certification framework 945 may be implemented as software, firmware, hardware, or a combination thereof.
  • the certification framework 945 may be installed in the user video device at manufacture or installed subsequently.
  • a software embodiment of certification framework 945 or software portions thereof may be delivered to the user video device via download.
  • FIGs. 6-8 may be implemented in the user video device in hardware, software, firmware, or any combination thereof, including discrete and integrated circuit logic, ASIC logic, and microcontrollers, and may be implemented as part of a domain-specific integrated circuit package, or a combination of integrated circuit packages.
  • a software or firmware embodiment is illustrated in the context of a computing system in FIG. 10, which may be incorporated in a user video device.
  • System 1000 of FIG. 10 corresponds to system 900 of FIG. 9, but is redrawn here to show modules of a software/firmware implementation of FIGs. 6-8.
  • This system includes one or more CPU(s) 910, plus memory 1010 that includes one or more transitory and/or non-transitory computer readable media storing computer program logic 1040 and data received or generated during the processing described above. Such data may include but is not limited to test logic, test input data, and test results for example.
  • memory 1010 may collectively represent RAM 930 and ROM 920 of FIG. 9.
  • CPU(s) 910 and memory 1010 may be in communication using any of several technologies known to one of ordinary skill in the art, such as a bus or a point-to-point interconnect.
  • Computer program logic 1040 contained in memory 1010 may be read and executed by CPU(s) 910. In an alternative embodiment, logic 1040 may be executed by a different processor.
  • I/O 1030 one or more I/O ports and/or I/O devices, shown collectively as I/O 1030, are connected to CPU(s) 910 and memory 1010.
  • I/O 1030 may include connectivity to a network 130, shown in FIG. 1 .
  • computer program logic 1040 may include a module 1050 responsible for receipt of test logic from the application server.
  • Test logic receipt module 1050 may include the functionality illustrated at 620 and 730 discussed above.
  • Computer program logic 1040 may also include a certification framework module 1055 responsible for the functionality of the certification framework 945. In an embodiment, this may include the interfacing between the test logic 950 and other memory locations and processors in the user video device.
  • Computer program logic 1040 may also include a module 1060 responsible for output of test results. Test results output module 1060 incorporates the functionality discussed above with respect to 640 and 750, according to an embodiment.
  • the certification framework module 1055 may incorporate the functionality of modules 1050 and/or 1060.
  • information about one or more user video devices may be obtained for certification purposes without the delivery of specific test logic or test input data.
  • testing may be a matter of collecting data that is generated by normal operation of the user video devices. For example, data in a user video device may be mined, e.g., CPU-related underflow that occurs under certain circumstances. Such information may be sent periodically (or aperiodically) to the application server for subsequent analysis. While the logic for such data collection and reporting may be sent to the user video device in the manner of test logic as described above, such logic may alternatively be installed at manufacture or installed subsequently.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

L'invention concerne des procédés et des systèmes destinés à l'essai de certification à distance de dispositifs utilisateur vidéo. Dans un mode de réalisation, une logique d'essai et toutes les données d'entrée d'essai associées sont envoyées par une source distante telle qu'un serveur d'applications à un dispositif utilisateur vidéo. Ladite logique d'essai est exécutée par le dispositif utilisateur vidéo, et des résultats d'essai sont émis. Les résultats d'essai sont renvoyés au serveur d'applications. Le fonctionnement du serveur d'applications peut être facilité par une base de données. Dans des modes de réalisation, la base de données mémorise la logique d'essai et les données d'entrée d'essai, et/ou mémorise les résultats d'essais à des fins d'analyse ultérieure.
PCT/US2014/044713 2013-06-28 2014-06-27 Système, procédé et produit programme d'ordinateur destinés à transmettre une logique d'essai à des dispositifs utilisateur WO2014210542A2 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US13/930,716 2013-06-28
US13/930,753 US9648086B2 (en) 2013-06-28 2013-06-28 System, method, and computer program product for providing test logic to user devices
US13/930,716 US20150002680A1 (en) 2013-06-28 2013-06-28 System, method, and computer program product for receiving and executing test logic at user devices
US13/930,666 US20150002679A1 (en) 2013-06-28 2013-06-28 System, method, and computer program product for providing test logic to user devices
US13/930,666 2013-06-28
US13/930,753 2013-06-28

Publications (2)

Publication Number Publication Date
WO2014210542A2 true WO2014210542A2 (fr) 2014-12-31
WO2014210542A3 WO2014210542A3 (fr) 2015-03-05

Family

ID=52142854

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/044713 WO2014210542A2 (fr) 2013-06-28 2014-06-27 Système, procédé et produit programme d'ordinateur destinés à transmettre une logique d'essai à des dispositifs utilisateur

Country Status (1)

Country Link
WO (1) WO2014210542A2 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9648086B2 (en) 2013-06-28 2017-05-09 Sonic Ip, Inc. System, method, and computer program product for providing test logic to user devices
EP3561676A1 (fr) * 2018-04-27 2019-10-30 Top Victory Investments Ltd. Système de détection de dispositif d'affichage
USRE47825E1 (en) 2013-12-31 2020-01-21 Divx, Llc Methods, systems, and media for certifying a playback device
WO2020227182A1 (fr) * 2019-05-03 2020-11-12 Rainforest Qa, Inc. Systèmes et procédés d'évaluation de test de produit

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8230265B2 (en) * 2010-03-31 2012-07-24 Microsoft Corporation Testing software in electronic devices
US8417478B2 (en) * 2010-09-23 2013-04-09 Ixia Network test conflict checking
US20120079606A1 (en) * 2010-09-24 2012-03-29 Amazon Technologies, Inc. Rights and capability-inclusive content selection and delivery
US8566442B2 (en) * 2011-02-08 2013-10-22 Netflix, Inc. Test environment for audio/video playback

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9648086B2 (en) 2013-06-28 2017-05-09 Sonic Ip, Inc. System, method, and computer program product for providing test logic to user devices
USRE47825E1 (en) 2013-12-31 2020-01-21 Divx, Llc Methods, systems, and media for certifying a playback device
EP3561676A1 (fr) * 2018-04-27 2019-10-30 Top Victory Investments Ltd. Système de détection de dispositif d'affichage
WO2020227182A1 (fr) * 2019-05-03 2020-11-12 Rainforest Qa, Inc. Systèmes et procédés d'évaluation de test de produit
US11947447B2 (en) 2019-05-03 2024-04-02 Rainforest Qa, Inc. Systems and methods for evaluating product testing

Also Published As

Publication number Publication date
WO2014210542A3 (fr) 2015-03-05

Similar Documents

Publication Publication Date Title
US12039557B2 (en) Methods and apparatus to monitor media presentations
US20180322419A1 (en) Model Driven Modular Artificial Intelligence Learning Framework
CN104579768B (zh) 客户端升级方法和装置
US20150002680A1 (en) System, method, and computer program product for receiving and executing test logic at user devices
US9648086B2 (en) System, method, and computer program product for providing test logic to user devices
CN112218115B (zh) 流媒体音视频同步的控制方法、装置、计算机设备
WO2014210542A2 (fr) Système, procédé et produit programme d'ordinateur destinés à transmettre une logique d'essai à des dispositifs utilisateur
US20160148232A1 (en) Using hashed media identifiers to determine audience measurement data including demographic data from third party providers
US10810332B2 (en) Method, apparatus, and computer program product for simulating client and application interface integration
US20240195881A1 (en) Methods and apparatus to collect media metrics on computing devices
US20170075748A1 (en) System for monitoring a plurality of distributed devices
US20240163344A1 (en) Methods and apparatus to perform computer-based community detection in a network
US20220198058A1 (en) Identifying and preventing access to aggregate pii from anonymized data
US20150002679A1 (en) System, method, and computer program product for providing test logic to user devices
CN113472564B (zh) 基于区块链的责任溯源方法及电子设备
US11153392B2 (en) Methods and apparatus for session building from ping-level data
CN111930736B (zh) 数据校验方法及装置
CN114968696A (zh) 指标监控方法、电子设备及芯片系统
US10958509B2 (en) Providing a new server alarm configuration based on the correlation of server alarms for varying metrics of monitored servers
US10075352B2 (en) Monitoring user terminal applications using performance statistics for combinations of different reported characteristic dimensions and values
US11500742B2 (en) Electronic device and control method thereof
CN106919788B (zh) 血糖仪数据处理的方法和装置
US20140358850A1 (en) Open feedback apparatus and method of operating the same
CN116954975A (zh) 软件产品的故障处理方法、系统、装置以及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14817766

Country of ref document: EP

Kind code of ref document: A2

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC, EPO FORM 1205N DATED 04.03.2016

122 Ep: pct application non-entry in european phase

Ref document number: 14817766

Country of ref document: EP

Kind code of ref document: A2