US11178394B2 - System and method of subjective video quality evaluation - Google Patents
System and method of subjective video quality evaluation Download PDFInfo
- Publication number
- US11178394B2 US11178394B2 US16/880,646 US202016880646A US11178394B2 US 11178394 B2 US11178394 B2 US 11178394B2 US 202016880646 A US202016880646 A US 202016880646A US 11178394 B2 US11178394 B2 US 11178394B2
- Authority
- US
- United States
- Prior art keywords
- video clip
- pairs
- test
- video
- quality
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000013441 quality evaluation Methods 0.000 title claims abstract description 37
- 238000012360 testing method Methods 0.000 claims abstract description 236
- 238000001303 quality assessment method Methods 0.000 claims abstract description 22
- 238000010200 validation analysis Methods 0.000 claims description 27
- 230000015556 catabolic process Effects 0.000 claims description 14
- 238000006731 degradation reaction Methods 0.000 claims description 14
- 238000003860 storage Methods 0.000 claims description 14
- 230000000007 visual effect Effects 0.000 description 13
- 230000008569 process Effects 0.000 description 9
- 238000013459 approach Methods 0.000 description 8
- 238000012790 confirmation Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 238000012795 verification Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000009826 distribution Methods 0.000 description 6
- 238000011156 evaluation Methods 0.000 description 4
- 239000008186 active pharmaceutical agent Substances 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000013442 quality metrics Methods 0.000 description 3
- 238000012372 quality testing Methods 0.000 description 3
- 238000010998 test method Methods 0.000 description 3
- 241000282412 Homo Species 0.000 description 2
- 230000009471 action Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000006735 deficit Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000006866 deterioration Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 210000003414 extremity Anatomy 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 210000004124 hock Anatomy 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000001771 impaired effect Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000005304 joining Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000012358 sourcing Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012956 testing procedure Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/004—Diagnosis, testing or measuring for television systems or their details for digital television systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04B—TRANSMISSION
- H04B17/00—Monitoring; Testing
- H04B17/30—Monitoring; Testing of propagation channels
- H04B17/309—Measuring or estimating channel quality parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/239—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests
- H04N21/2393—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests involving handling client requests
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
- H04N21/25808—Management of client data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
- H04N21/2668—Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/475—End-user interface for inputting end-user data, e.g. personal identification number [PIN], preference data
- H04N21/4756—End-user interface for inputting end-user data, e.g. personal identification number [PIN], preference data for rating content, e.g. scoring a recommended movie
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/658—Transmission by the client directed to the server
- H04N21/6582—Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number
Definitions
- the presently disclosed subject matter relates generally to the field of assessment of video quality, and more specifically, to methods and systems of assessment of subjective quality of video content.
- the goal is to minimize costs while maximizing visual quality.
- the costs are composed of both the computational resources and power consumption required to perform all processing, from content acquisition, whether on camera or synthetically generated, and up until content consumption, as well as the obtained video file size or bitrate which impacts the cost of storage and required transmission bandwidth.
- Many complimenting approaches are taken to reduce costs: development of new encoding schemes and standards, optimizing encoders to reduce power consumption, creating more efficient storage and transmission platforms etc.
- the counter challenge—maximizing visual quality makes subjective video quality assessment a necessity, requiring availability of a fast, reliable and robust subjective quality evaluation process. This need is answered by the system described herein.
- a commonly used approach to evaluating video quality is by calculating objective quality metrics.
- the main advantages of this approach are the low cost of procuring the quality assessments and the repeatability of the results.
- Objective metrics belong to different categories according to their reliance on a reference or original video, to which the target video is compared. In this paper we limit the discussion to full-reference metrics, i.e. assessing quality of a particular video stream or video clip when compared to a corresponding reference or source video stream or clip.
- the full reference quality metrics vary from the most simple such as Peak-Signal-to-Noise-Ratio, which is just a logarithmically scaled Mean-Square-Error, through metrics such as Structural-SIMilarity (SSIM) and Visual Information Fidelity (VIF) and up to sophisticated metrics that have been shown to offer higher correlation with subjective quality such as Netflix's video quality assessment based on multi-method fusion known as VMAF.
- SSIM Structural-SIMilarity
- VIP Visual Information Fidelity
- ITU-R BT.500 The most commonly used standard for performing subjective testing of visual quality is ITU-R BT.500.
- This ITU recommendation describes standardized methods for subjective quality testing of television pictures, and defines the following test features: General viewing conditions such as display luminance and brightness, maximum observation angle, background chromaticity, room illumination, monitor resolution, monitor contrast and preferred viewing distance, source signals, selection of test materials and observers, test duration, instructions, order of test material and finally, how to present the test results.
- ITU-R BT.710 is a short document that further describes some viewing conditions and assessment methods which are specific to High Definition TV.
- BT.500 sets forth five different test methods, and points to ITU-R BT.1082 for the definition of five additional test methods.
- DSIS Double-Stimulus Impairment Scale
- DSCQS double-stimulus continuous quality-scale
- BT.1082 is not a normative document, but describes studies related to the development of new methods for subjective quality testing. Once these studies are finalized, candidate methods may be proposed for inclusion in future versions of BT.500.
- One of the proposed testing methods is the Forced-Choice Double-Stimulus (FCDS) method which has been specifically designed for assessing thresholds of visibility. This method is similar to DSIS and DSCQS tests, in that the observer views a pair of clips, e.g. original and processed, in random order, but instead of marking the quality the observer is simply asked which of them is impaired. The choice is described as forced since the observer must always select one of the clips, even if they seem identical. In studies comparing subjective methods for image quality assessment the forced-choice pairwise comparison method was found to produce the most accurate results.
- FCDS Forced-Choice Double-Stimulus
- ITU-T P.910 describes non-interactive subjective assessment methods for evaluating the one-way overall video quality for multimedia applications and is beneficial in adapting the testing procedures defined in the previous standards and recommendations to more dynamic testing content.
- ITU-T P.913 recommendation describes methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment, enabling evaluation of video quality on multiple devices in various “real-life” situations.
- the invention consists of two components: the client or application, and the backend or server. Each test session is performed by running a client, described below. This client communicates with the backend, or server, to receive test clips and upload the user feedback or test results. Once enough tests are performed, the results are collected from the backend and analyzed. We will now provide the details of these two components.
- a computerized system for video quality assessment comprising: a server comprising a database and a processor operatively connected thereto, wherein the database is configured to store pointers to a plurality of video clip pairs to be used in one or more test sessions performed by one or more users, each video clip pair including a target video clip and a reference video clip corresponding thereto; and the processor is configured to: Create, for each test session performed by a respective user, a test set including a set of video clip pairs selected from the plurality of video clip pairs to be displayed to the respective user, a display order of the video clip pairs in the set and display positions of the target video clip and the reference video clip in each video clip pair on a display screen of the respective user, giving rise to one or more test sets corresponding to the one or more test sessions; Send the one or more test sets to the one or more users for assessing quality of the target video clip and the reference video clip in each video clip pair in one or more sets of video clip pairs
- system can comprise one or more of features (a) to (g) listed below, in any desired combination or permutation which is technically possible:
- a computerized system for video quality assessment comprising a processor configured to: Receive a test set including a set of video clip pairs selected from a plurality of video clip pairs to be displayed to a user, each video clip pair including a target video clip and a reference video clip, a display order of the video clip pairs in the set, and display positions of the target video clip and the reference video clip in each video clip pair on a display screen of the user; Display the video clip pairs in the set according to the display order and the display positions in the test set, thereby enabling the user to perform a test session and assess quality of the target video clip and the reference video clip in each video clip pair; and Collect feedback regarding the assessed quality for each video clip pair and send the feedback to a server, wherein the feedback is usable for providing subjective quality evaluation of the plurality of video clip pairs.
- system according to this aspect of the presently disclosed subject matter can comprise one or more of features (i) to (vi) listed below, in any desired combination or permutation which is technically possible:
- a computerized method for video quality assessment comprising: Generating, by a processor in a server, one or more test sets corresponding to one or more test sessions performed by one or more users, comprising: Creating, for each test session performed by a respective user, a test set including a set of video clip pairs selected from a plurality of video clip pairs stored in a database to be displayed to the respective user, a display order of the video clip pairs in the set and display positions of a target video clip and a corresponding reference video clip included in each video clip pair on a display screen of the respective user, giving rise to the one or more test sets corresponding to the one or more test sessions; Sending, by the processor, the one or more test sets to the one or more users for assessing quality of the target video clip and the reference video clip in each video clip pair in one or more sets of video clip pairs in the one or more test sets; Receiving, by the processor, feedback regarding the assessed quality from the one or more users
- This aspect of the disclosed subject matter can comprise one or more of features (a) to (g) listed above with respect to the system, mutatis mutandis, in any desired combination or permutation which is technically possible.
- a computerized method for video quality assessment comprising: Receiving, by a processor, a test set including a set of video clip pairs selected from a plurality of video clip pairs to be displayed to a user, each video clip pair including a target video clip and a reference video clip, a display order of the video clip pairs in the set, and display positions of the target video clip and the reference video clip in each video clip pair on a display screen of the user; Displaying, by the processor, the video clip pairs in the set according to the display order and the display positions in the test set, thereby enabling the user to perform a test session and assess quality of the target video clip and the reference video clip in each video clip pair; Collecting, by the processor, feedback regarding the assessed quality for each video clip pair and sending the feedback to a server, wherein the feedback is usable for providing subjective quality evaluation of the plurality of video clip pairs.
- this aspect of the disclosed subject matter can comprise one or more of features (i) to (vi) listed above with respect to the system, mutatis mutandis, in any desired combination or permutation which is technically possible.
- the computerized method for video quality assessment may further comprise an automation unit which is further configured to perform: Receiving or obtaining a configuration comprising source files and indication of corresponding frames to be used, encoder binaries and encoding specifications; Encoding of indicated frames from said source files using said encoder binaries according to said specifications, giving rise to a plurality of video clip pairs to be used in the process of video quality assessment.
- the automation unit may be further configured to write the information pertaining to the plurality of video clip pairs into the server database.
- the automation unit may be further configured to control one or more aspects of performing user test sessions or a test round including: deployment of one or more test sessions, posting a job or multiple jobs on a crowdsourcing marketplaces, collecting and managing completed test sessions and managing payment to the one or more users who completed a test session.
- a non-transitory computer readable storage medium tangibly embodying a program of instructions that, when executed by a computer, cause the computer to perform the method steps of any of the methods disclosed above.
- FIG. 1 illustrates a block diagram of a computerized system for subjective quality evaluation in accordance with certain embodiments of the presently disclosed subject matter
- FIG. 1B illustrates the data flow in a block diagram of a computerized system for subjective quality evaluation in accordance with certain embodiments of the presently disclosed subject matter
- FIG. 3 illustrates a generalized flowchart of the client or user-side application component of a computerized system for subjective quality evaluation in accordance with certain embodiments of the presently disclosed subject matter
- FIGS. 4 and 4B Illustrate example flows of video pair display in the client or user-side application component of a computerized system for subjective quality evaluation in accordance with certain embodiments of the presently disclosed subject matter;
- FIG. 5 illustrates a block diagram of a computerized system for automatic large-scale subjective quality evaluation in accordance with certain embodiments of the presently disclosed subject matter
- FIGS. 6A, 6B and 6C illustrate example screen captures from the client or user-side application component of a computerized system for subjective quality evaluation in accordance with certain embodiments of the presently disclosed subject matter
- FIG. 7 illustrates an example of a scenes file used for automated subjective quality evaluation in accordance with certain embodiments of the presently disclosed subject matter
- FIG. 9 illustrates an example of distribution of subjective quality results for a set of video pairs, obtained in accordance with certain embodiments of the presently disclosed subject matter
- computer should be expansively construed to cover any kind of hardware-based electronic device with data processing capabilities including, by way of non-limiting example, the system/apparatus and parts thereof as well as the control circuit/circuitry therein disclosed in the present application.
- non-transitory memory and “non-transitory storage medium” used herein should be expansively construed to cover any volatile or non-volatile computer memory suitable to the presently disclosed subject matter.
- the input to the system described herein are a set of video streams, video clips, or video files and the output of the system is configured to provide information regarding the subjective opinions of viewers who viewed video stream pairs.
- the system includes a server with a primary task to create test sets comprising multiple side by side video pairs, and a client application with the primary role of displaying video pairs to a viewer and receiving the viewer subjective opinion regarding relative quality of the videos in the pair.
- client and the server are implemented by way of a processor configured to perform the relevant operations, for example by way of a software that generates commands executed by a CPU.
- the computerized system for visual subjective quality evaluation may consist of two components: A client and a server.
- these components may each be installed and activated on a separate general purpose computer, in other embodiments they may be located on the same computer and in yet other embodiments the server may be located on a virtual machine or in the cloud, while the client is installed and run on multiple user computers or devices.
- test set refers to a set of video pairs that are to be viewed in a specific test session.
- a test set may have associated with it a set of instructions, which indicate as well as the clip pairs to be used in the test set, also instructions regarding display order of these pairs i.e. which pair is played first, which second etc. and may or may not include also accompanying metadata on the test set such as display position—which video shall be displayed on the left and which on the right for each pair, how many replays of each pair are supported etc.
- test session refers to a complete test performed by a single user and may or may not include also the feedback from the user and metadata related to the test session such as a number of times each video pair was displayed to the user.
- test round refers to multiple test sessions performed by one or more users.
- the system described herein enables collecting user opinions for subjective assessment of visual quality. This can be used for the purpose of development, calibration and verification of video content quality, video compression quality and/or video quality metrics.
- the system comprises a front-end or client application—which is what users see and experience, and the backend or server which is responsible for example for providing the clips, or links to the clips, to be viewed, collecting the user inputs, and/or creating a results database.
- FIG. 1 there is illustrated a block diagram of a computerized system for subjective quality evaluation in accordance with certain embodiments of the presently disclosed subject matter.
- this system may comprise two primary building blocks, A Visual Subjective Testing Application (VISTA) server 110 and a Visual Subjective Testing Application (VISTA) client 160 .
- this system may further comprise a video storage unit 105 , which may be part of the server block 110 or external to it.
- the video storage unit 105 contains the actual video streams or video files that are to be compared during the viewing session.
- video stream video content and video files are used interchangeably herein, and all terms refer to the video content that is to be evaluated.
- the videos in the storage unit 105 may reside on the same physical computer, on different physical computers, or use cloud resources—which is in particular beneficial for the server block 110 and storage block 105 .
- cloud resources which is in particular beneficial for the server block 110 and storage block 105 .
- a single server entity is used to serve multiple client invocations—serially, sequentially, in parallel, or a combination thereof.
- each of the server 110 and client 160 may comprise a number of internal blocks, details of which follow shortly. However first, in order to help clarify some of the terms used, an explanation on the system is provided, in accordance with certain embodiments of the presently disclosed subject matter.
- a method of quality grading or scaling should be selected.
- two types of subjective video quality evaluation tasks may be defined as: 1. Making sure that a ‘target’ encode has the same perceptual quality as a ‘source’—be that a source video (compressed or uncompressed) or a source encode. 2. Comparing two encodes, obtained for example by two different encoders or by an encoder used with two different configurations, to see which provides better subjective quality.
- These two testing modes can for example use the Forced-Choice Double-Stimulus test to perform the quality evaluation, as this test compares two video sequences to each other, and the quality observation is relative rather than absolute.
- the VISTA client It is desired to make the VISTA client a seamless and easy to use application, which operates in a fully automated manner, and does not require the user to go through multiple or complicated steps to perform the evaluation, thus allowing the user to focus purely on the video quality assessment. Therefore, it is recommended to design the client application so that the usage is very simple, with video retrieval and playback being automatic and using only a minimalistic API.
- the client may include display of a brief introduction on how to perform the test.
- the BeamrView video viewer, or a sub-set of that package, which supports video pair decode and synchronized side-by-side playback, may be integrated into the client application and used for displaying the video pairs.
- test set configuration in a non-limiting example of a possible embodiment.
- the videos on one of the video sets as ‘reference’, or ‘source’
- the videos of the other video set as ‘target’.
- Most of the video pairs in each test set may be associated with the desired comparison or evaluation.
- additional pairs may be created for which the target clip is visibly degraded, meaning it (clearly) has lower visual quality than the source.
- each test set may consist of 20 video pairs. These 20 pairs in turn may consist of fourteen pairs of the actual target vs. source comparisons with difficulty 5, four user validation pairs with difficulty 0 and two cases where both sides show the same stream for statistics purposes, with difficulty 10.
- the VISTA backend may construct each test set to be semi-random, or pseudo-random, thus avoiding long sequences of source being displayed on the same side and dispersing the user validation tests semi-uniformly along the test.
- the selection of videos to use in each test is also semi-randomized, in a way that equalizes the numbers of viewings of each source-target video pair across multiple client sessions.
- the Video pairs Database 130 is a block that may store the aforementioned links or pointers to the video files or streams, and optionally video pair metadata which may comprise of one or more of the following: pair difficulty level, video resolution (video frame dimensions), video frame rate, video duration indicated by number of frames or as measured in a time unit such as milliseconds, and possibly other properties of the video content or its creation process.
- a processor unit configured to generate test sets is termed ‘test set Generator’ 120 . It generates the test set for each user session, using a configuration either stored within the Generator or supplied externally as explained in regard to FIG. 1B .
- the output from system 100 may comprise results collected in the test round directly from a Result Database 150 , such as but not limited to, the feedback collected from each user or viewer on each displayed clip pair, possibly with additional data such as viewing duration, number of replays etc.
- the output 165 may be provided after processing in an Analysis unit 140 and comprise the processed result(s) of the subjective quality evaluation test round, for example information on the statistical certainty of perceptual identity of the target video pairs, or in another example per user and/or per clip statistics such as percentage of viewings where target clip was selected as having lower quality for each target clip pair.
- the interfaces between the units may comprise flow 145 illustrating a Server 110 providing test set information to a client 160 , wherein this information is used by a session Controller 170 to control the test session.
- the interfaces may further comprise flow 135 wherein a Video storage 105 provides files required for the test session to a client 160 for example using http download (or any other transport protocol e.g. “copy”, “secure copy”, “ftp” etc.) and may further comprise flow 155 wherein a client 160 reports test results to a Server 110 and wherein the test results may comprise selections made by the user, number of replays per clip, user screen resolution, selected video frame area in the case cropping was used etc.
- each session can consist of certain number of video pairs as determined for example by an input configuration.
- the pairs used in a test set will be selected randomly from a prepared set of video pairs. Generally, it is advisable that in about 50% of the cases the display position will be such that target clip shall be on the left, and in about 50% of the cases the target clip shall be on the right.
- each session some, for example 4, of the pairs may comprise a ‘source’ or reference video and a ‘target’ video that has been significantly degraded intentionally, to check the reliability of the user. If the user does not correctly identify the degraded video in this pair, the results of the whole session may be ignored.
- FIG. 2 there is illustrated a generalized flowchart of a server component, or backend, of a computerized system for subjective quality evaluation in accordance with certain embodiments of the presently disclosed subject matter.
- the server receives a user ID in block 210 transmitted from the client.
- This ID may for example comprise a user email, a user worker ID assigned by an underlying crowd-sourcing framework or any such similar user ID.
- This ID may be used within VISTA for example to identify the results associated with a specific user, and possibly to manage user compensation.
- the Test set Generator prepares a test set and sends it to the client.
- the test set includes indications of which video files or streams to use in the test session which for example may be presented as URLs pointing to the video files location, and may also include further information in the test set such as one or more of the following non limiting examples: Information for each pair re which clip to show on which side of the screen, cropping coordinates of the video frames when video dimensions are such that side by side view of the full frame on left and right exceed the user screen resolution and information re configuration of the test such as how many replays are allowed.
- each test set contains 20 pairs of videos, which are selected from a pre-determined set of video-pairs.
- the selection of videos from the full set or database is semi-randomized, so that all videos are included in about the same number of test-sets.
- each test set includes exactly 4 verification pairs, in one embodiment of the subject matter presented herein, it is possible that these pairs will be placed in display order within the test set as follows: 1st verification video in random slot chosen from slots 2-5; 2nd verification video in random slot chosen from slots 6-10; 3rd verification video in random slot chosen from slots 11-15; 4th verification video in random slot chosen from slots 16-20.
- the client Upon completion of the test session by the user, the client informs the server that the session is complete and in cases where the user requests a confirmation code—for example, in order to receive compensation for the viewing task, the server in 230 “calculates” or uses some hash algorithm to create a confirmation token which is sent to the client and displayed to the user.
- the results of the test session comprising at least the selections made by the user, and possibly further information such as number of replays, actual viewed area of cropped frames, duration of test etc. are sent by the client and received and stored by the server in 240 .
- the receiving and storing is performed at the end of the test session
- the client-server communication is ongoing throughout the test session so that after the user makes a selection at the end of each clip pair this information is aggregated and stored by the server.
- the client may download all the video files for the test, as indicated in the test set instructions provided by the backend.
- the client may download all the video files for the test, as indicated in the test set instructions provided by the backend.
- the clips By downloading the clips and then playing them back from local storage rather than streaming them over the Internet, network issues which may impact video quality are avoided.
- a stage corresponding to 340 the user goes through one or more brief explanation screens, such as the screen shown in FIG. 6A . More details on this block are provided below in reference to FIG. 4 and FIG. 4B .
- the user can submit the test results and in 360 the client sends results of the test session to the server.
- results include at least the user selections or feedback and optionally also additional information such as but not limited to number of views per clip pair, duration of the viewings etc.
- this data it is possible for this data to be transmitted during the test session, for example for each pair after a decision is provided by the user, in which case block 360 is effectively ongoing throughout the test session.
- the client may optionally display to the user a confirmation token received from the server, for example to use when requesting reimbursement for the task. The test session then concludes.
- FIG. 6B An example screen shot of the viewer screen at this stage is shown in FIG. 6B .
- the client Upon completion of playback, when maximum replays was not reached, in block 440 the client requests the user to select among options “left has lower quality”, “right has lower quality” and “replay” as illustrated in the screen capture shown in FIG. 6C . If the user chooses to replay, the client returns to block 410 . Otherwise, the flow proceeds to block 460 where the information regarding the user selection and accompanying data such as number of replays, duration, processing applied etc. is stored by the client and transmitted back to the server, at some time prior to session completion.
- the client UX flow may be as follows.
- the user When the user launches the app, the user may be requested to provide an ID, which can be either an email address or Amazon Mechanical Turk WorkerID. This component is completely optional and may not be used in some embodiments.
- the top bar may display “Video x out of y” and an ‘X” button on the top right corner for closing the app.
- a confirmation dialog may be displayed when the user presses the “X” button. It is recommended that the bar with the selection buttons is not shown during playback, to avoid distraction.
- a grey screen may be shown, with 3 buttons: “Left is lower quality”, “Right is lower quality” and “Replay video”.
- the text message on the screen may be “Select an option below to continue”.
- the user shall indicate which video has lower quality in their opinion, by clicking on one of the selection buttons “Left is lower quality” and “Right is lower quality”. After the user selects the lower quality video, the next clip pair will be loaded and played.
- the client application may be built for a variety of operating systems including but in no way limited to, Windows, Mac OS and Linux.
- FIG. 5 there is illustrated a block diagram of a computerized system for automatic large-scale subjective quality evaluation in accordance with certain embodiments of the presently disclosed subject matter.
- Auto-VISTA is a combination of the VISTA server, VISTA client and an automation unit.
- the Auto-VISTA question is represented by block 510 .
- the goal of an Auto-VISTA testing round is to answer a question, so defining the question is crucial for building the test correctly and for finding the answer we seek.
- the question is generally aimed at comparing two encoding solutions, which may be the same encoder with two different configurations, or two different encoders. The comparison seeks to determine whether one provides superior visual quality to the other, or whether the encodes are perceptually identical to each other. The latter case is of interest for example when the encoding solutions differ in CPU performance or in the bitrate they provide.
- one of the encoders to be encoder A associated with a set of N video streams, or video clips ⁇ VA ⁇ , and the second encoder to be encoder B creating video streams labeled as ⁇ VB ⁇ .
- parameters controlling the test set structure need to be configured. These include the number of pairs of each difficulty level to include in a test set, e.g. number of target pair comparisons, number of user-validation pairs and number of statistical-validation pairs, per each test session. They also include other parameters controlling the test sessions, such as the number of replays allowed prior to forcing the choice of the lower quality clip.
- the VISTA server or backend stores pointers or links to all clips used for testing.
- these clips will be created in the next step—the automated session creation.
- This for example may be done by employing a cloud-based encoding solution which for example may use Dockers containing encoder binaries and supports deployment of the encoding tasks on the cloud, for example using AWS EC2 instances.
- the appropriate Docker images and corresponding configuration for each of the ‘source’, ‘target’ and ‘user-validation’ encodes must be prepared and specified.
- a CSV file describing the scenes to encode e.g. the selected frames for each source file, is required.
- This CSV contains the location of the source files and descriptors such as which frames should be encoded, dimensions and frame rate of the input and encoded streams, and optional additional descriptors such as target bitrate per scene.
- An example of such a file is provided in FIG. 7 .
- the S3videoPath indicates location of the source content from which the VISTA source, target and user validation clips will be created
- srcWidth srcHeight are the dimensions of the source clip
- width and height are the dimensions to encode the scene to
- fps is the frame rate of the video
- numFrames indicates how many frames should be encoded, starting from startOffset provided in format hh::mm:ss.
- Additional fields include the format of the video pixels in the source clip and encoded scenes and possible additional encoding parameters such as target bitrate for scene encoding.
- an Auto-VISTA session is automatically created with the following steps. First a new Auto-VISTA session ID is created, and all required resources in the backend are allocated for the session. Then the user inputs are mapped to provide the full set of inputs and configurations required to automatically run the automated video encoding utility we developed.
- This utility receives as inputs JSON files describing the required run, and pointers to Docker images containing the binaries and environment on which to perform the run. It then launches an Amazon EC2 spot instance for each run, executes the run, and uploads results to the Amazon S3 bucket specified in the JSON file. Once scene creation is complete, pointers to all the video stream pairs and their accompanying metadata such as duration, resolution and difficulty level of the clip pair are stored in the VISTA backend DynamoDB.
- crowdsourcing marketplaces are considered by many a way to make humans and their computers an available, scalable, resource.
- One such platform is Amazon Mechanical Turk, or MTurk, which is very well suited for large-scale user testing tasks.
- the participants, termed ‘Workers’ receive a Human Intelligence Task or HIT, which in this case involves downloading the VISTA client and performing a test session, and are paid when the results are submitted. This provides an almost infinite resource of prospective viewers and enables fast turnaround at relatively low costs. It also enables very easy scaling of testing group size as desired for each round of testing.
- the session can be started, or set as active. This will cause all VISTA clients starting the test process to receive test files belonging to this active session.
- MTurk this is done by creating a HIT consisting of instructions to download and run the VISTA client. Workers who then accept this HIT follow the instructions, and upon reaching the submit stage, receive a confirmation token from the VISTA backend. The users provide this confirmation token to get paid or for the task. It is also possible to automatically filter out users who did not correctly select the degraded clip in the user validation pairs, thus guaranteeing that the human performing the test actually payed attention when watching the videos, and did not just randomly press the buttons to complete the test as quickly as possible.
- block 570 is activated. Note that it is also possible to activate this block periodically and obtain intermediate results and use these to determine when enough results have been collected.
- the test reports are retrieved from the backend. These reports contain all the information on each clip in each testing session. An example of a result set example is provided in FIG. 8 .
- the Video ID is an internal label associated with the clip pair
- Original and Degraded are paths to the file locations on Amazon S3
- Difficulty is as explained above
- #Plays indicates how many times the user viewed the clip pair
- X0 Y0 and X1 Y1 are the displayed cropped frame upper left and lower right coordinates respectively
- Width and Height are the full clip dimensions
- Duration is the clip length in milliseconds.
- Target on describes on which side clip “B” was presented, and Selected is the side the user selected as having lower quality.
- User provides the user ID.
- At least some of the analysis of the test results may be performed within the server 110 . In other embodiments this analysis is done externally, entirely in block 570 .
- the analysis may be done using some automated tool, for example implemented by way of Python scripts. Results from all the test sets performed are collected and tests from non-valid users—for example users who made the incorrect selection in the user-validation pairs, may be discarded or filtered out.
- the analysis unit then analyzes the results for the valid pairs—target pairs viewed by valid users, and calculates the statistical distribution of the user selections—as to which clip was has lower quality in each pair, and may also calculate the distribution of selections overall for all the evaluated A/B comparisons. This leads directly to the ‘answer’ depicted in block 590 .
- Block 590 represents the answer or the output of the system in the example embodiment of FIG. 5 .
- the answer to the initial question is immediate.
- the answer is “yes” if the percentage of selecting B as lower quality is close enough to 50%, with the distance from 50% which is acceptable. Determining what “close enough” is, can possibly be derived from the statistics of the pairs used for statistical significance, and possibly can be calculated using other statistics based tools or methodologies.
- the answer is “yes” if A was selected as lower quality in more than 50% of the cases, with the certainty of that answer possibly depending on the distance from 50% compared to the deviation from 50% of the files used for statistical purposes, or possibly using other statistical analysis tools.
- FIG. 9 there is illustrated an example of the statistical distribution of subjective quality results for a set of video pairs, obtained in accordance with certain embodiments of the presently disclosed subject matter. Percentage of views where ‘target’ clip was selected as lower quality per clip pair, shown for pairs with difficulty 5—depicted as blue (dark) bars, and difficulty 10—depicted as green (light) bars. As can be seen, the distribution is close to 50:50 and further analysis of the obtained results for this example showed that the two video sets being compared were indeed perceptually identical with a confidence level within the 95 th percentile.
- these teachings provide for video quality assessment such that reliable information regarding perceived subjective quality of a set of video clips, or comparative perceived subjective quality of pairs of video clips, can be obtained in a fast, scalable and cheap procedure.
- the system according to the invention may be, at least partly, implemented on a suitably programmed computer.
- the invention contemplates a computer program being readable by a computer for executing the method of the invention.
- the invention further contemplates a non-transitory computer-readable storage medium tangibly embodying a program of instructions executable by the computer for executing the method of the invention.
Abstract
Description
-
- (a) A client unit operatively connected to the server, the client unit comprising an additional processor configured to perform a given test session by a given user, wherein the client unit is configured to: Receive said test set; Display the video clip pairs in the set according to the display order and the display positions in the test set, thereby enabling the given user to perform the given test session and assess quality of the target video clip and reference video clip in each video clip pair; Collect feedback regarding the assessed quality for each video clip pair and send the feedback to said server, wherein the feedback is usable for providing subjective quality evaluation of the plurality of video clip pairs.
- (b) The set of video clip pairs includes one or more video clip pairs each with visible degradation in quality between the target video clip and the corresponding reference video clip, wherein the one or more video clip pairs are used for user validation.
- (c) The set of video clip pairs include one or more video clip pairs used for statistical validation, wherein the target video clip and the reference video clip in each of the one or more video clip pairs are identical.
- (d) The processor is further configured to select the set of video clips pairs from the database for each test set, so that each of the plurality of video clip pairs is included in a pre-determined number of the one or more test sets.
- (e) The display positions of the target video clip and the reference video clip in each video clip pair are determined in a semi-randomized manner.
- (f) The display order of the video clip pairs in the set is determined in a semi-randomized manner.
- (g) The display order of the video clip pairs in the set is determined at least by inserting the one or more video clip pairs with visible degradation in one or more semi-randomized slots along the test set.
-
- (i) The processor is a client application or unit configured to perform said receiving, displaying, collecting and sending in a fully automated manner.
- (ii) The set of video clip pairs include one or more video clip pairs each with visible degradation in quality between the target video clip and the corresponding reference video clip, wherein the one or more video clip pairs are used for user validation.
- (iii) The set of video clip pairs include one or more video clip pairs used for statistical validation, wherein the target video clip and the reference video clip in each of the one or more video clip pairs are identical.
- (iv) The display positions of the target video clip and the reference video clip in each video clip pair are determined in a semi-randomized manner
- (v) The display order of the video clip pairs in the set is determined in a semi-randomized manner.
- (vi) The display order of the video clip pairs in the set is determined at least by inserting the one or more video clip pairs with visible degradation in one or more semi-randomized slots along the test set.
Claims (17)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/880,646 US11178394B2 (en) | 2019-07-16 | 2020-05-21 | System and method of subjective video quality evaluation |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962874522P | 2019-07-16 | 2019-07-16 | |
US16/880,646 US11178394B2 (en) | 2019-07-16 | 2020-05-21 | System and method of subjective video quality evaluation |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210021805A1 US20210021805A1 (en) | 2021-01-21 |
US11178394B2 true US11178394B2 (en) | 2021-11-16 |
Family
ID=74346248
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/880,646 Active US11178394B2 (en) | 2019-07-16 | 2020-05-21 | System and method of subjective video quality evaluation |
Country Status (2)
Country | Link |
---|---|
US (1) | US11178394B2 (en) |
IL (1) | IL274803B2 (en) |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100053300A1 (en) * | 2007-02-02 | 2010-03-04 | Einarsson Torbjoern | Method And Arrangement For Video Telephony Quality Assessment |
US7683931B2 (en) * | 2006-03-06 | 2010-03-23 | Dell Products L.P. | Image artifact detection in video quality benchmarks |
US8654181B2 (en) * | 2011-03-28 | 2014-02-18 | Avid Technology, Inc. | Methods for detecting, visualizing, and correcting the perceived depth of a multicamera image sequence |
US8817110B1 (en) * | 2013-10-07 | 2014-08-26 | Advanced Testing Technologies, Inc. | Instrument card for video applications |
US8917777B2 (en) * | 2008-07-21 | 2014-12-23 | Huawei Technologies Co., Ltd. | Method, system and apparatus for evaluating video quality |
US9064154B2 (en) * | 2012-06-26 | 2015-06-23 | Aol Inc. | Systems and methods for associating electronic content |
US9319670B2 (en) * | 2011-04-11 | 2016-04-19 | Huawei Technologies Co., Ltd. | Video data quality assessment method and apparatus |
-
2020
- 2020-05-20 IL IL274803A patent/IL274803B2/en unknown
- 2020-05-21 US US16/880,646 patent/US11178394B2/en active Active
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7683931B2 (en) * | 2006-03-06 | 2010-03-23 | Dell Products L.P. | Image artifact detection in video quality benchmarks |
US20100053300A1 (en) * | 2007-02-02 | 2010-03-04 | Einarsson Torbjoern | Method And Arrangement For Video Telephony Quality Assessment |
US8917777B2 (en) * | 2008-07-21 | 2014-12-23 | Huawei Technologies Co., Ltd. | Method, system and apparatus for evaluating video quality |
US8654181B2 (en) * | 2011-03-28 | 2014-02-18 | Avid Technology, Inc. | Methods for detecting, visualizing, and correcting the perceived depth of a multicamera image sequence |
US9319670B2 (en) * | 2011-04-11 | 2016-04-19 | Huawei Technologies Co., Ltd. | Video data quality assessment method and apparatus |
US9064154B2 (en) * | 2012-06-26 | 2015-06-23 | Aol Inc. | Systems and methods for associating electronic content |
US8817110B1 (en) * | 2013-10-07 | 2014-08-26 | Advanced Testing Technologies, Inc. | Instrument card for video applications |
Non-Patent Citations (20)
Title |
---|
"Amazon DynamoDB Fast and Flexible NoSQL database service for any scale", 14. https://aws.amazon.com/dynamodb/ (May 23, 2019). |
"Amazon Mechanical Turk", https://www.mturk.com/ (Jul. 6, 2019). |
"Beamr Optimizer", https://beamr.com/video-content-adaptive-optimizer (Oct. 30, 2018). |
"Beamr Riew Video Comparison Tool", https://beamr.com/h264-hevc-video-comparison-player/ (May 23, 2019). |
"Cisco Visual Networking Index: Forecast and Trends, 2017-2022 White Paper" https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-741490.html (Feb. 27, 2019 ). |
Binomial Distribution Confident Interval, www.wolframalpha.com/entities/calculators/binomial_distribution_confidence_interval/ (Jul. 6, 2019). |
Crowd-sourced subjective quality evaluation platform, www.subjectify.us, Jun. 30, 2019, pp. 1-6. |
ITU-T p. 910 "Subjective video quality assessment methods for multimedia applications", 2008. |
ITU-T p. 913, "Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment", 2016. |
Marr, "How Much Data Do We Create Every Day? The Mind-Blowing Stats Everyone Should Read", https://www.forbes.com/sites/bernardmarr/2018/05/21/how-much-data-do-we-create-every-day-the-mind-blowing-stats-everyone-should-read, May 21, 2018. |
R. Rassool, "VMAF reproducibility: Validating a perceptual practical video quality metric," 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), Cagliari, 2017. |
Rafal K. Mantiuk, Anna Tomaszewska, and Radosław Mantiuk, "Comparison of Four Subjective Methods for Image Quality Assessment", Computer Graphics Forum Dec. 31, 2012, pp. 2478-2491. |
Recommendation ITU-R BT.500-13, "Methodology for the subjective assessment of the quality of television pictures", 2012. |
Recommendation ITU-R BT.710-4, "Subjective Assessment Methods for Image Quality in High-Definition Television", 1990-1998. |
Report ITU-R BT.1082-1, "Studies Towards the Unification of Picture Assessment Methodologies", 1986-1990. |
Sheikh, Hamid; Bovik, Alan. "Image Information and Visual Quality". IEEE Transactions on Image Processing. 15 (2): 430-444. |
Shoham, T., Gill, D., Carmel, S., Terterov, N. and Tiktov, P., "Content-adaptive frame level rate control for video encoding using a perceptual video quality measure," Proc. SPIE 11137, Applications of Digital Image Processing XLII, Aug. 2019. |
Tamar Shoham, Dror Gill, Sharon Carmel, "A novel perceptual image quality measure for block based image compression," Proc SPIE 7867, Image Quality and System Performance VIII, 786709, 2011. |
Wang, Zhou, et al. "Image quality assessment: from error visibility to structural similarity." IEEE transactions on image processing 13.4 (Apr. 2004): 600-612. |
Zhi Li, Anne Aaron et al., "Toward A Practical Perceptual Video Quality Metric," Netflix Te chBlog, Jun. 2016. |
Also Published As
Publication number | Publication date |
---|---|
IL274803B2 (en) | 2023-06-01 |
US20210021805A1 (en) | 2021-01-21 |
IL274803A (en) | 2021-01-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Ghadiyaram et al. | In-capture mobile video distortions: A study of subjective behavior and objective algorithms | |
US11405695B2 (en) | Training an encrypted video stream network scoring system with non-reference video scores | |
Barman et al. | QoE modeling for HTTP adaptive video streaming–a survey and open challenges | |
US11216698B2 (en) | Training a non-reference video scoring system with full reference video scores | |
Bampis et al. | Towards perceptually optimized end-to-end adaptive video streaming | |
Upenik et al. | Testbed for subjective evaluation of omnidirectional visual content | |
Rehman et al. | Display device-adapted video quality-of-experience assessment | |
Pinson et al. | Selecting scenes for 2D and 3D subjective video quality tests | |
Bampis et al. | Towards perceptually optimized adaptive video streaming-a realistic quality of experience database | |
Tavakoli et al. | Quality of experience of adaptive video streaming: Investigation in service parameters and subjective quality assessment methodology | |
EP3427481A1 (en) | Method and system for automatic user quality-of-experience measurement of streaming video | |
Duanmu et al. | A knowledge-driven quality-of-experience model for adaptive streaming videos | |
Lopes et al. | Subjective and objective quality assessment of omnidirectional video | |
Søgaard et al. | On subjective quality assessment of adaptive video streaming via crowdsourcing and laboratory based experiments | |
Devlic et al. | QoE-aware optimization for video delivery and storage | |
Huang et al. | Perceptual quality driven frame-rate selection (PQD-FRS) for high-frame-rate video | |
Shang et al. | Subjective and objective video quality assessment of high dynamic range sports content | |
Guo et al. | Perceptual quality assessment of immersive images considering peripheral vision impact | |
Zhang et al. | Quality-of-Experience Evaluation for Digital Twins in 6G Network Environments | |
US11178394B2 (en) | System and method of subjective video quality evaluation | |
Li et al. | Quantifying the influence of devices on Quality of Experience for Video streaming | |
Xie et al. | Determining bitrate requirement for UHD video content delivery | |
US20170194035A1 (en) | Video Capture and Testing of Media Player Playback of a Video | |
Sugito et al. | Practical use suggests a re-evaluation of HDR objective quality metrics | |
Shoham et al. | Overnight large-scale subjective video quality assessment using automatic test generation and crowdsourcing Internet marketplaces |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
AS | Assignment |
Owner name: BEAMR IMAGING LTD, ISRAEL Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GILL, DROR;SHOHAM, TAMAR;JULIUS, DAN;AND OTHERS;SIGNING DATES FROM 20200526 TO 20200603;REEL/FRAME:053290/0231 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
AS | Assignment |
Owner name: BEAMR IMAGING LTD, ISRAEL Free format text: CHANGE OF ASSIGNEE ADDRESS;ASSIGNOR:BEAMR IMAGING LTD;REEL/FRAME:057542/0118 Effective date: 20200825 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: SILICON VALLEY BANK, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:BEAMR IMAGING LTD.;REEL/FRAME:059097/0195 Effective date: 20220217 |