CN108712645B - Video quality parameter acquisition method, device, equipment, system and storage medium - Google Patents
Video quality parameter acquisition method, device, equipment, system and storage medium Download PDFInfo
- Publication number
- CN108712645B CN108712645B CN201810399205.XA CN201810399205A CN108712645B CN 108712645 B CN108712645 B CN 108712645B CN 201810399205 A CN201810399205 A CN 201810399205A CN 108712645 B CN108712645 B CN 108712645B
- Authority
- CN
- China
- Prior art keywords
- video
- parameter
- quality
- pilot
- videos
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
Abstract
The invention provides a method, a device, equipment, a system and a storage medium for acquiring video quality parameters, wherein the method comprises the following steps: splitting each section of video in a video group to be evaluated into two sections of videos, and respectively dividing the two sections of videos into a test set and a guide set; acquiring a target quality parameter of the video in the pilot set, and extracting a pilot video subset for representing the video quality range of the pilot set from the pilot set according to the target quality parameter; and determining the test quality parameters of the videos in the test set based on the video quality parameters obtained when the test set is played, wherein the pilot video subset is played before the test set is tested. According to the embodiment of the disclosure, a group of video groups similar to the video quality range of the test set is played before the test set video is played, so that the video quality range of the test set is reminded, and the accuracy of obtaining the video quality parameters can be improved.
Description
Technical Field
The present invention relates to the field of video technologies, and in particular, to a method, an apparatus, a device, a system, and a storage medium for acquiring video quality parameters.
Background
With the explosion of internet and multimedia technology, watching video has become one of the main entertainment ways in daily life, and especially as a product of both the live broadcast industry and the game industry, live game broadcast is more and more popular with users. The pursuit of users to the watching experience is also continuously improved along with the popularization of online videos, the video quality determined by the factors such as video definition and the like is a key factor influencing the watching experience of the users, and therefore, the method has important significance on how to effectively obtain the video quality parameters.
At present, an evaluating end often directly displays a video to be evaluated and receives video quality parameters input by an evaluating party. However, the video quality parameters obtained by the evaluation terminal have low accuracy due to different video quality ranges of different video groups to be evaluated.
Disclosure of Invention
The invention provides a method, a device, equipment, a system and a storage medium for acquiring video quality parameters, which aim to overcome the defect of low accuracy of the video quality parameters in the prior art.
According to a first aspect of the embodiments of the present invention, there is provided a method for obtaining video quality parameters, the method including:
splitting each section of video in a video group to be evaluated into two sections of videos, and respectively dividing the two sections of videos into a test set and a guide set;
acquiring a target quality parameter of the video in the pilot set, and extracting a pilot video subset for representing the video quality range of the pilot set from the pilot set according to the target quality parameter;
and determining the test quality parameters of the videos in the test set based on the video quality parameters obtained when the test set is played, wherein the pilot video subset is played before the test set is tested.
In one embodiment, the obtaining the target quality parameter of the video in the preamble set includes:
sending the leader set to at least one preliminary evaluation terminal, and receiving parameter groups fed back by each preliminary evaluation terminal, wherein the parameter groups comprise initial quality parameters of each section of video in the leader set;
converting the initial quality parameters of each group of parameter groups to convert the quality parameter range of each group of parameter groups into a specified quality parameter range;
and determining data representing the quality of the video corresponding to the parameter group from the converted parameter group to obtain the target quality parameter of the video in the leader set.
In one embodiment, the initial quality parameter of each set of parameters is converted using the following formula, such that the quality parameter range of each set of parameters is converted into a specified quality parameter range:
wherein x is
maxRepresenting the maximum value, x, in the parameter set
minRepresenting the minimum value in the parameter set, x representing the initial quality parameter in the parameter set, x
newRepresenting data obtained by converting x, k
maxUpper limit value, k, representing a range of specified quality parameters
minRepresenting a lower value of the specified quality parameter range.
In one embodiment, the determining the test quality parameter of the video in the test set based on the video quality parameter obtained when the test set is played includes:
sending the pilot video subset and the test set to at least one target evaluation terminal, wherein the target evaluation terminal meets the following requirements: playing a pilot video subset before playing a test set, and receiving input video quality parameters of videos in the test set when playing the videos in the test set;
receiving video quality parameters of the videos in the test set fed back by the target evaluation terminal;
and merging the video quality parameters corresponding to each section of video in the test set to obtain the test quality parameters of the video in the test set.
In one embodiment, the subset of the pilot videos at least includes videos corresponding to the highest, lowest and middle values of the target quality parameter, and the video corresponding to the middle value of the target quality parameter is marked with the target quality parameter.
In one embodiment, the videos in the subset of the pilot videos are arranged according to a designated playing sequence, the video corresponding to the intermediate value of the target quality parameter is arranged at the head, and other videos except the video corresponding to the intermediate value of the target quality parameter in the subset of the pilot videos are randomly arranged.
According to a second aspect of the embodiments of the present invention, there is provided a video quality parameter acquisition system, including a processing terminal, a preliminary evaluation terminal, and a target evaluation terminal;
the processing end divides each section of video in the video group to be evaluated into two sections of video, and divides the two sections of video into a test set and a pilot set respectively, and sends the pilot set to at least one preliminary evaluation end;
the preliminary evaluation end plays the pilot set and obtains a parameter set based on an input instruction, wherein the parameter set comprises initial quality parameters of each video segment in the pilot set;
the processing end determines target quality parameters of videos in the pilot set according to parameter groups fed back by each pre-evaluation end, extracts a pilot video subset used for representing the video quality range of the pilot set from the pilot set according to the target quality parameters, and sends the pilot video subset and the test set to at least one target evaluation end;
the target evaluation terminal plays the pilot video subset before playing the test set and receives the input video quality parameters of the videos in the test set when playing the videos in the test set;
the processing end receives the video quality parameters of the test concentrated video fed back by the target evaluation end; and merging the video quality parameters corresponding to each section of video in the test set to obtain the test quality parameters of the video in the test set.
In one embodiment, the subset of the pilot videos at least includes videos corresponding to the highest, lowest and intermediate values of the target quality parameter;
and when the target evaluation terminal plays the video of the test set, the video corresponding to the intermediate value of the target quality parameter is played at the same time, and the configuration parameters of the display equipment playing the video of the test set are the same as the configuration parameters of the display equipment playing the video corresponding to the intermediate value of the target quality parameter. According to a third aspect of the embodiments of the present invention, there is provided a video quality parameter acquisition apparatus, including:
the video splitting module is used for splitting each section of video in the video group to be evaluated into two sections of video and dividing the two sections of video into a test set and a guide set respectively;
the subset construction module is used for acquiring a target quality parameter of the video in the pilot set and extracting a pilot video subset used for representing the video quality range of the pilot set from the pilot set according to the target quality parameter;
and the parameter obtaining module is used for determining the test quality parameters of the videos in the test set according to the video quality parameters obtained when the test set is played, and the pilot video subset is played before the test set is tested.
In an embodiment, the subset construction module is specifically configured to:
sending the leader set to at least one preliminary evaluation terminal, and receiving parameter groups fed back by each preliminary evaluation terminal, wherein the parameter groups comprise initial quality parameters of each section of video in the leader set;
converting the initial quality parameters of each group of parameter groups to convert the quality parameter range of each group of parameter groups into a specified quality parameter range;
and determining data representing the quality of the video corresponding to the parameter group from the converted parameter group to obtain the target quality parameter of the video in the leader set.
In one embodiment, the subset construction module converts the initial quality parameter of each set of parameters using the following formula, such that the quality parameter range of each set of parameters is converted into the specified quality parameter range:
wherein x is
maxRepresenting the maximum value, x, in the parameter set
minRepresenting the minimum value in the parameter set, x representing the initial quality parameter in the parameter set, x
newRepresenting data obtained by converting x, k
maxUpper limit value, k, representing a range of specified quality parameters
minRepresenting a lower value of the specified quality parameter range.
In one embodiment, the parameter obtaining module is specifically configured to:
sending the pilot video subset and the test set to at least one target evaluation terminal, wherein the target evaluation terminal meets the following requirements: playing a pilot video subset before playing a test set, and receiving input video quality parameters of videos in the test set when playing the videos in the test set;
receiving video quality parameters of the videos in the test set fed back by the target evaluation terminal;
and merging the video quality parameters corresponding to each section of video in the test set to obtain the test quality parameters of the video in the test set.
In one embodiment, the subset of the pilot videos at least includes videos corresponding to the highest, lowest and middle values of the target quality parameter, and the video corresponding to the middle value of the target quality parameter is marked with the target quality parameter.
In one embodiment, the videos in the subset of the pilot videos are arranged according to a designated playing sequence, the video corresponding to the intermediate value of the target quality parameter is arranged at the head, and other videos except the video corresponding to the intermediate value of the target quality parameter in the subset of the pilot videos are randomly arranged.
According to a fourth aspect of embodiments of the present invention, there is provided an electronic apparatus, including:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
splitting each section of video in a video group to be evaluated into two sections of videos, and respectively dividing the two sections of videos into a test set and a guide set;
acquiring a target quality parameter of the video in the pilot set, and extracting a pilot video subset for representing the video quality range of the pilot set from the pilot set according to the target quality parameter;
and determining the test quality parameters of the videos in the test set based on the video quality parameters obtained when the test set is played, wherein the pilot video subset is played before the test set is tested.
According to a fifth aspect of embodiments of the present invention, there is provided a computer readable storage medium, on which a computer program is stored, which when executed by a processor, performs the steps of any of the above methods.
The technical scheme provided by the embodiment of the invention can have the following beneficial effects:
according to the embodiment of the disclosure, a group of video groups similar to the video quality range of the test set is played before the test set video is played, so that the evaluating party obtains the video quality range of the test set before evaluating the test set video, and the accuracy of obtaining the video quality parameters can be improved by reminding the video quality range.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
Fig. 1A is a flowchart illustrating a video quality parameter acquisition method according to an exemplary embodiment of the present invention.
Fig. 1B is a schematic diagram illustrating a leader video subset arrangement order according to an exemplary embodiment of the present invention.
Fig. 2 is a diagram illustrating a video quality assessment scenario in accordance with an exemplary embodiment of the present invention.
Fig. 3A is a schematic structural diagram illustrating a video quality parameter acquisition system according to an exemplary embodiment of the present invention.
Fig. 3B is a timing diagram illustrating a video quality parameter acquisition method according to an exemplary embodiment of the invention.
Fig. 4 is a block diagram illustrating a video quality parameter acquisition apparatus according to an exemplary embodiment of the present invention.
Fig. 5 is a hardware configuration diagram of an electronic device in which the video quality parameter acquisition apparatus of the present invention is located.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
With the explosion of internet and multimedia technology, watching video has become one of entertainment ways in daily life. Particularly, with the rapid development of the live broadcast industry and the game industry, the game live broadcast is used as a product of the live broadcast and the game industry, and the scale of the game live broadcast is greatly improved. The video quality directly affects the watching experience of users, different users may pay different attention points to the same video picture, and the subjective feelings of the video quality are different. Therefore, the subjective evaluation of the video quality is of great significance. According to the video quality parameters obtained by subjective evaluation, the algorithm and the parameters can be adjusted in time, so that the video quality can better meet the requirements of the public.
At present, an evaluation terminal often directly displays a video to be evaluated, an evaluation party evaluates the video to be evaluated, and video quality parameters obtained through evaluation are input into the evaluation terminal. However, the video quality ranges of different video groups to be evaluated are different, which results in low accuracy of the video quality parameters obtained by the evaluation terminal.
In view of this, the present invention provides a method for obtaining video quality parameters, in which a group of video groups similar to the video quality range of a test set is played before a test set video is played, so that an evaluating party obtains the video quality range of the test set before evaluating the test set video, and the video quality range is used as a reference factor for evaluating video quality, thereby improving the accuracy of obtaining video quality parameters. Embodiments of the present invention are described below with reference to the accompanying drawings.
As shown in fig. 1A, fig. 1A is a flowchart illustrating a video quality parameter obtaining method according to an exemplary embodiment of the present invention, which may include the following steps 101 to 103:
in step 101, dividing each section of video in a video group to be evaluated into two sections of videos, and dividing the two sections of videos into a test set and a guide set respectively;
in step 102, obtaining a target quality parameter of the video in the pilot set, and extracting a pilot video subset for characterizing a video quality range of the pilot set from the pilot set according to the target quality parameter;
in step 103, the test quality parameters of the videos in the test set are determined based on the video quality parameters obtained when the test set is played, and the subset of the leader videos is played before the test set is tested.
The embodiment of the disclosure can be applied to an electronic device, and the electronic device can be a desktop computer, a laptop computer, a tablet computer, a PDA (Personal Digital Assistant), a multimedia player, and other devices with data processing capability.
The video group to be evaluated is a set formed by videos to be evaluated, and the video group to be evaluated at least comprises one video to be evaluated. According to the embodiment of the invention, each video segment in the video group to be evaluated can be divided into two video segments, one video segment is used for forming the guide set, and the other video segment is used for forming the test set. In one example, each video segment in the video group to be evaluated can be split into two video segments according to a preset ratio (e.g., N: M). Splitting (clipping) any video to be evaluated, and dividing the split video into the proportion of the videos in the pilot set
Less than or equal to the proportion of the video divided into the test set after splitting
Namely, it is
Therefore, the number of the videos in the test set is the same as that of the videos in the pilot set, and the videos are obtained by splitting the same videos to be evaluated.
It can be understood that other splitting manners may be adopted to split each video segment in the video group to be evaluated into two video segments, for example, the video segments in the video group to be evaluated are split according to the playing time, and are cut at the tth minute, and the two video segments obtained by cutting are respectively divided into the leader set and the test set. Other splitting modes are not exemplified here.
After the test set and the lead set are obtained, target quality parameters of the video in the lead set can be obtained. The target quality parameter may be a parameter for evaluating video quality. Video quality may be determined by factors such as video sharpness, fluency, etc., and in particular by parameters that affect the viewing experience of the user. The target quality parameters of the videos in the pilot set can be obtained through subjective evaluation by a preliminary evaluating party or through intelligent evaluation by equipment. The embodiment of the present invention is exemplified by one implementation manner, and the obtaining of the target quality parameter of the video in the pilot set may include:
sending the leader set to at least one preliminary evaluation terminal, and receiving parameter groups fed back by each preliminary evaluation terminal, wherein the parameter groups comprise initial quality parameters of each section of video in the leader set;
converting the initial quality parameters of each group of parameter groups to convert the quality parameter range of each group of parameter groups into a specified quality parameter range;
and determining data representing the quality of the video corresponding to the parameter group from the converted parameter group to obtain the target quality parameter of the video.
After receiving the pilot set, the preliminary evaluation end can play the video in the pilot set. For example, after the initial quality parameters of the currently played video are obtained, the next video segment can be played, so that the initial quality parameters of each video segment in the leader set are obtained at the pre-evaluation end. For the same preliminary evaluation end, the initial quality parameters of different videos in the leader set may constitute a parameter set. Thus, the different preliminary evaluation terminals have corresponding parameter sets. The mapping relation between each video and the initial quality parameter can be recorded in the parameter group. For example, each video segment in the leader set is played in a fixed sequence, and as one of the mapping relationship representing modes, the initial quality parameters of the videos may be arranged according to the video arrangement sequence in the leader set, so as to implement the mapping relationship between the videos and the initial quality parameters by using the arrangement sequence. As another example, a mapping of video identification to initial video quality may be included in the parameter set. The video identifier may be a video name, or any combination of numbers, character strings, symbols, and the like, as long as the video identifier can uniquely identify the video.
The execution end of the method can receive the parameter group fed back by each pre-evaluation end. In view of the fact that the quality evaluation results of different pre-evaluation terminals for the same video may be different and the quality parameter ranges of the evaluation results may not be uniform, the initial quality parameters of each group of parameter groups may be converted, so that the quality parameter ranges of each group of parameter groups are converted into the specified quality parameter ranges.
Therefore, the initial quality parameters of each group of parameter sets are converted, the purpose is to convert the quality parameter range of each group of parameter sets into the specified quality parameter range, and the parameter sets in different quality parameter ranges are mapped into the same quality parameter range, so that the standard is judged uniformly.
In one embodiment, the following formula may be used to convert the initial quality parameter of each set of parameters, so that the quality parameter range of each set of parameters is converted into the specified quality parameter range:
wherein x is
maxRepresents the maximum value in the parameter set, i.e. the largest initial quality parameter in the parameter set. x is the number of
minRepresents the minimum value in the parameter set, i.e. the smallest initial quality parameter in the parameter set. x denotes an initial quality parameter in the parameter set, x
newRepresenting data obtained by converting x, k
maxUpper limit value, k, representing a range of specified quality parameters
minRepresenting a lower value of the specified quality parameter range.
Taking the specified quality parameter range as 1-5 as an example, the following formula can be adopted to transform the initial quality parameters in the parameter set, so that the values of different domains are mapped into 1-5 domains:
therefore, the embodiment realizes parameter conversion by using a formula mode, can quickly convert different quality parameter ranges into the specified quality parameter range, and improves the conversion efficiency.
After the quality parameter range of each group of parameter sets is converted into the specified quality parameter range, data representing the quality of the video corresponding to the parameter set can be determined from the converted parameter sets, the target quality parameter of the video is obtained, and the representative data representing the quality of the video is achieved. For example, an averaging or median method may be used to determine a target quality parameter that characterizes the quality of the video corresponding to the set of parameters. Specifically, the converted parameter set is averaged to obtain the target quality parameter of the video segment. For another example, the intermediate value is extracted from the converted parameter set as the target quality parameter of the video segment.
Therefore, the target quality parameters of each video segment in the pilot set can be obtained by integrating the parameter sets fed back by different pre-evaluation terminals, and the accuracy of the target quality parameters is improved.
After obtaining the target quality parameters of the videos in the pilot set, a subset of the pilot videos from the pilot set can be extracted for characterizing the video quality range of the pilot set according to the target quality parameters, for example, one or more videos that can represent the video quality range are extracted from the leader set to form a leader video subset, therefore, the video quality range of the pilot set is embodied by using the pilot video subset, and because the pilot set and the test set are obtained by splitting the same video to be evaluated, the video quality range of the test set can be embodied by using the pilot video subset, and the pilot video subset is played before the test set is played, thereby prompting the video quality range of the test set of the evaluating party before the evaluating party evaluates the video quality of the test set, because the display information is more comprehensive, the video quality range is beneficial to improving the evaluation accuracy of an evaluating party, and further the accuracy of the execution end for obtaining the video quality parameters is improved.
In one embodiment, the subset of the pilot videos includes at least videos corresponding to the highest, lowest and intermediate values of the target quality parameter.
The video corresponding to the highest target quality parameter, the video corresponding to the lowest target quality parameter and the video corresponding to the intermediate value target quality parameter can be extracted from the pilot set, and the extracted videos are used for constructing the pilot video subset.
It can be seen that the present embodiment utilizes the limit value and the intermediate value to characterize the video quality range of the guide set, which is easy to implement.
Further, other videos in the leader set may also be included in the leader video subset. For example, n-3 videos may also be randomly drawn from the remaining videos in the guide set. Wherein n is the number of videos pre-designated in the subset of the pilot videos. Taking the target quality parameters as the highest 5, the lowest 1 and n as 6 as examples, extracting the video corresponding to 1 point, the video corresponding to 3 points and the video corresponding to 5 points from the pilot set, randomly extracting 3 videos from the rest videos, and forming the extracted 6 videos into the pilot video subset.
In one example, the video corresponding to the intermediate value of the target quality parameter is marked with the target quality parameter.
The embodiment performs parameter marking on the video corresponding to the intermediate value of the target quality parameter, so that the target quality parameter can be displayed simultaneously when the video corresponding to the intermediate value of the target quality parameter is displayed, and the video corresponding to the intermediate value of the target quality parameter can be reminded. Although the leader video subset definitely has videos corresponding to the maximum value (the highest target quality parameter) and the minimum value (the lowest target quality parameter), the videos corresponding to the maximum value and the minimum value may not be subjected to parameter marking, so that an evaluating party can evaluate the unmarked videos.
In constructing the subset of leader videos, in one example, the extracted videos may be randomly ordered. In another example, the videos in the subset of the pilot videos are arranged according to a designated playing sequence, the video corresponding to the intermediate value of the target quality parameter is arranged at the head, and the videos other than the video corresponding to the intermediate value of the target quality parameter in the subset of the pilot videos are randomly arranged.
Still taking the target quality parameters as highest 5, lowest 1, and n as 6 as an example, as shown in fig. 1B, fig. 1B is a schematic diagram illustrating an arrangement sequence of the leader video subsets according to an exemplary embodiment of the present invention. In the schematic diagram, the leader video subset is composed of videos corresponding to the intermediate value (e.g. 3 points), the maximum value (e.g. 5 points), the minimum value (e.g. 1 point) and three unknown quality parameters. And arranging the video corresponding to the intermediate value at the head, randomly arranging other videos, marking the target quality parameters of the video arranged at the head, and not marking the quality parameters of the videos arranged at other ranks.
Therefore, when the pilot video subset is played, the video corresponding to the intermediate value is played first, and the target quality parameter of the video can be reminded during playing, so that the video with the intermediate quality is provided for reference of an evaluating party.
It is understood that the pilot video subset may also be extracted from the pilot set by other manners, as long as the video subset can be used to characterize the video quality range of the pilot set, and therefore, the description is not repeated here.
After the pilot video subset is obtained, the test quality parameters of the videos in the test set may be determined based on the video quality parameters obtained while the test set is played. The quality parameter of the video in the test set is named as the test quality parameter in order to distinguish the test quality parameter from the target quality parameter of the video in the pilot set.
As one of the determination methods of the test quality parameters, the method may include:
sending the pilot video subset and the test set to at least one target evaluation terminal, wherein the target evaluation terminal meets the following requirements: playing a pilot video subset before playing a test set, and receiving input video quality parameters of videos in the test set when playing the videos in the test set;
receiving video quality parameters of the videos in the test set fed back by the target evaluation terminal;
and merging the video quality parameters corresponding to each section of video in the test set to obtain the test quality parameters of the video in the test set.
The target evaluation terminal is a terminal used for obtaining video quality parameters of videos in the test set. And the target evaluation terminal plays the pilot video subset before playing the test set to remind a user of the video quality range of the test set, and receives the input video quality parameters of the video in the test set when playing the video in the test set.
Because a plurality of target evaluation terminals can exist, different video quality parameters can be fed back by different target evaluation terminals aiming at the same video in the test set, so that one video can correspond to a plurality of video quality parameters, and the video quality parameters corresponding to each section of video in the test set can be combined to obtain the test quality parameters of the video in the test set. The merging process may be a process of obtaining a mean value and a variance, and the like, so as to obtain a parameter capable of evaluating the video quality of the test set.
Therefore, in the embodiment, the target evaluation terminal obtains the video quality parameters of the concentrated test video, and the target evaluation terminals can be multiple, so that the concentrated test video can be evaluated by using different target evaluation terminals.
Furthermore, the target evaluation terminal can also play the pilot video subset when playing the test set video, especially play the video corresponding to the intermediate value of the target quality parameter in the pilot video subset, so as to realize that the pilot video subset is used for reminding the video quality range when playing the test set video, especially reminding the intermediate value of the video quality range of an evaluation party.
In addition, for ease of understanding, one of the video quality assessment scenarios is exemplified. As shown in fig. 2, fig. 2 is a diagram illustrating a video quality assessment scenario according to an exemplary embodiment of the present invention. In this scenario, the processing end and the evaluation end are respectively disposed on different devices, and it can be understood that, as an implementation manner, the processing end and the evaluation end may also be disposed on the same device, and this embodiment is described by taking the location on different devices as an example. The evaluation end can also be divided into a preparation evaluation end and a target evaluation end. The preliminary evaluation terminal and the target evaluation terminal can be two different applications or two different modes in the same application. The device at the processing end may be a device with data processing capability, for example, a desktop computer, a handheld computer, a tablet computer, and the like. The device where the evaluation terminal is located may be a smart phone, a tablet computer, a PDA (Personal Digital Assistant), a multimedia player, and the like. The processing end may be used to generate the test set and the leader video subset. The preliminary evaluation end is used for playing the pilot set and obtaining a parameter group comprising the initial quality parameters of each video segment in the pilot set. The target evaluation terminal can be used for playing the test set and the pilot video subset and obtaining the test quality parameters of the videos in the test set when the videos in the test set are played.
As shown in fig. 3A, fig. 3A is a schematic structural diagram of a video quality parameter obtaining system according to an exemplary embodiment of the present invention. As shown in fig. 3B, fig. 3B is a timing diagram illustrating a video quality parameter obtaining method according to an exemplary embodiment of the present invention. The system comprises a processing end, at least one preliminary evaluation end and at least one target evaluation end.
The following steps may be performed in the processing side:
301. splitting each section of video in a video group to be evaluated into two sections of videos, and respectively dividing the two sections of videos into a test set and a guide set;
302. sending the leader set to at least one preliminary evaluation terminal;
the following steps may be performed in the preliminary evaluation terminal:
303. and playing the pilot set, and obtaining a parameter group based on an input instruction, wherein the parameter group comprises the initial quality parameters of each video segment in the pilot set.
The following steps can also be executed in the processing end:
304. determining target quality parameters of the videos in the pilot set according to parameter groups fed back by each preliminary evaluation end;
305. extracting a leader video subset from the leader set for characterizing a video quality range of the leader set according to the target quality parameter;
306. and sending the pilot video subset and the test set to at least one target evaluation terminal.
The following steps may be performed in the target evaluation terminal:
307. and playing the pilot video subset before playing the test set, and receiving input video quality parameters of the videos in the test set when playing the videos in the test set.
The following steps can also be executed in the processing end:
308. receiving video quality parameters of the videos in the test set fed back by the target evaluation terminal;
309. and merging the video quality parameters corresponding to each section of video in the test set to obtain the test quality parameters of the video in the test set.
Fig. 3B is the same as the related art in fig. 1A, and is not repeated herein.
According to the embodiment, the video group similar to the video quality range of the test set is played before the test set video is played, so that the evaluating party obtains the video quality range of the test set before evaluating the test set video, and the accuracy of obtaining the video quality parameters can be improved by reminding the video quality range.
In one example, the subset of the pilot videos at least includes videos corresponding to the highest, lowest and intermediate values of the target quality parameter; and when the target evaluation terminal plays the videos of the test set, the videos corresponding to the intermediate values of the target quality parameters are played at the same time.
Therefore, the method and the device can remind the middle value of the video quality range of the evaluating party when the test set video is played.
Further, if two display devices are used to play the test set video and the video corresponding to the intermediate value of the target quality parameter, the configuration parameters of the display device playing the test set video and the display device playing the video corresponding to the intermediate value of the target quality parameter are the same.
The configuration parameters are the same, and the two display devices belong to the same model, the same brightness, the same contrast, the same gamma value and the like, so that the two display devices have the same display effect when displaying the same video, and the external factors of different display effects caused by different display devices are eliminated.
Therefore, in the whole evaluation process, one display device with the same model and parameters as those of the test machine (the display device for playing the test set video) can be used for playing the video with the fixed score for reference of an evaluating party, so that the result deviation caused by standard inconsistency due to limited memory of the evaluating party is prevented.
When the target evaluation terminal plays the test set video, the playing time of each video can be limited, a shortest playing time is set, namely at least the playing time of each video, and when the playing time arrives, the quality parameter input control is displayed for the evaluation party to input the video quality parameters of the video. For example, the playing time length can be set to 20s, and can also be flexibly set to other values. After each video scoring is finished, preset questions and answer input controls can be displayed for an evaluating party to input answers corresponding to the questions. For example, the question may be a scoring reason, or the like.
At the target evaluation end, in order to prevent the result deviation caused by too little comparison, a review mechanism can be provided for the evaluating party to review the result from beginning to end, and the number of times of modifying the result by the evaluating party can be set, for example, 1 time. And (3) reviewing from the first video, displaying the evaluation result of the video, and if an evaluating party disagrees with the evaluation result, directly inputting new quality parameters and inputting a modification reason.
Therefore, after the positive evaluation of the evaluating party is finished, the evaluating party can review the test result, modify the problematic result and ensure the reliability of the result.
The various technical features in the above embodiments can be arbitrarily combined, so long as there is no conflict or contradiction between the combinations of the features, but the combination is limited by the space and is not described one by one, and therefore, any combination of the various technical features in the above embodiments also belongs to the scope disclosed in the present specification.
Corresponding to the embodiment of the video quality parameter acquisition method, the invention also provides embodiments of a video quality parameter acquisition device, a video quality parameter acquisition system, electronic equipment applied to the video quality parameter acquisition device and the system, and a computer storage medium.
As shown in fig. 4, fig. 4 is a block diagram of a video quality parameter obtaining apparatus according to an exemplary embodiment of the present invention, the apparatus including:
the video splitting module 410 is configured to split each video segment in the video group to be evaluated into two video segments, and divide the two video segments into a test set and a guide set respectively;
a subset construction module 420, configured to obtain a target quality parameter of the video in the pilot set, and extract, according to the target quality parameter, a pilot video subset used for characterizing a video quality range of the pilot set from the pilot set;
a parameter obtaining module 430, configured to determine, according to the video quality parameter obtained when the test set is played, a test quality parameter of a video in the test set, where the pilot video subset is played before the test set is tested.
In an embodiment, the subset construction module 420 is specifically configured to:
sending the leader set to at least one preliminary evaluation terminal, and receiving parameter groups fed back by each preliminary evaluation terminal, wherein the parameter groups comprise initial quality parameters of each section of video in the leader set;
converting the initial quality parameters of each group of parameter groups to convert the quality parameter range of each group of parameter groups into a specified quality parameter range;
and determining data representing the quality of the video corresponding to the parameter group from the converted parameter group to obtain the target quality parameter of the video in the leader set.
In one embodiment, the subset construction module 420 converts the initial quality parameter of each set of parameters using the following formula, such that the quality parameter range of each set of parameters is converted to the specified quality parameter range:
wherein x is
maxRepresenting the maximum value, x, in the parameter set
minRepresenting the minimum value in the parameter set, x representing the initial quality parameter in the parameter set, x
newRepresenting data obtained by converting x, k
maxUpper limit value, k, representing a range of specified quality parameters
minRepresenting a lower value of the specified quality parameter range.
In an embodiment, the parameter obtaining module 430 is specifically configured to:
sending the pilot video subset and the test set to at least one target evaluation terminal, wherein the target evaluation terminal meets the following requirements: playing a pilot video subset before playing a test set, and receiving input video quality parameters of videos in the test set when playing the videos in the test set;
receiving video quality parameters of the videos in the test set fed back by the target evaluation terminal;
and merging the video quality parameters corresponding to each section of video in the test set to obtain the test quality parameters of the video in the test set.
In one embodiment, the subset of the pilot videos at least includes videos corresponding to the highest, lowest and middle values of the target quality parameter, and the video corresponding to the middle value of the target quality parameter is marked with the target quality parameter.
In one embodiment, the videos in the subset of the pilot videos are arranged according to a designated playing sequence, the video corresponding to the intermediate value of the target quality parameter is arranged at the head, and other videos except the video corresponding to the intermediate value of the target quality parameter in the subset of the pilot videos are randomly arranged.
Correspondingly, the embodiment of the invention also provides electronic equipment, which comprises a processor; a memory for storing processor-executable instructions; wherein the processor is configured to:
splitting each section of video in a video group to be evaluated into two sections of videos, and respectively dividing the two sections of videos into a test set and a guide set;
acquiring a target quality parameter of the video in the pilot set, and extracting a pilot video subset for representing the video quality range of the pilot set from the pilot set according to the target quality parameter;
and determining the test quality parameters of the videos in the test set based on the video quality parameters obtained when the test set is played, wherein the pilot video subset is played before the test set is tested.
Correspondingly, an embodiment of the present invention further provides a computer storage medium, where the storage medium stores program instructions, and the program instructions include:
splitting each section of video in a video group to be evaluated into two sections of videos, and respectively dividing the two sections of videos into a test set and a guide set;
acquiring a target quality parameter of the video in the pilot set, and extracting a pilot video subset for representing the video quality range of the pilot set from the pilot set according to the target quality parameter;
and determining the test quality parameters of the videos in the test set based on the video quality parameters obtained when the test set is played, wherein the pilot video subset is played before the test set is tested.
The present invention may take the form of a computer program product embodied on one or more storage media including, but not limited to, disk storage, CD-ROM, optical storage, and the like, having program code embodied therein. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of the storage medium of the computer include, but are not limited to: phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technologies, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic tape storage or other magnetic storage devices, or any other non-transmission medium, may be used to store information that may be accessed by a computing device.
The specific details of the implementation process of the functions and actions of each module in the device are referred to the implementation process of the corresponding step in the method, and are not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, wherein the modules described as separate parts may or may not be physically separate, and the parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the invention. One of ordinary skill in the art can understand and implement it without inventive effort.
The embodiment of the video quality parameter acquisition device can be applied to electronic equipment. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. Taking a software implementation as an example, as a logical device, the device is formed by reading, by a processor of the electronic device where the device is located, a corresponding computer program instruction in the nonvolatile memory into the memory for operation. From a hardware aspect, as shown in fig. 5, it is a hardware structure diagram of an electronic device where the video quality parameter obtaining apparatus 532 is located in the present invention, except for the processor 510, the memory 530, the network interface 520, and the nonvolatile memory 540 shown in fig. 5, the electronic device where the apparatus 532 is located in the embodiment may also include other hardware according to an actual function of the electronic device, which is not described again.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (9)
1. A method for obtaining video quality parameters, the method comprising:
splitting each section of video in a video group to be evaluated into two sections of videos, and respectively dividing the two sections of videos into a test set and a guide set;
acquiring target quality parameters of videos in the pilot set, and extracting a pilot video subset used for representing a video quality range of the pilot set from the pilot set according to the target quality parameters, wherein the pilot video subset at least comprises videos with the highest target quality parameters, the lowest target quality parameters and corresponding intermediate values, and the videos corresponding to the intermediate values of the target quality parameters are marked with the target quality parameters;
determining the test quality parameters of the videos in the test set based on the video quality parameters obtained when the test set is played, wherein the pilot video subset is played before the test set is tested;
wherein the step of obtaining the target quality parameter of the video in the leader set comprises:
sending the leader set to at least one preliminary evaluation terminal, and receiving parameter groups fed back by each preliminary evaluation terminal, wherein the parameter groups comprise initial quality parameters of each section of video in the leader set;
converting the initial quality parameters of each group of parameter groups to convert the quality parameter range of each group of parameter groups into a specified quality parameter range;
and determining data representing the quality of the video corresponding to the parameter group from the converted parameter group to obtain the target quality parameter of the video in the leader set.
2. The method of claim 1, wherein the initial quality parameter for each set of parameter sets is transformed using the following formula such that the quality parameter range for each set of parameter sets is transformed into the specified quality parameter range:
wherein x is
maxRepresenting the maximum value, x, in the parameter set
minRepresenting the minimum value in the parameter set, x representing the initial quality parameter in the parameter set, x
newRepresenting data obtained by converting x, k
maxUpper limit value, k, representing a range of specified quality parameters
minRepresenting a lower value of the specified quality parameter range.
3. The method of claim 1, wherein determining the test quality parameters of the videos in the test set based on the video quality parameters obtained when the test set is played comprises:
sending the pilot video subset and the test set to at least one target evaluation terminal, wherein the target evaluation terminal meets the following requirements: playing a pilot video subset before playing a test set, and receiving input video quality parameters of videos in the test set when playing the videos in the test set;
receiving video quality parameters of the videos in the test set fed back by the target evaluation terminal;
and merging the video quality parameters corresponding to each section of video in the test set to obtain the test quality parameters of the video in the test set.
4. The method according to claim 3, wherein the videos in the subset of pilot videos are arranged in a designated playing order, the video corresponding to the intermediate value of the target quality parameter is arranged at the head, and other videos in the subset of pilot videos except the video corresponding to the intermediate value of the target quality parameter are randomly arranged.
5. A video quality parameter acquisition system is characterized by comprising a processing end, a preparation evaluation end and a target evaluation end;
the processing end divides each section of video in the video group to be evaluated into two sections of video, and divides the two sections of video into a test set and a pilot set respectively, and sends the pilot set to at least one preliminary evaluation end;
the preliminary evaluation end plays the pilot set and obtains a parameter set based on an input instruction, wherein the parameter set comprises initial quality parameters of each video segment in the pilot set;
the processing terminal converts the initial quality parameters of each group of parameter groups to convert the quality parameter range of each group of parameter groups into an appointed quality parameter range, determines data representing the quality of the video corresponding to the parameter group from the converted parameter groups to obtain target quality parameters of the video in the pilot set, extracts a pilot video subset used for representing the video quality range of the pilot set from the pilot set according to the target quality parameters, and sends the pilot video subset and the test set to at least one target evaluation terminal, wherein the pilot video subset at least comprises the videos corresponding to the highest, the lowest and the middle values of the target quality parameters, and the video corresponding to the middle value of the target quality parameters is marked with the target quality parameters;
the target evaluation terminal plays the pilot video subset before playing the test set and receives the input video quality parameters of the videos in the test set when playing the videos in the test set;
the processing end receives the video quality parameters of the test concentrated video fed back by the target evaluation end; and merging the video quality parameters corresponding to each section of video in the test set to obtain the test quality parameters of the video in the test set.
6. The video quality parameter acquisition system according to claim 5,
and when the target evaluation terminal plays the video of the test set, the video corresponding to the intermediate value of the target quality parameter is played at the same time, and the configuration parameters of the display equipment playing the video of the test set are the same as the configuration parameters of the display equipment playing the video corresponding to the intermediate value of the target quality parameter.
7. A video quality parameter acquisition apparatus, characterized in that the apparatus comprises:
the video splitting module is used for splitting each section of video in the video group to be evaluated into two sections of video and dividing the two sections of video into a test set and a guide set respectively;
the subset construction module is used for acquiring target quality parameters of videos in the pilot set, and extracting a pilot video subset used for representing a video quality range of the pilot set from the pilot set according to the target quality parameters, wherein the pilot video subset at least comprises videos corresponding to the highest, the lowest and the middle values of the target quality parameters, and the videos corresponding to the middle values of the target quality parameters are marked with the target quality parameters;
the parameter obtaining module is used for determining the test quality parameters of the videos in the test set according to the video quality parameters obtained when the test set is played, and the pilot video subset is played before the test set is tested;
wherein the step of obtaining the target quality parameter of the video in the leader set comprises:
sending the leader set to at least one preliminary evaluation terminal, and receiving parameter groups fed back by each preliminary evaluation terminal, wherein the parameter groups comprise initial quality parameters of each section of video in the leader set;
converting the initial quality parameters of each group of parameter groups to convert the quality parameter range of each group of parameter groups into a specified quality parameter range;
and determining data representing the quality of the video corresponding to the parameter group from the converted parameter group to obtain the target quality parameter of the video in the leader set.
8. An electronic device, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
splitting each section of video in a video group to be evaluated into two sections of videos, and respectively dividing the two sections of videos into a test set and a guide set;
acquiring target quality parameters of videos in the pilot set, and extracting a pilot video subset used for representing a video quality range of the pilot set from the pilot set according to the target quality parameters, wherein the pilot video subset at least comprises videos with the highest target quality parameters, the lowest target quality parameters and corresponding intermediate values, and the videos corresponding to the intermediate values of the target quality parameters are marked with the target quality parameters;
determining the test quality parameters of the videos in the test set based on the video quality parameters obtained when the test set is played, wherein the pilot video subset is played before the test set is tested;
wherein the step of obtaining the target quality parameter of the video in the leader set comprises:
sending the leader set to at least one preliminary evaluation terminal, and receiving parameter groups fed back by each preliminary evaluation terminal, wherein the parameter groups comprise initial quality parameters of each section of video in the leader set;
converting the initial quality parameters of each group of parameter groups to convert the quality parameter range of each group of parameter groups into a specified quality parameter range;
and determining data representing the quality of the video corresponding to the parameter group from the converted parameter group to obtain the target quality parameter of the video in the leader set.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810399205.XA CN108712645B (en) | 2018-04-28 | 2018-04-28 | Video quality parameter acquisition method, device, equipment, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810399205.XA CN108712645B (en) | 2018-04-28 | 2018-04-28 | Video quality parameter acquisition method, device, equipment, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108712645A CN108712645A (en) | 2018-10-26 |
| CN108712645B true CN108712645B (en) | 2020-02-11 |
Family
ID=63868671
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810399205.XA Active CN108712645B (en) | 2018-04-28 | 2018-04-28 | Video quality parameter acquisition method, device, equipment, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108712645B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101540896A (en) * | 2008-03-17 | 2009-09-23 | 中国移动通信集团公司 | Method, device and system for testing video service quality |
| CN100584047C (en) * | 2008-06-25 | 2010-01-20 | 厦门大学 | Video quality automatic evaluation system oriented to wireless network and evaluation method thereof |
| CN101998137B (en) * | 2009-08-21 | 2016-09-07 | 华为技术有限公司 | Video quality parameter acquisition methods and device and electronic equipment |
| WO2015106383A1 (en) * | 2014-01-14 | 2015-07-23 | 华为技术有限公司 | Method for assessing video quality and related device |
| US9922411B2 (en) * | 2015-11-30 | 2018-03-20 | Disney Enterprises, Inc. | Saliency-weighted video quality assessment |
-
2018
- 2018-04-28 CN CN201810399205.XA patent/CN108712645B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN108712645A (en) | 2018-10-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230012732A1 (en) | Video data processing method and apparatus, device, and medium | |
| US10579654B2 (en) | Method and device for generating online question paths from existing question banks using a knowledge graph | |
| US10368123B2 (en) | Information pushing method, terminal and server | |
| CN108900923B (en) | Method and device for recommending live broadcast template | |
| CN107832437B (en) | Audio/video pushing method, device, equipment and storage medium | |
| CN112533051B (en) | Barrage information display method, barrage information display device, computer equipment and storage medium | |
| CN110049318B (en) | Learning quality evaluation method and related device | |
| CN111191067A (en) | Picture book identification method, terminal device and computer readable storage medium | |
| US9146989B2 (en) | Analytic comparison of libraries and playlists | |
| CN108519998B (en) | Problem guiding method and device based on knowledge graph | |
| CN104980790A (en) | Voice subtitle generating method and apparatus, and playing method and apparatus | |
| CN111479129A (en) | Live broadcast cover determining method, device, server, medium and system | |
| CN107968953A (en) | Anti- cheating user method and device | |
| CN112507163A (en) | Duration prediction model training method, recommendation method, device, equipment and medium | |
| CN111107442A (en) | Method and device for acquiring audio and video files, server and storage medium | |
| CN112104909A (en) | Interactive video playing method and device, computer equipment and readable storage medium | |
| CN109410972B (en) | Method, device and storage medium for generating sound effect parameters | |
| CN108921138B (en) | Method and apparatus for generating information | |
| CN108712645B (en) | Video quality parameter acquisition method, device, equipment, system and storage medium | |
| CN111049735B (en) | Group head portrait display method, device, equipment and storage medium | |
| CN113747233B (en) | Music replacement method and device, electronic equipment and storage medium | |
| CN113301362B (en) | Video element display method and device | |
| CN114745594A (en) | Method and device for generating live playback video, electronic equipment and storage medium | |
| CN112652281A (en) | Music chord identification method and device, electronic equipment and storage medium | |
| CN112287173A (en) | Method and apparatus for generating information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210115 Address after: 511442 3108, 79 Wanbo 2nd Road, Nancun Town, Panyu District, Guangzhou City, Guangdong Province Patentee after: GUANGZHOU CUBESILI INFORMATION TECHNOLOGY Co.,Ltd. Address before: 511442 24 floors, B-1 Building, Wanda Commercial Square North District, Wanbo Business District, 79 Wanbo Second Road, Nancun Town, Panyu District, Guangzhou City, Guangdong Province Patentee before: GUANGZHOU HUADUO NETWORK TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20181026 Assignee: GUANGZHOU HUADUO NETWORK TECHNOLOGY Co.,Ltd. Assignor: GUANGZHOU CUBESILI INFORMATION TECHNOLOGY Co.,Ltd. Contract record no.: X2021440000053 Denomination of invention: Video quality parameter acquisition method, device, equipment, system and storage medium Granted publication date: 20200211 License type: Common License Record date: 20210208 |
|
| EE01 | Entry into force of recordation of patent licensing contract |