CN112843677B - System delay determining method, device, equipment and storage medium - Google Patents

Abstract

The embodiment of the disclosure provides a system delay determining method, a device, equipment and a storage medium, comprising the following steps: responding to a user operation instruction acting on a multimedia signal of a background window, and acquiring the input moment of the user operation instruction; detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment; determining the response time of the user operation instruction based on the mutation condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window; and determining the system delay of the system according to the input time and the response time. According to the embodiment of the disclosure, the system delay is determined by acquiring the input time of the user operation instruction and the response time corresponding to the user operation instruction, and the scheme does not need expensive hardware equipment or excessive manual participation, so that the interaction delay of the remote rendering system can be conveniently and rapidly measured.

Description

A system delay determination method, device, equipment and storage medium

technical field

The embodiments of the present disclosure relate to the technical field of data processing, and in particular, to a method, device, device and storage medium for determining system delay.

Background technique

In the remote rendering system, the user inputs an operation instruction through the client and sends the operation instruction to the server. The server obtains the media signal corresponding to the operation instruction and returns the media signal to the client. The client responds to the above media signal and displays it. In the remote rendering system, since the client needs to request the media signal from the server, there is a certain delay between the operation command and the display of the corresponding media signal. Common remote rendering system delays include cloud game delays, screen-casting application delays, or remote desktop delays, etc.

In the prior art, a manual measurement method based on a high-speed camera is often used to determine the delay of the remote rendering system. Fig. 1 is a schematic diagram of a delay determination method based on a high-speed camera; as shown in Fig. 1 , a high-speed camera is used to record the user's input to the remote rendering system and the response of the remote system. Then manually find the video frame input by the user and the video frame where the system responds. The time difference between the two is the system delay.

The above method can measure the delay of the remote system more accurately, but requires an expensive high-speed camera, and at the same time, manually searching for user input frames and system response frames is also a time-consuming and laborious work.

Contents of the invention

Embodiments of the present disclosure provide a method, device, device and storage medium for determining system delay, so as to conveniently and quickly measure the interaction delay of a remote rendering system.

In a first aspect, an embodiment of the present disclosure provides a system delay determination method, including:

Responding to a user operation instruction acting on the multimedia signal of the background window, acquiring the input moment of the user operation instruction;

detecting a multimedia signal in a background window and a foreground window, wherein the background window is a first time window before the input moment and the foreground window is a second time window after the input moment;

Determine the response time of the user operation instruction based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window;

A system delay of the system is determined according to the input time and the response time.

In the second aspect, the embodiment of the present disclosure also provides a device for determining system delay, which is characterized in that it includes:

The input time acquisition module is used to acquire the input time of the user operation command in response to the user operation command acting on the multimedia signal of the background window;

A multimedia signal detection module, configured to detect multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time after the input moment window;

A response time determination module, configured to determine the response time of the user operation instruction based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window;

A system delay determination module, configured to determine the system delay of the system according to the input time and the response time.

In a third aspect, an embodiment of the present disclosure also provides a device for determining system delay, including:

one or more processors;

memory for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the method for determining system delay according to any one of the embodiments of the present disclosure.

In a fourth aspect, the embodiments of the present disclosure further provide a computer storage medium, the medium stores a computer program, and when the computer program is executed by a processor, the system delay as described in any one of the embodiments of the present disclosure is implemented. Determine the method.

The present disclosure provides a system delay determination method, device, device and storage medium. The system delay determination method includes: responding to a user operation instruction acting on a multimedia signal in a background window, obtaining the input time of the user operation instruction; detecting The multimedia signal in the background window and the foreground window, wherein, the background window is the first time window before the input moment, and the foreground window is the second time window after the input moment; based on the multimedia signal in the foreground window relative to the multimedia in the background window The sudden change of the signal determines the response time of the user's operation command; the system delay of the system is determined according to the input time and the response time. The embodiment of the present disclosure determines the system delay by obtaining the input time of the user operation command and the corresponding response time of the user operation command. This solution does not require expensive hardware equipment or excessive manual participation, and can measure remote rendering conveniently and quickly. System interaction delay.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

Fig. 1 is a schematic diagram of a delay determination method based on a high-speed camera;

FIG. 2 is a schematic diagram of a topology structure of a remote rendering system;

FIG. 3 is a flow chart of a method for determining system delay provided by an embodiment of the present disclosure;

FIG. 4 is a flow chart of another method for determining system delay provided by an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a system delay determination method provided by an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of MoSIFT and MFCC provided by an embodiment of the present disclosure;

Fig. 7 is a schematic diagram of delay results on two cloud games provided by an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of an apparatus for determining system delay provided by an embodiment of the present disclosure;

Fig. 9 is a structural diagram of a device for determining system delay provided by an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method implementations of the present disclosure may be executed in different orders, and/or executed in parallel. Additionally, method embodiments may include additional steps and/or omit performing illustrated steps. The scope of the present disclosure is not limited in this respect.

As used herein, the term "comprise" and its variations are open-ended, ie "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one further embodiment"; the term "some embodiments" means "at least some embodiments." Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the sequence of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more" multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are used for illustrative purposes only, and are not used to limit the scope of these messages or information.

First, take cloud game delay as an example for illustration. Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing technology. Cloud gaming technology enables thin clients with relatively limited graphics processing and data computing capabilities to run high-quality games. In the cloud game scenario, the game is not running on the player's game client, but in the cloud server, and the cloud server renders the game scene into a video and audio stream, which is transmitted to the player's game client through the network. The player's game client does not need to have powerful graphics computing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to obtain player input instructions and send them to the cloud server. In order to ensure the smoothness of the cloud game process, a cloud game delay test is required.

Secondly, a brief introduction to the delay of the remote rendering system. As shown in Figure 2, a remote rendering system usually includes a local client and a remote server, and its end-to-end delay includes input hardware delay t1, main process delay and output hardware delay t2. Wherein the master process delay includes local client processing delay (not shown in Fig. 2), network transmission delay (local area network upload delay t2 in Figure 2, WAN upload delay t3, WAN download delay t5, The local area network download delay t6) and the processing delay t7 of the remote server are the most important parts of the delay from the client to the server. The input hardware delay t1 and the output hardware delay t2 are usually a constant value that does not change, and the delay of the main process is often much greater than the input hardware delay t1 and the output hardware delay t2. Therefore, this disclosure mainly considers determining the delay of the main process as shown in FIG. 2 .

It should be noted that the system delay in the following embodiments refers to the delay of the main process in FIG. 2 , excluding the input hardware delay t1 and the output hardware delay t2 .

In this embodiment, the server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, and network services Cloud servers for basic cloud computing services such as cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Clients can be smartphones, tablets, laptops, desktop computers, smart speakers, smart watches, etc., but are not limited thereto. The client and the server may be connected directly or indirectly through wired or wireless communication, which is not limited in this embodiment.

Fig. 3 is a flow chart of a method for determining system delay provided by an embodiment of the present disclosure. This embodiment is applicable to determining the response delay of a remote rendering interactive system, and the method can be executed by a system delay determining device. The device for determining the system delay can be implemented by means of software and/or hardware. The method for determining the system delay is applied to the client.

It should be noted that the system delay determination method provided in this embodiment can be used on the client, and can be considered to be executed by a system delay determination device integrated on the client, wherein the client can include A computer device with a processor, memory, input devices, and output devices. Such as notebook computers, desktop computers, tablet computers and smart terminals.

As shown in FIG. 3 , the system delay determination method provided in this embodiment mainly includes steps S11 , S12 , S13 and S14 .

S11. In response to a user operation instruction acting on a multimedia signal in a background window, acquire an input time of the user operation instruction.

Wherein, the user operation instruction may be an operation instruction input through an input device of the client, and the above-mentioned input device may be an input device built into the client, such as: a touch screen, a built-in voice input device, etc.; External input devices connected to the client via cable, such as mouse, keyboard, gamepad, etc.

Exemplarily, the operation instruction may be a random click operation on the mouse, keyboard or gamepad. The operation instruction can also be a random click on an operable point in the operation interface, or a specified position of the operable point, for example: the four corners, the center of the operation interface, or the corresponding position of other virtual elements. The specific form depends on the actual scene.

Wherein, the background window is a first time window before the input moment. The multimedia signal in the background window may be a video and audio signal in a time window before the input moment.

Further, the input time may be the time when the user operation instruction is responded to, or the time when the user performs an operation. Specifically, if the input moment is the moment when the user operates an instruction, the measured system delay includes the input hardware delay; if the input moment is the moment when the user performs an operation, the measured system delay does not include the input delay. hardware delay.

In a preferred embodiment, the input time refers to the time when the processor of the client responds to the operation instruction. Since the time from the user's operation to the response time refers to the input hardware delay, the input hardware delay is often a fixed value, and it is easier to obtain the time from the response to the operation command than the time from the user's operation. Therefore, this implementation In the example, the input time refers to the time when the operation instruction is responded to.

Among them, multimedia refers to the synthesis of multiple media, generally including text, pictures, photos, audio and video and other media forms. A multimedia signal can generally be understood as a data signal corresponding to the above-mentioned various media forms.

Acquiring the input time may be to acquire the system time at that time when an operation instruction inputted by the user is detected. Acquiring the input time may also be to read the time stamp corresponding to the operation instruction after responding to the operation instruction input by the user, so as to obtain the input time. It should be noted that this embodiment only illustrates the manner of acquiring the input time, but does not limit it.

S12. Detect multimedia signals in the background window and the foreground window.

Wherein, the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment.

Among them, the time window, usually for some concepts in real-time information display, such as maintaining a five-minute transaction detail time window, it is necessary to record the current time, all transaction details up to five minutes ago, and the data before five minutes, then discarded . In this embodiment, a time window may be the time of multiple video frames or the time of multiple audio frames.

Specifically, the multimedia signal of the detection background window may be a video image or audio in a time window before the detection input moment. The multimedia signal of the detection foreground window may be a video image or audio in a time window before the detection input moment. Specifically, after the input time is determined, multiple frames of video or audio before the input time may be extracted as the multimedia signal of the background window; after the input time is determined, multiple frames of video or audio after the input time may be extracted as the multimedia signal of the foreground window .

S13. Based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window, determine a response time for the user operation instruction.

The user operation instruction is sent to the cloud server, and the cloud server renders the game scene into a video and audio stream based on the user operation instruction, and transmits it to the client through the network, and the client receives and displays the rendered video and audio stream.

The response time of the user operation command can be understood as the time when the client displays the multimedia signal through the output device, that is, the time when the processor of the client sends the multimedia signal to the output device, and the output device obtains the multimedia signal. The above-mentioned output device can be an output device built in the computer equipment, such as a touch screen, a built-in speaker, etc.; it can also be an external output device connected to the computer equipment through a communication line, such as a projector, a digital TV, an external speakers, etc.

Exemplarily, the response time of the above-mentioned user operation instruction may be the time when the video signal is received and rendered and is about to be displayed through a touch screen or a digital TV. The response time of the above-mentioned user operation instruction may also be the time when the audio signal is about to be played through the speaker after the audio signal is received and rendered.

In this embodiment, the determination of the response time of the user operation instruction may be when a response to the multimedia signal is detected, and the system time at this time is acquired to determine the response time of the response to the multimedia signal. The impression timestamp for the input moment. It should be noted that this embodiment only illustrates the manner of determining the response time, rather than limiting it.

In one embodiment, a starting point detection method is used to determine the response time of the user operation instruction.

Specifically, it is detected whether there is an obvious mutation of the multimedia signal in the background window and the foreground window. For example, if there is no sudden change in the energy of the audio in the background window but there is an obvious sudden change in the foreground window, then the sudden change is considered to be caused by the user’s operation command, which is the system’s response to the user’s operation command, and the moment corresponding to the sudden change is obtained and determined as a definite response Response time for multimedia signals.

S14. Determine a system delay of the system according to the input time and the response time.

As shown in FIG. 2 , the difference between the response time of the multimedia signal on the local client and the input time corresponding to the user's input operation command is the calculated system delay.

The system delay determination method provided by the embodiment of the present disclosure includes: responding to the user operation instruction acting on the multimedia signal of the background window, obtaining the input time of the user operation instruction; detecting the multimedia signal in the background window and the foreground window, wherein, the background The window is the first time window before the input moment, and the foreground window is the second time window after the input moment; based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window, determine the response time of the user operation instruction; Determine the system delay of the system according to the input time and response time. The embodiment of the present disclosure determines the system delay by obtaining the input time of the user operation command and the corresponding response time of the user operation command. This solution does not require expensive hardware equipment or excessive manual participation, and can measure remote rendering conveniently and quickly. System interaction delay.

On the basis of the foregoing embodiments, embodiments of the present disclosure optimize the method for determining system delay. Fig. 4 is a flowchart of another method for determining system delay provided by an embodiment of the present disclosure; as shown in Fig. 4 , the method for determining system delay provided by an embodiment of the present disclosure mainly includes the following steps:

S21. In response to a user operation instruction acting on a multimedia signal in a background window, acquire an input time of the user operation instruction.

S22. Detect multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment.

In this embodiment, the time window, usually for some concepts in real-time information display, such as maintaining a five-minute transaction detail time window, needs to record all transaction details up to five minutes before the current time, and five minutes ago data is discarded. In this embodiment, a time window may be the time of multiple video frames or the time of multiple audio frames.

S23. Determine whether the sudden change of the multimedia signal in the foreground window is caused by the user operation instruction.

In this embodiment, a sudden change in the signal can be understood as a great change in the signal waveform within a relatively short period of time. The absence of a sudden change in the multimedia signal in the background window may immediately mean that the waveform of the multimedia signal in the background window does not change, or the change is smaller than the change threshold. A sudden change in the multimedia signal in the foreground window may immediately cause a great change in the waveform of the multimedia signal in the background window, or the change is greater than or equal to the change threshold. The foregoing change threshold may be set according to actual conditions, and is not limited in this embodiment.

Further, detect whether there are obvious signal mutations in the background window and the foreground window. For example, if the energy of the audio has no sudden change in the background window but has an obvious sudden change in the foreground window, it is considered that the sudden change is caused by user input and is the system’s response to the operation command, and the moment when the energy of the audio in the foreground window suddenly changes is determined. The response time for the response audio. In the same way, if the energy of the video has no sudden change in the background window but has an obvious sudden change in the foreground window, it is considered that the sudden change is caused by user input and is the system's response to the operation command, and the energy of the video in the foreground window is suddenly changed. The moment is determined as the response moment of the response video.

In one embodiment, determining whether the sudden change of the multimedia signal in the foreground window is caused by the user operation instruction includes: combining the sudden change signal of the multimedia signal in the foreground window with the user operation instruction The instruction is input into a pre-trained consistent classifier; and based on the output result of the consistent classifier, it is determined whether the sudden change of the multimedia signal in the foreground window is caused by the user operation instruction.

In this embodiment, the consistency classifier is used to detect whether the mutation signal of the multimedia signal in the foreground window is consistent with the user operation instruction input, that is, the mutation signal of the multimedia signal in the foreground window Whether the signal is caused by a user action command. Specifically, if the mutation signal of the multimedia signal in the foreground window is caused by a user operation instruction, then it has consistency; if the mutation signal of the multimedia signal in the foreground window is not caused by a user operation instruction, then it is not Be consistent.

Exemplarily, in the shooting cloud game, if the user operation instruction is a shooting instruction, and the mutation signal of the multimedia signal in the foreground window is a shooting action (including shooting video and/or shooting audio), then it is determined The mutation signal of the multimedia signal in the foreground window is consistent with the shooting instruction input by the user. If the user operation instruction is a shooting instruction, and the mutation signal of the multimedia signal in the foreground window is an aiming action, then it is determined that the mutation signal of the multimedia signal in the foreground window is consistent with the shooting instruction input by the user .

In one embodiment, the training process of the consistency classifier includes: obtaining a training sample set, the training sample set including: a plurality of historical operation instructions and mutation signals of a plurality of historical multimedia signals; extracting each of the historical The feature of the mutation signal of the multimedia signal; The number of times that the first feature occurs is counted, and the first feature refers to the feature of the mutation signal of the historical multimedia signal corresponding to the historical operation instruction; The number of times that the first feature occurs is determined as the history A normalized value corresponding to the operation instruction; inputting the historical operation instruction and the normalized value corresponding to the historical operation instruction into a preset classifier for training to obtain a consistent classifier.

Specifically, the preset classifiers include one or more of the following: support vector machine SVM classifier, nearest neighbor classifier, naive Bayesian classifier, random forest classifier, and linear discriminant classifier.

Wherein, the pre-trained consistency classifier is trained through the bag-of-features model.

In this embodiment, the same user input behavior will bring about similar system response. In order to detect the consistency of the system response, for a remote rendering, a part of the data will be collected to pre-train a consistency classifier, and the pre-trained consistency classifier will be used in the test phase to judge whether the detected multimedia signal of the response meets the response consistency. Specifically, a bag of feature model (Bag of Feature) is used to train a classifier for system response.

Wherein, the multimedia signal includes one or more of the following modalities: video signal, audio signal, and somatosensory signal.

In one embodiment, extracting the feature of each abrupt signal of the historical multimedia signal includes: when the multimedia signal includes a video signal, extracting a scale-invariant feature transformation SIFT feature corresponding to the abrupt signal of each video signal.

In one embodiment, extracting the feature of each abrupt signal of the historical multimedia signal includes: when the multimedia signal includes an audio signal, extracting the cepstral mel coefficients corresponding to the abrupt signal of each audio signal.

Specifically, when training a consistent classifier, different features are used for data of different modalities. For video data, the scale-invariant feature transform (SIFT) features are used for training. For audio data, Mel-Frequency Cepstral Coefficients (MFCC) are used for training. The default classifier uses a single-class Support Vector Machine (Support Vector Machine, SVM), nearest neighbor classifier, etc.

S24. If the sudden change of the multimedia signal in the foreground window is caused by the user operation instruction, perform a change based on the multimedia signal in the foreground window relative to the multimedia signal in the background window In case of a sudden change, determine the operation at the time when the system responds to the user operation instruction.

In one embodiment, based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window, determining the response time for the system to respond to the user operation instruction includes: if the When the multimedia signal in the foreground window changes abruptly relative to the multimedia signal in the background window, the moment when the multimedia signal in the foreground window changes suddenly is determined as the response time for responding to the operation instruction.

Further, if there is no sudden change in the energy of the audio in the background window but there is an obvious sudden change in the foreground window, it is considered that the sudden change is caused by user input, which is the system’s response to the operation command, and the energy of the audio in the foreground window is suddenly changed. time is determined as the response time of the response audio. In the same way, if the energy of the video has no sudden change in the background window but has an obvious sudden change in the foreground window, it is considered that the sudden change is caused by user input and is the system's response to the operation command, and the energy of the video in the foreground window is suddenly changed. The moment is determined as the response moment of the response video.

S25. For each mode signal, determine a response delay corresponding to the mode based on the input time and output time corresponding to the mode.

In this embodiment, the system delay determined in the above manner may be a single-mode delay or a multi-mode delay. When the multimedia signal is a single video signal or a single audio signal, the difference between the response time and the input time is directly determined as the system delay.

When the multimedia signal includes multiple media signals, such as video signals or audio signals, a signal fusion algorithm is used to fuse the system delays of the multiple modalities to obtain the system delay.

S26. Use a signal fusion algorithm to fuse response delays corresponding to multiple modes to determine a system delay of the system.

The system delays of multiple modes are fused to obtain the final delay result. This part can be fused using the pre-calibrated modal delay deviation, or the signal fusion algorithm can be used on the basis of the modal delay deviation ( Such as Kalman filtering, etc.) for further fusion.

For example, for the same operation command input by the user, the sound effect is often slower than the change of the screen, so after the delay corresponding to the sound effect is increased, it can be weighted and averaged with the delay of the screen to obtain a new system delay, that is, the final system delay. Wherein, the weight value corresponding to each media signal may be set according to the actual situation, and is not specifically limited in this embodiment.

On the basis of the foregoing embodiments, the method further includes: optimizing system design parameters based on the system delay.

In the embodiment of the present disclosure, after the client determines the system delay through the aforementioned method embodiments for the client, it can upload the delay to the server through the system delay, and the server can display the delay to the user on the server after receiving the system delay, which is convenient The user performs further data statistics, analysis and optimization after a delay according to the system. Specifically, in the application of system development and testing, the developer can obtain the system delay determined by the terminal of the embodiment of the present application through the server to perform system performance testing, and can adjust or optimize the design indicators or parameters of the system according to the test results. In actual operation and maintenance applications, users such as system operators and system operation and maintenance personnel can obtain the system delay determined by the embodiment of the present application through the server to count the system delay index data of actual users in products in the live network environment, Assist in locating problems such as user freezes, and optimize the system based on the system delay data, such as optimizing access deployment for areas with high delay or operators.

On the basis of the foregoing embodiments, the embodiments of the present disclosure provide another preferred example. In this embodiment, a multimedia signal including a video signal and an audio signal is taken as an example for illustration. Fig. 5 is a schematic diagram of a method for determining system delay provided by an embodiment of the present disclosure.

As shown in Figure 5, the background window audio signal is obtained from the background window of the audio, the foreground window audio signal is obtained from the foreground window of the audio, and the sudden change of the background window audio signal and the foreground window audio signal is judged. If the background window audio signal If there is no mutation and the mutation of the audio signal of the foreground window, the audio event monitoring is carried out. If not, the audio signal of the background window and the audio signal of the foreground window are discarded. After the audio event monitoring, the MFCC in the audio signal is extracted, and the MFCC in the audio signal is used for consistency. Line detection, judging whether the MFCC in the audio signal meets the consistency detection, if it is satisfied, the delay of the audio signal is sent to the fusion module, if not, the MFCC in the audio signal is discarded.

Obtain the video signal of the background window from the background window of the video, obtain the video signal of the foreground window from the foreground window of the video, and judge the sudden change of the video signal of the background window and the video signal of the foreground window. If there is a sudden change in the signal, the video event monitoring is performed. If not, the background window video signal and the foreground window video signal are discarded. After the video event monitoring, the MoSIFT/SURF in the video signal is extracted, and the consistent row detection is performed through MoSIFT/SURF in the video signal. , to judge whether the MoSIFT/SURF in the video signal meets the consistency detection, if so, send the delay of the video signal to the fusion module, if not, discard the MoSIFT/SURF in the video signal.

The fusion module performs multimodal fusion of the delay of the audio signal and the delay of the video signal to obtain the system delay.

Further, the left figure in Figure 6 is MoSIFT in video data, and the right figure in Figure 6 is MFCC in audio data, and the delay results for the video data and audio data in Figure 6 are shown in Figure 7 . Figure 7 shows how the delay measured by the algorithm varies with artificial delay after artificial delay is added to the test data. The results show that the delay estimation results of the present disclosure better reflect the artificially added delay.

Fig. 8 is a schematic diagram of a device for determining system delay provided by an embodiment of the present disclosure. This embodiment is applicable to determining the delay of a remote rendering interactive system, and the device for determining system delay can be implemented through software and/or hardware way to achieve. The system delay determination device is configured in the client.

As shown in FIG. 8 , the system delay determination device provided by this embodiment mainly includes an input time acquisition module 81 , a multimedia signal detection module 82 , a response time determination module 83 and a system delay determination module 84 .

The input time acquisition module 81 is configured to acquire the input time of the user operation command in response to the user operation command acting on the multimedia signal of the background window;

A multimedia signal detection module 82, configured to detect multimedia signals in a background window and a foreground window, wherein the background window is the first time window before the input moment, and the foreground window is the second time window after the input moment. time window;

A response time determining module 83, configured to determine the response time of the user operation instruction based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window;

A system delay determination module 84, configured to determine the system delay of the system according to the input time and the response time.

The device for determining the system delay provided by the embodiment of the present disclosure performs the following operations: in response to the user operation instruction acting on the multimedia signal of the background window, obtain the input time of the user operation instruction; detect the multimedia signal in the background window and the foreground window, wherein , the background window is the first time window before the input time, and the foreground window is the second time window after the input time; based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window, determine the response of the user operation command Time: Determine the system delay of the system according to the input time and response time. The embodiment of the present disclosure determines the system delay by obtaining the input time of the user operation command and the corresponding response time of the user operation command. This solution does not require expensive hardware equipment or excessive manual participation, and can measure remote rendering conveniently and quickly. System interaction delay.

Further, the response time determination module 83 is specifically configured to, if the multimedia signal in the foreground window changes abruptly relative to the multimedia signal in the background window, then change the multimedia signal in the foreground window to a sudden change. The time of is determined as the response time of responding to the operation instruction.

Further, the device further includes: a sudden signal judging module, configured to determine that the system responds to the user based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window. Before the response time of the operation instruction, determine whether the mutation of the multimedia signal in the foreground window is caused by the user operation instruction; if the mutation of the multimedia signal in the foreground window is caused by the user operation instruction If yes, the operation of determining the response time when the system responds to the user operation instruction is performed based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window.

Further, the mutation signal judging module is specifically configured to input the mutation signal of the multimedia signal in the foreground window and the user operation instruction into a pre-trained consistency classifier; The output result determines whether the sudden change of the multimedia signal in the foreground window is caused by the user operation instruction.

Further, the training process of the consistency classifier includes:

Obtain a training sample set, the training sample set includes: a plurality of historical operation instructions and mutation signals of a plurality of historical multimedia signals;

Extracting the features of the mutation signals of each of the historical multimedia signals;

Counting the number of occurrences of the first feature, where the first feature refers to the feature of a sudden change signal of the historical multimedia signal corresponding to the historical operation instruction;

Determining the number of occurrences of the first feature as a normalized value corresponding to the historical operation instruction;

Inputting the historical operation instructions and the normalized values corresponding to the historical operation instructions into a preset classifier for training to obtain a consistent classifier.

Further, the preset classifiers include one or more of the following:

Support Vector Machine SVM Classifier, Nearest Neighbor Classifier, Naive Bayes Classifier, Random Forest Classifier, Linear Discriminant Classifier.

Further, the multimedia signal includes one or more of the following modalities: video signal, audio signal, and somatosensory signal.

Further, extracting the features of the mutation signals of each of the historical multimedia signals, including:

When the multimedia signal includes a video signal, the scale-invariant feature transformation SIFT feature corresponding to the sudden change signal of each video signal is extracted.

Further, extracting the features of the mutation signals of each of the historical multimedia signals, including:

When the multimedia signal includes an audio signal, the cepstral mel coefficients corresponding to the sudden change signal of each audio signal are extracted.

Further, the system delay determination module 84 is specifically used for each mode signal, based on the input time and output time corresponding to the mode to determine the response delay corresponding to the mode; using the signal fusion algorithm to combine multiple modes The corresponding response delays are fused to determine the system delay of the system.

Further, the device further includes: a parameter optimization module, configured to optimize system design parameters based on the system delay.

The device for determining system delay provided in this embodiment can execute the method for determining system delay provided in any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the method for determining system delay.

Referring now to FIG. 9 , it shows a schematic structural diagram of an electronic device (such as the client device or server in FIG. 9 ) 900 suitable for implementing the embodiments of the present disclosure. Client devices in embodiments of the present disclosure may include, but are not limited to, devices such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), car client Mobile clients such as car navigation clients, etc. and stationary clients such as digital TVs, desktop computers, and the like. The electronic device shown in FIG. 9 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.

As shown in FIG. 9, an electronic device 900 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 901, which may be randomly accessed according to a program stored in a read-only memory (ROM) 902 or loaded from a storage device 908. Various appropriate actions and processes are executed by programs in the memory (RAM) 903 . In the RAM 903, various programs and data necessary for the operation of the electronic device 900 are also stored. The processing device 901 , ROM 902 , and RAM 903 are connected to each other through a bus 904 . An input/output (I/O) interface 905 is also connected to the bus 904 .

Typically, the following devices can be connected to the I/O interface 905: input devices 906 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration an output device 907 such as a computer; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication means 909 may allow the electronic device 900 to perform wireless or wired communication with other devices to exchange data. While FIG. 9 shows electronic device 900 having various means, it is to be understood that implementing or having all of the means shown is not a requirement. More or fewer means may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, where the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 909 , or from storage means 908 , or from ROM 902 . When the computer program is executed by the processing device 901, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the server can communicate with any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol, Hypertext Transfer Protocol), and can communicate with digital data in any form or medium ( For example, communication networks) interconnect. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device, or may exist independently without being incorporated into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device:

Responding to a user operation instruction acting on the multimedia signal of the background window, acquiring the input moment of the user operation instruction;

detecting a multimedia signal in a background window and a foreground window, wherein the background window is a first time window before the input moment and the foreground window is a second time window after the input moment;

Determine the response time of the user operation instruction based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window;

A system delay of the system is determined according to the input time and the response time.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or combinations thereof, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of a unit does not constitute a limitation of the unit itself under certain circumstances.

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chips (SOCs), Complex Programmable Logical device (CPLD) and so on.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device and storage medium are provided, including:

Responding to a user operation instruction acting on the multimedia signal of the background window, acquiring the input moment of the user operation instruction;

detecting a multimedia signal in a background window and a foreground window, wherein the background window is a first time window before the input moment and the foreground window is a second time window after the input moment;

Determine the response time of the user operation instruction based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window;

A system delay of the system is determined according to the input time and the response time.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device, and storage medium are provided, based on the multimedia signal in the foreground window relative to the background window The sudden change of the multimedia signal, and determine the response time when the system responds to the user operation instruction, including:

If the multimedia signal in the foreground window changes abruptly relative to the multimedia signal in the background window, determine the moment when the multimedia signal in the foreground window changes suddenly as the response to the operation instruction Response time.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device, and storage medium are provided, based on the multimedia signal in the foreground window relative to the background window Before determining the sudden change of the multimedia signal in the system to respond to the user operation instruction, it also includes:

determining whether the sudden change in the multimedia signal in the foreground window is caused by the user operation instruction;

If the sudden change in the multimedia signal in the foreground window is caused by the user operation instruction, then execute based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window , determining an operation at a time when the system responds to the user operation instruction.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device and storage medium are provided, and it is determined whether the sudden change of the multimedia signal in the foreground window is the Caused by user operation instructions, including:

inputting the mutation signal of the multimedia signal in the foreground window and the user operation instruction into a pre-trained consistent classifier;

It is determined based on the output result of the consistency classifier whether the sudden change of the multimedia signal in the foreground window is caused by the user operation instruction.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device, and storage medium are provided. The training process of the consistency classifier includes:

Obtain a training sample set, the training sample set includes: a plurality of historical operation instructions and mutation signals of a plurality of historical multimedia signals;

Extracting the features of the mutation signals of each of the historical multimedia signals;

Counting the number of occurrences of the first feature, where the first feature refers to the feature of a sudden change signal of the historical multimedia signal corresponding to the historical operation instruction;

Determining the number of occurrences of the first feature as a normalized value corresponding to the historical operation instruction;

Inputting the historical operation instructions and the normalized values corresponding to the historical operation instructions into a preset classifier for training to obtain a consistent classifier.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device, and storage medium are provided, and the preset classifier includes one or more of the following:

Support Vector Machine SVM Classifier, Nearest Neighbor Classifier, Naive Bayes Classifier, Random Forest Classifier, Linear Discriminant Classifier.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device and storage medium are provided, the multimedia signal includes one or more of the following modalities: video signal , audio signal, somatosensory signal.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device, and storage medium are provided, and the characteristics of each abrupt change signal of the historical multimedia signal are extracted, including:

When the multimedia signal includes a video signal, the scale-invariant feature transformation SIFT feature corresponding to the sudden change signal of each video signal is extracted.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device, and storage medium are provided, and the characteristics of each abrupt change signal of the historical multimedia signal are extracted, including:

When the multimedia signal includes an audio signal, the cepstral mel coefficients corresponding to the sudden change signal of each audio signal are extracted.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device and storage medium are provided, and the system delay of the system is determined according to the input time and the response time, including :

For each mode signal, determine the response delay corresponding to the mode based on the input time and output time corresponding to the mode;

A signal fusion algorithm is used to fuse response delays corresponding to multiple modes to determine the system delay of the system.

According to one or more embodiments of the present disclosure, a system delay determination method, device, device, and storage medium are provided, and the method further includes: optimizing system design parameters based on the system delay .

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principle. Those skilled in the art should understand that the disclosure scope involved in this disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, but also covers the technical solutions formed by the above-mentioned technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

In addition, while operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or performed in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims (13)

1. A system delay determination method, characterized in that, comprising: Responding to a user operation instruction acting on the multimedia signal of the background window, acquiring the input time of the user operation instruction, the input time being the time when the client responds to the user operation instruction; Detecting multimedia signals in the background window and the foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment; Determine the response time of the user operation instruction based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window; determining a system delay of the system according to the input time and the response time; If the multimedia signal in the foreground window changes abruptly relative to the multimedia signal in the background window, determine the moment when the multimedia signal in the foreground window changes suddenly as the response to the operation instruction Response time. 2. The method according to claim 1, wherein, based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window, it is determined that the system responds to the user operation Before the response time of the command, it also includes: determining whether the sudden change in the multimedia signal in the foreground window is caused by the user operation instruction; If the sudden change in the multimedia signal in the foreground window is caused by the user operation instruction, then execute based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window , determining an operation at a time when the system responds to the user operation instruction. 3. The method according to claim 2, wherein determining whether the sudden change in the multimedia signal in the foreground window is caused by the user operation instruction comprises: inputting the mutation signal of the multimedia signal in the foreground window and the user operation instruction into a pre-trained consistent classifier; It is determined based on the output result of the consistency classifier whether the sudden change of the multimedia signal in the foreground window is caused by the user operation instruction. 4. method according to claim 3, is characterized in that, the training process of described consistent classifier, comprises: Obtain a training sample set, the training sample set includes: a plurality of historical operation instructions and mutation signals of a plurality of historical multimedia signals; Extracting the features of the mutation signals of each of the historical multimedia signals; Counting the number of occurrences of the first feature, where the first feature refers to the feature of a sudden change signal of the historical multimedia signal corresponding to the historical operation instruction; Determining the number of occurrences of the first feature as a normalized value corresponding to the historical operation instruction; Inputting the historical operation instructions and the normalized values corresponding to the historical operation instructions into a preset classifier for training to obtain a consistent classifier. 5. The method according to claim 4, wherein the preset classifiers include one or more of the following: Support Vector Machine SVM Classifier, Nearest Neighbor Classifier, Naive Bayes Classifier, Random Forest Classifier, Linear Discriminant Classifier. 6. The method according to any one of claims 1-4, wherein the multimedia signal includes one or more of the following modalities: video signal, audio signal, and somatosensory signal. 7. method according to claim 6, is characterized in that, extracting the feature of the mutation signal of each history multimedia signal, comprises: When the multimedia signal includes a video signal, the scale-invariant feature transformation SIFT feature corresponding to the sudden change signal of each video signal is extracted. 8. method according to claim 6, is characterized in that, extracting the feature of the mutation signal of each history multimedia signal, comprises: When the multimedia signal includes an audio signal, the cepstral mel coefficients corresponding to the sudden change signal of each audio signal are extracted. 9. The method according to any one of claims 7-8, wherein determining the system delay of the system according to the input time and the response time comprises: For each mode signal, determine the response delay corresponding to the mode based on the input time and output time corresponding to the mode; A signal fusion algorithm is used to fuse response delays corresponding to multiple modes to determine the system delay of the system. 10. The method of claim 1, further comprising: System design parameters are optimized based on the system delay. 11. A system delay determining device, characterized in that, comprising: The input time acquisition module is used to obtain the input time of the user operation command in response to the user operation command acting on the multimedia signal of the background window, and the input time is the time when the client responds to the user operation command; A multimedia signal detection module, configured to detect multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time after the input moment window; A response time determination module, configured to determine the response time of the user operation instruction based on the sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window; A system delay determination module, configured to determine the system delay of the system according to the input time and the response time; The response time determination module is specifically configured to determine the moment when the multimedia signal in the foreground window changes abruptly relative to the multimedia signal in the background window as The response time for responding to the operation instruction. 12. A system delay determination device, characterized in that, comprising: one or more processors; memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the method for determining system delay according to any one of claims 1-10. 13. A computer storage medium, wherein the medium stores a computer program, and when the computer program is executed by a processor, the system delay determination method according to any one of claims 1-10 is implemented.

Images