CN112843677B - System delay determining method, device, equipment and storage medium - Google Patents
System delay determining method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112843677B CN112843677B CN202110172567.7A CN202110172567A CN112843677B CN 112843677 B CN112843677 B CN 112843677B CN 202110172567 A CN202110172567 A CN 202110172567A CN 112843677 B CN112843677 B CN 112843677B
- Authority
- CN
- China
- Prior art keywords
- window
- operation instruction
- signal
- time
- multimedia signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/30—Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
- A63F13/35—Details of game servers
- A63F13/358—Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/50—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by details of game servers
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- User Interface Of Digital Computer (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The embodiment of the disclosure provides a system delay determining method, a device, equipment and a storage medium, comprising the following steps: responding to a user operation instruction acting on a multimedia signal of a background window, and acquiring the input moment of the user operation instruction; detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment; determining the response time of the user operation instruction based on the mutation condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window; and determining the system delay of the system according to the input time and the response time. According to the embodiment of the disclosure, the system delay is determined by acquiring the input time of the user operation instruction and the response time corresponding to the user operation instruction, and the scheme does not need expensive hardware equipment or excessive manual participation, so that the interaction delay of the remote rendering system can be conveniently and rapidly measured.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of data processing, in particular to a system delay determining method, a system delay determining device and a storage medium.
Background
In the remote rendering system, a user inputs an operation instruction through a client, the operation instruction is sent to a server, the server acquires a media signal corresponding to the operation instruction, the media signal is returned to the client, and the client responds to the media signal and displays the media signal. In a remote rendering system, a client needs to go to a server to request a media signal, so that a certain delay exists between an operation instruction and a corresponding media signal presentation. Common remote rendering system delays include cloud gaming delays, screen-drop applications delays, or remote desktop delays, etc.
In the prior art, a manual measurement method based on a high-speed camera is mostly adopted to determine the time delay of a remote rendering system. FIG. 1 is a schematic diagram of a high-speed camera-based delay determination method; as shown in fig. 1, the user's input to the remote rendering system and the remote system's response are recorded using a high-speed camera. Then manually finding the video frame input by the user and the video frame generated by the system response, wherein the time difference between the video frame input by the user and the video frame generated by the system response is the system delay.
The method can accurately measure the delay of a remote system, but requires an expensive high-speed camera, and meanwhile, manually searching for a user input frame and a system response frame is a time-consuming and labor-consuming task.
Disclosure of Invention
The embodiment of the disclosure provides a system delay determining method, device, equipment and storage medium, so as to conveniently and quickly measure the interaction delay of a remote rendering system.
In a first aspect, an embodiment of the present disclosure provides a system delay determining method, including:
responding to a user operation instruction acting on a multimedia signal of a background window, and acquiring the input moment of the user operation instruction;
detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment;
determining a response time of the user operation instruction based on a sudden change condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window;
and determining the system delay of the system according to the input time and the response time.
In a second aspect, an embodiment of the present disclosure further provides a system delay determining apparatus, which is characterized in that the system delay determining apparatus includes:
an input time acquisition module, configured to respond to a user operation instruction applied to a multimedia signal of a background window, and acquire an input time of the user operation instruction;
The multimedia signal detection module is used for detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment;
a response time determining module, configured to determine a response time of the user operation instruction based on a sudden change situation of the multimedia signal in the foreground window relative to the multimedia signal in the background window;
and the system delay determining module is used for determining the system delay of the system according to the input time and the response time.
In a third aspect, an embodiment of the present disclosure further provides a system delay determining apparatus, including:
one or more processors;
a memory for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement a system latency determination method as in any of the embodiments of the present disclosure.
In a fourth aspect, the disclosed embodiments also provide a computer storage medium storing a computer program which, when executed by a processor, implements a system delay determination method according to any one of the disclosed embodiments.
The present disclosure provides a system delay determining method, device, equipment and storage medium, where the system delay determining method includes: responding to a user operation instruction acting on a multimedia signal of a background window, and acquiring the input moment of the user operation instruction; detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment; determining the response time of the user operation instruction based on the mutation condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window; and determining the system delay of the system according to the input time and the response time. According to the embodiment of the disclosure, the system delay is determined by acquiring the input time of the user operation instruction and the response time corresponding to the user operation instruction, and the scheme does not need expensive hardware equipment or excessive manual participation, so that the interaction delay of the remote rendering system can be conveniently and rapidly measured.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
FIG. 1 is a schematic diagram of a high-speed camera-based delay determination method;
FIG. 2 is a schematic diagram of a topology of a remote rendering system;
FIG. 3 is a flow chart of a system delay determination method provided by an embodiment of the present disclosure;
FIG. 4 is a flow chart of another system delay determination method provided by an embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a system delay determination method provided by an embodiment of the present disclosure;
FIG. 6 is a schematic diagram of MoSIFT and MFCC provided by an embodiment of the present disclosure;
FIG. 7 is a schematic diagram of a delayed outcome on two cloud games provided by an embodiment of the present disclosure;
fig. 8 is a schematic diagram of a system delay determining device according to an embodiment of the disclosure;
fig. 9 is a block diagram of a system delay determining device according to an embodiment of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.
It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.
The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
First, a cloud game delay will be described as an example. Cloud gaming (Cloud gaming), which may also be referred to as game on demand, is an online gaming technology based on Cloud computing technology. Cloud gaming technology enables lightweight devices (thin clients) with relatively limited graphics processing and data computing capabilities to run high quality games. In a cloud game scene, the game is not run in a player game client, but runs in a cloud server, the cloud server renders the game scene into a video and audio stream, and the video and audio stream is transmitted to the player game client through a network. The player game client does not need to have strong graphic operation and data processing capability, and only needs to have basic streaming media playing capability and the capability of acquiring player input instructions and sending the player input instructions to the cloud server. In order to ensure smoothness of the cloud game process, a cloud game delay test is required.
Secondly, the delay of the remote rendering system is simply introduced. As shown in fig. 2, a remote rendering system often includes a local client and a remote server, where the end-to-end delay includes an input hardware delay t1, a main process delay and an output hardware delay t2. The main process delay includes local client processing delay (not shown in fig. 2), network transmission delay (local area network uploading delay t2, wide area network uploading delay t3, wide area network downloading delay t5, local area network downloading delay t 6) and remote server processing delay t7 in fig. 2, which are the most important parts of the client-to-server delay. The input hardware delay t1 and the output hardware delay t2 are typically substantially constant values, while the main process delay tends to be much greater than the input hardware delay t1 and the output hardware delay t2. The determination of the main process delay as shown in fig. 2 is therefore mainly considered in this disclosure.
It should be noted that, the system delays in the following embodiments refer to the main process delays in fig. 2, and do not include the input hardware delay t1 and the output hardware delay t2.
In this embodiment, the server may be an independent physical server, or may be a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The client may be, but is not limited to, a smart phone, tablet, notebook, desktop, smart speaker, smart watch, etc. The client and the server may be directly or indirectly connected through wired or wireless communication, which is not limited herein.
Fig. 3 is a flowchart of a system delay determining method provided by an embodiment of the present disclosure, where the embodiment may be adapted to determine a response delay condition of a remote rendering interaction system, and the method may be performed by a system delay determining device, where the system delay determining device may be implemented by using software and/or hardware. The system delay determining method is applied to the client.
It should be noted that, the system delay determining method provided in this embodiment may be specifically used on a client, and may be considered to be specifically executed by a system delay determining device integrated on the client, where the client may be a computer device including a processor, a memory, an input device, and an output device. Such as notebook computers, desktop computers, tablet computers, intelligent terminals, and the like.
As shown in fig. 3, the system delay determining method provided in this embodiment mainly includes steps S11, S12, S13, and S14.
S11, responding to a user operation instruction acted on a multimedia signal of a background window, and acquiring the input moment of the user operation instruction.
The user operation instruction may be an operation instruction input through an input device of the client, where the input device may be an input device built in the client, such as: touch display screen, built-in voice input device, etc.; or an external input device connected with the client through a communication line, such as: a mouse, keyboard, gamepad, etc.
Illustratively, the operation instruction may be a random click operation of a mouse, a keyboard, or a joystick. The operation instruction may also be a random click of an operable point in the operation interface, or a specified operable point position, such as: the four corners, the center or other virtual element of the operation interface correspond to the positions, and the specific form is determined according to the actual scene.
Wherein the background window is a first time window before the input time. The multimedia signal of the background window may be a video audio signal within a time window before the input instant.
Further, the input time may be a time in response to a user operation instruction, or may be a time when the user performs an operation. Specifically, if the input time is the time in response to the user operation instruction, the measured system delay includes the input hardware delay, and if the input time is the time in which the user operates, the measured system delay does not include the input hardware delay.
In a preferred embodiment, the input time refers to the time when the processor of the client responds to the operation instruction. Since the time from the time when the user operates to the time when the user responds refers to the input hardware delay, the input hardware delay is often a fixed value, and the time when the user responds to the operation instruction is easier than the time when the user operates is acquired, in this embodiment, the input time refers to the time when the user responds to the operation instruction.
Multimedia refers to the integration of various media, generally including text, pictures, photographs, audio and video, and other forms of media. A multimedia signal is generally understood to be a data signal corresponding to the various media forms described above.
The acquisition input timing may be a system time at which an operation instruction in response to a user input is detected. The input time may also be obtained by reading a time stamp corresponding to an operation instruction after the operation instruction is input by a user. The present embodiment is described with reference to the manner in which the input time is acquired, and is not limited thereto.
And S12, detecting the multimedia signals in the background window and the foreground window.
Wherein the background window is a first time window before the input time and the foreground window is a second time window after the input time.
Where the time window, typically for some concepts in real-time information presentation, such as maintaining a five minute transaction detail time window, requires recording the current time, all transaction details up to five minutes ago, and data up to five minutes ago is discarded. In this embodiment, one time window may be the time of a plurality of video frames or the time of a plurality of audio frames.
In particular, the multimedia signal for detecting the background window may be a video image or audio within a time window before the detection of the input moment. The multimedia signal for detecting the foreground window may be a video image or audio within a time window immediately after the detection of the input moment. Specifically, after the input time is determined, a multi-frame video or audio before the input time is extracted as a multimedia signal of a background window, and after the input time is determined, a multi-frame video or audio after the input time is extracted as a multimedia signal of a foreground window.
S13, determining the response time of the user operation instruction based on the abrupt change condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window.
The method comprises the steps that a user operation instruction is sent to a cloud server, the cloud server renders a game scene into a video and audio stream based on the user operation instruction, the video and audio stream is transmitted to a client through a network, and the client receives and displays the rendered video and audio stream.
The response time of the user operation instruction can be understood as the time when the client side displays the multimedia signal through the output device, that is, the time when the processor of the client side sends the multimedia signal to the output device and the output device obtains the multimedia signal. The output means may be an output device built in a computer device, such as: touch display screens, built-in speakers, etc.; or an external output device connected with the computer device through a communication line, such as: projector, digital TV, external speakers, etc.
For example, the response time of the user operation instruction may be a time when the display is about to be performed by touching the display screen or the digital TV after receiving the video signal and rendering. The response time of the user operation instruction may also be the time when the audio signal is received and rendered, and the audio signal is to be played through the sound box.
In this embodiment, the determining of the response time of the user operation instruction may be that when the response to the multimedia signal is detected, the acquiring of the system time at this time determines the response time of the response to the multimedia signal, or may be that after the response to the multimedia signal, the presentation time stamp corresponding to the multimedia signal is read to acquire the input time. The present embodiment is described with reference to the manner in which the response time is determined, and is not limited thereto.
In one embodiment, a starting point detection method is used to determine the response time of the user operation instruction.
Specifically, whether there is a clear multimedia signal mutation in the background window and the foreground window is detected. For example, if the energy of the audio has no mutation in the background window and has obvious mutation in the foreground window, the mutation is considered to be caused by the user operation instruction, the system responds to the user operation instruction, the moment corresponding to the mutation is obtained, and the response moment for responding to the multimedia signal is determined.
S14, determining the system delay of the system according to the input time and the response time.
As shown in fig. 2, the difference between the response time of the multimedia signal on the local client and the input time corresponding to the user input operation instruction is the calculated system delay.
The system delay determining method provided by the embodiment of the disclosure comprises the following steps: responding to a user operation instruction acting on a multimedia signal of a background window, and acquiring the input moment of the user operation instruction; detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment; determining the response time of the user operation instruction based on the mutation condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window; and determining the system delay of the system according to the input time and the response time. According to the embodiment of the disclosure, the system delay is determined by acquiring the input time of the user operation instruction and the response time corresponding to the user operation instruction, and the scheme does not need expensive hardware equipment or excessive manual participation, so that the interaction delay of the remote rendering system can be conveniently and rapidly measured.
Based on the above embodiments, the embodiments of the present disclosure optimize the system delay determination method. FIG. 4 is a flow chart of another system delay determination method provided by an embodiment of the present disclosure; as shown in fig. 4, the system delay determining method provided by the embodiment of the disclosure mainly includes the following steps:
S21, responding to a user operation instruction acted on the multimedia signal of the background window, and acquiring the input moment of the user operation instruction.
S22, detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment.
In this embodiment, the time window, for some concepts in real-time information presentation, such as maintaining a five minute transaction detail time window, is needed to record the current time, all transaction details up to five minutes ago, and the data up to five minutes ago is discarded. In this embodiment, one time window may be the time of a plurality of video frames or the time of a plurality of audio frames.
S23, determining whether the mutation of the multimedia signal in the foreground window is caused by the user operation instruction.
In this embodiment, the abrupt change of the signal is understood to be a great change in the waveform of the signal in a short time. The absence of a sudden change in the multimedia signal in the background window may be immediately an absence of a change in the waveform of the multimedia signal in the background window or a change less than a change threshold. The abrupt change of the multimedia signal in the foreground window may be immediately that the multimedia signal in the background window has a great change in waveform, or that the change is greater than or equal to a change threshold. The above-mentioned change threshold may be set according to actual situations, and is not limited in this embodiment.
Further, it is detected whether there is a significant signal mutation in the background window and the foreground window. For example, if there is no abrupt change in the energy of the audio in the background window and there is a significant abrupt change in the foreground window, the abrupt change is considered to be caused by the user input, and is a response of the system to the operation instruction, and the time when the abrupt change occurs in the energy of the audio in the foreground window is determined as the response time when the audio is responded. Similarly, if the energy of the video has no mutation in the background window and has obvious mutation in the foreground window, the mutation is considered to be caused by user input and is the response of the system to the operation instruction, and the moment when the energy of the video in the foreground window is mutated is determined to be the response moment of the response video.
In one embodiment, determining whether the mutation of the multimedia signal in the foreground window is caused by the user operation instruction comprises: inputting a sudden change signal of the multimedia signal in the foreground window and the user operation instruction into a pre-trained consistency classifier; determining whether the multimedia signal mutation in the foreground window is caused by the user operation instruction based on an output result of the consistency classifier.
In this embodiment, the consistency classifier is configured to detect whether the abrupt signal of the multimedia signal in the foreground window and the user operation instruction input have consistency, that is, whether the abrupt signal of the multimedia signal in the foreground window is caused by the user operation instruction. Specifically, the abrupt change signal of the multimedia signal in the foreground window is caused by a user operation instruction and has consistency, and the abrupt change signal of the multimedia signal in the foreground window is not caused by a user operation instruction and does not have consistency.
In the shooting cloud game, if the user operation instruction is a shooting instruction and the abrupt change signal of the multimedia signal in the foreground window is a shooting action (including a shooting video and/or a shooting audio), it is determined that the abrupt change signal of the multimedia signal in the foreground window has consistency with the shooting instruction input by the user. If the user operation instruction is a shooting instruction and the abrupt change signal of the multimedia signal in the foreground window is an aiming action, determining that the abrupt change signal of the multimedia signal in the foreground window has consistency with the shooting instruction input by the user.
In one embodiment, the training process of the consistency classifier includes: obtaining a training sample set, the training sample set comprising: a plurality of historical operating instructions and a plurality of abrupt signals of the historical multimedia signals; extracting the characteristics of abrupt signals of each historical multimedia signal; counting the occurrence times of a first feature, wherein the first feature is the feature of a sudden change signal of a historical multimedia signal corresponding to a historical operation instruction; determining the number of times of occurrence of the first feature as a standardized value corresponding to the historical operating instruction; and inputting the historical operation instruction and the standardized value corresponding to the historical operation instruction into a preset classifier for training to obtain a consistency classifier.
Specifically, the preset classifier includes one or more of the following: a Support Vector Machine (SVM) classifier, a nearest neighbor classifier, a naive Bayes classifier, a random forest classifier and a linear discriminant classifier.
Wherein the pre-trained consistency classifier is trained through a feature word bag model.
In this embodiment, the same user input behavior will bring about a similar system response. To detect consistency of the system response, a consistency classifier is pre-trained for a remote rendering by collecting a portion of the data, and the pre-trained consistency classifier is used during the test phase to determine whether the detected response multimedia signal satisfies the consistency of the response. Specifically, a Feature Bag of words model (Bag of Feature) is used to train a classifier for the system response.
Wherein the multimedia signal comprises one or more of the following modalities: video signal, audio signal, somatosensory signal.
In one embodiment, extracting the characteristics of the abrupt signal of each of the historical multimedia signals includes: and when the multimedia signals comprise video signals, extracting scale invariant feature transform SIFT features corresponding to abrupt signals of the video signals.
In one embodiment, extracting the characteristics of the abrupt signal of each of the historical multimedia signals includes: and when the multimedia signals comprise audio signals, extracting the mel-frequency cepstrum coefficients corresponding to abrupt signals of the audio signals.
Specifically, when training a consistency classifier, different features are used for data of different modalities. For video data, scale-invariant feature transform (SIFT) features are used for training. For audio data, mel-frequency cepstral coefficients (Mel-Frequency Cepstral Coefficients, MFCC) are used for training. The preset classifier uses a single class support vector machine (Support Vector Machine, SVM), nearest neighbor classifier, etc.
And S24, if the mutation of the multimedia signal in the foreground window is caused by the user operation instruction, executing the operation of determining the response time of the system in response to the user operation instruction based on the mutation condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window.
In one embodiment, determining a response time of a system in response to the user operation instruction based on a sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window includes: and if the multimedia signal in the foreground window is mutated relative to the multimedia signal in the background window, determining the moment of the mutation of the multimedia signal in the foreground window as the response moment responding to the operation instruction.
Further, if there is no abrupt change in the energy of the audio in the background window and there is a significant abrupt change in the foreground window, the abrupt change is considered to be caused by the user input, and is a response of the system to the operation instruction, and the time when the abrupt change occurs in the energy of the audio in the foreground window is determined as the response time for responding to the audio. Similarly, if the energy of the video has no mutation in the background window and has obvious mutation in the foreground window, the mutation is considered to be caused by user input and is the response of the system to the operation instruction, and the moment when the energy of the video in the foreground window is mutated is determined to be the response moment of the response video.
S25, determining response delay corresponding to each mode based on the input time and the output time corresponding to the mode aiming at each mode signal.
In this embodiment, the system delay may be a single-mode delay or a multi-mode delay determined in the above manner. When the multimedia signal is a single video signal or a single audio signal, the difference between the response time and the input time is directly determined as a system delay.
When the multimedia signal comprises a plurality of media signals, for example, a video signal or an audio signal, the system delays of the plurality of modes are fused by using a signal fusion algorithm, so as to obtain the system delay.
S26, fusing response delays corresponding to multiple modes by using a signal fusion algorithm, and determining the system delay of the system.
The system delay of a plurality of modes is fused to obtain a final delay result, and the part can be fused by using a pre-calibrated mode delay deviation, or can be further fused by using a signal fusion algorithm (such as Kalman filtering and the like) on the basis of the mode delay deviation.
For example: for the same operation instruction input by the user, the sound effect is slower than the change of the picture, so that after the delay corresponding to the sound effect is increased, the time delay of the picture is weighted and averaged to obtain a new system delay, namely the final system delay. The weight value corresponding to each media signal may be set according to the actual situation, which is not specifically limited in this embodiment.
On the basis of the above embodiment, the method further includes: and optimizing the system design parameters based on the system delay.
In the embodiment of the disclosure, after the client determines the system delay through the embodiment of the method for the client, the system delay can be uploaded to the server, and the server can display the system delay to a user for watching after receiving the system delay, so that the user can perform further data statistics, analysis and optimization according to the system delay. Specifically, in the system development test application, a developer may acquire the system delay determined by the terminal in the embodiment of the present application through a server to perform a system performance test, and may adjust or optimize a design index or parameter of the system according to a test result. In practical operation and maintenance application, users such as a system operator, a system operation and maintenance personnel and the like can acquire the system delay determined by the embodiment of the application through a server to count the system delay index data of the practical user, assist in positioning the problem of user blocking and the like in the product of the current network environment, and optimize the system according to the system delay data, such as optimizing access deployment aiming at areas or operators with higher delay.
On the basis of the above-described embodiments, the embodiments of the present disclosure further provide a preferred example, and the present embodiment is described by taking a multimedia signal including a video signal and an audio signal as an example. Fig. 5 is a schematic diagram of a system delay determining method according to an embodiment of the disclosure.
As shown in fig. 5, a background window audio signal is obtained from a background window of audio, a foreground window audio signal is obtained from a foreground window of audio, the mutation situation of the background window audio signal and the foreground window audio signal is judged, if the background window audio signal is not mutated and the foreground window audio signal is mutated, audio event monitoring is performed, if the background window audio signal is not mutated, the background window audio signal and the foreground window audio signal are discarded, MFCCs in the audio signal are extracted after the audio event monitoring, consistent line detection is performed through MFCCs in the audio signal, whether the MFCCs in the audio signal meet the consistent detection is judged, if the MFCCs meet the consistent detection, the delay of the audio signal is sent to a fusion module, and if the MFCCs in the audio signal are discarded.
The method comprises the steps of respectively obtaining background window video signals from background windows of a video, obtaining foreground window video signals from foreground windows of the video, judging mutation conditions of the background window video signals and the foreground window video signals, if the background window video signals are not mutated and the foreground window video signals are mutated, then carrying out video event monitoring, if the background window video signals are not mutated, discarding the background window video signals and the foreground window video signals, extracting MoSIFT/SURF in the video signals after the video event monitoring, carrying out consistent line detection through MoSIFT/SURF in the video signals, judging whether the MoSIFT/SURF in the video signals meets the consistent detection, if the MoSIFT/SURF in the video signals meets the consistent detection, sending delay of the video signals to a fusion module, and if the MoSIFT/SURF in the video signals is discarded.
And the fusion module fuses the time delay of the audio signal and the time delay of the video signal in a multi-mode manner to obtain the system time delay.
Further, the left graph of fig. 6 is MoSIFT in video data, the right graph of fig. 6 is MFCC in audio data, and the delay results for the video data and the audio data in fig. 6 are shown in fig. 7. Fig. 7 shows the variation of the algorithm measured delay with manual delay after the addition of the manual delay to the test data. The results show that the delay estimation results of the present disclosure better reflect the artificially added delay.
Fig. 8 is a schematic diagram of a system delay determining device provided in an embodiment of the present disclosure, where the embodiment may be adapted to determine a delay condition of a remote rendering interaction system, and the system delay determining device may be implemented by using software and/or hardware. The system delay determining device is configured in the client.
As shown in fig. 8, the system delay determining apparatus provided in this embodiment mainly includes an input time acquisition module 81, a multimedia signal detection module 82, a response time determining module 83, and a system delay determining module 84.
An input time acquisition module 81, configured to acquire an input time of a user operation instruction in response to the user operation instruction acting on a multimedia signal of a background window;
A multimedia signal detection module 82, configured to detect multimedia signals in a background window and a foreground window, where the background window is a first time window before the input time, and the foreground window is a second time window after the input time;
a response time determining module 83, configured to determine a response time of the user operation instruction based on a sudden change situation of the multimedia signal in the foreground window relative to the multimedia signal in the background window;
a system delay determining module 84, configured to determine a system delay of the system according to the input time and the response time.
The system delay determining device provided by the embodiment of the disclosure executes the following operations: responding to a user operation instruction acting on a multimedia signal of a background window, and acquiring the input moment of the user operation instruction; detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment; determining the response time of the user operation instruction based on the mutation condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window; and determining the system delay of the system according to the input time and the response time. According to the embodiment of the disclosure, the system delay is determined by acquiring the input time of the user operation instruction and the response time corresponding to the user operation instruction, and the scheme does not need expensive hardware equipment or excessive manual participation, so that the interaction delay of the remote rendering system can be conveniently and rapidly measured.
Further, the response time determining module 83 is specifically configured to determine, as the response time to the operation instruction, a time when the multimedia signal in the foreground window is mutated with respect to the multimedia signal in the background window.
Further, the device further comprises: a sudden change signal judging module, configured to determine, before a response time of a system in response to the user operation instruction based on a sudden change condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window, whether the sudden change of the multimedia signal in the foreground window is caused by the user operation instruction; and if the mutation of the multimedia signal in the foreground window is caused by the user operation instruction, executing an operation of determining the response time of the system in response to the user operation instruction based on the mutation condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window.
Further, the abrupt change signal judging module is specifically configured to input an abrupt change signal of the multimedia signal in the foreground window and the user operation instruction into a consistency classifier trained in advance; determining whether the multimedia signal mutation in the foreground window is caused by the user operation instruction based on an output result of the consistency classifier.
Further, the training process of the consistency classifier includes:
obtaining a training sample set, the training sample set comprising: a plurality of historical operating instructions and a plurality of abrupt signals of the historical multimedia signals;
extracting the characteristics of abrupt signals of each historical multimedia signal;
counting the occurrence times of a first feature, wherein the first feature is the feature of a sudden change signal of a historical multimedia signal corresponding to a historical operation instruction;
determining the number of times of occurrence of the first feature as a standardized value corresponding to the historical operating instruction;
and inputting the historical operation instruction and the standardized value corresponding to the historical operation instruction into a preset classifier for training to obtain a consistency classifier.
Further, the preset classifier includes one or more of the following:
a Support Vector Machine (SVM) classifier, a nearest neighbor classifier, a naive Bayes classifier, a random forest classifier and a linear discriminant classifier.
Further, the multimedia signal comprises one or more of the following modalities: video signal, audio signal, somatosensory signal.
Further, extracting the characteristics of the abrupt change signals of the historical multimedia signals includes:
And when the multimedia signals comprise video signals, extracting scale invariant feature transform SIFT features corresponding to abrupt signals of the video signals.
Further, extracting the characteristics of the abrupt change signals of the historical multimedia signals includes:
and when the multimedia signals comprise audio signals, extracting the mel-frequency cepstrum coefficients corresponding to abrupt signals of the audio signals.
Further, the system delay determining module 84 is specifically configured to determine, for each mode signal, a response delay corresponding to the mode based on an input time and an output time corresponding to the mode; and fusing response delays corresponding to the multiple modes by using a signal fusion algorithm, and determining the system delay of the system.
Further, the device further comprises: and the parameter optimization module is used for optimizing the system design parameters based on the system delay.
The system delay determining device provided by the embodiment can execute the system delay determining method provided by any embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of executing the system delay determining method.
Referring now to fig. 9, a schematic diagram of an electronic device (e.g., client device or server in fig. 9) 900 suitable for use in implementing embodiments of the present disclosure is shown. Client devices in embodiments of the present disclosure may include, but are not limited to, mobile clients such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), on-board clients (e.g., car navigation clients), and the like, as well as stationary clients such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 9 is merely an example, and should not impose any limitations on the functionality and scope of use of embodiments of the present disclosure.
As shown in fig. 9, the electronic device 900 may include a processing means (e.g., a central processor, a graphics processor, etc.) 901, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 902 or a program loaded from a storage means 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data necessary for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input/output (I/O) interface 905 is also connected to the bus 904.
In general, the following devices may be connected to the I/O interface 905: input devices 906 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 907 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 908 including, for example, magnetic tape, hard disk, etc.; and a communication device 909. The communication means 909 may allow the electronic device 900 to communicate wirelessly or by wire with other devices to exchange data. While fig. 9 shows an electronic device 900 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When executed by the processing device 901, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
In some implementations, the server may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:
responding to a user operation instruction acting on a multimedia signal of a background window, and acquiring the input moment of the user operation instruction;
detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment;
Determining a response time of the user operation instruction based on a sudden change condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window;
and determining the system delay of the system according to the input time and the response time.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
According to one or more embodiments of the present disclosure, there is provided a system delay determination method, apparatus, device, and storage medium, including:
responding to a user operation instruction acting on a multimedia signal of a background window, and acquiring the input moment of the user operation instruction;
detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment;
determining a response time of the user operation instruction based on a sudden change condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window;
and determining the system delay of the system according to the input time and the response time.
According to one or more embodiments of the present disclosure, there is provided a system delay determining method, apparatus, device and storage medium, for determining a response time of a system in response to the user operation instruction based on a sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window, including:
And if the multimedia signal in the foreground window is mutated relative to the multimedia signal in the background window, determining the moment of the mutation of the multimedia signal in the foreground window as the response moment responding to the operation instruction.
According to one or more embodiments of the present disclosure, there is provided a system delay determining method, apparatus, device, and storage medium, before determining a response time of a system in response to the user operation instruction based on a sudden change of the multimedia signal in the foreground window relative to the multimedia signal in the background window, further comprising:
determining whether the multimedia signal mutation in the foreground window is caused by the user operation instruction;
and if the mutation of the multimedia signal in the foreground window is caused by the user operation instruction, executing an operation of determining the response time of the system in response to the user operation instruction based on the mutation condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window.
According to one or more embodiments of the present disclosure, there is provided a system delay determination method, apparatus, device, and storage medium, determining whether a mutation of the multimedia signal in the foreground window is caused by the user operation instruction, including:
Inputting a sudden change signal of the multimedia signal in the foreground window and the user operation instruction into a pre-trained consistency classifier;
determining whether the multimedia signal mutation in the foreground window is caused by the user operation instruction based on an output result of the consistency classifier.
According to one or more embodiments of the present disclosure, there is provided a system delay determination method, apparatus, device and storage medium, the training process of the consistency classifier includes:
obtaining a training sample set, the training sample set comprising: a plurality of historical operating instructions and a plurality of abrupt signals of the historical multimedia signals;
extracting the characteristics of abrupt signals of each historical multimedia signal;
counting the occurrence times of a first feature, wherein the first feature is the feature of a sudden change signal of a historical multimedia signal corresponding to a historical operation instruction;
determining the number of times of occurrence of the first feature as a standardized value corresponding to the historical operating instruction;
and inputting the historical operation instruction and the standardized value corresponding to the historical operation instruction into a preset classifier for training to obtain a consistency classifier.
According to one or more embodiments of the present disclosure, there is provided a system delay determination method, apparatus, device and storage medium, where the preset classifier includes one or more of the following:
a Support Vector Machine (SVM) classifier, a nearest neighbor classifier, a naive Bayes classifier, a random forest classifier and a linear discriminant classifier.
According to one or more embodiments of the present disclosure, there is provided a system delay determination method, apparatus, device and storage medium, the multimedia signal comprising one or more of the following modalities: video signal, audio signal, somatosensory signal.
According to one or more embodiments of the present disclosure, there is provided a system delay determining method, apparatus, device and storage medium for determining a system delay, extracting characteristics of abrupt signals of respective ones of the historical multimedia signals, including:
and when the multimedia signals comprise video signals, extracting scale invariant feature transform SIFT features corresponding to abrupt signals of the video signals.
According to one or more embodiments of the present disclosure, there is provided a system delay determining method, apparatus, device and storage medium for determining a system delay, extracting characteristics of abrupt signals of respective ones of the historical multimedia signals, including:
And when the multimedia signals comprise audio signals, extracting the mel-frequency cepstrum coefficients corresponding to abrupt signals of the audio signals.
According to one or more embodiments of the present disclosure, there is provided a system delay determining method, apparatus, device, and storage medium, for determining a system delay of a system according to the input time and the response time, including:
for each mode signal, determining response delay corresponding to the mode based on the input time and the output time corresponding to the mode;
and fusing response delays corresponding to the multiple modes by using a signal fusion algorithm, and determining the system delay of the system.
According to one or more embodiments of the present disclosure, there is provided a system delay determination method, apparatus, device, and storage medium, the method further comprising: and optimizing the system design parameters based on the system delay.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).
Moreover, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims.
Claims (13)
1. A system delay determination method, comprising:
responding to a user operation instruction acting on a multimedia signal of a background window, and acquiring the input moment of the user operation instruction, wherein the input moment is the moment of a client responding to the user operation instruction;
Detecting multimedia signals in the background window and the foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment;
determining a response time of the user operation instruction based on a sudden change condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window;
determining the system delay of the system according to the input time and the response time;
and if the multimedia signal in the foreground window is mutated relative to the multimedia signal in the background window, determining the moment of the mutation of the multimedia signal in the foreground window as the response moment responding to the operation instruction.
2. The method of claim 1, wherein prior to determining a response time at which a system responds to the user operation instruction based on a sudden change in the multimedia signal in the foreground window relative to the multimedia signal in the background window, further comprising:
determining whether the multimedia signal mutation in the foreground window is caused by the user operation instruction;
And if the mutation of the multimedia signal in the foreground window is caused by the user operation instruction, executing an operation of determining the response time of the system in response to the user operation instruction based on the mutation condition of the multimedia signal in the foreground window relative to the multimedia signal in the background window.
3. The method of claim 2, wherein determining whether the mutation of the multimedia signal in the foreground window is caused by the user operation instruction comprises:
inputting a sudden change signal of the multimedia signal in the foreground window and the user operation instruction into a pre-trained consistency classifier;
determining whether the multimedia signal mutation in the foreground window is caused by the user operation instruction based on an output result of the consistency classifier.
4. A method according to claim 3, wherein the training process of the consistency classifier comprises:
obtaining a training sample set, the training sample set comprising: a plurality of historical operating instructions and a plurality of abrupt signals of the historical multimedia signals;
extracting the characteristics of abrupt signals of each historical multimedia signal;
Counting the occurrence times of a first feature, wherein the first feature is the feature of a sudden change signal of a historical multimedia signal corresponding to a historical operation instruction;
determining the number of times of occurrence of the first feature as a standardized value corresponding to the historical operating instruction;
and inputting the historical operation instruction and the standardized value corresponding to the historical operation instruction into a preset classifier for training to obtain a consistency classifier.
5. The method of claim 4, wherein the pre-set classifier comprises one or more of:
a Support Vector Machine (SVM) classifier, a nearest neighbor classifier, a naive Bayes classifier, a random forest classifier and a linear discriminant classifier.
6. The method according to any of claims 1-4, wherein the multimedia signal comprises one or more of the following modalities: video signal, audio signal, somatosensory signal.
7. The method of claim 6, wherein extracting characteristics of the abrupt signal of each of the historical multimedia signals comprises:
and when the multimedia signals comprise video signals, extracting scale invariant feature transform SIFT features corresponding to abrupt signals of the video signals.
8. The method of claim 6, wherein extracting characteristics of the abrupt signal of each of the historical multimedia signals comprises:
and when the multimedia signals comprise audio signals, extracting the mel-frequency cepstrum coefficients corresponding to abrupt signals of the audio signals.
9. The method according to any of claims 7-8, wherein determining a system delay of a system from the input time instant and the response time instant comprises:
for each mode signal, determining response delay corresponding to the mode based on the input time and the output time corresponding to the mode;
and fusing response delays corresponding to the multiple modes by using a signal fusion algorithm, and determining the system delay of the system.
10. The method according to claim 1, wherein the method further comprises:
and optimizing the system design parameters based on the system delay.
11. A system delay determining device, comprising:
the input time acquisition module is used for responding to a user operation instruction acting on a multimedia signal of a background window and acquiring the input time of the user operation instruction, wherein the input time is the time when a client responds to the user operation instruction;
The multimedia signal detection module is used for detecting multimedia signals in a background window and a foreground window, wherein the background window is a first time window before the input moment, and the foreground window is a second time window after the input moment;
a response time determining module, configured to determine a response time of the user operation instruction based on a sudden change situation of the multimedia signal in the foreground window relative to the multimedia signal in the background window;
the system delay determining module is used for determining the system delay of the system according to the input time and the response time;
and the response time determining module is specifically configured to determine, as the response time for responding to the operation instruction, a time when the multimedia signal in the foreground window is mutated with respect to the multimedia signal in the background window.
12. A system delay determination apparatus, comprising:
one or more processors;
a memory for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the system latency determination method of any of claims 1-10.
13. A computer storage medium storing a computer program which when executed by a processor implements the system delay determination method of any one of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110172567.7A CN112843677B (en) | 2021-02-08 | 2021-02-08 | System delay determining method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110172567.7A CN112843677B (en) | 2021-02-08 | 2021-02-08 | System delay determining method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112843677A CN112843677A (en) | 2021-05-28 |
| CN112843677B true CN112843677B (en) | 2023-07-11 |
Family
ID=75989245
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110172567.7A Active CN112843677B (en) | 2021-02-08 | 2021-02-08 | System delay determining method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112843677B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113876437B (en) * | 2021-09-13 | 2024-02-23 | 上海微创医疗机器人（集团）股份有限公司 | Storage medium, robot system, and computer device |
| CN114422401B (en) * | 2021-12-07 | 2024-08-30 | 宁波路特斯机器人有限公司 | Remote control delay measurement method, device and medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103222262A (en) * | 2010-11-19 | 2013-07-24 | 思科技术公司 | System and method for skipping video coding in a network environment |
| JP5602963B1 (en) * | 2014-01-30 | 2014-10-08 | グリー株式会社 | GAME MANAGEMENT PROGRAM, GAME MANAGEMENT METHOD, AND GAME MANAGEMENT SYSTEM |
| CN107181906A (en) * | 2016-03-11 | 2017-09-19 | 深圳市骄阳数字图像技术有限责任公司 | Image pickup method and device |
| CN110855907A (en) * | 2019-10-19 | 2020-02-28 | 中国航空工业集团公司洛阳电光设备研究所 | Low-delay video overlay frame buffer scheduler based on prediction |
| JP6809731B1 (en) * | 2019-12-05 | 2021-01-06 | ネットスマイル株式会社 | Video signal processing system and video signal processing method |
| CN112256372A (en) * | 2020-10-20 | 2021-01-22 | 北京字跳网络技术有限公司 | Information processing method and device and electronic equipment |
Family Cites Families (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5745204B2 (en) * | 2008-07-28 | 2015-07-08 | 株式会社バンダイナムコエンターテインメント | Program, information storage medium and game machine |
| CN102710898B (en) * | 2011-08-26 | 2015-02-18 | 新奥特（北京）视频技术有限公司 | Method and system for synchronously switching foreground and background of color key device |
| US8812725B2 (en) * | 2012-05-18 | 2014-08-19 | Cisco Technology Inc. | System and method for latency reduction in a network environment |
| CN102801923B (en) * | 2012-08-30 | 2015-09-09 | 广东威创视讯科技股份有限公司 | A kind of video matrix switching Delay time defining method, equipment and system |
| US9603049B2 (en) * | 2013-07-22 | 2017-03-21 | Seven Networks, Llc | Extending delay tolerance of mobile applications for optimizing mobile traffic management |
| US9886094B2 (en) * | 2014-04-28 | 2018-02-06 | Microsoft Technology Licensing, Llc | Low-latency gesture detection |
| CN104253996B (en) * | 2014-09-18 | 2018-02-09 | 中安消技术有限公司 | The sending, receiving method and its device and Transmission system of video data |
| CN105261202B (en) * | 2015-11-05 | 2019-03-19 | 江苏省电力公司电力科学研究院 | Transmission delay measurement method based on B-spline interpolating wavelet Singularity detection |
| US10390082B2 (en) * | 2016-04-01 | 2019-08-20 | Oath Inc. | Computerized system and method for automatically detecting and rendering highlights from streaming videos |
| WO2017203096A1 (en) * | 2016-05-27 | 2017-11-30 | Picturall Oy | A computer-implemented method for reducing video latency of a computer video processing system and computer program product thereto |
| CN107222274B (en) * | 2017-07-11 | 2020-07-07 | 成都德芯数字科技股份有限公司 | Delay detection method and system |
| CN108988974B (en) * | 2018-06-19 | 2020-04-07 | 远形时空科技（北京）有限公司 | Time delay measuring method and device and system for time synchronization of electronic equipment |
| CN110795298B (en) * | 2019-11-06 | 2021-05-11 | 腾讯科技（深圳）有限公司 | Touch display delay determining method and device, storage medium and electronic device |
| CN111245680B (en) * | 2020-01-10 | 2021-10-08 | 腾讯科技（深圳）有限公司 | Method, device, system, terminal and server for detecting cloud game response delay |
| CN111367434B (en) * | 2020-03-06 | 2023-02-10 | Oppo广东移动通信有限公司 | Touch delay detection method and device, electronic equipment and storage medium |
| CN112055237B (en) * | 2020-08-31 | 2022-07-19 | 北京爱奇艺科技有限公司 | Method, system, apparatus, device and storage medium for determining screen-to-screen delay |
-
2021
- 2021-02-08 CN CN202110172567.7A patent/CN112843677B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103222262A (en) * | 2010-11-19 | 2013-07-24 | 思科技术公司 | System and method for skipping video coding in a network environment |
| JP5602963B1 (en) * | 2014-01-30 | 2014-10-08 | グリー株式会社 | GAME MANAGEMENT PROGRAM, GAME MANAGEMENT METHOD, AND GAME MANAGEMENT SYSTEM |
| CN107181906A (en) * | 2016-03-11 | 2017-09-19 | 深圳市骄阳数字图像技术有限责任公司 | Image pickup method and device |
| CN110855907A (en) * | 2019-10-19 | 2020-02-28 | 中国航空工业集团公司洛阳电光设备研究所 | Low-delay video overlay frame buffer scheduler based on prediction |
| JP6809731B1 (en) * | 2019-12-05 | 2021-01-06 | ネットスマイル株式会社 | Video signal processing system and video signal processing method |
| CN112256372A (en) * | 2020-10-20 | 2021-01-22 | 北京字跳网络技术有限公司 | Information processing method and device and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 基于光学相控阵器研究中的特性与实现;蒋宏彬;刘广智;杨筱舟;;绵阳师范学院学报(第08期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112843677A (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111314733A (en) | Method and apparatus for evaluating video sharpness | |
| CN112488783B (en) | Image acquisition method and device and electronic equipment | |
| CN112153460B (en) | Video dubbing method and device, electronic equipment and storage medium | |
| CN111104980B (en) | Method, device, equipment and storage medium for determining classification result | |
| CN112843677B (en) | System delay determining method, device, equipment and storage medium | |
| CN110287810B (en) | Vehicle door motion detection method, device and computer readable storage medium | |
| US10129504B2 (en) | Method and system for measuring quality of video call | |
| CN110287816B (en) | Vehicle door motion detection method, device and computer readable storage medium | |
| CN112637631B (en) | Code rate determining method and device, electronic equipment and storage medium | |
| CN114339294B (en) | Method, device and equipment for confirming network jitter and storage medium | |
| CN111416996B (en) | Multimedia file detection method, multimedia file playing device, multimedia file equipment and storage medium | |
| CN111432245A (en) | Multimedia information playing control method, device, equipment and storage medium | |
| CN114154068A (en) | Media content recommendation method and device, electronic equipment and storage medium | |
| CN113144620B (en) | Method, device, platform, readable medium and equipment for detecting frame synchronous game | |
| CN112990176A (en) | Writing quality evaluation method and device and electronic equipment | |
| CN112507884A (en) | Live content detection method and device, readable medium and electronic equipment | |
| CN112000251A (en) | Method, apparatus, electronic device and computer readable medium for playing video | |
| CN114528433B (en) | Template selection method and device, electronic equipment and storage medium | |
| CN110177270A (en) | Video head frame test method and device | |
| CN112418233B (en) | Image processing method and device, readable medium and electronic equipment | |
| CN113240108B (en) | Model training method and device and electronic equipment | |
| CN115272760A (en) | Small sample smoke image fine classification method suitable for forest fire smoke detection | |
| CN111625745B (en) | Recommendation method, recommendation device, electronic equipment and computer readable medium | |
| CN113515329B (en) | Special effect attribute setting method and device | |
| CN114298403A (en) | Method and device for predicting attention degree of work |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |