CN113589919A

CN113589919A - Image processing method and device

Info

Publication number: CN113589919A
Application number: CN202010366426.4A
Authority: CN
Inventors: 黄茵; 熊华平
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-04-30
Filing date: 2020-04-30
Publication date: 2021-11-02
Also published as: WO2021218683A1

Abstract

The application discloses an image processing method and device, and belongs to the technical field of network transmission. The method is applied to the network equipment and comprises the following steps: first pose information indicating a pose and a position of the target object at a first time is acquired, and historical pose information indicating a pose and a position of the target object at historical times is acquired. And then, compensating the posture and the position indicated by the first posture information according to the time delay information and the historical posture information, thereby obtaining second posture information. Then, a target instruction matched with the second pose information can be obtained, and the target instruction is used for obtaining an image to be displayed. According to the method and the device, the instruction error caused by time delay is calibrated, and the accuracy of the action response of the object is improved. Not only the self-adaptability is stronger, can handle moreover and obtain the higher image of precision, promoted user's operation experience.

Description

Image processing method and device

Technical Field

The present application relates to the field of network transmission technologies, and in particular, to a method and an apparatus for image processing.

Background

With the development of network transmission technology, cloud virtual reality (CloudVR) based on network transmission is gradually applied to the lives of people. In the application process of CloudVR, the first device often transmits the instruction to the second device through the network, the second device performs image processing according to the received instruction, and returns the obtained image to the first device, so that the first device displays the image. It can be seen that the instruction sent by the first device is the key to performing image processing.

In the related art, CloudVR is applied to a human-computer interaction scene. Taking a man-machine interaction game as an example, the second device firstly obtains a first image according to game logic processing, the first device displays the first image, and a user can act on a game object in the first image. The first device generates an action instruction according to the detected user action, and transmits the action instruction to the second device through the network. And the second equipment obtains a second image according to the received action command and the game logic processing, and returns the second image to the first equipment. And then, the first equipment displays the received second image, thereby completing the game process of man-machine interaction.

However, due to the limitations of the network transmission rate and transmission distance, there is inevitably a delay in transmitting the motion command to the second device through the network, that is, the motion command received by the second device is a command that lags behind in time. Because the second device runs the game logic uninterruptedly, when the second device delays to receive the action command, the game object deviates, so that the user action and the game object in the second image rendered by the second device according to the action command and the game logic are not matched with each other, the image accuracy is low, and the display effect is poor. Therefore, it is desirable to provide an image processing method to improve the accuracy of an image and avoid affecting the user experience.

Disclosure of Invention

The embodiment of the application provides an image processing method and device, which aim to solve the problem of low image precision provided by the related technology, and the technical scheme is as follows:

in a first aspect, a method for image processing is provided, where the method is applied to a network device, and the method includes: first pose information indicating a pose and a position of the target object at a first time is acquired, and historical pose information indicating a pose and a position of the target object at historical times is acquired. And then, compensating the posture and the position indicated by the first posture information according to the time delay information and the historical posture information, thereby obtaining second posture information. Then, a target instruction matched with the second pose information can be obtained, and the target instruction is used for obtaining an image to be displayed.

According to the method and the device, the first position and posture information is adjusted through the time delay information and the historical position and posture information, the hysteresis of the target instruction obtained based on the first position and posture information is improved, the instruction error caused by time delay is calibrated, and after the target instruction reaches the second equipment in a delayed mode due to the time delay, the action described by the second position and posture information carried by the target instruction can still be matched with the logic of the operation of the second equipment, so that the action response accuracy is improved. The scheme has strong self-adaptability, can generate images or video streams with high precision, and improves the operation experience of users.

In an exemplary embodiment, the compensating the pose and the position indicated by the first pose information according to the time delay information and the historical pose information to obtain the second pose information includes: and determining pose change information according to the time delay information and the historical pose information, wherein the pose change information comprises pose change amount and pose change direction. And based on the pose change direction, compensating the pose and the position indicated by the first pose information according to the pose change amount, and obtaining second pose information according to the compensated pose and position. According to the embodiment, the attitude and the position indicated by the first attitude information are compensated under the condition that the time delay information is larger, so that the waste of processing resources is avoided.

In an exemplary embodiment, determining pose change information from the time delay information and the historical pose information includes: and determining the pose change direction and the pose change rate according to the historical pose information. And determining the pose variation according to the time delay information and the pose variation rate.

In an exemplary embodiment, after obtaining the first pose information of the target object, the method further comprises: acquiring time delay information; and responding to the time delay information being larger than the time delay threshold value, and executing the step of compensating the posture and the position indicated by the first posture information according to the time delay information and the historical posture information to obtain second posture information.

In an exemplary embodiment, the network device is a terminal, and after acquiring the target instruction matched with the second position information, the method further includes: and sending a target instruction to the server, receiving an image to be displayed returned by the server according to the target instruction, and displaying the image.

In an exemplary embodiment, the network device is a server, and after acquiring the target instruction matched with the second position information, the method further includes: and acquiring an image to be displayed according to the target instruction, and sending the image to be displayed to the terminal for displaying.

In a second aspect, there is provided an image processing apparatus comprising: the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring first attitude information of a target object, and the first attitude information is used for indicating the attitude and the position of the target object at a first moment;

the acquisition module is further configured to acquire historical pose information of the target object, where the historical pose information is used to indicate a pose and a position of the target object at a historical time;

the compensation module is used for compensating the posture and the position indicated by the first pose information according to time delay information and the historical pose information to obtain second pose information, the time delay information is obtained based on a time interval between the acquisition time of reference pose information and the display time of a reference image, the reference image is an image corresponding to the reference pose information, and the acquisition time of the reference pose information is earlier than the first time;

the acquisition module is further configured to acquire a target instruction matched with the second posture information, where the target instruction is used to acquire an image to be displayed.

In an exemplary embodiment, the compensation module is configured to determine pose change information according to the time delay information and the historical pose information, where the pose change information includes a pose change amount and a pose change direction; and compensating the posture and the position indicated by the first posture information according to the posture change amount based on the posture change direction, and obtaining second posture information according to the compensated posture and position.

In an exemplary embodiment, the compensation module is configured to determine the pose change direction and the pose change rate according to the historical pose information; and determining the pose variation according to the time delay information and the pose variation rate.

In an exemplary embodiment, the obtaining module is further configured to obtain the time delay information; and the compensation module is further used for responding to the time delay information being larger than a time delay threshold value, and executing the step of compensating the posture and the position indicated by the first pose information according to the time delay information and the historical pose information to obtain second pose information.

In an exemplary embodiment, the apparatus further comprises: and the display module is used for sending the target instruction to a server, receiving the image to be displayed returned by the server according to the target instruction and displaying the image.

In an exemplary embodiment, the apparatus further comprises: and the sending module is used for obtaining the image to be displayed according to the target instruction and sending the image to be displayed to a terminal for displaying.

In a third aspect, a communication apparatus is provided, the apparatus comprising: a transceiver, a memory, and a processor. Wherein the transceiver, the memory and the processor are in communication with each other via an internal connection path, the memory is configured to store instructions, and the processor is configured to execute the instructions stored by the memory to control the transceiver to receive signals and control the transceiver to transmit signals, and when the processor executes the instructions stored by the memory, to cause the processor to perform the method provided by the first aspect or any one of the exemplary embodiments of the first aspect.

Optionally, there are one or more processors and one or more memories.

Alternatively, the memory may be integrated with the processor, or provided separately from the processor.

In a specific implementation process, the memory may be a non-transient memory, such as a Read Only Memory (ROM), which may be integrated on the same chip as the processor, or may be separately disposed on different chips.

In a fourth aspect, there is provided a computer program (product) comprising: computer program code which, when run by a computer, causes the computer to perform the method of the above-mentioned aspects.

In a fifth aspect, a readable storage medium is provided, which stores a program or instructions, which when executed on a computer, performs the method of the above aspects.

In a sixth aspect, a chip is provided, which includes a processor for calling up and executing instructions stored in a memory from the memory, so that a communication device in which the chip is installed executes the method in the above aspects.

In a seventh aspect, a chip is provided, which includes: the system comprises an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected through an internal connection path, the processor is used for executing codes in the memory, and when the codes are executed, the processor is used for executing the method in the aspects.

Drawings

FIG. 1 is a flowchart of a method for image processing according to an embodiment of the present disclosure;

FIG. 2 is a flowchart of a method for image processing according to an embodiment of the present disclosure;

FIG. 3 is a schematic flow chart of image processing provided in an embodiment of the present application;

FIG. 4 is a schematic flowchart of image processing provided in an embodiment of the present application;

fig. 5 is a schematic diagram of pose information provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of compensating for first pose information according to an embodiment of the present disclosure;

FIG. 7 is a flowchart of a method for image processing according to an embodiment of the present disclosure;

FIG. 8 is a schematic flowchart of image processing provided in an embodiment of the present application;

fig. 9 is a block diagram of an apparatus for image processing according to an embodiment of the present application.

Detailed Description

The terminology used in the description of the embodiments section of the present application is for the purpose of describing particular embodiments of the present application only and is not intended to be limiting of the present application.

Virtual Reality (VR) technology is a technology that combines virtual content generated by a computer with a real environment, and is currently applied to a plurality of fields such as video, games, education, and medical care, and has a huge market space. The cloud VR introduces the concept and technology of cloud computing into the VR technology, so that the processes of computing, rendering and the like which are originally required to be executed by the first equipment are executed by the second equipment, and the first equipment only needs to have decoding capability, so that the configuration requirement on the first equipment is reduced. Therefore, CloudVR can reduce the consumption cost of users, is beneficial to quickening the popularization of VR technology, and has attracted much attention in recent years.

In the application process of CloudVR, the first device often transmits an instruction to the second device through the network, the second device processes the instruction according to the received instruction to obtain one or more frames of images, and returns the one or more frames of images or a video stream formed by the multiple frames of images to the first device so that the first device can display the images. However, there is inevitably a delay in transmitting the command to the second device via the network, and thus the command received by the second device is a delayed command in time, and the image and video streams generated by the second device according to such delayed command are less accurate. Therefore, the above application process needs to be adjusted so as to obtain images and video streams with higher precision.

An embodiment of the present application provides an image processing system, and referring to fig. 1, the system includes: a terminal 101, a network node 102 and a server 103, wherein the terminal 101 is connected to the server 103 through the network node 102.

The terminal 101 includes, but is not limited to, a Virtual Reality (VR) helmet and a handle, where the VR helmet is used for browsing images and video streams of a target object, and the handle is used for operating the target object. The VR helmet may be a VR all-in-one machine, may be a split VR including a mobile phone and a VR head display, and may also be a Personal Computer (PC) VR. One or more components such as a button, a trigger and the like can be arranged on the handle according to requirements.

The network node 102 may be an Optical Network Terminal (ONT), a switch, a router, or a wireless Access Point (AP), which can access the terminal 101 to a communication network. The communication network may be, for example, the internet, an intranet of an enterprise organization, or a local network within a certain area.

The server 103 may be a Personal Computer (PC) local to the terminal, may be a single independent server, may be a server cluster including a plurality of servers, may be a virtual machine or a computer provided with a cloud service, and the like.

The terminal 101 is configured to acquire pose information of the target object, where the pose information describes an action performed by the target object. The pose information is used for indicating the pose and the position of the target object at a certain moment, the pose describes the rotation action performed by the target object, and the position describes the static action or the translation action performed by the target object. Illustratively, the motion performed by the target object includes a motion resulting from movement of a body part of the target object and an operation performed by the target object with respect to the handle. The motion of the body part of the target object is, for example, head rotation, body movement, and waving of the hand holding the handle of the target object. The operation of the target object on the handle, such as pressing a button provided on the handle, pulling a trigger provided on the handle, or the like, for the target object.

It should be noted that the operation of the target object on the handle can be directly captured by the sensor built in the handle, and for the motion generated by the motion of the body part of the target object, any one of the inside-out (inside-outside) mode and the outside-in (outside-inside) mode needs to be used for capturing, so as to realize the acquisition of the pose information. In response to the capture by the inside-out method, the change of the target object relative to the environment of the target object is detected through built-in equipment of the VR helmet and the handle, and the motion generated by the motion of the body part of the target object is calculated according to the change of the target object relative to the environment of the target object. The built-in devices include, but are not limited to, sensors, cameras, and the like. For example, the VR headset can detect the movement of the VR headset through the built-in device, and the head movement of the target object can be calculated through the movement of the VR headset. The handle can detect the movement of the handle by the built-in device, and the swing of the hand of the handle held by the target object can be estimated by the movement of the handle. In response to capturing by the outside-in method, it is necessary to track the movement of the target body part by means of an external tracking device, such as a positioning base point/base station, an optical camera, or the like, in addition to the terminal.

The terminal 101 is further configured to access a communication network through the network node 102, and send the collected pose information to the server 103. Accordingly, the server 103 is configured to obtain an instruction matching the received first pose information, perform calculation and rendering according to the instruction matching the pose information, and generate an image or video stream corresponding to the pose information. The server 103 encodes the image or video stream, and then returns the encoded image or video stream to the terminal 101 through the communication network. The terminal 101 is then further configured to decode the encoded image or video stream, project the image or video stream through the VR headset, and display the image or video stream, that is, the target object can browse the image or video stream through the VR headset.

In an exemplary embodiment, either the terminal 101 or the network node 102 and the server 103 form a latency detection subsystem for detecting latency information. That is, in this embodiment, the terminal 101 and the server 103 may form a delay detection subsystem, or the network node 102 and the server 103 may form a delay detection subsystem.

The time delay information is obtained based on a time interval between the acquisition time of the pose information and the display time of the image corresponding to the pose information. Since the terminal 101 can be used to collect pose information, the collection time of the pose information is the time when the terminal 101 collects the pose information. For example, the present embodiment may regard the time when the terminal 101 collects the pose information and the time when the pose information is generated (i.e., the object performs the motion) as the same time, or the time interval between the collection time of the pose information and the generation time of the pose information may be ignored.

In addition, as described above, after the terminal 101 acquires the pose information, the pose information is sent to the server 103, the server 103 performs processing based on the received pose information, so as to return an image or video stream corresponding to the pose information, and a time when the terminal 101 displays the image or video stream returned by the server 103 is a display time of the image corresponding to the pose information. Because the processes of information transmission, processing and the like are involved between the acquisition time of the pose information and the display time of the image corresponding to the pose information, and the processes of information transmission, processing and the like need to consume time, a time interval exists between the acquisition time and the display time, and the time interval can be used as the time delay information.

When processing an image corresponding to the pose information of the target object at the current time, the time delay information may be determined based on the pose information of the target object at the historical time, or may be determined based on the pose information of an object other than the target object at the historical time. Of course, in this embodiment, the time delay information may be determined not only according to the pose information based on the historical time, but also according to the pose information of the target object or other objects at the current time in real time, and the time at which the pose information used for determining the time delay information is located is not limited in this embodiment.

The time delay detection subsystem comprises a client and a server, wherein the client is used for triggering the detection of the time delay information, and the server is used for completing the detection of the time delay information in cooperation with the client. In the latency detection subsystem including the terminal 101 and the server 103, the terminal 101 may be a client and the server 103 may be a server, or the terminal 101 may be a server and the server 103 may be a client. In the delay detection subsystem including the network node 101 and the server, the network node 102 may be a client and the server 103 may be a server, or the network node 102 may be a server and the server 103 may be a client.

It is understood that the delay information detected by the delay detection subsystem may be network delay generated by the pose information transmitted between the terminal 101 and the server 103, or may be other delay generated during the operation of the terminal 101 or the server 103. Other delays include, but are not limited to, queuing delay of the pose information waiting to be sent in a sending queue of the terminal 101 when the terminal 101 sends the pose information to the server 103, and sending delay generated when the terminal 101 sends the pose information from the sending queue to the network.

Next, the following describes the manner of detecting and obtaining the network delay and other delays:

network delay: firstly, a client in a time delay detection subsystem sends a detection signal containing sending time to a server so as to trigger detection of time delay information. Then, the server can calculate the difference between the time of receiving the detection signal and the time before the sending time contained in the detection signal. Since the difference is the time required for the detection signal to be transmitted between the terminal 101 and the server 103 in the communication network, the difference can be used as the network delay.

It should be noted that, since the network delay to be detected is the network delay between the terminal 101 and the server 103, in the case that the client or the server in the delay detection subsystem is the network node 102, the location of the network node 102 in the communication network needs to be close to the terminal. Thereby, the difference between the time length of the transmission of the detection signal between the network node 102 and the server 103 and the time length of the transmission between the terminal 101 and the server 103 can be made to approach zero, so that the network delay between the network node 102 and the server 103 can be substituted for the network delay between the terminal 101 and the server 103.

Other time delays: for example, the present embodiment may employ a software probe to perform other time delay detection. The software probe can add timestamps to services executed by the equipment, and other time delays can be calculated by calculating the timestamp difference between different services. Taking the queuing delay as an example, in the terminal 101, a first timestamp can be added to record the current time when the terminal 101 executes a service of "adding the collected pose information into the sending queue", and a second timestamp can be added to record the current time when the terminal 101 executes a service of "about to send the pose information in the sending queue". And then, calculating a timestamp difference value between the first timestamp and the second timestamp to obtain the queuing time delay of the position and pose information waiting for sending in the sending queue. For different time delays other than the queuing time delay, no further description is given here.

In addition to the time delay detection subsystem, the system provided in this embodiment further includes a computation node, where the computation node is configured to obtain a pose and a position indicated by the pose information of the target object according to the time delay information and the historical pose information, and compensate the pose and the position indicated by the pose information of the target object to obtain second pose information, so as to further obtain a target instruction, which is matched with the second pose information and is used for obtaining an image to be displayed. And the historical pose information is used for indicating the pose and the position of the target object at the historical moment. In an exemplary embodiment, the computing node may be a terminal 101 or a server 103. Next, a description will be given of a process of acquiring the history pose information, and two cases where the calculation nodes are the terminal 101 and the server 103 will be described:

the historical pose information is information acquired by the terminal 101. After acquiring the historical pose information, the terminal 101 may store the historical pose information locally in the terminal 101, send the historical pose information to the server 103 for storage, and send the historical pose information to a storage device other than the server 103 for storage. Therefore, regardless of whether the computing node is the terminal 101 or the server 103, the history pose information can be acquired from the device in which the history pose information is stored.

As described above, the pose information of the target object is used to indicate the pose and position of the target object at the current time, and the historical pose information is used to indicate the pose and position of the target object at the historical time. The "current time", and "historical time" may refer to one instant time, or may refer to a period of time composed of a plurality of instant times.

Taking the target object to swing manually as an example, when a hand swinging action is performed, the hand of the target object moves from one place to another place, and the moving process is not always completed within one instant time, but needs to occupy a period of time consisting of a plurality of instant times. Therefore, when the current time and the historical time represent one instantaneous time, the motion described by the posture information is a motion of the hand at a certain point that the hand passes through while moving. In this case, the historical posture information and the posture information of the target object may both be motions at a certain point in the process of waving the hand, except that the motion described by the historical posture information occurs before the motion described by the posture information of the target object.

Or, in the case where the current time and the historical time represent a period of time, the gesture information of the target object describes a complete motion of moving the hand of the target object from one place to another. Thus, in this case, the historical pose information describes one or more other complete actions that occurred prior to the complete action of the hand swing of the target object. The time indicated by the time is not limited in this embodiment, and the two cases can be selected as needed. Next, a procedure of performing calculation by the terminal 101 and the server 103, respectively, will be described:

in the first case: the computing node is terminal 101. After the terminal 101 locally acquires the pose information of the target object, when the terminal 101 acquires the delay information, in response to that the terminal 101 is a server side in the delay detection subsystem, the terminal 101 may receive a detection signal including transmission time sent by a client side, so as to obtain the delay information through calculation. In response to the terminal 101 being a client in the delay detection subsystem, the terminal 101 may obtain the delay information from a server in the delay detection subsystem. In addition, in the case where the terminal 101 does not constitute the delay detection subsystem, the terminal 101 may also acquire the delay information from any one of a network node and a server in the delay detection subsystem.

In addition, when the terminal 101 acquires the historical pose information, the terminal 101 may acquire the historical pose information from a local server 103 or other device storing the historical pose information. Then, the terminal can compensate the pose and the state indicated by the acquired pose information according to the acquired time delay information and the historical pose information to obtain new pose information, so that a target instruction matched with the new pose information is determined.

After the terminal 101 obtains the target instruction, the terminal 101 further needs to send the target instruction to the server 103, so that the server 103 performs calculation and rendering according to the target instruction, and encodes and returns the rendered image or video stream to the terminal 101. The terminal 101 may then display the received image or video stream.

In the second case: the computing node is a server 103. The pose information of the target object is still collected by the terminal, so that the server 103 needs to receive the pose information sent by the terminal 101, thereby achieving the acquisition of the pose information. When the delay information is obtained, in response to that the server 103 is a server side in the delay detection subsystem, the server 103 may receive a detection signal including sending time sent by a client side, so as to obtain the delay information through calculation. In response to the server 103 being a client in the latency detection subsystem, the server 103 may obtain latency information from a server in the latency detection subsystem. As for the historical motion information of the target object, as is understood from the description in the above first case, the server 103 may acquire the historical motion information from the terminal 101, locally or other device storing the historical pose information.

After compensating the pose and position indicated by the pose information through the time delay information and the historical pose information to obtain new pose information, the server 103 may obtain a target instruction matched with the new pose information, thereby directly performing calculation and rendering according to the target instruction to obtain an image or video stream, and pushing the image or video stream to the terminal 101. It should be noted that the server 103 may invoke one or more wayside devices to perform the detection of the time delay information and the compensation of the pose information, or may directly perform the detection and the compensation without invoking the wayside devices, and the embodiment does not limit whether the server 103 invokes the wayside devices.

Based on the above description, the terminal 101, the network node 102, and the server 103 shown in fig. 1 can constitute four systems as follows. It is understood that the following four systems are merely examples. In practical applications, the system provided in this embodiment is not limited to the following four exemplary systems, and may also include other different systems obtained by combining the terminal 101, the network node 102, and the server 103:

the first system: the terminal 101 and the server 103 are used for forming a time delay detection subsystem to detect time delay, and the terminal 101 is used for performing compensation calculation of pose information. Considering that the latency detection subsystem includes a client and a server, the first system can be further divided into two combinations in table 1:

TABLE 1

Time delay detection subsystem client	Service terminal of time delay detection subsystem	Compensation calculation of pose information
			Terminal device	Server	Terminal device
Server	Terminal device	Terminal device

The second system: the terminal 101 and the server 103 are used for forming a time delay detection subsystem, and the server 103 is used for performing compensation calculation of pose information. The second system can be divided into two combinations as shown in table 2:

TABLE 2

Time delay detection subsystem client	Service terminal of time delay detection subsystem	Compensation calculation of pose information
			Terminal device	Server	Server
Server	Terminal device	Server

The third system: the network node 102 and the server 103 form a time delay detection subsystem, and the terminal 101 is used for performing compensation calculation of pose information. The third system can be divided into two combinations as shown in table 3:

TABLE 3

Time delay detection subsystem client	Service terminal of time delay detection subsystem	Compensation calculation of pose information
			Network node	Server	Terminal device
Server	Network node	Terminal device

A fourth system: the network node 102 and the server 103 form a time delay detection subsystem, and the server 103 is used for performing compensation calculation of pose information. The fourth system can be divided into two combinations as shown in table 4:

TABLE 4

Time delay detection subsystem client	Service terminal of time delay detection subsystem	Compensation calculation of pose information
			Network node	Server	Server
Server	Network node	Server

Based on the image processing system shown in fig. 1, the present embodiment further provides an image processing method, which is applicable to a network device, where the network device may be a terminal shown in fig. 1 or a server shown in fig. 1. Referring to fig. 2, in response to the network device being a terminal, the method includes:

step 201, first pose information of the target object is obtained, and the first pose information is used for indicating the pose and the position of the target object at a first time.

The first position and orientation information describes the action of the target object at the first moment by indicating the position and the orientation of the target object at the first moment. The motion performed by the target object includes a motion generated by a body part of the target object moving and an operation performed by the target object on a VR device such as a handle. The motion of the body part of the target object is, for example, head rotation, body movement, and waving of the hand holding the handle of the target object, and the like, and the operation of the target object on the VR device is, for example, the target object pressing a button provided on the VR device, and the like.

It should be noted that the motion performed by the target object can be categorized into three different types, including a stationary motion, a translational motion, and a rotational motion. For example, a hand swing of a target object may be considered a translational motion and a head turn of the target object may be considered a rotational motion. For example, the first pose information may include one or both of first pose information describing a rotational motion performed by the target object and first position information describing a stationary motion and a translational motion performed by the target object.

The space in which the target object is located can be represented by three-dimensional coordinates including two perpendicular X, Y and the Z-axis. When any body part of the target object is in a static state and in a different state, the any body part is fixed at a position, and the first position information can be represented by a coordinate point in a three-dimensional coordinate system. When any body part of the target object is subjected to translational change, the position of the any body part is continuously changed, and the first position information comprises a plurality of coordinate points in a three-dimensional coordinate system. It is to be understood that the translational changes can be viewed as translational changes along at least one of the X, Y, and Z axes in a three-dimensional coordinate system. When the position of any body part of the target object is unchanged but the any body part rotates, the first posture information comprises rotation angles in all directions at one coordinate point. It is to be understood that the rotation process can be regarded as rotation around at least one of the X-axis, the Y-axis and the Z-axis in the three-dimensional coordinate system.

Step 202, obtaining historical pose information of the target object, wherein the historical pose information is used for indicating the posture and the position of the target object at the historical moment.

The history time is any time earlier than the first time. Thus, the action described by the historical pose information is an action that has been completed by the target object before the action described by the first pose information. For example, the first time and the historical time may represent one instant time, or may represent a period of time composed of a plurality of instant times.

Taking the target object to swing manually as an example, when a hand swinging action is performed, the hand of the target object moves from one place to another place, and the moving process is not always completed within one instant time, but needs to be completed within a period of time consisting of a plurality of instant times. Therefore, when the first time and the historical time represent one instantaneous time, the motion described by the first posture information and the historical posture information is the motion of the hand at a certain point passing through the hand during the movement. Assuming that the hand of the target object passes through the first point first and then the second point in the moving process, the historical pose information describes the motion at the first point and the first pose information describes the motion at the second point because the motion described by the historical pose information is completed earlier than the motion described by the first pose information.

Or, in the case that the current time and the historical time represent a period of time, the first posture information describes a complete motion of moving the hand of the target object from one place to another place. Thus, the historical pose information describes one or more other complete motions that occurred prior to the complete motion of the hand swing of the target object. The embodiment is not limited to the above two cases, and the selection can be performed according to the action type of the target object and the actual requirement.

And 203, compensating the posture and the position indicated by the first pose information according to the time delay information and the historical pose information to obtain second pose information, wherein the time delay information is obtained based on the time interval between the acquisition time of the reference pose information and the display time of a reference image, the reference image is an image corresponding to the reference pose information, and the acquisition time of the reference pose information is earlier than the first time.

The reference pose information may be used to indicate a pose and a position of the target object at a time earlier than the first time, or the reference pose information may also be used to indicate a pose and a pose of another object other than the target object at a time earlier than the first time. It should be noted that, in this embodiment, the acquisition time of the reference pose information and the generation time of the reference pose information may be regarded as the same time, or a time interval between the acquisition time and the generation time may be ignored.

After the reference pose information is acquired at a certain moment earlier than the first moment, the terminal acquires an instruction matched with the reference pose information (or the compensated reference pose information), the instruction is sent to the server, the server acquires a reference image (or a video stream) corresponding to the reference pose information based on the instruction, and the terminal can receive and display the reference image returned by the server. And the time when the terminal displays the reference image is the display time of the reference image. Because at least the processes of acquiring the instruction, transmitting the instruction, generating the reference image, transmitting the reference image and the like are involved between the acquisition time and the display time, and the compensation process of the reference pose information can also be involved, a time interval exists between the acquisition time and the display time, and the time interval can be used as time delay information for compensating the pose and the position indicated by the first pose information.

Based on the above description, when compensating for the posture and the position indicated by the first posture information, the time delay information may be determined from the reference posture information preceding the first posture information. For example, in this embodiment, the time delay information may be calculated after the acquisition time of the reference pose information and the display time of the reference image are acquired, and the time delay information is stored. And then, when the attitude and the position indicated by the first attitude information need to be compensated, directly reading the stored time delay information so as to complete the compensation process.

Of course, in addition to the above case, the embodiment may also detect the time delay information in real time when the posture and the position indicated by the first posture information need to be compensated. The delay information may be a network delay between the terminal and the server, or an equipment delay generated in the operation process of the terminal and the server. Illustratively, the latency information between the terminal and the server is determined by a latency detection subsystem. Referring to fig. 3, the latency detection subsystem may be formed by a terminal and a server. Alternatively, referring to fig. 4, the latency detection subsystem may also be formed by a network node and a server.

For the case that the terminal and the server are used as the delay detection subsystem shown in fig. 3, in response to that the terminal is a delay detection client and the server is a server, the terminal sends a detection signal to the server, and the server calculates a difference between a sending time of the detection signal sent by the terminal and a receiving time of the local reception detection signal, and uses the difference as delay information between the terminal and the server. And then, the server can send the calculated delay information to the terminal, so that the terminal can acquire the delay information.

In response to the situation that the terminal is a time delay detection server and the server is a client, the terminal receives a detection signal sent by the server, and the terminal calculates a difference between the sending time of the detection signal sent by the server and the receiving time of the detection signal locally received. After the terminal calculates the difference, the difference can be used as the time delay information, thereby realizing the acquisition of the time delay information.

For the case that the network node and the server shown in fig. 4 are used as the delay detection subsystem, the network node or the server calculates the difference between the sending time and the receiving time of the detection signal, so as to obtain the delay information. And then, the network node or the server which obtains the delay information through calculation can directly send the delay information to the terminal, so that the terminal can conveniently obtain the delay information. Or, the network node or the server that obtains the delay information through calculation may also wait for an acquisition request sent by the terminal, and then return the delay information to the terminal according to the acquisition request, so that the terminal can also obtain the delay information.

After the time delay information is obtained, the terminal may compensate the pose and the position indicated by the first pose information according to the time delay information and the historical pose information obtained in step 202, so as to avoid that a command generated based on the first pose information lags in time, so that a subsequent image processing process is affected, and the accuracy of the generated image is low.

Illustratively, the compensating the pose and the position indicated by the first pose information according to the time delay information and the historical pose information to obtain the second pose information includes the following steps 2031 and 2032:

step 2031, determining pose change information according to the time delay information and the historical pose information, wherein the pose change information comprises pose change amount and pose change direction.

In an exemplary embodiment, determining pose change information from the time delay information and the historical pose information includes: and determining the pose change direction and the pose change rate according to the historical pose information, and determining the pose change direction according to the time delay information and the pose change rate. The embodiment can determine the action trend of the target object according to the historical pose information of the target object, so that the pose change direction is determined according to the action trend. Illustratively, in response to the action trend determined from the historical pose information being in a straight line shape, an arc shape, or the like in a certain direction, the direction of the pose and the position indicated by the historical pose information may be taken as the pose change direction. In response to the action trend determined from the historical pose information being wave, zigzag, or the like whose direction is constantly changing or even turning, the pose change direction may be the same as or opposite to the direction of the pose and position indicated by the historical pose information, or may deviate from the direction of the pose and position indicated by the historical pose information. For the latter case, for example, a direction change rule of the action trend may be induced according to at least three pieces of historical pose information, for example, a period of direction change or turning is generated, and then the pose change direction is determined by combining the direction change rule.

Taking fig. 5 as an example, the process of determining the pose change direction is described: t1 and t2 are two adjacent history times, and t1 is a time earlier than t 2. The coordinates of the position a where the target object is located at the time t1 in the three-dimensional coordinate system are (x1, y1, z1), and the coordinates of the position B where the target object is located at the time t2 in the three-dimensional coordinate system are (x2, y2, z 2). In response to the action trend being a linear, arc or other trajectory along a certain direction, the direction pointing from the position a to the position B may be directly taken as the pose change direction. In response to the action trend is a sawtooth-shaped track, a wave-shaped track and the like which are changed constantly, other historical pose information or historical pose information earlier than t1 is needed to be obtained, so that the direction change rule of the action trend is determined through the historical pose information, and the pose change direction is determined according to the direction change rule. In the latter case, the posture change direction may be a direction in which the position a points to the position B, may be a direction in which the position B points to the position a, or may be a direction in which the position B points to an arbitrary position other than the position a.

When the pose change rate is determined according to the historical pose information corresponding to the time t1 and the time t2, the determination can be performed according to the following formula:

the pose change rate is [ (x2, y2, z2) - (x1, y1, z1) ]/(t2-t 1).

After the pose change rate is determined, the product of the time delay information and the pose change rate is calculated, and the change amount of the pose in the time length of the time delay can be obtained. That is, the pose change amount can be calculated according to the following formula:

the pose variation is [ (x2, y2, z2) - (x1, y1, z1) ] × (t3-t2)/(t2-t1), wherein (t3-t2) is the time delay information.

And 2032, compensating the posture and the position indicated by the first posture information according to the posture change amount based on the posture change direction, and obtaining second posture information according to the compensated posture and the compensated position.

For example, in the pose change direction, the first pose information and the pose change amount information are summed, and the sum of the first pose information and the pose change amount is used as the second pose information, so that the first pose information is compensated, and the second pose information is obtained. Based on the above description, the second posture information can be calculated according to the following formula:

wherein, (x3, y3, z3) is the second attitude information.

As described above, the purpose of compensating the first posture information is to: the method improves the hysteresis of the subsequently generated instruction, and avoids the influence on the use experience of the user due to the fact that the accuracy of the subsequently generated image or video stream is reduced by the instruction hysteresis. However, if the lag of the command is weak, the accuracy of the image or video stream is not affected even if the command has a lag. Therefore, if the first pose information is compensated even when the command lag is weak and the accuracy of the image or video stream is not affected, the processing resources of the terminal, the network node, and the server are wasted.

Therefore, in an exemplary embodiment, after acquiring the first pose information of the target object, the method further includes: and acquiring time delay information, responding to the time delay information being larger than a time delay threshold value, responding to the time delay information being larger than the time delay threshold value, and compensating the posture and the position indicated by the first posture information according to the time delay information and the historical posture information to obtain second posture information.

In response to the delay information not being greater than the delay threshold, the delay of the instruction is proved to be weak, so that compensation calculation is not needed, and waste of processing resources is avoided. And under the condition that the time delay information is larger than the time delay threshold value, the instruction has stronger hysteresis, and if the server directly applies the instruction to generate the image or video stream, the image or video stream with lower accuracy is generated. Therefore, the first instruction needs to be adjusted. The present embodiment does not limit the value of the delay threshold, and the delay threshold may be set according to actual needs or experience. For example, the delay threshold may be 30 milliseconds, or may be a value other than 30 milliseconds.

And 204, acquiring a target instruction matched with the second posture information, wherein the target instruction is used for acquiring an image to be displayed.

After the terminal acquires the second position information in step 203, the terminal may encapsulate the second position information and other information that needs to be used in the transmission process to obtain a target instruction matched with the second position information. Other information that may need to be used during transmission includes, but is not limited to, transmission protocol information, routing information, and the like.

In an exemplary embodiment, after obtaining the target instruction matching the second pose information, the method further comprises: and sending a target instruction to the server, receiving an image to be displayed returned by the server according to the target instruction, and displaying the image. As shown in fig. 6, before the compensation process for the first pose information is not performed, the first pose information is the pose information at time t 2. Because there is a delay between the terminal and the server and the logic on the server runs continuously, when the server receives the command matching the first attitude information, the logic run by the server is the logic at time t3, and the first attitude information at time t2 has a lag compared with the logic run by the server. Therefore, the server generates an image or video stream according to the logic executed at the time t3 and the first pose information at the time t2, which may cause the action and the logic described by the first pose information at the time t2 to be mismatched, and the image or video stream may have low precision. For example, in a human-computer interaction game, if the action and the logic are not matched, the game object which can be hit by the target object cannot be hit, and the use experience of the target object is influenced.

Through the calculation process, the target pose information included in the target instruction sent to the server by the terminal at the time t2 is the second pose information corresponding to the time t 3. There is still a delay in the transmission of the target command, so that the target command can be received by the server at time t3, and the logic executed by the server is the logic at time t 3. The second pose information included in the target instruction is pose information at the time t3, so that the second pose information is matched with the logic of the server operation, the object action of the target object is matched with the logic, the accuracy of the video stream is high, and the use experience of the target object is guaranteed.

After receiving a target instruction sent by a terminal, a server performs calculation and rendering based on the target instruction to obtain an image or video stream, encodes the image or video stream, and sends the encoded image or video stream to the terminal so as to be displayed by the terminal conveniently. Since the server transmits the encoded image or video stream, the terminal may receive the encoded image or video stream accordingly. After receiving, the terminal may decode the encoded image or video stream to obtain the original target video stream. And after receiving the image or video stream sent by the server, the terminal also displays the target video stream. That is, the terminal may display the target video stream through the VR headset or other display device, so that the user can browse the target video stream with higher accuracy.

Based on the above description, referring to fig. 3, the embodiment may provide a system in which the terminal serves as a client of the delay detection subsystem, the server serves as a server of the delay detection subsystem, and the terminal is used for compensating the posture and the position indicated by the first posture information. In step 301, the terminal may acquire first pose information, and in step 302, the first device further needs to detect delay information with the server through interaction with the server, so as to obtain the delay information. In addition to acquiring the time delay information, in step 303, since the terminal needs to perform compensation calculation, the terminal also needs to acquire historical pose information required by the compensation calculation. After the terminal acquires the time delay information and the historical pose information, the terminal may perform compensation adjustment on the first pose information according to the time delay information and the historical pose information in step 304, so as to obtain second pose information.

Next, since the image and video stream need to be generated on the server side, the terminal acquires the target command matching the second pose information, encapsulates the target command into a packet, and transmits the packet to the server in step 305. In step 306, the server may perform calculations and rendering based on the received target instructions, generating an image or video stream for display. After the server generates the image or video stream, referring to step 307, the image or video stream needs to be encoded, and the encoded image or video stream is returned to the terminal. In the subsequent process, the terminal can also decode, display and the like the received encoded image or video stream.

In addition, referring to fig. 4, the embodiment may further provide a system in which the network node serves as a client of the delay detection subsystem, the server serves as a server of the delay detection subsystem, and the terminal is used to compensate the posture and the position indicated by the first posture information. In the acquisition process, referring to step 401, the terminal may acquire first position information. Since the network node and the server are the client and the server in the delay detection subsystem, respectively, the delay information between the terminal and the server is obtained through the interaction between the network node and the server in step 402. In step 403, the terminal may obtain the time delay information from any device in the network node or the server, and the terminal needs to further obtain historical pose information, so that in step 404, the pose and the position indicated by the first pose information are adjusted according to the time delay information and the historical pose information to obtain the second pose information.

Next, since the video stream needs to be generated on the server side, referring to step 405, the terminal obtains the target command matched with the second pose information, encapsulates the target command into a data packet, and sends the data packet to the server. In step 406, the server may perform calculations and rendering based on the received target instructions, generating an image or video stream for display. In step 407, after the server generates the image or video stream, it needs to encode the image or video stream, and return the encoded image or video stream to the terminal. In the subsequent process, the terminal can also decode, display and the like the received encoded image or video stream.

In summary, according to the embodiment of the application, the first pose information is compensated through the time delay information and the historical pose information, the hysteresis of the target instruction is improved, and the instruction error caused by time delay is calibrated, so that after the target instruction reaches the server in a delayed manner due to the time delay, the posture and the position indicated by the second pose information matched with the target instruction can still be matched with the running logic of the server, and the accuracy of action response is improved. The scheme has strong self-adaptability, can generate images or video streams with high precision, and improves the operation experience of users.

Based on the image processing system shown in fig. 1, the present embodiment provides an image processing method, which is applied to a network device, where the network device may be a server shown in fig. 1. Referring to fig. 7, the method includes:

step 701, acquiring first pose information, where the first pose information is used to indicate a pose and a position of a target object at a first time.

The terminal has the function of collecting the first position information, so that the first position information is obtained by the terminal side. Since the server needs to compensate the first pose information, the server needs to first obtain the first pose information to be adjusted from the terminal. Illustratively, the first pose information acquired by the server may be the first pose information pushed by the terminal, and also starts to be the first pose information returned by the terminal according to the acquisition request sent by the server.

Step 702, historical pose information is obtained, and the historical pose information is used for indicating the posture and the position of the target object at the historical moment.

The historical pose information is acquired by the terminal, and is stored locally in the terminal after the terminal acquires the historical pose information, so that the server needs to acquire the historical pose information from the terminal. After the terminal acquires the historical pose information, the historical pose information is uploaded to a server to be stored, and the server can acquire the historical pose information locally. In response to the terminal storing the historical pose information in the terminal and another device other than the server, the server needs to acquire the historical pose information from the other device.

And 703, compensating the posture and the position indicated by the first posture information according to the time delay information and the historical posture information to obtain second posture information.

For the description of the time delay information, see step 203. For example, in the case that the delay information is network delay between the terminal and the server, or device delay generated by the terminal and the server during operation, the delay information may be determined by the delay detection subsystem. The delay detection subsystem may be formed by a terminal and a server. Alternatively, referring to fig. 8, the latency detection subsystem may also be formed by a network node and a server.

For the condition that the terminal and the server are used as the time delay detection subsystem, in response to the fact that the terminal is a time delay detection client and the server is a server, the terminal sends a detection signal to the server, the server calculates a difference value between sending time of the detection signal sent by the terminal and receiving time of a local received detection signal, and the difference value is used as time delay information between the terminal and the server, so that the server can obtain the time delay information.

In response to the situation that the terminal is a time delay detection server and the server is a client, the server first sends a detection signal to the terminal, the terminal receives the detection signal sent by the server, and the terminal calculates a difference value between the sending time of the detection signal sent by the server and the receiving time of the detection signal received locally. After the terminal calculates the difference, the difference can be returned to the server, so that the server can acquire the delay information.

As shown in fig. 8, for the case that the network node and the server are used as the delay detection subsystem, in response to that the network node is a delay detection client and the server is a server, the network node sends a detection signal to the server, the server calculates a difference between a sending time of the detection signal sent by the network node and a receiving time of a local reception detection signal, and uses the difference as delay information between the network node and the server, so that the server realizes acquisition of the delay information.

In response to the situation that the network node is a time delay detection server and the server is a client, the server first sends a detection signal to the network node, the network node receives the detection signal sent by the server, and the network node calculates a difference between sending time of the detection signal sent by the server and receiving time of the detection signal locally received by the server. After the difference is obtained by the calculation of the network node, the difference can be returned to the server, so that the server can obtain the time delay information.

For the process of compensating the pose and the position indicated by the first pose information according to the time delay information and the historical pose information, reference may be made to the description in step 203, which is not described herein again.

And step 704, acquiring a target instruction matched with the second posture information, wherein the target instruction is used for acquiring an image to be displayed.

The process of obtaining the target instruction may refer to the description in step 204 above, and is not described here again. In an exemplary embodiment, after the server obtains the target instruction matching the second pose information, the method further includes: and acquiring an image to be displayed according to the target instruction, and sending the image to be displayed to the terminal for displaying. The server performs calculation, rendering and other processes based on the target instruction to obtain an image or video stream to be displayed. The terminal may be configured to display the image or video stream after receiving the image or video stream transmitted by the server.

Based on the above description, referring to fig. 8, this embodiment may provide a system in which the network node serves as a client of the delay detection subsystem, the server serves as a server of the delay detection subsystem, and the server is used to compensate the posture and the position indicated by the first posture information. In step 801, collecting first pose information is performed by a terminal, since the terminal is a device capable of human-computer interaction with a target object. In step 802, the terminal sends the first pose information to the server, and accordingly, the server may receive the first pose information and the historical pose information to be sent by the terminal. In addition, in step 803, the server needs to detect the delay information between the server and the terminal through interaction with the network node, so as to achieve acquisition of the delay information. In step 804, in addition to obtaining the time delay information, the server needs to obtain historical pose information required to compensate for the first pose information. In step 805, after the server obtains the time delay information and the historical pose information, the server may compensate the first pose information according to the time delay information and the historical pose information, so as to obtain the second pose information.

In step 806, since the image or video stream can be generated on the server side, after the server acquires the target command matching the second pose information, the server can continue to perform calculation and rendering based on the target command to generate the image video stream for display. After the server generates the image video stream, the server further needs to encode the image or video stream, and return the encoded image or video stream to the terminal, in step 807. In the subsequent process, the terminal can also decode, display and the like the received encoded image or video stream.

As shown in fig. 9, an embodiment of the present application further provides an apparatus for image processing, including:

an obtaining module 901, configured to obtain first pose information of a target object, where the first pose information is used to indicate a pose and a position of the target object at a first time;

the obtaining module 901 is further configured to obtain historical pose information of the target object, where the historical pose information is used to indicate a pose and a position of the target object at a historical time;

a compensation module 902, configured to compensate the pose and the position indicated by the first pose information according to the time delay information and the historical pose information to obtain second pose information, where the time delay information is obtained based on a time interval between an acquisition time of the reference pose information and a display time of a reference image, the reference image is an image corresponding to the reference pose information, and the acquisition time of the reference pose information is earlier than the first time;

the obtaining module 901 is configured to obtain a target instruction matched with the second pose information, where the target instruction is used to obtain an image to be displayed.

In an exemplary embodiment, the compensation module 902 is configured to determine pose change information according to the time delay information and the historical pose information, where the pose change information includes a pose change amount and a pose change direction; and based on the pose change direction, compensating the pose and the position indicated by the first pose information according to the pose change amount, and obtaining second pose information according to the compensated pose and position.

In an exemplary embodiment, the compensation module 902 is configured to determine a pose change direction and a pose change rate according to historical pose information; and determining the pose variation according to the time delay information and the pose variation rate.

In an exemplary embodiment, the obtaining module 901 is further configured to obtain time delay information; and the compensation module is also used for responding to the time delay information being larger than the time delay threshold value, and executing the step of compensating the posture and the position indicated by the first posture information according to the time delay information and the historical posture information to obtain second posture information.

In an exemplary embodiment, the apparatus further comprises: and the display module is used for sending the target instruction to the server, receiving the image to be displayed returned by the server according to the target instruction and displaying the image.

In an exemplary embodiment, the apparatus further comprises: and the sending module is used for obtaining the image to be displayed according to the target instruction and sending the image to be displayed to the terminal for displaying.

It should be understood that, when the apparatus provided in fig. 9 implements its functions, it is only illustrated by the division of the functional modules, and in practical applications, the above functions may be distributed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to implement all or part of the functions described above. In addition, the apparatus and method embodiments provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments for details, which are not described herein again.

An embodiment of the present application provides a communication apparatus, including: a transceiver, a memory, and a processor. Wherein the transceiver, the memory and the processor are in communication with each other through an internal connection path, the memory is used for storing instructions, the processor is used for executing the instructions stored in the memory to control the transceiver to receive signals and control the transceiver to transmit signals, and when the processor executes the instructions stored in the memory, the processor is enabled to execute the method provided in any one of the exemplary embodiments of step 201 and step 204.

An embodiment of the present application provides a communication apparatus, including: a transceiver, a memory, and a processor. Wherein the transceiver, the memory and the processor are in communication with each other via an internal connection path, the memory is used for storing instructions, the processor is used for executing the instructions stored in the memory to control the transceiver to receive signals and control the transceiver to transmit signals, and when the processor executes the instructions stored in the memory, the processor is enabled to execute the method provided in any one of the exemplary embodiments in

steps

701 and 704.

Optionally, there are one or more processors and one or more memories.

An embodiment of the present application provides a computer program (product), including: computer program code which, when executed by a computer, causes the computer to perform the method as provided by any of the exemplary embodiments of this application.

Embodiments of the present application provide a readable storage medium storing a program or instructions, which when executed on a computer, performs the method in the above aspects.

The embodiment of the present application provides a chip, which includes a processor, and is configured to call and execute instructions stored in a memory, so that a communication device in which the chip is installed executes a method provided in any exemplary embodiment of the present application.

The embodiment of the present application provides a chip, including: the system comprises an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected through an internal connection path, the processor is used for executing codes in the memory, and when the codes are executed, the processor is used for executing the method provided by any one of the exemplary embodiments of the application.

It should be understood that the processor may be a Central Processing Unit (CPU), other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or any conventional processor or the like. It is noted that the processor may be an advanced reduced instruction set machine (ARM) architecture supported processor.

Further, in an alternative embodiment, the memory may include both read-only memory and random access memory, and provide instructions and data to the processor. The memory may also include non-volatile random access memory. For example, the memory may also store device type information.

The memory may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available. For example, Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), double data rate synchronous SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct memory bus RAM (DR RAM).

The present application provides a computer program, which when executed by a computer, may cause the processor or the computer to perform the respective steps and/or procedures corresponding to the above-described method embodiments.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedures or functions described in accordance with the present application are generated, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk), among others.

The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for image processing, the method being applied to a network device, the method comprising:

acquiring first attitude information of a target object, wherein the first attitude information is used for indicating the attitude and the position of the target object at a first moment;

acquiring historical pose information of the target object, wherein the historical pose information is used for indicating the posture and the position of the target object at historical time;

compensating the posture and the position indicated by the first pose information according to time delay information and the historical pose information to obtain second pose information, wherein the time delay information is obtained based on a time interval between the acquisition time of reference pose information and the display time of a reference image, the reference image is an image corresponding to the reference pose information, and the acquisition time of the reference pose information is earlier than the first time;

and acquiring a target instruction matched with the second posture information, wherein the target instruction is used for acquiring an image to be displayed.

2. The method of claim 1, wherein the compensating the pose and the position indicated by the first pose information according to the time delay information and the historical pose information to obtain second pose information comprises:

determining pose change information according to the time delay information and the historical pose information, wherein the pose change information comprises pose change amount and pose change direction;

and compensating the posture and the position indicated by the first posture information according to the posture change amount based on the posture change direction, and obtaining second posture information according to the compensated posture and position.

3. The method of claim 2, wherein determining pose change information from the time delay information and the historical pose information comprises:

determining the pose change direction and the pose change rate according to the historical pose information;

and determining the pose variation according to the time delay information and the pose variation rate.

4. The method of any of claims 1-3, wherein after obtaining the first pose information of the target object, the method further comprises:

acquiring the time delay information;

and responding to the time delay information being larger than a time delay threshold value, and executing the step of compensating the posture and the position indicated by the first posture information according to the time delay information and the historical posture information to obtain second posture information.

5. The method according to any one of claims 1-4, wherein the network device is a terminal, and after the obtaining of the target command matching the second position information, the method further comprises:

and sending the target instruction to a server, receiving an image to be displayed returned by the server according to the target instruction, and displaying the image.

6. The method according to any one of claims 1-4, wherein the network device is a server, and after obtaining the target instruction matching the second pose information, the method further comprises:

and acquiring an image to be displayed according to the target instruction, and sending the image to be displayed to a terminal for displaying.

7. An apparatus for image processing, the apparatus comprising:

the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring first attitude information of a target object, and the first attitude information is used for indicating the attitude and the position of the target object at a first moment;

8. The apparatus of claim 7, wherein the compensation module is configured to determine pose change information according to the time delay information and the historical pose information, and the pose change information includes pose change amount and pose change direction; and compensating the posture and the position indicated by the first posture information according to the posture change amount based on the posture change direction, and obtaining second posture information according to the compensated posture and position.

9. The apparatus of claim 8, wherein the compensation module is configured to determine the pose change direction and the pose change rate according to the historical pose information; and determining the pose variation according to the time delay information and the pose variation rate.

10. The apparatus according to any one of claims 7-9, wherein the obtaining module is further configured to obtain the delay information; and the compensation module is further used for responding to the time delay information being larger than a time delay threshold value, and executing the step of compensating the posture and the position indicated by the first pose information according to the time delay information and the historical pose information to obtain second pose information.

11. The apparatus of any of claims 7-10, further comprising: and the display module is used for sending the target instruction to a server, receiving the image to be displayed returned by the server according to the target instruction and displaying the image.

12. The apparatus of any of claims 7-10, further comprising: and the sending module is used for obtaining the image to be displayed according to the target instruction and sending the image to be displayed to a terminal for displaying.

13. A network device, the network device comprising a memory and a processor; the memory has stored therein at least one instruction that is loaded and executed by the processor to implement the method of image processing as claimed in any of claims 1-6.

14. A computer-readable storage medium having stored therein at least one instruction, which is loaded and executed by a processor to implement the method of image processing according to any one of claims 1-6.