CN111583350A - Image processing method, device and system and server - Google Patents
Image processing method, device and system and server Download PDFInfo
- Publication number
- CN111583350A CN111583350A CN202010472745.3A CN202010472745A CN111583350A CN 111583350 A CN111583350 A CN 111583350A CN 202010472745 A CN202010472745 A CN 202010472745A CN 111583350 A CN111583350 A CN 111583350A
- Authority
- CN
- China
- Prior art keywords
- rendering
- image
- rendered
- visual angle
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
- G06T9/004—Predictors, e.g. intraframe, interframe coding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
- G06T9/001—Model-based coding, e.g. wire frame
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
- G06T9/008—Vector quantisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The application provides an image processing method, an image processing device, an image processing system and a server, wherein after the server acquires first rendering visual angle information of a current frame, second rendering visual angle information of a previous frame is directly called, image rendering is carried out by using model data of an object to be rendered according to the first rendering visual angle information, a rendering image of the current frame is obtained, meanwhile, a motion vector of the rendering image of the current frame relative to the rendering image of the previous frame is obtained by directly using the first rendering visual angle information and the second rendering visual angle information, accordingly, coding compression of the rendering image of the current frame is completed, and the compressed rendering image is output. Therefore, a large amount of time consumed by estimating the motion vector based on image block search is greatly saved, the rendering image compression efficiency is improved, the loss of the compressed rendering image is avoided, and the definition and the fluency of the displayed image after the electronic equipment decompresses are ensured.
Description
Technical Field
The present application relates to the field of virtual reality applications, and in particular, to an image processing method, apparatus, system, and server.
Background
Nowadays, with the development and popularization of a 5G network (fifth generation mobile communication network), an AR (augmented Reality) device/VR (Virtual Reality) device can perform image rendering on a complex model (i.e. a 3D model of an object to be rendered, such as a monster, a building and the like in a game scene) by using a cloud rendering technology, so that a device wearer can see detailed information of the complex model in real time, and user experience is improved.
The cloud rendering technology is that a 3D program is placed in a remote server for rendering, a user initiates a control instruction through a terminal, the server responds to the control instruction, corresponding rendering tasks are executed, and an obtained rendering image is fed back to the user terminal for displaying.
In practical application, in order to meet the real-time requirement of the device on display, in the cloud rendering process, after a rendered image of any frame is obtained, the rendered image needs to be compressed first and then transmitted to the device. In particular, a video compression-based method is usually adopted to implement compression and real-time transmission of rendered images obtained by rendering consecutive frames,
however, after the rendered image of the current frame is obtained, in the prior art, a motion vector estimation is usually performed on the rendered image of the previous frame and the rendered image of the current frame in an image block search manner, so as to obtain motion vector information of the current frame, and then coding and compression of the rendered image are realized according to the motion vector information.
Disclosure of Invention
In view of the above, in order to solve the above technical problems, the present application provides the following technical solutions:
in one aspect, the present application provides an image processing method, including:
acquiring first rendering visual angle information of a current frame of an object to be rendered;
calling second rendering visual angle information of a last frame of the object to be rendered;
according to the first rendering visual angle information, image rendering is carried out by using the model data of the object to be rendered to obtain a rendering image of the current frame;
in the process of image rendering by using the model data, obtaining a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame by using the first rendering visual angle information and the second rendering visual angle information;
and encoding and compressing the rendering image of the current frame by using the motion vector, and outputting the compressed rendering image.
Optionally, the obtaining a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame by using the first rendering perspective information and the second rendering perspective information includes:
acquiring offset information between the two-dimensional projection point of the current frame and the two-dimensional projection point of the previous frame when the model data is projected on the two-dimensional plane by using the first rendering visual angle information and the second rendering visual angle information;
and determining a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame according to the offset information.
Optionally, the obtaining, by using the first rendering perspective information and the second rendering perspective information, offset information between the two-dimensional projection point of the current frame and the two-dimensional projection point of the previous frame, where the model data is projected onto a two-dimensional plane, includes:
comparing the first rendering visual angle information with the second rendering visual angle information to obtain rendering visual angle variable quantities of the current frame and the previous frame;
acquiring a first position of a two-dimensional projection point of the model data projected on a two-dimensional plane in the process of utilizing the model data to render the current frame image;
calling a second position of a two-dimensional projection point projected by the model data onto a two-dimensional plane in the process of rendering the previous frame of image by using the model data;
and obtaining offset information between the first position and the second position according to the rendering visual angle variable quantity.
Optionally, the method further includes:
acquiring third rendering visual angle information of a next frame aiming at the object to be rendered in the process of image rendering by using the model data or compressing the rendering image of the current frame;
and performing image rendering by using the model data according to the third rendering visual angle information to obtain a rendering image of the next frame, and obtaining a motion vector of the rendering image of the next frame relative to the rendering image of the current frame by using the first rendering visual angle information and the third rendering visual angle information in the process of performing image rendering of the next frame by using the model data.
Optionally, the method further includes:
the acquired first rendering visual angle information and the frame information of the current frame are stored in an associated mode;
the retrieving second rendering perspective information for a last frame of the object to be rendered includes:
and calling second rendering visual angle information stored in association with frame information of a previous frame of the current frame from rendering visual angle information stored in association with each of at least one historical frame.
Optionally, the rendering an image by using the model data of the object to be rendered according to the first rendering perspective information to obtain a rendered image of the current frame includes:
acquiring data to be rendered corresponding to the first rendering visual angle information from the model data of the object to be rendered;
and performing image rendering by using the data to be rendered to obtain a rendered image of the current frame.
In yet another aspect, the present application further proposes an image processing apparatus, the apparatus comprising:
the information acquisition module is used for acquiring first rendering visual angle information of a current frame of an object to be rendered;
the information calling module is used for calling second rendering visual angle information of a previous frame of the object to be rendered;
the image rendering module is used for performing image rendering by using the model data of the object to be rendered according to the first rendering visual angle information to obtain a rendering image of the current frame;
a motion vector obtaining module, configured to obtain a motion vector of a rendered image of a current frame with respect to a rendered image of a previous frame by using the first rendering perspective information and the second rendering perspective information in an image rendering process performed by the image rendering module using the model data;
and the coding and compressing module is used for coding and compressing the rendering image of the current frame by using the motion vector and outputting the compressed rendering image.
In another aspect, the present application further provides a server, including:
a communication module;
a memory for storing a program for implementing the image processing method as described above;
and the processor is used for loading and executing the program stored in the memory so as to realize the steps of the image processing method.
In yet another aspect, the present application further proposes an image processing system, the system comprising at least one electronic device and at least one server;
the electronic equipment is used for acquiring rendering visual angle information corresponding to each of continuous multiframes of an object to be rendered under the condition that a user wears the electronic equipment;
the server is the server.
Optionally, the electronic device includes multiple types of sensors, and the multiple types of sensors are configured to sense different degrees of freedom spatial poses of the electronic device, so as to obtain rendering perspective information of a corresponding frame.
Therefore, compared with the prior art, the application provides an image processing method, device, system and server, after the server acquires the first rendering visual angle information of the current frame, the server directly calls the second rendering visual angle information of the previous frame, and performs image rendering by using the model data of the object to be rendered according to the first rendering visual angle information, so that while the rendering image of the current frame is obtained, the motion vector of the rendering image of the current frame relative to the rendering image of the previous frame is obtained by directly using the first rendering visual angle information and the second rendering visual angle information, so that the motion vector is used to complete coding compression of the rendering image of the current frame, and the compressed rendering image is output. Compared with the prior art, the method greatly saves a large amount of time consumed by estimating the motion vector based on image block search, improves the compression efficiency of the rendered image, solves the uncertainty caused by the image block search matching mode, ensures the reliability and accuracy of the obtained motion vector, avoids the loss of the compressed rendered image, and improves the definition and smoothness of the displayed image after the electronic equipment decompresses.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a schematic structural diagram of an alternative image processing system for implementing the image processing method proposed in the present application;
FIG. 2 illustrates a flow diagram of cloud rendering an application scene;
fig. 3 shows a schematic flow chart of an alternative example of the image processing method proposed by the present application;
FIG. 4 is a flow chart diagram illustrating a prior art image processing method;
FIG. 5 shows a schematic flow diagram of yet another alternative example of the image processing method proposed by the present application;
FIG. 6 shows a schematic flow diagram of yet another alternative example of the image processing method proposed by the present application;
fig. 7 shows a schematic flow diagram of yet another alternative example of the image processing method proposed by the present application;
fig. 8 is a schematic structural diagram showing an alternative example of the image processing apparatus proposed by the present application;
fig. 9 shows a schematic configuration diagram of yet another alternative example of the image processing apparatus proposed in the present application;
fig. 10 is a hardware configuration diagram showing an alternative example of a server that implements the image processing method proposed by the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present application may be combined with each other without conflict. Moreover, as used in this application, a "system," "apparatus," "unit" and/or "module" is a method for distinguishing between different components, elements, components, parts or assemblies of different levels and may be replaced by other language if it is done so.
As used in this application and the appended claims, the terms "a," "an," "the," and/or "the" are not intended to be inclusive in the singular, but rather are intended to be inclusive in the plural unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements. An element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
In the description of the embodiments herein, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two. The terms "first", "second" and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.
Additionally, flow charts are used herein to illustrate operations performed by systems according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, the various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to the processes, or a certain step or several steps of operations may be removed from the processes.
Referring to fig. 1, in order to implement a schematic structural diagram of an optional image processing system of the image processing method provided in the present application, the image processing system may be applicable to various cloud rendering application scenarios, as shown in fig. 1, the system may include at least one electronic device 100 and at least one server 200, where:
the electronic device 100 may be configured to acquire rendering perspective information corresponding to each of consecutive multiple frames of an object to be rendered when a user wears the electronic device.
In this embodiment, the electronic device 100 may include various types of AR (Augmented Reality) devices, VR (Virtual Reality) devices, and the like, such as VR smart glasses, a helmet, a handle, and the like, and the specific device types of the electronic device 100 are not limited in this application, and for different users, the electronic device used by the user to view an object to be rendered in a Virtual scene may be determined according to actual needs, usage habits, and the like, which are not described in detail herein.
The server 200 may be a service device that supports cloud computing and implements cloud rendering, that is, a cloud server deployed in a cloud end, and may specifically be composed of one or more servers, and the specific composition structure of the server 200 is not limited in the present application. In the present application, the image processing method and apparatus provided in the following embodiments of the present application may be implemented, and the specific implementation process may refer to the description of the corresponding parts in the following embodiments.
In practical applications, in order to improve interactive experience and control over a virtual world, in a cloud rendering application scene, referring to a flow chart of the cloud rendering application scene shown in fig. 2, it is generally required that the electronic device 100 collects rendering view angle information at different times and sends the rendering view angle information to the server 200, the server 200 performs image rendering by using model data according to the rendering view angle information, obtains a rendering image of a model of an object to be rendered at the rendering view angle, and then feeds the rendering image back to the electronic device 100 after compressing the rendering image, and the electronic device 100 decompresses and displays the compressed rendering image.
The rendering visual angle information acquired by the electronic device 100 at different times may be a multiple degree of freedom (dof) spatial pose, such as a 3dof spatial pose, a 6dof spatial pose, and the like, and the 3dof may refer to degrees of freedom of 3 rotation angles, such as rotation of the head of a user wearing the electronic device in different directions, but cannot detect front, back, left, and right spatial displacements of the head, and may be suitable for viewing an application scene of a VR movie; 6dof can be on the basis of 3dof, increase and worn the electronic equipment user's health and moved the change that brings about from top to bottom, from front to back and from left to right, can realize better tracking location to the user, if under the game application scene, can make the electronic equipment user cross the obstacle, avoid scenes such as strange animal, mountain-climbing interdynamic, obtain more true, immersive experience and experience.
Based on this, the electronic device 100 may include various types of sensors for sensing spatial poses of different degrees of freedom of the electronic device 100 to form rendering perspective information of corresponding frames, and a detailed detection process of the rendering perspective information is not described in detail.
In the embodiment, in order to improve the cloud rendering efficiency, a rendering process of a frame of rendered image and a motion vector calculation process between the frame and an adjacent frame are combined, rendering visual angle information obtained by two adjacent frames is directly used for calculating the motion vector, compared with the prior art that a large amount of time is spent on image block searching of rendered images obtained by respective rendering of the two adjacent frames, the motion vector is estimated according to an image block searching result, the operation vector calculation efficiency is greatly improved, the compression speed of the rendered images of each frame is accelerated, and the purpose of improving the model rendering efficiency is achieved. The specific implementation process can refer to the description of the corresponding parts of the following embodiments.
Referring to fig. 3, a flowchart of an alternative example of an image processing method proposed in an embodiment of the present application, which may be applied to a server in an image processing system, as shown in fig. 3, the method may include:
step S11, acquiring first rendering visual angle information of a current frame of an object to be rendered;
in connection with the above description of the cloud rendering application scenario, the user may, via an electronic device (such as the above-mentioned AR device or VR device, etc., which are not explained one by one below) or other terminal, through interaction with the server, an object to be rendered, such as an entire virtual scene or a certain virtual object contained therein, which needs to be rendered and presented currently, is determined, so that after the server obtains model data of the object to be rendered, when the user wears the electronic equipment and interacts with the currently presented virtual scene, each sensor in the electronic equipment can collect first rendering visual angle information, such as 3dof space attitude, 6dof space attitude and other multi-degree-of-freedom space attitude data, is uploaded to a server in real time, therefore, the server obtains the first rendering view angle information of the current frame of the object to be rendered, and can determine the rendering image at which rendering view angle of the object to be rendered needs to be presented to the user.
It can be seen that, in step S11, the server receives, through a wireless communication network, such as a 5G (fifth generation mobile communication) network, a WIFI network, or the like, the first rendering perspective information of the current frame, which is collected and sent by the electronic device during the period that the electronic device is worn by the user.
It should be noted that, regarding the collection of the first rendering perspective information, the collection is not limited to the collection of multiple types of sensors configured in the electronic device, and may also be performed on the user wearing the electronic device by combining with other collection devices configured in the space where the user is located, such as image collection devices like a camera, and an intelligent bracelet, so as to determine the first rendering perspective information for the object to be rendered, which is needed by the current frame.
The frames generally refer to a single image frame of the minimum unit in the image animation, one frame is a static image frame, consecutive frames can form animation, such as presenting a dynamic game scene, playing a VR movie, and the like, the number of frames is the number of frames of pictures transmitted in 1 second, and it can be understood that the graphics processor can refresh the number of times per second.
Based on this, in order to ensure the real-time performance and reliability of the interaction between the user and the virtual scene, after the user wears the electronic device, the view angle in the virtual scene usually changes continuously, that is, at least one part of the body of the user wearing the electronic device or moves, such as the head, the body, the eyes, the hands and the like, and rendering view angle information of different frames needs to be continuously collected.
Step S12, retrieving second rendering perspective information of the previous frame of the object to be rendered;
step S13, according to the first rendering visual angle information, image rendering is carried out by using the model data of the object to be rendered, and a rendering image of the current frame is obtained;
step S14, in the process of image rendering by using model data, obtaining a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame by using the first rendering visual angle information and the second rendering visual angle information;
in order to meet the high requirement of the virtual reality scene on the display real-time performance, such as from the motion of a user wearing the electronic device, a link to display a corresponding virtual picture can be preferably controlled within 10ms (unit: millisecond), but the link is not limited to this, and the rendered images of each frame need to be compressed and then transmitted in real time, so as to improve the data transmission speed, shorten the time spent in the whole process from rendering to displaying each frame of rendered image, and further improve the display effect of the electronic device on the virtual reality scene.
In the process of compressing the rendered images of different frames, motion vectors between two consecutive frames are usually required to be calculated, so as to realize coding compression of the rendered image of the current frame. In the prior art, as shown in a schematic diagram of an encoding compression process shown in fig. 4, after a rendered image of a current frame is obtained according to first rendering perspective information of the current frame, a server calls a rendered image of an adjacent previous frame, and then performs image block search matching on the rendered image of the previous frame and the rendered image of the current frame based on an image block search algorithm to obtain a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame, where the motion vector is actually estimated according to an image block search matching result, which is not only time-consuming but also often suboptimal, so that a certain loss is generated in a compressed picture.
In order to solve the problems, the method directly utilizes rendering visual angle information of two adjacent frames to directly calculate a motion vector, the motion vector is used for representing the relative displacement between a current coding block and an optimal matching block in a reference image of the current coding block in interframe coding, and the visual angle change between the rendering visual angle information of the two adjacent frames can indicate the relative poses with different degrees of freedom by combining the description of the rendering visual angle information and the above description, and after conversion processing, the same part of an object to be rendered and the relative variable of the multi-degree-of-freedom space pose between the two adjacent frames can be directly determined, so that the motion vector between the rendering images of the two adjacent frames can be accurately calculated.
It can be seen that in the method for calculating the motion vector between two adjacent frames of rendered images provided in this embodiment, the rendered images do not need to be divided into image blocks, and then a plurality of image blocks are searched and matched, so that a large amount of time consumed for searching the image blocks in the conventional method for estimating the motion vector based on image block search is saved, the compression efficiency of the rendered images is improved, meanwhile, the uncertainty caused by the image block search and matching method is also solved, the reliability and accuracy of the obtained motion vector are ensured, and the loss of the compressed images is avoided.
In addition, in order to further shorten the time from the process of rendering a frame of image to the process of transmitting the frame of image to the electronic equipment, the application further provides that in the process of rendering the image of the current frame, the motion vector is simultaneously acquired, and the motion vector is calculated after the rendering of the image of the current frame is not required to be completed. Therefore, in the application of this embodiment, after the first rendering perspective information of the current frame is obtained, as shown in the flowchart of fig. 5, the server directly retrieves the second rendering perspective information of the previous frame, and then, during the image rendering process by using the model data of the object to be rendered according to the first rendering perspective information, the server may simultaneously use the first rendering perspective information and the second rendering perspective information to obtain the motion vector of the current frame, that is, the motion vector of the rendered image of the current frame relative to the rendered image of the previous frame.
It should be noted that, the implementation process of how to use the model data of the object to be rendered to perform image rendering according to the rendering view angle information to obtain the corresponding rendering image and the implementation process of how to use the rendering view angle of the adjacent frame to calculate the motion vector between the rendering images of the adjacent frame are not limited.
Step S15 is to encode and compress the rendered image of the current frame using the motion vector, and output the compressed rendered image.
The present application does not describe the specific implementation process of step S15 in detail.
In this embodiment, after acquiring the first rendering perspective information of the current frame, the server directly retrieves the second rendering perspective information of the previous frame, and performs image rendering by using the model data of the object to be rendered according to the first rendering perspective information to obtain the rendering image of the current frame, and simultaneously, directly obtains the motion vector of the rendering image of the current frame relative to the rendering image of the previous frame by using the first rendering perspective information and the second rendering perspective information, thereby completing encoding and compression of the rendering image of the current frame by using the motion vector, and outputting the compressed rendering image. Compared with the prior art, the method greatly saves a large amount of time consumed by estimating the motion vector based on image block search, improves the compression efficiency of the rendered image, solves the uncertainty caused by the image block search matching mode, ensures the reliability and accuracy of the obtained motion vector, avoids the loss of the compressed rendered image, and improves the definition and smoothness of the displayed image after the electronic equipment decompresses.
Referring to fig. 6, a schematic flowchart of yet another optional example of the image processing method proposed in the embodiment of the present application, where the present embodiment may be an optional refined implementation of the image processing method described in the foregoing embodiment, but is not limited to such a refined implementation described in the present embodiment, and as shown in fig. 6, the method may include:
step S21, acquiring first rendering visual angle information of a current frame of an object to be rendered;
step S22, retrieving second rendering perspective information of the previous frame of the object to be rendered;
step S23, according to the first rendering visual angle information, image rendering is carried out by using the model data of the object to be rendered, and a rendering image of the current frame is obtained;
for specific implementation processes of step S21 to step S23, reference may be made to the description of corresponding parts in the foregoing embodiments, and this embodiment is not described herein again.
Step S24, acquiring offset information between the two-dimensional projection point of the current frame and the two-dimensional projection point of the previous frame when the model data is projected on the two-dimensional plane by using the first rendering visual angle information and the second rendering visual angle information;
in the process of rendering a three-dimensional model, three-dimensional model data of an object to be rendered is generally required to be projected onto a two-dimensional plane to obtain a picture of the object to be rendered at a corresponding view angle of the three-dimensional model, and specifically, the three-dimensional model data in a three-dimensional coordinate system can be converted into corresponding projection data in a two-dimensional coordinate system, that is, coordinates of a two-dimensional projection point obtained by converting three-dimensional point cloud data are determined. Moreover, because the rendering visual angle information obtained by two adjacent frames generally changes, but the change is not very large, so that the rendering images of the adjacent frames contain the same pixel point, in this embodiment, the rendering visual angle information obtained by two adjacent frames can be utilized to calculate and obtain the offset information of the two-dimensional projection point of the same pixel point of the adjacent frame, like the coordinate change of the same pixel point, and the specific calculation process is not described in detail.
Step S25, determining a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame according to the offset information;
in this embodiment, the obtained offset information of different two-dimensional projection points may be directly determined as a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame. Of course, the motion vector is obtained by converting the information format of the offset information obtained by the above calculation according to the requirement of encoding compression, which is not limited in the present application.
Step S26 is to encode and compress the rendered image of the current frame using the motion vector, and output the compressed rendered image.
In summary, in this embodiment, after the first rendering perspective information of the current frame is obtained, the current frame of the object to be rendered and the known rendering perspective information of the previous frame of the object to be rendered may be used to directly render the current frame image, and at the same time, accurately calculate offset information of two-dimensional projection points rendered at different previous and subsequent perspectives, obtain a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame, and further use the motion vector to perform encoding compression on the rendered image of the current frame, and output the compressed rendered image. The motion vector estimation process based on image block search is omitted, the calculation speed and precision of the motion vector are improved, and the whole cloud rendering experience is improved.
Referring to fig. 7, a schematic flowchart of a further optional example of an image processing method proposed in an embodiment of the present application, where the present embodiment may be a further optional refined implementation of the image processing method described in the foregoing embodiment, but is not limited to such a refined implementation described in the present embodiment, and as shown in fig. 7, the method may include:
step S31, acquiring first rendering visual angle information of a current frame of an object to be rendered;
step S32, the first rendering visual angle information and the frame information of the current frame are stored in a correlation mode;
the frame information of the current frame may be used to distinguish different frames, and therefore, the frame information may include a frame number of the current frame in the rendering display process of the whole object to be rendered, but is not limited thereto, and the application does not limit the association storage manner of the first rendering perspective information and the corresponding frame information.
Step S33, calling second rendering visual angle information stored in association with frame information of a previous frame of the current frame from rendering visual angle information stored in association with each of at least one historical frame;
in combination with the above analysis, in this embodiment, each time rendering perspective information of one frame is obtained, the rendering perspective information and corresponding frame information may be stored in association for calculating a motion vector of a next frame, so that after the first rendering perspective information of the current frame, in order to obtain the motion vector of the current frame, frame information of a previous frame of the current frame may be determined, and then rendering information corresponding to the frame information of the previous frame is searched from corresponding rendering perspective information stored in association with each historical frame, that is, second rendering perspective information of the previous frame.
Step S34, acquiring data to be rendered corresponding to the first rendering perspective information from the model data of the object to be rendered;
step S35, image rendering is carried out by using the data to be rendered to obtain a rendered image of the current frame;
the model data may be data obtained by three-dimensional modeling of the object to be rendered, and details about the process of obtaining the model data of the object to be rendered are not described. After the first rendering visual angle information of the current frame of the object to be rendered is obtained, model data of a three-dimensional model of the object to be rendered under the rendering visual angle, namely data to be rendered for rendering a picture of the object to be rendered under the rendering visual angle can be obtained, then image rendering can be carried out by using the data to be rendered, and a rendering image of the current frame can be obtained, and the specific implementation process of the image rendering is not described in detail.
Step S36, comparing the first rendering visual angle information with the second rendering visual angle information in the process of rendering the image of the current frame by using the data to be rendered to obtain the rendering visual angle variable quantity of the current frame and the previous frame;
it should be understood that specific contents included in the rendering view angle variation of the current frame and the previous frame may be determined according to the contents of the rendering view angle information, and if the rendering view angle information is a 3dof space pose or a 6dof space pose, the rendering view angle variation of two adjacent frames may include space pose variations with different degrees of freedom.
Step S37, acquiring a first position of a two-dimensional projection point of the data to be rendered projected onto a two-dimensional plane in the process of image rendering by using the data to be rendered of the current frame;
step S38, in the process of image rendering by using the data to be rendered of the previous frame, the data to be rendered is projected to a second position of the two-dimensional projection point on the two-dimensional plane;
in order to distinguish the two-dimensional projection points of the data to be rendered of different frames projected onto the two-dimensional plane, the embodiment may record the position information of the two-dimensional projection point of the current frame as the first position, and record the position information of the two-dimensional projection point of the previous frame as the second position.
Step S39, obtaining offset information between the first position and the second position according to the rendering perspective variation;
step S310, determining a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame according to the offset information;
based on the variation of the respective degree-of-freedom space poses included in the variation of the rendering visual angle, the coordinate variation of the two-dimensional projection point of the same pixel point of the adjacent frames in the two-dimensional coordinate system can be determined, and therefore the motion vector between the adjacent frames is determined.
Step S311, using the motion vector, encodes and compresses the rendered image of the current frame, and outputs the compressed rendered image.
It should be noted that, regarding the step of storing the first rendering perspective information of the current frame, the step may be completed before the rendering perspective information of the next frame is acquired, and is not limited to the above-described step execution sequence of the embodiment.
In some embodiments, since the motion vector of a certain frame and the previous frame adjacent to the certain frame is obtained, and the rendering perspective information of the previous frame is utilized through the above analysis, the rendering perspective information of other historical frames is not required to be used, so the present application may retain the rendering perspective information of several adjacent historical frames of the current frame, and the rendering perspective information of all historical frames is not required to be retained, specifically, the rendering perspective information of the current frame may be sequentially written into the storage space after the number of the rendering perspective information that may exist is determined, and if the storage space is full, the rendering perspective information that is written first may be covered, but is not limited to this storage manner.
In summary, in this embodiment, after the rendering perspective information of the current frame is obtained, the rendering perspective information of the current frame and the frame information of the frame may be stored, so as to be called when a motion vector of a next frame is obtained, and in the process of performing image rendering by using to-be-rendered data in model data corresponding to the current frame according to the rendering perspective information of the current frame, a position variation of a two-dimensional projection point under the rendering perspective of the current frame and the previous frame may also be obtained by using the rendering perspective variation of the current frame and the previous frame, so that a motion vector between the rendering images of the current frame and the previous frame is obtained through fast and accurate calculation, so as to implement encoding and compression of the rendering image of the current frame by using the motion vector, and then send the compressed rendering image to the electronic device. Compared with the existing image processing method, the motion vector estimation process based on image block search is omitted, the calculation speed and precision of the motion vector are greatly improved, the loss of the compressed rendering image is avoided, and the clear and smooth image presented by the electronic equipment is further ensured.
In some embodiments, in combination with the description of the embodiments, to further shorten the time for obtaining the rendered image between two adjacent frames, the present application may obtain the third rendering perspective information of the next frame of the object to be rendered during the process of rendering the image of the current frame by using the model data or compressing the obtained rendered image of the current frame, and in this case, in combination with the above analysis, the server may perform image rendering by using the model data according to the third rendering perspective information to obtain the rendered image of the next frame, and in the process of rendering the image of the next frame by using the model data, by using the first rendering perspective information and the third rendering perspective information, the motion vector of the rendered image of the next frame relative to the rendered image of the current frame is obtained, so as to implement compression of the rendered image of the next frame.
Referring to fig. 8, a schematic structural diagram of an alternative example of an image processing apparatus provided in an embodiment of the present application, where the apparatus may be applied to a server in an image processing system, as shown in fig. 8, the apparatus may include:
an information obtaining module 210, configured to obtain first rendering perspective information of a current frame of an object to be rendered;
an information retrieving module 220, configured to retrieve second rendering perspective information of a previous frame of the object to be rendered;
in some embodiments, the image processing apparatus proposed in the present application may further include:
the rendering visual angle information storage module is used for storing the acquired first rendering visual angle information and the frame information of the current frame in a correlation manner;
accordingly, the information retrieving module 220 may include:
and the second rendering visual angle information calling unit is used for calling second rendering visual angle information stored in association with frame information of a previous frame of the current frame from rendering visual angle information stored in association with each of at least one historical frame.
An image rendering module 230, configured to perform image rendering by using the model data of the object to be rendered according to the first rendering perspective information, so as to obtain a rendered image of the current frame;
in one possible implementation, the image rendering module 230 may include:
a to-be-rendered data obtaining unit, configured to obtain, from the model data of the to-be-rendered object, to-be-rendered data corresponding to the first rendering perspective information;
and the image rendering unit is used for rendering the image by using the data to be rendered to obtain a rendered image of the current frame.
A motion vector obtaining module 240, configured to obtain a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame by using the first rendering perspective information and the second rendering perspective information in an image rendering process performed by the image rendering module using the model data;
and an encoding and compressing module 250, configured to perform encoding and compression on the rendered image of the current frame by using the motion vector, and output a compressed rendered image.
In some embodiments, as shown in fig. 9, the motion vector obtaining module 240 may include:
an offset information obtaining unit 241, configured to obtain offset information between a two-dimensional projection point of a current frame and a two-dimensional projection point of a previous frame when the model data is projected onto a two-dimensional plane by using the first rendering perspective information and the second rendering perspective information;
in a possible implementation manner, the offset information obtaining unit 241 may include:
a rendering visual angle variation obtaining unit, configured to compare the first rendering visual angle information with the second rendering visual angle information to obtain a rendering visual angle variation of a current frame and a previous frame;
the first position acquisition unit is used for acquiring a first position of a two-dimensional projection point of the model data projected onto a two-dimensional plane in the process of rendering the current frame image by using the model data;
a second position retrieving unit, configured to retrieve a second position of a two-dimensional projection point at which the model data is projected onto a two-dimensional plane in a process of rendering a previous frame of image by using the model data;
and the offset information obtaining unit is used for obtaining offset information between the first position and the second position according to the rendering visual angle variable quantity.
A motion vector determining unit 242, configured to determine a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame according to the offset information.
Based on the image processing apparatus described in each of the above embodiments, in order to further shorten a time difference between rendering images of adjacent frames to improve a cloud rendering experience, the image processing apparatus provided in the present application may further include:
a third rendering perspective information obtaining module, configured to obtain third rendering perspective information for a next frame of the object to be rendered in a process of performing image rendering using the model data or compressing the rendering image of the current frame;
a next frame rendering image obtaining module, configured to perform image rendering by using the model data according to the third rendering perspective information, so as to obtain a rendering image of the next frame;
and a next frame motion vector obtaining module, configured to obtain a motion vector of the rendered image of the next frame relative to the rendered image of the current frame by using the first rendering perspective information and the third rendering perspective information in a process of rendering an image of the next frame by using the model data.
It should be noted that, various modules, units, and the like in the embodiments of the foregoing apparatuses may be stored in the memory as program modules, and the processor executes the program modules stored in the memory to implement corresponding functions, and for the functions implemented by the program modules and their combinations and the achieved technical effects, reference may be made to the description of corresponding parts in the embodiments of the foregoing methods, which is not described in detail in this embodiment.
The present application also provides a storage medium on which a computer program can be stored, the computer program being called and loaded by a processor to implement the steps of the image processing method described in the above embodiments.
Referring to fig. 10, in order to illustrate a hardware structure of an alternative example of a server for implementing the image processing method proposed by the present application, the server may include a communication module 31, a memory 32, and a processor 33, wherein:
the number of the communication module 31, the memory 32, and the processor 33 may be at least one, and the communication module 31, the memory 32, and the processor 33 may all be connected to a communication bus, so as to implement data interaction therebetween through the communication bus, and a specific implementation process may be determined according to requirements of a specific application scenario, which is not described in detail herein.
The communication module 31 may include a communication module capable of implementing data interaction by using a wireless communication network, such as a WIFI module, a 5G/6G (fifth generation mobile communication network/sixth generation mobile communication network) module, and the communication module 31 may further include a communication interface for implementing data interaction between internal components of the server, such as a USB interface, a serial/parallel interface, and the like, and the specific content included in the communication module 31 is not limited in this application.
In the present embodiment, the memory 32 may include a high speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device or other volatile solid state storage device. The processor 33 may be a Central Processing Unit (CPU), an application-specific integrated circuit (ASIC), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic device.
In practical applications of the present embodiment, the memory 32 may be used for storing a program for implementing the image processing method described in any of the above method embodiments; the processor 33 may load and execute the program stored in the memory 32 to implement each step of the image processing method provided in any of the above method embodiments of the present application, and the specific implementation process may refer to the description of the corresponding part in the corresponding embodiment above, which is not described again.
It should be understood that the structure of the server shown in fig. 10 does not constitute a limitation to the server in the embodiment of the present application, and in practical applications, the server may include more or less components than those shown in fig. 10, or some components may be combined, and the present application is not specifically described herein.
Finally, it should be noted that, in the present specification, the embodiments are described in a progressive or parallel manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. The device, the system and the server disclosed by the embodiment correspond to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method of image processing, the method comprising:
acquiring first rendering visual angle information of a current frame of an object to be rendered;
calling second rendering visual angle information of a last frame of the object to be rendered;
according to the first rendering visual angle information, image rendering is carried out by using the model data of the object to be rendered to obtain a rendering image of the current frame;
in the process of image rendering by using the model data, obtaining a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame by using the first rendering visual angle information and the second rendering visual angle information;
and encoding and compressing the rendering image of the current frame by using the motion vector, and outputting the compressed rendering image.
2. The method of claim 1, wherein the obtaining a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame by using the first rendering perspective information and the second rendering perspective information comprises:
acquiring offset information between the two-dimensional projection point of the current frame and the two-dimensional projection point of the previous frame when the model data is projected on the two-dimensional plane by using the first rendering visual angle information and the second rendering visual angle information;
and determining a motion vector of the rendered image of the current frame relative to the rendered image of the previous frame according to the offset information.
3. The method of claim 2, wherein the obtaining of the offset information between the two-dimensional projection point of the current frame and the two-dimensional projection point of the previous frame by using the first rendering perspective information and the second rendering perspective information to project the model data onto the two-dimensional plane comprises:
comparing the first rendering visual angle information with the second rendering visual angle information to obtain rendering visual angle variable quantities of the current frame and the previous frame;
acquiring a first position of a two-dimensional projection point of the model data projected on a two-dimensional plane in the process of utilizing the model data to render the current frame image;
calling a second position of a two-dimensional projection point projected by the model data onto a two-dimensional plane in the process of rendering the previous frame of image by using the model data;
and obtaining offset information between the first position and the second position according to the rendering visual angle variable quantity.
4. The method of any of claims 1-3, further comprising:
acquiring third rendering visual angle information of a next frame aiming at the object to be rendered in the process of image rendering by using the model data or compressing the rendering image of the current frame;
and performing image rendering by using the model data according to the third rendering visual angle information to obtain a rendering image of the next frame, and obtaining a motion vector of the rendering image of the next frame relative to the rendering image of the current frame by using the first rendering visual angle information and the third rendering visual angle information in the process of performing image rendering of the next frame by using the model data.
5. The method of any of claims 1-3, further comprising:
the acquired first rendering visual angle information and the frame information of the current frame are stored in an associated mode;
the retrieving second rendering perspective information for a last frame of the object to be rendered includes:
and calling second rendering visual angle information stored in association with frame information of a previous frame of the current frame from rendering visual angle information stored in association with each of at least one historical frame.
6. The method according to any one of claims 1 to 3, wherein the performing image rendering by using the model data of the object to be rendered according to the first rendering perspective information to obtain a rendered image of the current frame comprises:
acquiring data to be rendered corresponding to the first rendering visual angle information from the model data of the object to be rendered;
and performing image rendering by using the data to be rendered to obtain a rendered image of the current frame.
7. An image processing apparatus, the apparatus comprising:
the information acquisition module is used for acquiring first rendering visual angle information of a current frame of an object to be rendered;
the information calling module is used for calling second rendering visual angle information of a previous frame of the object to be rendered;
the image rendering module is used for performing image rendering by using the model data of the object to be rendered according to the first rendering visual angle information to obtain a rendering image of the current frame;
a motion vector obtaining module, configured to obtain a motion vector of a rendered image of a current frame with respect to a rendered image of a previous frame by using the first rendering perspective information and the second rendering perspective information in an image rendering process performed by the image rendering module using the model data;
and the coding and compressing module is used for coding and compressing the rendering image of the current frame by using the motion vector and outputting the compressed rendering image.
8. A server, the server comprising:
a communication module;
a memory for storing a program for implementing the image processing method according to any one of claims 1 to 6;
a processor for loading and executing the program stored in the memory to realize the steps of the image processing method according to any one of claims 1 to 6.
9. An image processing system, the system comprising at least one electronic device and at least one server;
the electronic equipment is used for acquiring rendering visual angle information corresponding to each of continuous multiframes of an object to be rendered under the condition that a user wears the electronic equipment;
the server of claim 8.
10. The system of claim 9, the electronic device comprising a plurality of types of sensors to sense different degrees of freedom spatial poses of the electronic device to derive rendering perspective information for a respective frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010472745.3A CN111583350A (en) | 2020-05-29 | 2020-05-29 | Image processing method, device and system and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010472745.3A CN111583350A (en) | 2020-05-29 | 2020-05-29 | Image processing method, device and system and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111583350A true CN111583350A (en) | 2020-08-25 |
Family
ID=72114247
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010472745.3A Pending CN111583350A (en) | 2020-05-29 | 2020-05-29 | Image processing method, device and system and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111583350A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022151972A1 (en) * | 2021-01-14 | 2022-07-21 | 华为云计算技术有限公司 | Video encoding method and apparatus, device, and storage medium |
| WO2023284504A1 (en) * | 2021-07-12 | 2023-01-19 | 华为技术有限公司 | Data processing method, system and electronic apparatus |
| WO2024002065A1 (en) * | 2022-06-30 | 2024-01-04 | 维沃移动通信有限公司 | Video encoding method and apparatus, electronic device, and medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105263050A (en) * | 2015-11-04 | 2016-01-20 | 山东大学 | Mobile terminal real-time rendering system and method based on cloud platform |
| WO2017131977A1 (en) * | 2016-01-25 | 2017-08-03 | Microsoft Technology Licensing, Llc | Frame projection for augmented reality environments |
| CN107274472A (en) * | 2017-06-16 | 2017-10-20 | 福州瑞芯微电子股份有限公司 | A kind of method and apparatus of raising VR play frame rate |
| CN108810554A (en) * | 2018-06-15 | 2018-11-13 | 腾讯科技（深圳）有限公司 | Scene image transmission method, computer equipment and the storage medium of virtual scene |
| CN110351480A (en) * | 2019-06-13 | 2019-10-18 | 歌尔科技有限公司 | Image processing method, device and electronic equipment for electronic equipment |
-
2020
- 2020-05-29 CN CN202010472745.3A patent/CN111583350A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105263050A (en) * | 2015-11-04 | 2016-01-20 | 山东大学 | Mobile terminal real-time rendering system and method based on cloud platform |
| WO2017131977A1 (en) * | 2016-01-25 | 2017-08-03 | Microsoft Technology Licensing, Llc | Frame projection for augmented reality environments |
| CN107274472A (en) * | 2017-06-16 | 2017-10-20 | 福州瑞芯微电子股份有限公司 | A kind of method and apparatus of raising VR play frame rate |
| CN108810554A (en) * | 2018-06-15 | 2018-11-13 | 腾讯科技（深圳）有限公司 | Scene image transmission method, computer equipment and the storage medium of virtual scene |
| CN110351480A (en) * | 2019-06-13 | 2019-10-18 | 歌尔科技有限公司 | Image processing method, device and electronic equipment for electronic equipment |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022151972A1 (en) * | 2021-01-14 | 2022-07-21 | 华为云计算技术有限公司 | Video encoding method and apparatus, device, and storage medium |
| WO2023284504A1 (en) * | 2021-07-12 | 2023-01-19 | 华为技术有限公司 | Data processing method, system and electronic apparatus |
| WO2024002065A1 (en) * | 2022-06-30 | 2024-01-04 | 维沃移动通信有限公司 | Video encoding method and apparatus, electronic device, and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111627116B (en) | Image rendering control method and device and server | |
| KR102362001B1 (en) | Method and system for providing eye tracking based information about a user behavior, client device, server and computer program product | |
| KR102215166B1 (en) | Providing apparatus, providing method and computer program | |
| KR20220009393A (en) | Image-based localization | |
| CN111583350A (en) | Image processing method, device and system and server | |
| CN107911737B (en) | Media content display method and device, computing equipment and storage medium | |
| CN112738010A (en) | Data interaction method and system, interaction terminal and readable storage medium | |
| JP7200935B2 (en) | Image processing device and method, file generation device and method, and program | |
| CN103650001A (en) | Moving image distribution server, moving image playback device, control method, program, and recording medium | |
| CN111669561B (en) | Multi-angle free view image data processing method and device, medium and equipment | |
| CN111813689B (en) | Game testing method, apparatus and medium | |
| Zerman et al. | User behaviour analysis of volumetric video in augmented reality | |
| CN113965773A (en) | Live broadcast display method and device, storage medium and electronic equipment | |
| CN113220251A (en) | Object display method, device, electronic equipment and storage medium | |
| KR102503337B1 (en) | Image display method, apparatus and system | |
| CN109978945B (en) | Augmented reality information processing method and device | |
| CN108632661A (en) | Playback method and playing device | |
| CN115761190A (en) | Multi-user augmented reality photo browsing method and system based on scene mapping | |
| CN115131528A (en) | Virtual reality scene determination method, device and system | |
| EP4036858A1 (en) | Volumetric imaging | |
| EP3742404A1 (en) | Content coding system and method | |
| CN111669570B (en) | Multi-angle free view video data processing method and device, medium and equipment | |
| EP3716217A1 (en) | Techniques for detection of real-time occlusion | |
| KR20210055381A (en) | Apparatus, method and computer program for providing augmented reality contentes through smart display | |
| CN112738646A (en) | Data processing method, device, system, readable storage medium and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |