WO2022042413A1

WO2022042413A1 - Image reconstruction method and apparatus, and computer readable storage medium, and processor

Info

Publication number: WO2022042413A1
Application number: PCT/CN2021/113469
Authority: WO
Inventors: 盛骁杰
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2020-08-24
Filing date: 2021-08-19
Publication date: 2022-03-03
Also published as: CN114092315A

Abstract

Disclosed in the present invention are an image reconstruction method and apparatus, and a computer readable storage medium, and a processor. The method comprises: upon detecting that an interaction operation occurs on an operation interface, and obtaining a target position after displacement of a virtual viewpoint occurs on the operation interface; controlling transition of the virtual viewpoint from the target position to a predetermined position, wherein the predetermined position is a camera position that coincides with the position of the virtual viewpoint in spatial degree of freedom; and processing, by using an image deformation model, an image when the virtual viewpoint is located at the predetermined position, and obtaining a reconstructed image. The present invention solves the technical problem of poor timeliness in reconstructing an image at a virtual viewpoint.

Description

Method, apparatus, computer readable storage medium and processor for reconstructing an image

This application claims the priority of the Chinese Patent Application No. 202010857911.1 filed on August 24, 2020 and the invention title is "Method, Apparatus, Computer-readable Storage Medium and Processor for Reconstructing Images", the entire contents of which are incorporated by reference in this application.

technical field

The present invention relates to the field of computers, and in particular, to a method, an apparatus, a computer-readable storage medium and a processor for reconstructing an image.

Background technique

At present, the user uses gesture interaction to determine the virtual viewpoint position for viewing video images on the terminal. During the viewing process, due to the low computing resources of the terminal or the poor video quality, when the user stays on the virtual viewpoint for a long time to watch, the video is stuck, and the timeliness when reconstructing the image from the virtual viewpoint is poor.

For the above-mentioned technical problem of poor timeliness when reconstructing an image from a virtual viewpoint, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a method, an apparatus, a computer-readable storage medium, and a processor for reconstructing an image, so as to at least solve the technical problem of poor timeliness when reconstructing an image at a virtual viewpoint.

According to an aspect of the embodiments of the present invention, a method for reconstructing an image is provided. The method may include: detecting that an interactive operation occurs on the operation interface, and obtaining a target position after the virtual viewpoint is displaced on the operation interface; controlling the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a space free from the virtual viewpoint The position of the camera whose degree and position are coincident; use the image deformation model to process the image with the virtual viewpoint at the predetermined position, and obtain the reconstructed image.

According to another aspect of the embodiments of the present invention, another method for reconstructing an image is also provided. The method may include: detecting that an interactive operation of controlling the virtual viewpoint to move from a first position to a second position occurs on the operation interface; and controlling the virtual viewpoint to transition from the second position to a third position, wherein the third position is a camera deployed with a camera. The location of the true viewpoint; read the image of the camera at the third position; process the image using a plane-based image warping model to obtain a reconstructed image.

According to another aspect of the embodiments of the present invention, another method for reconstructing an image is also provided. The method may include: sensing the interactive operation of the operating object on the operating interface; responding to the interactive operation, displaying the displacement of the virtual viewpoint on the operating interface based on the interactive operation, wherein the displacement is moving from a first position to a second position; displaying The virtual viewpoint transitions from a second location to a third location, where the third location is the location of the real viewpoint where the camera is deployed; displaying a reconstructed image generated by processing the target image using a plane-based image warping model, where the target image is located at Image captured by the camera in the third position.

According to another aspect of the embodiments of the present invention, another method for reconstructing an image is also provided. The method may include: sensing the movement operation of the operation object on the operation interface; if the suspension of the movement operation is sensed, obtaining the stop position of the operation object on the operation interface when the movement operation is stopped; controlling the virtual viewpoint located at the stop position to transition to a predetermined position, Wherein, the predetermined position is the position of the camera that coincides with the position of the spatial degrees of freedom of the virtual viewpoint; the image of the camera at the predetermined position is processed using a plane-based image deformation model to obtain a reconstructed image.

According to another aspect of the embodiments of the present invention, another method for reconstructing an image is also provided. The method may include: sensing the movement operation of the operation object on the operation interface; if the suspension of the movement operation is sensed, obtaining the stop position of the operation object on the operation interface when the movement operation is stopped; controlling the virtual viewpoint located at the stop position to transition to a predetermined position, The predetermined position is the position of the real viewpoint where the camera is deployed; the image of the camera located at the predetermined position is read; the image is processed using a plane-based image deformation model to obtain a reconstructed image.

According to another aspect of the embodiments of the present invention, another method for reconstructing an image is also provided. The method may include: during the live broadcast, detecting that a moving operation is received on the live screen; acquiring a target position after the virtual viewpoint is displaced on the live screen; controlling the virtual viewpoint to move from the target position to a predetermined position, wherein the predetermined position is The camera position coincides with the position of the spatial degrees of freedom of the virtual viewpoint; the image deformation model is used to process the image of the virtual viewpoint at the predetermined position, and the reconstructed image is obtained.

According to another aspect of the embodiments of the present invention, an apparatus for reconstructing an image is also provided. The device may include: a first acquisition module for detecting an interactive operation on the operation interface, and acquiring a target position after the virtual viewpoint is displaced on the operation interface; a first control module for controlling the virtual viewpoint to transition from the target position to the The predetermined position, wherein the predetermined position is the camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint; the second processing module is used for using the image deformation model to process the image of the virtual viewpoint at the predetermined position to obtain the reconstructed image.

According to another aspect of the embodiments of the present invention, another apparatus for reconstructing an image is also provided. The device may include: a detection module for detecting that an interactive operation of controlling the virtual viewpoint to move from the first position to the second position occurs on the operation interface; and a second control module for controlling the virtual viewpoint to transition from the second position to the third position position, where the third position is the position of the real viewpoint where the camera is deployed; the first reading module is used to read the image of the camera located at the third position; the second processing module is used to use the plane-based image warping model Process the image to obtain a reconstructed image.

According to another aspect of the embodiments of the present invention, another apparatus for reconstructing an image is also provided. The device may include: a first sensing module for sensing the movement operation of the operation object on the operation interface; a second acquisition module for obtaining the stoppage of the operation object on the operation interface when the movement operation is stopped if it is sensed that the movement operation is stopped position; a third control module for controlling the virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is a camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint; a third processing module for using a plane-based The image warping model processes the image of the camera at the predetermined position and obtains the reconstructed image.

According to another aspect of the embodiments of the present invention, another apparatus for reconstructing an image is also provided. The device may include: a second sensing module for sensing the movement operation of the operation object on the operation interface; a third acquisition module for obtaining the stay of the operation object on the operation interface when the movement operation is stopped if it is sensed that the movement operation is stopped position; a fourth control module for controlling the virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is the position of the real viewpoint where the camera is deployed; a second reading module for reading the camera at the predetermined position The fourth processing module is used to process the image by using the plane-based image deformation model to obtain the reconstructed image.

According to another aspect of the embodiments of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored program, wherein when the program is executed by the processor, the device where the computer-readable storage medium is located is controlled to execute the method for reconstructing an image according to the embodiment of the present invention.

According to another aspect of the embodiments of the present invention, a processor is also provided. The processor is used for running a program, wherein the method for reconstructing an image according to the embodiment of the present invention is executed when the program is running.

According to another aspect of the embodiments of the present invention, a system for reconstructing an image is also provided. The system may include: a processor; a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: detecting that an interactive operation occurs on the operation interface, and acquiring a target after the virtual viewpoint is displaced on the operation interface position; control the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is the camera position that coincides with the position of the spatial degrees of freedom of the virtual viewpoint; use the image deformation model to process the image of the virtual viewpoint at the predetermined position to obtain the reconstructed image.

In the embodiment of the present invention, an interactive operation is detected on the operation interface to obtain the target position after the virtual viewpoint is displaced on the operation interface; the virtual viewpoint is controlled to transition from the target position to a predetermined position, wherein the predetermined position is the same as the virtual viewpoint. The position of the camera with the spatial degrees of freedom coincides with the position of the camera; the image deformation model is used to process the image with the virtual viewpoint at the predetermined position, and the reconstructed image is obtained. That is to say, the present application obtains the target position after the virtual viewpoint is displaced on the operation interface, and when the virtual viewpoint transitions from the target position to the predetermined position, the image deformation model can be used to process the image of the virtual viewpoint at the predetermined position, so as to achieve In order to obtain the reconstructed image, because the calculation process of the image deformation model is simple, the calculation speed is block, and it adapts to the computing resources of the low-end computer, it solves the technical problem of poor timeliness when reconstructing the image at the virtual viewpoint, and achieves the improvement of the virtual image. The technical effect of the timeliness when reconstructing images from viewpoints.

Description of drawings

The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the attached image:

1A is a block diagram of a hardware structure of a computer terminal (or mobile device) for implementing a method for reconstructing an image according to an embodiment of the present invention;

1B is a schematic structural diagram of a reconstructed image in a specific application scenario according to an embodiment of the present invention;

2 is a flowchart of a method for reconstructing an image according to an embodiment of the present invention;

3 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention;

4 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention;

5 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention;

6 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention;

7 is a schematic diagram of a DIBR virtual viewpoint difference according to the related art;

8 is a schematic diagram of virtual viewpoint interpolation based on Warping and DIBR according to an embodiment of the present invention;

9 is a flowchart of a method for virtual viewpoint interpolation based on Warping and DIBR according to an embodiment of the present invention;

10 is a schematic diagram of an apparatus for reconstructing an image according to an embodiment of the present invention;

11 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention;

12 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention;

FIG. 13 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention; and

FIG. 14 is a structural block diagram of a computer terminal according to an embodiment of the present invention.

detailed description

In order to make those skilled in the art better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only Embodiments are part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

First of all, some nouns or terms that appear in the process of describing the embodiments of the present application are suitable for the following explanations:

Depth Image Based Rendering (DIBR for short), first uses the depth information to project the reference image to the 3D Euclidean space, and then projects the 3D space points to the imaging plane of the virtual camera;

Warping is an operation to deform a two-dimensional image according to the three-dimensional information of the space;

An image deformation model, which is a model for processing a two-dimensional image by using an image deformation technology, wherein the image deformation technology includes the above deformation operation;

For free viewpoint videos, in order to provide a high degree of freedom viewing experience, users can adjust the viewing angle through interactive means during viewing, and watch from the free viewpoint they want to watch;

The 6DoF parameter refers to six degrees of freedom in directions, specifically, translation along three directions and rotation parameters around three axes.

Example 1

According to an embodiment of the present invention, an embodiment of a method for reconstructing an image is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and , although a logical order is shown in the flowcharts, in some cases steps shown or described may be performed in an order different from that herein.

The method embodiment provided in Embodiment 1 of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. 1A is a block diagram of a hardware structure of a computer terminal (or mobile device) for implementing a method for reconstructing an image according to an embodiment of the present invention. As shown in FIG. 1A , the computer terminal 10 (or mobile device 10 ) may include one or more processors 102 (represented by 102a, 102b, . processing means such as a processor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission means 106 for communication functions. In addition, may also include: display, input/output interface (I/O interface), universal serial bus (USB) port (may be included as one of the ports of the I/O interface), network interface, power supply and/or camera. Those of ordinary skill in the art can understand that the structure shown in FIG. 1A is only a schematic diagram, which does not limit the structure of the above electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in FIG. 1A , or have a different configuration than that shown in FIG. 1A .

It should be noted that the one or more processors 102 and/or other data processing circuits described above may generally be referred to herein as "data processing circuits." The data processing circuit may be embodied in whole or in part as software, hardware, firmware or any other combination. Furthermore, the data processing circuitry may be a single stand-alone processing module, or incorporated in whole or in part into any of the other elements in the computer terminal 10 (or mobile device). As referred to in the embodiments of the present application, the data processing circuit acts as a kind of processor control (eg, selection of a variable resistance termination path connected to an interface).

The memory 104 can be used to store software programs and modules of application software, such as a program instruction/data storage device corresponding to the method for reconstructing an image in the embodiment of the present invention. The processor 102 runs the software programs and modules stored in the memory 104, thereby Execute various functional applications and data processing, that is, realize the method of reconstructing images of the above-mentioned application programs. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, memory 104 may further include memory located remotely from processor 102, which may be connected to computer terminal 10 through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

Transmission means 106 are used to receive or transmit data via a network. A specific example of the above-mentioned network may include a wireless network provided by a communication provider of the computer terminal 10 . In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Radio Frequency, RF) module, which is used for wirelessly communicating with the Internet.

The display may be, for example, a touch screen type liquid crystal display (LCD) that enables a user to interact with the user interface of the computer terminal 10 (or mobile device).

It should be noted here that, in some optional embodiments, the computer device (or mobile device) shown in FIG. 1A may include hardware elements (including circuits), software elements (including a computer stored on a computer-readable medium) code), or a combination of both hardware and software elements. It should be noted that FIG. 1A is only one example of a specific embodiment, and is intended to illustrate the types of components that may be present in a computer device (or mobile device) as described above.

1B is a schematic structural diagram of a reconstructed image in a specific application scenario according to an embodiment of the present invention, which shows an arrangement scene of a system 20 for reconstructing an image. The system 20 for reconstructing an image may include a plurality of acquisition devices. The acquisition array 21 , the data processing device 22 , the server cluster 23 in the cloud (may include: server 231 , server 232 , server 233 , server 234 ), playback control device 24 , playback terminal 25 and interactive terminal 26 . The system 20 for reconstructing an image can acquire the target position after the virtual viewpoint is displaced on the operation interface, and when the virtual viewpoint transitions from the target position to the predetermined position, the image deformation model can be used to process the image in which the virtual viewpoint is located at the predetermined position , so as to achieve the purpose of obtaining reconstructed images.

Specifically, referring to FIG. 1B , the acquisition array 21 may include a plurality of cameras, which may be fan-shaped and placed at different positions in the field acquisition area according to a preset multi-angle free viewing angle range.

The data processing device 22 can respectively send instructions to each camera in the acquisition array 21 through a wireless local area network, and each acquisition device in the acquisition array 21 transmits the obtained position of the camera based on the instructions sent by the data processing device 22 . to the data processing device 22 .

The interactive terminal 26 in this embodiment determines the target position after the virtual viewpoint is displaced on the operation interface based on the interactive operation. When the data processing device 22 detects that an interactive operation occurs on the operation interface of the interactive terminal 26, the interactive terminal can be obtained. 26. The target position after the virtual viewpoint sent on the operation interface is displaced, and the virtual viewpoint is controlled to transition from the target position to a predetermined position. The image with the virtual viewpoint at the predetermined position is processed, the reconstructed image is obtained, and the obtained reconstructed image can be uploaded to the server cluster 23 in the cloud, and the server cluster 23 can send the reconstructed image to the interactive terminal 26 for display.

As another optional implementation, when the data processing device 22 detects that an interactive operation occurs on the operation interface of the interactive terminal 26, obtains the target position of the virtual viewpoint sent by the interactive terminal 26 after the displacement on the operation interface, and uses The target position is uploaded to the server cluster 23 in the cloud, and the virtual viewpoint is controlled to transition from the target position to the predetermined position through the server cluster 23, and then the image with the virtual viewpoint located at the predetermined position is processed by the image deformation model, and the reconstructed image is obtained, and the obtained data can be obtained. The reconstructed image is sent to the interactive terminal 26 for display.

Then, the playback control device 24 may receive the reconstructed image sent by the server cluster 23, and the playback terminal 25 receives the reconstructed image from the playback control device 24 and plays the reconstructed image in real time. The playback control device 24 may be a manual playback control device or a virtual playback control device. In a specific implementation, a broadcast director control device, such as a broadcast director station, may be used as a playback control device in the embodiment of the present invention.

In this embodiment, the reconstructed image may be a reconstructed image in a video. Generally speaking, the entities in the video will not be completely static. For example, using the above-mentioned reconstructed image system, the reconstructed image will also change continuously over time. change.

This embodiment adopts the above-mentioned system for reconstructing an image. On the one hand, the user can directly view the reconstructed image through the playback terminal 25; To control the transition of the virtual viewpoint from the target position to the predetermined position, use the image deformation model to process the image with the virtual viewpoint located at the predetermined position, and obtain the reconstructed image. It can be understood that, the above system 20 for reconstructing an image may also include only the playback terminal 25 or only the interaction terminal 26, or use the same terminal device as the playback terminal 25 and the interaction terminal 26.

It can be understood by those skilled in the art that the user uses gesture interaction to determine the virtual viewpoint position for viewing the video image on the terminal. During the viewing process, due to the low computing resources of the terminal or the poor video quality, when the user stays on the virtual viewpoint for a long time to watch, the video is stuck, and the timeliness when reconstructing the image from the virtual viewpoint is poor. Therefore, how to ensure the timeliness when reconstructing images using virtual viewpoints has become a difficult problem to solve.

In view of this, an embodiment of the present specification provides a solution. Under the operating environment shown in FIG. 1A or FIG. 1B , the present application provides a method for reconstructing an image as shown in FIG. 2 . It should be noted that, the method for reconstructing an image in this embodiment may be performed by the mobile terminal of the embodiment shown in FIG. 1A or the system for reconstructing an image shown in FIG. 1B .

FIG. 2 is a flowchart of a method for reconstructing an image according to an embodiment of the present invention. As shown in Figure 2, the method may include the following steps:

In step S202, an interactive operation is detected on the operation interface, and a target position after the virtual viewpoint is displaced on the operation interface is acquired.

In the technical solution provided by the above step S202 of the present invention, the interactive operation may be the interaction performed by the terminal user for the virtual viewpoint, and may also be called user free viewpoint interaction (DIBR), wherein the interactive operation occurs on the operation interface. , the virtual viewpoint will be displaced on the operation interface, therefore, the above-mentioned interactive operation may be a movement operation for the virtual viewpoint, for example, a movement operation starting from point A on the operation interface.

In this embodiment, an interactive operation can be detected on the operation interface, and then the target position where the virtual viewpoint stays after the displacement on the operation interface can be obtained. For example, the target position is point B on the operation interface, and the interactive operation stops at point B. It is the operation that the user's finger lifts up the moment it reaches point B, then the displacement of the virtual viewpoint on the operation interface is the displacement between point A and point B.

Step S204 , controlling the virtual viewpoint to transition from the target position to a predetermined position, where the predetermined position is a camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint.

In the technical solution provided by the above step S204 of the present invention, after obtaining the target position of the virtual viewpoint displaced on the operation interface, the target position does not necessarily correspond to the real camera position, so that image reconstruction cannot be performed directly at the target position . In this embodiment, the virtual viewpoint can be controlled to transition from the target position to the predetermined position, and the virtual viewpoint can be transitioned from the stayed target position along the interactive inertia according to a certain path until the transitioned position coincides with the position of the spatial degrees of freedom of the virtual viewpoint. , the position coincident with the position of the spatial degree of freedom of the virtual viewpoint is the predetermined position. It should be noted that the above-mentioned predetermined position in this embodiment may be a position where a camera is deployed, wherein the camera may be a real camera, that is, the predetermined position in this embodiment may be a real camera position, which may also be called a real viewpoint Location. Optionally, the predetermined position in this embodiment is a real viewpoint position closest to the target position.

In this embodiment, after acquiring the target position of the virtual viewpoint after the displacement on the operation interface, the virtual viewpoint is no longer controlled by the interactive operation, so this embodiment controls the virtual viewpoint to transition from the target position to the predetermined position, so The interactive inertia involved can be understood as a rule for transitioning the virtual viewpoint, which can also be called inertial transition, which conforms to the user's interaction habits and experience. For example, the virtual viewpoint is controlled to slide from the target position to a predetermined position at a certain speed and then stop. , the predetermined position may be a preferred position to which the control target position transitions on the operation interface, or a position that conforms to the user's interaction habits and experience.

In an optional embodiment shown in FIG. 7 and FIG. 8, the above-mentioned embodiment will be described to a certain extent. The black solid points (such as A1 and A2) can represent virtual viewpoints, and the white hollow positions (such as B1, A2, etc.) B2, B3, etc.) can represent the position of the real viewpoint where the real camera is located. After the movement operation occurs on the operation interface, after the virtual viewpoint is moved from A1 to A2, the virtual viewpoint at the A2 position will select a real viewpoint where the real camera is located based on the predetermined control mode, and move to this real viewpoint. At this time, The control mode can include any one of the following: select the real viewpoint from the virtual viewpoint A2; select a real viewpoint with free computing resources and the closest to the virtual viewpoint; when the camera where the selected real viewpoint is located is damaged, waiting for processing time When any one or more of the situations in the predetermined duration exceed, select the real viewpoint with the highest preset weight value.

Optionally, the above-mentioned certain path in this embodiment may be a predetermined smooth path. The above-mentioned spatial degrees of freedom positions may be three spatial degrees of freedom positions (x, y, z).

Step S206 , using the image deformation model to process the image with the virtual viewpoint located at the predetermined position to obtain the reconstructed image.

In the technical solution provided in the above step S206 of the present invention, the image warping model may also be called a Warping model, which can be used to process the two-dimensional image by using the image warping technology, that is, the two-dimensional image is processed according to the three-dimensional information of the space. Deformation operations. In this embodiment, after the virtual viewpoint transitions to the predetermined position, there is an image with the virtual viewpoint at the predetermined position. In this embodiment, in the image reconstruction scene, the above-mentioned image deformation model can be used to process the virtual viewpoint at the above-mentioned predetermined position. image to obtain a reconstructed image.

In this embodiment, the calculation process of the above-mentioned image deformation model is simple, so this embodiment is faster than the interpolation method using depth map-based virtual viewpoint reconstruction (DIBR), adapts to the computing resources of low-end computers, and ensures the virtual viewpoint Timeliness (real-time) when reconstructing images, and image quality in complex image reconstruction scenarios.

Through the above-mentioned steps S204 to S206 of the present application, an interactive operation on the operation interface is detected, and the target position after the virtual viewpoint is displaced on the operation interface is obtained; the virtual viewpoint is controlled to transition from the target position to a predetermined position, wherein the predetermined position is the same as the target position. The position of the camera where the spatial degrees of freedom of the virtual viewpoint coincide; the image deformation model is used to process the image with the virtual viewpoint at a predetermined position, and the reconstructed image is obtained. That is to say, in this embodiment, the target position after the virtual viewpoint is displaced on the operation interface is obtained. When the virtual viewpoint transitions from the target position to the predetermined position, the image deformation model can be used to process the image in which the virtual viewpoint is located at the predetermined position. The purpose of obtaining the reconstructed image is achieved. Because the calculation process of the image deformation model is simple, the calculation speed is block, and it adapts to the computing resources of the low-end computer, it solves the technical problem of poor timeliness when reconstructing the image at the virtual viewpoint, and achieves the improvement of The technical effect of time-sensitiveness when reconstructing images from virtual viewpoints.

The above method of this embodiment will be further described below.

As an optional implementation manner, if it is detected that the interaction operation at the target position is interrupted, a transition operation of transitioning the virtual viewpoint from the target position to the predetermined position is started.

In this embodiment, the interactive operation occurring on the operation interface may be a continuous operation process. During the displacement of the virtual viewpoint on the operation interface, it is detected that an interactive operation occurs on the operation interface. If it is detected that the virtual viewpoint moves to the target position, the interactive operation is interrupted, for example, the interactive operation stays at the target position. At this point, the transition operation of executing the transition of the virtual viewpoint from the target position to the predetermined position is started, that is, once the interruption of the interactive operation at the target position is detected, the execution of the above transition operation is triggered.

As an optional implementation manner, in step S204, in the process of controlling the virtual viewpoint to transition from the target position to the predetermined position, the method further includes: using the DIBR virtual viewpoint interpolation algorithm to generate interpolation of the transition image frame; based on the transition image frame interpolation to reconstruct the image during the transition.

In this embodiment, in the middle process of controlling the virtual viewpoint to transition from the target position to the predetermined position, the DIBR virtual viewpoint interpolation algorithm may be used to generate the interpolation of the transition image frame until the virtual viewpoint transitions from the target position to the predetermined position, for example, until The spatial three-DOF position (x, y, z) of the virtual viewpoint coincides with the real camera position. Among them, after the spatial three-degree-of-freedom position (x, y, z) of the virtual viewpoint coincides with the real camera position, the difference between the interpolated planned 6DoF position and the real camera position is only in the remaining three rotational degrees of freedom (respectively are the degrees of freedom of rotation around the x, y, and z axes).

After the interpolation of the transition image frame is generated, an image during the transition of the virtual viewpoint from the target position to the predetermined position may be reconstructed based on the interpolation of the transition image frame.

As an optional implementation manner, in step S204, after the virtual viewpoint is controlled to transition from the target position to the predetermined position, the Warping image deformation algorithm is invoked to switch to use the image deformation model for image reconstruction.

In this embodiment, the image warping algorithm can be the Warping image warping algorithm, which can also be called Warping operation. Because of its simple calculation, only one projection needs to be performed, and there are no other pre- and post-processing operations. Therefore, it is more efficient than DIBR. Interpolation is fast. In this embodiment, after controlling the virtual viewpoint to transition from the target position to the predetermined position, the Warping image warping algorithm may be called, that is, the DIBR virtual viewpoint interpolation algorithm is switched to the Warping image warping algorithm, so that the image warping model is used for image reconstruction. It is because of the above-mentioned characteristics of the operation of Warping, which ensures the timeliness when reconstructing images from virtual viewpoints and the image quality in complex image reconstruction scenarios.

This embodiment provides the DIBR method combined with the Warping function, which can support richer terminal interaction design, and can flexibly configure different interaction methods in combination with actual terminal and scene conditions, thereby creating a good experience for users.

As an optional implementation manner, using an image deformation model to process an image where the virtual viewpoint is located at a predetermined position includes: reading a corresponding image from a camera based on the predetermined position where the virtual viewpoint is located; according to the space where the virtual viewpoint is located Rotational degrees of freedom on the coordinate system, using the image warping model to process the read image.

In this implementation, when the image deformation model is used to process the image in which the virtual viewpoint is located at the predetermined position, the corresponding image may be read from the camera based on the predetermined position of the virtual viewpoint, and the corresponding image is read from the camera. The image of the virtual viewpoint is located at a predetermined position, which can also be called the original camera image, and then determine the spatial coordinate system where the virtual viewpoint is located, and determine the rotational degrees of freedom on the spatial coordinate system, for example, three rotational degrees of freedom , according to the rotational degrees of freedom on the spatial coordinate system where the virtual viewpoint is located, use the image deformation model to process the read image, and then update the next viewpoint reconstruction function, so that the timeliness problem of terminal reconstruction DIBR can be solved at the same time, and Image quality issues in complex reconstruction scenarios. At the same time, this embodiment can solve the problem of smoothness when the user switches between free viewpoints through the inertial DIBR interpolation transition from the virtual viewpoint position interacted by the user to the real camera position.

As an optional implementation manner, the spatial degree of freedom is the coordinate value of the virtual viewpoint on the spatial coordinate system, and the rotational degree of freedom is the degree of freedom of rotating around the coordinate axis of the spatial coordinate system.

In the above solution of this embodiment, the spatial degree of freedom is the coordinate value of the virtual viewpoint on the spatial coordinate system, for example, (x, y, z), and the rotational degree of freedom is the degree of freedom to rotate around the coordinate axis of the spatial coordinate system , for example, there are 3 degrees of freedom, which can be rotation degrees of freedom around the three axes of x, y, and z respectively.

As an optional implementation manner, in step S204, before controlling the virtual viewpoint to transition from the target position to the predetermined position, the method further includes: acquiring the distance between the target position where the virtual viewpoint is currently located and the predetermined position; if the distance exceeds a threshold value , then control the virtual viewpoint to move from the target position to the viewpoint position where the camera with the highest priority is located according to the preset camera priority; if the distance does not exceed the threshold, execute the step of transitioning the virtual viewpoint from the target position to the predetermined position.

In this embodiment, before controlling the virtual viewpoint to transition from the target position to the predetermined position, the distance between the current target position after the virtual viewpoint is displaced on the operation interface and the predetermined position can also be obtained, and the predetermined position is The location where the camera is deployed. Then it is judged whether the distance between the target position and the predetermined position exceeds the threshold value, and if it is judged that the distance between the target position and the predetermined position exceeds the threshold value, the virtual viewpoint transition from the target position to the predetermined position is not executed, but the preset The camera priority is controlled to control the virtual viewpoint to move from the target position to the viewpoint position where the camera with the highest priority is located according to the above-mentioned preset camera priority; If the distance does not exceed the threshold, you can continue to perform the transition of the virtual viewpoint from the target position to the predetermined position, which can be to transition the virtual viewpoint along the interactive inertia according to a certain path from the stayed target position until the transition to the space between the position and the virtual viewpoint The position of the degrees of freedom coincides, and the position that coincides with the position of the spatial degrees of freedom of the virtual viewpoint is also the predetermined position, so that this embodiment achieves the control of the virtual viewpoint from the target position according to the distance between the target position where the virtual viewpoint is currently located and the predetermined position. The purpose of the location to which the location transitions.

As an optional implementation manner, in step S202, after detecting that an interactive operation occurs on the operation interface, the method further includes: popping up prompt information for indicating at least one selection control; selecting the reconstructed image by triggering any selection control display resolution.

In this embodiment, the interaction operation may be the interaction performed by the end user for the virtual viewpoint, in order to make the user more convenient to deal with the acquired reconstructed image and set to meet the requirements, for example, in order to make the user more convenient to deal with the acquired reconstructed image The display resolution of the obtained reconstructed image is set to meet the requirements for the resolution of the reconstructed image to be obtained. In this embodiment, after an interactive operation is detected on the operation interface, a pop-up is used to indicate at least one selection control. The prompt information, wherein the selection control may be a functional control for selecting the resolution of the reconstructed image to be acquired, and the prompt information is also interactive indication information to prompt the user to perform a selection operation.

After popping up the prompt information indicating at least one selection control, the user's selection operation on any selection control can be received, and then in response to the selection operation on any selection control, the display resolution of the reconstructed image is selected, wherein each Each selection control may correspond to the display resolution of one reconstructed image, and multiple selection controls may correspond to different display resolutions of the reconstructed image respectively, so as to meet the requirement of diverse settings for the display resolution of the reconstructed image to be acquired, for example, In the scenario where the user is watching a movie, there will be a reminder of the low display resolution of the reconstructed image. In this way, when the image deformation model is used to process the image with the virtual viewpoint located at the predetermined position, the obtained reconstructed image is the display resolution corresponding to the selection control triggered by the user.

The embodiment of the present invention also provides another method for reconstructing an image.

FIG. 3 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention. As shown in Figure 3, the method may include the following steps:

Step S302, it is detected that an interactive operation of controlling the virtual viewpoint to move from the first position to the second position occurs on the operation interface.

In the technical solution provided by the above step S302 of the present invention, the interactive operation may be an interaction performed by the user with respect to the virtual viewpoint. The virtual viewpoint is controlled to move from the first position to the second position, where the second position may be the position where the virtual viewpoint stays after the displacement on the operation interface, for example, the position where the user's finger is lifted on the operation interface.

Step S304, controlling the virtual viewpoint to transition from the second position to the third position, where the third position is the position of the real viewpoint where the camera is deployed.

In the technical solution provided in the above step S304 of the present invention, after an interactive operation of controlling the virtual viewpoint to move from the first position to the second position is detected on the operation interface, the virtual viewpoint can be controlled to transition from the second position to the third position, It can be that the virtual viewpoint transitions from the second position where it stayed along a certain path along the interactive inertia until it transitions to a third position, where the real viewpoint of the camera can be deployed, which is different from the position of the spatial degrees of freedom of the virtual viewpoint. coincide.

Optionally, the above-mentioned certain path in this embodiment may be a predetermined smooth path. The spatial DOF position may be a spatial three DOF position (x, y, z).

Step S306, read the image of the camera located at the third position.

In the technical solution provided by the above step S306 of the present invention, after controlling the virtual viewpoint to transition from the second position to the third position, the image of the camera located at the third position can be read.

In this embodiment, after the virtual viewpoint transitions to the third position, there is an image of the camera whose virtual viewpoint is at the third position, and the image of the camera at the third position is read.

Step S308, using the plane-based image deformation model to process the image to obtain a reconstructed image.

In the technical solution provided in the above step S308 of the present invention, after the image of the camera located at the third position is read, the image can be processed using a plane-based image deformation model to obtain a reconstructed image.

In this embodiment, after the virtual viewpoint transitions to the third position and the image of the camera located at the third position is read, a plane-based image deformation model can be used in the image reconstruction scene to process the image of the third position of the virtual viewpoint. The image of the camera is obtained, and then the reconstructed image is obtained, and the image deformation model of the plane is a two-dimensional image deformation model.

In this embodiment, the calculation process of the above-mentioned plane-based image deformation model is simple, so this embodiment is faster than the interpolation method using the DIBR based on the depth map, thus adapting to the computing resources of low-end computers, It ensures the timeliness of image reconstruction from virtual viewpoints and the image quality in complex image reconstruction scenarios.

As an optional implementation manner, the position of the real viewpoint closest to the second position is selected as the third position.

In this embodiment, before controlling the virtual viewpoint to transition from the second position to the third position, at least one real viewpoint position may be acquired, and then from the at least one real viewpoint position, the closest real viewpoint position to the second position is determined, And use it as the third position that needs to control the transition of the virtual viewpoint from the second position to.

FIG. 4 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention. As shown in Figure 4, the method may include the following steps:

Step S402, an interactive operation of the operation object on the operation interface is sensed.

In the technical solution provided in the above step S402 of the present invention, the interactive operation may be the interaction of the virtual viewpoint performed by the operation object. During the interactive operation on the operation interface, the virtual viewpoint will be displaced on the operation interface, wherein , the operation object can be the user. This embodiment senses the above-mentioned interactive operation of the operation object on the operation interface.

Step S404 , in response to the interactive operation, display the displacement of the virtual viewpoint on the operation interface based on the interactive operation, wherein the displacement is moving from the first position to the second position.

In the technical solution provided in the above step S404 of the present invention, the virtual viewpoint can be displaced on the operation interface based on the interactive operation. In this embodiment, after sensing the interactive operation of the operation object on the operation interface, the virtual viewpoint can be displayed in response to the interactive operation. The displacement occurs on the operation interface based on the interactive operation, wherein the displacement is the displacement from the first position to the second position.

In this embodiment, in response to the above-mentioned interactive operation, the second position where the virtual viewpoint stays after the displacement from the first position on the operation interface can be obtained, and the displacement between the first position and the second position can be determined, which is also an interactive operation. The displacement occurred on the operation interface, and then the displacement occurred in the operation interface of the interactive operation is displayed.

Step S406, displaying that the virtual viewpoint transitions from the second position to the third position, where the third position is the position of the real viewpoint where the camera is deployed.

In the technical solution provided by the above step S406 of the present invention, after the displacement of the interactive operation on the operation interface is displayed, the displayed virtual viewpoint transitions from the second position to the third position.

In this embodiment, the virtual viewpoint can be controlled to transition from the second position to the third position, and the virtual viewpoint can be transitioned from the second position where it stayed along the interactive inertia according to a certain path until it transitions to the third position, The third position may be deployed with the real viewpoint of the camera, which coincides with the position of the spatial degree of freedom of the virtual viewpoint, thereby displaying the result of the transition of the virtual viewpoint from the second position to the third position.

Step S408 , displaying a reconstructed image generated by processing the target image using the plane-based image deformation model, where the target image is an image captured by a camera located at a third position.

In the technical solution provided by the above step S408 of the present invention, after the displayed virtual viewpoint transitions from the second position to the third position, the reconstructed image generated by processing the target image using the plane-based image deformation model is displayed.

In this embodiment, after the virtual viewpoint transitions to the third position, there is an image of the camera with the virtual viewpoint at the third position, the target image captured by the camera at the third position is read, and the plane-based The image warping model processes the above target image, acquires and displays the reconstructed image.

In this embodiment, the calculation process of the above-mentioned plane-based image deformation model is simple, so this embodiment is faster than the interpolation method using DIBR based on depth map, thus adapting to the computing resources of low-end computers and ensuring virtual viewpoint reconstruction The timeliness of images, and the image quality in complex image reconstruction scenarios.

FIG. 5 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention. As shown in Figure 5, the method may include the following steps:

Step S502, sensing the movement operation of the operation object on the operation interface.

In the technical solution provided by the above step S502 of the present invention, the movement operation may be a movement operation performed by the operation object for the virtual viewpoint. During the movement operation on the operation interface, the virtual viewpoint will occur and move on the operation interface. The displacement corresponding to the trajectory of the operation, wherein the operation object may be the user's finger, and the movement operation may be the sliding operation of the user's finger. This embodiment senses the above-mentioned movement operation of the operation object on the operation interface.

Step S504, if it is sensed that the movement operation is suspended, obtain the stop position where the operation object stays on the operation interface when the movement operation is suspended.

In the technical solution provided by the above step S504 of the present invention, after sensing the movement operation of the operation object on the operation interface, if the suspension of the movement operation is sensed, the stop position of the operation object on the operation interface when the movement operation is stopped is obtained.

In this embodiment, the moving operation that occurs on the operation interface is a continuous operation process. If the suspension of the moving operation is sensed, the stop position of the operation object on the operation interface when the moving operation is stopped can be obtained, that is, the moving operation An interruption occurred at the above stop position.

Step S506 , controlling the virtual viewpoint located at the stop position to transition to a predetermined position, where the predetermined position is a camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint.

In the technical solution provided by the above step S506 of the present invention, after obtaining the stop position where the operation object stays on the operation interface when the moving operation is suspended, the virtual viewpoint at the stop position can be controlled to transition to a predetermined position.

In this embodiment, the virtual viewpoint moves with the moving operation, and after acquiring the stop position where the operation object stays on the operation interface when the moving operation is stopped, the virtual viewpoint at the stop position can be triggered to transition to a predetermined position, the predetermined position The camera position that is coincident with the position of the spatial degrees of freedom of the virtual viewpoint may be the real camera position, and may also be referred to as the real viewpoint position.

Step S508 , using the plane-based image deformation model to process the image of the camera at the predetermined position to obtain a reconstructed image.

In the technical solution provided in the above step S508 of the present invention, after the virtual viewpoint at the stop position is controlled to transition to a predetermined position, a plane-based image deformation model can be used to process the image of the camera at the predetermined position to obtain a reconstructed image.

In this embodiment, after the virtual viewpoint transitions to the predetermined position, there is an image of the camera with the virtual viewpoint at the predetermined position, the image captured by the camera at the third position is read, and the plane-based image deformation model is used Process this image to obtain a reconstructed image.

FIG. 6 is a flowchart of another method for reconstructing an image according to an embodiment of the present invention. As shown in Figure 6, the method may include the following steps:

Step S602, sensing the movement operation of the operation object on the operation interface.

In the technical solution provided in the above step S602 of the present invention, the movement operation may be a movement operation performed by the operation object for the virtual viewpoint. During the movement operation on the operation interface, the virtual viewpoint will occur and move on the operation interface. The displacement corresponding to the trajectory of the operation, where the operation object may be the user's finger, and the movement operation may be the user's sliding operation. This embodiment senses the above-mentioned movement operation of the operation object on the operation interface.

Step S604, if it is sensed that the movement operation is suspended, obtain the stop position where the operation object stays on the operation interface when the movement operation is suspended.

In the technical solution provided by the above step S604 of the present invention, after sensing the movement operation of the operation object on the operation interface, if the suspension of the movement operation is sensed, the stop position of the operation object on the operation interface when the movement operation is stopped is obtained.

In this embodiment, the movement operation that occurs on the operation interface is a continuous operation process. If the suspension of the movement operation is sensed, the stop position of the operation object on the operation interface when the movement operation is stopped can be obtained, that is, the movement operation An interruption occurred at the above stop position.

Step S606, controlling the virtual viewpoint located at the staying position to transition to a predetermined position, where the predetermined position is the position of the real viewpoint where the camera is deployed.

In the technical solution provided by the above step S606 of the present invention, after obtaining the stop position where the operation object stays on the operation interface when the moving operation is suspended, the virtual viewpoint at the stop position is controlled to transition to a predetermined position.

In this embodiment, the virtual viewpoint moves with the moving operation, and after acquiring the stop position where the operation object stays on the operation interface when the moving operation is stopped, the virtual viewpoint at the stop position can be triggered to transition to a predetermined position, the predetermined position is the position of the real viewpoint where the camera is deployed, the position of the camera that can coincide with the position of the spatial degrees of freedom of the virtual viewpoint.

In step S608, the image of the camera located at the predetermined position is read.

In the technical solution provided in the above step S608 of the present invention, after controlling the virtual viewpoint located at the stop position to transition to the predetermined position, the image of the camera located at the predetermined position is read.

In this embodiment, after the virtual viewpoint transitions to the predetermined position, there is an image of the camera whose virtual viewpoint is located at the predetermined position, and the image of the camera located at the predetermined position is read.

Step S610, using a plane-based image deformation model to process the image to obtain a reconstructed image.

In the technical solution provided by the above step S610 of the present invention, after the image of the camera located at the predetermined position is read, the image is processed using a plane-based image deformation model to obtain a reconstructed image.

In this embodiment, after the virtual viewpoint transitions to a predetermined position and the image of the camera at the predetermined position is read, the image of the camera at the predetermined position of the virtual viewpoint can be processed using a plane-based image deformation model in the image reconstruction scene , and then obtain the reconstructed image.

The embodiment of the present invention also provides another method for reconstructing an image, and the method may include: during the live broadcast, detecting that a movement operation is received on the live broadcast screen; obtaining the target position of the virtual viewpoint after the displacement on the live broadcast screen; controlling The virtual viewpoint is moved from the target position to a predetermined position, where the predetermined position is the camera position that coincides with the position of the virtual viewpoint's spatial degrees of freedom; the image deformation model is used to process the image of the virtual viewpoint at the predetermined position to obtain a reconstructed image.

The method for reconstructing an image in this embodiment may be applied to a live broadcast scenario, for example, the live broadcast scenario may be a transaction type live broadcast scenario, which is not specifically limited here. The movement operation in this embodiment may be a movement operation of a virtual viewpoint performed by a terminal user on the live screen in the live broadcast scene, and may also be referred to as user free viewpoint interaction. The viewpoint will be displaced on the live screen, for example, the displacement corresponding to the movement operation starting from point A on the live screen.

In this embodiment, the interactive operation on the live screen can be detected, and then the target position where the virtual viewpoint stays after the displacement on the live screen is obtained. For example, the target position is point B on the live screen, and the moving operation stops at point B. It is the operation that the user's finger lifts up when it reaches point B, and the displacement of the virtual viewpoint on the live screen is the displacement between point A and point B.

After obtaining the target position of the virtual viewpoint displaced on the live screen, the target position does not necessarily correspond to the real camera position, so that image reconstruction cannot be performed directly at the target position. In this embodiment, the virtual viewpoint can be controlled to transition from the target position to the predetermined position, and the virtual viewpoint can be transitioned from the stayed target position along the interactive inertia according to a certain path until the transitioned position coincides with the position of the spatial degrees of freedom of the virtual viewpoint. , the position coincident with the position of the spatial degree of freedom of the virtual viewpoint is the predetermined position. Optionally, the predetermined position in this embodiment is a real viewpoint position closest to the target position.

In this embodiment, after obtaining the target position of the virtual viewpoint after the displacement on the live screen, the virtual viewpoint is no longer controlled by the interactive operation, so this embodiment controls the virtual viewpoint to transition from the target position to the predetermined position, so The interactive inertia involved can be understood as a rule for transitioning the virtual viewpoint.

Optionally, the above-mentioned certain path in this embodiment may be a predetermined smooth path on the live broadcast screen. The above-mentioned spatial degrees of freedom positions may be three spatial degrees of freedom positions (x, y, z).

In this embodiment, after the virtual viewpoint transitions to the predetermined position, there is an image with the virtual viewpoint at the predetermined position. In this embodiment, an image deformation model can be used to process the image with the virtual viewpoint at the predetermined position to obtain a reconstructed image. .

In this embodiment, the calculation process of the above-mentioned image deformation model is simple, so this embodiment is faster than the interpolation method using depth map-based virtual viewpoint reconstruction, adapts to the computing resources of low-end computers, and ensures that the virtual viewpoint reconstructs the image when the image is reconstructed. The timeliness (real-time), and the image quality in complex image reconstruction scenarios.

This embodiment uses the combined solution of inertial DIBR transition and real viewpoint position Warping to avoid the problem of low time-efficiency in the calculation of the low-end computer after staying at the virtual viewpoint position, and the related problem of low image quality in complex scenes, It can adapt to the computing resources of low-end computers and improve the image quality in complex scenes at the same time, so that the free-view video experience has better universality.

Example 2

The preferred implementation of the above-mentioned method of this embodiment will be further described with examples below.

FIG. 7 is a schematic diagram of a DIBR virtual viewpoint difference value according to the related art. As shown in Figure 7, as shown in Figure 8, the black thin solid arrow is used to represent the true orientation of the original camera, the hollow circle is used to represent the original camera, and the black solid circle is used to represent the virtual camera. In this embodiment, a camera array with sparse viewpoints is used to obtain a wide range of free viewpoints, wherein the black solid circles are used to indicate the position where the virtual viewpoints stay when performing virtual viewpoint interaction, and the hollow circles are used to represent the virtual viewpoints through DIBR. method to interpolate the determined real camera position. In the related art, the main interaction method is to determine the position of the virtual viewpoint where the user wishes to stay through the gesture interaction of the end user, and to perform interpolation through the DIBR method according to the 6DoF parameters of the virtual viewpoint to obtain the user's position at the virtual viewpoint. View images.

However, a single DIBR function cannot meet the diverse needs of users. This is mainly due to the following two reasons: on some mid-to-low-end models, the terminal computing resources are not abundant, which will lead to high complexity of DIBR. The calculation cannot achieve the purpose of real-time. Therefore, if the user stays in the virtual viewpoint for a long time to watch, the video will be stuck; As a result, the quality of the depth map cannot be guaranteed, which further causes the user to stay at the virtual viewpoint position continuously, and the subjective experience of the interpolated image quality is seriously degraded compared with the original viewpoint.

Based on the above two reasons, this embodiment proposes that a simplified version of Warping-based method needs to be added to the DIBR reference software to meet diverse and universal interaction requirements.

FIG. 8 is a schematic diagram of virtual viewpoint interpolation based on Warping and DIBR according to an embodiment of the present invention. As shown in Figure 8, the black thin solid line arrows are used to represent the real orientation of the original camera, and the black thick solid line arrows are used to represent the virtual camera orientation after the path is planned (what the user sees is the black thick solid line arrows) ). Open circles are used to represent the original camera, and black filled circles are used to represent the virtual camera. FIG. 9 is a flowchart of a method for virtual viewpoint interpolation based on Warping and DIBR according to an embodiment of the present invention. 8 and 9 , in the virtual viewpoint interpolation based on Warping, when step S901 is executed, the user performs virtual viewpoint interaction (DIBR) and stays at the position of the black solid circle, the virtual viewpoint can be planned along the path. The interactive inertia performs transition interpolation through a smooth path until the three-degree-of-freedom position (x, y, z) of the virtual viewpoint coincides with the real camera position. For example, the virtual viewpoint moves from the second black circle to its adjacent At the position of the left hollow circle of the adjacent left side, the position of the adjacent left hollow circle corresponds to the position of the real camera, so that step S902 is realized, and the virtual viewpoint inertial transitions to the real viewpoint, and then step S903 is executed, through Warping Determine the true viewpoint location. After the spatial three-degree-of-freedom position (x, y, z) of the virtual viewpoint coincides with the real camera position, the difference between the interpolated planned 6DoF position and the real camera position is only in the remaining three rotational degrees of freedom (respectively is the degree of freedom of rotation around the three axes of x, y, and z), after step S903 is executed, S901 is executed again. Among them, in FIG. 8 , the solid thin arrows corresponding to the hollow circles, the solid thick arrows corresponding to the black solid circles, and the paths corresponding to the solid dashed arrows.

After the three-degree-of-freedom position (x, y, z) of the virtual viewpoint coincides with the real camera position, the next viewpoint reconstruction function can be updated through the Warping operation of the original camera image according to the three rotational degrees of freedom, so that The above two problems can be solved at the same time: the problem of timeliness of terminal reconstruction of DIBR, and the problem of image quality in complex reconstruction scenarios. At the same time, in this embodiment, the inertial DIBR interpolation transition is performed from the virtual viewpoint position interacted by the user to the real camera position, thereby solving the problem of smoothness when the user switches between free viewpoints.

7 and 8, the black solid points (such as A1 and A2) can represent virtual viewpoints, and the white hollow points (such as B1, B2, B3, etc.) can represent the position of the real viewpoint where the real camera is located. After the movement operation occurs on the operation interface, after the virtual viewpoint is moved from A1 to A2, the virtual viewpoint at the A2 position will select a real viewpoint where the real camera is located based on the predetermined control mode, and move to this real viewpoint. At this time, The control mode can include any one of the following: select the real viewpoint from the virtual viewpoint A2; select a real viewpoint with free computing resources and the closest to the virtual viewpoint; when the camera where the selected real viewpoint is located is damaged, waiting for processing time When any one or more of the situations in the predetermined duration exceed, select the real viewpoint with the highest preset weight value.

In this embodiment, the above-mentioned Warping operation only needs to perform one projection because of its simple calculation, and there are no other pre- and post-processing operations, so it is faster than the DIBR interpolation. Considering the ubiquity of the application of the standard in different scenarios and models, this embodiment provides a DIBR method combined with the Warping function, which can support richer terminal interaction design, and can be flexibly configured in combination with actual terminal and scenario conditions Different interaction methods to enhance user experience.

In this embodiment, the combination scheme of inertial DIBR transition and real viewpoint position Warping is used to avoid the problem of real-time calculation of low-end computers after staying at the virtual viewpoint position, and the problem of image quality in related complex scenes.

This embodiment provides a DIBR synthesis method supported by a Warping function with low computational complexity, through which the computational resources of low-end and mid-end computers and image quality problems in complex scenes can be simultaneously adapted, so that free-view video can be Experience is more universal.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence. As in accordance with the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

From the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solutions of the present invention can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

Example 3

According to an embodiment of the present invention, an apparatus for implementing the above method for reconstructing an image is also provided. It should be noted that the apparatus for reconstructing an image in this embodiment can be used to execute the method for reconstructing an image shown in FIG. 2 in the embodiment of the present invention.

FIG. 10 is a schematic diagram of an apparatus for reconstructing an image according to an embodiment of the present invention. As shown in FIG. 10 , the apparatus 100 for reconstructing an image may include: a first acquisition module 101, a first control module 102 and a second processing module 103.

The first obtaining module 101 is configured to detect an interactive operation on the operation interface, and obtain the target position after the virtual viewpoint is displaced on the operation interface.

The first control module 102 is configured to control the virtual viewpoint to transition from a target position to a predetermined position, wherein the predetermined position is a camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint.

The second processing module 103 is configured to use the image deformation model to process the image with the virtual viewpoint located at the predetermined position, and obtain the reconstructed image.

It should be noted here that the above-mentioned first acquisition module 101 , first control module 102 and second processing module 103 correspond to steps S202 to S206 in Embodiment 1, and the three modules and corresponding steps are implemented by the examples and The application scenarios are the same, but are not limited to the content disclosed in the first embodiment. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal 10 provided in the first embodiment.

According to an embodiment of the present invention, another apparatus for implementing the above method for reconstructing an image is also provided. It should be noted that the apparatus for reconstructing an image in this embodiment can be used to execute the method for reconstructing an image shown in FIG. 3 in the embodiment of the present invention.

According to an embodiment of the present invention, another apparatus for implementing the above method for reconstructing an image is also provided. It should be noted that the apparatus for reconstructing an image in this embodiment may be used to execute the method for reconstructing an image shown in FIG. 4 in the embodiment of the present invention.

FIG. 11 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention. As shown in FIG. 11 , the apparatus 110 for reconstructing an image may include: a detection module 111 , a second control module 112 , a first reading module 113 and a second processing module 114 .

The detection module 111 is configured to detect that an interactive operation of controlling the virtual viewpoint to move from the first position to the second position occurs on the operation interface.

The second control module 112 is configured to control the virtual viewpoint to transition from the second position to the third position, where the third position is the position of the real viewpoint where the camera is deployed.

The first reading module 113 is used for reading the image of the camera located at the third position.

The second processing module 114 is configured to process the image by using the plane-based image deformation model to obtain the reconstructed image.

It should be noted here that the detection module 111 , the second control module 112 , the first reading module 113 and the second processing module 114 correspond to steps S302 to S308 in Embodiment 1, and the four modules correspond to the corresponding steps The implemented examples and application scenarios are the same, but are not limited to the content disclosed in the first embodiment above. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal 10 provided in the first embodiment.

According to an embodiment of the present invention, another apparatus for implementing the above method for reconstructing an image is also provided. It should be noted that the apparatus for reconstructing an image in this embodiment can be used to execute the method for reconstructing an image shown in FIG. 5 in the embodiment of the present invention.

FIG. 12 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention. As shown in FIG. 12 , the apparatus 120 for reconstructing an image may include: a first sensing module 121, a second acquiring module 122, a third control module 123, and a third processing module 124.

The first sensing module 121 is used for sensing the movement operation of the operation object on the operation interface.

The second obtaining module 122 is configured to obtain the stop position where the operation object stays on the operation interface when the movement operation is stopped when it is sensed that the movement operation is stopped.

The third control module 123 is configured to control the virtual viewpoint located at the stop position to transition to a predetermined position, wherein the predetermined position is a camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint.

The third processing module 124 is configured to process the image of the camera at the predetermined position by using the plane-based image deformation model to obtain the reconstructed image.

It should be noted here that the above-mentioned first sensing module 121, second acquisition module 122, third control module 123 and third processing module 124 correspond to steps S502 to S508 in Embodiment 1, and the four modules correspond to the corresponding steps S502 to S508. The examples and application scenarios implemented by the steps are the same, but are not limited to the content disclosed in the first embodiment above. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal 10 provided in the first embodiment.

According to an embodiment of the present invention, another apparatus for implementing the above method for reconstructing an image is also provided. It should be noted that the apparatus for reconstructing an image in this embodiment can be used to execute the method for reconstructing an image shown in FIG. 6 in the embodiment of the present invention.

FIG. 13 is a schematic diagram of another apparatus for reconstructing an image according to an embodiment of the present invention. As shown in FIG. 13 , the apparatus 130 for reconstructing an image may include: a second sensing module 131 , a third acquiring module 132 , a fourth control module 133 , a second reading module 134 and a fourth processing module 135 .

The second sensing module 131 is used for sensing the movement operation of the operation object on the operation interface.

The third obtaining module 132 is configured to obtain the stop position where the operation object stays on the operation interface when the movement operation is stopped if it is sensed that the movement operation is stopped.

The fourth control module 133 is configured to control the virtual viewpoint located at the stop position to transition to a predetermined position, where the predetermined position is the position of the real viewpoint where the camera is deployed.

The second reading module 134 is used for reading the image of the camera located at the predetermined position.

The fourth processing module 135 is configured to process the image by using the plane-based image deformation model to obtain the reconstructed image.

It should be noted here that the above-mentioned second sensing module 131 , third acquisition module 132 , fourth control module 133 , second reading module 134 and fourth processing module 135 correspond to steps S602 to S610 in Embodiment 1 , the examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the content disclosed in the first embodiment above. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal 10 provided in the first embodiment.

In the apparatus for reconstructing an image of this embodiment, the target position after the virtual viewpoint is displaced on the operation interface is obtained, and when the virtual viewpoint transitions from the target position to the predetermined position, an image deformation model can be used to process the virtual viewpoint at the predetermined position. The image above can achieve the purpose of obtaining the reconstructed image. Because the calculation process of the image deformation model is simple, the calculation speed is block, and it adapts to the computing resources of the low-end computer, thus solving the technical problem of poor timeliness when reconstructing the image at the virtual viewpoint, The technical effect of improving the timeliness when reconstructing an image at a virtual viewpoint is achieved.

Example 4

Embodiments of the present invention may provide a system for reconstructing an image, and the system for reconstructing an image may include a computer terminal, and the computer terminal may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the above-mentioned computer terminal may also be replaced by a terminal device such as a mobile terminal.

Optionally, in this embodiment, the above-mentioned computer terminal may be located in at least one network device among multiple network devices of a computer network.

In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the method for reconstructing an image: detecting that an interactive operation occurs on the operation interface, obtaining the target position after the virtual viewpoint is displaced on the operation interface; controlling the virtual viewpoint from the target The position transitions to a predetermined position, where the predetermined position is the camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint; the image deformation model is used to process the image of the virtual viewpoint at the predetermined position to obtain a reconstructed image.

Optionally, FIG. 14 is a structural block diagram of a computer terminal according to an embodiment of the present invention. As shown in FIG. 14 , the computer terminal A may include: one or more (only one is shown in the figure) processor 142 , memory 144 , and transmission device 146 .

The memory can be used to store software programs and modules, such as program instructions/modules corresponding to the method and apparatus for reconstructing an image in the embodiment of the present invention, and the processor executes various functions by running the software programs and modules stored in the memory. Application and data processing, that is, to realize the above-mentioned method of reconstructing an image. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory located remotely from the processor, and these remote memories may be connected to the computer terminal A through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: detecting that an interactive operation occurs on the operation interface, obtaining the target position after the virtual viewpoint is displaced on the operation interface; controlling the virtual viewpoint from the target position Transition to a predetermined position, where the predetermined position is the camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint; use the image deformation model to process the image with the virtual viewpoint at the predetermined position to obtain a reconstructed image.

Optionally, the above-mentioned processor may further execute the program code of the following steps: if it is detected that the interaction operation at the target position is interrupted, start and execute a transition operation of transitioning the virtual viewpoint from the target position to the predetermined position.

Optionally, the above-mentioned processor can also execute the program code of the following steps: in the process of controlling the virtual viewpoint to transition from the target position to the predetermined position, use the DIBR virtual viewpoint interpolation algorithm to generate the interpolation of the transition image frame; Interpolate to reconstruct the image during transition.

Optionally, the above-mentioned processor can also execute the program code of the following steps: after controlling the virtual viewpoint to transition from the target position to the predetermined position, call the Warping image warping algorithm to switch to use the image warping model for image reconstruction.

Optionally, the above-mentioned processor can also execute the program code of the following steps: based on the predetermined position where the virtual viewpoint is located, read the corresponding image from the camera; Image warping model to process the read image.

Optionally, the above-mentioned processor can also execute the program code of the following steps: before controlling the virtual viewpoint to transition from the target position to the predetermined position, obtain the distance between the target position where the virtual viewpoint is currently located and the predetermined position; if the distance exceeds a threshold, Then control the virtual viewpoint to move from the target position to the viewpoint position where the camera with the highest priority is located according to the preset camera priority; if the distance does not exceed the threshold, execute the step of transitioning the virtual viewpoint from the target position to the predetermined position.

Optionally, the above-mentioned processor can also execute the program code of the following steps: after detecting that an interactive operation occurs on the operation interface, pop up prompt information for indicating at least one selection control; by triggering any selection control, select the reconstructed image. display resolution.

As an optional example, the processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: detecting that an interactive operation of controlling the virtual viewpoint to move from the first position to the second position occurs on the operation interface control the virtual viewpoint to transition from a second position to a third position, where the third position is the position of the real viewpoint where the camera is deployed; read the image of the camera at the third position; process the image using a plane-based image warping model, Acquire reconstructed images.

Optionally, the above-mentioned processor may further execute the program code of the following steps: selecting the position of the real viewpoint closest to the second position as the third position.

As an optional example, the processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: sensing an interaction operation of the operation object on the operation interface; responding to the interaction operation, displaying the virtual viewpoint based on the interaction The displacement generated by the operation on the operation interface, wherein the displacement is the movement from the first position to the second position; the transition of the displayed virtual viewpoint from the second position to the third position, wherein the third position is the position of the real viewpoint where the camera is deployed ; displaying a reconstructed image generated by processing the target image using a plane-based image warping model, where the target image is an image captured by a camera located at a third position.

As an optional example, the processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: sensing the movement operation of the operation object on the operation interface; if it senses that the movement operation is terminated, obtain the movement operation The stop position where the operation object stays on the operation interface when it is aborted; control the virtual viewpoint at the stop position to transition to a predetermined position, where the predetermined position is the camera position that coincides with the position of the spatial degrees of freedom of the virtual viewpoint; use a plane-based image deformation model to process The image of the camera at the predetermined position, and the reconstructed image is obtained.

As an optional example, the processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: sensing the movement operation of the operation object on the operation interface; if it senses that the movement operation is terminated, obtain the movement operation The stop position where the operation object stays on the operation interface when it is suspended; control the virtual viewpoint at the stop position to transition to a predetermined position, wherein the predetermined position is the position of the real viewpoint where the camera is deployed; read the image of the camera at the predetermined position; The planar image warping model processes the image to obtain a reconstructed image.

As an optional example, the processor may call the information and application programs stored in the memory through the transmission device to perform the following steps: during the live broadcast process, detecting that a moving operation is received on the live broadcast screen; obtaining a virtual viewpoint on the live broadcast screen control the virtual viewpoint to move from the target position to a predetermined position, wherein the predetermined position is the camera position that coincides with the position of the spatial degrees of freedom of the virtual viewpoint; use the image deformation model to process the image of the virtual viewpoint at the predetermined position , to obtain the reconstructed image.

Using the embodiments of the present invention, a method for reconstructing an image is provided. By detecting the interactive operation on the operation interface, the target position after the virtual viewpoint is displaced on the operation interface is obtained; the virtual viewpoint is controlled to transition from the target position to a predetermined position, wherein the predetermined position is the position that coincides with the position of the spatial degrees of freedom of the virtual viewpoint Camera position; use the image deformation model to process the image with the virtual viewpoint at the predetermined position, and obtain the reconstructed image. That is to say, the target position after the virtual viewpoint is displaced on the operation interface is obtained. When the virtual viewpoint transitions from the target position to the predetermined position, the image deformation model can be used to process the image of the virtual viewpoint at the predetermined position, so as to obtain the The purpose of reconstructing the image, because the calculation process of the image deformation model is simple, the calculation speed is block, and it adapts to the computing resources of the low-end computer, so as to solve the technical problem of poor timeliness when reconstructing the image in the virtual viewpoint, and achieve the improvement of the reconstruction in the virtual viewpoint. Time-sensitive technical effects of images.

Those of ordinary skill in the art can understand that the structure shown in FIG. 14 is only a schematic diagram, and the computer terminal A can also be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, an applause computer, and a mobile Internet device (Mobile Internet Devices, MID), PAD and other terminal equipment. FIG. 14 does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal A may also include more or less components than those shown in FIG. 14 (eg, a network interface, a display device, etc.), or have a different configuration than that shown in FIG. 14 .

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can Including: flash disk, read-only memory (Read-Only Memory, ROM), random access device (Random Access Memory, RAM), magnetic disk or optical disk, etc.

Example 5

Embodiments of the present invention also provide a computer-readable storage medium. Optionally, in this embodiment, the above-mentioned storage medium may be used to store the program code executed by the method for reconstructing an image provided in the above-mentioned first embodiment.

Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for performing the following steps: detecting that an interactive operation occurs on the operation interface, and acquiring the target position after the virtual viewpoint is displaced on the operation interface. ; Control the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is the camera position coincident with the position of the spatial degrees of freedom of the virtual viewpoint; use the image deformation model to process the image of the virtual viewpoint at the predetermined position, and obtain the reconstructed image.

Optionally, the computer-readable storage medium is further configured to store program code for performing the steps of: initiating performing a transition operation of transitioning the virtual viewpoint from the target position to the predetermined position if an interruption of the interactive operation at the target position is detected.

Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: in the process of controlling the virtual viewpoint to transition from the target position to the predetermined position, use the DIBR virtual viewpoint interpolation algorithm to generate a transition image frame. Interpolation; based on the interpolation of the transition image frames, to reconstruct the image in the transition process.

Optionally, the computer-readable storage medium is further configured to store program codes for executing the following steps: after controlling the virtual viewpoint to transition from the target position to the predetermined position, call the Warping image warping algorithm, and switch to use the image warping model to perform the transformation. Image reconstruction.

Optionally, the computer-readable storage medium is further configured to store program codes for performing the following steps: reading a corresponding image from the camera based on the predetermined position where the virtual viewpoint is located; according to the spatial coordinate system where the virtual viewpoint is located The rotational degrees of freedom on the image warping model are used to process the read image.

Optionally, the computer-readable storage medium is further configured to store program codes for executing the following steps: before controlling the virtual viewpoint to transition from the target position to the predetermined position, acquiring the distance between the target position where the virtual viewpoint is currently located and the predetermined position. Distance; if the distance exceeds the threshold, control the virtual viewpoint to move from the target position to the viewpoint position where the camera with the highest priority is located according to the preset camera priority; if the distance does not exceed the threshold, execute the virtual viewpoint transition from the target position to the predetermined location steps.

Optionally, the computer-readable storage medium is further configured to store program codes for executing the following steps: after detecting that an interactive operation occurs on the operation interface, pop up prompt information for indicating at least one selection control; by triggering any one of the Select the control to choose the display resolution of the reconstructed image.

As an optional example, the computer-readable storage medium is configured to store program codes for performing the following steps: detecting that an interactive operation of controlling the virtual viewpoint to move from the first position to the second position occurs on the operation interface; controlling the virtual The viewpoint transitions from the second position to the third position, where the third position is the position of the real viewpoint where the camera is deployed; the image of the camera at the third position is read; the image is processed using a plane-based image warping model to obtain a reconstructed image .

Optionally, the computer-readable storage medium is further configured to store program code for performing the steps of: selecting a real viewpoint position closest to the second position as the third position.

As an optional example, the computer-readable storage medium is configured to store program codes for executing the following steps: sensing an interactive operation of the operating object on the operating interface; responding to the interactive operation, displaying that the virtual viewpoint is operating based on the interactive operation. The displacement that occurs on the interface, wherein the displacement is moving from the first position to the second position; the transition of the virtual viewpoint from the second position to the third position is displayed, wherein the third position is the position of the real viewpoint where the camera is deployed; the display uses A reconstructed image generated by processing a target image based on a plane-based image deformation model, wherein the target image is an image captured by a camera located at a third position.

As an optional example, the computer-readable storage medium is configured to store program codes for performing the following steps: sensing the movement operation of the operation object on the operation interface; if the suspension of the movement operation is sensed, obtaining the operation when the movement operation is stopped The stop position where the object stays on the operation interface; control the virtual viewpoint located at the stop position to transition to a predetermined position, where the predetermined position is the camera position that coincides with the position of the spatial degree of freedom of the virtual viewpoint; use the plane-based image deformation model to process the camera at the predetermined position. position of the image to obtain a reconstructed image.

As an optional example, the computer-readable storage medium is configured to store program codes for performing the following steps: sensing the movement operation of the operation object on the operation interface; if the suspension of the movement operation is sensed, obtaining the operation when the movement operation is stopped The stop position where the object stays on the operation interface; control the virtual viewpoint at the stop position to transition to a predetermined position, where the predetermined position is the position of the real viewpoint where the camera is deployed; read the image of the camera at the predetermined position; use the plane-based image The deformed model processes the image to obtain a reconstructed image.

As an optional example, the computer-readable storage medium is configured to store program codes for executing the following steps: during the live broadcast, it is detected that a movement operation is received on the live broadcast screen; the virtual viewpoint is obtained to shift on the live broadcast screen control the virtual viewpoint to move from the target position to a predetermined position, where the predetermined position is the camera position that coincides with the position of the spatial degrees of freedom of the virtual viewpoint; use the image deformation model to process the image with the virtual viewpoint at the predetermined position, and obtain the reconstruction image.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content may be implemented in other ways. The device embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. It should be regarded as the protection scope of the present invention.

Claims

A method of reconstructing an image, comprising:

It is detected that an interactive operation occurs on the operation interface, and the target position after the virtual viewpoint is displaced on the operation interface is obtained;

controlling the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a camera position coincident with the position of the spatial degrees of freedom of the virtual viewpoint;

Using an image deformation model to process the image where the virtual viewpoint is located at the predetermined position to obtain a reconstructed image.
The method according to claim 1, wherein if the interruption of the interactive operation at the target position is detected, a transition operation for performing the transition of the virtual viewpoint from the target position to the predetermined position is initiated.
The method according to claim 1, wherein, in the process of controlling the virtual viewpoint to transition from the target position to a predetermined position, the method further comprises:

Use the DIBR virtual viewpoint interpolation algorithm to generate the interpolation of the transition image frame;

An image in the transition process is reconstructed based on the interpolation of the transition image frames.
The method according to claim 3, wherein after controlling the virtual viewpoint to transition from the target position to a predetermined position, a Warping image warping algorithm is invoked to switch to use the image warping model for image reconstruction.
The method according to any one of claims 1 to 4, wherein, using an image deformation model to process the image with the virtual viewpoint at the predetermined position, comprising:

Based on the predetermined position where the virtual viewpoint is located, a corresponding image is read from the camera;

The read image is processed by using the image deformation model according to the rotational degrees of freedom on the space coordinate system where the virtual viewpoint is located.
The method according to claim 5, wherein the spatial degree of freedom is the coordinate value of the virtual viewpoint on the spatial coordinate system, and the rotational degree of freedom is the freedom of rotation around the coordinate axis of the spatial coordinate system Spend.
The method of claim 1, wherein before controlling the virtual viewpoint to transition from the target position to a predetermined position, the method further comprises:

obtaining the distance between the target position where the virtual viewpoint is currently located and the predetermined position;

If the distance exceeds the threshold, controlling the virtual viewpoint to move from the target position to the viewpoint position where the camera with the highest priority is located according to the preset camera priority;

If the distance does not exceed the threshold, the step of transitioning the virtual viewpoint from the target position to a predetermined position is performed.
The method according to claim 1, wherein after detecting that an interactive operation occurs on the operation interface, the method further comprises:

pop up a prompt message indicating at least one selection control;

By triggering any one of the selection controls, the display resolution of the reconstructed image is selected.
A method of reconstructing an image, comprising:

During the live broadcast, it is detected that a mobile operation is received on the live broadcast screen;

acquiring the target position after the virtual viewpoint is displaced on the live screen;

controlling the virtual viewpoint to move from the target position to a predetermined position, wherein the predetermined position is a camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint;

Using an image deformation model to process the image where the virtual viewpoint is located at the predetermined position to obtain a reconstructed image.
A method of reconstructing an image, comprising:

Detecting that an interactive operation of controlling the virtual viewpoint to move from the first position to the second position occurs on the operation interface;

controlling the virtual viewpoint to transition from the second position to a third position, wherein the third position is the position of the real viewpoint where the camera is deployed;

reading the image of the camera located at the third position;

The image is processed using a plane-based image warping model to obtain a reconstructed image.
The method of claim 10, wherein a true viewpoint position closest to the second position is selected as the third position.
A method of reconstructing an image, comprising:

Sensing the interactive operation of the operation object on the operation interface;

In response to the interactive operation, displaying the displacement of the virtual viewpoint on the operation interface based on the interactive operation, wherein the displacement is moving from a first position to a second position;

displaying that the virtual viewpoint transitions from the second position to a third position, wherein the third position is the position of the real viewpoint where the camera is deployed;

A reconstructed image generated by processing a target image using a plane-based image warping model is displayed, wherein the target image is an image captured by a camera located at the third position.
A method of reconstructing an image, comprising:

Sensing the movement of the operation object on the operation interface;

If it is sensed that the movement operation is suspended, obtain the stop position of the operation object on the operation interface when the movement operation is suspended;

controlling the virtual viewpoint located at the stop position to transition to a predetermined position, wherein the predetermined position is a camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint;

The image of the camera at the predetermined position is processed using a plane-based image warping model to obtain a reconstructed image.
A method of reconstructing an image, comprising:

Sensing the movement of the operation object on the operation interface;

If it is sensed that the movement operation is suspended, obtain the stop position of the operation object on the operation interface when the movement operation is suspended;

controlling the virtual viewpoint located at the stop position to transition to a predetermined position, wherein the predetermined position is the position of the real viewpoint where the camera is deployed;

reading the image of the camera located at the predetermined position;

The image is processed using a plane-based image warping model to obtain a reconstructed image.
A device for reconstructing an image, comprising:

a first acquisition module, configured to detect that an interactive operation occurs on the operation interface, and acquire the target position after the virtual viewpoint is displaced on the operation interface;

a first control module, configured to control the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is a camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint;

The second processing module is configured to use an image deformation model to process the image where the virtual viewpoint is located at the predetermined position to obtain a reconstructed image.
A device for reconstructing an image, comprising:

a detection module, configured to detect that an interactive operation of controlling the virtual viewpoint to move from the first position to the second position occurs on the operation interface;

a second control module, configured to control the virtual viewpoint to transition from the second position to a third position, wherein the third position is the position of the real viewpoint where the camera is deployed;

a first reading module for reading the image of the camera located at the third position;

The second processing module is configured to process the image by using a plane-based image deformation model to obtain a reconstructed image.
A device for reconstructing an image, comprising:

The first sensing module is used for sensing the movement operation of the operation object on the operation interface;

a second obtaining module, configured to obtain the stop position of the operation object on the operation interface when the movement operation is stopped when it is sensed that the movement operation is stopped;

a third control module, configured to control the virtual viewpoint located at the stop position to transition to a predetermined position, wherein the predetermined position is a camera position coincident with the position of the spatial degree of freedom of the virtual viewpoint;

The third processing module is configured to process the image of the camera at the predetermined position by using a plane-based image deformation model to obtain a reconstructed image.
A device for reconstructing an image, comprising:

The second sensing module is used for sensing the movement operation of the operation object on the operation interface;

a third obtaining module, configured to obtain the stop position of the operation object on the operation interface when the movement operation is stopped if it is sensed that the movement operation is stopped;

a fourth control module, configured to control the virtual viewpoint located at the stay position to transition to a predetermined position, wherein the predetermined position is the position of the real viewpoint where the camera is deployed;

a second reading module, configured to read the image of the camera located at the predetermined position;

The fourth processing module is configured to process the image by using a plane-based image deformation model to obtain a reconstructed image.
A computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is run by a processor, a device where the computer-readable storage medium is located is controlled to execute the programs in claims 1 to 14 any of the methods described.
A processor, wherein the processor is used to run a program, wherein the program executes the method of any one of claims 1 to 14 when the program is run.
A system for reconstructing an image, comprising:

processor;

a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps:

Detecting that an interactive operation occurs on the operation interface, and obtaining a target position after the virtual viewpoint is displaced on the operation interface; controlling the virtual viewpoint to transition from the target position to a predetermined position, wherein the predetermined position is the same as the target position. The position of the camera where the spatial degrees of freedom of the virtual viewpoint coincide; the image deformation model is used to process the image where the virtual viewpoint is located at the predetermined position to obtain a reconstructed image.