CN114241127A

CN114241127A - Panoramic image generation method and device, electronic equipment and medium

Info

Publication number: CN114241127A
Application number: CN202111457949.0A
Authority: CN
Inventors: 黄梦辉
Original assignee: Vivo Mobile Communication Co Ltd
Current assignee: Vivo Mobile Communication Co Ltd
Priority date: 2021-12-01
Filing date: 2021-12-01
Publication date: 2022-03-25

Abstract

The application discloses a panoramic image generation method, a panoramic image generation device, electronic equipment and a panoramic image generation medium, and belongs to the technical field of image processing. The method comprises the following steps: performing three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image except an elevation angle image in the shooting scene; adjusting a first observation angle of the first local three-dimensional model to a second observation angle to obtain a second local three-dimensional model, wherein a shooting object corresponding to the first image is in a shooting range of the panoramic image generation device at a corresponding position of the second observation angle; and generating a target panoramic image based on the second local three-dimensional model and the positive visual angle image.

Description

Panoramic image generation method and device, electronic equipment and medium

Technical Field

The embodiment of the invention belongs to the technical field of image processing, and particularly relates to a panoramic image generation method, a panoramic image generation device, electronic equipment and a panoramic image generation medium.

Background

With the rapid development of terminal technology, more and more users tend to use cameras in more portable terminal devices for photographing and recording. The panoramic photography mode of the camera provides the user with a better choice when the user wants to record a scene where the field of view of the camera does not cover all.

However, in the process of shooting the panoramic image, the panoramic image generated finally is easily deformed obviously due to the different relative angles between different shooting objects and the camera.

Disclosure of Invention

The embodiment of the application provides a panoramic image generation method, a panoramic image generation device, an electronic device and a panoramic image generation medium, which can solve the problem that image deformation is easy to occur when a panoramic image is shot.

In order to solve the technical problem, the invention is realized as follows:

in a first aspect, an embodiment of the present application provides a method for generating a panoramic image, where the method includes:

performing three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image except an elevation angle image in the shooting scene;

adjusting a first observation angle of the first local three-dimensional model to a second observation angle to obtain a second local three-dimensional model, wherein a shooting object corresponding to the first image is in a shooting range of the panoramic image generation device at a corresponding position of the second observation angle;

and generating a target panoramic image based on the second local three-dimensional model and the positive visual angle image.

In a second aspect, an embodiment of the present application provides a panoramic image generation apparatus, including:

the system comprises a model establishing unit, a first local three-dimensional model obtaining unit and a second local three-dimensional model obtaining unit, wherein the model establishing unit is used for carrying out three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image to obtain the first local three-dimensional model, and the first image is an image except an orthoscopic image in the shooting scene;

an adjusting unit, configured to adjust a first observation angle of the first local three-dimensional model to a second observation angle to obtain a second local three-dimensional model, where, at a corresponding position of the second observation angle, a photographic object corresponding to the first image is within a shooting range of the panoramic image generation apparatus;

an image generating unit, configured to generate a target panoramic image based on the second local three-dimensional model and the forward view image.

In a third aspect, an embodiment of the present application provides a terminal device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the steps of the panoramic image generation method according to the first aspect.

In a fourth aspect, the present application further provides a readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the panoramic image generation method according to the first aspect.

In a fifth aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to execute a program or instructions to implement the method for generating a panoramic image according to the first aspect.

In the embodiment of the application, when a panoramic image is shot, three-dimensional reconstruction can be performed based on a first image of a shot scene and depth information of the first image to obtain a first local three-dimensional model, the first image is an image except an elevation angle image in the shot scene, a first observation angle of the first local three-dimensional model is adjusted to a second observation angle to obtain a second local three-dimensional model, wherein a shot object corresponding to the first image is in a shooting range of a panoramic image generation device at a corresponding position of the second observation angle, and finally a target panoramic image is generated based on the second local three-dimensional model and the elevation angle image. On one hand, the depth information of the image of the shooting scene can be obtained, and when a local three-dimensional model is established, the establishment of the three-dimensional model of the front view angle image is omitted, so that the establishment efficiency of the three-dimensional model is improved; on the other hand, the observation angle of view of the local three-dimensional model is adjusted, so that the panoramic image generation device can contain all the shooting objects in the target panoramic image generated finally at the corresponding position of the adjusted observation angle of view, the panoramic image generated finally cannot be obviously deformed in vision, and the quality of the panoramic image is improved.

Drawings

Fig. 1 is a schematic flowchart of a specific implementation of a panoramic image generation method according to an embodiment of the present application;

fig. 2 is a schematic diagram of depth information of an acquired image of a depth panorama obtained by opening a camera of a terminal device in a panoramic image generation method provided in an embodiment of the present application;

fig. 3 is a schematic diagram of images of multiple viewing angles acquired in a panoramic image generation method provided in an embodiment of the present application;

fig. 4 is a schematic diagram of a local three-dimensional model established in the panoramic image generation method according to the embodiment of the present application;

fig. 5 is a schematic diagram illustrating adjustment of a vertex of a viewing cone in a panoramic image generation method according to an embodiment of the present application so that a first observation angle of a first local three-dimensional model is adjusted to a second observation angle;

fig. 6 is a schematic diagram of a camera space and a world space in a panoramic image generation method provided in an embodiment of the present application;

fig. 7 is a schematic diagram of a panoramic image generated before the panoramic image generation method provided by the embodiment of the present application is adopted and a panoramic image generated after the panoramic image generation method provided by the embodiment of the present application is adopted;

fig. 8 is a schematic structural diagram of a panoramic image generation apparatus according to an embodiment of the present application;

fig. 9 is a schematic hardware structure diagram of an electronic device according to an embodiment of the present disclosure;

fig. 10 is a schematic hardware structure diagram of another electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art without any inventive work according to the embodiments of the present application are within the scope of the present application.

The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application may be practiced in sequences other than those illustrated or described herein, and that the terms "first," "second," and the like are generally used herein in a generic sense and do not limit the number of terms, e.g., the first term can be one or more than one. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.

The method for generating a panoramic image according to the embodiments of the present application will be described in detail with reference to the accompanying drawings through specific embodiments and application scenarios thereof.

In order to solve the problem that the generated panoramic image is easily deformed when the relative angle between a shooting object and a camera is changed in the existing panoramic image generation method, the application provides a panoramic image generation method, and an execution main body of the method can be but is not limited to a mobile phone, a tablet personal computer and the like which can be configured to execute at least one of the user terminals of the method provided by the embodiment of the invention.

For convenience of description, the following description will be made of an embodiment of the method taking as an example that an execution subject of the method is a terminal device capable of executing the method. It is understood that the implementation of the method by the terminal device is only an exemplary illustration, and should not be construed as a limitation of the method.

At present, when a panoramic image is generated, sub-images of a plurality of collected visual angles need to be spliced, and when the visual angle range of the panoramic image is large, the spliced panoramic image is prone to deformation. When the shot object is a crowd, the crowd in the same plane in the spliced panoramic image is usually small in the middle view field and large in the people display, and the view fields on the two sides are large and small in the people display. When the shooting objects are building communities on the same plane, the spliced panoramic image also has the phenomenon that the middle building is high and the buildings on the two sides become low gradually, and therefore misleading can be caused to an observer of the panoramic image.

The existing panoramic image correction method aiming at deformation has stretching means such as affine transformation and the like, and can carry out targeted correction on a specific scene to restore the actual effect, but the method needs to calibrate the specific shooting scene, so the method is only effective on part of the specific shooting scene, and when the shooting scene is not the specific shooting scene, the image correction method adopting the affine transformation cannot eliminate the deformation generated by the panoramic image.

In view of the above, the method for generating a panoramic image provided by the present invention includes: firstly, three-dimensional reconstruction is carried out on the basis of a first image of a shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image except an elevation angle image in the shooting scene; then, adjusting the first observation angle of the first local three-dimensional model to a second observation angle to obtain a second local three-dimensional model, wherein a shooting object corresponding to the first image is in the shooting range of the panoramic image generation device at the corresponding position of the second observation angle; and finally, generating a target panoramic image based on the second local three-dimensional model and the front view angle image.

On one hand, the depth information of the image of the shooting scene can be obtained, and when a local three-dimensional model is established, the establishment of the three-dimensional model of the front view angle image is omitted, so that the establishment efficiency of the three-dimensional model is improved; on the other hand, the observation angle of view of the local three-dimensional model is adjusted, so that the panoramic image generation device can contain all the shooting objects at the corresponding position of the adjusted observation angle of view, the finally generated panoramic image cannot be obviously deformed in vision, and the quality of the panoramic image is improved.

The following describes in detail an implementation process of the panoramic image generation method shown in fig. 1 with reference to a schematic flowchart of a specific implementation process of the method, including:

and S110, performing three-dimensional reconstruction based on a first image of the shooting scene and the depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image except for the front view angle image in the shooting scene.

The three-dimensional reconstruction is performed based on the first image of the shooting scene and the depth information of the first image, and specifically, the three-dimensional reconstruction is performed according to the gray value of each pixel point corresponding to the depth information of the first image in the first image and the depth information of the first image to obtain a first local three-dimensional model.

Optionally, before performing three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image to obtain a first local three-dimensional model, the method provided in an embodiment of the present specification further includes:

depth information of the first image is acquired by TOF lens or multi-ocular vision techniques.

Fig. 2 is a schematic diagram of depth information of an acquired image of a depth panorama obtained by opening a camera of a terminal device in a panoramic image generation method provided in an embodiment of the present application. In fig. 2, when a user wants to take a panoramic image, the camera of the terminal device may be turned on, the "more" function button in fig. 2 is clicked, and a "depth panoramic image" mode is selected to perform image acquisition of multiple viewing angles on a subject that wants to generate a panoramic image. In the "depth panorama" mode, the user may fix the position of the camera, change the shooting angle of view of the camera clockwise or counterclockwise, and collect an image within a desired angle of view range, where the image within the angle of view range includes all the shot objects. In this case, in the shooting mode, the panoramic image generation device, which is a camera of the terminal device, automatically acquires depth information of images at a plurality of angles of view of the panoramic image.

Optionally, the number of the first images may also be multiple, and the three-dimensional reconstruction is performed based on the first image of the shooting scene and the depth information of the first image to obtain the first local three-dimensional model, including:

and performing three-dimensional reconstruction based on at least two first images of a shooting scene and the depth information of the at least two first images to obtain a first local three-dimensional model.

Fig. 3 is a schematic diagram of images of multiple viewing angles acquired in a panoramic image generation method provided in an embodiment of the present application. In fig. 3, the acquired images of multiple viewing angles include one image of a positive viewing angle (i.e., the image (b) in the middle of fig. 3) and the first image (i.e., the images (a) and (c) in fig. 3) having an angle with the image of the positive viewing angle.

Wherein the front view angle image is an image in which an angle with a plane projected by all the photographic subjects in the target panoramic image is zero. Because the observation visual angle of the generated target panoramic image is consistent with the direction of the normal visual angle image, the establishment of the three-dimensional model of the normal visual angle image can be omitted in the process of establishing the three-dimensional model, so that the computing resource is saved, and the modeling efficiency of the first local three-dimensional model is improved.

Fig. 4 is a schematic diagram of a local three-dimensional model established in the panoramic image generation method according to the embodiment of the present application. In fig. 4, in order to improve the efficiency of building the three-dimensional model and save the computing resources, the front view image may not participate in building the local three-dimensional model.

In addition, the specific three-dimensional model building process for obtaining the first local three-dimensional model by performing three-dimensional reconstruction based on the first image of the shooting scene and the depth information of the first image may refer to the existing three-dimensional model building process, and as long as the method for building the three-dimensional model based on the depth information of the image and the two-dimensional image acquired by the camera is applicable to the present application, the embodiment of the present application is not specifically limited to this.

And S120, adjusting the first observation visual angle of the first local three-dimensional model to a second observation visual angle to obtain a second local three-dimensional model.

And at the corresponding position of the second observation visual angle, the shooting object corresponding to the first image is in the shooting range of the panoramic image generation device.

Optionally, after the first local three-dimensional model is converted into the two-dimensional image by means of projection transformation, in order to avoid a significant deformation problem occurring in the target panoramic image generated after the converted two-dimensional image is spliced, in the embodiment of the present application, before the first local three-dimensional model is converted into the two-dimensional image by means of projection transformation, the observation angle of the first local three-dimensional model may be adjusted, so that the panoramic image generation apparatus can include all the objects in the target panoramic image generated last at the corresponding position of the adjusted observation angle. Specifically, adjusting a first observation angle of the first local three-dimensional model to a second observation angle to obtain a second local three-dimensional model, including:

taking the position of a camera of a panoramic image generation device for acquiring a first image as the vertex of the visual cone;

and moving the vertex of the viewing cone in a direction away from the shooting object corresponding to the first image, namely moving the panoramic image generation device in a direction away from the shooting object corresponding to the first image so as to adjust the observation angle of the first local three-dimensional model to a second observation angle, so that the panoramic image generation device can shoot all the shooting objects in the target panoramic image at the corresponding position of the second observation angle.

Fig. 5 is a schematic diagram illustrating that a vertex of a viewing cone is adjusted so that a first observation angle of a first local three-dimensional model is adjusted to a second observation angle in a panoramic image generation method according to an embodiment of the present application. In fig. 5, when a camera of a terminal device serving as a panoramic image generation apparatus takes a photograph, since the camera is closer to a subject corresponding to a first image in an initial state, a single field of view of the camera cannot cover all the subjects corresponding to the first image and a front view angle image. While the field angle of the camera is usually not changed, the embodiment of the present application chooses to change the scene range shot by the camera in the case of changing the object distance, i.e. to increase the object distance, so as to expand the scene range shot by the camera.

Specifically, the panoramic image generation device may be a viewing cone vertex, and in order to obtain a larger viewing field, the viewing cone vertex is translated backwards until the shooting viewing field of the panoramic image generation device can cover all the objects in the target panoramic image, so that the target panoramic image finally presented is visually prevented from being deformed, and the image quality of the generated panoramic image is improved.

And S130, generating a target panoramic image based on the second local three-dimensional model and the front view angle image.

Optionally, since the obtained second local three-dimensional model is three-dimensional, and the target panoramic image that needs to be generated is a two-dimensional image, the second local three-dimensional model may be converted into a two-dimensional image by means of projection transformation. Specifically, generating a target panoramic image based on the second local three-dimensional model and the front view angle image includes:

obtaining a first two-dimensional image according to the first image and the second local three-dimensional model;

and carrying out image synthesis on the first two-dimensional image and the front view angle image to obtain a target panoramic image.

Optionally, obtaining a first two-dimensional image according to the first image and the second local three-dimensional model includes:

acquiring a first distance between a vertex of a viewing cone at a second observation angle in the second local three-dimensional model and a camera in the panoramic image generation device, a second distance between the vertex of the viewing cone at the second observation angle and a shot object, and an observation angle in the horizontal direction and an observation angle in the vertical direction in the second observation angle;

and obtaining a first two-dimensional image based on the first distance, the second distance, the observation visual angle in the horizontal direction and the observation visual angle in the vertical direction in the second observation visual angle, the first image and the second local three-dimensional model.

Projective transformation is a process of projectively transforming a three-dimensional object into a two-dimensional image. Fig. 6 is a schematic diagram of a camera space and a world space in a panoramic image generation method provided in an embodiment of the present application. In fig. 6, all objects in the three-dimensional scene will only participate in rendering if they are in the viewing cone. And if the part exceeds the view cone, the part does not participate in the rendering. In fig. 6, the specific geometry of the "viewing cone" is limited to two angles between the observation viewing angle in the horizontal direction (english name fov h) and the observation viewing angle in the vertical direction (english name fov). Furthermore, the position of the pyramid top and bottom can also be determined, which can also be referred to as "clipping plane". The plane where the camera is located is generally referred to as a pyramid top surface (english name near plane), and the plane where the object is located is generally referred to as a pyramid bottom surface (english name far plane). The distance from the top surface of the pyramid to the camera is n, and the distance from the bottom surface of the pyramid to the camera is f.

The representation of an object in three-dimensional space is usually represented by world coordinates, which can be converted to a camera coordinate system when the camera is used to photograph the object. This requires a conversion between two coordinate systems, specifically a translation and a rotation to transform world coordinates into camera coordinates.

The coordinate system in the camera space takes the direction in which the camera lens is facing as the Z-axis (which may be a positive or negative direction) in order to calculate the "Z/depth value" of an object in three-dimensional space, when there are multiple objects in the three-dimensional scene, there may be multiple objects mapped to the same pixel from the viewpoint of the viewing cone. At this time, the color of the object with the minimum absolute value of "Z value/depth value" may be selected as the color of the pixel, that is, the color of the object close to the camera may be selected as the color of the pixel.

Wherein, the matrix formula of projective transformation is as follows:

and (x, y and z) are coordinates of an object in the second local three-dimensional model, n is a first distance between a vertex of the viewing cone at the second observation angle and the panoramic image generation device, and f is a second distance between the vertex of the viewing cone at the second observation angle and all the shot objects in the generated target panoramic image.

The result after normalizing the calculation result is as follows:

resulting coordinates in two-dimensional space

I.e. corresponding to the coordinates (x, y, z) in the second local three-dimensional model.

Optionally, in order to improve the generation efficiency of the panoramic image, the front view angle image may be retained, and after the three-dimensional model is converted into the two-dimensional image by means of projective transformation, the front view angle image is spliced into the two-dimensional image according to the complete shape of the object in the image, so as to generate the target panoramic image. Specifically, the image synthesis of the first two-dimensional image and the front view angle image to obtain the target panoramic image includes:

and splicing the first two-dimensional image and the front view angle image to generate a target panoramic image.

Fig. 7 is a schematic diagram of a panoramic image generated before the panoramic image generation method provided by the embodiment of the present application is adopted and a panoramic image generated after the panoramic image generation method provided by the embodiment of the present application is adopted. The left diagram (a) of fig. 7 is a schematic diagram of a panoramic image generated without adjusting the observation angle of the first partial three-dimensional model, and the right diagram (b) of fig. 7 is a schematic diagram of a panoramic image generated with adjusting the first observation angle to the second observation angle of the first partial three-dimensional model. Obviously, the panoramic image generated from the first observation angle to the second observation angle of the first local three-dimensional model is adjusted, so that obvious image deformation can be avoided, and the image quality of the panoramic image is improved.

In the embodiment of the application, when a panoramic image is shot, three-dimensional reconstruction can be performed based on a first image of a shot scene and depth information of the first image to obtain a first local three-dimensional model, the first image is an image except an elevation angle image in the shot scene, a first observation angle of the first local three-dimensional model is adjusted to a second observation angle to obtain a second local three-dimensional model, wherein a shot object corresponding to the first image is in a shooting range of a panoramic image generation device at a corresponding position of the second observation angle, and finally a target panoramic image is generated based on the second local three-dimensional model and the elevation angle image. On one hand, the depth information of the image of the shooting scene can be obtained, and when a local three-dimensional model is established, the establishment of the three-dimensional model of the front view angle image is omitted, so that the establishment efficiency of the three-dimensional model is improved; on the other hand, the observation angle of view of the local three-dimensional model is adjusted, so that the panoramic image generation device can contain all the shooting objects at the corresponding position of the adjusted observation angle of view, the finally generated panoramic image cannot be obviously deformed in vision, and the quality of the panoramic image is improved.

In the panoramic image generation method provided in the embodiment of the present application, the execution subject may be a panoramic image generation apparatus, or a control module in the panoramic image generation apparatus for executing the panoramic image generation method. The panoramic image generation apparatus provided in the embodiment of the present application will be described with reference to an example in which a panoramic image generation apparatus executes a panoramic image generation method.

Referring to fig. 8, a schematic structural diagram of a panoramic image generation apparatus 800 according to an embodiment of the present invention is shown in fig. 8, which includes a model building unit 802, an adjusting unit 803, and an image generating unit 804, where:

a model establishing unit 802, configured to perform three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image to obtain a first local three-dimensional model, where the first image is an image of the shooting scene except for an orthoscopic image;

an adjusting unit 803, configured to adjust a first observation angle of the first local three-dimensional model to a second observation angle, so as to obtain a second local three-dimensional model, where, at a corresponding position of the second observation angle, a shooting object corresponding to the first image is within a shooting range of the panoramic image generation apparatus;

an image generating unit 804, configured to generate a target panoramic image based on the second local three-dimensional model and the forward view image.

Optionally, in an embodiment, the image generating unit 804 is configured to:

and carrying out image synthesis on the first two-dimensional image and the positive visual angle image to obtain the target panoramic image.

Optionally, in an embodiment, the image generating unit 804 is configured to:

acquiring a first distance between a vertex of a viewing cone at a second observation angle in the second local three-dimensional model and a camera in the panoramic image generation device, a second distance between the vertex of the viewing cone at the second observation angle and a shooting object, and an observation angle in the horizontal direction and an observation angle in the vertical direction in the second observation angle;

and obtaining the first two-dimensional image based on the first distance, the second distance, an observation visual angle in the horizontal direction and an observation visual angle in the vertical direction in the second observation visual angle, the first image and the second local three-dimensional model.

Optionally, in an embodiment, before the model building unit 802 performs three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image, the apparatus further includes:

an information obtaining unit 801, configured to obtain depth information of the first image through a TOF lens or a multi-view vision technique.

Optionally, in an embodiment, the model establishing unit 802 is configured to:

and performing three-dimensional reconstruction based on at least two first images of a shooting scene and depth information of the at least two first images to obtain the first local three-dimensional model.

The panoramic image generation apparatus 800 according to the embodiment of the present application, when a panoramic image is captured, can perform three-dimensional reconstruction based on a first image of a captured scene and depth information of the first image to obtain a first local three-dimensional model, the first image being an image other than an elevation angle image in the captured scene, adjust a first observation angle of the first local three-dimensional model to a second observation angle to obtain a second local three-dimensional model, wherein a captured object corresponding to the first image is within a capture range of the panoramic image generation apparatus at a position corresponding to the second observation angle, and finally generate a target panoramic image based on the second local three-dimensional model and the elevation angle image. On one hand, the depth information of the image of the shooting scene can be obtained, and when a local three-dimensional model is established, the establishment of the three-dimensional model of the front view angle image is omitted, so that the establishment efficiency of the three-dimensional model is improved; on the other hand, the observation angle of view of the local three-dimensional model is adjusted, so that the panoramic image generation device can contain all the shooting objects at the corresponding position of the adjusted observation angle of view, the finally generated panoramic image cannot be obviously deformed in vision, and the quality of the panoramic image is improved.

The panoramic image generation apparatus in the embodiment of the present application may be an apparatus, or may be a component, an integrated circuit, or a chip in a terminal. The device can be mobile electronic equipment or non-mobile electronic equipment. By way of example, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a palm top computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and the non-mobile electronic device may be a server, a Network Attached Storage (NAS), a Personal Computer (PC), a Television (TV), a teller machine or a self-service machine, and the like, and the embodiments of the present application are not particularly limited.

The panoramic image generation apparatus in the embodiment of the present application may be an apparatus having an operating system. The operating system may be an Android (Android) operating system, an ios operating system, or other possible operating systems, and embodiments of the present application are not limited specifically.

The panoramic image generation apparatus provided in the embodiment of the present application can implement each process implemented by the method embodiments of fig. 1 to 7, and is not described herein again to avoid repetition.

Optionally, as shown in fig. 9, an electronic device M09 is further provided in an embodiment of the present application, and includes a memory M91, a processor M92, and a program or an instruction stored in the memory M91 and executable on the processor M92, where the program or the instruction when executed by the processor M92 implements each process of the above-described panoramic image generation method embodiment, and can achieve the same technical effect, and details are not repeated here to avoid repetition.

It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

Fig. 10 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 1000 includes, but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010.

Those skilled in the art will appreciate that the electronic device 1000 may further comprise a power source (e.g., a battery) for supplying power to various components, and the power source may be logically connected to the processor 1010 through a power management system, so as to implement functions of managing charging, discharging, and power consumption through the power management system. The electronic device structure shown in fig. 10 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than those shown, or combine some components, or arrange different components, and thus, the description is not repeated here.

The processor 1010 performs three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene except for an orthoscopic image; adjusting a first observation angle of the first local three-dimensional model to a second observation angle to obtain a second local three-dimensional model, wherein a shooting object corresponding to the first image is in a shooting range of the panoramic image generation device at a corresponding position of the second observation angle; and generating a target panoramic image based on the second local three-dimensional model and the positive visual angle image.

Optionally, the processor 1010 is further configured to obtain a first two-dimensional image according to the first image and the second local three-dimensional model;

Optionally, the processor 1010 is further configured to obtain a first distance between a vertex of a viewing cone at a second observation angle in the second local three-dimensional model and a camera in the panoramic image generation apparatus, a second distance between the vertex of the viewing cone at the second observation angle and a shooting object, and an observation angle in a horizontal direction and an observation angle in a vertical direction in the second observation angle;

Optionally, the processor 1010 is further configured to obtain depth information of the first image through a TOF lens or a multi-view vision technique.

Optionally, the processor 1010 is further configured to perform three-dimensional reconstruction based on at least two first images of a shooting scene and depth information of the at least two first images, so as to obtain the first local three-dimensional model.

By adopting the method provided by the application, on one hand, the depth information of the image of the shooting scene can be obtained, and when the local three-dimensional model is established, the establishment of the three-dimensional model of the front view angle image is omitted, so that the establishment efficiency of the three-dimensional model is improved; on the other hand, the observation angle of view is adjusted for the projection of the local three-dimensional model, so that the panoramic image generation device can contain all the shooting objects at the corresponding position of the adjusted observation angle of view, and further, the finally generated panoramic image is not obviously deformed in vision, and the quality of the panoramic image is improved.

It should be understood that in the embodiment of the present application, the input Unit 1004 may include a Graphics Processing Unit (GPU) 10041 and a microphone 10042, and the Graphics Processing Unit 10041 processes image data of still pictures or videos obtained by an image capturing device (such as a camera) in a video capturing mode or an image capturing mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 1007 includes a touch panel 10071 and other input devices 10072. The touch panel 10071 is also referred to as a touch screen. The touch panel 10071 can include two portions, a touch detection device and a touch processor. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail herein. The memory 1009 may be used to store software programs as well as various data, including but not limited to application programs and operating systems. Processor 1010 may integrate an application processor that handles primarily operating systems, user interfaces, applications, etc. and a modem processor that handles primarily wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 1010.

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the above-mentioned panoramic image generation method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.

The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and so on.

The embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement each process of the above panoramic image generation method embodiment, and can achieve the same technical effect, and in order to avoid repetition, the description is omitted here.

It should be understood that the chips mentioned in the embodiments of the present application may also be referred to as system-on-chip, system-on-chip or system-on-chip, etc.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Further, it should be noted that the scope of the methods and apparatus of the embodiments of the present application is not limited to performing the functions in the order illustrated or discussed, but may include performing the functions in a substantially simultaneous manner or in a reverse order based on the functions involved, e.g., the methods described may be performed in an order different than that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.

While the present embodiments have been described with reference to the accompanying drawings, it is to be understood that the invention is not limited to the precise embodiments described above, which are meant to be illustrative and not restrictive, and that various changes may be made therein by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A panoramic image generation method applied to a panoramic image generation device is characterized by comprising the following steps:

2. The method of claim 1, wherein generating a target panoramic image based on the second local three-dimensional model and the forward perspective image comprises:

3. The method of claim 2, wherein deriving a first two-dimensional image from the first image and the second local three-dimensional model comprises:

4. The method of claim 1, wherein prior to the three-dimensional reconstruction based on the first image of the captured scene and the depth information of the first image, the method further comprises:

and acquiring the depth information of the first image through a TOF lens or a multi-view vision technology.

5. The method according to any one of claims 1 to 4, wherein performing three-dimensional reconstruction based on a first image of a captured scene and depth information of the first image to obtain a first local three-dimensional model comprises:

6. A panoramic image generation apparatus, comprising:

7. The apparatus of claim 6, wherein the image generation unit is to:

8. The apparatus of claim 7, wherein the image generation unit is to:

9. The apparatus of claim 6, wherein the model building unit, prior to three-dimensional reconstruction based on a first image of a captured scene and depth information of the first image, further comprises:

and the information acquisition unit is used for acquiring the depth information of the first image through a TOF lens or a multi-view vision technology.

10. The apparatus according to any one of claims 6 to 9, wherein the model building unit is configured to:

11. An electronic device comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, the program or instructions when executed by the processor implementing the steps of the panoramic image generation method of any one of claims 1-5.

12. A readable storage medium, on which a program or instructions are stored, which when executed by a processor, implement the steps of the panoramic image generation method according to any one of claims 1 to 5.