WO2018068719A1

WO2018068719A1 - Image stitching method and apparatus

Info

Publication number: WO2018068719A1
Application number: PCT/CN2017/105657
Authority: WO
Inventors: 袁梓瑾; 简伟华
Original assignee: 腾讯科技（深圳）有限公司
Priority date: 2016-10-12
Filing date: 2017-10-11
Publication date: 2018-04-19
Also published as: CN106331527A; CN106331527B

Abstract

Disclosed are an image stitching method and apparatus. The method is applied to an image pick-up device comprising at least two image pick-up apparatuses, and comprises: acquiring images respectively picked up by at least two image pick-up apparatuses; with regard to each image pick-up apparatus, constructing a three-dimensional coordinate system of the image pick-up apparatus by taking a pre-set common optical center of the at least two image pick-up apparatuses as an origin; with regard to each pixel in an image picked up by each image pick-up apparatus, executing the following processing: converting a first coordinate of the pixel in a two-dimensional coordinate system of the image into a second coordinate in the three-dimensional coordinate system, and according to the optical center of the image pick-up apparatus and a specific target object point in the image, correcting the second coordinate so as to obtain a third coordinate; and stitching all images according to the third coordinate of each pixel in all the images.

Description

Image mosaic method and device

The present application claims priority to Chinese Patent Application No. 201610890008.9, filed on Jan. 12,,,,,,,,,,,,,,,,,, .

Technical field

The present application relates to the field of image processing technologies, and in particular, to an image stitching method and apparatus.

Background of the invention

At present, 360-degree panoramic video has gradually become one of the main contents in the field of virtual reality. Compared to traditional limited-view video, this panoramic video provides users with a more realistic and immersive viewing experience. Since the single-lens system for collecting panoramic video is still rare, it is generally composed of video captured by multiple camera devices or multiple lens systems.

Summary of the invention

In view of this, the present invention provides an image splicing method and apparatus, which can provide a spliced image without parallax and improve resource utilization of the image splicing device.

The technical solution of the present invention is implemented as follows:

The present invention provides an image stitching method for an image pickup apparatus including at least two image pickup apparatuses, the method comprising:

Acquiring an image captured by each of the at least two camera devices;

For each imaging device, constructing a three-dimensional coordinate system of the imaging device with a common optical center of the at least two imaging devices as a starting point;

For each pixel in an image captured by each camera, the following processing is performed:

Converting the first coordinate of the pixel in the two-dimensional coordinate system of the image into the three-dimensional coordinate system The second coordinate below;

Correcting the second coordinate according to an optical center of the imaging device and a target object point specified in the image to obtain a third coordinate; and

All images are stitched according to the third coordinate of each pixel in all images.

The present invention also provides an image splicing apparatus comprising a processor and a memory, wherein the memory stores instructions executable by the processor, and when the instruction is executed, the processor is configured to:

Acquiring images captured by at least two camera devices;

For each pixel in an image captured by each camera device, performing a process of converting a first coordinate of the pixel in a two-dimensional coordinate system of the image to a second coordinate in the three-dimensional coordinate system; Correcting the second coordinate to obtain a third coordinate by the optical center of the imaging device and the target object point specified in the image; and

The invention further provides a computer readable storage medium storing computer readable instructions for causing at least one processor to perform the method described above.

The present invention further provides an image pickup apparatus comprising at least two image pickup apparatuses, an image display apparatus, a processor, and a memory, wherein the memory stores instructions executable by the processor, when the instructions are executed, The processor is used to:

Acquiring an image captured by each of the at least two camera devices;

For each pixel in an image captured by each camera, a process of converting the first coordinate of the pixel in the two-dimensional coordinate system of the image into the three-dimensional coordinate system is performed a second coordinate; correcting the second coordinate according to an optical center of the imaging device and a target object point specified in the image to obtain a third coordinate; and

All images are stitched according to the third coordinate of each pixel in all images;

The stitched image is displayed by the image display device.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following drawings will be briefly described, and the drawings in the following description are only some embodiments of the present application. Other drawings may also be obtained from those of ordinary skill in the art in light of the inventive work. among them,

FIG. 1a is a schematic diagram of an implementation environment according to an embodiment of the invention; FIG.

FIG. 1b is an exemplary flowchart of an image stitching method according to an embodiment of the invention; FIG.

2 is a schematic diagram of constructing a Cartesian coordinate system in accordance with an embodiment of the present invention;

FIG. 3 is an exemplary flowchart of a method for compensating an optical center offset according to an embodiment of the invention; FIG.

4a is a schematic diagram of coordinates for correcting a second coordinate according to an embodiment of the invention;

4b is a schematic diagram of coordinates for determining an offset according to an embodiment of the invention;

FIG. 5 is an exemplary flowchart of an image stitching method according to another embodiment of the present invention; FIG.

6a is a schematic diagram of a two-dimensional image before splicing according to an embodiment of the invention;

6b is a schematic diagram of a two-dimensional image after splicing according to an embodiment of the invention;

FIG. 7 is a schematic structural diagram of an image splicing apparatus according to an embodiment of the present invention; FIG.

FIG. 8 is a schematic structural diagram of an image splicing apparatus according to another embodiment of the present invention.

Implementation

The technical solutions in the embodiments of the present application are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present application. It is obvious that the described embodiments are a part of the embodiments of the present application. Rather than all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

According to the optical perspective geometry of the lens, two-dimensional imaging captured by two non-co-optic lenses always has a certain parallax in their common field of view. Moreover, at different depths, the degree of parallax is different, which ultimately leads to visually unacceptable flaws in the stitched image, such as ghosting, ghosting, and continuous line misalignment. Therefore, the spliced image has a poor effect, affects the user's viewing experience, and reduces the resource utilization of the imaging device.

The image splicing method and apparatus in the embodiments of the present invention are applicable to any image pickup apparatus having at least two image pickup apparatuses, wherein the angles of view of two adjacent image pickup apparatuses have a common portion, that is, a common view portion, and images taken by the two Has overlapping parts. According to the method in the embodiment of the present invention, the images captured by each camera device are respectively processed, and then the images are stitched in the entire imaging device, and the complete target object point (or depth surface) can be obtained completely. A panoramic image without parallax.

FIG. 1a is a schematic diagram of an implementation environment according to an embodiment of the invention. As shown in FIG. 1a, the imaging system 100 includes a target object 200 and an imaging device 300. The imaging device 300 further includes an image splicing device 310, an image display device 320, and imaging devices 331-335. All the imaging devices 331-335 are combined. Together, it can form a 360-degree panoramic shot.

According to an embodiment of the present invention, when the user moves to a certain scene using the imaging apparatus 300, the target object 100 is photographed in response to a user operation, and each of the imaging apparatuses 331 to 335 captures an image for the target object 100, and The captured image is transmitted to the image splicing device 310 for splicing, and then the image splicing device 310 transmits the spliced panoramic image to the image display device 320 for display for viewing by the user.

In practical applications, the imaging device 300 may be a wearable smart terminal, each camera The device is a single camera lens that can take a single image or take multiple consecutive images. The image display device 320 is a display screen, and provides a visual interface for the user to display the stitched panoramic image.

FIG. 1b is an exemplary flowchart of an image stitching method according to an embodiment of the invention. The method is applied to an image pickup apparatus including at least two image pickup apparatuses, as shown in FIG. 1b, comprising the following steps:

Step 101: Acquire images captured by at least two imaging devices.

In this step, first, images captured by all the imaging devices in the imaging device are acquired.

Step 102: For each imaging device, construct a three-dimensional coordinate system of the imaging device with the common optical center of the preset at least two imaging devices as an origin.

Since each camera device has an optical center of its own lens, in this step, a common optical center is first preset, which is for the entire imaging device, that is, all the imaging devices have such an ideal. The optical center is used as a starting point to construct a three-dimensional coordinate system for each camera.

If the three-dimensional coordinate system is represented as (X, Y, Z), when the three-dimensional coordinate system of the imaging device is constructed with the preset common optical center as the origin, the specific method includes: taking the common optical center as the origin, and the imaging surface of the imaging device The two-dimensional coordinate system (X, Y) is established on the parallel plane, and then the Z-axis is determined according to the two-dimensional coordinate system (X, Y) and the right-hand rule.

In an embodiment, the three-dimensional coordinate system is a Cartesian coordinate system. This Cartesian coordinate system is also referred to as a Cartesian world coordinate system with respect to the coordinate system of the camera. 2 is a schematic diagram of constructing a Cartesian coordinate system in accordance with an embodiment of the present invention. As shown in FIG. 2, the X-axis, the Y-axis, and the Z-axis together constitute a Cartesian coordinate system of the image pickup device A, and the common optical center O is the origin of the coordinate system. Incident light

The lens system of the imaging device A is entered at an angle θ, and after being refracted by the lens, it is imaged on the imaging surface x'o'y' of the imaging device A. Among them, the XOY face and the x'o'y' face are parallel. Thus, a two-dimensional coordinate system (X, Y) is established on the parallel plane XOY plane of the imaging plane x'o'y', and then the Z-axis is determined based on the two-dimensional coordinate system (X, Y) and the right-hand rule.

Step 103: Performing the following processing for each pixel in an image captured by each camera:

Step 1031: Convert the first coordinate of the pixel in the two-dimensional coordinate system of the image to the second coordinate in the three-dimensional coordinate system;

Here, the captured image is two-dimensional, and each pixel has a two-dimensional coordinate, that is, a first coordinate, in the two-dimensional coordinate system of the image.

Step 1032: Correct the second coordinate according to the optical center of the imaging device and the target object point specified in the image to obtain a third coordinate.

For step 1031, converting the first coordinate to the second coordinate specifically includes: determining an angular coordinate of the pixel according to the first coordinate, determining incident light and the three-dimensional coordinate according to the lens imaging geometric function and the first coordinate of the imaging device. The angle between the Z axes in the system (X, Y, Z), and then calculate the second coordinates based on the angle and the angular coordinates.

If the first coordinate of a pixel is represented as (x ₁ , y ₁ ), the angular coordinate is expressed as

Determining an angular coordinate of the pixel according to the first coordinate comprises determining

The following trigonometric values are:

If the second coordinate is expressed as (x ₂ , y ₂ , z ₂ ) and the angle is expressed as θ, x ₂ , y ₂ and z ₂ in the second coordinate are calculated according to the following formula:

If the lens imaging geometric function of the imaging device is r(θ), when the lens of the imaging device is rectilinear, r(θ)=f·tan(θ), the angle is

When the lens of the camera device is equidistant, there is r(θ)=f·θ, then the angle is

Among them, atan (·) represents the inverse tangent function, pw, ph represents the width and height of the pixel, respectively, f is the focal length of the lens (as shown in Figure 2).

Corresponding to FIG. 2, one pixel p ₁ ' in the imaging plane x'o'y', the first coordinate is (x ₁ , y ₁ ), the connection between p ₁ ' and the origin o' and x' The angle between the o' axes is

Converted to the Cartesian coordinate system (X, Y, Z), corresponding to the object point P ₁ , its three-dimensional coordinates are shown in formula (2). Wherein, the projection of P ₁ on the two-dimensional plane of XOY is p ₁ , and the angle between the line between p ₁ and the origin O and the XO axis is also

The above-mentioned common optical center O is unique to all the imaging devices, but considering that each of the imaging devices has its own optical center O' in practice, it is necessary to perform imaging on the image according to the deviation between the optical centers. The compensation is such that it is consistent with the imaging of O under the origin.

In this regard, FIG. 3 is an exemplary flowchart of an optical center offset compensation method according to an embodiment of the present invention. For the step 1032, the second coordinate is corrected according to the optical center of the imaging device and the target object point specified in the image, and the third coordinate is obtained. As shown in FIG. 3, the method includes the following steps:

Step 301: Obtain a distance between the public optical center and the target object point, that is, obtain a depth of the target object point.

In this step, the target object point may be specified by the user according to the object point of interest in the captured image, or may be specified according to the main target object or the content in the scene. After specifying the target point, estimate the common optical center and target point on the XOZ plane. The distance between them. For example, the depth of the target point in a specific scene is estimated to be 10 m, or 20 m, etc. according to third party software.

4a is a schematic diagram of coordinates for correcting a second coordinate according to an embodiment of the invention. As shown in Figure 4a, the target point is incident light.

On the object point P ₁ , the above distance is the projection of P ₁ on the XOZ plane

The length, that is, the length between 0 and P', is denoted as R ₀ , which is also referred to as the depth of the object point P ₁ .

Step 302: Obtain an offset of the optical center of the imaging device relative to the common optical center.

In this step, considering that there is an overlapping portion between images captured by two adjacent imaging devices in one imaging device, regression or simulation estimation may be performed according to sample data of the overlapping image and corresponding/matching relationship with the imaging device. The above offset is given. For example, a panoramic (ie 360°) video system in which three cameras are placed in a three-dimensional space, each camera capturing an image within a certain range of viewing angles.

4b is a schematic diagram of coordinates for determining an offset according to an embodiment of the invention. As shown in Fig. 4b, in the ABC coordinate system constructed by the three-dimensional spherical surface 400,

cameras

401 and 402 are arranged at different positions, and the images taken by the two have overlapping portions. The offset between the optical center O' of each camera and the origin O can be determined from the sample data of the superimposed image. Back to the figures, the optical center O 4a 'with respect to the origin O in the offset X axis, Y axis and Z axis are _{_{_{T x, T y, T z}}} .

Step 303, calculating a third coordinate according to the distance, the offset, and the second coordinate.

Correcting the second coordinate, each coordinate value x ₃ , y _{3 ,} and z _{3 in the} third coordinate (x ₃ , y ₃ , z ₃ ) can be calculated according to the following formula:

among them,

b=2·(T _z ·z ₂ +T _x ·x ₂ ).

Step 104, splicing all images according to the third coordinate of each pixel in all images.

After performing the above processing for each pixel in each image, according to the position of each imaging device in the imaging device, all the processed images are spliced according to a certain projection type, thereby obtaining the target object point. A panoramic image without any parallax on the depth surface.

In this embodiment, by acquiring images captured by at least two imaging devices, for each imaging device, a three-dimensional coordinate system of the imaging device is constructed with a common optical center of at least two preset imaging devices as an origin, Each pixel in an image captured by each camera device performs a process of converting a first coordinate of the pixel in a two-dimensional coordinate system of the image to a second coordinate in the three-dimensional coordinate system; The optical center of the device and the target object point specified in the image, the second coordinate is corrected to obtain a third coordinate, and all the images are spliced according to the third coordinate of each pixel in all images, thereby providing a non-parallax splicing The depth surface technology can adaptively select the depth position of the main content in the scene as the non-parallax stitching depth surface, so that the main content in the scene presents a non-parallas stitching effect.

In addition, the coordinate conversion and the compensation of the optical center offset in the above method are independent of the geometric characteristics of the target object point, and are not dependent on the shape of the specific target object point, and are more suitable for video applications whose content is constantly changing in the time dimension. Compared with the prior art, the above method does not need to perform feature detection and feature matching on the scene content, so that the user can specify the target object quickly and flexibly. The point (or the specified non-parallax stitching depth surface) provides a complete alignment of the object point or scene content at the desired location, providing a stitched image with no parallax. Moreover, the above method and the specific imaging geometric formula of the imaging device and the projection type of the final stitching are also irrelevant, and therefore, have versatility, and improve resource utilization of the image splicing device.

FIG. 5 is an exemplary flowchart of an image stitching method according to another embodiment of the present invention. As shown in FIG. 5, the method is applied to an image pickup apparatus including at least two image pickup apparatuses, and includes the following steps:

Step 501: Acquire an image captured by each of the at least two imaging devices.

Step 502: For each camera device, construct a Cartesian coordinate system of the camera device with the common optical center of the preset at least two camera devices as an origin.

Step 503, performing the following processing for each pixel in an image captured by each camera:

Step 5031, performing coordinate conversion:

Converting the first coordinate of the pixel in the two-dimensional coordinate system of the image to the second coordinate in the Cartesian coordinate system;

In step 5032, the optical center offset compensation is performed:

The second coordinate is corrected based on the optical center of the imaging device and the target object point specified in the image to obtain a third coordinate.

It can be seen from the above formula (2) that the modulus of the second coordinate is 1, that is, the established Cartesian coordinate system is a normalized Cartesian coordinate system. Since the normalized Cartesian coordinate system does not contain depth information, it is in the same incident light.

The upper two object points with different depths have the same normalized Cartesian coordinate values. As shown, p ₁ '2 converted to the corresponding normalized Cartesian coordinate system (X, Y, Z) is not only the object point P _1, except that P _1, may also be along the incident light

Other objects on the point, as shown in Figure 2 P ₂ . The depths of the object points P ₁ and P ₂ are different, that is, the distances between the XOZ planes and the optical center O are different, but the two have the same normalized Cartesian coordinate values (x ₂ , y ₂ , z ₂ ). Both correspond to p ₁ ' on the imaging plane x'o'y'.

Step 504: Project the third coordinate into the unit panoramic sphere according to a preset projection type according to the position of each camera in the imaging device.

When all of the cameras constitute a panoramic camera device, the third coordinates are projected into a unit of panoramic sphere. Preset projection types include, but are not limited to, rectilinear, fisheye, equirectangular, orthographic, stereographic, and the like.

Step 505, splicing all the images in the unit panoramic spherical surface to obtain a panoramic image.

Through the above steps, in the spliced panoramic image, the splicing depth surface without parallax can be reached at the specified target object point position, and the adjacent images are completely aligned, and the effect of no splicing 瑕疵 is obtained. When an image is presented to a user, the three-dimensional panoramic image can be reconverted into a two-dimensional image.

FIG. 6a is a schematic diagram of a two-dimensional image before splicing according to an embodiment of the invention. Wherein, in the left diagram 600, the target point is the first flagpole closest to the lens (as indicated by arrow 601), corresponding to P1-P' shown in Figure 4a. Before the optical center offset compensation, the up, down, left and right image misalignment due to parallax occurs at the flagpole. As can be clearly seen in the right diagram 610, the extra point 611' appears at the lower left of the top end 611 of the flagpole. The flag is originally the image shown at 612, but due to the parallax, the final image is 612' (as indicated by the dotted line). Show).

FIG. 6b is a schematic diagram of a two-dimensional image after splicing according to an embodiment of the invention. Correspondingly, the left picture 620 is the image after the coordinate transformation and the optical center offset compensation, and the upper and lower images are perfectly aligned at the flagpole. As can be clearly seen in the right diagram 630, images that are not aligned outside of the top 611 and the flag 612 disappear, showing a clear flagpole. Visible, implemented in the scene The perfect alignment of the main contents (ie, the flagpole), at the position of the flagpole, becomes the non-parallax stitching depth surface.

In a specific application, reverse processing may also be adopted, that is, the inverse processing is performed pixel by pixel on a blank panoramic canvas (ie, the optical center offset compensation described in step 5032 is sequentially performed, step 5031). The coordinate conversion operation described) finds the pixel position of the image captured by the camera device corresponding thereto, and then interpolates to obtain the actual value of the pixel on the current panoramic canvas.

FIG. 7 is a schematic structural diagram of an image splicing apparatus according to an embodiment of the present invention. As shown in FIG. 7, the image splicing device 700 includes an obtaining module 710, a coordinate system building module 720, a coordinate processing module 730, and a splicing module 740, where

The acquiring module 710 is configured to acquire an image captured by each of the at least two camera devices;

a coordinate system construction module 720, configured to construct, for each camera device, a three-dimensional coordinate system of the camera device with a common optical center of at least two preset imaging devices as an origin;

The coordinate processing module 730 is configured to, for each pixel in an image captured by each camera device, perform a process of converting the first coordinate of the pixel in the two-dimensional coordinate system of the image into the three-dimensional coordinate system. a second coordinate; correcting the second coordinate according to the optical center of the imaging device and the target object point specified in the image to obtain a third coordinate; and

The splicing module 740 is configured to splicing all the images according to the third coordinate of each pixel in all the images.

In an embodiment, the coordinate processing module 730 includes a conversion unit 731 for determining an angular coordinate of the pixel according to the first coordinate, and determining the incident light and the three-dimensional coordinate system according to the lens imaging geometric function and the first coordinate of the imaging device ( The angle between the Z axes in X, Y, Z); the second coordinate is calculated from the angular coordinates and the included angle.

In an embodiment, if the first coordinate is represented by (x ₁ , y ₁ ), the angular coordinate is expressed as

The converting unit 731 is configured to determine:

The three-dimensional coordinate system is a Cartesian coordinate system. If the second coordinate is represented by (x ₂ , y ₂ , z ₂ ) and the angle is represented by θ, the conversion unit 731 is configured to calculate x ₂ , y ₂ and according to the following formula. z ₂ :

z ₂ =cos(θ)

In an embodiment, the coordinate processing module 730 includes a correction unit 732 for acquiring a distance between the common optical center and the target object point; acquiring an offset of the optical center of the imaging device with respect to the common optical center; The offset and the second coordinate calculate the third coordinate.

In an embodiment, if the distance is represented as R ₀ , the offset is expressed as (T _x , T _y , T _z ), the second coordinate is represented as (x ₂ , y ₂ , z ₂ ), and the third coordinate is expressed as (x ₃ , y ₃ , z ₃ ), the correcting unit 732 is configured to calculate x ₃ , y ₃ and z ₃ according to the following formula:

among them,

b=2·(T _z ·z ₂ +T _x ·x ₂ ).

In an embodiment, the splicing module 740 is configured to project the third coordinate into the unit panoramic spherical surface according to a preset projection type according to the position of each camera device in the imaging device; The images are stitched together to obtain a panoramic image.

FIG. 8 is a schematic structural diagram of an image splicing apparatus according to another embodiment of the present invention. The image splicing apparatus 800 can include a processor 810, a memory 820, a port 830, and a bus. 840. Processor 810 and memory 820 are interconnected by a bus 840. Processor 810 can receive and transmit data through port 830. among them,

The processor 810 is configured to execute a machine readable instruction module stored by the memory 820.

The memory 820 stores machine readable instruction modules executable by the processor 810. The instruction module executable by the processor 810 includes an acquisition module 821, a coordinate system construction module 822, a coordinate processing module 823, and a splicing module 824. among them,

The acquiring module 821 may be executed by the processor 810 to: acquire an image captured by each of the at least two camera devices;

The coordinate system construction module 822 may be configured by the processor 810 to: for each camera device, construct a three-dimensional coordinate system of the camera device with the common optical center of the preset at least two camera devices as an origin;

The coordinate processing module 823 may be executed by the processor 810 to: for each pixel in an image captured by each camera device, perform the following process: converting the pixel to the first coordinate in the two-dimensional coordinate system of the image a second coordinate in the three-dimensional coordinate system; correcting the second coordinate according to the optical center of the imaging device and the target object point specified in the image to obtain a third coordinate;

The splicing module 824, when executed by the processor 810, can splicing all of the images based on the third coordinate of each pixel in all of the images.

It can be seen that when the instruction modules stored in the memory 820 are executed by the processor 810, various functions of the acquisition module, the coordinate system construction module, the coordinate processing module, and the splicing module in the foregoing various embodiments can be implemented.

According to still another embodiment of the present invention, an image pickup apparatus includes at least two image pickup apparatuses, an image display apparatus, a processor, and a memory, and the memory stores instructions executable by the processor, and when executing the instructions, the processor is configured to:

Acquiring images captured by at least two camera devices;

For each imaging device, a three-dimensional coordinate system of the imaging device is constructed with a common optical center of at least two preset imaging devices as an origin;

For each pixel in an image captured by each camera device, performing a process of converting a first coordinate of the pixel in a two-dimensional coordinate system of the image to a second coordinate in the three-dimensional coordinate system; The optical center of the imaging device and the target object point specified in the image are corrected to obtain the third coordinate; and,

The stitched image is displayed by the image display device.

In the foregoing apparatus and system embodiments, specific methods for implementing functions of the respective modules and units are described in the method embodiments, and details are not described herein again.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

Additionally, each of the embodiments of the present invention can be implemented by a data processing program executed by a data processing device such as a computer. Obviously, the data processing program constitutes the present invention. Further, a data processing program usually stored in a storage medium is executed by directly reading a program out of a storage medium or by installing or copying the program to a storage device (such as a hard disk and or a memory) of the data processing device. Therefore, such a storage medium also constitutes the present invention. The storage medium can use any type of recording method, such as paper storage medium (such as paper tape, etc.), magnetic storage medium (such as floppy disk, hard disk, flash memory, etc.), optical storage medium (such as CD-ROM, etc.), magneto-optical storage medium ( Such as MO, etc.).

Accordingly, the present invention also discloses a storage medium in which is stored a data processing program for performing any of the above-described embodiments of the present invention.

The above description is only the preferred embodiment of the present application and is not intended to limit the application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and scope of the present application are intended to be included within the scope of the present disclosure.

Claims

An image splicing method is applied to an image pickup apparatus including at least two image pickup apparatuses, and the method includes:

Acquiring an image captured by each of the at least two camera devices;

For each imaging device, constructing a three-dimensional coordinate system of the imaging device with a common optical center of the at least two imaging devices as a starting point;

For each pixel in an image captured by each camera, the following processing is performed:

Converting the pixel to a first coordinate in the two-dimensional coordinate system of the image to a second coordinate in the three-dimensional coordinate system;

Correcting the second coordinate according to an optical center of the imaging device and a target object point specified in the image to obtain a third coordinate; and

All images are stitched according to the third coordinate of each pixel in all images.
The method according to claim 1, wherein if the three-dimensional coordinate system is represented as (X, Y, Z), the image pickup device is constructed with a predetermined common optical center of the at least two camera devices as an origin The three-dimensional coordinate system includes:

Establishing a two-dimensional coordinate system (X, Y) on a parallel plane of the imaging surface of the imaging device with the common optical center as an origin;

The Z axis is determined based on the two-dimensional coordinate system (X, Y) and the right hand rule.
The method of claim 1, wherein the converting the first coordinate of the pixel in the two-dimensional coordinate system of the image to the second coordinate in the three-dimensional coordinate system comprises:

Determining an angular coordinate of the pixel according to the first coordinate;

Determining an angle between incident light and a Z-axis in the three-dimensional coordinate system (X, Y, Z) according to a lens imaging geometric function of the imaging device and the first coordinate;

The second coordinate is calculated according to the angular coordinate and the included angle.
The method according to claim 3, wherein if the first coordinate is expressed as (x 1 , y 1 ), the angular coordinate is expressed as
The determining the angular coordinates of the pixel according to the first coordinate comprises:

determine
The trigonometric values are:

The three-dimensional coordinate system is a Cartesian coordinate system. If the second coordinate is represented by (x 2 , y 2 , z 2 ), the angle is represented by θ, and the first coordinate is calculated according to the angle and the angular coordinate. The two coordinates include:

Calculate x 2 , y 2 and z 2 according to the following formula:

z 2 =cos(θ)
The method according to claim 3 or 4, wherein said incident light and said three-dimensional coordinate system are determined according to a lens imaging geometric function r(θ) of said imaging device and said first coordinate (x 1 , y 1 ) The angle θ between the Z axes in X, Y, Z) includes:

When the lens of the imaging device is linear, r(θ)=f·tan(θ), then

When the lens of the imaging device is of an equidistant type, r(θ)=f·θ, then

Where atan(·) represents the inverse tangent function, pw, ph represents the width and height of the pixel, respectively, and f is the focal length of the lens.
The method according to claim 1, wherein said correcting said second coordinates according to an optical center of said image pickup device and a target object point specified in said image, to obtain a third coordinate include:

Obtaining a distance between the public optical center and the target object point;

Obtaining an offset of an optical center of the camera device relative to the common optical center;

The third coordinate is calculated based on the distance, the offset, and the second coordinate.
The method according to claim 6, wherein said according to said distance R 0 , said offset (T x , T y , T z ) and said second coordinate (x 2 , y 2 , z 2 Calculating the third coordinate (x 3 , y 3 , z 3 ) includes:

Calculate x 3 , y 3 and z 3 according to the following formula:

among them,
b=2·(T z ·z 2 +T x ·x 2 ).
The method according to any one of claims 1 to 7, wherein said splicing all images according to said third coordinate of each pixel in all images comprises:

And projecting the third coordinate into the unit panoramic spherical surface according to a preset projection type according to a position of each camera device in the imaging device;

All the images are spliced in the unit panoramic spherical surface to obtain a panoramic image.
An image splicing apparatus, comprising: a processor and a memory, wherein the memory stores instructions executable by the processor, and when the instructions are executed, the processor is configured to:

Acquiring images captured by at least two camera devices;

For each camera device, the common optical center of the at least two camera devices is preset Constructing a three-dimensional coordinate system of the camera device at an origin;

For each pixel in an image captured by each camera device, performing a process of converting a first coordinate of the pixel in a two-dimensional coordinate system of the image to a second coordinate in the three-dimensional coordinate system; Correcting the second coordinate to obtain a third coordinate by the optical center of the imaging device and the target object point specified in the image; and

All images are stitched according to the third coordinate of each pixel in all images.
The apparatus according to claim 9, wherein, when the instruction is executed, the processor is further configured to: determine an angular coordinate of the pixel according to the first coordinate; according to a lens imaging geometric function and a location of the imaging device The first coordinate determines an angle between the incident light and the Z axis in the three-dimensional coordinate system (X, Y, Z); the second coordinate is calculated according to the angular coordinate and the included angle.
The apparatus according to claim 10, wherein if the first coordinate is expressed as (x 1 , y 1 ), the angular coordinate is expressed as
When the instructions are executed, the processor is further configured to: determine:

The three-dimensional coordinate system is a Cartesian coordinate system. If the second coordinate is represented by (x 2 , y 2 , z 2 ), the angle is represented by θ. When the instruction is executed, the processor is further configured to: The formula calculates x 2 , y 2 and z 2 :

z 2 =cos(θ)
The apparatus according to claim 9, wherein, when the instruction is executed, the processor is further configured to: acquire a distance between the common optical center and the target object point; acquire an optical center of the imaging device An offset from the common optical center; according to the distance, The offset and the second coordinate calculate the third coordinate.
The apparatus according to claim 12, wherein said distance is expressed as (T x , T y , T z ) if said distance is represented as R 0 and said second coordinate is represented as (x 2 , y 2 , z 2 ), the third coordinate is represented as (x 3 , y 3 , z 3 ), when executing the instruction, the processor is further configured to: calculate x 3 , y 3 and according to the following formula z 3 :

among them,
b=2·(T z ·z 2 +T x ·x 2 ).
The apparatus according to any one of claims 9 to 13, wherein, when the instruction is executed, the processor is further configured to follow a preset according to a position of each camera in the image pickup apparatus The projection type projects the third coordinate into the unit panoramic spherical surface; all the images are spliced in the unit panoramic spherical surface to obtain a panoramic image.
A computer readable storage medium, characterized by storing computer readable instructions, which may cause at least one processor to perform the method of any one of claims 1 to 8.
An image pickup apparatus comprising at least two image pickup apparatuses, an image display apparatus, a processor, and a memory, wherein the memory stores an instruction executable by the processor, and when the instruction is executed, the processing Used for:

Acquiring an image captured by each of the at least two camera devices;

For each imaging device, constructing a three-dimensional coordinate system of the imaging device with a common optical center of the at least two imaging devices as a starting point;

For each pixel in an image captured by each camera device, performing a process of converting a first coordinate of the pixel in a two-dimensional coordinate system of the image to a second coordinate in the three-dimensional coordinate system; Correcting the second coordinate to obtain a third coordinate by the optical center of the imaging device and the target object point specified in the image; and

All images are stitched according to the third coordinate of each pixel in all images;

The stitched image is displayed by the image display device.