TWI762353B - Method for generating projection image with scaling adjustment and seam cut stitching - Google Patents

Abstract

An image processing method for receiving M lens images and generating a projection image is disclosed. The image processing method comprises: determining P optimal warping coefficients of P control regions in the projection image according to a 2D error table and the M lens images from an image capture module; generating M face images according to the M lens images, a first vertex list and the P optimal warping coefficients; determining a seam for each of N seam regions; and, stitching two overlapping seam images to generate a stitched seam image for each seam region according to its corresponding seam. The 2D error table comprises multiple test warping coefficients and multiple accumulation pixel value differences in the P control regions. The P control regions are respectively located in the N seam regions that are respectively located in N overlap regions of the M face images, where M＞=2, N＞=1 and P＞=3.

Description

Method for generating projected images with scaling and seam stitching

The present invention relates to image processing, and more particularly, to a method for generating projected images with scaling and seam stitching. The present invention facilitates the generation of projected images in an image processing system.

FIG. 1A shows the relationship between a cube frame 11 and a sphere 12 . As shown in FIG. 1A , a conventional camera has four lenses and is mounted on four working surfaces (right, left, front, and back) of a cube frame 11 respectively, and any two adjacent surfaces of the four working surfaces are Substantially orthogonal, for example, the four working surfaces face the longitudes of 0, 90, 180 and 270 degrees of the virtual sphere 12 respectively, so as to simultaneously capture a 360-degree horizontal field of view (FOV) ) and a 90-degree vertical FOV, and produce four lens images. A necessary condition for setting up the camera with four lenses is that there should be sufficient overlap between the fields of view of any two adjacent lenses to facilitate image mosaicking. Referring to FIG. 1B , the pixels in areas A( 1 ) to A( 2 ) are formed by overlapping two lenses/texture images, while the pixels in area 15 are from a single lens/texture image. Therefore, a stitching operation can be performed on the overlapping regions 13 to form an equirectangular panoramic image.

A perfect projected image processing system should make it impossible for the viewer to perceive that the video is being captured by multiple cameras, so the viewer can experience a seamless experience where multiple cameras look like a single camera.

In view of the above problems, one of the objectives of the present invention is to provide an image processing method to generate a projected image by scaling adjustment and seam splicing.

According to an embodiment of the present invention, an image processing method is provided for receiving M lens images and generating a projection image, the method comprising: determining P in the projection image according to a 2D error table and the M lens images P optimal bonding coefficients of each control area, wherein an image capture module is used to capture a field of view with a horizontal field of view of X degrees and a vertical field of view of Y degrees to generate the M lens images; according to a first The vertex list, the P optimal joint coefficients, and the M lens images, generate M work surface images; determine N seams in the N seam areas; and, according to the seams in each seam area, splicing Two overlapping seam images in the seam area to generate a stitched seam image. Wherein, each working surface image includes z seam images and a main image, and each seam area is located between two adjacent main images in the projected image; the 2D error table includes a plurality of test joint coefficients and A plurality of accumulated pixel value error amounts corresponding to the P control areas; the P control areas are respectively located in the N seam areas, and the N seam areas are respectively located in the M working surfaces within the N overlapping regions of ; the first vertex list includes a plurality of first vertices with data structures that define the first vertex mapping between a single lens image and the projected image; wherein, 1<=z <=2, M>=2, N>=1, P>=3, X<=360, and Y<180.

The above-mentioned and other objects and advantages of the present invention will be described in detail below in conjunction with the following figures, detailed descriptions of the embodiments and the scope of the patent application.

11: Cube Architecture

12: Sphere

15: Non-overlapping regions

21: Image capture module

22: Compensation device

23: Correspondence Generator

40: Ideal imaging position

41, 44: Image Center

42: The location of the control area

43: System Center of Image Capture Module 21

45: Objects

46, 47: Lens Center

48: Actual imaging position

55: Image Engine

56: Seam Engine

71: Intersection/Vertex

72: Quadrilateral

78: Seams

81: Update Windows

82: Separation point

200: Projection Image Processing System

210: Vertex Processing Device 230

220: Image processing device

230: Splicing Decision Unit

510: Destination Buffer

520: Temporary Buffer

560: Rasterization Engine

570: Texture Mapping Circuit

571, 572: Texture mapping engine

580: Measurement unit

[FIG. 1A] shows the relationship between a cube frame and a sphere.

FIG. 1B shows an equidistant rectangular panoramic image, which is derived from the equidistant rectangular projection of four lens images (back, left, right, and front) of an image capture module 21 .

2 is a block diagram showing a projection image processing system according to an embodiment of the present invention.

[FIG. 3A] shows a triangular mesh used to model the surface of a sphere.

[FIG. 3B] shows a polygon mesh used to compose/model the isometric panorama.

[ FIG. 3C ] An example of an equidistant rectangular panoramic image with four overlapping regions and twenty control regions is shown.

[ FIG. 4A ] An example of the relationship between the target vertex P and the ten control regions R( 6 ) to R( 15 ) in the frontal lens image is shown.

[FIG. 4B] shows how the mismatched image defect of an object is improved after the vertex processing device 210 modifies the texture coordinates of all vertices according to the joint coefficient.

5 is a schematic diagram showing the image processing apparatus according to an embodiment of the present invention.

FIG. 6A is a flowchart showing a method for determining the optimal bonding coefficients of all control regions according to an embodiment of the present invention.

FIG. 6B is a flowchart showing a method for the splicing decision unit 230 to perform the coefficient decision operation of step S612 according to an embodiment of the present invention.

[ FIG. 6C ] It is shown that among all the paths of the connection meter, the sum of the connection meter values of the path 0-1-1 (solid line path) is the smallest.

FIG. 7A shows an example of an equidistant rectangular panorama image including four main images and their seam areas.

[ FIG. 7B ] An enlarged view showing the seam area S( 2 ) of FIG. 7A .

[FIG. 7C] A flow chart showing a method of determining an error-minimizing path (or a seam) for a seam region according to an embodiment of the present invention.

[FIG. 7D] shows an example where a seam 78 is located in a seam area and the seam area is located between a left main image and a right main image.

8A is a flowchart showing an image processing method according to an embodiment of the present invention.

[ FIG. 8B ] An example of a section of a seam area is shown, and the seam area is related to a seam 78 , a seam image A, and a seam image B. [ FIG.

[FIG. 8C] shows the size of a seam area and the size and different positions of the update windows 81 in three columns.

[FIG. 8D] shows that a seam area is located at the leftmost and rightmost sides of the equidistant rectangular panoramic image at the same time.

References in the singular form of "a" and "the" throughout this specification and subsequent claims shall include both the singular and the plural, unless the specification expressly states otherwise. Throughout the specification, circuit elements with the same function use the same reference symbols.

FIG. 2 is a block diagram showing a projection image processing system according to an embodiment of the present invention. The projection image processing system 200 of the present invention includes an image capture module 21 , a compensation device 22 and a correspondence generator 23 . The compensation device 22 receives the original vertex list from the correspondence generator 23 and a plurality of lens images from the image capture module 21 to generate a projected image. Please note that the image capture module 21 is a multi-lens camera, which can be a panoramic camera or a wide-angle camera. Therefore, the above-mentioned projected image can be a panoramic image or a wide-angle camera image.

Many projection methods are applicable to the projection image processing system 200 of the present invention. The term "projection" refers to flattening the surface of a sphere into a two-dimensional plane, such as a projected plane. The projections include, but are not limited to, equirectangular projections, cylindrical projections, and modified cylindrical projections. The modified cylindrical projection includes, but is not limited to, Miller projection, Mercator projection, Lambert cylindrical equal area projection, Pannini projection, and the like. Accordingly, the above-mentioned projection images include, but are not limited to, an equidistant rectangular projection image, a cylindrical projection image, and a modified cylindrical projection image. 1A-1B and 3A-3C are related equirectangular projections. The implementations of cylindrical projection and modified cylindrical projection are well known to those skilled in the art, and will not be repeated here. Please note that no matter which projection method is adopted by the projection image processing system 200 of the present invention, the correspondence generator 23 will correspondingly generate an original vertex list (eg, Table 1), which defines the relationship between the lens images and the projection image. vertex mapping relationship. For the sake of clarity and convenience, the following examples and embodiments only use equidistant rectangular projections and equidistant rectangular panoramic images for illustration. It should be noted that the projection image processing system of the present invention The operation of 200 and the image processing method (FIGS. 6A-6B, 7C, and 8A) are also applicable to the wide-angle camera, cylindrical projection, and modified cylindrical projection described above.

The image capture module 21 is a multi-lens camera, which can simultaneously capture a field of view covering X-degree horizontal FOV and Y-degree vertical FOV to generate multiple lens images, where X<=360, Y< 180, such as 360×160 or 180×90, etc. For example, as shown in FIG. 1A , the image capturing module 21 includes four lenses (not shown in the figure) and is respectively erected on four working surfaces of a cube structure 11 to simultaneously capture an image with a 360-degree horizontal view. Field FOV and 90 degree vertical FOV field of view to produce four lens images. Please note that the present invention does not limit the number of lenses included in the image capturing module 21 as long as a field of view of an X degree horizontal FOV and a Y degree vertical FOV can be captured, where X<=360 and Y<180. A necessary condition for setting up the image capture module 21 is that there should be sufficient overlap between the fields of view of any two adjacent lenses to facilitate image stitching.

Relevant terms referred to throughout the specification and subsequent claims are defined below, unless otherwise specifically indicated in this specification. The term "texture coordinates" refers to coordinates in a texture space (such as a texture image or lens image); the term "rasterization" refers to the Image) is mapped to the calculation process of the texture coordinates of each lens image. The processing pipeline of the projection image processing system 200 is divided into an offline phase and an online phase. In the offline stage, the four lenses of the image capture module 21 are respectively calibrated, the correspondence generator 23 uses a suitable image registration technique to generate an original vertex list, and each vertex in the original vertex list Provide the distance between the equidistant rectangular panorama image and the lens images (or between the equidistant rectangular coordinates and the texture coordinates ) mapping relationship. For example, many circles are drawn on the surface of the sphere 12 with a radius of 2 meters (r=2), which are regarded as longitude and latitude, and multiple intersection points thereof are regarded as multiple correction points. The four lenses capture the correction points, and the positions of the correction points on the lens images are known. Then, since the view angles of the correction points and the texture coordinates are linked, a mapping relationship between the equidistant rectangular panoramic image and the lens images can be established. In this specification, a correction point with the above-mentioned mapping relationship is defined as a "vertex". In short, the correspondence generator 23 corrects the relationship between the equidistant rectangular panoramic image and the lens images for each vertex to obtain the original vertex list. In the offline phase, the correspondence generator 23 performs all necessary calculations.

Figure 3A shows a triangular mesh used to model the surface of a sphere. Referring to Figure 3A, the surface of a sphere 12 is modeled using a triangular mesh. Figure 3B shows a polygon mesh used to compose/model the isometric panorama. The polygonal mesh of FIG. 3B is generated by performing an equirectangular projection on the triangular mesh of FIG. 3A , and the polygonal mesh of FIG. 3B is a collection of quadrilaterals and/or triangles.

In the offline stage, according to the geometric shapes of the equidistant rectangular panoramic image and the lens images, the correspondence generator 23 calculates the equidistant rectangular coordinates and texture coordinates for each vertex of the polygonal mesh (FIG. 3B), to produce this original vertex list. Afterwards, the correspondence generator 23 transmits the original vertex list to the vertex processing device 210 . The original vertex list is a list of vertices that form the quads and/or triangles of the polygon mesh (FIG. 3B), and each vertex is defined by a corresponding data structure. The data structure defines between a destination space and a texture space (or the equidistant rectangle and the texture base The vertex mapping relationship between the markers). Table 1 shows an example of the data structure of each vertex in the original vertex list.

For clarity and convenience of description, the following examples and embodiments assume that the image capture module 21 includes four lens images and is respectively erected on the four working surfaces (right, left, front, and back) of the cube frame 11 .

FIG. 3C shows an example of an equidistant rectangular panoramic image with four overlapping regions and twenty control regions R(1)-R(20). Please also refer to FIG. 1B , each of the overlapping areas A( 1 ) to A( 4 ) includes P1 control areas, where P1 >=3. The following examples and embodiments are described by taking as an example that each overlapping area of the equidistant rectangular panoramic image includes five (P1=5) control areas. In the example of FIG. 3C, the equidistant rectangular panoramic image has twenty control regions R(1)-R(20). And the control regions R(1)-R(20) respectively have twenty warping coefficients C(1)-C(20), and the twenty warping coefficients respectively represent the twenty control regions R( 1) The different degrees of engagement of ~R(20).

The compensation device 22 includes a vertex processing device 210 , a splicing decision unit 230 and an image processing device 220 . In a measurement mode, the vertex processing device 210 receives the original vertex list and the test joint coefficients C(1)-C(20) from the splicing decision unit 320, and corrects each vertex in the original vertex list in a vertex-by-vertex manner. at all texture coordinates of all lens images, and generating a corrected vertex list (to be described later); based on the corrected vertex list and the four lens images from the image capture module 21, the image processing device 210 measures the The regional error amounts E(1)-E(20) of all control regions R(1)-R(20) in the isometric rectangular panoramic image, and output these regional error amounts, and then the splicing decision unit 230 according to the Some test joint coefficients C(1)-C(20), receive the regional error amounts to generate a 2D error table, and then generate optimal joint coefficients C(1)-C(20) according to the 2D error table. On the other hand, in a rendering mode, the splicing decision unit 320 is disabled. Therefore, according to the optimal splicing coefficients of the control regions output by the splicing decision unit 230, the vertex processing device 210 and the image processing device 220 operate together to generate the isometric panoramas.

In the measurement or visualization mode, the vertex processing device 210 receives the target vertex P (and its data structure) from the original vertex list, and twenty test/optimal joint coefficients C from the stitching decision unit 230 (1)~C(20); Next, the vertex processing device 210, for each lens image related to the target vertex P, according to the original texture coordinates of the target vertex P, calculates the rest of the twenty test/optimal joint coefficients. 2. Perform an interpolation operation to generate the interpolation joint coefficient C' of each lens image. Taking the front lens image as an example, according to the original vertex list, it is assumed that the target vertex P has the original texture coordinates (u _P , v _P ) in the front lens image. Fig. 4A shows an example of the relationship between the target vertex P and the ten control regions R(6)~R(15) in the frontal lens image, wherein the ten control regions R(6)~R(15) are respectively There are ten tests/best degree of engagement/factors C(6)~C(15). In one embodiment, according to the angle θ, the vertex processing device 210 obtains the two joint coefficients of the two control regions closest to the target vertex P (with the original texture coordinates (u _P , v _P ) in the frontal lens image) , and perform an interpolation operation to generate the interpolated joint coefficient C' of the target vertex P in the front lens image. Wherein, the angle θ is clockwise and is formed between a first vector V1 and a second vector V2; the first vector V1 takes the image center 41 (with texture coordinates (u _center , v _center )) as the starting point, The end point is the position 42 of the start control region R(13), and the second vector V2 has the image center 41 as the start point and the target vertex P(u _P , v _P ) as the end point. Assuming θ=119.5°, since there are five control areas on the left and right sides of the frontal lens image, 90°/4=22.5°, idx=θ/22.5°=5 and θ mod 22.5°=θ-idx×22.5 °=7°. Therefore, the two control regions closest to the vertex P are R(9) and R(10), and the interpolated joint coefficient C' of the target vertex P in the frontal lens image is defined/calculated by the following equation: C' =C(9)×(7/22.5)+C(10)×(1-7/22.5). The location 42 of each control area includes, but is not limited to, the gravity center of the control area, the regional center of the control area, and the median point of the control area. Please note that the start control area R(13) is only an example and not a limitation of the present invention. In another embodiment, another control area, such as R(11), can be selected as the start of the front lens image The control area, as long as the angle is still clockwise, and from the first vector V1 (with the image center 41 (with texture coordinates (u _center , v _center )) as the starting point, pointing to the starting control area R(11) Position 42) is measured to the second vector V2.

Taking the above target vertex P as an example, in order to simplify the above calculation, the correspondence generator 23 predetermines which two control regions (ie, R(9) and R(10)) are the closest to the target vertex P, and assigns its index (9 and 10) are written into the "joint coefficient index" field of the front lens image of the data structure of the target vertex P in the original vertex list; in addition, the correspondence generator 23 also pre-calculates the joint coefficient (C( 9) and C(10)) of the blending weight (=7/22.5), and write the blending weight to the blending coefficient of the front lens image of the data structure of the target vertex P in the original vertex list weight (Alpha)" field. Please note that the test/optimal joint coefficients (C(1)~C(20)) output from the splicing decision unit 230 are arranged as a 1D joint coefficient array or a 1D data stream. After the vertex processing device 210 receives the original vertex list, according to the "joint coefficient index" field (ie, 9 and 10) of the front lens image in the data structure of the target vertex P, the output from the stitching decision unit 230 Two joint coefficients (C(9) and C(10)) are extracted from the 1D joint coefficient array, and then according to the data structure of the target vertex P in the front lens image (please refer to Table 1) "joint coefficients" Mixing weight (Alpha)" field (ie 7/22.5), calculate the interpolated joint coefficient C'.

After receiving the interpolated joint coefficient C', the vertex processing device 210 calculates the modified texture coordinates (u _P ', v _P ') of the target vertex P in the frontal lens image according to the following equation: u _P '=(u _P -u _center )* C'+u _center ; v _P '=(v _P -v _center )*C'+v _center . In this way, the original texture coordinates (u2, v2) of the target vertex P in the frontal lens image become the modified texture coordinates (u2', v2'). In this way, according to the twenty test/best joint coefficients C(1)-C(20), the vertex processing device 210 sequentially corrects all texture coordinates of the four lens images from each vertex of the original vertex list , to generate the corrected vertex list. Table 2 shows an example of the data structure of each vertex in the modified vertex list.

After the vertex processing device 210 corrects all texture coordinates of all vertices from the original vertex list according to the test/optimal joint coefficients C(1)-C(20), the center of the lens of the image capture module 21 The problem of mismatched image defects caused by offset (that is, a distance of a lens center 46 relative to its system center 43 by an offset ofs) can be greatly improved (that is, the actual imaging position 48 will be pushed to the ideal imaging position 40 ). ), as shown in Figure 4B. Note that since the sphere 12 is virtual, it is possible for the object 45 to be located outside or inside or on the surface of the sphere 12 .

After receiving the corrected vertex list and the four lens images, the image processing device 220 generates the regional error amounts E(1)-E( of the twenty control regions R(1)-R(20) in the measurement mode 20), or generate an equidistant rectangular panorama image in developing mode. FIG. 5 is a schematic diagram showing the image processing apparatus according to an embodiment of the present invention. Please refer to Figure 5, the The image processing device 220 includes a graphics engine 55 , a seam engine 56 , a destination buffer 510 and a temporary buffer 520 . The image engine 55 includes a rasterization engine 560 , a texture mapping circuit 570 and a measurement unit 580 . The texture mapping circuit 570 includes two texture mapping engines 571-572. The polygon mesh of FIG. 3B is a collection of multiple quadrilaterals or/and multiple triangles. Therefore, the rasterization engine 560 may perform a quadrangular rasterization operation on each pixel in one of the quadrilaterals (as shown in FIG. 3B ) formed by any four vertices from the modified vertex list, or on any three of the modified vertex lists. Each pixel in a triangle formed by the vertices (as shown in FIG. 3B ) is subjected to a triangle rasterization operation.

In the case of quadrilaterals, it is assumed that four vertices (A, B, C, D) from the modified vertex list (forming a quadrilateral of the polygonal mesh of FIG. 3B ) are located in the aforementioned twenty control regions R(1) Within a range of ~R(20) and overlapped by two lens images (front and right; N=2), the four vertices (A, B, C, D) in the vertex list respectively contain the following data structures: Vertex A: {(x _A ,y _A ),2,ID _Front ,(u _1A ,v _1A ),w _1A ,ID _Right ,(u _2A ,v _2A ),w _2A }; Vertex B: {(x _B ,y _B ),2,ID _Front ,(u _1B ,v _1B ),w _1B ,ID _Right ,(u _2B ,v _2B ),w _2B }; vertex C: {(x _C ,y _C ),2, ID _Front ,(u _1C ,v _1C ),w _1C ,ID _Right ,(u _2C ,v _2C ),w _2C ,}; vertex D: {(x _D ,y _D ),2,ID _Front ,(u _1D ,v _1D ),w _1D ,ID _Right ,(u _2D ,v _2D ),w _2D }. The rasterization engine 560 directly performs a quadrilateral rasterization operation on the quadrilateral ABCD. Specifically, the rasterization engine 560 uses the following steps to calculate the texture coordinates of each lens image for a point Q (with equidistant rectangular coordinates (x, y) and located within the quadrilateral ABCD of the polygon mesh) : 1. Using a bi-linear interpolation method, according to the equidistant rectangular coordinates (x _A , y _A , x _B , y _B , x _C , y _C , x _D , y _D , x ,y), calculate four spatial weights (a,b,c,d); 2. Calculate the texture coordinates of the sampling point Q _F (corresponding to the point Q) in the frontal lens image: (u1,v1)=( a*u _1A +b*u _1B +c*u _1C +d*u _1D ,a*v _1A +b*v _1B +c*v _1C +d*v _1D ); calculate the sampling point in the right lens image The texture coordinates of Q _R (corresponding to the point Q): (u2,v2)=(a*u _2A +b*u _2B +c*u _2C +d*u _2D ,a*v _2A +b*v _2B +c *v _2C +d*v _2D ). Finally, the rasterization engine 560 transmits the two texture coordinates (u1, v1) and (u2, v2) to the two texture mapping engines 571-572 in parallel. Among them, a+b+c+d=1. According to the two texture coordinates (u1, v1) and (u2, v2), the two texture mapping engines 571-572 use any suitable method (eg, nearest-neighbour interpolation, bilinear interpolation) Interpolation, or trilinear interpolation), texture maps the texture data of the front and right lens images to generate two sample values s1, s2. Each of the sample values may be a luma value, a chroma value, an edge value, a pixel color value (RGB) or a motion vector.

In the case of triangles, the rasterization engine 560 and the two texture mapping engines 571 to 572 perform similar operations on each pixel in a triangle (as shown in FIG. 3B ) formed by any three vertices from the modified vertex list (similar to quadrilateral) to generate two corresponding sample values s1, s2, except that the rasterization engine 560 uses a barycentric weighting method in step (1) instead of the above-mentioned bilinear interpolation method, According to the equidistant rectangular coordinates (x _A , y _A , x _B , y _B , x _C , y _C , x, y), three spatial weights (a, b, c) are calculated.

In the measurement mode, the measurement unit 580 measures/estimates the equidistant rectangular panoramic image according to the equidistant rectangular coordinates (x, y) and the two sample values (s1-s2) of the point Q. The area error amounts E(1)~E(20) of the twenty control areas R(1)~R(20). According to the equidistant rectangular coordinates (x, y) of the point Q, the measurement unit 580 determines whether the point Q falls within the twenty control areas (please Referring to one of FIG. 3C ), if it is determined that the point Q falls within any control area, the estimation/measurement of the area error of the control area is started. The measurement unit 580 may utilize any known algorithm, such as sum of absolute differences (SAD), sum of squared differences (SSD), median absolute deviation (median absolute deviation) , MAD), etc., to estimate/measure the area errors of these control areas. For example, if it is determined that the point Q falls within the control region R(11), that is, N=2, the measurement unit 580 uses the following equation: E=|s1-s2|; E(11)+=E, and accumulates the The absolute value of the difference in sampling values between each point in the control area R(11) in the right lens image and the corresponding point in the control area R(11) in the front lens image, to obtain a SAD value as the control area R( 11) of the area error amount E(11). In this way, the measuring unit 580 measures the regional error amounts E(1)-E(20) of the twenty control regions R(1)-R(20).

One of the features of the present invention is to minimize mismatched image defects within a predetermined number of loops (Q1 in FIG. 6A). The predetermined number of loops is related to an offset ofs, and the offset ofs is the distance that the lens center 46 of the image capturing module 21 deviates from the system center 43 thereof (refer to FIG. 4B ).

Returning to FIG. 2 , in the measurement mode, according to the offset ofs in FIG. 4B , the splicing decision unit 230 sets the twenty joint coefficients C( 1 ) to C( 20 ) to different value ranges to measure The amount of regional error, and each time (or each loop) the twenty engagement coefficients are set to the same value. For example, assuming ofs=3 cm, the twenty joint coefficients C(1)~C(20) are set to a value range of 0.96~1.04, if each increment is 0.01, there will be a total of nine measurements; Assuming ofs=1 cm, the twenty joint coefficients C(1)~C(20) are set to the value range of 0.99~1.00. If each increment is 0.001, there will be a total of ten measurements.

FIG. 6A is a flowchart showing a method for determining the optimal bonding coefficients of all control regions according to an embodiment of the present invention. 4B, 5 and 6A, assuming ofs=3 cm, the method of determining the optimal joint coefficients of all control regions in the present invention will be described (applicable to the splicing decision unit 230 in the measurement mode).

Step S602: Set the number of loops Q1 and the test engagement coefficient to new values respectively. In one embodiment, Q1 is set to 1 in the first loop, and Q1 is increased by 1 for each subsequent loop; if ofs=3 cm, these test joint coefficients C(1) are set in the first loop ~C(20) are all set to 0.96 (ie, C(1)=...=C(20)=0.96), and in the subsequent loops, these joint coefficients C(1)~C(20 ) is set to 0.97,....,1.04.

Step S604: Clear all the regional error amounts E(m) to 0, where m=1, 2, . . . , 20.

Step S606 : After the vertex processing device 210 completes the operation according to the test joint coefficient set in step S602 , the image engine 56 of the image processing device 220 measures/obtains twenty equidistant rectangular panoramic images. The area error amount E(1)~E(20) of the control area R(1)~R(20). For the convenience of description, this step S606 is represented by E(m)=f(C(m)), where m=1, 2, . . . , 20, and f() represents (by the image engine 56 ) According to the corresponding test joint coefficient C(m) and the four lens images output by the image capturing module 21, the regional error amounts E(m) are measured.

Step S608: Store all the regional error quantities E(1)-E(20) in a 2D error table. Table 3 shows that when ofs=3 cm (the numerical range of the test joint coefficient 0.96~1.04), an example of this 2D error table.

In the 2D error table of Table 3, there are a total of twenty regional error quantities and nine different test joint coefficients. Due to space limitations, Table 4 only shows eight regional error quantities and six different test junction coefficients, and the rest are temporarily skipped. Please note that the area error amount and the number of test joint coefficients in the above 2D error table are only an example, and not a limitation of the present invention. In practical implementation, the above 2D error table can use any number of regional error quantities and joint coefficients.

Step S610: Determine whether the number of loops Q1 reaches the upper limit 9. If yes, go to step S612, otherwise, go back to step S602.

Step S612: Perform a coefficient decision operation according to the above 2D error table.

Step S614 : output the optimal joint coefficient C(m), where m=1, 2, . . . , 20. In the developing mode, output the optimal joint coefficients C(m) to the vertex The processing device 210 enables the image processing device 220 to generate a corresponding isometric rectangular panoramic image.

FIG. 6B is a flowchart showing a method for the splicing decision unit 230 to perform the coefficient decision operation of step S612 according to an embodiment of the present invention. In the following, please refer to FIGS. 3C and 6A-6B, which describe all the steps of performing the coefficient decision operation.

Step S661: Set Q2 to 0 for initialization.

Step S662: Extract a selected decision group from the 2D error table. 3C and 7B, usually each control area is adjacent to two control areas, a selected control area and its adjacent two control areas form a selected decision group to determine the best control area for the selected control area engagement factor. For example, a selected control region R(9) and its adjacent two control regions R(8) and R(10) form a decision group. However, if a selected control region (eg, R(1)) is located at the top or bottom of the overlapping region, then the selected control region R(1) will only form a single adjacent control region R(2). Decision group to determine its best engagement coefficient C(1). The description of the subsequent steps assumes that a control region R(2) is selected, and that R(2) and its adjacent two control regions R(1) and R(3) form a selected decision group to determine its optimal bonding Coefficient C(2).

Step S664: Determine a local minimum value among the regional error amounts of each control area of the selected decision group. Table 4 shows an example of the area error amounts of R(1)~R(3) and the test joint coefficient.

As shown in Table 4, there is only one local minimum in the nine regional error quantities of R(1), and there are two local minima in each of the nine regional error quantities of R(2) and R(3). Among them, each local minimum value in Table 4 is marked with an asterisk (*).

Step S666: Select candidates according to the local minima. Table 5 shows that candidates are selected from the local minima in Table 4, where ID represents the index, WC represents the junction coefficient, and RE represents the regional error amount. The number of candidates is equal to the number of the local minima in Table 4.

Step S668: Create a link metric according to the candidates in Table 5. As shown in FIG. 6C , according to the candidates in Table 5, a link meter is established.

及

，其二者 間的最小值

。關於二個連結計量值

及

，其二者間的最小值

。之後，分別計算路徑0-0-0及路徑0-1- 1的連結計量值的總和如下：

及

。因為

>

，故可決定

(路徑0-1-1)是 該連結計量的所有路徑中，連結計量值的最小總和，如第6C圖中的實線路徑。 Step S670 : In all the paths of the connection measurement, determine the minimum sum of connection measurement values. About two linked metrics

and

, the minimum value between the two

. About two linked metrics

and

, the minimum value between the two

. After that, calculate the sum of the connection meter values of path 0-0-0 and path 0-1-1 respectively as follows:

and

. because

>

, so it can be determined

(Path 0-1-1) is the smallest sum of connection meter values among all paths of the connection meter, such as the solid line path in Figure 6C.

(路徑0-1-1)是所有路徑中連結計量值的最小總和，故決定1.02為控制區R(2)之最佳接合係數。然而，若結束計算時有二條或更多路徑的連結計量值總和相同，就選擇具最小區域誤差量之節點的接合係數，當作該選定控制區之最佳接合係數。在此，將迴圈次數Q2的值遞增1。 Step S672: Determine the optimal bonding coefficient of the selected control area. In the example given in step S670, because

(Path 0-1-1) is the minimum sum of connection metering values in all paths, so 1.02 is determined as the optimal bonding coefficient for the control region R(2). However, if there are two or more paths with the same sum of link metrics at the end of the calculation, the joint coefficient of the node with the smallest area error is selected as the best joint coefficient for the selected control area. Here, the value of the number of loops Q2 is incremented by one.

Step S674: Determine whether the number of laps Q2 reaches the upper limit of 20. If yes, end the flow, otherwise, go back to step S662 to process the next control area.

Relevant terms referred to throughout the specification and subsequent claims are defined below, unless otherwise specifically indicated in this specification. The term "face image" refers to an image derived from the projection of a corresponding lens image of the image capture module 21; the projection such as equirectangular projection, cylindrical projection, Miller projection , Mercator projection, Lambert cylinder equal area projection or Pannini projection, etc. The term "seam area" refers to a projection image that is located in the middle of two adjacent first and second main images, and is surrounded by a seam image A and a seam image B overlaps, wherein the first main image and the seam image A are part of a first working image, and the second main image and the seam image B are part of a second working image. Generally speaking, the working surface image is divided into three parts for subsequent splicing processing. For example, in FIG. 7A, the first part of the left working surface image is located in the seam area S(2) or located in the main left side. The right side of the image is called seam image A; the second part of the left working surface image is located in the seam area S(1) or on the left side of the left main image, called seam image B; the left working surface image The third part is located in the middle of the seam area S(1) and S(2), and is called the left main image. The same segmentation method also applies to the right work surface image, the front work surface image, and the back work surface image.

FIG. 7A shows an example of an equidistant rectangular panoramic image including four main images and four seam regions. FIG. 7B shows an enlarged view of the seam area S( 2 ) of FIG. 7A . The quadrilaterals 72 in FIG. 7B correspond to the quadrilaterals of the polygon mesh of FIG. 3B , and the intersections 71 of the quadrilaterals 72 are vertices. According to the FOV of the lens in the image capturing module 21, the resolution of the lens sensor and the angle of the lens, the size of the overlapping area will vary accordingly. According to the present invention, P1 control areas are located in a seam area, and the seam area is located in a corresponding overlap area, where P1>=3. Generally, the width of the overlapping region is greater than or equal to the width of the seam region, and the width of the seam region is greater than or equal to the width of the P1 control regions. For example, the seam area S(2) is located in the overlapping area A(2) (not shown), the width of A(2) is greater than or equal to the width of S(2), and the width of S(2) is greater than or equal to The width of R(6)~R(10). In a preferred embodiment, the ratio of the widths of the overlapping area (eg A(2)), the seam area (eg S(2)) and the control area (eg R(6)-R(10)) is 7:5 :1. In another embodiment, the ratio of the widths of the overlapping area (eg A(2)), the seam area (eg S(2)) and the control area (eg R(6) to R(10)) is 9:7: 1, 9:9:1, 9:5:1, 7:7:1 or 5:5:1.

In order to generate the equidistant rectangular panorama image shown in FIG. 7A , the image engine 56 generates four working surface images according to the equidistant rectangular projection of the four lens images from the image capturing module 21 . The nine images related to the above four working surface images, namely one primary image (including the rear main image, front main image, left main image and right main image) and eight seam images (including four seam images A and four a seam image B). In order to store the above-mentioned nine images in nine different buffers for subsequent stitching operations, in the development mode, the correspondence generator 23 generates (m1+8) original single-shot vertex lists, that is, m1 original primary vertices The list and the eight original seam vertex lists correspond to the above four work plane images and eight seam images respectively, where 1<=m1<=4. For example, if m1=1, use one original main vertex list to generate four main images at a time; if m1=2, use either of the two original main vertex lists to generate two main images at a time; if m1 = 4, then use any of the four original primary vertex lists to generate the four primary images one at a time. Correspondingly, in order to minimize the problem of mismatched image defects caused by the offset of the lens center of the image capture module 21, according to the (m1+8) original single-lens vertex list and the above twenty optimal joint coefficients C (1) to C(20), the vertex processing device 210 generates (m1+8) modified single-lens vertex lists, that is, m1 modified main vertex lists and eight modified seam vertex lists respectively corresponding to the above four Work plane image and eight seam images. Note that the format of each of the (m1+8) original single-shot vertex lists is similar to Table 1, but only contains one set of parameters for a single shot, namely ID _k , (u _k ,v _k ), (idx _k0 ,idx _k1 ) and Alpha _k , where k=0, 1, 2 or 3. Please note that the format of each of the (m1+8) corrected single-shot vertex lists is similar to Table 2, but only contains a set of parameters for a single shot, namely ID _k and (u' _k ,v' _k ), where k= 0, 1, 2 or 3.

In the development mode, according to the m1 corrected main vertex lists and the four lens images, the image engine 56 performs rasterization and texture mapping to generate the above-mentioned main images (including the back main image, the front main image, the left main image and the left main image). main image and right main image). Then, according to the above-mentioned eight modified seam vertex lists and the four lens images, the image engine 56 "sequentially" performs rasterization and texture mapping to generate the above-mentioned four seam images A and four seam images B, that is, one seam image is generated at a time. For example, the image engine 56 performs rasterization and texture mapping according to a corresponding list of modified seam vertices and a frontal lens image to generate a seam image A associated with the frontal face image one at a time. In another embodiment, the image engine 56 can generate the above-mentioned images in reverse order, that is, the above-mentioned eight seam images can be generated first, and then the above-mentioned primary image can be generated.

There is a special case that when the image capture module 21 includes a wide-angle camera, the leftmost working surface image and the rightmost working surface image of the equidistant rectangular projection image only include a main image and a seam image, and also That is, one of the seam image A and the seam image B of the leftmost working face image and the rightmost working face image will be merged into the main image. For example, assuming that the image capture module 21 includes a dual-lens wide-angle camera (eg, 180°×60°) to generate a left-lens image and a right-lens image, the image engine 56 generates a left-lens image and a right-lens image according to the two lens images, One or two modified primary vertex lists and two modified seam vertex lists, resulting in a primary image (including a left main image and a left main image) and a seam image A and a seam image B related to a single seam area. Finally, the image processing device 220 further forms a wide-angle image from the primary image, the seam image A, and the seam image B.

7C is a flowchart illustrating a method of determining an error-minimizing path (or a seam) for a seam region, according to an embodiment of the present invention. Hereinafter, referring to FIGS. 5 and 7A-7D, the method of determining the minimizing error path of a seam region (performed by the seam engine 55 of the image processing device 220 in the developing mode) will be described.

For the convenience of description, the coordinates (x, y) refer to the position of a pixel in an equidistant rectangular projection field.

Step S702 : In the same seam area, calculate the absolute value of the pixel value difference between each pixel position in the seam image A relative to the adjacent pixel position in the seam image B. In one embodiment, the following code is provided to calculate the proximity of each pixel position (x, y) in the seam image A relative to the seam image B in the same seam area (range from Y0 to Y1 and X0 to X1) The absolute value of the pixel value difference at the pixel position (x+1,y) |A(x,y)-B(x+1,y)|, where A(x,y) indicates that it is related to a first working plane image The pixel value at the position (x, y) in the seam image A, and B(x+1, y) represents the position (x+1, y) in the seam image B related to a second working plane image pixel value. Please note that the two pixels A(x, y) and B(x+1, y) must be located on the same row y, and the first working plane image and the second working plane image overlap in the seam area.

In another embodiment, the squared error of the pixel value of each pixel position in the seam image A relative to the adjacent pixel position in the seam image B is calculated, rather than the absolute value of the pixel value difference or error.

Step S704: Calculate the minimum cumulative error (or the absolute value of the minimum cumulative pixel value difference) of each pixel in a current row in the seam area, and find the X coordinate with the minimum cumulative error in the previous row. In one embodiment, the following code is provided to calculate the minimum cumulative error of each pixel in a current row y, and find the X coordinate with the smallest cumulative error in the previous row (y-1), where y ranges from Y0 to Y1.

Step S706: Find the X coordinate with the smallest accumulated error in the bottom row of the seam area. In one embodiment, the following code is provided to find the X coordinate with the smallest accumulated error in the bottom row of the seam region.

Step S708: Trace/backtrack from the bottom row of the seam area to the top row to form an error-minimizing path (or a seam). As shown in FIG. 7D, a seam 78 spans the seam region S(2) and has a pixel position, ie, the x-coordinate, in each row. Therein, the seam 78 is a collection of a series of separation points (or seam points) on all columns of the seam area. In one embodiment, the following code is provided to form a seam in a seam area:

At the end of this step, the array Seam[y] will contain a series of separation points on all columns (Y0~Y1) of the seam region, ie the minimized error path or the seam. In another embodiment, those skilled in the art can appropriately modify the contents of steps S704-S708 to form the seam of the seam area in the reverse order (trace from the uppermost row to the lowermost row of the seam area). , which also falls within the scope of the patent application of the present invention.

8A is a flowchart illustrating an image processing method according to an embodiment of the present invention. Hereinafter, please refer to FIGS. 2 , 5 and 8A-8D to describe the image processing method (which is applicable to the projection image processing system 200 of the present invention).

Step S802: Obtain a 2D error table with all the test bonding coefficients and the regional error amounts of all the control regions. Please refer to steps S602 to S610 in FIG. 6A . At the end of this step S802, a 2D error table (see Table 3) including all the test bonding coefficients and the regional error amounts E(1)-E(20) of all the control regions R(1)-R(20) can be obtained. Note that all tests engage The coefficient is related to the offset ofs from the center 46 of the lens of the image capturing module 21 from the center 43 of the system, as shown in FIG. 4B .

Step S804: According to the above-mentioned 2D error table, determine the optimal bonding coefficients of all control regions. Please refer to steps S612 - S61 in FIG. 6A and steps S661 - S674 in FIG. 6B . At the end of this step S804, the optimal bonding coefficients C(1)-C(20) of all the control regions R(1)-R(20) can be obtained, and sent to the vertex processing device 210; at the same time, the measurement is ended mode and start developing mode.

Step S806 : According to the plurality of lens images, the plurality of original single-lens vertex lists, and the optimal joint coefficient obtained in step S804 , each working surface image is generated. As described above, the vertex processing device 210 obtains the best single-camera vertex list (including at least one original main vertex list and a plurality of original seam vertex lists) outputted by the correspondence generator 23 and the best one obtained in step S804. The joint coefficient is used to sequentially generate a plurality of modified single-lens vertex lists (including at least one modified main vertex list and a plurality of modified seam vertex lists). In the example of the equidistant rectangular panoramic image with four seam areas in FIG. 7, in the measurement mode, the image engine 56 in FIG. m1+8) corrected single-camera vertex lists and the twenty optimal joint coefficients C(1)-C(20) obtained in step S804, to generate four working surface images (including four main images, four seams Image A and four seam images B), where 1<=m1<=4. After generating the above four working plane images, the image engine 56 uses one of the following two methods to store nine images (including a primary image (including the four main images in total), the four seam images A and the Four seam images B). Method 1: First, the image engine 56 stores the nine images in nine different temporary buffers 520, and the nine images do not overlap each other; then, the primary image is sent to a destination buffer 510 , in which the four seam areas are empty; finally, the seam engine 55 according to the above four After the four seam images A and the four seam images B are generated, the four seam images after splicing are stored in the four seam areas of the destination buffer 510 for display. . Method 2: First, the image engine 56 directly stores the primary image in the destination buffer 510 and stores the four seam images A and the four seam images B in eight different temporary buffers respectively 520; Next, after the seam engine 55 generates four seam images after splicing according to the four seam images A and the four seam images B, and then stores the four seam images after splicing in the destination The four seam areas of bumper 510. For example, seam image A, which is part of the left work surface image, and seam image B, which is part of the front work surface image, are fed to the seam engine 55 for stitching, and a stitch is generated After the seam image is created, the seam image after splicing is stored in the seam area S( 2 ) of the destination buffer 510 , as shown in FIG. 7A .

Assuming that the above-mentioned image capture module 21 includes a wide-angle camera with three lenses and generates three lens images (ie, a left lens image, a front lens image, and a right lens image), the image engine 56 in FIG. The above-mentioned three lens images, at least five modified single-lens vertex lists and a plurality of optimal joint coefficients, generate three work plane images (including three main images and two seam images A and Two seam images B). After generating the above-mentioned three working plane images, the image engine 56 uses the above-mentioned method 1 or method 2 to store five images (including a primary image (including the three main images in total), the above-mentioned two seam images A and two A seam image B) for subsequent image stitching.

Step S808: Determine the minimum error path (or seam) of each seam area. In one embodiment, the method of determining an error minimization path for a seam region of FIG. 7C is performed on the four seam regions S( 1 ) to S( 4 ) of FIG. 7A . At the end of this step S808, four error-minimizing paths or seams of the four seam regions S(1)-S(4) will be generated.

Step S810: splicing the overlapping seam image A and seam image B according to the minimized error path or seam of each seam area. In the example of FIG. 7D, seam 78 is used to stitch the seam image A associated with the left working face image and the seam image B associated with the front working face image to produce a stitched seam image. Finally, a total of four spliced seam images are stored in the four seam areas S( 1 ) to S( 4 ) of the destination storage 510 respectively. Therefore, the four spliced seam images and the above four main images A complete isometric rectangular panoramic image is formed.

FIG. 8B shows an example of a section of a seam area, and the seam area is associated with a seam 78 , a seam image A, and a seam image B. FIG. FIG. 8C shows an example of the size of a seam area and the sizes and different positions of the update windows 81 in three columns. In step S810, the present invention provides the following three methods (1) to (3) for splicing the seam image A and the seam image B to generate a corresponding seam image after splicing. (1) Traditional method: According to the minimized error path or seam of each seam area, the corresponding seam image A and the seam image B are spliced together. As shown in FIG. 8B, for each row of the seam image or seam area after splicing, if a pixel position (x, y) is located on the left side of the seam 78, the pixel value in the seam image A is designated as The pixel value of the same pixel position (x, y) in the seam image after stitching; on the contrary, if a pixel position (x, y) is located on the right side of the seam 78, then specify its pixel in the seam image B The value will be taken as the pixel value at the same pixel position (x,y) in the stitched seam image.

(2) Seam blending method (each seam area is a continuous integral area, not cut): according to the seam 78 of a seam area, apply an update window 81 to each row of the seam area to splicing Corresponding seam image A and seam image B. In order to smooth the transition of pixel values near the seam 78, the present invention provides a seam blending method to update pixel values in an update window 81, wherein the width of the update window 81 is equal to 2*Ra and a Separation point 82 is located at the The center of the window 81 is updated, as shown in FIG. 8C. For each column of the seam area S(2), if a pixel position (x, y) is located on the left side of the update window 81, the pixel value of the seam image A is designated as the same in the seam image after splicing The pixel value of the pixel position (x, y); on the contrary, if a pixel position (x, y) is located on the right side of the update window 81, the pixel value in the seam image B is designated as the seam image after stitching The pixel value of the same pixel position (x, y); furthermore, if a pixel position (x, y) is located within the update window 81, according to the last width of the update window 81 and the pixel position (x, y) With respect to the distance from the left edge of the update window 81, a blending weight is calculated, and then according to the blending weight and two corresponding pixel values of the seam image A and the seam image B (ie ImgA(x, y) and ImgB (x,y)), calculate the mixed pixel value of the same pixel position (x,y). In one embodiment, the following code is provided to update the update window 81 according to the last width of the update window 81 in each row of the seam area and the position of each pixel position (x, y) relative to the update window 81 to update the update window 81 Pixel values.

(3) Wrap around seam blending method (one of the above-mentioned seam areas is divided into two discontinuous parts and located on opposite sides of the panoramic image): In a special case, a seam The area may be located near the left and right edges of the equirectangular panorama at the same time. As shown in FIG. 8D , the seam area S( 3 ) is located at the leftmost and rightmost sides of the equidistant rectangular panoramic image, in other words, the seam area S(3) is divided into two parts. To address this situation, the present invention provides a wraparound seam blend. In one embodiment, the following code is provided to form a stitched seam image of a seam region (ranging from Y0 to Y1 and X0 to X1) according to the wraparound seam blending method:

Therefore, even if the seam area S(3) in FIG. 8D is located at the leftmost and rightmost sides of the equidistant rectangular panoramic image, the above-mentioned wraparound seam blending method can still generate a correct stitched seam image .

The compensation device 22 and the corresponding generator 23 of the present invention can be implemented by software, hardware, or a combination of software (or firmware) and hardware. An example of a simple solution is a field programmable gate array. or an application specific integrated circuit. In a preferred embodiment, the vertex processing device 210 and the image processing device 220 are implemented by a graphics processing unit (Graphics Processing Unit) and a first program memory; the splicing decision unit 230 and the correspondence generator 23 is implemented using a first general-purpose processor and a second program memory. The first program memory stores a first processor executable program, and the second program memory stores a second processor executable program. When the graphics processing unit executes the first processor executable program, the graphics processing unit is configured to operate as the vertex processing device 210 and the image processing device 220 . When the first general purpose processor executes the second processor executable program, the first general purpose processor is configured to operate as: the splicing decision unit 230 and the correspondence generator 23 .

In another embodiment, the compensation device 22 and the correspondence generator 23 are implemented using a second general purpose processor and a third program memory. The third program memory stores a third processor executable program. When the second general purpose processor executes the third processor executable, the second general purpose processor is configured to operate as: the vertex processing device 210, the tiling decision unit 230, the correspondence generator 23 and the image processing device 220.

The above are only preferred embodiments of the present invention, and are not intended to limit the scope of the patent application of the present invention; all other equivalent changes or modifications without departing from the spirit disclosed in the present invention shall be included in the following application. within the scope of the patent.

Claims (22)

An image processing method for receiving M lens images to generate a projection image, the method comprising: determining P optimal bonding coefficients of P control regions in the projection image according to a 2D error table and the M lens images, Wherein, an image capture module is used to capture a field of view with a horizontal field of view of X degrees and a vertical field of view of Y degrees to generate the M lens images; according to a first vertex list, the P optimal joint coefficients and The M lens images generate M working surface images; N seams of the N seam areas are determined; and according to the seams of each seam area, two overlapping seam images of each seam area are spliced to generate A seam image after splicing; wherein, each working surface image includes z seam images and a main image, and each seam area is located between two adjacent main images in the projected image; wherein, the 2D image The error table includes a plurality of test joint coefficients and a plurality of accumulated pixel value error amounts corresponding to the P control regions; wherein, the P control regions are respectively located in the N joint regions, and the N joint regions are respectively located. The seam regions are respectively located in the N overlapping regions of the M working surfaces; wherein the first vertex list includes a plurality of first vertices with data structures, and the data structures define the relationship between a single lens image and the projected image. and where 1<=z<=2, M>=2, N>=1, P>=3, X<=360, and Y<180. As in the method of claim 1, it further includes: The projection image is formed according to the N seam images after splicing and the M main images of the N seam regions. The method of claim 1, wherein the step of determining the P optimal bonding coefficients of the P control regions comprises: according to an offset of a lens center of the image capture module relative to a system center of the image capture module determine the test joint coefficients; according to the test joint coefficients, modify the texture coordinates of each lens image in the second data structure of each second vertex in a second vertex list to generate a third vertex list; The third vertex list and the M lens images are used to measure the accumulated pixel value errors in the P control regions; the correction step and the measurement step are repeated until all the test joint coefficients are processed to form the 2D error table; and according to at least a local minimum value of the accumulated pixel value error amount of one or two adjacent control regions of each control region in the 2D error table, determine the optimal bonding coefficient of each control region; wherein, the The second vertex list includes a plurality of second vertices with data structures defining second vertex mappings between the M lens images and the projected image; and wherein the third vertex list includes a plurality of second vertices with data structures the third vertex of the structure, the data structures define the third vertex mapping between the M lens images and the projection image. The method of claim 1, wherein each of the M working images is derived from a preset projection of a corresponding lens image of the image capture module. The method of claim 6, wherein the preset projection is one of an equirectangular projection, a Miller projection, a Mercator projection, a Lambert cylinder equal-area projection, and a Panini projection. The method of claim 1, wherein if z=2, the main image of each working plane image is positioned between the z seam images. The method of claim 1, wherein the first vertex list includes m1 first main vertex lists and 2×N first seam vertex lists, and wherein the step of generating the M working surface images comprises: according to the P an optimal joint coefficient, sequentially correcting the texture coordinates of all the first vertices in the first vertex list in a single lens image to generate a fourth vertex list, wherein the fourth vertex list includes m1 second main vertex lists and 2×N second seam vertex lists, where 1<=m1<=M; and according to the M lens images, for each point in each polygon formed by a set of fourth vertices from the fourth vertex list , perform rasterization and texture mapping operations to sequentially generate M main images, N first seam images, and N second seam images; wherein each of the m1 first main vertex lists and each of the 2 The ×N first seam vertex lists include the first vertices having a first data structure defining a first vertex mapping between the single shot image and the projected image; and wherein each of the The m1 second main vertex lists and each of the 2×N second seam vertex lists include a plurality of fourth vertices having fourth data structures defining a space between a corresponding lens image and the projected image The fourth vertex map of . The method of claim 7, wherein the step of correcting the texture coordinates comprises: Modify the texture of the single-shot image in the first data structure of each first vertex according to the test joint coefficients and the two joint coefficient indices and a blending weight of the single-lens image in the first data structure of each first vertex coordinates to generate the fourth vertex list; wherein the two joint coefficient indices are used to select two of the test joint coefficients, and the blend weight is used to blend the two selected test joint coefficients. The method of claim 8, wherein the step of modifying the texture coordinates further comprises: according to the test stitching coefficients and the two stitching coefficient indices and the blending weight of the single-lens image in the first data structure of a target first vertex value, calculate an interpolation joint coefficient of the target first vertex in the single lens image; according to the interpolation joint coefficient, the interpolation joint coefficient of the target first vertex in the single lens image and the texture coordinate, calculate the The modified texture coordinates of the target first vertex in the single lens image; and the modified texture coordinates of the target first vertex in the single lens image are stored in a fourth data structure corresponding to the fourth vertex in the fourth vertex list. The method of claim 7, wherein the step of generating the M working surface images further comprises: storing the M main images in a first buffer for display; and storing the N first seam images and the N first seam images Two seam images in 2×N different second buffers, wherein the 2×N second buffers are separated from the first buffer. The method of claim 7, wherein the step of generating the M work surface images further comprises: storing the M main images, the N first seam images and the N second seam images in (m2+2×N) different first buffers, where m2<=M; and the M images The main image is moved to a second buffer for display. The method of claim 1, wherein the step of determining the seam of each of the N seam regions comprises: in a target seam region, calculating the position of each first pixel in a third seam image relative to a fourth seam region A pixel value error amount corresponding to a second pixel position in the slit image is stored in a fifth data structure, wherein each first pixel position and its corresponding second pixel position are located in the same row and adjacent; according to the first pixel position The data of the five data structures and a preset search range are calculated, and the minimum accumulated pixel value error amount of each first pixel position in the current row y of the target seam area is calculated to be stored in a sixth data structure, and the previous In a row (y-1), the x-coordinate with the minimum accumulated pixel value error is stored in a seventh data structure; according to the sixth data structure, find out the minimum accumulated pixel in the bottom row of the target seam area the x-coordinate of the difference in value; and according to the seventh data structure, backtracking from the x-coordinate of the lowermost row of the target seam area to the uppermost row to form the seam of the target seam area, and storing that each row is penetrated The x-coordinate of an eighth data structure. The method of claim 1, wherein the two overlapping seam images comprise a fifth seam image and a sixth seam image, wherein the fifth seam image is related to a first one of the M working images The working image and the sixth seam image are related to a second working image of the M working images, wherein the main image of the first working image and the main image of the second working image are respectively located at The left and right sides of a corresponding seam area, and the seam of the corresponding seam area includes a plurality of separation points, which are respectively located in each row of the corresponding seam area. The method of claim 13, wherein the splicing step comprises: for each row of the corresponding seam area, according to the relative position of a pixel position in the spliced seam image and the corresponding separation point on the same row, the first The fifth seam image and the sixth seam image assign a pixel value to the pixel position in the stitched seam image. The method of claim 14, wherein the specifying step comprises: if the pixel position is located on the left side of the seam, specifying the pixel value of the pixel position in the fifth seam image to be the same in the seam image after splicing the pixel position of is located on the seam, and assigns the pixel position to the corresponding pixel value of one of the fifth seam image and the sixth seam image to the same pixel position in the stitched seam image. The method of claim 13, wherein the splicing step comprises: according to the seam of the corresponding seam area and whether one of the N seam areas is divided into two separate parts, by respectively providing an update window to the corresponding seam area Each row of the seam area is used to stitch the two overlapping seam images; wherein, the separation point on each row of the corresponding seam area is located at the center of the update window. The method of claim 16, wherein the splicing step comprises: for each column of the corresponding seam area, If a pixel position is located between the left edge of the seam area and the left edge of the update window, assign the pixel value of the pixel position in the fifth seam image to the same pixel in the stitched seam image position; if the pixel position is between the right edge of the seam and the right edge of the update window, assign the pixel value of the pixel position in the sixth seam image to the same pixel value in the seam image after stitching pixel position; and if the pixel position is located within the update window, calculating a blending weight based on the final width of the update window and the distance of the pixel position relative to the left edge of the update window, and according to the blending weight and the corresponding pixel value of the pixel position in the fifth seam image and the sixth seam image, and calculate the mixed pixel value of the same pixel position in the seam image after splicing. The method of claim 17, wherein each seam area is a complete area and the splicing step further comprises: before the specifying step and the calculating the blending weight and the blending pixel value, for each row of the corresponding seam area , according to the original width of the update window, determine the coordinates of the left edge and the right edge of the update window; and if one of the left and right edges of the update window extends beyond the corresponding seam area, correct the update window The edge of the window is aligned with the border of the corresponding seam area to obtain the final width of the update window. The method of claim 17, wherein one of the N seam regions is divided into two separate parts and the two separate parts are located at the leftmost and the rightmost of the projected image, wherein the splicing step comprises: Before the specifying step and before calculating the blending weight and the blending pixel value, for each column of the corresponding seam region, subtract the x-coordinate of the left edge of the corresponding seam region from the x-coordinate of the corresponding separation point, to obtain the relative distance of the corresponding separation point; if the relative distance is less than 0, add the relative distance to the width of the projected image to correct the relative distance; according to the relative distance and the original width of the update window, determine the width of the update window The position of the left and right edges; if one of the left and right edges of the update window protrudes beyond the corresponding seam area, modify the edge of the update window to align with the boundary of the corresponding seam area to Obtain the final width of the update window; calculate the distance of a pixel position in the corresponding seam area relative to the left edge of the projected image; and if the pixel position in the corresponding seam area is relative to the distance of the left edge of the projected image If the width of the projected image is greater than the width of the projected image, the distance of the pixel location relative to the left edge of the projected image is subtracted from the width of the projected image to correct the pixel location. The method of claim 1, wherein each seam area includes P1 control areas arranged in a line, and P1>=3. The method of claim 20, wherein for a seam area located in an overlap area and the P1 control areas located in the seam area, the width of the overlap area is greater than or equal to the width of the seam area, and The width of the seam area is greater than or equal to the width of the P1 control areas. The method of claim 1, wherein the projected image is one of a panoramic image and a wide-angle image.

Images