TWI837563B - Image processing system and method thereof for generating projection images based on a multiple-lens camera - Google Patents

Abstract

An image processing system is disclosed, comprising: an M-lens camera, a compensation device and a correspondence generator. The M-lens camera generates M lens images. The compensation device generates a projection image according to a first vertex list and the M lens images. The correspondence generator is configured to conduct calibration for vertices to define vertex mappings, horizontally and vertically scan each lens image to determine texture coordinates of its image center, determine texture coordinates of control points according to the vertex mappings, and P1 control points in each overlap region in the projection image; and, determine two adjacent control points and a coefficient blending weight for each vertex in each lens image according to the texture coordinates of the control points and the image center in each lens image to generate the first vertex list.

Description

Image processing system and method for generating projection images using a multi-lens camera

The present invention relates to wide-angle images, and more particularly, to an image processing system and method thereof, which utilizes an inward-facing or outward-facing multi-lens camera to generate a projected image.

FIG. 1A shows the relationship between a cubic structure 11A and a sphere 12 disclosed in the patent document No. 1 728620 of the Republic of China (the contents of the above patent are hereby quoted as a part of the contents of this specification). FIG. 1B shows an equirectangular panoramic image, which is derived from the equirectangular projection of six lens images (top, bottom, back, left, right, front) of a six-lens camera set up on the six working surfaces of the cubic structure 11A. Since there are six lens images in FIG. 1B, it can be inferred that the six lenses (top, bottom, back, left, right, front) of the camera set up on the six working surfaces of the cubic structure 11A are facing outwards relative to the center of gravity of the cubic structure 11A, as shown in FIG. 2A. The internal angle formed by any two adjacent surfaces/edges of the cubic structure 11A (used to set up the above six lenses) 1 is equal to 90 ^° . Hereinafter, the six-lens camera in FIG. 2A is referred to as an “external-facing six-lens camera”. FIG. 2B illustrates an external-facing three-lens camera mounted on a frame 11B. The external-facing three-lens camera in FIG. 2B is used to generate three lens images, and the three lens images are used to form a wide-angle image. The inner angle formed by any two adjacent surfaces/edges of the frame 11B (used to mount the three lenses) 2 is less than 180 ^° . However, the disadvantage of the aforementioned outward-facing multi-lens camera is that the lens will protrude outside the body 22 of the electronic product, so the electronic product is not convenient to carry and the lens itself is easily worn.

Therefore, the industry is in urgent need of an image processing system with an inward-looking multi-lens camera to produce wide-angle images and prevent the lens from wear and tear.

In view of the above problems, one of the objects of the present invention is to provide an image processing system having an inward-looking multi-lens camera for generating a projected image and preventing the lens from being worn.

According to an embodiment of the present invention, an image processing system is provided, comprising: a camera with M lenses, a compensation device, and a correspondence generator. The camera with M lenses is used to capture a field of view covering an X-degree horizontal field of view and a Y-degree vertical field of view to generate M lens images. The compensation device is used to generate a projection image according to a first vertex list and the M lens images. A correspondence generator is used to generate a set of operations, including: correcting multiple vertices to define multiple first vertex mappings between the M lens images and the projection image; scanning each lens image horizontally and vertically to determine the image center of each lens image; determining the texture coordinates of all control points based on the first vertex mappings and P1 control points in each overlapping area in the projection image; and, based on the texture coordinates of all control points and the image center of each lens image, determining two neighboring control points of each vertex in each lens image and a coefficient mixing weight to generate the first vertex list, where X＜=360, Y＜180, M＞=2 and P1＞=3.

Another embodiment of the present invention provides an image processing method, comprising: calibrating a plurality of vertices to define a plurality of first vertex mappings between M lens images and a projection image, wherein the M lens images are generated by a camera having M lenses capturing a field of view covering an X-degree horizontal field of view and a Y-degree vertical field of view; scanning each lens image horizontally and vertically to determine an image center of each lens image; and determining a plurality of first vertex mappings between the plurality of lens images and a projection image. Determine the texture coordinates of all control points based on P1 control points in each overlapping area of the projection image; determine two neighboring control points of each vertex in each lens image and a coefficient mixing weight based on the texture coordinates of all control points and the image center of each lens image to generate a first vertex list; and generate the projection image based on the first vertex list and the M lens images, wherein X＜=360, Y＜180, M＞=2 and P1＞=3.

The above and other objects and advantages of the present invention are described in detail below with reference to the following drawings, detailed description of embodiments and patent claims.

The singular forms of "a", "an", "the" and the like mentioned in the entire specification and the subsequent claims include both the singular and the plural, unless otherwise specified in the specification. The relevant terms mentioned in the entire specification and the subsequent claims are defined as follows, unless otherwise specified in the specification. Throughout the specification, circuit elements with the same function use the same reference symbols.

One of the features of the present invention is to use an inward-looking multi-lens camera to generate multiple lens images, and then use stitching and blending methods to generate a wide-angle image.

FIG. 3A is a diagram showing two side views of an inward-facing three-lens camera mounted on a frame 11C according to an embodiment of the present invention. FIG. 3B is a diagram showing two side views of an inward-facing two-lens camera mounted on a frame 11D according to an embodiment of the present invention. The reflex angle formed by any two adjacent surfaces/edges of the frame 11C is 3 and the reflex angle formed by any two adjacent surfaces/edges of the structure 11D 4 are all greater than 180° ^and less than 270 ^° . Each outward-facing multi-lens camera has an internal center, and each inward-facing multi-lens camera has an external center. For example, the intersection O1 of the three optical axes D1~D3 of the three lenses (left lens, front lens, right lens) of the outward-facing three-lens camera in FIG2B is formed inside the frame 11B or below the three lenses, so it is called the "inner center"; in contrast, the two optical axes D1~D3 of the dual lenses (lens A, lens B) of the inward-facing dual-lens camera in FIG3B The intersection O2 of D4~D5 is formed outside the structure 11D or located above the double lens. Furthermore, the intersection O3 of the three optical axes D6, D2 and D7 of the three lenses (lens A, lens C, lens B) of the inward-looking three-lens camera in Figure 4A is formed outside the structure 11C or located above the three lenses, so the intersection points O2 and O3 are called "external centers".

The advantage of the aforementioned inward-facing multi-lens camera is that the lens will be enclosed within the body 32/33 of the electronic product, so the electronic product can be convenient to carry and the lens itself can be protected from wear and tear.

FIG. 4A illustrates the setup of an inward-facing three-lens camera and an outward-facing three-lens camera. FIG. 4B illustrates a wide-angle image including three lens images (output by the outward-facing three-lens camera of FIG. 4A). FIG. 4C illustrates a wide-angle image including three lens images (output by the inward-facing three-lens camera of FIG. 4A). FIG. 4D illustrates a wide-angle image including two lens images (output by the inward-facing dual-lens camera of FIG. 3B). The setup of the two cameras in FIG. 4A is for comparison only. The order in which the three lenses of the outward-facing three-lens camera in FIG. 4A (from left to right: left lens, front lens, right lens) are mounted on the frame 11B is the same as the order in which the three lens images are arranged in the wide-angle image of FIG. 4B (from left to right: left lens image, front lens image, right lens image). However, the order in which the three lenses of the inward-facing three-lens camera in FIG. 4A (from left to right: lens A, lens C, lens B) are mounted on the frame 11C is opposite to the order in which the three lens images are arranged in the wide-angle image of FIG. 4C (from left to right: lens B image, lens C image, lens A image); in addition, the order in which the two lenses of the inward-facing two-lens camera in FIG. 3B (from left to right: lens A, lens B) are mounted on the frame 11D is also opposite to the order in which the two lens images are arranged in the wide-angle image of FIG. 4D (from left to right: lens B image, lens A image). In the example of FIG. 4A, by appropriately selecting the inner angle in the frame 11B, the lens images are arranged in the wide-angle image of FIG. 2 and the internal angles in structure 11C 3, so that the angle of the optical axis D7 of the lens B of the inward-looking three-lens camera relative to the horizontal line (not shown) is the same as the angle of the optical axis D1 of the left-pointing lens of the outward-looking three-lens camera relative to the horizontal line, and the angle of the optical axis D6 of the lens A of the inward-looking three-lens camera relative to the horizontal line is the same as the angle of the optical axis D3 of the right-pointing lens of the outward-looking three-lens camera relative to the horizontal line. However, even if the angles of the optical axes are the same, there is still a displacement between the left-pointing lens image and the lens B image, and between the right-pointing lens image and the lens A image. Therefore, it is necessary to properly splice and mix the multiple lens images output by the inward-looking multi-lens camera to form a high-quality wide-angle image.

According to the present invention, each lens of the above-mentioned inward-looking or outward-looking multi-lens camera simultaneously captures a field of view covering a horizontal field of view (HFOV) of x1 degree and a VFOV of y1 degree to generate a lens image. Thereafter, multiple lens images from the above-mentioned inward-looking or outward-looking multi-lens camera form a projected image with a HFOV of x2 degree and a VFOV of y2 degree, wherein 0＜x1＜x2＜=360, 0＜y1＜y2＜180. For example, each lens of the inward-looking three-lens camera in FIG. 4A can simultaneously capture a field of view covering a 70-degree HFOV and a 70-degree VFOV to generate a lens image, and then the lens B image, the lens C image, and the lens A image form a wide-angle image with a 160-degree HFOV and a 60-degree VFOV, as shown in FIG. 4C ; each lens of the inward-looking dual-lens camera in FIG. 3B can simultaneously capture a field of view covering a 100-degree HFOV and a 70-degree VFOV to generate a lens image, and then the lens B image and the lens A image form a wide-angle image with a 160-degree HFOV and a 60-degree VFOV, as shown in FIG. 4D . A necessary condition for setting up the multi-lens camera is that there should be enough overlap between the fields of view of any two adjacent lenses to facilitate image stitching. Referring to Figures 4C-4D, the pixels in the overlap area 41 are formed by the overlap of two lens/texture images, while the pixels in the non-overlap area 43 are from a single lens/texture image, and the pixels in area 42 are discarded. Therefore, the image processing device 520 needs to perform blending and stitching operations on the overlap areas 41 to form a wide-angle image (to be explained later).

FIG5A is a block diagram showing a projection image processing system according to an embodiment of the present invention. Referring to FIG5A , the projection image processing system 500 includes an image capture module 51, a compensation device 52, and a correspondence generator 53. The compensation device 52 receives the original vertex list from the correspondence generator 23 and multiple lens images from the image capture module 51 to generate a projection image, such as a wide-angle image or a panoramic image.

Many projection methods are applicable to the projection image processing system 500 of the present invention. The term "projection" refers to: flattening the surface of a sphere into a two-dimensional plane, such as a projection plane. The projection includes, but is not limited to, equirectangular projection, cylindrical projection, and modified cylindrical projection. Modified cylindrical projection includes, but is not limited to, Miller projection, Mercator projection, Lambert cylindrical equal area projection, Pannini projection, etc. Accordingly, the above-mentioned projection image includes, but is not limited to, an equirectangular projection image, a cylindrical projection image, and a modified cylindrical projection image. Figures 1A-1B, 6A-6C, 8A-8D, and 10B-10C are related to equirectangular projection. As for the implementation methods of cylindrical projection and modified cylindrical projection, they are already well known to technical personnel in this field and will not be elaborated here. Please note that no matter which projection method is adopted by the projection image processing system 500 of the present invention, the correspondence generator 53 will correspondingly generate an original vertex list (such as Table 1), which defines the vertex mapping relationship between the lens images and the projection image.

The image capture module 51 is a camera with multiple lenses, such as an extroverted multi-lens camera, an introverted dual-lens camera or an introverted three-lens camera, which can simultaneously capture a field of view covering an X-degree HFOV and a Y-degree VFOV to generate multiple lens images, where X<=360, Y<180. For clarity and convenience of description, the following examples and embodiments are only illustrated with equirectangular projection and assume that the image capture module 51 is an introverted three-lens camera and the projected image is an equirectangular wide-angle image. It should be noted that the operation of the projection image processing system 500 of the present invention and the related descriptions and methods of FIGS. 7B-7C , 8A-8D and 9A-9C are also applicable to the above-mentioned exotropic multi-lens camera, introtropic dual-lens camera, cylindrical projection and modified cylindrical projection.

The definitions of terms used throughout this specification and in the claims that follow are as follows, unless otherwise specified in this specification. The term "texture coordinates" refers to coordinates in a texture space (such as a texture image or a lens image); the term "rasterization" refers to the computational process of mapping scene geometry (or a projection image) to texture coordinates of each lens image.

The processing pipeline of the projection image processing system 500 is divided into an offline phase and an online phase. In the offline phase, the three inward-facing lenses of the image capture module 51 are calibrated respectively, and the correspondence generator 53 uses a suitable image registration technology to generate an original vertex list, and each vertex in the original vertex list provides a mapping relationship between the equidistant rectangular projection image and the lens images (or between the equidistant rectangular coordinates and the texture coordinates). For example, a sphere 62 with a radius of 2 meters (r=2) in FIG. 6A is marked with many circles on its surface, which are used as longitude and latitude, and its multiple intersections are regarded as multiple calibration points. The three lenses of the image capture module 51 mounted on the frame 11D capture the calibration points, and the positions of the calibration points on the lens images are known. Then, because the view angles of the calibration points and the texture coordinates are linked, a mapping relationship between the equirectangular panoramic image and the lens images can be established. In this specification, a calibration point with the above mapping relationship is defined as a "vertex". In short, the correspondence generator 53 calibrates each vertex to define the vertex mapping relationship between the equirectangular projection image and the lens images, and then obtains the original vertex list. In the offline stage, the correspondence generator 53 completes all necessary calculations.

FIG6B shows a triangular mesh used to model a spherical surface. Referring to FIG6B , a triangular mesh is used to model the surface of a sphere 62. FIG6C shows a polygonal mesh used to form/model the equirectangular projection image. The polygonal mesh of FIG6C is generated by performing an equirectangular projection on the triangular mesh of FIG6B , and the polygonal mesh of FIG6C is a collection of multiple quadrilaterals or/and multiple triangles defined by the above vertices.

In the offline stage, according to the geometric shapes of the equirectangular projection image and the lens images, the correspondence generator 53 calculates the equirectangular coordinates and texture coordinates for each vertex of the polygonal grid (FIG. 6C) to generate the original vertex list. Afterwards, the correspondence generator 53 transmits the original vertex list to the vertex processing device 510. The original vertex list is a list of multiple vertices, which form multiple quadrilaterals and/or triangles of the polygonal grid (FIG. 6C), and each vertex is defined by a corresponding data structure. The data structure defines the vertex mapping relationship between a destination space and a texture space (or between the equirectangular coordinates and the texture coordinates). Table 1 shows an example of the data structure of each vertex in the original vertex list. Table I Attributes instruction (x, y) Equirectangular coordinates N Number of overlapping camera images ID ₁ The ID of the first camera image (u ₁ , v ₁ ) Texture coordinates in the first shot image (idx ₁₀ , idx ₁₁ ) Index of stitching factor for the first lens image Alpha ₁ Blending weights for the stitching coefficients in the first shot image w ₁ The stitching blending weights of the first lens image ….. ……. ID _N The ID of the Nth lens image (u _N , v _N ) Texture coordinates in the Nth lens image (idx _N0 , idx _N1 ) Index of stitching coefficient in the Nth lens image Alpha _N The blending weight of the stitching coefficient in the Nth lens image _1v The stitching blending weight of the Nth lens image

In an ideal situation, the three lenses of the image capture module 51 are simultaneously located at the camera system center 73 of the frame 11C (not shown), so a single ideal imaging point 70 of a distant object 75 is located on an image plane 62 with a radius of 2 meters (r=2). Taking lens B and lens C as an example, because the ideal imaging point 70 of the lens B image matches the ideal imaging point 70 of the lens C image, after completing the image stitching/blending operation, the equidistant rectangular wide-angle image will present a perfect stitching/blending result. However, in actual conditions, the lens center 76 of the lens B image and the lens C image has an offset ofs relative to the system center 73, as shown on the left side of FIG. 7A . As a result, after the image stitching/blending operation is completed, the equidistant rectangular wide-angle image will clearly show mismatched image defects.

Due to the optical properties of the lens, such as lens shading and luma shading, the size is equal to Wi The image center 74 of the lens image Hi may not be located in the middle of the lens image (Wi/2, Hi/2), where Wi and Hi represent the width and height of the lens image, respectively. In the offline stage, the correspondence generator 53 performs the following five steps to determine the texture coordinates of the actual image center 74 of a lens image. (i) Determine a brightness threshold TH to define the boundary of the lens image; (ii) Scan each row of pixels from left to right to determine the left boundary point of each row, as shown in Figure 7B. During scanning, if the brightness value of a pixel is less than TH, it means that the pixel is outside the lens image. Otherwise, it means that the pixel is inside the lens image. Afterwards, the left boundary point of each row is stored. Similarly, scan each column of pixels from right to left to determine and store the right boundary point of each column; scan each row of pixels from top to bottom to determine and store the upper boundary point of each row; scan each row of pixels from bottom to top to determine and store the lower boundary point of each row, as shown in Figure 7C. (iii) Calculate the column center Uc(i) of the i-th column based on the left boundary point and the right boundary point of the i-th column; and, calculate the row center Vc(j) of the j-th row based on the upper boundary point and the lower boundary point of the j-th row, where 0＜=i＜=(Wi-1) and 0＜=j＜=(Hi-1). (iv) Average the column centers of all columns to obtain an average u value, and average the row centers of all rows to obtain an average v value. (v) The texture coordinates (u _center , v _center ) of the real image center 74 of the lens image are respectively set equal to the average u value and the average v value. If any image center 74 cannot be calculated in the above manner, the texture coordinates (u _center , v _center ) of the image center 74 of the lens image are respectively set equal to (Wi/2, Hi/2).

FIG8A illustrates a grid for modeling an equirectangular wide-angle image. Referring to FIG8A , a grid can be viewed as a rectangular structure with five columns and six rows of quadrilaterals within the equirectangular wide-angle image. The grid example of FIG8A corresponds to the polygonal grid of FIG6C , and thus, the vertices/intersections 81 on the grid of FIG8A correspond to the vertices of the polygonal grid of FIG6C . In an equirectangular coordinate system, the horizontal distance between any two adjacent vertices on the horizontal axis is the same, and the vertical distance between any two adjacent vertices on the vertical axis is the same. The horizontal axis and the vertical axis in FIG8A are respectively denoted by and In the example of FIG8A , it is assumed that the three lenses of the image capture module 51 simultaneously capture a field of view covering a 160-degree HFOV and a 50-degree VFOV to generate three lens images, thereby forming the equidistant rectangular wide-angle image.

According to the present invention, each overlapping area in a projected image includes a row of P1 control points, where P1>=3. The size of these overlapping areas will change with the FOV of the three lenses of the image capture module 51, the lens sensor resolution and the lens installation angle. Generally speaking, in the projected image, the width of each overlapping area will be greater than or equal to a multiple of the width of a row of quadrilaterals. If each overlapping area includes multiple rows of quadrilaterals, one row of quadrilaterals (hereinafter referred to as the "default row") is preset in the multiple rows of quadrilaterals to accommodate P1 control points. In the example of FIG. 8A , the equirectangular wide-angle image includes two overlapping areas A( 1 ) and A( 2 ), and each overlapping area includes only one row of quadrilaterals, which are directly regarded as the default row for accommodating P1 control points.

In the offline stage, the correspondence generator 53 executes the following four steps (a) to (d) to determine the texture coordinates of each control point in the equidistant rectangular wide-angle image. (a) The leftmost row of quadrilaterals, the rightmost row of quadrilaterals and the default rows in each overlapping area of the equidistant rectangular wide-angle image are defined as "control rows". (b) In each control row, the top and bottom two control points are placed at the center of the top and bottom quadrilaterals respectively. In this step (b), each control row determines the x coordinates of each control point in the same control row according to the x coordinates of the left and right boundaries of its own quadrilateral in the equidistant rectangular region (e.g., x1 and x2), for example, x=(x1+x2)/2. (c) In each control row, the distance between the top and bottom two control points is divided by (P1-2) to obtain the y coordinates of the (P1-2) control points in the equidistant rectangular region. In other words, in each control row, the (P1-2) control points are evenly distributed between the top and bottom two control points. FIG. 8B illustrates two preset rows with different numbers of quadrilaterals and P1=3. As shown in FIG. 8B , two different control rows have five and six quadrilaterals, respectively. Therefore, the y coordinates of the three control points A~C of the two different preset rows are different. The range definition of the measurement area (used to measure the area error in the measurement mode) is as follows: for a quadrilateral i in the default row of each overlapping area, search for the closest control point j and classify or define the quadrilateral i as the measurement area j (corresponding to the control point j), where 1<=i<=n1, 1<=j<=P1 and n1 represents the number of quadrilaterals contained in each default row. (d) According to the equirectangular coordinates of each control point and the original vertex list (which defines the vertex mapping relationship between the equirectangular coordinates and the texture coordinates), interpolate to obtain the texture coordinates of each control point.

FIG8C is related to FIG8A , and illustrates an example in which an equidistant rectangular wide-angle image includes twenty control points R(1) to R(20) and ten measurement areas M(1) to M(10), and each control row includes five control points (P1=5). In FIG8C , the twenty control points R(1) to R(20) have twenty joint coefficients C(1) to C(20), and the joint coefficients C(1) to C(20) represent different degrees of joint of the twenty control points R(1) to R(20). In addition, ten control points R(1) to R(10) are located in two overlapping areas A(1) to A(2), and the other ten control points R(11) to R(20) are located in non-overlapping areas. Since the control points R(11)-R(20) are located in a non-overlapping region, the corresponding ten stitching coefficients C(11)-C(20) are set equal to a constant (e.g., 1), and are referred to as "constant stitching coefficients" in this specification. The following examples and implementation examples are illustrated using the twenty control points R(1)-R(20) and ten measurement regions M(1)-M(10) of the equal-distance rectangular wide-angle image in FIG. 8C as an example.

Referring back to FIG. 5A , the compensation device 52 includes a vertex processing device 510, a stitching decision unit 530, and an image processing device 520. In a measurement mode, the vertex processing device 510 receives the original vertex list and the test stitching coefficients C _t (1)~C _t from the stitching decision unit 520. (10) and constant stitching coefficients C(11)~C(20), and the texture coordinates of each vertex in the original vertex list in all lens images are corrected vertex by vertex, and a corrected vertex list (as shown in Table 2) is generated; according to the corrected vertex list and the three lens images from the image capture module 51, the image processing device 520 measures the regional errors E(1)~E(10) of all measurement areas M(1)~M(10) in the equidistant rectangular wide-angle image, and outputs the regional errors E(1)~E(10). Thereafter, the stitching decision unit 530 sets the test stitching coefficients C _t (1)~C _t according to the offset ofs. (10), and then receive the regional error amounts E(1)~E(10) to generate a 2D error table, and then generate the optimal stitching coefficients C(1)~C(10) of the ten control points R(1)~R(10) according to the 2D error table. On the other hand, in a display mode, the stitching decision unit 530 is disabled, and at the same time, the vertex processing device 510 and the image processing device 520 operate together to generate an equidistant rectangular wide-angle image according to the three lens images output by the image capture module 51 and the ten optimal stitching coefficients C(1)~C(10) and the ten constant stitching coefficients C(11)~C(20) output by the stitching decision unit 530.

In the measurement mode, the vertex processing device 510 corrects the texture coordinates of each vertex in each lens image according to the stitching coefficient mixing weights corresponding to each vertex in the original vertex list in each lens image and the "two test stitching coefficients" or "one test stitching coefficient and a constant stitching coefficient" of the two nearest control points, so that the image processing device 520 measures the regional errors E(1)~E(10) (such as steps S905~S906 in Figure 9A). In the display mode, the vertex processing device 510 corrects the texture coordinates of each vertex in each lens image according to the blending weights of the stitching coefficients corresponding to each vertex in the original vertex list in each lens image and the "two best stitching coefficients" or "one best stitching coefficient and a constant stitching coefficient" of the two nearest control points to minimize the above-mentioned mismatched image defects. FIG8D illustrates the positional relationship between a target vertex P and ten control points R(1) to R(10) in the lens C image. In the example of FIG8D , the first angle The first vector V1 is formed in a clockwise direction between a first vector V1 and a second vector V2; the first vector V1 starts from the image center 74 (with texture coordinates (u _center , v _center )) and ends at the position of the control point R(4), while the second vector V2 starts from the image center 74 and ends at the target vertex P (u _P , v _P ). Second angle is in a clockwise direction and is formed between a third vector V3 and the second vector V2; the third vector V3 starts from the image center 74 and ends at the position of the control point R(5). In the offline stage, the correspondence generator 53 pre-determines which two control points (i.e., R(4) and R(5)) are the closest to the target vertex P, and writes their indexes (4 and 5) to the "joining coefficient index" field of the lens C image in the data structure of the target vertex P in the original vertex list; in addition, the correspondence generator 53 also pre-calculates the blending weights (= /( + )), and the mixed weight ( /( + )) is written into the "blending weight (Alpha) of stitching coefficient" field of the lens C image in the data structure of the target vertex P in the original vertex list. Please note that a set of test stitching coefficients C _t (1) to C _t (10) (measurement mode) or a set of optimal stitching coefficients C (1) to C (10) (imaging mode) output from the stitching decision unit 530 is arranged as a 1D stitching coefficient array or a 1D data stream. Furthermore, in the measurement mode, according to the displacement ofs in Figure 7A, the stitching decision unit 530 sets the values of the test bonding coefficients C _t (1) ~ C _t (10) (such as step S902 in Figure 9A; to be explained later); when the measurement mode ends, the stitching decision unit 530 determines the values of the optimal bonding coefficients C(1) ~ C(10) (such as step S972 in Figure 9B; to be explained later), and uses the values of the optimal bonding coefficients C(1) ~ C(10) in the display mode.

One of the features of the present invention is to minimize the mismatched image defects within a preset number of loops (max in FIG. 9A ). The preset number of loops is related to an offset ofs, which is the distance that the lens center 76 of the image capture module 51 deviates from its system center 73 (see FIG. 7A ). In the measurement mode, according to the offset ofs in FIG. 7A , the stitching decision unit 530 sets ten test joint coefficients C _t (1) to C _t (10) to different value ranges to measure the regional errors E (1) to E (10), and the ten test joint coefficients are set to the same value each time (or each loop). For example, assuming ofs = 3 cm, the ten test joint coefficients C _t (1) to C _t (10) are set to a value range of 0.96 to 1.04. If each increment is 0.01, there will be a total of nine measurements (i.e., max = 9 in FIG. 9A ); assuming ofs = 1 cm, the ten test joint coefficients C _t (1) to C _t (10) are set to a value range of 0.99 to 1.00. If each increment is 0.001, there will be a total of ten measurements (i.e., max = 10 in FIG. 9A ). Please note that the offset ofs has been detected/determined in the offline stage, so the values of the ten test joint coefficients C _t (1) to C _t (10) can be predetermined.

FIG. 9A is a flowchart showing a method for determining the optimal joint coefficient of a control point according to an embodiment of the present invention (executed by the stitching decision unit 530 in the measurement mode). Assuming ofs=3 cm, the method for determining the optimal joint coefficients C(1)~C(10) of the ten control points R(1)~R(10) in FIG. 9A and the method flowchart of the coefficient decision step (step S912) in FIG. 9B are described below.

Step S902: Set the number of loops Q1 and the test bonding coefficient to new values. In one embodiment, Q1 is set to 1 in the first loop, and Q1 is increased by 1 in each subsequent loop; if ofs=3 cm, the test bonding coefficients C _t (1)~C _t (10) are all set to 0.96 in the first loop (i.e., C _t (1)=…=C _t (10)=0.96), and in subsequent loops, the test bonding coefficients C _t (1)~C _t (10) are set to 0.97,…., 1.04 in sequence.

Step S904: Clear all regional errors E(i) to 0, where i=1, 2, ..., 10.

Step S905: Generate a modified vertex list according to the values of the test stitching coefficients C _t (1) to C _t (10) and the original vertex list. The following is again explained using FIG. 8D as an example. After receiving the original vertex list from the correspondence generator 53, the vertex processing device 510 extracts two test stitching coefficients (C t (4) and C t (5)) from the one-dimensional test stitching coefficient array according to the "stitching coefficient index" field (i.e., 4 and 5) in the data structure of the target vertex P in the lens C image, and then extracts two test stitching coefficients (C _t (4) and C _t (5)) according to the "blending weight (Alpha) of the stitching coefficient" field (i.e., 4 and 5) in the data structure of the target vertex P in the lens C image (please refer to Table 1). /( + )), the interpolation coefficient C' is calculated according to the following equation: C'=C _t (4) ( /( + ))+C _t (5) ( /( + )). Thereafter, the vertex processing device 510 calculates the modified texture coordinates (u _P ', v _P ') of the target vertex P in the lens C image according to the following equations: u _P '=(u _P - u _center )* C' + u _center ; v _P '=(v _P - v _center )*C' + v _center . In this way, the vertex processing device 510 sequentially modifies the texture coordinates of each vertex from the original vertex list in the lens C image according to the ten test joint coefficients C _t (1)~C _t (10) to generate a portion of a modified vertex list. Similarly, the vertex processing device 510 also sequentially modifies the texture coordinates of each vertex from the original vertex list in the lens A image and the lens B image according to the ten test joint coefficients C _t (1) to C _t (10) and the ten constant joint coefficients C (11) to C (20) to complete the modified vertex list. Table 2 shows an example of the data structure of each vertex in the modified vertex list. Table II Attributes instruction (x, y) Equirectangular coordinates N Number of overlapping camera images ID ₁ The ID of the first camera image (u ₁ ', v ₁ ') Corrected texture coordinates in the first camera image w ₁ The stitching blending weights of the first lens image ….. ……. ID _N The ID of the Nth lens image (u _N ', v _N ') Corrected texture coordinates in the Nth lens image _1v The stitching blending weight of the Nth lens image

Step S906: The image processing device 520 measures the regional errors E(1)-E(10) of the ten measurement areas M(1)-M(10) of the equidistant rectangular wide-angle image according to the corrected vertex list and the three lens images from the image capture module 51 (described in detail in FIG. 5B ). For ease of description, E(i)=f(C _t (i)) is used to represent this step S906, where i=1,…,10, and f() represents (measured by the image processing device 520) according to the corrected vertex list and the three lens images.

Step S908: Store all the regional errors E(1)~E(10) in a two-dimensional (2D) error table. Table 3 shows an example of the 2D error table when ofs=3 cm (the value range of the test joint coefficient is 0.96~1.04). In the 2D error table of Table 3, there are a total of ten regional errors E(1)~E(10) and nine different values of the test joint coefficient (0.96~1.04). Table 3 1st 2nd 3rd ---- 7th 8th 9th Bonding coefficient 0.96 0.97 0.98 ---- 1.02 1.03 1.04 E(1) E(2) ----------------------------- E(7) ---- E(8) E(9) E(10)

Step S910: Determine whether the number of loops Q1 has reached the maximum value of 9. If yes, jump to step S912, otherwise, return to step S902.

Step S912: Perform coefficient decision operation according to the above 2D error table.

Step S914: Output the best stitching coefficient C(i), where i=1, 2, ..., 10. In the imaging mode, the best stitching coefficients C(i) are output to the vertex processing device 510 to generate a corresponding modified vertex list, so that the image processing device 520 generates a corresponding wide-angle image (to be described later) according to the corresponding modified vertex list and the three lens images from the image capture module 51.

FIG9B is a flowchart showing a method for the splicing decision unit 530 to perform the coefficient decision operation of step S912 according to an embodiment of the present invention.

Step 961: Set Q2 to 0 for initialization.

Step S962: Extract a selected decision group from the above 2D error table. Returning to FIG8C, each control region is usually adjacent to two control regions, and a selected control region and its two adjacent control regions form a selected decision group to determine the optimal joint coefficient of the selected control region. For example, a selected control point R(9) and its two adjacent control points R(8) and R(10) form a decision group, as shown in FIGS. 8A and 8C. However, if a selected control point (such as R(6)) is located at the top or bottom of the overlap region A(2), the selected control point R(6) will only form a decision group with its only adjacent control point R(7) to determine its optimal joint coefficient C(6). The description of the subsequent steps assumes that a control point R(7) is selected, and R(7) and its two adjacent control points R(6) and R(8) form a selected decision group to determine its optimal joint coefficient C(7).

Step S964: Determine the local minimum value in the regional error of each control point of the selected decision group. Table 4 shows an example of the regional error E(6)~E(8) and the test joint coefficient of R(6)~R(8). Table 4 index Test bonding coefficient E(6) E(7) E(8) 1 0.96 1010 2600(*) 820 2 0.97 1005 2650 750 3 0.98 1000 2800 700 4 0.99 900 3000 600(*) 5 1.00 800(*) 2700 650 6 1.01 850 2500 580 7 1.02 950 2400(*) 500(*) 8 1.03 960 2820 700 9 1.04 975 2900 800

As shown in Table 4, there is only one local minimum value among the nine regional errors of R(6), while there are two local minimum values among the nine regional errors of R(7) and R(8), respectively. An asterisk (*) is marked next to each local minimum value in Table 4.

Step S966: Select candidates based on the local minima. Table 5 shows the candidates selected from the local minima in Table 4, where ID represents the index, WC represents the join coefficient, and RE represents the regional error. The number of candidates is equal to the number of local minima in Table 4. Table 5 R(6) R(7) R(8) Number of local minima 1 2 2 ID WC RE ID WC RE ID WC RE Candidates[0] 5 1.00 800 1 0.96 2600 4 0.99 600 Candidate[1] 7 1.02 2400 7 1.02 500

Step S968: Create a link metric based on the candidates in Table 5. As shown in FIG9C , a link metric is created based on the candidates in Table 5.

Step S970: Determine the minimum sum of the link metric values among all paths of the link metric. =0.03 and =0.06, the minimum value between the two =min( )=0.03. About the two link measurements =0.03 and =0.00, the minimum value between the two =min( )=0.00. Then, the sum of the link metrics of path 0-0-0 and path 0-1-1 is calculated as follows: + =0.04+0.03=0.07 and + =0.02+0.00=0.02. Because , so we can decide (Path 0-1-1) is the minimum sum of link metric values among all paths of the link metric, such as the solid line path in FIG. 9C .

Step S972: Determine the optimal bonding coefficient for the selected control region. In the example of step S970, because (Path 0-1-1) is the minimum sum of the link metrics of all paths, so 1.02 is determined as the optimal joint coefficient for control region R(7). However, if the sum of the link metrics of two or more paths is the same at the end of the calculation, the joint coefficient of the node with the minimum regional error is selected as the optimal joint coefficient for the selected control region. Here, the value of the loop number Q2 is increased by 1.

Step S974: Determine whether the number of loops Q2 reaches the upper limit TH1 (=10). If so, end this process, otherwise, return to step S962 to process the next control area.

After the vertex processing device 510 corrects all texture coordinates of all vertices from the original vertex list according to the test/optimal joint coefficients C(1)-C(20), the mismatch image defect problem caused by the lens center offset of the image capture module 51 (i.e., a lens center 76 has an offset ofs relative to the system center 73) can be greatly improved (i.e., the actual imaging position 78 will be pushed toward the ideal imaging position 70), as shown in FIG7A. Please note that because the sphere 62 is virtual, the object 75 may be located outside, inside, or on the surface of the sphere 62.

FIG5B is a schematic diagram showing the image processing device according to an embodiment of the present invention. Referring to FIG5B , the image processing device 520 includes a rasterization engine 521, a texture mapping circuit 522, a blending unit 523 (controlled by a control signal CS2), a destination buffer 524, and a measurement unit 525 (controlled by a control signal CS1). Please note that in the measurement mode, if the equidistant rectangular coordinates of a pixel/point fall within any measurement area, the blending unit 523 will be disabled through the control signal CS2, and the measurement unit 524 will be enabled through the control signal CS1; in the display mode, the blending unit 523 will be enabled and the measurement unit 524 will be disabled through the two control signals CS1 and CS2. The texture mapping circuit 522 includes two texture mapping engines 52a-52b. The rasterization engine 521 can perform a quadrilateral rasterization operation on each pixel in a quadrilateral (such as FIG. 6C ) formed by each group of four vertices from a modified vertex list, or perform a triangle rasterization operation on each pixel in a triangle (such as FIG. 6C ) formed by each group of three vertices from the modified vertex list.

Please go back to FIG. 8C . For the case of a quadrilateral, assume that a set of four vertices (E, F, G, H) from the modified vertex list (forming a quadrilateral in the polygonal grid) are located within one of the five measurement areas M(1) to M(5) of the overlap area A(1) and are overlapped by the lens B image and the lens C image (N=2). The four vertices (E, F, G, H) respectively include the following data structures: Vertex E: {(x _E , y _E ), 2, ID _lens-B , (u _1E , v _1E ), w _1E , ID _lens-c , (u _2E , v _2E ), w _2E }; Vertex F: {(x _F , y _F ), 2, ID _lens-B , (u _1F , v _{1F )} , ), w _1F , ID _lens-C , (u _2F , v _2F ), w _2F }; vertex G: {(x _G , y _G ), 2, ID _lens-B , (u _1G , v _1G ), w _1G , ID _lens-C , (u _2G , v _2G ), w _2G }; vertex H: {(x _H , y _H ), 2, ID _lens-B , (u _1H , v _1H ), w _1H , ID _lens-C , (u _2H , v _2H ), w _2H }. The rasterization engine 521 directly performs a quadrilateral rasterization operation on each point/pixel within the quadrilateral EFGH. Specifically, the rasterization engine 521 calculates the texture coordinates of each lens image for a point Q (having equidistant rectangular coordinates (x, y) and located in the quadrilateral EFGH of the polygonal grid) using the following steps: (1) using a bilinear interpolation method, four spatial weights (e, f, g, h) are calculated based on the equidistant rectangular coordinates (x _E , y _E , x _F , y _F , x _G , y _G , x _H , y _H , x, y); (2) the working surface blending weight of a sampling point Q _B in the lens B image (corresponding to the point Q) is calculated: fw ₁ =e*w _1E +f*w _1F +g*w _1G +h*w _1H ; the working surface blending weight of a sampling point Q _C in the lens C image is calculated: The working surface blending weight (corresponding to the point Q) is: fw ₂ = e*w _2E +f*w _2F +g*w _2G +h*w _2H ; (3) Calculate the texture coordinates of the sampling point Q _B (corresponding to the point Q) in the lens image B: (u1, v1) =(e*u _1E +f*u _1F +g*u _1G +h*u _1H , e*v _1E +f*v _1F +g*v _1G +h*v _1H ); Calculate the texture coordinates of the sampling point Q _C (corresponding to the point Q) in the lens image C: (u2, v2) =(e*u _2E +f*u _2F +g*u _2G +h*u _2H , e*v _2E +f*v _2F +g*v _2G +h*v _2H ). Finally, the rasterization engine 521 transmits the two texture coordinates (u1, v1) and (u2, v2) in parallel to the two texture mapping engines 52a-52b, wherein e+f+g+h = 1 and _fw1 + _fw2 = 1. According to the two texture coordinates (u1, v1) and (u2, v2), the two texture mapping engines 52a-52b use any suitable method (such as nearest-neighbor interpolation, bilinear interpolation, or trilinear interpolation) to texture map the texture data of the lens B image and the lens C image to generate two sample values s1 and s2. Each of the sample values may be a luma value, a chroma value, an edge value, a pixel color value (RGB) or a motion vector.

For the case of triangles, similar to the above-mentioned operations for quadrilaterals, the rasterization engine 521 and the two texture mapping engines 52a~52b perform triangle rasterization operations and texture mapping on each pixel in a triangle formed by each group of three vertices from the modified vertex list (as shown in Figure 6C) to generate two corresponding sample values s1 and s2, except that step (1) is modified as follows: the rasterization engine 521 does not use _the above-mentioned bilinear interpolation method, but instead uses a barycentric weighting method to calculate the three spatial weights ( _e , _f , g) of the three vertices (E, _F , _G ) (forming a triangle of the polygonal grid of Figure 6C) based on equidistant rectangular coordinates ( _xE , yE, xF, yF, xG, yG, x, y).

Next, the rasterization engine 521 determines whether the point Q falls into one of the five measurement areas M(1)-M(5) according to the equidistant rectangular coordinates (x, y) of the point Q. If the point Q falls into one of the five measurement areas M(1)-M(5), the control signal CS1 is set to be valid (asserted) to enable the measurement unit 525 to start measuring the regional error of the measurement area. The measurement unit 525 can use any known algorithm, such as sum of absolute differences (SAD), sum of squared differences (SSD), median absolute deviation (MAD), etc., to estimate/measure the regional errors of the control areas. For example, if the point Q is determined to fall into the measurement area M(1), the measurement unit 525 uses the following equation: E= |s1-s2|; E(1) += E to accumulate the absolute value of the difference between the sample values of each point in the measurement area M(1) in the lens B image and the corresponding point in the measurement area M(1) in the lens C image to obtain a SAD value as the regional error E(1) of the measurement area M(1). In this way, the measurement unit 525 measures the regional errors E(1) to E(5) of the five measurement areas M(1) to M(5). In the same manner, the measuring unit 525 measures the regional errors E(6)~E(10) of the five measuring areas M(6)~M(10) according to the corrected vertex list, the lens C image and the lens A image.

In the display mode, the rasterization engine 521 and the texture mapping circuit 522 operate in the same manner as in the measurement mode. Hereinafter, the above example (a point Q has equidistant rectangular coordinates (x, y) and is located in the quadrilateral EFGH of the polygonal grid, and the quadrilateral EFGH is overlapped by the lens B image and the lens C image (N=2)) is used for explanation. After the two texture mapping engines 52a~52b texture map the texture data of the lens B image and the lens C image to generate two sample values s1 and s2, the blending unit 523 blends the two sample values s1 and s2 according to the following equation to generate the blend value Vb of the point Q: Vb= fw ₁ *s1+ fw ₂ *s2. Finally, the blending unit 523 stores the blending value Vb of point Q in the destination buffer 524. In this way, the blending unit 523 stores all the blending values Vb in the destination buffer 524 in sequence until all the points in the quadrilateral EFGH are processed. On the other hand, assuming that a point Q' has equidistant rectangular coordinates (x', y') and is located in the quadrilateral E'F'G'H' of the polygonal grid of Figure 6C, and the quadrilateral E'F'G'H' is located in the non-overlapping area of the lens B image (N=1), the rasterization engine 521 only transmits the texture coordinates (such as (u1, v1)) to a texture mapping engine 52a, and a working surface blending weight fw ₁ (=1) to the blending unit 523. Correspondingly, after the texture mapping engine 52a texture maps the texture data of the lens B image to generate a sample value s1, the blending unit 523 generates a blending value Vb (= fw ₁ * s1) of the point Q'. In this way, the blending unit 523 stores all the blending values Vb in sequence in the destination buffer 524 until all the points in the quadrilateral E'F'G'H' are processed. Once all the quadrilaterals and triangles are processed, a projected image is stored in the destination buffer 524.

When the image capture module 51 is an outward-facing M-lens camera and the projected image is a panoramic image, in the offline stage, the correspondence generator 53 also executes the aforementioned four steps (a) to (d) to determine the texture coordinates of the control points, except that step (a) is modified as follows: the default rows in each overlapping area of the equidistant rectangular panoramic image are defined as "control rows". Taking M=4 (as shown in Figure 10A) as an example, Figure 10A illustrates an outward-facing four-lens camera mounted on the frame 11E. Assume that the outward-facing four-lens camera of Figure 10A can simultaneously capture a field of view covering a 360-degree HFOV and a 50-degree VFOV to generate four lens images (left lens, front lens, right lens, and back lens). FIG. 10B illustrates a grid for modeling an equirectangular panoramic image, and the equirectangular panoramic image includes four overlapping areas A(1) to A(4) and four lens images (generated by the outward-facing four-lens camera of FIG. 10A). Assume that the grid example of FIG. 10B corresponds to the polygonal grid of FIG. 6C, so the vertices/intersections on the grid of FIG. 10B correspond to the vertices of the polygonal grid of FIG. 6C. FIG. 10C illustrates an equirectangular panoramic image including twenty control points R(1) to R(20) and twenty measurement areas M(1) to M(20) according to FIG. 10B, and each control row includes five control points (P1=5). In the example of FIGS. 10B-10C , since the twenty control points R(1) to R(20) are all located in the overlapping area A(1) to A(4), the method of FIGS. 9A to 9B is used to determine the optimal joint coefficients C(1) to C(20) of the twenty control points R(1) to R(20) and TH1=20.

The compensation device 52 and the correspondence generator 53 of the present invention can be implemented by software, hardware, or a combination of software (or firmware) and hardware. An example of a simple solution is a field programmable gate array or an application specific integrated circuit. In a preferred embodiment, the vertex processing device 510 and the image processing device 520 are implemented using a graphics processing unit and a first program memory; the splicing decision unit 530 and the correspondence generator 53 are implemented using a first general-purpose processor and a second program memory. The first program memory stores a first processor executable program, and the second program memory stores a second processor executable program. When the graphics processing unit executes the first processor executable program, the graphics processing unit is configured to operate as: the vertex processing device 510 and the image processing device 520. When the first general-purpose processor executes the second processor executable program, the first general-purpose processor is configured to operate as: the splicing decision unit 530 and the correspondence generator 53.

In another embodiment, the compensation device 52 and the correspondence generator 53 are implemented using a second general purpose processor and a third program memory. The third program memory stores a third processor executable program. When the second general purpose processor executes the third processor executable program, the second general purpose processor is configured to operate as: the vertex processing device 510, the splicing decision unit 530, the correspondence generator 53 and the image processing device 520.

The above are only preferred embodiments of the present invention and are not intended to limit the scope of the patent application of the present invention; any other equivalent changes or modifications that do not deviate from the spirit disclosed by the present invention should be included in the scope of the following patent application.

11A Cube structure 11B~11E Structure 12, 62 Sphere 22, 32, 33 Body 41, A(1)~A(4) Overlapping area 42, 43 Non-overlapping area 70 Ideal imaging position 74 Image center of lens image 73 Camera system center of image acquisition module 51 75 Object 76 Lens center 78 Actual imaging position 81 Intersection point/vertex 500 Projection image processing system 51 Image acquisition module 52 Compensation device 53 Correspondence generator 510 Vertex processing device 520 Image processing device 530 Stitching decision unit 521 Rasterization engine 522 Texture mapping circuit 52a, 52b Texture mapping engine 523 Mixing unit 524 Destination buffer 525 Measuring unit D1~D7 Optical axis of lens R(1)~R(20) Control point M(1)~M(20) Measuring area First Angle Second angle V1 First vector V2 Second vector V3 Third vector O1~O3 Intersection of lens optical axis

[FIG. 1A] shows the relationship between a cubic structure 11A and a sphere 12, which has been disclosed in the patent document No. 1 728620 of the Republic of China. [FIG. 1B] shows an equidistant rectangular panoramic image, which is an equidistant rectangular projection of six lens images (top, bottom, back, left, right, front) of a six-lens camera mounted on six working surfaces of the cubic structure 11A. [FIG. 2A] illustrates an outward-facing six-lens camera mounted on the structure 11A of FIG. 1A. [FIG. 2B] illustrates an outward-facing three-lens camera mounted on the structure 11B and protruding from the fuselage 22. [FIG. 3A] is a diagram showing two side views of an inward-facing three-lens camera mounted on a frame 11C, located inside a body 32 and having a lens free from wear and tear according to an embodiment of the present invention. [FIG. 3B] is a diagram showing two side views of an inward-facing dual-lens camera mounted on a frame 11D according to an embodiment of the present invention. [FIG. 4A] shows a mounting method of an inward-facing three-lens camera and an outward-facing three-lens camera. [FIG. 4B] shows a wide-angle image including three lens images (output by the outward-facing three-lens camera of FIG. 4A). [FIG. 4C] shows a wide-angle image including three lens images (output by the inward-facing three-lens camera of FIG. 4A). [FIG. 4D] illustrates a wide-angle image including two lens images (output by the inward-facing dual-lens camera of FIG. 3B). [FIG. 5A] is a block diagram of a projection image processing system according to an embodiment of the present invention. [FIG. 5B] is a schematic diagram of an image processing device 520 according to an embodiment of the present invention. [FIG. 6A] shows the relationship between a structure 11C and a sphere 62. [FIG. 6B] shows a triangular mesh for modeling a sphere surface. [FIG. 6C] shows a polygonal mesh for forming/modeling an equirectangular projection image. [FIG. 7A] shows that the vertex processing device 510 improves the mismatched image defects caused by the offset ofs (the distance of the lens center 76 from the camera system center 73) after modifying the texture coordinates of all vertices in each lens image according to the optimal joint coefficient. [FIG. 7B] shows that each column of pixels is scanned from left to right to obtain the left boundary point of each column, and each column of pixels is scanned from right to left to obtain the right boundary point of each column. [FIG. 7C] shows that each row of pixels is scanned from top to bottom to obtain the upper boundary point of each row, and each row of pixels is scanned from bottom to top to obtain the lower boundary point of each row. [FIG. 8A] illustrates a grid for modeling an equidistant rectangular wide-angle image, and the equidistant rectangular wide-angle image includes two overlapping areas A(1)-A(2) and a three-lens image from the inward-facing three-lens camera of FIG. 3A. [FIG. 8B] illustrates two preset rows with different numbers of quadrilaterals and P1=3. [FIG. 8C] is related to FIG. 8A, illustrating an equidistant rectangular wide-angle image including twenty control points R(1)-R(20) and ten measurement areas M(1)-M(10) and each control row includes five control points (P1=5). [FIG. 8D] illustrates the positional relationship between a target vertex P and ten control points R(1)-R(10) in the lens C image. [FIG. 9A] is a flow chart showing a method for determining the optimal stitching coefficient of control points according to an embodiment of the present invention. [FIG. 9B] is a flowchart showing a method for performing the coefficient decision operation of step S912 according to an embodiment of the present invention. [FIG. 9C] shows that among all the paths of the link measurement, the sum of the link measurement values of path 0-1-1 (solid line path) is the smallest. [FIG. 10A] illustrates an external-facing four-lens camera mounted on frame 11E. [FIG. 10B] illustrates a grid for modeling an equidistant rectangular panoramic image, and the equidistant rectangular panoramic image includes four overlapping areas A(1) to A(4) and four lens images generated by the external-facing four-lens camera of FIG. 10A. [Figure 10C] is based on Figure 10B, illustrating an equidistant rectangular panoramic image including twenty control points R(1)~R(20) and twenty measurement areas M(1)~M(20), and each control row includes 5 control points (P1=5).

74 Image center R(1)~R(10) of lens C image Control point _1First Angle ₂ Second angle V1 First vector V2 Second vector V3 Third vector

Claims (25)

An image processing system includes: a camera with M lenses, for capturing a field of view covering an X-degree horizontal field of view and a Y-degree vertical field of view to generate M lens images; a compensation device, for generating a projection image according to a first vertex list and the M lens images; and a correspondence generator, for generating a set of operations, including: correcting a plurality of vertices to define a plurality of first vertex mappings between the M lens images and the projection image; horizontally and vertically Scan the pixels of each lens image to determine the image center of each lens image; determine the texture coordinates of all control points based on the first vertex mappings and P1 control points in each overlapping area in the projection image; and determine the two neighboring control points of each vertex in each lens image and a coefficient mixing weight based on the texture coordinates of all control points and the image center of each lens image to generate the first vertex list, where X<=360, Y<180, M>=2 and P1>=3. A system as claimed in claim 1, wherein the first vertex list includes the vertices having a first data structure, and the first data structure of each vertex includes a first vertex mapping between the M lens images and the projection image, two indices of the two neighboring control points in each lens image, and the coefficient blending weight. The system of claim 1, wherein the projection image is a set of multiple quadrilaterals defined by the vertices, and the texture coordinate operation for determining all control points includes: In each overlapping area of the projection image, select one row of quadrilaterals as a default row; when the projection image is a wide-angle image, define each default row, the leftmost row of quadrilaterals and the rightmost row of quadrilaterals of the projection image as control rows; when the projection image is a panoramic image, define each default row as a control row; The top control point and a bottom control point are placed at the center of the topmost quadrilateral and the bottommost quadrilateral of each control row to determine the x coordinates of the P1 control points in each control row of the projection image; the distance between the top control point and the bottom control point is divided by (P1-2) to determine the y coordinates of the P1 control points in each control row of the projection image; and interpolation is performed based on the first vertex mappings and the x coordinates and y coordinates of each control point to obtain the texture coordinates of each control point. The system of claim 3, wherein the operation of determining the texture coordinates of all control points further comprises: for a quadrilateral i in each default row, searching for a closest control point j; and classifying the quadrilateral i into a measurement area j; wherein 1<=i<=n1, 1<=j<=P1, and n1 represents the number of quadrilaterals in each default row. A system as claimed in claim 1, wherein the projected image is derived from a preset projection of the M lens images, and the projected image is one of a panoramic image and a wide-angle image. A system as claimed in claim 5, wherein the default projection is one of the following: equirectangular projection, Miller projection, Mercator projection, Lambert cylindrical equal area projection, and Panini projection. The system of claim 1, wherein the operation of scanning each lens image horizontally and vertically includes: for a target lens image, scanning each row of pixels from left to right and from left to right respectively according to a brightness threshold value to determine a left boundary point and a right boundary point of each row; scanning each row of pixels from top to bottom and from bottom to top respectively according to the brightness threshold value to determine an upper boundary point and a lower boundary point of each row; calculating the row center of each row according to the left boundary point and the right boundary point of each row; calculating the row center of each row according to the upper boundary point and the lower boundary point of each row; averaging the row centers of all rows to obtain the x coordinate of the image center of the target lens image; and averaging the row centers of all rows to obtain the y coordinate of the image center of the target lens image. A system as claimed in claim 1, wherein the coefficient blending weight at a target vertex in a specific lens image is related to a first angle and a second angle, wherein the first angle is formed between a first vector and a second vector, and the second angle is formed between a third vector and the second vector, wherein the first vector starts from the image center of the specific lens image and ends at a first control point of the two neighboring control points, and the second vector starts from the image center of the specific lens image and ends at the target vertex, and wherein the third vector starts from the image center of the specific lens image and ends at a second control point of the two neighboring control points. The system of claim 1, wherein the compensation device comprises: a splicing decision unit, in a measurement mode, determining a set of optimal stitching coefficients for the control points according to a two-dimensional error table, wherein the two-dimensional error table comprises a plurality of test stitching coefficients and a plurality of accumulated pixel value errors of a plurality of measurement areas corresponding to the control points in each overlap area; a vertex processing device, in accordance with the test stitching coefficients in the measurement mode or the set of optimal stitching coefficients in the imaging mode, correcting the texture coordinates of all vertices from the first vertex list in each lens image to generate a second vertex. point list; and an image processing device, receiving the M lens images and the second vertex list, forming the projection image in the display mode, and measuring the accumulated pixel value errors of the measurement areas in the measurement mode; wherein the values of the test joint coefficients are related to the offset of the center of a lens of the M lenses relative to the center of the camera system of the camera having the M lenses; and wherein the second vertex list includes the vertices having a second data structure, and the second data structure of each vertex defines the second vertex mapping between the M lens images and the projection image. The system of claim 9, wherein the vertex processing device is further used to perform the following operations in the display mode: for a target vertex in the first vertex list, (1) in each lens image, based on the two indices of the two adjacent control points in the first data structure of the target vertex, two selected coefficients are taken from the optimal joint coefficients; (2) in each lens image, based on the two selected coefficients and (3) in each lens image, according to the interpolation joint coefficient and the texture coordinates in the first data structure of the target vertex, calculate a modified texture coordinate; and (4) repeat steps (1) to (3) until the modified texture coordinates of all vertices are calculated, so as to generate the second vertex list. A system as claimed in claim 1, wherein the camera having M lenses is an inward-facing M-lens camera, and the intersection of the optical axes of the inward-facing M lenses is formed above the inward-facing M lenses. A system as claimed in claim 1, wherein the camera having M lenses is an outward-facing M-lens camera, and the intersection of the optical axes of the outward-facing M lenses is formed below the outward-facing M lenses. An image processing method includes: calibrating a plurality of vertices to define a plurality of first vertex mappings between M lens images and a projected image, wherein the M lens images are generated by a camera having M lenses capturing a field of view covering an X-degree horizontal field of view and a Y-degree vertical field of view; horizontally and vertically scanning pixels of each lens image to determine an image center of each lens image; and determining a plurality of first vertex mappings between the first vertex mappings and the projected image. The P1 control points in each overlapping area determine the texture coordinates of all control points; and based on the texture coordinates of all control points and the image center of each lens image, determine the two neighboring control points of each vertex in each lens image and a coefficient mixing weight to generate a first vertex list; and based on the first vertex list and the M lens images, generate the projected image, where X<=360, Y<180, M>=2 and P1>=3. The method of claim 13, wherein the first vertex list includes the vertices having a first data structure, and the first data structure of each vertex includes a first vertex mapping between the M lens images and the projection image, two indices of the two neighboring control points in each lens image, and the coefficient blending weight. The method of claim 13, wherein the projection image is a set of multiple quadrilaterals defined by the vertices, and the step of determining the texture coordinates of all control points includes: selecting a row of quadrilaterals in each overlapping area of the projection image as a default row; when the projection image is a wide-angle image, defining each default row, the leftmost row of quadrilaterals and the rightmost row of quadrilaterals of the projection image as control rows; when the projection image is a panoramic image, defining each default row as a control row; defining a vertex as a default row. The top control point and a bottom control point are placed at the center of the topmost quadrilateral and the bottommost quadrilateral of each control row to determine the x coordinates of the P1 control points in each control row of the projection image; the distance between the top control point and the bottom control point is divided by (P1-2) to determine the y coordinates of the P1 control points in each control row of the projection image; and interpolation is performed based on the first vertex mappings and the x coordinates and y coordinates of each control point to obtain the texture coordinates of each control point. The method of claim 15, wherein the step of determining the texture coordinates of all control points further comprises: for a quadrilateral i in each default row, searching for a closest control point j; and classifying the quadrilateral i into a measurement area j; wherein 1<=i<=n1, 1<=j<=P1, and n1 represents the number of quadrilaterals in each default row. The method of claim 13, wherein the projected image is derived from a preset projection of the M lens images, and the projected image is one of a panoramic image and a wide-angle image. The method of claim 17, wherein the default projection is one of the equirectangular projection, the Miller projection, the Mercator projection, the Lambert cylindrical equal area projection, and the Panini projection. As in the method of claim 13, the step of scanning each lens image horizontally and vertically includes: for a target lens image, scanning each row of pixels from left to right and from left to right respectively according to a brightness threshold value to determine a left boundary point and a right boundary point of each row; scanning each row of pixels from top to bottom and from bottom to top respectively according to the brightness threshold value to determine an upper boundary point and a lower boundary point of each row; calculating the row center of each row according to the left boundary point and the right boundary point of each row; calculating the row center of each row according to the upper boundary point and the lower boundary point of each row; averaging the row centers of all rows to obtain the x coordinate of the image center of the target lens image; and averaging the row centers of all rows to obtain the y coordinate of the image center of the target lens image. The method of claim 13, wherein the coefficient blending weight of a target vertex in a specific lens image is related to a first angle and a second angle, wherein the first angle is formed between a first vector and a second vector, and the second angle is formed between a third vector and the second vector, wherein the first vector starts from the image center of the specific lens image and ends at a first control point of the two neighboring control points, and the second vector starts from the image center of the specific lens image and ends at the target vertex, and wherein the third vector starts from the image center of the specific lens image and ends at a second control point of the two neighboring control points. The method of claim 13, wherein the step of generating the projection image comprises: determining a set of optimal stitching coefficients for the control points based on a two-dimensional error table, wherein the two-dimensional error table comprises a plurality of test stitching coefficients and a plurality of accumulated pixel value errors of a plurality of measurement areas corresponding to the control points in each overlap area; and correcting the texture coordinates of all vertices from the first vertex list in each lens image based on the set of optimal stitching coefficients to generate a second Vertex list; and receiving the M lens images and the second vertex list to form the projection image; wherein the values of the test joint coefficients are related to the offset of a lens center of the M lenses relative to the camera system center of the camera having the M lenses; and wherein the second vertex list includes the vertices having a second data structure, and the second data structure of each vertex defines a second vertex mapping between the M lens images and the projection image. The method of claim 21, wherein the step of modifying all vertices from the first vertex list comprises: for a target vertex in the first vertex list, (1) for each lens image, based on two indices of the two adjacent control points in the first data structure of the target vertex, taking two selected coefficients from the optimal joint coefficients; (2) for each lens image, based on the two selected coefficients and (3) for each lens image, a modified texture coordinate is calculated according to the interpolation joint coefficient and the texture coordinate in the first data structure of the target vertex; and (4) steps (1) to (3) are repeated until the modified texture coordinates of all vertices are calculated to generate the second vertex list. The method of claim 21, wherein the step of determining the set of optimal stitching coefficients of the control points comprises: (a) setting the values of the test stitching coefficients to be equal to one of a plurality of preset values within a preset value range according to the offset of the lens center of the M lenses relative to the camera system center of the camera having the M lenses; (b) correcting the texture coordinates of all vertices from the first vertex list in each lens image according to the values of the test stitching coefficients; (c) ) calculate the accumulated pixel value errors of the measurement areas; (d) repeat steps (a) to (c) until the preset values within the preset value range are processed to generate the two-dimensional error table; and (e) determine the optimal joint coefficient of each target control point in a group-by-group manner according to the local minimum value between the accumulated pixel value errors of each control point in each decision group, wherein each decision group includes the target control point and one or two neighboring control points. As in the method of claim 13, the camera with M lenses is an inward-facing M lens camera, and the intersection of the optical axes of the inward-facing M lenses is formed above the inward-facing M lenses. As in the method of claim 13, wherein the camera with M lenses is an outward-facing M lens camera, and the intersection of the optical axes of the outward-facing M lenses is formed below the outward-facing M lenses.