TW202248963A - Compound eyes system, the vehicle using the compound eyes systme and the image processing method for the compound eyes system - Google Patents

Abstract

A compound eye camera system includes a first lens, at least four second lenses, a storage unit and an image processing controller. The storage unit is used to store multiple source image files captured by the first lens or the second lens, The image processing controller is used to identify the image features of at least one object to be measured in the source image files captured in the first shooting area, and uses the reference length as a ruler to construct a 3D spatial digital model; Thus, it can be used to assist vehicle monitoring and automatic driving, so that it has 3D space recognition and multi object monitoring in 3D space, so as to improve the level of unmanned monitoring.

Description

Compound eye camera system, vehicle using compound eye camera system and image processing method thereof

The present invention relates to a multi-lens compound eye camera system, especially a compound eye camera system and its images that can be used in different industrial equipment such as vehicle monitoring, AI robots, automatic driving, sweeping robots, aerial drones, and multi-axis processing machines and instruments. Approach.

Self-driving cars have become a hot topic in recent years. Whether it is traditional car manufacturers such as GM, Volvo, Toyota, or newcomers such as Telsa, UBER, Waymo, Nuro.ai, etc., they are actively collecting environmental information through road tests of experimental vehicles to build self-driving cars. Traffic data required for AI deep learning. As the "eyes" of the self-driving car, the self-driving vehicle measured on the road is also equipped with multiple perception systems such as image sensing, among which the LiDAR sensor plays a key role. The function of the lidar is to measure the distance of surrounding vehicles and objects, etc., and create and identify 3D spatial images. At this stage, the main application areas include telemetry, factory automation equipment, and disaster prevention and monitoring of social infrastructure such as railways and tunnels.

Compared with image sensing devices that use the visible light range, LiDAR uses near-infrared rays and is less susceptible to interference from light in the environment. LiDAR sensing has many advantages, including It can be identified within the range of receiving reflected light (approximately 200 meters or even 300 meters for automotive products), and is not affected by the intensity of light or shadows in the environment; sensors such as infrared rays and millimeter waves are mainly used to measure distances. If the 3D vision is established by image sensing, it needs to rely on multiple image sensing devices; LiDAR can use one sensor to create three-dimensional environment scanning information, so that it can maintain the quality in the long-distance use situation Measurement accuracy. In addition, some international companies such as Google, Waymo, and Uber have also developed sensor fusion technology that includes LiDAR, integrating the information returned by LiDAR sensing technology with the information detected by other types of sensors, and giving different errata Logic, to improve the overall recognition accuracy, as the basic information for artificial intelligence deep learning training and inference required for future self-driving cars.

Here, the main structure of the LiDAR is composed of an emitting light module that irradiates near-infrared laser light to the surroundings, and a receiving light module that receives light reflected from objects. The time difference calculates the distance. However, LiDAR is easily affected by rain and fog and loses its accuracy, and it cannot identify the material of objects. As a result, LiDAR cannot accurately interpret signboards, billboards, or physical images. In addition, LiDAR adjustment is time-consuming and difficult to mass-produce, resulting in high cost and difficult to promote on a large scale. This is also its important shortcoming.

Therefore, how to achieve measurable and measurable surrounding scenery under the premise of controllable costs, clearly identify the surrounding scenery materials, and establish a 3D digital space perception system, so that the system can be used for vehicle monitoring, automatic driving, AI robots, Different industrial equipment such as sweeping robots, aerial drones, and multi-axis processing machines and instruments are the goals of those with ordinary knowledge in the field.

The main purpose of the present invention is to achieve quantifiable and measurable surrounding scenery and clearly identify the material of surrounding scenery under the premise of controllable cost, so as to establish a 3D digital space perception system.

Another object of the present invention is to assist different industrial equipment such as vehicle monitoring, AI robots, automatic driving, sweeping robots, aerial drones, and multi-axis processing machine instruments, so that the equipment can have 3D space recognition and 3D space multi-object monitoring. Improve the level of industrial unmanned monitoring.

In order to solve the above and other problems, the present invention provides a compound eye camera system, which includes a first lens, at least four second lenses, a storage unit and an image processing controller. The first lens has a fan-shaped first shooting area, the second lens is distributed on the periphery of the first lens, each second lens has a fan-shaped second shooting area, and the center of the first shooting area There is an angle between the shooting direction and the central shooting direction of the second shooting area, and the second shooting area partially overlaps with the first shooting area; A plurality of source image files; the image processing controller is used to identify the image features of at least one object under test in the source image files, and the image processing controller is used to analyze the multiple source image files taken at the same time point and A corresponding 3D primitive is generated, and a portable image file with 3D spatial information is analyzed through multiple 3D primitives generated at different time points.

In the above-mentioned compound eye camera system, a reference length is displayed in the source image files captured by the first shooting area, and the image processing controller uses the reference length as a scale to construct a 3D spatial digital model.

In the above-mentioned compound eye camera system, wherein the storage unit is coupled to the image processing controller, so that the 3D graphic element or the portable image file is transmitted and stored in the storage unit.

In the above-mentioned compound eye imaging system, at least one graphic element template is stored in the storage unit, and the graphic element template is a two-dimensional image of all or partial features of the object under test.

The above-mentioned compound eye camera system further includes at least one warning light coupled to the image processing controller, so that the image processing controller can calculate and analyze the source image files It is directly used to control the warning light after the object under test in it.

In order to solve the above and other problems, the present invention further provides a vehicle with a compound-eye camera system as described above, wherein multiple compound-eye camera systems are distributed on the roof, front edge, rear edge or both sides of the vehicle.

In order to solve the above and other problems, the present invention further provides an image processing method of a compound eye camera system, the compound eye camera system includes a first lens with a first shooting area and a second lens with a second shooting area, the image processing The method includes the following steps: Step A01: intercept source image files of multiple cameras and multiple time points; Step A02: identify and analyze the source image files to generate 3D primitives corresponding to at least one object under test ; Step A03: Calculate the distance of the object under test; Step A04: Calculate the 3D motion vector of the object under test; Step A05: Selectively compensate and correct the error of the 3D motion vector of the object under test; Step A06: Combine the Analyze the 3D primitive of the object under test and its corresponding 3D movement information to obtain a portable image file with 3D spatial information; and step A07: establish a 3D spatial digital model of the movement of the object under test, and convert the The portable graphic file is superimposed on the 3D space digitized model.

The image processing method of the above-mentioned compound eye camera system further includes step A08: sending out a deceleration warning signal, a brake warning signal, a steering prompt signal or a steering control signal.

The image processing method of the above-mentioned compound eye camera system, wherein, step A02 further includes the following sub-steps: Step A021: extract an image feature of an object under test from the source image files; step A022: extract these images Feature comparison of multiple graphic element templates from different viewing angles in a storage unit; step A023: generating a 3D graphic element of the object to be tested; in a further embodiment, the graphic element template in step A022 is all the features of the object to be measured or Two-dimensional image of local features.

The image processing method of the above-mentioned compound eye camera system, wherein, step A03 is further The following sub-steps are included: step A031: take a source image file at the same time point through the first lens or the second lens to measure the distance of the object under test; step A032: measure the object to be measured from the source image file The azimuth and elevation angle of the object to be measured; step A033 : calculating and confirming the spatial relationship of the object to be measured.

In the image processing method of the above-mentioned compound eye camera system, the distance measurement in step A031 is obtained by comparing the reference length of the vehicle in the source image file, or through the multiple of the reference length in the source image file In a further embodiment, the image processing controller uses the position of the center point of the outline appearance of the object to be measured to compare the multiple ruler marking line.

In the image processing method of the compound eye camera system described above, the distance measurement in step A031 is obtained through triangulation measurement of the viewing angles of the first lens and the second lens.

In the image processing method of the above-mentioned compound eye camera system, wherein the azimuth or elevation angle measurement in step A032 is measured through the azimuth ruler marking line or the elevation ruler marking line in the source image file; further In an embodiment, the image processing controller compares the marked line of the azimuth ruler or the marked line of the pitch ruler with the position of the shape center point of the outline appearance presented by the object under test.

As mentioned above in the image processing method of the compound eye camera system, step A04 further includes the following sub-steps: Step A041: Obtain the position of the object under test at different time points; Step A042: Calculate the motion vector of the object under test; Step A043: Continue Displays motion vectors for multiple time points.

In the image processing method of the compound eye camera system described above, step A05 further includes the following sub-steps: Step A051: Extract at least one turning feature of the object under test from the source image file; Step A052: Calibrate and generate the object under test The compensation correction vector of the object; step A053: assign The compensation correction vector is weighted to the object under test to correct the moving path of the object under test.

In this way, the compound eye camera system and its image processing method described in the present invention can achieve measurable and measurable surrounding scenery and clearly identify the surrounding scenery material under the premise of controllable cost, so as to establish a 3D digital space perception system, And then assist vehicle monitoring, automatic driving, AI robots, sweeping robots, aerial drones, multi-axis processing machine instruments and other industrial equipment, so that the equipment can have 3D space recognition and 3D space multi-object monitoring to improve industrial unmanned monitoring Level.

In order to further understand the features and technical content of the present invention, please refer to the following detailed description and drawings related to the present invention. However, the attached drawings are only for reference and illustration, and are not used to limit the present invention. In order to further understand the features and technical content of the present invention, please refer to the following detailed description and accompanying drawings of the present invention. However, the accompanying drawings are provided for reference and illustration only, and are not intended to limit the present invention.

50 : compound eye camera system

51 : first shot

52 : second shot

53 : The first shooting area

53A : Center shooting direction

54 : Second shooting area

54A : Center shooting direction

61 : source image file

62 : 3D digital space model

63 : image features

64 : 3D primitives

65 : Portable graphic file

66 : primitive template

71 : Reference length

55 : storage unit

56 : image processing controller

57 : warning lights

91 : vehicle

92 : object to be tested

72 : Marking line of azimuth ruler

74 : Marking line of multiple ruler gauge

75 : Spatial grid lines

h1 : vertical distance

d : lens distance

α, β, θ : angle

FIG. 1A is a schematic structural diagram of a compound eye camera system.

FIG. 1B is a diagram showing the usage state of the compound eye camera system applied to a vehicle.

FIG. 1C is a functional block diagram of the compound eye camera system.

2A to 2E are flowcharts of the image processing method of the compound eye camera system.

FIG. 3 is a schematic diagram of the image processing controller identifying image features in the source image file.

FIG. 4A is a schematic diagram of a vehicle installed with the compound eye camera system calculating a distance through the reference length.

FIG. 4B is a schematic diagram of calculating the distance by the image processing controller through the reference length of the vehicle in the source image file.

FIG. 5 is a schematic diagram showing the application of the triangulation method of the compound eye camera system.

FIG. 6A is a schematic diagram of measuring and calculating the azimuth angle of the object to be measured by the vehicle installed with the compound eye camera system.

FIG. 6B is a schematic diagram of calculating the azimuth angle of the object under test by the image processing controller through the source image file.

FIGS. 7A to 7C are schematic diagrams of situational awareness of surrounding scenes of a vehicle installed with the compound eye camera system in the 3D space digital model.

FIG. 8 is a schematic diagram of a demand scene for the image processing controller to perform residual image correction.

FIG. 9 is a functional block diagram of another embodiment of the compound eye camera system.

Please refer to Figures 1A to 1C at the same time. Figure 1A shows the schematic structure of the compound eye camera system, Figure 1B shows the application state of the compound eye camera system in vehicles, and Figure 1C shows the function of the compound eye camera system block diagram. As shown in the figure, a compound eye camera system 50 includes a first lens 51 , four second lenses 52 , a storage unit 55 and an image processing controller 56 . The first lens 51 has a fan-shaped first shooting area 53, a plurality of second lenses 52 are distributed around the first lens 51, and each second lens 52 has a fan-shaped second shooting area 54 The central shooting direction 53A of the first shooting area 53 and the central shooting direction 54A of the second shooting area 54 form an angle θ, and the second shooting area 54 and the first shooting area 53 are partially overlapped. Wherein, the first lens 51 and the second lens 52 of the compound eye camera system 50 are designed with arc surfaces, which can make the central shooting direction 53A of the first shooting area 53 and the central shooting direction 54A of the second shooting area 54 be incompatible. pointing in the same direction, so that the overall coverage area of the first shooting area 53 and the second shooting area 54 is larger, and more dead spots for shooting are avoided. The storage unit 55 is used for storing a plurality of source image files 61 captured by the first lens 51 or the second lens 52 . The storage unit 55 is coupled with the image processing controller 56, and the image processing controller 56 is used to analyze multiple source image files 61 shot at the same time point and generate a corresponding 3D primitive 64, and then through multiple 3D primitives 64 generated at different time points are analyzed to obtain a portable graphic file 65 with 3D spatial information. The storage unit 55 stores a plurality of primitive templates 66 , and the primitive templates 66 are two-dimensional images of local features or all features of a certain object under test 92 in the source image file 61 . The 3D graphic element 64 or the portable graphic file 65 is transmitted and stored in the storage unit 55 . Wherein, the source image file 61 is a file in image format captured by the first lens 51 or the second lens 52, and its format includes but not limited to JPG, JPEG, PSD, TIFF, PDF, BMP, EPS, PNG, GIF, PCX and other formats. The 3D graphic element 64 is a digitized file with multi-view visualization in 3D space, vectorizable and analyzable. The portable image file 65 is a portable electronic file format with 3D spatial information, which can be transmitted to the cloud or other machine equipment for storage, analysis and application by the network; the 3D spatial information includes the position of the 3D space Information (such as GPS positioning, Beidou positioning or other satellite system positioning), velocity vector information or acceleration vector information with direction.

The above-mentioned compound eye camera system 50 can be applied to various industrial equipment such as automobile monitoring assistance, automatic driving, sweeping robot, AI robot, aerial drone, multi-axis processing machine instrument, etc., so that these equipment can have 3D space recognition, 3D space Multi-object monitoring to improve the level of industrial unmanned monitoring. Hereinafter, taking the application of the compound eye camera system 50 to the vehicle 91 monitoring assistance as an example, To illustrate the image processing method of the compound eye camera system 50 of the present invention, as shown in FIG. The 3D three-dimensional space conditions around and above the vehicle 91 are monitored. In particular, the installation of the compound eye camera system 50 around the vehicle 91 can allow the vehicle 91 to establish a 3D space situation awareness around the vehicle 91, and know the size, outline, shape, and speed of objects within 200 meters around the vehicle 91. , acceleration, so that the vehicle 91 can respond to the surrounding traffic conditions in advance to prevent traffic accidents. In addition, the compound eye camera system 50 is installed on the top of the vehicle 91, which can be used to monitor objects above the vehicle 91. For example, if the vehicle 91 often travels in areas where rockfall or mountainous landslides frequently occur, the compound eye camera system 50 will Just can warn in advance and know rockfall, landslide, mountain walk, landslide, and then allow this vehicle 91 to dodge directly or stop. The value of the compound eye camera system 50 and its image processing method for the vehicle 91 lies in that it can enable the vehicle 91 to have situational awareness of the surrounding 3D space scenery, thereby improving the controllability and accuracy of the automatic driving of the vehicle.

In order to achieve the above-mentioned purpose of improving the controllability and precision of the automatic driving of the automobile, the present invention further provides an image processing method of the compound eye camera system 50 . Please refer to Fig. 2A ~ Fig. 2E at the same time, Fig. 2A ~ Fig. 2E are depicted as the flow chart of the image processing method of compound eye camera system 50; , source image files 61 at multiple time points (step A01), and then identify and analyze the source image files 61 through the image processing controller 56 to generate a 3D image corresponding to at least one object under test 92 element 64 (step A02). Here, as shown in FIG. 2B , the detailed implementation of step A02 includes the following sub-steps: extracting an image feature 63 of an object under test 92 from the source image files 61 (step A021), wherein, as As shown in FIG. 3 , from the source image files 61 captured by the compound eye camera system 50 , the image features 63 of a plurality of objects to be tested 92 in the source image files 61 can be identified through the image processing controller 56 . The DUT 92 can be So cars, trucks, locomotives, traffic signs, telephone poles, overpasses, road trees...etc. Different test objects 92 will have different image features 63, and the image features 63 are plane image features of the test object 92, which include but not limited to color features, texture features, gray value features, shape features, Spatial correspondence features, local features or global features...etc. Taking specific objects as an example, the image features 63 of road trees are leaves or trunks; the image features 63 of automobiles are the outline of the car shell and tires; . Therefore, by recognizing the image feature 63 of the object under test 92 , the compound eye camera system 50 can distinguish whether the object under test 92 in front of the vehicle 91 is a locomotive, a car or a pedestrian. Next, as shown in FIG. 2B , these image features 63 are compared with a plurality of primitive templates 66 of different viewing angles in the storage unit 55 (step A022), and it is judged whether the image features 63 of the object to be tested 92 are Attach the graphic element template 66; if it matches, then generate the 3D graphic element 64 (shown in FIG. 1C, see step A023) of the object under test 92, so that the 3D graphic element 64 is consistent with the image in the storage unit 55 Each meta-template 66 corresponds to the specific analyte 92 . Here, the primitive template 66 is a file formed by combining a plurality of two-dimensional images of the object 92 with different viewing angles (that is, a collection of complete images of the object 92 with different viewing angles). ), which can be a built-in or image-based template file captured from big data; for example, the graphic element template 66 can be a specific object (such as a car, a locomotive, a truck, a traffic signal, a road tree, etc. ) Comparable image files at different viewing angles, the purpose of which is to provide the compound eye camera system 50 with a reference comparison of image features 63 from multiple different viewing angles. Therefore, only a few source image files 61 of specific angles, after comparison by the image processing controller 56 , can allow the compound eye camera system 50 to identify and confirm what kind of object the object under test 92 is. Even the type and style of car. Furthermore, the primitive template 66 can be a local feature of a certain object 92 in the source image file 61, so afterimage comparison and afterimage compensation can be performed through the primitive template 66 of the local feature ; Please also Referring to FIG. 8 , FIG. 8 is a schematic diagram of a demand scene for the image processing controller to perform residual image correction. As shown in FIG. 8 , in the first photographing area 53 of the compound eye camera system 50 , the pedestrian to be tested 92 is partially blocked by the van ahead of the image, causing the image processing controller 56 to be unable to fully identify the pedestrian. The analyte92. At this time, through the creation of afterimages and afterimages (that is, images of local features of the pedestrian object to be tested 92 ) of the primitive template 66, the image processing controller 56 can be obtained and obtained through feature comparison. What is the object that is blocked in the first shooting area 53? In this way, the compound eye camera system 50 can know in advance what objects are behind the occluded area, so as to achieve the functions of advance prediction and early warning.

It should be supplemented here that the above-mentioned analysis of the source image file 61 , or the identification and comparison of the image features 63 , are all achieved through image matching. The image matching refers to identifying homogeneous features or homogeneous features between two or more images through a certain matching algorithm, such as comparing the window of the same size in the target area and the search area in two-dimensional image matching For the correlation coefficient, the center point of the window corresponding to the maximum correlation coefficient in the search area is taken as the homogeneity or homogeneity feature, that is, a statistical method is used to find the correlation matching degree between signals. Its essence is to use matching criteria to achieve the best search effect under the condition that the basic primitives are related and similar. In general, image matching can be divided into grayscale-based matching and feature-based matching.

Next, after confirming the object to be tested 92 and generating its corresponding 3D primitive 64, the distance of the object to be tested 92 can be calculated (step A03); wherein, the distance of the object to be tested 92 is calculated, as shown in FIG. As shown in 2C, the source image file 61 of "the same time point" can be shot through the first lens 51 or the second lens 52 to measure the distance of the object 92 to be measured (step A031), and then the source image Measure the azimuth and elevation angles of the object under test 92 in the image file 61 (step A032 ), so that the spatial relationship of the object under test 92 can be calculated and confirmed (step A033 ). Wherein, as shown in Fig. 4A and Fig. 4B, this step The distance measurement in step A031 can be through the reference length 71 of the vehicle 91 in the source image file 61, using the reference length 71 to compare and measure the truck, the vehicle under test 92 and the installation in the source image file 61. The relative distance of the vehicle 91 of the compound eye camera system 50; that is, at one time of reference length 71, two times of reference length 71, three times of reference length 71, and four times of reference length 71 in Fig. 4A, all will be shown in Fig. 4B In the source image file 61 of the source image, the multiple ruler marking line 74 of the reference length 71 is displayed or marked, so that the image processing controller 56 of the compound eye camera system 50 can compare or calculate the distance of the object 92 of the truck or automobile . Wherein, as shown in Fig. 4A and Fig. 4B, the reference length 71 of the vehicle 91 is preferably the distance from the installation point of the compound eye camera system 50 to the front end of the vehicle 91; The reference length 71 and its multiples of the vehicle 91 in the captured source image file 61 can also be achieved through the software built-in scale (built-in standard fixed length), or on the lens surface of the first lens 51 Achieved by engraving physical graduation marks. In addition, as shown in FIG. 4B , viewed from the source image file 61, if the area of the object under test 92 is large (maybe the actual volume of the object under test 92 is large, or it is relatively close to the device If the vehicle 91 of the compound eye camera system 50 crosses a plurality of multiplier scale marking lines 74, the image processing controller 56 will compare the position of the shape center point of the outline appearance of the object 92 presented by the object under test 92. The marking line 74 of the multiple ruler is used for judging the distance of the object 92 to be measured.

In addition to using the reference length 71 of the vehicle 91 to calculate, the distance of the truck or the vehicle to be measured 92 can also be calculated by triangulation. As shown in FIG. 5 , the lens distance d between the first lens 51 and the second lens 52 of the compound eye camera system 50, and the vertical distance h1 between a locomotive-like object 92 to be tested and the compound eye camera system 50, can be obtained through trigonometric functions and trigonometric functions. The vertical distance h1=d*[(sinα*sinβ)/sin(α+β)] can be obtained by positioning measurement method. That is to say, the lens distance d between the first lens 51 and the second lens 52 is known, and then the angle is observed and measured through the compound eye camera system 50 The size of α and β can be used to obtain the vertical distance h1 through calculation. Wherein, the locomotive-shaped object 92 to be tested can of course also be a car, a truck, a pedestrian, a road tree or a traffic signal...etc.

The azimuth or elevation angle measurement in step A032 can be measured through the azimuth ruler marking line 72 or the elevation ruler marking line in the source image file 61; for example, as shown in FIGS. 6A and 6B , from When the compound eye camera system 50 looks toward the front of the vehicle 91, the frame of the source image file 61 can be divided into a plurality of regions by the azimuth ruler marking line 72. From the position of the object to be measured 92, it can be obtained The corresponding azimuth angle (Azimuth angle) of the object under test 92 relative to the compound eye camera system 50 is known. If the area of the object-to-be-tested 92 presented in the source image file 61 is larger (it may be that the actual volume of the object-to-be-tested 92 is large, or it is relatively close to the vehicle 91 on which the compound eye camera system 50 is installed, such as As shown in FIG. 6B ) and straddles a plurality of azimuth ruler marking lines 72 or pitch ruler marking lines, then the image processing controller 56 will use the shape center point of the outline appearance presented by the object under test 92 as the Azimuth/pitch angle judgment of the object 92 to be measured. In the same way, the image processing controller 56 can also divide the source image file 61 into a plurality of regions with different pitch angles through a plurality of pitch ruler marking lines, and then judge the pitch of the object 92 to be measured. Location. In this way, the image processing controller 56 is used to analyze the distance, orientation, and pitch of the object under test 92, and the corresponding spatial relationship between the object under test 92 and the vehicle 91 is known and confirmed according to the principle of spherical coordinates. The execution of step A033 is completed.

Next, calculate the 3D motion vector of the object under test 92 (step A04), wherein the position of the object under test 92 at different time points is obtained through the aforementioned step A03 (step A041), and then calculate the movement of the object under test 92 vector (step A042), that is, to continuously display multiple moving vectors at different time points (step A043). Among them, in step A03, the multiple source image files 61 obtained from "same time point, different shots" are aimed at passing through multiple first shots 51 and second shots at different positions. The difference in the spatial position of the two lenses 52 is used to refine the position calculation of the distant object 92 to be measured; its essence is to obtain the "positioning" of the object 92 to be measured multiple times through multiple different lens positions, and through multiple calculations. achieve the goal of improving accuracy. And step A04 is to use the source image files 61 captured at "different time points" to obtain the moving track and moving vector of a certain object under test 92 (∵ the position change at different time points, that is, the object under test 92 motion vector).

Next, selectively compensate and correct the error of the 3D motion vector of the object under test 92 (step A05). Steering characteristics (step A051 ), the steering characteristics include but are not limited to tire steering of a car, head steering of pedestrians on the road, or a staggered angle between the car body and the road lane. These turning characteristics mean that the cars or pedestrians around the vehicle 91 on which the compound eye camera system 50 is installed have strong steering intentions, may change their direction of travel significantly, turn suddenly, change lanes suddenly, and cause the vehicle 91 to chase. crash accident. Therefore, if the compound-eye camera system 50 can predict the turning intentions of surrounding cars and pedestrians in advance, corresponding measures can be taken in advance to reduce the probability of the vehicle 91 colliding with the surrounding object 92 to be tested. After confirming that the surrounding cars and pedestrians of the vehicle 91 intend to turn, the compensation and correction vector of the object under test 92 can be calibrated and generated (step A052), and weights are assigned to the compensation and correction vector of the object under test 92 to correct the The moving path of the object under test 92 (step A053 ), that is to say, is used to correct the moving vector generated in the aforementioned step A04 , so as to predict in advance the sudden turning and lane changing of surrounding pedestrians and cars. It is specifically explained here that the selective execution of step A05 means that it may or may not be executed. In addition, as shown in FIG. 2A , if it is found that the compensation correction vector is too large after calculation in step A05 , it may be fed back to step A04 to re-execute the calculation of the motion vector of the object under test 92 in step A04 .

At this moment, the compound eye camera system 50 has completed the photographing of surrounding automobiles, pedestrians or traffic signals. For the calculation of the distance and movement vector of the vehicle 91, the image processing controller 56 is used to combine the 3D primitive 64 of the object 92 and its corresponding 3D movement information, and then analyze the 3D spatial information. Portable graphic file 65 (step A06); Next, establish the 3D space digital model 62 of the movement of the object 92 to be tested, and superimpose the portable graphic file 65 on the 3D space digital model 62 (Step A07 ), so that the image processing controller 56 of the compound-eye camera system 50 can superimpose all scenes such as people, vehicles and traffic signs around the vehicle 91 on the 3D space digitized model 62 . Please refer to Figures /A~7C at the same time. Figures 7A~7C are schematic diagrams of situational awareness of the surrounding scenes of the vehicle installed with the compound eye camera system in the 3D space digital model; as shown in Figures 7A and 7B, The compound eye camera system 50 can detect possible objects 92 such as cars, people, road trees, traffic signs, etc., and turn them into corresponding 3D primitives 64, and perceive their 3D spatial positions, moving vectors, and accelerations, and finally Then the 3D graphic element 64 is converted into a portable graphic file 65 with 3D space information and superimposed on a 3D space digital model 62, so that the compound eye camera system 50 establishes the 3D space situation awareness around the vehicle 91, The 3D spatial depth of field estimation can detect the size, speed, and acceleration of objects within 200 meters around it, so that the vehicle 91 has a powerful ability to monitor surrounding objects. As shown in FIG. 7A , the vehicle 91 installed with the compound eye camera system 50 can detect and perceive the locomotive to be tested 92 on the left rear and the lane marking line on the road, and then judge whether to avoid dodge or accelerate away; As shown in FIG. 7B , the vehicle 91 installed with the compound eye camera system 50 can detect and perceive a plurality of automobile objects to be tested 92 around it, and let the image processing controller 56 establish the 3D space around the vehicle 91 The digitized model 62 and the spatial coordinates make the 3D spatial digitized model 62 have a virtual spatial grid line 75, so that the compound eye camera system 50 can know and perceive the relative coordinates of all objects to be measured 92 around it, thereby using To allow the image processing controller 56 to plan the best route to travel, avoid or even go around, and even decide whether to slow down, stop and wait, or overtake. Finally, please also refer to Figure 7C, the The vehicle 91 can send a deceleration warning signal, a brake warning signal, a steering prompt signal or a steering control signal according to the image monitoring and judgment of the compound eye camera system 50 or the image processing controller 56 (step A08), so that the vehicle 91 has Functions of autonomous control and automatic driving. As shown in the left half of Figure 7C, the compound eye camera system 50 can also integrate a map system (such as Google Maps, Baidu Maps, Gaode Maps, etc.) to know the road direction of tens of kilometers around the vehicle 91, and at the same time A plurality of spatial grid lines 75 generated by the image processing controller 56 are displayed. Also, after the integration, the compound eye camera system 50 can present the road direction, road planning, detected and perceived surrounding scenery and the object to be measured 92 of the map system together on the 3D space digital model 62 In this way, as shown in the right half of Figure 7C, the compound eye camera system 50 of the present invention can achieve: the perception and prediction of the coordinates, relative distance, and motion vector of the object to be measured 92, and early detection of possible impacts , Collision warning.

Please refer to FIG. 9 , which is a functional block diagram of another embodiment of the compound eye camera system. As shown in Figure 9, the compound eye camera system 50 of the present invention can further increase at least one warning light 57, and the warning light 57 is coupled to the image processing controller 56, so the image processing controller 56 can be used to control The warning lamp 57 lights up, goes out or flickers. As shown in Figure 7A again, when the locomotive to be tested 92 at the left rear is close to the vehicle 91, when the distance of the locomotive to be tested 92 is too short through the calculation and analysis of the image processing controller 56, the image processing control The device 56 can autonomously drive the warning light 57 to flash (that is, without the need for the driver of the vehicle 91 to control), reminding the locomotive to be tested 92 to keep the distance between vehicles. That is to say, after the image processing controller 56 of the compound eye camera system 50 calculates and analyzes the objects 92 in the source image files 61, the image processing controller 56 can directly control the warning light 57 to issue a warning. light. Therefore, the function of step A08 is to control Control, drive this warning light 57 to send deceleration early warning signal, brake early warning signal, turn to prompt signal, reach the purpose of preventing collision.

Thereby, the compound-eye camera system 50 of the present invention, the vehicle 91 using the compound-eye camera system 50 and its image processing method can not use expensive equipment such as laser radar, infrared radar, lidar, etc., but under the premise of controllable cost , so that the surrounding scenery can be measured and measured, the material of the surrounding scenery can be clearly identified, and a 3D digital space perception system is established, so that the system can be used for car monitoring, AI robots, automatic driving, sweeping robots, aerial drones, and multi-axis processing Different industrial equipment such as machines and instruments. Therefore, it has great potential for commercial application.

The present invention is described above with examples, but they are not intended to limit the scope of patent rights claimed by the present invention. The scope of its patent protection shall depend on the scope of the appended patent application and its equivalent fields. All changes or modifications made by those with common knowledge in the field without departing from the spirit or scope of this patent belong to the equivalent change or design completed under the spirit disclosed by the present invention, and should be included in the scope of the following patent application Inside.

Step A01~Step A08

Claims (18)

A compound eye camera system (50), comprising: A first lens (51), which has a fan-shaped first shooting area (53); At least four second lenses (52), distributed on the periphery of the first lens (51), each second lens (52) has a fan-shaped second shooting area (54), the first shooting area ( 53) the central shooting direction (53A) and the central shooting direction (54A) of the second shooting area (54) have an angle (θ), and the second shooting area (54) and the first shooting area (53) is partially overlapping; A storage unit (55), used for storing a plurality of source image files (61) captured by the first lens (51) or the second lens (52); and An image processing controller (56), used to identify the image features (63) of at least one object under test (92) in the source image files (61), the image processing controller (56) used to analyze the same A plurality of source image files (61) taken at time points and generate a corresponding 3D primitive (64), and then analyze a 3D spatial information through the 3D primitives (64) generated at different time points Portable image files (65). The compound eye camera system (50) as described in claim 1, wherein, a reference length (71) is displayed in the source image file (61) captured by the first shooting area (53), and the image processing controller ( 56) Using the reference length (71) as a scale to construct a 3D spatial digital model (62). The compound eye camera system (50) according to claim 1, wherein, the storage unit (55) is coupled with the image processing controller (56), so that the 3D graphic element (64) or the portable graphic file (65) transfer and store to the storage unit (55). The compound eye camera system (50) according to claim 1, wherein at least one graphic element template (66) is stored in the storage unit (55), and the graphic element template (66) is the object under test (92) Two-dimensional images of all features or partial features. The compound eye camera system (50) according to claim 1, further comprising at least one warning light (57) coupled to the image processing controller (56), so that the image processing controller (56) calculates the solution After analyzing the objects to be detected (92) in the source image files (61), they are directly used to control the warning light (57). A vehicle (91) using a plurality of compound eye camera systems (50) as claimed in claim 1, wherein the plurality of compound eye camera systems (50) are distributed on the roof of the vehicle (91), the front edge of the vehicle body, the vehicle body trailing edge or sides. An image processing method for a compound eye camera system (50), the compound eye camera system (50) comprising a first lens (51) having a first shooting area (53) and a second lens having a second shooting area (54) (52), the image processing method includes the following steps: Step A01: Intercept source image files of multiple shots and multiple time points (61); Step A02: identifying and analyzing the source image files (61) to generate 3D primitives (64) corresponding to at least one object under test (92); Step A03: Calculate the distance of the object under test (92); Step A04: calculating the 3D motion vector of the object to be measured (92); Step A05: selectively compensating and correcting the error of the 3D motion vector of the object under test (92); Step A06: Combining the 3D primitive (64) of the object under test (92) and its corresponding 3D movement information to analyze a portable graphic file (65) with 3D spatial information; and Step A07: Establish a 3D spatial digital model (62) of movement of the object under test (92), and superimpose the portable image file (65) on the 3D spatial digital model (62). The image processing method of the compound eye camera system (50) according to claim 7, further comprising step A08: sending out a deceleration warning signal, a brake warning signal, a steering prompt signal or a steering control signal. The image processing method of the compound eye camera system (50) as described in Claim 7, wherein step A02 further includes the following sub-steps: Step A021: extracting image features (63) of an object under test (92) from the source image files (61); Step A022: Compare these image features (63) with a plurality of different viewing angles in a storage unit (55) primitivetemplate(66); Step A023: Generate a 3D primitive (64) of the object under test (92). The image processing method of the compound eye camera system (50) according to Claim 9, wherein the primitive template (66) in step A022 is a two-dimensional image of all or local features of the object under test (92). The image processing method of the compound eye camera system (50) as described in Claim 7, wherein step A03 further includes the following sub-steps: Step A031: shooting the source image file (61) at the same time point through the first lens (51) or the second lens (52) to measure the distance of the object under test (92); Step A032: Measure the azimuth and elevation angles of the object under test (92) from the source image file (61); Step A033: Calculate and confirm the spatial relationship of the object under test (92). The image processing method of the compound eye camera system (50) as described in claim 11, wherein the distance measurement in step A031 is achieved by comparing a reference length (71) of the vehicle (91) in the source image file (61) obtained, or measured through a ruler marking line (74) that is a multiple of a reference length (71) in the source image file (61). The image processing method of the compound eye camera system (50) as described in Claim 12, wherein the image processing controller (56) compares the multiplier with the position of the center point of the outline appearance of the object under test (92) Ruler marking line (74). The image processing method of the compound eye camera system (50) as described in claim 11, wherein the distance measurement in step A031 is the triangulation method measurement of the viewing angles of the first lens (51) and the second lens (52). measured. The image processing method of the compound eye camera system (50) as described in claim 11, wherein the azimuth angle or elevation angle measurement in step A032 is marked through the azimuth ruler in the source image file (61) Line (72) or pitch gauge mark line and measure. The image processing method of the compound eye camera system (50) as described in Claim 15, wherein the image processing controller (56) uses the position of the center point of the outline appearance of the object (92) to compare the orientation Ruler marking line (72) or this pitch ruler rule marking line. The image processing method of the compound eye camera system (50) as described in Claim 7, wherein step A04 further includes the following sub-steps: Step A041: obtaining the position of the object to be tested (92) at different time points; Step A042: Calculating the motion vector of the object under test (92); Step A043: Continuously display motion vectors at multiple time points. The image processing method of the compound eye camera system (50) as described in Claim 7, wherein step A05 further includes the following sub-steps: Step A051: extracting at least one turning feature of the object under test (92) from the source image file (61); Step A052: Calibrate and generate a compensation correction vector for the object under test (92); Step A053: assigning weights to the compensation correction vector of the object under test (92), so as to correct the moving path of the object under test (92).

Images