JP2023047328A - Image processing system, image processing method, imaging apparatus, optical system, and computer program - Google Patents

Abstract

To provide an image processing system that allows easy adjustment of the location and posture of a camera or other devices.SOLUTION: The image processing system includes: image acquisition means for acquiring an image signal formed by imaging means for capturing an optical image having a low distortion area and a high distortion area; characteristic information acquisition means for acquiring characteristic information on characteristics of the optical image; and display signal generation means for forming information indicating a boundary between the low distortion area and the high distortion area on the basis of the characteristic information acquired by the characteristic information acquisition means and generating a display signal to be superimposed on the image signal.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing system, an image processing method, an imaging device, an optical system, and a computer program capable of image recognition.

In recent years, there has been a demand to replace rearview mirrors (rearview mirrors) mounted on vehicles with electronic rearview mirrors. For example, in Patent Document 1, an image capturing means having an image capturing range outside the vehicle and a display means inside the vehicle are configured. An electronic rearview mirror system is disclosed that allows for confirmation of what is happening behind.

On the other hand, there is a rear confirmation system that enables the driver to check the blind spots behind the vehicle when the vehicle is backing up. Patent document 2 discloses a rear confirmation system for enabling the driver to check the blind spot behind the vehicle when backing up by installing a camera so as to image the rear of the vehicle and displaying the captured image in the vehicle interior. is disclosed.

JP 2010-95202 A JP-A-2004-345554

A camera as an imaging unit that captures the image for the electronic rearview mirror described above is required to have a high resolution so that the driver can more precisely confirm the situation in the rear relatively far away. On the other hand, the camera for the rear confirmation system must take a wider range of images to confirm safety in a wider range, including the blind spots behind the vehicle and the rear sides, in order to avoid collisions when reversing. is required.

Therefore, when an electronic rearview mirror system and a rearview confirmation system are installed in a vehicle at the same time, if the camera for the electronic rearview mirror system and the camera for the rearview confirmation system are installed separately, the in-vehicle image processing system becomes complicated. Such a problem also occurs, for example, in an automatic driving system that automatically drives a vehicle by arranging a plurality of cameras to photograph the surroundings of the vehicle.

On the other hand, the number of cameras installed in the vehicle can be reduced by adopting a camera using a special ultra-wide-angle lens, for example. However, although a wide angle of view can be obtained, there are high-resolution areas (low-distortion areas) and low-resolution areas (high-distortion areas). It is difficult to tell whether There was a problem that adjustment for installation of the camera was very difficult.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image processing system that facilitates adjustment of at least one of the camera installation position and orientation.

In order to achieve the above object, an image processing system according to one aspect of the present invention includes:
an image acquisition means for acquiring an image signal formed by an imaging means for imaging an optical image having a low distortion area and a high distortion area;
a characteristic information obtaining means for obtaining characteristic information relating to the characteristics of the optical image;
forming information indicating a boundary between the low distortion region and the high distortion region based on the characteristic information obtained by the characteristic information obtaining means, and generating a display signal for superimposing the information indicating the boundary on the image signal; a display signal generating means for
characterized by having

According to the present invention, it is possible to realize an image processing system that facilitates adjustment of at least one of the camera installation position and orientation.

4 is a diagram for explaining the positional relationship between a vehicle and an imaging unit in the image processing system of Embodiment 1; FIG. 4A and 4B are diagrams for explaining optical characteristics of an imaging unit according to the first embodiment; FIG. 1 is a functional block diagram for explaining the configuration of an image processing system according to Embodiment 1; FIG. 4 is a flow chart for explaining a processing flow of a camera processing unit according to the first embodiment; FIG. 5 is a flowchart for explaining processing continued from FIG. 4; FIG. FIG. 5 is a flowchart for explaining branch processing from FIG. 4; FIG. FIG. 7 is a flowchart for explaining branch processing from FIG. 6; FIG. FIG. 5 is a flowchart for explaining another branching process from FIG. 4; FIG. FIG. 10 is a diagram for explaining an example of superimposition of the display state of an image and the display resolution boundary; FIG. 11 is a diagram for explaining another example of a display resolution boundary 93 superimposed on an image; FIG. 10 is a diagram showing an example of a reference frame and the like displayed on an adjustment screen according to a flag F; FIG. 4 is a diagram showing an example of the installation position of a reference frame;

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of the present invention will be described below using examples with reference to the accompanying drawings. In each figure, the same members or elements are denoted by the same reference numerals, and overlapping descriptions are omitted or simplified.

[Embodiment 1]
In Embodiment 1, an improved method for achieving both a high-definition display for an electronic rearview mirror and a display for checking the surroundings of a vehicle, such as a wide area behind the vehicle, with a small number of cameras will be described. FIG. 1 is a diagram for explaining the positional relationship between a vehicle and an imaging unit in the image processing system of Embodiment 1. FIG.

In this embodiment, as shown in FIG. 1, camera units 11, 12, 13, and 14 are installed on the front, right, rear, and left sides of a vehicle 1, such as an automobile, which is a moving object. Although four camera units are provided in this embodiment, the number of camera units is not limited to four, and at least one or more camera units may be provided.

The camera units 11 to 14 are installed so as to cover the front, right, rear, and left sides of the vehicle 1 as a moving body, respectively, and function as imaging units. The camera units 11 to 14 have substantially the same configuration, each having an imaging device for capturing an optical image and an optical system for forming an optical image on the light receiving surface of the imaging device.

For example, the optical axes of the optical systems of the camera units 11 and 13 are installed so as to be substantially horizontal, and the optical axes of the optical systems of the camera units 12 and 14 are installed so as to point slightly below the horizontal. there is

In addition, the optical systems of the camera units 11 to 14 used in this embodiment can obtain high-definition images at narrow angles of view around the optical axis, and obtain low-resolution captured images at wide angles of view. is configured so that Note that 11a to 14a are imaging field angles capable of imaging a high-resolution, low-distortion image, and 11b to 14b are imaging field angles capable of imaging a low-resolution, high-distortion image.

Optical systems of the camera units 11 to 14 in this embodiment will be described with reference to FIG. Although the characteristics of the optical systems of the camera units 11 to 14 may not be the same, in the present embodiment, the optical systems of the camera units 11 to 14 have substantially the same characteristics. An optical system possessed by is exemplarily described.

2A and 2B are diagrams for explaining the optical characteristics of the imaging unit according to Embodiment 1 of the present invention. FIG. 2A shows the optical system of the camera unit 11 according to this embodiment. 2 is a diagram showing contour lines of the image height y at each half angle of view on the light receiving surface of the imaging element.

FIG. 2B is a diagram showing the projection characteristics representing the relationship between the image height y and the half angle of view θ of the optical system of the camera unit 11 in this embodiment. In FIG. 2B, the half angle of view (the angle formed by the optical axis and the incident light beam) θ is taken as the horizontal axis, and the imaging height (image height) y on the sensor surface (image plane) of the camera unit 11. is shown as the vertical axis.

As shown in FIG. 2B, the optical system of the camera unit 11 according to the present embodiment has different projection characteristics y(θ) between a region smaller than a predetermined half angle of view θa and a region larger than the half angle of view θa. is configured as Therefore, when the amount of increase in the image height y with respect to the half angle of view θ per unit is referred to as the resolution, the resolution differs depending on the area.

It can also be said that this local resolution is represented by the differential value dy(θ)/dθ of the projection characteristic y(θ) at the half angle of view θ. That is, it can be said that the larger the slope of the projection characteristic y(θ) in FIG. 2B, the higher the resolution. Further, it can be said that the larger the interval of the image height y at each half angle of view of the contour lines in FIG. 2A, the higher the resolution.

In the present embodiment, when the half angle of view θ is less than the predetermined half angle of view θa, the area near the center formed on the sensor surface is the high resolution area 10a, and the half angle of view θ is equal to or greater than the predetermined half angle of view θa. is called a low-resolution area 10b. In this embodiment, the circle of the boundary between the high-resolution area 10a and the low-resolution area 10b is called the resolution boundary, and the boundary image on the display screen corresponding to the resolution boundary is called the display resolution boundary or simply the boundary image. Note that the boundary image (display resolution boundary) displayed on the display screen does not have to be circular. For convenience, it may be rectangular or the like.

In this embodiment, the high-resolution area 10a is a low-distortion area with relatively little distortion, and the low-resolution area 10b is a high-distortion area with relatively many distortions. Therefore, in the present embodiment, the high-resolution area and the low-resolution area correspond to the low-distortion area and the high-distortion area, respectively, and the high-resolution area and the low-resolution area can be called the low-distortion area and the high-distortion area, respectively. be. Conversely, the low distortion area and the high distortion area are sometimes called the high resolution area and the low resolution area, respectively.

The optical system of the camera unit 11 in this embodiment is configured such that the projection characteristic y(θ) is greater than f×θ in the high resolution area (low distortion area) 10a (f is the camera unit 11 focal length of the optical system with). Also, the projection characteristic y(θ) in the high resolution area (low distortion area) is set to be different from the projection characteristic in the low resolution area (high distortion area).

Further, when θmax is the maximum half angle of view of the optical system of the camera unit 11, the ratio θa/θmax between θa and θmax is desirably equal to or greater than a predetermined lower limit value, for example, 0.05 as the predetermined lower limit value. 15 to 0.16 is desirable.

Also, the ratio θa/θmax of θa to θmax is desirably equal to or less than a predetermined upper limit value, such as 0.25 to 0.35. For example, when θmax is 90°, the predetermined lower limit is 0.15, and the predetermined upper limit is 0.35, θa is desirably determined within the range of 13.5 to 31.5°. Furthermore, the optical system of the camera unit 11 is configured such that its projection characteristic y(θ) also satisfies the following equation (1).

f is the focal length of the optical system of the camera unit 11 as described above, and A is a predetermined constant. By setting the lower limit to 1, the central resolution can be made higher than that of an orthographic projection (y = f x sin θ) fisheye lens having the same maximum imaging height, and by setting the upper limit to A, the fisheye lens Good optical performance can be maintained while obtaining an equivalent angle of view. The predetermined constant A may be determined in consideration of the resolution balance between the high-resolution area and the low-resolution area, and is preferably 1.4 to 1.9.

By configuring the optical system as described above, high resolution can be obtained in the high resolution region 10a, while in the low resolution region 10b, the increase in the image height y with respect to the half angle of view θ per unit is reduced, It becomes possible to image a wider angle of view. Therefore, it is possible to obtain a high resolution in the high resolution area 10a while making the imaging range as wide as a fisheye lens.

Furthermore, in the present embodiment, in the high resolution area (low distortion area), the central projection method (y=f×tan θ) or the equidistant projection method (y=f× Since the characteristics are close to θ), optical distortion is small, and fine display is possible. Therefore, it is possible to obtain a natural perspective when viewing surrounding vehicles such as a preceding vehicle and a following vehicle, and to obtain good visibility by suppressing deterioration of image quality.

It should be noted that the present invention is not limited to the projection characteristics shown in FIG. 2 because the same effect can be obtained if the projection characteristics y(θ) satisfy the condition of Equation 1 above. In this embodiment, the optical system having the projection characteristic y(θ) that satisfies the condition of Equation 1 may be called a different angle of view lens.

The high resolution area 10a of the optical system of each of the camera units 11 to 14 corresponds to the imaging field angles 11a to 14a, and the low resolution area 10b corresponds to the imaging field angles 11b to 14b.

For example, the camera units 11, 12, and 14 attached to the front (front), left side, and right side of the moving object are not lenses with different angles of view as shown in FIG. An optical system having a projection characteristic y(θ) that satisfies condition 2 can also be used. In this optical system, the high resolution area and the low resolution area are roughly opposite compared to the different angle of view lens.

That is, the above-described different angle of view lens has a high resolution area 10a in the central area, and a low resolution area 10b in the outer area where the half angle of view θ is equal to or greater than the predetermined half angle of view θa. However, in an optical system that satisfies the condition of Expression 2, the central area is the low resolution area 10b, and the outer area where the half angle of view θ is equal to or greater than the predetermined half angle of view θa is the high resolution area 10a. Such an optical system is sometimes called a reverse different angle of view lens.
0.2<2×f×tan(θmax/2)/y(θmax)<0.92 (Equation 2)

In this manner, the camera unit having the different angle of view lens and the camera unit having the reverse different angle of view lens may be installed at different positions on the moving body. That is, at least one of the camera units has a focal length of its optical system of f, a half angle of view of θ, an image height of y on the image plane, and a projection characteristic representing the relationship between the image height y and the half angle of view θ. When y([theta]) and [theta]max are the maximum half angle of view of the optical system, the above equation 2 may be satisfied.

Next, the configuration of the image processing system according to this embodiment will be described with reference to FIG. FIG. 3 is a functional block diagram for explaining the configuration of the image processing system according to the first embodiment.

In FIG. 3, the image processing system 100 is mounted on a vehicle 1 as a moving body, and imaging units 21 to 24 and camera processing units 31 to 34 are arranged in housings of camera units 11 to 14, respectively.

The image capturing units 21 to 24 have different field angle lenses 21c to 24c and image capturing elements 21d to 24d such as CMOS image sensors and CCD image sensors, respectively. Here, the imaging units 21 to 24 each function as an image acquisition unit, and each image acquisition unit acquires an image signal from the imaging unit that captures an optical image having a low distortion area and a high distortion area. . In addition, the present embodiment has a plurality of image acquisition units arranged at different positions.

The different view angle lenses 21c to 24c as an optical system are composed of one or more optical lenses, have a projection characteristic y(θ) that satisfies the condition of Equation 1, and have a low distortion area and a high distortion area. The optical images are formed on the light receiving surfaces of the imaging elements 21d to 24d, respectively. The imaging devices 21d to 24d function as imaging units, photoelectrically convert optical images, and output imaging signals. For example, RGB color filters are arranged for each pixel on the light receiving surfaces of the imaging elements 21d to 24d. The arrangement of RGB is, for example, a Bayer arrangement.

It should be noted that an image acquisition unit having a reverse different angle of view lens can be arranged in place of the image acquisition unit having a different angle of view lens on the front or side. The arrangement position of the image acquisition unit having the different angle of view lens is not limited to this. At least two image acquisition units of the plurality of image acquisition units (an image acquisition unit having a different angle of view lens and an image acquisition unit having a reverse different angle of view lens) are arranged so that the photographing ranges of at least two of the image acquisition units overlap each other. It is assumed that there is

Therefore, from the imaging device, signals of R, G, R, and G are sequentially output from a predetermined row, and signals of G, B, G, and B are sequentially output from the adjacent row in accordance with the Bayer arrangement. is configured as follows.

Camera processing units 31 to 34 are housed in the same housings of the camera units 11 to 14 together with the imaging units 21 to 24, respectively, and process imaging signals output from the imaging units 21 to 24, respectively. In FIG. 3, the details of the imaging unit 24 and the camera processing unit 34 and their wiring are omitted for the sake of convenience.

The camera processing units 31 to 34 have image processing units 31a to 34a, recognition units 31b to 34b, and camera information units 31c to 34c, respectively. The image processing units 31a to 34a process the imaging signals output from the imaging units 21 to 24, respectively. Note that part or all of the camera processing section 31 may be performed by stacked signal processing sections in the imaging elements 21d to 24d.

Specifically, the image processing units 31a to 34a debayer the image data input from the imaging units 21 to 24 according to the Bayer array, and convert the image data into RGB raster format image data. Furthermore, various correction processes such as white balance adjustment, gain/offset adjustment, gamma processing, color matrix processing, and reversible compression processing are performed. However, irreversible compression processing is not performed, and a so-called RAW image signal is formed.

The recognizing units 31b to 34b perform image recognition of predetermined objects (for example, automobiles, people, obstacles, etc.) from the RAW image signals that have not undergone distortion correction and have been image-processed by the image processing units 31a to 34a. That is, the recognizing units 31b to 34b perform image recognition in the state of the RAW image signal without performing distortion correction on the image signal corresponding to the low distortion area, and output the first image recognition result. That is, the recognizing units 31b to 34b of the present embodiment perform image recognition processing on at least the RAW image signal obtained from the high-resolution area 10a to recognize a predetermined target object.

At this time, the recognition units 31b to 34b may also perform image recognition processing on the RAW image signal obtained from the low resolution area 10b. However, since the distortion of the RAW image signal is not corrected, the distortion of the peripheral image of the different angle of view lens is large, and the reliability of recognition is lowered.

Alternatively, the recognizing units 31b to 34b may cut out the RAW image signal obtained from the high resolution area 10a and perform image recognition processing only on the RAW image signal obtained from the high resolution area 10a.

It should be noted that the area to be cut out at that time is desirably rectangular for image recognition processing. The rectangular area to be cut out may be only a part of the high-resolution area 10a (for example, a rectangle inscribed in the high-resolution area 10a), or may be a rectangle including both the high-resolution area 10a and the low-resolution area 10b.

Here, the recognizing units 31b to 34b recognize the image signal of at least a partial area among the image signals acquired by the imaging unit (image acquiring unit), and output the first image recognition result as a first image. It functions as a recognition part. In addition, in the present embodiment, the partial area is an area corresponding to the low distortion area.

The recognizing units 31b to 34b transmit the type of object and a set of coordinates to the integration processing unit 40 as a recognition result. On the other hand, the recognizing units 31b to 34b are a set of information on the type of object straddling the camera units and the moving direction of the object or priority recognition region information from the integrated control unit 41c of the integrated processing unit 40. Receive forecast information. This prediction information will be described later.

Here, the output of the recognition section 31b of the camera unit 11 installed facing the front is also directly supplied to the running control section (ECU) 60. FIG. This is because it may be necessary to immediately stop traveling based on the recognition result of an obstacle or the like by the recognizing unit 31b, or to control traveling to avoid the obstacle.

The camera information units 31c to 34c store camera information unique to the camera units 11 to 14 in advance in memory. The camera information section can also temporarily hold information from various sensors provided in the camera units 11-14.

The camera information includes, for example, characteristic information (resolution boundary information, etc.) as shown in FIG. It also includes the number of pixels of the imaging elements 21d to 24d, the mounting position coordinates and orientation (pitch, roll, yaw, etc.) information of the camera unit in the vehicle coordinates, the imaging direction, and the like. Camera information may include information such as gamma characteristics, sensitivity characteristics, and frame rate.

Furthermore, the camera information may include information about image processing methods such as lossless compression methods and image formats when generating RAW image signals in the image processing units 31a to 34a. Note that the mounting position coordinates may be stored in the memory in the camera information section in advance, since the mounting position with respect to the vehicle is often determined for each camera unit.

The orientation coordinates of the camera unit are coordinates relative to the vehicle 1, and may be obtained from an encoder (not shown) provided in the camera unit. Alternatively, it may be acquired using a three-dimensional acceleration sensor or the like.

Also, the information regarding the imaging direction may be acquired using, for example, a geomagnetic sensor. Since the resolution boundary information of the camera is determined by the lens design, it is assumed to be stored in advance in the memory within the camera information section.

The camera information is specific to the imaging units 21 to 24 and different from each other. The information is transmitted to the integration processing unit 40 and referred to when the integration processing unit 40 performs image processing.

The camera processing units 31 to 34 incorporate a CPU as a computer and a memory storing a computer program as a storage medium. Also, the CPU is configured to control each section in the camera processing sections 31 to 34 by executing a computer program in the memory.

In this embodiment, the image processing units 31a to 34a and the recognition units 31b to 34b use hardware such as dedicated circuits (ASIC) and processors (reconfigurable processors, DSPs). As a result, it is possible to speed up the image recognition in the high-resolution area, and increase the possibility of avoiding accidents such as contact with a moving object. Note that the image processing units 31a to 34a may have a distortion correction function.

Some or all of the functional blocks inside the camera processing units 31 to 34 may be realized by causing the CPU to execute a computer program stored in the memory. It is desirable to increase the speed.

An integrated processing unit 40 has an SOC (System On Chip)/FPGA (Field Programmable Gate Array) 41, a CPU 42 as a computer, and a memory 43 as a storage medium. The CPU 42 performs various controls over the entire image processing system 100 by executing computer programs stored in the memory 43 . Note that in this embodiment, the integrated processing unit 40 is housed in a housing separate from the camera unit.

The SOC/FPGA 41 has an image processing section 41a, a recognition section 41b, and an integrated control section 41c. The image processing unit 41a acquires RAW image signals from the camera processing units 31 to 34, and acquires camera information of the camera units 11 to 14 from the camera information units 31c to 34c.

As described above, the camera information includes the optical characteristics of the different angle lenses 21c to 24c, the number of pixels of the imaging devices 21d to 24d, photoelectric conversion characteristics, gamma characteristics, sensitivity characteristics, RAW image signal format information, camera unit vehicle It includes mounting coordinates and attitude information in coordinates.

The image processing unit 41a performs resolution conversion on the respective RAW image signals from the camera processing units 31 to 34 based on the camera information, and converts the images obtained from the low resolution regions 10b of the imaging units 21 to 24, respectively. Perform distortion correction on the signal.

In this embodiment, the image processing unit 41a does not perform distortion correction because the image signal obtained from the high-resolution area 10a has almost no distortion. However, the image processing unit 41a may also perform simple distortion correction on the image signal obtained from the high resolution area 10a. Further, the image processing unit 41a appropriately performs irreversible compression processing and the like on the respective RAW image signals from the camera processing units 31-34.

Further, the image processing unit 41a synthesizes the distortion-corrected image signals of the low-resolution regions 10b of the imaging units 21 to 24 and the image signals of the high-resolution regions 10a so as to connect them smoothly. Every 24 full images are formed.

When performing distortion correction on both the image signal of the low resolution area 10b and the image signal obtained from the high resolution area 10a, the RAW image signals obtained by the image processing units 31a to 34a are directly distorted. can be corrected.

The recognition unit 41b performs image recognition processing on the entire image of each of the imaging units 21 to 24 after distortion correction at least in the low-resolution area, and recognizes a predetermined object (for example, an automobile) in the entire image of each of the imaging units 21 to 24. , people, obstacles, etc.). That is, the recognition unit 41b corrects the distortion of the image signal corresponding to at least the low-resolution area (high-distortion area), performs image recognition, and outputs the second image recognition result.

At that time, the recognition results (object type and coordinates) by the recognition units 31b to 34b are also referred to. In this embodiment, the recognition unit 41b performs image recognition on the entire image of each of the imaging units 21 to 24, but the image recognition does not necessarily have to be performed on the entire image. For example, image recognition may not be performed for the peripheral portion of the image.

That is, the image recognition area of the recognition section 41b may be an area wider than the areas recognized by the recognition sections 31b to 34b, for example. That is, the recognizing unit 41b performs image recognition on an image signal of an area wider than the partial area including the partial area subjected to image recognition by the first image recognizing unit, among the image signals acquired by the image acquiring unit. It functions as a second image recognition unit for outputting a second image recognition result.

The second image recognition unit performs image recognition on both the image signals corresponding to the high resolution area 10a as the low distortion area and the image signal corresponding to the low resolution area 10b as the high distortion area, and outputs the second image recognition result. ing.

In this embodiment, the image processing section 41a synthesizes images from the camera units 12 to 14 as a plurality of imaging sections so as to join them together to form a panoramic synthetic image. In this case, the images of the plurality of imaging units to be spliced are set so that at least a part of each shooting angle of view has an overlapping area of a predetermined amount or more.

That is, the camera units 12 and 13 are arranged so that the shooting ranges of the camera units 12 and 13 overlap each other. Also, the camera units 13 and 14 are arranged so that the shooting ranges of the camera units 13 and 14 overlap each other. Moreover, in the present embodiment, the imaging ranges of the low-distortion regions of at least two image acquisition units are arranged so as to overlap each other.

The recognition unit 41b also performs image recognition on the panoramic composite image. As a result, for example, it is possible to recognize an image of an object photographed so as to extend over the angles of view of a plurality of imaging units. In other words, there are cases where the entire image of the object cannot be seen from the individual whole images from the respective imaging units, but in the panoramic composite image, almost the entire object is captured and image recognition becomes possible through image processing. This is because there are cases.

For example, when the recognition result by the recognition units 31b to 34b and the recognition result by the recognition unit 41b are different, the integrated control unit 41c outputs the integrated image recognition result by adopting the recognition result with higher reliability. do.

For example, the ratio of the objects recognized by the recognition units 31b to 34b in the screen is compared with the ratio of the same objects recognized by the recognition unit 41b that are closed in the screen. It may be adopted by judging that it is more reliable.

Alternatively, in the case of an object that straddles both the high-resolution area and the low-resolution area, the recognition result by the recognition unit 41b is judged to be more reliable than the recognition result by the recognition units 31b to 34b, and adopted. You may Alternatively, when the positions of the objects recognized by the recognition units 31b to 34b are in the periphery of the screen, it is determined that the reliability is low, and the recognition result by the recognition unit 41b is determined to be more reliable. may be adopted.

Alternatively, the recognizing unit 41b performs image recognition only in the low-resolution area while the distortion of the low-resolution area is corrected. You may make it image-recognize with respect.

That is, it is possible to perform control so that the recognition unit 41b does not perform image recognition processing, considering that the reliability of the recognition by the recognition units 31b to 34b is higher for the object existing only in the high-resolution area. Here, the integration control unit 41c functions as an integration processing unit that outputs an image recognition result integrated based on the reliability of the first image recognition result and the reliability of the second image recognition result.

In addition, the image processing unit 41a selects a desired image from among the entire image of each of the imaging units 21 to 24 and a panoramic composite image obtained by joining them together. 3 Forms a signal to be displayed on the display unit 52 or the like. It also generates a frame for highlighting the recognized target object, information about the target object type, size, position, speed, etc., and CG for warning.

Further, it is possible to generate a boundary image CG for displaying the boundary based on optical system characteristic information such as display resolution boundary information acquired from the camera information units 31c to 34c. In order to represent the boundary, it is sufficient to generate information indicating the boundary without generating a display image.

In addition, display processing for superimposing these CG and characters on the image is performed. Here, the first display section 50, the second display section 51, the third display section 52, and the like function as display sections, and display image signals, integrated image recognition results, and the like.

Furthermore, in this embodiment, the integrated control section 41c is configured to share information regarding recognized objects among a plurality of camera units. That is, for example, it is assumed that the object recognized by the camera unit 14 is recognized to be moving in the direction of the angle of view of the camera unit 11 . In that case, the integrated control section 41c transmits prediction information including the type of the object and the moving direction of the object or priority recognition area information to the recognition section 31b of the camera unit 11. FIG.

By sharing such prediction information among the recognition sections 31b-34b of the camera units 11-14, the accuracy of image recognition in the recognition sections 31b-34b of the camera units 11-14 can be improved. The advantage of sharing such prediction information is particularly large when the recognition units 31b to 34b of the camera units 11 to 14 are separate from the recognition unit 41b of the integrated processing unit 40. FIG.

Further, the integrated control unit 41c communicates with the traveling control unit (ECU) 60 and the like via a communication unit (not shown) provided therein using protocols such as CAN, FlexRay, and Ethernet. As a result, display processing for appropriately changing the information to be displayed based on the vehicle control signal received from the travel control unit (ECU) 60 or the like is performed. That is, for example, the range of the image displayed on the display unit is changed according to the moving state of the vehicle acquired by the vehicle control signal.

A travel control unit (ECU) 60 is mounted on the vehicle 1 and is a unit containing a computer and a memory for comprehensively performing drive control, direction control, etc. of the vehicle 1 . From the travel control unit (ECU) 60, vehicle control signals such as travel speed, travel direction, shift lever, shift gear, turn signal state, vehicle direction detected by a geomagnetic sensor, etc. are integrated. It is input to the processing unit 40 .

The integrated control unit 41c transmits information such as the type, position, moving direction, and moving speed of the predetermined object (obstacle, etc.) recognized by the recognition unit 41b to the running control unit (ECU) 60. FIG. Accordingly, the travel control unit (ECU) 60 performs necessary controls such as stopping the vehicle, driving the vehicle, changing the traveling direction, and avoiding obstacles. Here, the traveling control unit (ECU) 60 functions as a movement control unit that controls the movement of the vehicle as a moving object based on the integrated image recognition result.

The first display unit 50 is installed, for example, in the vicinity of the center in the vehicle width direction above the front of the driver's seat of the vehicle 1, and functions as an electronic rearview mirror with the display screen directed toward the rear of the vehicle. A half mirror or the like may be used so that it can be used as a mirror when it is not used as a display. Further, a touch panel or an operation button may be provided to acquire an instruction from the user and output it to the integrated control section 41c.

The second display unit 51 is installed, for example, around the operation panel near the center in the vehicle width direction in front of the driver's seat of the vehicle 1 . The vehicle 1 as a mobile body is equipped with a navigation system, an audio system, and the like (not shown).

For example, the second display section can also display various control signals from the navigation system, the audio system, and the travel control section (ECU) 60 . It also has a touch panel and operation buttons, and is configured to be able to acquire instructions from the user.

The third display unit 52 is, for example, a display unit of a tablet terminal. can also

When adjusting the positions and postures of the camera units 11 to 14, the adjustment operation is performed while displaying the adjustment screen on the display unit of the wirelessly connected tablet terminal. Of course, the functions of the second display section 51 and the third display section 52, which display control signals from the navigation system, can be realized by one display device.

As the display panels of the first display section 50, the second display section 51, and the third display section 52, a liquid crystal display, an organic EL display, or the like can be used. Note that the number of display units is not limited to three.

Some or all of the functional blocks included in the integrated processing unit 40 and the like may be realized by hardware, or may be realized by causing the CPU 42 to execute a computer program stored in the memory 43. good. As hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like can be used.

Note that part or all of the image processing performed by the image processing units 31 a to 34 a may be performed by the image processing unit 41 a of the integrated processing unit 40 . That is, in this embodiment, for example, the image acquisition section and the first image recognition section are housed in the same housing of the camera unit, and the camera unit and the second image recognition section are housed in separate housings. ing. However, for example, the first image recognition section may be housed in the housing of the integrated processing section 40 together with the second image recognition section.

In this embodiment, the integrated processing unit 40 is mounted on the vehicle 1 as a moving body, but part of the processing of the image processing unit 41a, the recognition unit 41b, and the integrated control unit 41c of the integrated processing unit 40 can be performed by, for example, a network. may be performed by an external server or the like.

In that case, for example, the imaging units 21 to 24 as image acquisition units are mounted on the vehicle 1 as a moving body, but part of the functions of the camera processing units 31 to 34 and the integrated processing unit 40 are processed by an external server or the like. it becomes possible to Also, it is possible to provide a running control unit (ECU) 60 with some or all of the functions of the integrated processing unit 40 .

A storage unit 61 stores the entire image of each of the imaging units 21 to 24 generated by the integration processing unit 40 and a panoramic synthesized image. Furthermore, CG such as a predetermined frame, characters, and warnings indicating the recognized object, and images superimposed on CG and displayed on the first display section 50 and the second display section 51 are displayed together with the time, GPS information, and the like. Remember. The integration processing unit 40 can also reproduce past information stored in the storage unit 61 and display it on the first display unit 50 and the second display unit 51 .

A communication unit 62 is for communicating with an external server or the like via a network, and transmits information before being stored in the storage unit 61 or past information stored in the storage unit 61 to the external server or the like. can be stored on an external server or the like.

Further, as described above, the image can be transmitted to an external tablet terminal or the like and displayed on the third display section 52, which is the display section of the tablet terminal. It is also possible to acquire traffic information and various types of information from an external server or the like and display them on the first display section 50 and the second display section 51 through the integrated processing section 40 .

4 to 8 are flowcharts for explaining a series of operations of the integration processing unit 40 of the first embodiment. 4 to 8 are sequentially performed by the CPU 42 of the integration processing unit 40 executing the computer program in the memory 43. FIG.

For example, a start button for starting the camera attitude adjustment mode is displayed on the third display unit 52 of the tablet terminal, and when the start button is clicked, the flows of FIGS. 4 to 8 are started.

FIG. 4 is a flow chart for explaining the processing flow of the camera processing section of the first embodiment. In step S41 of FIG. 4, it is determined whether or not the rear camera adjustment mode is set. If Yes, the process proceeds to step S42, and if No, the process proceeds to step S80 in FIG.

In step S42 (characteristic information acquisition step), the position, orientation, resolution boundary information, etc. of the rear camera are acquired from the camera information section 33c of the rear camera unit 13. FIG. Here, step S42 functions as a characteristic information acquiring section that acquires characteristic information regarding the characteristics of the optical image.

In step S43 (image acquisition step), an image is acquired from the rear camera unit 13. In step S44, it is determined whether or not to display the resolution boundaries of a plurality of camera units. If No, the process proceeds to step S45.

At step S45, it is determined whether or not distortion correction is to be performed. If Yes, in step S46, the image processing unit 41a corrects the distortion of the image of the rear camera unit 13, sets the flag F to 1, and proceeds to step S48.

In step S48, the display resolution boundary of the rear camera unit 13 is calculated based on the flag F, taking into consideration whether or not the distortion has been corrected. Then, in step S49 (display signal generation step), the calculated display resolution boundary is superimposed on the video from the rear camera unit 13 and displayed on the third display unit 52, for example.

That is, for example, it is displayed on the third display unit 52 of the tablet terminal used when adjusting the camera unit via the communication unit 62 . Here, step S48 forms a boundary image between the low distortion region and the high distortion region based on the characteristic information acquired by the characteristic information acquisition unit, and generates a display signal by superimposing the boundary image on the image signal. functioning as a department.

9A and 9B are diagrams for explaining an example of superimposition of the display state of the image and the display resolution boundary. FIG. It is a figure explaining the example which superimposed. That is, the image displayed when the flag F is 0 is shown.

On the other hand, FIG. 9B is a diagram for explaining an example in which the boundary image of the display resolution boundary 93 is superimposed on the distortion-corrected image 131 from the rear camera unit 13 . Both FIGS. 9A and 9B are examples of images displayed on the display section of the tablet terminal, which is the third display section 52 .

It should be noted that the display resolution boundary 93 displayed in the flag FIG. 9(B) may be coordinate transformed according to the distortion correction. Also, when superimposing the display resolution boundary 93 on the image, a CG of an additional boundary image as shown in FIG. 10 may be superimposed.

FIG. 10 is a diagram for explaining another example of the display resolution boundary 93 superimposed on the image. In order to distinguish the boundary between the high-resolution area and the low-resolution area, FIG. It is a figure which shows the example which enabled the outside to be known.

Note that when the luminance, color, and the like are different between the inside and outside of the display resolution boundary 93 as shown in FIG. 10A, the display resolution boundary line itself does not have to be displayed. That is, display of the display resolution boundary (boundary image) in this embodiment includes display of the boundary without using lines. In addition, the line representing the boundary can be represented by, for example, a double line, and the inner line (the side where the high-resolution area is located) can be made a dashed line to guide the line of sight inward, making it easier to distinguish between the inside and outside of the high-resolution area. can.

FIG. 10B is a diagram showing an example in which a horizontal axis and a vertical axis are added and superimposed on the display resolution boundary 93 to make it easier to understand when the camera unit is tilted with respect to the roll axis or the like. .

It should be noted that when displaying the display resolution boundary, for example, it may be displayed as a rectangle for convenience. By displaying in a rectangle, it is possible to more clearly indicate whether or not the roll is being performed. The rectangle in that case may be, for example, a rectangle that inscribes or circumscribes a circular display resolution boundary.

After the image, the display resolution boundary 93 and the like are output and displayed on the third display unit 52 in step S49 of FIG. 4, the process proceeds to step S50 of FIG.

FIG. 5 is a flowchart for explaining the processing continued from FIG.
At step S50, if the flag F is 0 or 1, the reference frame 200 and the reference frame center 203 as shown in FIG. 11A are displayed on the screen. When the flag F is 1, the distortion is corrected, so the coordinates of the image displayed on the third display unit 52 and the display resolution boundary 93 are also converted accordingly.

Note that the reference frame 200 may be, for example, a frame displaying only four corners or, for example, four dots. It should be noted that the frame of reference 200 is set to sufficiently satisfy the rear camera standard setting final rule (MVSS 111) by NHTSA (National Highway Traffic Safety Administration) of the United States. That is, for example, an area which is 3 m behind the rear end of the vehicle and which is 1.5 m away from the center axis of the vehicle in the left-right direction and which can sufficiently accommodate the head of an infant may be set as the reference frame.

11A and 11B are diagrams showing examples of reference frames and the like displayed on the adjustment screen according to the flag F, and FIG. FIG. 10 is a diagram showing an example of a reference frame and the like; FIG. 12 is a diagram showing an example of the installation position of the reference frame. When the flag F is 0, the screen in FIG. 11A is the screen before distortion correction, and when the flag F is 1, the screen is after the distortion correction.

In FIG. 11A, for example, a rectangular virtual reference frame 200 and a reference frame center 203 are displayed with respect to the display resolution boundary 93 . This reference frame 200 is located at a position 205 shown in FIG. 12 (for example, at a position 2 m away from the rear end of the vehicle 1, perpendicular to the traveling direction of the vehicle 1, and 4 m in the horizontal direction and 4 m in the horizontal direction from the ground). 1 m position) on the screen with the coordinates of the vehicle 1 as a reference.

The direction of the vehicle 1 may be obtained from the output of a geomagnetic sensor or the like via the travel control unit (ECU) 60 . Instead of displaying such a virtual image on the screen, the reference frame 200 and the reference frame center 203 may be displayed by actually placing the reference plate at the position 205 and taking an image.

Note that this reference frame 200 corresponds to the image area displayed on the electronic rearview mirror. For example, when the vehicle 1 moves backward, the image processing section 41a displays only the image signal obtained from the camera unit 13 on the electronic rearview mirror. At this time, only the image area corresponding to the reference frame 200 can be extracted from the image signal obtained from the camera unit 13 and displayed on the first display section 50 as an electronic rearview mirror, for example.

However, during the adjustment of the camera unit 13, the overall image as shown in FIG. 11A is displayed on the screen 520 of the third display unit 52 such as the display unit of the tablet terminal, and the position and orientation of the camera unit 13 can be adjusted. .

As described above and as shown in FIG. 12, at least two of the camera units 11 to 14, the camera units 12 and 13, and the camera units 13 and 14 are arranged so that their shooting ranges overlap each other. ing. Moreover, in the present embodiment, at least two image acquisition units are arranged so that the imaging ranges of the low-distortion regions overlap each other.

In step S51 of FIG. 5, it is determined whether or not the reference frame center 203 and the barycenter 93c of the display resolution boundary 93 match. If they match, the process proceeds to step S53. If No, the process proceeds to step S52, displays an arrow 303 (guide information) so as to move the center of gravity 93c of the display resolution boundary 93 in the direction of the reference frame center 203, and then proceeds to step S53.

In step S51, the reference frame center 203 and the center of gravity 93c do not necessarily have to match. It is sufficient if most (ideally, all) of the reference frame area can be adjusted to be a high resolution area. Therefore, a threshold may be provided, and if the area of the high-resolution area within the reference frame area exceeds the threshold, it may be determined as Yes in S51 and the process proceeds to S53.

Thus, in step S52, the direction and length of the arrow 303 can be used to display the direction and amount of movement for correcting the deviation. Alternatively, the direction of movement and instructions for movement may be given by text or voice.

Furthermore, in step S52, for example, with the reference frame center 203 as the center of the horizontal and vertical coordinates of the screen, the horizontal and vertical coordinates of the center of gravity 93c of the display resolution boundary 93 may be displayed numerically. .

These numerical values, the arrow 303, the reference frame 200, and the reference frame center 203 function as guide information for adjusting the orientation of the image acquisition unit based on the boundary image. At least one of them may be used as the guide information. For example, only the reference frame 200 may be displayed, and the user may visually adjust the boundary image and the reference frame 200 . Steps S50 and S52 also function as a display signal generation step (display signal generation unit).

In step S53, it is determined whether or not the adjustment of the posture of the camera unit 13 has been completed. For example, an adjustment completion button is displayed on the third display unit 52 of the tablet terminal. For example, when the user adjusts the posture of the camera unit 13 while looking at the screen of the tablet and clicks the adjustment completion button after completing the adjustment, it is determined that the adjustment is completed.

As described above, it is not always necessary to adjust until the deviation between the reference frame center 203 and the center of gravity 93c is eliminated. It is sufficient if most (ideally, all) of the reference frame area can be adjusted to be a high resolution area. Therefore, a threshold value is provided, and if the area of the high-resolution region within the reference frame region exceeds the threshold value, it is determined Yes in S53 and the process proceeds to S54. If No in step S53, the process returns to step S51, and if Yes, the process proceeds to step S54.

In step S54, it is determined whether or not there is a mode termination request. For example, an end button for ending the camera attitude adjustment mode is displayed on the third display unit 52 of the tablet terminal, and when the user clicks the end button, it is determined that a mode end request has been issued.

If Yes in step S54, the flow of FIGS. 4 to 8 ends. If No, the process returns to step S41.

At step S44 in FIG. 4, it is determined whether or not to display the resolution boundary of a plurality of camera units, and if Yes, the process proceeds to step S60 in FIG. 6 (characteristic information acquisition step).

FIG. 6 is a flowchart for explaining branch processing from FIG. Here, whether or not the area within the reference frame 200 can be photographed as a high-resolution area is displayed in an easy-to-understand manner. I'm trying to help you adjust.

In step S60, the positions, postures, resolution boundary information, etc. of the right camera unit 12 and the left camera unit 14 are obtained from the camera information sections 32c and 34c. That is, the boundary information in the camera units 12 and 14 as a plurality of image acquisition units is acquired.

Next, in step S61, it is determined whether or not images from a plurality of camera units are to be used. If No, the process proceeds to step S62 to determine whether or not distortion correction is to be performed.

If Yes in step S62, the image of the rear camera unit 13 is corrected for distortion in step S63, the flag F is set to 3, and the process proceeds to step S65. On the other hand, if No in step S62, the flag F is set to 2 in step S64, and then the process proceeds to step S65.

In step S65, the display resolution boundaries of the rear camera unit 13, the right camera unit 12, and the left camera unit 14 are calculated.

Then, in step S<b>66 (display signal generation step), a plurality of display resolution boundaries are superimposed on the image of the rear camera unit 13 and displayed on the third display unit 52 . That is, a display signal is generated by superimposing a plurality of boundary images on one image. The display signal is supplied to, for example, the third display section 52 of the tablet terminal used when adjusting the camera unit via the communication section 62 and displayed.

FIG. 9C shows a display resolution boundary 93 of the camera unit 13, a display resolution boundary 92 of the camera unit 12, and a display resolution boundary 94 of the camera unit 14 superimposed on an image 130 not corrected for distortion from the camera unit 13. FIG. 10 is a diagram showing an example of displaying by pressing; Both FIGS. 9C and 9D are examples of images displayed on the display section of the tablet terminal, which is the third display section 52 .

When an imaging unit including lenses with different angles of view on the sides is used, the inside of the resolution boundary 92 becomes a high-resolution area, and when an imaging unit including lenses with different angles of view on the sides is used, the outside of the resolution boundary 92 and the reverse angle of view. The area photographed by the lens becomes the high-resolution area.

That is, the image displayed when the flag F is 2 is shown. On the other hand, in FIG. 9D, a display resolution boundary 93 of the camera unit 13, a display resolution boundary 92 of the camera unit 12, and a display resolution boundary 94 of the camera unit 14 are superimposed on the distortion-corrected image 131 from the camera unit 13. It is a figure which shows the example which carried out.

That is, the image displayed when the flag F is 3 is shown. Note that display resolution boundaries 92 and 94 displayed in FIG. 9(D) are obtained by coordinate transformation according to distortion correction. The display resolution boundary 93 may also be coordinate-transformed according to the distortion correction.

After step S66, the process proceeds to step S50. The operations of steps S50 to S54 are the same as those already described, but when the flag F is 2 or 3, an adjustment screen such as that shown in FIG. 11B is displayed in step S50.

FIG. 11B is a diagram showing an example of a reference frame displayed on the adjustment screen when the flag F is 2, 3, or the like. When the flag F is 2, the screen in FIG. 11B is the screen before distortion correction, and when the flag F is 3, the screen is after the distortion correction.

When the flag F is 2 or 3, a reference point 202 for aligning the reference frame 200, the reference frame center 203, and the center of gravity 93c of the display resolution boundary 92 of the camera unit 12 on the right side is displayed. Furthermore, a reference point 204 for aligning the center of gravity 94c of the display resolution boundary 94 of the camera unit 14 on the left side is displayed.

Then, in step S51, the reference frame center 203 and the center of gravity 93c of the display resolution boundary 93 match, the reference point 202 and the center of gravity 93c of the display resolution boundary 92 match, and the reference point 204 and the center of gravity 94c of the display resolution boundary 94 match. If the

Here, the reference points 202 and 204 are set at a predetermined distance from the reference frame center 203, and the display resolution boundaries 92 and 93 are set so as to overlap each other by a predetermined amount. Also, the display resolution boundaries 94 and 93 are set so as to overlap each other by a predetermined amount. That is, as described above, in the present embodiment, the photographing ranges (in particular, the high resolution region (low distortion region)) of at least two image acquisition units are adjusted and arranged so as to overlap each other.

As described above, in step S51, the reference points 202, 204 and the centers of gravity 92c, 93c do not necessarily have to match. It is sufficient if most (ideally, all) of the reference frame area can be adjusted to be a high resolution area. Therefore, if a threshold value is provided and the area of the high-resolution region within the reference frame region exceeds the threshold value, it can be determined as Yes in S51. If it is equal to or less than the threshold value, No is determined in S51, and the process proceeds to S52.

Note that the reference frame 200 in FIG. 11B corresponds to the image area displayed on the electronic rearview mirror, as described above. Therefore, the image processing unit 41a synthesizes and displays the image signals obtained from the camera units 12 to 14 when the vehicle 1 is running normally.

Then, an image area corresponding to the reference frame 200 can be extracted from the composite image and displayed on the first display unit 50 as an electronic rearview mirror, for example. However, during the adjustment of the camera units 12-14, the overall image as shown in FIG. 11B is displayed on the third display unit 52 such as the display unit of the tablet terminal, and the posture of the camera units 12-14 can be adjusted. can be done.

In step S52, if there is a deviation between the reference frame center 203 and the center of gravity 93c of the display resolution boundary 93, the deviation is indicated by an arrow 303. FIG. Further, as shown in FIG. 11B, an arrow 302 indicates the deviation between the reference point 202 and the center of gravity 93c of the display resolution boundary 92, and an arrow 304 indicates the deviation between the reference point 204 and the center of gravity 94c of the display resolution boundary 94. FIG. Along with the arrow 302, it is also possible to display characters such as "Please turn the camera on the right side toward the vehicle side" or notify by voice.

As described above, it is not necessary to adjust the deviation until the reference points 202, 204 and the centers of gravity 92c, 93c match. It is sufficient if most (ideally, all) of the reference frame area can be adjusted to be a high resolution area. Therefore, a threshold is provided, and if the area of the high-resolution area within the reference frame area exceeds the threshold, it is determined Yes in S53 and the process proceeds to S54. In step S53, when the user adjusts the positions and orientations of the camera units 12 to 14 and clicks the adjustment completion button after the adjustment is completed, it is determined that the adjustment has been completed.

Next, in step S61 of FIG. 6, when it is determined whether or not to use images from a plurality of cameras, if Yes, the process proceeds to step S70 of FIG.

FIG. 7 is a flowchart for explaining branch processing from FIG. In step S703 (image acquisition step), images of the camera unit 12 on the right side and the camera unit 14 on the left side are acquired, and in step S71, it is determined whether or not to perform distortion correction.

In the case of Yes in step S71, the images of the rear camera unit 13, the right camera unit 12, and the left camera unit 14 are corrected for distortion in step S72, the flag F is set to 5, and the process proceeds to step S74. . On the other hand, if No in step S71, the flag F is set to 4 in step S73, and then the process proceeds to step S74.

In step S74, the images of the rear camera unit 13, the right camera unit 12, and the left camera unit 14 are synthesized. That is, the image signals obtained from the plurality of image obtaining units are synthesized to generate a synthesized image. Next, in step S75, display resolution boundaries of the rear camera unit 13, the right side camera unit 12, and the left side camera unit 14 are calculated.

Then, in step S76 (display signal generation step), the display resolution boundary is superimposed on the combined video and displayed on the third display unit 52 . That is, a display signal is generated in which a plurality of boundary images are superimposed on the synthesized image. The display signal is supplied to, for example, the third display section 52 of the tablet terminal used when adjusting the camera unit via the communication section 62 and displayed.

FIG. 9(E) is a diagram showing an example in which a display resolution boundary is superimposed on an image synthesized without distortion correction, and shows an image displayed when the flag F is 4. FIG. FIG. 9F is a diagram showing an example in which the display resolution boundary is superimposed on the image synthesized after the distortion correction, and shows an image displayed when the flag F is 5. In FIG. 520 is a screen frame of the third display unit 52 .

120 is a non-distortion-corrected image signal obtained from the right camera unit 12, 130 is a non-distortion-corrected image signal obtained from the rear camera unit 13, and 140 is obtained from the left camera unit 14. image signal without distortion correction. 900 is an image displaying them side by side.

The camera images are arranged so that the resolution boundaries are separated by the distance of the display resolution boundaries of each camera. For example, when the installation position or posture of the camera unit is changed according to the guides shown in FIGS. 11B and 11C and the display resolution boundary approaches, the information indicating the boundary (circular line) also approaches.

Also, 121 is a distortion-corrected image signal obtained from the right camera unit 12, 131 is a distortion-corrected image signal obtained from the rear camera unit 13, and 141 is obtained from the left camera unit 14. 2 shows a distortion-corrected image signal. 901 is an image in which they are displayed side by side. The camera images are arranged so that the resolution boundaries are separated by the distance of the display resolution boundaries of each camera.

For example, when the installation position or posture of the camera unit is changed according to the guides shown in FIGS. 11B and 11C and the display resolution boundary approaches, the information indicating the boundary (circular line) also approaches. Note that the display resolution boundaries 92 and 94 displayed in FIG. 9F are obtained by coordinate transformation according to the distortion correction. The display resolution boundary 93 may also be coordinate-transformed according to the distortion correction.

In this embodiment, the display resolution boundaries 92 and 94 are displayed on the screen 520 of the third display unit 52 so as not to overlap the display resolution boundary 93 in FIGS. 9E and 9F. . However, the image displayed on the electronic room monitor during normal driving is synthesized with a portion of the display resolution boundary 92 overlapping the display resolution boundary 93 . In addition, a part of the display resolution boundary 94 and the display resolution boundary 93 are synthesized while overlapping each other.

After step S76, the process proceeds to step S50. The operations of steps S50 to S54 are the same as those already described, but when the flag F is 4 or 5, an adjustment screen such as that shown in FIG. 11C is displayed in step S50.

FIG. 11C is a diagram showing an example of a screen for adjustment when the flag F is 4 or 5. When the flag F is 4, the screen is before distortion correction. is the image after distortion correction.

In FIG. 11C, the reference frame 210 displays a reference frame center 213 and a reference point 212 for aligning the center of gravity 93c of the display resolution boundary 92 of the camera unit 12 on the right side. Furthermore, a reference point 214 for aligning the center of gravity 94c of the display resolution boundary 94 of the camera unit 14 on the left side is displayed.

Then, in step S51, the reference frame center 213 and the center of gravity 93c of the display resolution boundary 93 match, the reference point 212 and the center of gravity 93c of the display resolution boundary 92 match, and the reference point 214 and the center of gravity 94c of the display resolution boundary 94 match. If the

Note that the reference frame 210 in FIG. 11C corresponds to the image area displayed on the electronic rearview mirror. However, in FIG. 11C, the images of the camera units 11 to 13 are displayed side by side without being synthesized, but the images of the camera units 11 to 13 are actually displayed on the electronic rearview mirror. It is a composite image obtained by superimposing and synthesizing. Therefore, the reference frame 210 is displayed at the time of adjustment, and has a longer and narrower shape than the reference frame 200 actually corresponding to the electronic rearview mirror shown in FIGS.

That is, when the image signals obtained from the camera units 12 to 14 are combined and displayed, the image processing unit 41a cuts out an image region corresponding to the reference frame 200 in FIG. 11A from the combined image. is displayed on the first display unit 50 as an electronic rearview mirror, for example.

Note that, in the present embodiment, the cropped region of the composite image displayed on the electronic rearview mirror corresponds to the high resolution region (low resolution region) of each of the camera units 12 to 14, and the high resolution region of the composite image. Only the (low-resolution area) is cut out and displayed.

However, as described above, during the adjustment of the camera units 12 to 14, the overall image and the reference frame 210 as shown in FIG. ~14 position and attitude adjustments can be made.

In step S52, if there is a deviation between the reference frame center 213 and the center of gravity 93c of the display resolution boundary 93, it is indicated by an arrow. The arrow indicates the deviation between the reference point 212 and the center of gravity 93c of the display resolution boundary 92, and the arrow indicates the deviation between the reference point 214 and the center of gravity 94c of the display resolution boundary 94. FIG. Further, the other operations in steps S50 to S54 are the same as those already explained, so the explanation will be omitted. That is, in a state in which a plurality of boundary images are displayed in the synthesized image, guide information such as arrows for adjusting the posture of the imaging unit is generated.

Next, in step S41 in FIG. 4, it is determined whether or not the rear camera unit is adjusted, and if No, the process proceeds to step S80 in FIG. 8 (characteristic information acquisition step).

FIG. 8 is a flow chart for explaining another branch process from FIG. In step S80, the position, orientation, resolution boundary information, etc. of the front camera unit 11 are acquired from the camera information section 31c of the front camera unit 11, and in step S813 (image acquisition step), the image of the front camera unit 11 is acquired. do.

Then, in step S82, it is determined whether or not distortion correction is to be performed. If Yes, distortion correction is performed on the image of the front camera unit 11 in step S83, the flag F is set to 7, and the process proceeds to step S85. On the other hand, if No in step S82, the flag F is set to 6 in step S84, and then the process proceeds to step S85.

In step S85, the display resolution boundary of the front camera unit 11 is calculated, and in step S86 (display signal generation step), the image of the front camera unit 11 is superimposed on the display resolution boundary and displayed on the third display unit 52. . That is, for example, it is displayed on the third display unit 52 of the tablet terminal used when adjusting the camera unit via the communication unit 62 .

FIG. 9G is a diagram showing an example in which the display resolution boundary 91 of the front camera unit 11 is superimposed on the image 110 of the front camera unit 11 without distortion correction, and is displayed when the flag F is 6. An example image is shown. FIG. 9H is a diagram showing an example in which the display resolution boundary 91 of the front camera unit 11 is superimposed on the image 111 of the front camera unit 11 after distortion correction. shows an example of It should be noted that the display resolution boundary 91 displayed in FIG. 9(H) may be coordinate-transformed according to the distortion correction.

After step S86, the process proceeds to step S50. The operations of steps S50 to S54 are the same as those already described, but when the flag F is 6 or 7, an adjustment screen such as that shown in FIG. 11A is displayed in step S50.

When the flag F is 6, the screen in FIG. 11A is the screen before distortion correction, and when the flag F is 7, it is the screen after distortion correction. Note that the display resolution boundary 93 and its center of gravity 93c in FIG. 11A are read as the display resolution boundary 91 and its center 91c, respectively.

Also, the virtual frame of reference for the front camera unit 11 may be of the same size as the frame of reference 200, and is positioned in front of the vehicle 1 in FIG. It can be displayed on the screen as a virtual arrangement.

Note that the reference frame 200 in FIG. 11A corresponds to the image area displayed on the electronic rearview mirror, as described above. Therefore, when displaying the image signal obtained from the camera unit 11, the image processing unit 41a cuts out the image area corresponding to the reference frame 200 and displays it on the first display unit 50 as an electronic rearview mirror, for example. can be done.

However, during the adjustment of the camera unit 11, the overall image as shown in FIG. 11A is displayed on the screen 520 of the third display unit 52 such as the display unit of the tablet terminal, and the position and orientation of the camera unit 11 can be adjusted. .

Thus, in the present embodiment, the high-resolution area (low-distortion area) 10a is formed by the central projection method (y=f×tan θ) or the equidistant projection method (y= It is configured to have a projection characteristic approximated to f×θ). Therefore, for example, an image for an electronic rearview mirror displayed on the first display unit 50 has a higher resolution than the low resolution area (high distortion area) 10b, and the front, sides, and rear of the vehicle 1 are far away. It can be displayed more precisely.

In addition, since the optical distortion of the high-resolution area 10a is small, the image for the electronic rearview mirror displayed on the first display unit 50 can be displayed with little distortion, and the driver can see the surroundings of the vehicle more naturally. It can be seen with a sense of distance.

In addition, in the adjustment example of FIG. 11, the case where the imaging unit including the lens with the different angle of view on the side is used has been described. In the case of using an image pickup unit including a reverse different angle of view lens on the side, adjustment is performed so that the high resolution area outside the resolution boundaries 92 and 94 fits within the reference frame 200 .

Further, the high-resolution area 10a in the present embodiment is configured so as to reduce optical distortion, and image recognition can be performed in the state of a RAW image signal without distortion correction, thereby reducing the processing load for image recognition. Image recognition can be performed at high speed. Therefore, the obstacle can be detected early based on the image recognition result, and the action for avoiding the obstacle can be performed in a timely manner. As described above, when the configuration of the present embodiment is used, a great effect can be obtained especially during high-speed driving on an expressway or the like.

In the above embodiment, an example using a plurality of camera units has been described, but a system with only one camera unit is also effective. In addition, according to this embodiment, in a system using a lens with a different angle of view as described above, the position and posture of installing the camera unit can be easily adjusted, so that the work efficiency for adjustment can be greatly improved. The performance of a system using lenses with different angles of view can be fully exploited.
In the above embodiment, the adjustment process can be shortened by displaying the boundary image between the low distortion area and the high distortion area when adjusting the camera unit. The boundary image may be displayed while the vehicle is running.

In the above-described embodiment, an example in which the image processing system is installed in a moving object such as a vehicle has been described. However, the mobile body of the present embodiment is not limited to vehicles such as automobiles, and may be any mobile device such as trains, ships, airplanes, robots, and drones. Further, the image processing system of this embodiment includes those mounted on such moving bodies. Also, this embodiment can be applied to remote control of a moving object.

[Embodiment 2]
At least one of the various functions, processes and methods described in Embodiment 1 above can be implemented using a program. Hereinafter, in the second embodiment, a program for realizing at least one of the various functions, processes, and methods described in the first embodiment will be referred to as "program X". Furthermore, in the second embodiment, a computer for executing program X is called "computer Y". Examples of the computer Y are a personal computer, a microcomputer, a CPU (Central Processing Unit), and the like. A computer such as the image processing system in the above-described embodiments is also an example of the computer Y.

At least one of the various functions, processes and methods described in Embodiment 1 above can be realized by the computer Y executing the program X. In this case, program X is supplied to computer Y via a computer-readable storage medium. A computer-readable storage medium in the second embodiment includes at least one of a hard disk device, a magnetic storage device, an optical storage device, a magneto-optical storage device, a memory card, a ROM, and a RAM. Furthermore, the computer-readable storage medium in Embodiment 2 is a non-transitory storage medium.

The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention. They are not excluded from the scope of the invention. In addition, the present invention includes the following combinations.

(Configuration 1) Image acquiring means for acquiring an image signal formed by an imaging means for taking an optical image having a low distortion region and a high distortion region; and characteristic information acquiring means for acquiring characteristic information about characteristics of the optical image. forming information indicating a boundary between the low distortion region and the high distortion region based on the characteristic information obtained by the characteristic information obtaining means, and generating a display signal for superimposing the information indicating the boundary on the image signal; and display signal generation means for generating an image processing system.

(Arrangement 2) The image according to Arrangement 1, wherein the image acquisition means includes an optical system that forms the optical image, and an imaging device that picks up the optical image formed by the optical system. processing system.

(Arrangement 3) A plurality of the image acquisition means are arranged at different positions, and the display signal generation means combines the image signals acquired from the plurality of image acquisition means to generate a composite image. 3. The image processing system according to the characterizing configuration 1 or 2.

(Arrangement 4) The image processing system according to Arrangement 3, wherein the characteristic information acquiring means acquires the characteristic information of a plurality of the image acquiring means.

(Configuration 5) In configuration 4, the display signal generation means generates a display signal in which information indicating a plurality of the boundaries is superimposed on one image based on the characteristic information in the plurality of image acquisition means. The described image processing system.

(Structure 6) The image processing system according to structure 5, wherein the display signal generating means generates a display signal for superimposing information indicating the plurality of boundaries on the synthesized image.

(Structure 7) The image processing according to any one of structures 1 to 6, wherein the display signal generation means generates a display signal including guide information for adjusting the posture of the image acquisition means. system.

(Structure 8) The image processing system according to structure 7, wherein the guide information includes a reference frame, and the reference frame corresponds to an image area displayed on an electronic rearview mirror.

(Arrangement 9) The image processing according to any one of Arrangements 1 to 8, wherein the low distortion area and the high distortion area respectively correspond to the high resolution area and the low resolution area of the optical image. system.

(Structure 10) Let f be the focal length of the optical system, θ be the half angle of view, y be the image height on the image plane, and y(θ) be the projection characteristic representing the relationship between the image height y and the half angle of view θ. when
3. The image processing system according to configuration 2, wherein y([theta]) in the low distortion region is larger than f*[theta] and is different from the projection characteristic in the high distortion region.

(Arrangement 11) The low-distortion region is characterized in that it is constructed so as to have a projection characteristic approximating a central projection method (y=f×tan θ) or an equidistant projection method (y=f×θ). An image processing system according to configuration 10.

を満足するように構成されていることを特徴とする構成１０又は１１に記載の画像処理システム。 (Structure 12) When θmax is the maximum half angle of view of the optical system and A is a predetermined constant,

12. The image processing system according to configuration 10 or 11, characterized in that it is configured to satisfy:

(Arrangement 13) At least one of the image acquisition means has f as the focal length of the optical system, θ as the half angle of view, y as the image height on the image plane, and the relationship between the image height y and the half angle of view θ as Letting y(θ) be the projection characteristic to be expressed and θmax be the maximum half angle of view of the optical system,
0.2<2×f×tan(θmax/2)/y(θmax)<0.92
13. The image processing system according to any one of configurations 3 to 12, wherein:

(Arrangement 14) A plurality of image acquiring means for acquiring image signals formed by image capturing means for capturing an optical image having a low distortion area and a high distortion area, and characteristics of the optical image of each of the image acquiring means and a characteristic information obtaining means for obtaining characteristic information about the display, and based on the characteristic information obtained by the characteristic information obtaining means, the image signals from the respective image obtaining means are subjected to image processing to generate a predetermined display signal. and display signal generation means, wherein at least two of the plurality of image acquisition means are arranged such that photographing ranges of at least two of the image acquisition means overlap each other.

(Arrangement 15) The image according to Arrangement 14, wherein the image acquisition means includes an optical system that forms the optical image, and an imaging device that captures the optical image formed by the optical system. processing system.

(Structure 16) Let f be the focal length of the optical system, θ be the half angle of view, y be the image height on the image plane, and y(θ) be the projection characteristic representing the relationship between the image height y and the half angle of view θ. 16, wherein y(θ) in the low distortion region is greater than f×θ and is different from the projection characteristic in the high distortion region.

(Structure 17) The low-distortion region is characterized in that it is constructed so as to have a projection characteristic approximating a central projection method (y=f×tan θ) or an equidistant projection method (y=f×θ). 17. An image processing system according to configuration 16.

を満足するように構成されていることを特徴とする構成１６又は１７に記載の画像処理システム。 (Structure 18) When θmax is the maximum half angle of view of the optical system and A is a predetermined constant,

18. The image processing system according to configuration 16 or 17, characterized in that it is configured to satisfy:

(Arrangement 19) At least one of the image acquisition means has f as the focal length of the optical system, θ as the half angle of view, y as the image height on the image plane, and the relationship between the image height y and the half angle of view θ as 0.2<2×f×tan(θmax/2)/y(θmax)<0.92 where y(θ) is the projection characteristic and θmax is the maximum half angle of view of the optical system. The image processing system according to any one of configurations 15 to 18, characterized in that:

(Arrangement 20) The image processing system according to any one of Arrangements 14 to 19, wherein at least two image acquisition means are arranged so that the imaging ranges of the low-distortion regions overlap each other.

(Structure 21) The image according to any one of structures 14 to 20, wherein the display signal generation means generates a composite image by combining the image signals from at least two image acquisition means. processing system.

(Configuration 22) Configurations 14 to 14, wherein the display signal generation means generates information indicating a boundary between the low distortion region and the high distortion region based on the characteristic information acquired by the characteristic information acquisition means. 22. The image processing system according to any one of 21.

(Arrangement 23) The image processing system according to Arrangement 22, wherein the display signal generation means generates a display signal for superimposing the information indicating the boundary on the image acquired from the image acquisition means.

(Arrangement 24) The image processing system according to Arrangement 22 or 23, wherein the display signal generation means generates a display signal including guide information for adjusting the attitude of the image acquisition means.

(Arrangement 25) The image processing system according to Arrangement 24, wherein the guide information includes a reference frame, and the reference frame corresponds to an image area displayed on an electronic rearview mirror.

(Arrangement 26) The image processing according to any one of Arrangements 14 to 25, wherein the low distortion area and the high distortion area respectively correspond to the high resolution area and the low resolution area of the optical image. system.

(Arrangement 27) A plurality of image acquiring means for acquiring image signals formed by image capturing means for capturing an optical image having a low distortion area and a high distortion area, and characteristics of the optical image of each of the image acquiring means and a characteristic information obtaining means for obtaining characteristic information about the display, and based on the characteristic information obtained by the characteristic information obtaining means, the image signals from the respective image obtaining means are subjected to image processing to generate a predetermined display signal. and a display signal generating means, wherein the optical system of the plurality of image acquisition means has the low distortion area in an area near the center of the optical system and the high distortion area in an outer area. and image acquisition means for obtaining an image in which an area near the center of the optical system is the high distortion area and an area near the outside of the optical system is the low distortion area.

(Arrangement 28) An area near the center of the optical system is the low distortion area, and an outer area is the high distortion area. The image processing system described in .

(Arrangement 29) The image acquiring means having the high distortion area in an area near the center of the optical system and the low distortion area in an outer area is arranged on the front or side of a moving object. 29. An image processing system according to configuration 27 or 28.

(Structure 30) The image processing according to any one of structures 27 to 29, characterized in that photographing ranges of at least two of the plurality of image acquisition means are arranged so as to overlap each other. system.

(Arrangement 31) An image obtaining means for obtaining an image signal formed by an imaging means for imaging an optical image having a low distortion area and a high distortion area, and characteristic information about characteristics of the optical image of the image obtaining means is obtained. a characteristic information acquiring means for acquiring; a display signal generating means for generating a predetermined display signal by image processing the image signal from the image acquiring means based on the characteristic information acquired by the characteristic information acquiring means; The image acquisition means has an optical system, the focal length of the optical system is f, the half angle of view is θ, the image height on the image plane is y, and the relationship between the image height y and the half angle of view θ 0.2<2×f×tan(θmax/2)/y(θmax)<0.92, where y(θ) is the projection characteristic representing An image processing system characterized by:

(Arrangement 32) An image acquiring means for acquiring an image signal formed by an imaging means for taking an optical image having a low distortion area and a high distortion area, the image acquiring means having an optical system, The optical system has f as the focal length of the system, θ as the half angle of view, y as the image height on the image plane, y(θ) as the projection characteristic representing the relationship between the image height y and the half angle of view θ, and θmax as An image pickup apparatus that satisfies 0.2<2×f×tan(θmax/2)/y(θmax)<0.92 when the maximum half angle of view is assumed.

(Structure 33) An optical system that forms an optical image having a low distortion region and a high distortion region on a light receiving surface, wherein f is the focal length, θ is the half angle of view, y is the image height on the image plane, and y is the image height. When y(θ) is the projection characteristic representing the relationship between y and the half angle of view θ, and θmax is the maximum half angle of view, 0.2<2×f×tan(θmax/2)/y(θmax)< An optical system characterized by satisfying 0.92.

(Method 1) An image acquisition step of acquiring an image signal formed by imaging means for capturing an optical image having a low distortion region and a high distortion region; and a characteristic information acquisition step of acquiring characteristic information regarding the characteristics of the optical image. forming information indicating a boundary between the low distortion region and the high distortion region based on the characteristic information obtained in the characteristic information obtaining step, and generating a display signal for superimposing the information indicating the boundary on the image signal; and a display signal generation step.

(program) in the computer of the image processing system,
an image acquisition step of acquiring an image signal formed by imaging means for imaging an optical image having a low distortion area and a high distortion area;
a characteristic information acquiring step of acquiring characteristic information relating to the characteristics of the optical image;
forming information indicating a boundary between the low distortion region and the high distortion region based on the characteristic information obtained in the characteristic information obtaining step, and generating a display signal for superimposing the information indicating the boundary on the image signal; a display signal generating step for
A computer program that causes

1: Vehicle 10a: High resolution area 10b: Low resolution area 11-14: Camera unit 21-24: Imaging unit 31-34: Camera processing unit 31b-34b: Recognition unit 40: Integration processing unit 41b: Recognition unit 50: Third 1 display unit 51: second display unit 60: traveling control unit ECU

Claims (35)

an image acquisition means for acquiring an image signal formed by an imaging means for imaging an optical image having a low distortion area and a high distortion area;
a characteristic information obtaining means for obtaining characteristic information relating to the characteristics of the optical image;
forming information indicating a boundary between the low distortion region and the high distortion region based on the characteristic information obtained by the characteristic information obtaining means, and generating a display signal for superimposing the information indicating the boundary on the image signal; a display signal generating means for
An image processing system comprising: The image acquisition means includes an optical system that forms the optical image,
2. The image processing system according to claim 1, further comprising an imaging device for imaging the optical image formed by the optical system. Having a plurality of the image acquisition means arranged at different positions,
2. The image processing system according to claim 1, wherein said display signal generating means generates a composite image by synthesizing said image signals obtained from said plurality of said image obtaining means. 4. The image processing system according to claim 3, wherein said property information acquisition means acquires said property information in a plurality of said image acquisition means. 5. The image according to claim 4, wherein said display signal generation means generates a display signal in which a plurality of pieces of information indicating said boundaries are superimposed on one image based on said characteristic information in said plurality of said image acquisition means. processing system. 6. The image processing system according to claim 5, wherein said display signal generating means generates a display signal for superimposing information indicating said plurality of boundaries on said synthesized image. 2. The image processing system according to claim 1, wherein said display signal generation means generates a display signal including guide information for adjusting the posture of said image acquisition means. 8. The image processing system according to claim 7, wherein said guide information includes a reference frame, and said reference frame corresponds to an image area displayed on an electronic rearview mirror. 2. The image processing system according to claim 1, wherein said low distortion area and said high distortion area respectively correspond to a high resolution area and a low resolution area of said optical image. When the focal length of the optical system is f, the half angle of view is θ, the image height on the image plane is y, and the projection characteristic representing the relationship between the image height y and the half angle of view θ is y(θ),
3. An image processing system according to claim 2, wherein y([theta]) in said low distortion region is greater than f*[theta] and is different from said projection characteristic in said high distortion region. 11. The method according to claim 10, wherein the low distortion region is configured to have a projection characteristic approximated to a central projection method (y=f×tan θ) or an equidistant projection method (y=f×θ). The described image processing system. When θmax is the maximum half angle of view of the optical system and A is a predetermined constant,

11. The image processing system according to claim 10, wherein the image processing system is configured to satisfy: At least one of the image acquisition means has f as the focal length of the optical system, θ as the half angle of view, y as the image height on the image plane, and y as the projection characteristic representing the relationship between the image height y and the half angle of view θ. When (θ) and θmax are the maximum half angle of view of the optical system,
0.2<2×f×tan(θmax/2)/y(θmax)<0.92
4. The image processing system according to claim 3, wherein: having a plurality of image acquisition means for acquiring image signals formed by imaging means for imaging an optical image having a low distortion region and a high distortion region;
a characteristic information acquiring means for acquiring characteristic information relating to the characteristics of the optical image of each of the image acquiring means;
display signal generating means for generating a predetermined display signal by performing image processing on the image signal from each of the image obtaining means based on the characteristic information obtained by the characteristic information obtaining means;
and
An image processing system, wherein at least two of said plurality of image acquisition means are arranged such that photographing ranges of said image acquisition means overlap each other. The image acquisition means includes an optical system that forms the optical image,
15. The image processing system according to claim 14, further comprising an imaging device for imaging the optical image formed by the optical system. When the focal length of the optical system is f, the half angle of view is θ, the image height on the image plane is y, and the projection characteristic representing the relationship between the image height y and the half angle of view θ is y(θ),
16. The image processing system of claim 15, wherein y([theta]) in the low distortion region is greater than f*[theta] and is different from the projection characteristic in the high distortion region. 17. The method according to claim 16, wherein the low-distortion region is configured to have projection characteristics approximate to a central projection method (y=f×tan θ) or an equidistant projection method (y=f×θ). The described image processing system. When θmax is the maximum half angle of view of the optical system and A is a predetermined constant,

17. The image processing system according to claim 16, wherein the image processing system is configured to satisfy: At least one of the image acquisition means has f as the focal length of the optical system, θ as the half angle of view, y as the image height on the image plane, and projection characteristics representing the relationship between the image height y and the half angle of view θ. When y(θ) and θmax are the maximum half angle of view of the optical system,
0.2<2×f×tan(θmax/2)/y(θmax)<0.92
16. The image processing system according to claim 15, wherein: 15. The image processing system according to claim 14, wherein at least two of said image acquisition means are arranged so that the photographing ranges of said low distortion areas overlap each other. 15. The image processing system according to claim 14, wherein said display signal generation means combines said image signals from at least two said image acquisition means to generate a composite image. 15. The image according to claim 14, wherein said display signal generating means generates information indicating a boundary between said low distortion area and said high distortion area based on said characteristic information acquired by said characteristic information acquiring means. processing system. 23. The image processing system according to claim 22, wherein said display signal generation means generates a display signal for superimposing information indicating said boundary on the image acquired from said image acquisition means. 23. The image processing system according to claim 22, wherein said display signal generation means generates a display signal including guide information for adjusting the attitude of said image acquisition means. 25. The image processing system according to claim 24, wherein said guide information includes a reference frame, said reference frame corresponding to an image area displayed on an electronic rearview mirror. 15. The image processing system according to claim 14, wherein said low distortion area and said high distortion area correspond to a high resolution area and a low resolution area of said optical image, respectively. having a plurality of image acquisition means for acquiring image signals formed by imaging means for imaging an optical image having a low distortion region and a high distortion region;
a characteristic information acquiring means for acquiring characteristic information relating to the characteristics of the optical image of each of the image acquiring means;
display signal generating means for generating a predetermined display signal by performing image processing on the image signal from each of the image obtaining means based on the characteristic information obtained by the characteristic information obtaining means;
has
The optical system of the plurality of image acquisition means has the low distortion area in the area near the center of the optical system and the high distortion area in the outer area, and the image acquisition means and the area near the center of the optical system. is the high distortion area, and an image acquisition means for the low distortion area is an outer area. 28. The image acquisition means according to claim 27, wherein the area near the center of the optical system is the low distortion area and the area near the outside is the high distortion area, and the image acquisition means is arranged behind the moving object. image processing system. 27. An area near the center of said optical system is said high distortion area, and an area near the outside of said optical system is said low distortion area. The image processing system described in . 28. The image processing system according to claim 27, wherein at least two of said plurality of image acquisition means are arranged such that photographing ranges of said image acquisition means overlap each other. An image acquiring means for acquiring an image signal formed by an imaging means for taking an optical image having a low distortion area and a high distortion area,
a characteristic information obtaining means for obtaining characteristic information relating to the characteristics of the optical image of the image obtaining means;
display signal generation means for generating a predetermined display signal by image processing the image signal from the image acquisition means based on the characteristic information acquired by the characteristic information acquisition means;
has
The image acquisition means has an optical system, and the focal length of the optical system is f, the half angle of view is θ, the image height on the image plane is y, and the projection characteristic represents the relationship between the image height y and the half angle of view θ. is y(θ) and θmax is the maximum half angle of view of the optical system,
0.2<2×f×tan(θmax/2)/y(θmax)<0.92
An image processing system characterized by satisfying An image acquiring means for acquiring an image signal formed by an imaging means for taking an optical image having a low distortion area and a high distortion area,
The image acquisition means has an optical system, the focal length of the optical system is f, the half angle of view is θ, the image height on the image plane is y, and the projection represents the relationship between the image height y and the half angle of view θ When the characteristic is y(θ) and θmax is the maximum half angle of view of the optical system,
0.2<2×f×tan(θmax/2)/y(θmax)<0.92
An imaging device characterized by satisfying: An optical system that forms an optical image having a low distortion area and a high distortion area on a light receiving surface,
Let f be the focal length, θ be the half angle of view, y be the image height on the image plane, y(θ) be the projection characteristic representing the relationship between the image height y and the half angle of view θ, and θmax be the maximum half angle of view. when
0.2<2×f×tan(θmax/2)/y(θmax)<0.92
An optical system characterized by satisfying an image acquisition step of acquiring an image signal formed by imaging means for imaging an optical image having a low distortion area and a high distortion area;
a characteristic information acquiring step of acquiring characteristic information relating to the characteristics of the optical image;
forming information indicating a boundary between the low distortion region and the high distortion region based on the characteristic information obtained in the characteristic information obtaining step, and generating a display signal for superimposing the information indicating the boundary on the image signal; a display signal generating step for
An image processing method characterized by comprising: In the computer of the image processing system,
an image acquisition step of acquiring an image signal formed by imaging means for imaging an optical image having a low distortion area and a high distortion area;
a characteristic information acquiring step of acquiring characteristic information relating to the characteristics of the optical image;
forming information indicating a boundary between the low distortion region and the high distortion region based on the characteristic information obtained in the characteristic information obtaining step, and generating a display signal for superimposing the information indicating the boundary on the image signal; a display signal generating step for
A computer program that causes

Images