JP2023109164A - Image processing system, mobile body, imaging system, image processing method, and computer program - Google Patents

Abstract

To suppress reduction in resolution when displaying a captured image of the periphery of a mobile body.SOLUTION: The image processing system has a first optical system that forms a first optical image having a low-resolution area corresponding to an angle of view of less than a first angle of view and a high-resolution area corresponding to an angle of view of equal to or greater than the first angle of view, first imaging means that generates first image data by capturing an image of the first optical image formed by the first optical system, and image processing means, that generates first deformed image data by deforming the first image data.SELECTED DRAWING: Figure 5

Description

The present invention relates to an image processing system, a mobile object, an imaging system, an image processing method, a computer program, and the like.

2. Description of the Related Art There is a system that, when an operator controls a mobile object such as a vehicle, photographs the surroundings of the mobile object and generates a bird's-eye view image (overhead view). Japanese Patent Laid-Open No. 2002-200003 discloses capturing an image of the periphery of a vehicle and displaying a bird's-eye view image.

JP 2008-283527 A

However, the technique disclosed in Patent Document 1 has a problem that when processing is performed to extend a distant area of a camera or a peripheral area of a camera image, the sense of resolution of the extended peripheral area decreases.
SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to suppress deterioration in resolution when displaying an image obtained by imaging the surroundings of a moving body.

In order to achieve the above object, the image processing system of the present invention includes:
a first optical system that forms a first optical image having a low resolution area corresponding to an angle of view less than a first angle of view and a high resolution area corresponding to an angle of view greater than or equal to the first angle of view; ,
a first imaging means for imaging the first optical image formed by the first optical system to generate first image data;
and image processing means for generating first modified image data obtained by modifying the first image data.

ADVANTAGE OF THE INVENTION According to this invention, when displaying the image which imaged the circumference|surroundings of a moving body, the fall of a feeling of resolution can be suppressed.

It is a figure explaining the vehicle (for example, automobile) and the imaging range of a camera in 1st Embodiment. 1 is a functional block diagram for explaining the configuration of an image processing system 100 according to a first embodiment; FIG. (A) is a diagram showing contour lines of the image height y1 at each half angle of view on the light receiving surface of the image sensor of the optical system 1 according to the first embodiment. (B) is a diagram showing projection characteristics representing the relationship between the image height y1 and the half angle of view θ1 of the optical system 1 in the first embodiment. (A) to (C) are diagrams showing contour lines of image heights at respective half angles of view on the light-receiving surface of the imaging element of each optical system. 5 is a graph showing an example of equidistant projection, resolution characteristics of optical system 1 and optical system 2 in the first embodiment. 4 is a flowchart for explaining the flow of an image processing method executed by an information processing section 21 of the first embodiment; FIG. 4 is a diagram for explaining a virtual viewpoint and image deformation according to the first embodiment; (A) is a schematic diagram showing the imaging range of the vehicle 10 on the road surface and the left side camera 14, and (B) is a schematic diagram of an image 70 acquired by the camera 14. FIG. (A) is a diagram showing an example of an image captured by the camera 11 while the vehicle 10 is running, and (B) is an image (orthographic projection) of FIG. ) is a diagram showing an example of an image subjected to variable coordinate transformation (deformation). (A) is a diagram showing examples of captured images 81a to 84a acquired by cameras 11 to 14, and (B) is a diagram showing a composite image 90 obtained by synthesizing the captured images. (A) to (D) are diagrams showing the positional relationship between an optical system (optical system 1 and optical system 2) and an imaging element according to the third embodiment. 11A is a schematic diagram showing an imaging range when the camera 11 having the optical system 2 and having the positional relationship between the optical system and the imaging element shown in FIG. 11D is arranged in front of the vehicle 10. FIG. FIG. 12B is a schematic diagram of image data acquired from the camera 11. FIG. (A) and (B) are schematic diagrams when the camera 11 is arranged in front in the third embodiment. (A) and (B) are schematic diagrams showing an example in which the camera 12 having the optical system 1 is arranged on the right side of the vehicle 10 in the third embodiment. (A) and (B) are schematic diagrams showing an example in which the camera 14 having the optical system 1 is arranged on the left side of the vehicle 10 in the third embodiment.

Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In the drawings, the same members or elements are denoted by the same reference numerals, and overlapping descriptions thereof will be omitted or simplified.

[First Embodiment]
In the first embodiment, four cameras are installed to capture images in four directions around an automobile as a moving object, and an image (overhead view) of looking down on the vehicle from a virtual viewpoint existing directly above the vehicle is generated. An imaging system will be described. Note that, in this embodiment, an area that can be acquired with high resolution (high-resolution area) is assigned to the area that is stretched when the camera image is converted from the viewpoint, so that the visibility of the video from the virtual viewpoint can be improved.

FIG. 1 is a diagram for explaining a vehicle (for example, an automobile) and an imaging range of a camera in the first embodiment.
As shown in FIG. 1, cameras 11, 12, 13, and 14 (imaging means) are installed at front, right, rear, and left positions of a vehicle 10 (moving body), respectively.

Cameras 11 to 14 are imaging units having optical systems and imaging elements. The imaging directions of the cameras 11 to 14 are set so that the imaging ranges are the front, right, rear, and left sides of the vehicle 10, and each has an imaging range with an angle of view of about 180 degrees, for example. . Further, the optical axes of the optical systems of the cameras 11 to 14 are installed so as to be horizontal with respect to the vehicle 10 when the vehicle 10 is placed on a horizontal road surface.

The imaging ranges 11a to 14a schematically show horizontal angles of view of the cameras 11 to 14, and 11b to 14b are high-resolution areas where high-resolution images can be obtained depending on the characteristics of the optical system of each camera. is schematically shown. The cameras 11 and 13, which are front and rear cameras, can acquire high-resolution areas near the optical axis, and the cameras 12 and 14, which are side cameras, can acquire high-resolution peripheral view angle areas away from the optical axis. Incidentally, although the imaging range and high-resolution area of the cameras 11 to 14 are actually three-dimensional ranges, they are schematically shown two-dimensionally in FIG. Also, the photographing range of each camera overlaps the photographing range of the other adjacent cameras at the periphery.

Next, FIG. 2 is a functional block diagram for explaining the configuration of the image processing system 100 according to the first embodiment. The image processing system 100 will be explained using FIG. Some of the functional blocks shown in FIG. 2 are realized by causing a computer (not shown) included in the image processing system 100 to execute a computer program stored in the storage unit 22 as a storage medium.
Moreover, each functional block shown in FIG. 2 may not be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

In FIG. 2, an image processing system 100 is installed in a vehicle 10 such as an automobile.
The cameras 11 to 14 respectively have imaging elements 11d to 14d for capturing optical images, and optical systems 11c to 14c for forming optical images on light receiving surfaces of the imaging elements (14c and 14d are not shown). As a result, the surrounding situation is obtained as image data.
The optical system 1 (first optical system) of the cameras 12 and 14 (first imaging means) disposed on the side forms a high-resolution optical image in the peripheral angle-of-view area away from the optical axis. , has an optical characteristic of forming a low-resolution optical image in a narrow field angle region around the optical axis.

Optical systems 2 (second optical systems) of cameras 11 and 13 (second imaging means) arranged in front and behind, which are different from the first imaging means, each have a narrow angle of view around the optical axis. It forms a high-resolution optical image in an area. In addition, it has an optical characteristic of forming a low-resolution optical image in a peripheral viewing angle area away from the optical axis. Details of the optical systems 11c to 14c will be described later.

The imaging elements 11d to 14d are, for example, CMOS image sensors or CCD image sensors, and photoelectrically convert optical images to output imaging data. In the imaging elements 11d to 14d, for example, RGB color filters are arranged in a Bayer array for each pixel. A color image can be obtained by demosaicing.

The image processing apparatus 20 (image processing means) includes an information processing section 21, a storage section 22, and various interfaces (not shown) for data and power input/output, and includes various hardware. Further, the image processing device 20 is connected to the cameras 11 to 14, and outputs image data obtained by synthesizing a plurality of image data obtained from each camera to a display section 30 (display means) as an image.

The information processing section 21 has an image transforming section 21a (image transforming means) and an image synthesizing section 21b (image synthesizing means). It also has, for example, an SOC (System On Chip), an FPGA (Field Programmable Gate Array), a CPU, an ASIC, a DSP, a GPU (Graphics Processing Unit), a memory, and the like. The CPU performs various controls of the entire image processing system 100 including the camera and the display unit by executing computer programs stored in the memory.

Incidentally, in the first embodiment, the image processing device and the camera are housed in separate housings. Further, the information processing unit 21 de-Bayers the image data input from each camera in accordance with the Bayer array, and converts the data into RGB raster format image data. Further, various image processing and image adjustments such as white balance adjustment, gain/offset adjustment, gamma processing, color matrix processing, reversible compression processing, and lens distortion correction processing are performed.
After the image transformation processing for viewpoint conversion is performed by the image transformation unit 21a, the image synthesis unit 21b synthesizes a plurality of images so as to join them together. Details will be described later.

The storage unit 22 is an information storage device such as a ROM, and stores information necessary for controlling the entire image processing system 100 . Note that the storage unit 22 may be a removable recording medium such as a hard disk or an SD card.
The storage unit 22 also stores, for example, camera information of the cameras 11 to 14, a coordinate transformation table for performing image deformation/compositing processing, and parameters for controlling the image processing system 100. FIG. Furthermore, image data generated by the information processing section 21 may be recorded.

The camera information includes the optical characteristics of the optical system 1 and the optical system 2, the number of pixels of the imaging elements 11d to 14d, the photoelectric conversion characteristics, the gamma characteristics, the sensitivity characteristics, the frame rate, the image format information, and the mounting position of the camera in the vehicle coordinate system. including coordinates, etc. In addition, the camera information may include not only the design values of the camera, but also adjustment values that are unique values for each individual camera.

The display unit 30 has a liquid crystal display or an organic EL display as a display panel, and displays video (images) output from the image processing device 20 . This allows the user to grasp the situation around the vehicle. Note that the number of display units is not limited to one. Two or more display units may output a pattern of different viewpoints of a synthesized image, a plurality of images acquired from a camera, and other information display to each display unit.

Next, the optical characteristics of the optical systems 1 and 2 of the cameras 11 to 14 will be described in detail.

First, the optical characteristics of the optical systems 1 and 2 will be described with reference to FIGS. 3 and 4. FIG. In the first embodiment, cameras 12 and 14 have optical systems 1 with the same characteristics, and cameras 11 and 13 have optical systems 2 with the same characteristics. However, the optical characteristics of the optical systems possessed by the cameras 11 to 14 may be different from each other.

FIG. 3A is a diagram showing contour lines of the image height y1 at each half angle of view on the light receiving surface of the image sensor of the optical system 1 according to the first embodiment. FIG. 3B is a diagram showing projection characteristics representing the relationship between the image height y1 and the half angle of view θ1 of the optical system 1 in the first embodiment. In FIG. 3B, the half angle of view (the angle formed by the optical axis and the incident light beam) θ1 is taken as the horizontal axis, and the imaging height (image height) on the light receiving surface (image plane) of the cameras 12 and 14 is y1 is shown as the vertical axis.

FIGS. 4A to 4C are diagrams showing contour lines of the image height at each half angle of view on the light-receiving surface of the image sensor of each optical system. FIG. , FIG. 4B shows an equidistant projection optical system, and FIG. 4C shows an optical system 2 . That is, FIG. 3(A) and FIG. 4(A) are the same. 3 and 4, reference numerals 40a and 41a denote high-resolution areas, which are indicated by being lightly painted. 40b and 41b are low resolution areas.

As shown in FIG. 4B, an equidistant projection type (y=fθ) lens, which is commonly used as a fisheye lens, has a constant resolution at each image height position and has proportional projection characteristics.

On the other hand, the optical system 1 of the cameras 12 and 14, as shown in the projection characteristics of FIG. The projection characteristic y1 (θ1) is configured to change. That is, when the amount of increase in the image height y1 with respect to the half angle of view θ1 per unit (that is, the number of pixels per unit angle) is called the resolution, the resolution differs depending on the area.

It can also be said that this local resolution is represented by the differential value dy1(θ1)/dθ1 of the projection characteristic y1(θ1) at the half angle of view θ1. That is, it can be said that the larger the slope of the projection characteristic y1 (θ1) in FIG. 3B, the higher the resolution. Further, it can be said that the larger the interval of the image height y1 at each half angle of view of the contour lines in FIG. 3A, the higher the resolution.

In the first embodiment, when the half angle of view θ1 is less than the predetermined half angle of view θ1a, the region near the center formed on the light receiving surface of the sensor is the low resolution region 40b, and the half angle of view θ1 is less than the predetermined half angle of view θ1a. An outer region with an angle of view θ1a or more is called a high-resolution region 40a. That is, the optical system 1 (first optical system) includes a low-resolution region 40b corresponding to an angle of view less than the first angle of view (half angle of view θ1a) and a to form a first optical image having a high resolution area corresponding to a field angle of .

Cameras 12 and 14 (first imaging means) capture a first optical image formed by the first optical system to generate first image data.
The value of the half angle of view θ1a is an example for explaining the optical system 1, and is not an absolute value. A high resolution area 40a corresponds to the high resolution areas 12b and 14b in FIG.

Looking at the projection characteristics of FIG. 3B, the rate of increase (inclination) of the image height y1 is small in the low-resolution region 40b with a small angle of view near the optical axis, and the rate of increase (slope) increases as the angle of view gradually increases. It can be seen that this is a projective characteristic in which the slope) increases. This is a characteristic projection characteristic in which the change in inclination is greater than that of the generally known stereographic projection (y=2f×tan(θ/2)).
In order to realize these characteristics, it is preferable to satisfy conditional expression 1 below.

y1 (θ1): projection characteristic representing the relationship between the half angle of view θ1 of the first optical system and the image height y1 on the image plane;
θ1max: maximum half angle of view of the first optical system (angle formed by the most off-axis principal ray from the optical axis),
f1: focal length of the first optical system,
A is a predetermined constant, which may be determined in consideration of the balance between the resolutions of the high-resolution area and the low-resolution area. It's good to

If the lower limit of formula (1) is exceeded, field curvature, distortion, and the like deteriorate, making it impossible to obtain good image quality. If the upper limit is exceeded, the difference in resolution between the central area and the peripheral area becomes small, making it impossible to achieve the desired projection characteristics.

The optical system 2 of the cameras 11 and 13 has a projective characteristic of having a high-resolution area near the optical axis as shown in FIG. 4(C). , the projection characteristics y2 (θ2) are different.

In the optical system 2 according to the first embodiment, the high-resolution area 41a is the area near the center generated on the sensor surface when the half angle of view θ2 is less than the predetermined half angle of view θ2b, and the half angle of view θ2 is less than the predetermined half angle of view θ2b. An outer region having a half angle of view θ2b or more is called a low resolution region 41b. That is, the optical system 2 (second optical system) has a high-resolution area 41a corresponding to an angle of view smaller than the second angle of view (half angle of view θ2b) and an angle of view equal to or larger than the second angle of view. A second optical image is formed having a low resolution area 41b.

Cameras 11 and 13 (second imaging means) capture a second optical image formed by a second optical system to generate second image data.
Here, the value of θ2 corresponding to the image height position of the boundary between 41a and 41b in FIG. .

In the high-resolution area 41a, the optical system 2 (second optical system) has a projection characteristic y2 (θ2) representing the relationship between the half angle of view θ2 of the second optical system and the image height y2 on the image plane. It is configured to be larger than θ2. However, f2 is the focal length of the second optical system that the cameras 11 and 13 have. Also, the projection characteristic y2(θ2) in the high resolution area is set to be different from the projection characteristic in the low resolution area.

When θ2max is the maximum half angle of view of the optical system 2, the ratio θ2b/θ2max between θ2b and θ2max is preferably equal to or greater than a predetermined lower limit. desirable. Also, the ratio θ2b/θ2max of θ2b and θ2max is desirably equal to or less than a predetermined upper limit value, such as 0.25 to 0.35. For example, when θ2max is 90°, the predetermined lower limit is 0.15, and the predetermined upper limit is 0.35, θ2b is preferably determined within the range of 13.5 to 31.5°.

Furthermore, the optical system 2 (second optical system) is configured to satisfy the following equation 2.

Here, B is a predetermined constant. By setting the lower limit to 1, the center resolution can be made higher than that of an orthographic projection (y = f x sin θ) fisheye lens having the same maximum image formation height, and by setting the upper limit to B, the fisheye lens Good optical performance can be maintained while obtaining an equivalent angle of view. The predetermined constant B may be determined in consideration of the resolution balance between the high-resolution area and the low-resolution area, and is preferably 1.9 to 1.4.

FIG. 5 is a graph showing an example of resolution characteristics of equidistant projection, optical system 1, and optical system 2 in the first embodiment. The horizontal axis is the half angle of view θ, and the vertical axis is the resolution, which is the number of pixels per unit angle of view. In equidistant projection, the resolution is constant at any half angle of view, whereas the optical system 1 has the characteristic that the resolution increases at positions with a large half angle of view, and the optical system 2 has a high resolution at positions with a small half angle of view. have characteristics.

By using the optical system 1 and the optical system 2 having the above characteristics, it is possible to acquire a high-resolution image in a high-resolution area while capturing a wide angle of view, such as 180 degrees, which is equivalent to a fisheye lens. can be done.

That is, in the optical system 1, the peripheral angle of view area away from the optical axis becomes a high-resolution area, and when placed on the side of the vehicle, it is possible to obtain a high-resolution image with little distortion in the longitudinal direction of the vehicle. . In the optical system 2, the vicinity of the optical axis becomes a high-resolution area, which is similar to the central projection method (y=f×tan θ) or equidistant projection method (y=f×θ), which is the projection characteristic of an optical system for normal imaging. Since it has such characteristics, optical distortion is small and it is possible to display finely. Therefore, it is possible to obtain a natural perspective when viewing surrounding vehicles such as a preceding vehicle and a following vehicle, and to obtain good visibility by suppressing deterioration of image quality.

It should be noted that the optical system 1 and the optical system 2 can obtain similar effects if the projection characteristics y1 (θ1) and y2 (θ2) that satisfy the conditions of the above formulas (1) and (2), respectively. , the optical system 1 and the optical system 2 of the first embodiment are not limited to the projection characteristics shown in FIGS.

FIG. 6 is a flow chart for explaining the flow of the image processing method executed by the information processing section 21 of the first embodiment. The contents of the processing to be executed will also be explained. The processing flow of FIG. 6 is controlled in units of frames, for example, by the CPU inside the information processing section 21 executing a computer program in the memory.

The processing flow in FIG. 6 is started when the image processing system 100 is powered on, a user operation, a change in running state, or the like is a trigger.
In step S11, the information processing section 21 acquires image data of the vehicle 10 captured by the cameras 11 to 14 in the four directions shown in FIG. The imaging by the cameras 11 to 14 is performed simultaneously (synchronously). That is, a first imaging step of capturing a first optical image to generate first image data and a second imaging step of capturing a second optical image to generate second image data are performed. done synchronously.

In step S<b>12 , the information processing section 21 performs image transformation processing for transforming the acquired image data into an image from a virtual viewpoint. That is, an image processing step is performed to transform the first image data and the second image data to generate the first transformed image data and the second transformed image data.

At this time, the image transforming unit transforms the images obtained from the cameras 11 to 14 based on the calibration data stored in the storage unit. It should be noted that transformation may be performed based on various parameters such as a coordinate conversion table based on calibration data. The contents of the calibration data include the internal parameters of the camera caused by the amount of lens distortion of each camera and the deviation from the sensor position, and the external parameters representing the relative positional relationship between the cameras and the vehicle.

Viewpoint conversion will be described with reference to FIG. FIG. 7 is a diagram for explaining the virtual viewpoint and image deformation of the first embodiment, in which the vehicle 10 is traveling on the road surface 60. FIG. Note that the side cameras 12 and 14 are not shown. The cameras 11 and 13 capture images of the front and rear, and the imaging range of the cameras 11 and 13 includes the road surface 60 around the vehicle 10 . The images acquired by the cameras 11 and 13 are projected on the position of the road surface 60 as a projection surface, and the image is coordinate-transformed as if the projection surface were captured by a virtual camera at a virtual viewpoint 50 directly above the vehicle. (transform. That is, the image is coordinate-transformed to generate a virtual viewpoint image from the virtual viewpoint.

By using various parameters included in the calibration data, an image can be projected onto a projection plane, and an image from another viewpoint can be coordinate-transformed and obtained. It is assumed that the calibration data is calculated by calibrating the camera in advance. Also, if the virtual camera is considered to be an orthographic camera, the image to be generated can be an image that is easy to grasp the sense of distance without distortion.

Also, the images of the cameras 12 and 14 (not shown) on the sides can be deformed by the same process. Also, the projection plane does not have to be a plane that imitates the road surface, and may be, for example, a bowl-shaped three-dimensional shape. Also, the position of the virtual viewpoint does not have to be directly above the vehicle.
The image transformation processing has been described above, but at this time, an area is generated that is greatly elongated due to the coordinate transformation of the image.

FIG. 8A is a schematic diagram showing the imaging range of the vehicle 10 on the road surface and the left side camera 14, and FIG. 8B is a schematic diagram of an image 70 acquired by the camera 14. FIG. A blackened region in the image 70 indicates the outside of the angle of view, indicating that the image has not been acquired. Areas 71 and 72 on the road surface have the same size, are included in the imaging range of the camera 14, respectively, and are displayed at positions 71a and 72a on the image 70, for example. Here, if the optical system of the camera 14 is an equidistant projection system, the area 72a far from the camera 14 is distorted and displayed in a small size (low resolution) on the image.

However, as described above, when viewpoint conversion processing is performed using an orthographic projection virtual camera, the regions 71 and 72 are stretched to the same size. At this time, the area 72 is stretched farther from the original image 70 than the area 71, so the visibility is lowered. That is, in the first embodiment, when the optical systems of the side cameras 12 and 14 are equidistant projection, the peripheral portion of the acquired image distant from the optical axis is stretched by the image deformation processing, so the image after deformation is Visibility is reduced.

On the other hand, the side cameras 12 and 14 in the first embodiment use the optical system 1 having the characteristics shown in FIG. Therefore, even when an image is stretched, deterioration in visibility can be suppressed compared to equidistant projection.

9A is a diagram showing an example of an image captured by the camera 11 while the vehicle 10 is running, and FIG. 9B is an image of FIG. 9A obtained by the camera 11 from a virtual viewpoint right above the vehicle. FIG. 10 is a diagram showing an example of an image that has undergone coordinate transformation (deformation) into a video (orthographic projection);

The image in FIG. 9A is an image of the vehicle 10 (own vehicle) traveling in the left lane of a long straight road with a constant road width. Although distortion actually occurs due to distortion in FIG. 9A, it is simplified. In FIG. 9A, the road width becomes smaller as the distance from the own vehicle increases due to the perspective effect.

However, as described above, when the virtual camera of the virtual viewpoint is converted into an orthographic projection, as shown in FIG. . In the first embodiment, when the optical systems of the cameras 11 and 13 arranged in the front-rear direction of the traveling direction of the vehicle perform equidistant projection, the central region of the image near the optical axis is greatly stretched, and the deformed image visibility is reduced.

On the other hand, since the cameras 11 and 13 arranged in the front-rear direction in the first embodiment have characteristics similar to those of the optical system 2, the area near the optical axis can be acquired with high resolution. Therefore, even when the central region of the image is extended, deterioration in visibility can be reduced compared to equidistant projection.

Returning to the description of the flow of FIG. 6, in step S13, the information processing section 21 synthesizes a plurality of images transformed and deformed in step S12. That is, the second image data captured and generated by the cameras 11 and 13 (second imaging means) and the first image data captured and generated by the cameras 12 and 14 (first imaging means) are combined. After transforming each of them, they are combined to generate a combined image.

FIG. 10A is a diagram showing examples of captured images 81a to 84a acquired by the cameras 11 to 14, and FIG. 10B is a diagram showing a synthesized image 90 obtained by synthesizing the captured images. In step S12, after deformation processing by viewpoint conversion is performed on each of the captured images 81a to 84a, the images are combined according to the respective camera positions.

The images at this time are synthesized at the positions of the regions 81b to 84b of the synthesized image 90, and the upper surface image 10a of the vehicle 10 stored in advance in the storage unit 22 is superimposed on the vehicle position.
As a result, it is possible to create an image that looks down on the own vehicle from a virtual viewpoint, and it is possible to grasp the situation around the own vehicle. In addition, as shown in FIG. 1, the captured images 81a to 84a have overlapping regions when the images are combined because the peripheral portions of the adjacent captured regions are overlapped with each other.

However, as indicated by the dotted lines in FIG. 10B, by performing mask processing and alpha blend processing on each image at the joint position, the synthesized image 90 is displayed as a single image viewed from a virtual viewpoint. can do. Also, the combining position of each camera can be deformed and combined by using the calibration data in the same manner as when the image is transformed in step S12.

In the first embodiment, the regions 82b and 84b use the optical system 1 capable of acquiring regions away from the optical axis with high resolution. Therefore, since the resolution of the regions 82b and 84b in the synthetic image 90, which is the obliquely forward and obliquely rearward regions of the upper surface image 10a of the vehicle 10, is increased by image deformation, an image with high visibility can be generated. can.

Further, in the first embodiment, the optical system 2 capable of acquiring the vicinity of the optical axis with high resolution is used for the regions 81b and 83b. Therefore, in the synthetic image 90, the resolution of the front and rear regions that are stretched by image deformation in the regions 81b and 83b apart from the upper surface image 10a of the vehicle 10 is increased, so that an image with high visibility is generated. be able to. Since there is a high possibility that a moving object will collide with an obstacle in its traveling direction, there is a need to display a farther distance. Therefore, the configuration of the first embodiment is effective because it is possible to improve the visibility of the moving object, particularly in the front and rear of the vehicle.

Although the images 81a and 83a acquired via the optical system 2 have a lower resolution in the peripheral portion away from the optical axis, the images 82a and 84a acquired through the optical system 1 are located away from the optical axis. The peripheral area has high resolution. Therefore, by preferentially using the images 82a and 84a acquired via the optical system 1 in the superimposed area of the respective images when synthesizing the images, the resolution of the peripheral portion away from the optical axis by the optical system 2 can be improved. can compensate for the decline in

For example, the areas 82b and 84b may be increased at the joints, which are dotted lines shown in the synthesized image 90. FIG. That is, the regions 81b and 83b may be narrowed and the regions 82b and 84b may be widened. Alternatively, the weight of the image acquired by the optical system 1 around the joint indicated by the dotted line shown in the synthesized image 90 may be increased by changing the alpha blend ratio or the like between the images.

Returning to FIG. 6 , in step S<b>14 , the information processing section 21 outputs the image synthesized in step S<b>13 and displays it on the display section 30 . As a result, the user can check the image from the virtual viewpoint in high resolution.
By repeatedly executing the flow of FIG. 6 for each frame, a moving image can be displayed, and the relative positions of obstacles can be grasped with high resolution.

[Second embodiment]
In the first embodiment, an example in which the image processing system 100 is installed in a vehicle such as an automobile as a moving object has been described. However, the mobile object of the first embodiment is not limited to vehicles, and may be any mobile device that moves, such as trains, ships, airplanes, robots, and drones. Also, the image processing system 100 of the first embodiment includes those mounted on these mobile devices.
Also, the first embodiment can be applied to remote control of a moving object.

In the first embodiment, the information processing unit 21 is mounted in the image processing device 20 of the vehicle 10, but part of each processing of the information processing unit 21 may be performed inside the cameras 11-14. In that case, the cameras 11 to 14 are also provided with information processing units such as CPUs and DSPs, and after various image processing and image adjustments are performed, the images are output to the image processing device. Also, part of each process of the information processing section 21 may be performed by an external server or the like via a network, for example. In that case, for example, the cameras 11 to 14 are mounted on the vehicle 10, but for example, part of the functions of the information processing section 21 can be processed by an external device such as an external server.

Further, although the storage unit 22 is included in the image processing device 20, the cameras 11 to 14 and the display unit 30 may have storage units. If the cameras 11 to 14 have storage units, the parameters specific to each camera can be linked to each camera body and managed.

Also, some or all of the constituent elements included in the information processing section 21 may be realized by hardware. As hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like can be used. This enables high-speed processing.

Further, the image processing system 100 may be provided with an operation input unit for inputting user operations, for example, an operation panel including buttons and a touch panel in the display unit. Thus, the image processing apparatus mode can be switched, and the camera video (image) desired by the user can be switched, and the virtual viewpoint position can be switched.

Further, the image processing system 100 is provided with a communication unit that performs communication conforming to a protocol such as CAN or Ethernet, and is configured to communicate with a travel control unit (not shown) provided inside the vehicle 10. Also good. Information related to the running (moving) state of the vehicle 10, such as the running speed, the running direction, the state of the shift lever, the shift gear, the winkers, the direction of the vehicle 10 detected by a geomagnetic sensor, etc., is acquired as a control signal from the running control unit. can be

Then, the mode of the image processing device 20 may be switched according to the control signal indicating the movement state thereof, and the camera video (image) may be switched according to the running state, or the virtual viewpoint position may be switched. . That is, it may be controlled according to a control signal indicating the moving state of the moving object whether or not to generate a composite image by combining the first image data and the second image data after transforming them.

Specifically, for example, when the moving speed of the moving object is less than a predetermined speed (for example, less than 10 km), the first image data and the second image data are deformed and combined to generate a composite image. , may be displayed. This allows you to fully understand your surroundings.

On the other hand, if the moving speed of the moving body is equal to or higher than a predetermined speed (for example, 10 km or higher), the second image data from the camera 11 that captures the moving direction of the moving body may be processed and displayed. good. This is because, when the moving speed is high, it is necessary to preferentially grasp an image at a distant position in front.

Further, the image processing system 100 does not have to display an image on the display unit 30, and may be configured to record the generated image in the storage unit 22 or a storage medium of an external server.

Also, in the first embodiment, an example in which images are acquired by connecting the cameras 11 to 14 and the image processing device 20 has been described. However, the camera captures an optical image having a low-resolution area and a high-resolution area by the optical system 1, the optical system 2, etc., and transmits the acquired image data to the external image processing device 20 via, for example, a network. It may be configured as follows. Alternatively, the composite image may be generated by reproducing the above image data once recorded on the recording medium by the image processing device 20 .

Also, in the first embodiment, the image processing system has four cameras, but the number of cameras that the image processing system has is not limited to four. The number of cameras that the image processing system has may be, for example, two or six. Furthermore, an effect can also be obtained in an image processing system having one or more cameras (first imaging means) having the optical system 1 (first optical system).

That is, even when deforming an image acquired from one camera, there is a problem that the resolution of the periphery of the imaging screen is lowered. Visibility can be improved. If there is only one camera, there is no need to synthesize images, so the image synthesizer 21b is not required.

Further, in the first embodiment, the image processing system 100 has two cameras each having the optical system 1 on the side of the moving body and two cameras each having the optical system 2 at the front and rear of the moving body. That is, the first imaging means is arranged on at least one of the right side and the left side with respect to the moving direction of the moving body, and the second imaging means is arranged at least on the front side and the rear side with respect to the moving direction of the moving body. are placed in

However, it is not limited to this configuration. For example, one or more cameras having the optical system 1 may be provided, and the other cameras may have a general fisheye lens or a camera configuration combining various lenses, or one camera having the optical system 1 and one camera having the optical system 2 may be combined. It may be a combination.

Specifically, for example, the imaging areas of two adjacent cameras (the imaging area of the first imaging means and the imaging area of the second imaging means) are arranged so that a part thereof overlaps. When the respective images are combined to generate one continuous image, the optical system 1 is used for one camera and the optical system 2 is used for the other camera to combine the images. At this time, the image of the optical system 1 is preferentially used in the overlapping area of the two images.

As a result, while using the high resolution area near the optical axis of the optical system 2, it is possible to synthesize a video (image) in which the low resolution of the peripheral portion of the optical system 2 is compensated for by the high resolution area of the optical system 1. . That is, the first and second image data obtained from the first and second imaging means are deformed by the image processing means, respectively, and the display section displays high-resolution synthesized data obtained by synthesizing the deformed image data. can be displayed.

Also, in the first embodiment, a camera having the optical system 1 is used as the side camera of the moving body. However, the position of the first imaging means is not limited to the side. For example, even when the camera having the optical system 1 is arranged in front or behind, the image peripheral portion is similarly stretched, so it is effective when it is desired to improve the visibility of the image peripheral portion. In that case, the first image data obtained from the first imaging means is transformed by the image processing means, and the display section displays the transformed image data.

In addition, the camera arrangement directions in the first embodiment are not premised on the four directions of front, back, left, and right. They may be arranged at various positions depending on the oblique direction and the shape of the moving body. For example, in a moving object such as an airplane or a drone, one or more cameras for capturing downward images may be arranged.

Further, in the first embodiment, the image transformation means is image transformation by coordinate transformation for transforming into an image from a virtual viewpoint, but the invention is not limited to this. Any image transformation that is processing for expanding, contracting, or enlarging an image may be used. In this case as well, by arranging the high-resolution areas of the optical system 1 and the optical system 2 in the area where the image is stretched, the visibility of the deformed image can be similarly improved.

Also, in the first embodiment, the optical axes of the cameras 11 to 14 are arranged horizontally with respect to the moving body, but the present invention is not limited to this. For example, the optical axis of the optical system 1 may be arranged in a direction parallel to the vertical direction, or may be arranged in a direction oblique to the vertical direction.

The optical axis of the optical system 2 does not have to be horizontal with respect to the moving object, but it is preferable that it is arranged in front or behind the moving object so that the position far from the moving object is included in the high resolution area. desirable. Optical system 1 can acquire an image distant from the optical axis with high resolution, and optical system 2 can acquire an image near the optical axis with high resolution. All you have to do is arrange them so that they can be assigned.

Further, in the first embodiment, the calibration data is stored in advance in the storage unit 22, and the image is deformed/synthesized based on the data. However, the calibration data may not necessarily be used. In that case, for example, the image may be deformed in real time by a user's operation so that the desired amount of deformation can be adjusted.

[Third Embodiment]
11A to 11D are diagrams showing the positional relationship between the optical system (optical system 1 and optical system 2) and the imaging device according to the third embodiment. In FIGS. 11A to 11D, each square frame represents the imaging surface (light receiving surface) of the imaging element, each concentric circle represents the half angle of view θ, and the outermost circle represents the maximum value θmax. When the imaging surface of the imaging device is larger than θmax, pixel data can be acquired as an image in the area inside θmax.

On the other hand, light does not enter the range outside θmax, and pixel data cannot be acquired in this area. In other words, image data can be acquired within the area inside θmax in the imaging plane. Let θvmax be the maximum half angle of view at which an image in the vertical direction can be acquired on the imaging plane, and let θhmax be the maximum half angle of view at which the image can be acquired in the horizontal direction. At this time, θvmax and θhmax are the imaging range (half angle of view) of image data that can actually be acquired.

In FIG. 11A, the imaging surface is square and the range of the half angle of view θ is entirely within the imaging surface, so θvmax=θhmax=θmax. For example, when .theta.max=90 degrees, a camera having this characteristic can capture an image in a range from the camera position to a horizontal angle of view of 180 degrees and a vertical angle of view of 180 degrees.

In FIG. 11B, the range of θmax is wider than the imaging surface. .theta.hmax<.theta.max and .theta.vmax<.theta.max, light is incident on all areas of the imaging surface, and there is no area on the imaging surface where pixel data cannot be obtained. On the other hand, the imaging range (imaging angle of view) of image data that can be acquired is narrowed.

In FIG. 11C, θhmax=θmax and θvmax<θmax, and an image can be obtained up to θmax in the horizontal direction, but only up to θvmax in the vertical direction. The images described with reference to FIGS. 8 to 10 correspond to the positional relationship shown in FIG. 11(C).

In FIG. 11D, θhmax=θmax in the horizontal direction, but in the vertical direction, the optical axis of the optical system and the center of the imaging surface are displaced (shifted) and are not vertically symmetrical. When θvmax is not vertically symmetrical, the downward direction is expressed as θv1max and the upward direction is expressed as θv2max. In that case, θv1max=θmax in FIG. 11D, but θv2max<θmax in the upward direction.

A case where the optical axis shifts in the horizontal direction can be similarly considered. By shifting the optical axis with respect to the center of the imaging surface in this manner, the imaging range can be changed. If θhmax=θmax and θv1max=θmax, the angle of view in the horizontal direction and the vertical downward direction can be widened, which is desirable. .

FIG. 12A is a schematic diagram showing the imaging range when the camera 11 having the optical system 2 and having the positional relationship between the optical system and the imaging element shown in FIG. 11D is arranged in front of the vehicle 10. be. That is, in FIG. 12A, the forward direction of the moving body is included in the high-resolution area of the second image pickup means, and the second image pickup means is arranged such that the optical axis of the second optical system is aligned with the second image pickup means. 2 is arranged at a position deviated from the center of the image pickup surface of the image pickup means.

A fan-shaped solid line 121 extending from the camera 11 indicates the imaging range of the high-resolution area of the camera 11, a fan-shaped dotted line 122 indicates the entire imaging range including the low-resolution area, and a dashed line indicates the direction of the optical axis. Although the actual imaging range is represented three-dimensionally, it is displayed two-dimensionally for the sake of simplicity.

FIG. 12B is a schematic diagram of image data acquired from the camera 11. FIG. In the horizontal direction and the vertical downward direction, the maximum range up to the half angle of view θmax is imaged, but in the vertical upward direction the image is taken only up to the range up to θv2max because θv2max<θmax.

As shown in FIGS. 12A and 12B, a camera 11 having an optical system 2 and having an optical axis shifted toward the lower portion of the vehicle with respect to the imaging surface is arranged in the front direction of the vehicle 10. The optical axis of 11 is arranged horizontally on the ground and in the direction of travel in front of the vehicle. As a result, the horizontal field angle and vertical downward field angle of the camera can be widened, and the road near the vehicle, which is the driver's blind spot, can be imaged. Furthermore, the camera 11 can capture an image of a distant area in front of the vehicle 10 in the direction of travel in the high-resolution area.

Although the example of arranging the camera in front of the vehicle has been described with FIGS. That is, when the imaging system is mounted, the second imaging means may be arranged at least one of the front side and the rear side of the moving body. By arranging the camera having the optical system 2 behind the vehicle 10, it is possible to capture an image of the far side (rear) in the opposite direction of the traveling direction of the vehicle 10 in a high-resolution area.

In order to capture images of the road surface near the vehicle, it is desirable to place the camera at the front end of the exterior of the vehicle. You may Even in this case, it is possible to image (photograph) a distant object in front with high resolution.

A preferred camera arrangement example will be described below with reference to the drawings. FIGS. 13A and 13B are schematic diagrams when the camera 11 is arranged at the front end of the vehicle 10 in the third embodiment. The direction parallel to the traveling direction of the vehicle is the Y-axis, the direction perpendicular to the ground (horizontal plane) is the Z-axis, and the axis perpendicular to the YZ plane is the X-axis.

In FIGS. 13A and 13B, the absolute value of the angle on the XY plane between a straight line passing through the arrangement position of the camera 11 and parallel to the Y axis and the optical axis 130 is θ2h, and the angle on the YZ plane is θ2h. Let θ2v be the absolute value of . At this time, it is desirable that θ2h≦θ2b and θ2v≦θ2b. As a result, the high-resolution area of the optical system 2 can be accommodated in the forward traveling direction.

When the image pickup system is mounted, the second image pickup means is arranged so that the optical axis of the second optical system deviates downward from the center of the image pickup surface of the second image pickup means. can be By arranging in such a manner, it is possible to image a wide area around the road surface below the moving body.

14A and 14B are schematic diagrams showing an example in which the camera 12 having the optical system 1 is arranged on the right side of the vehicle 10 in the third embodiment. , and FIG. 14B is a front view of the vehicle 10. FIG. 15A and 15B are schematic diagrams showing an example in which the camera 14 having the optical system 1 is arranged on the left side of the vehicle 10 in the third embodiment, and FIG. A left side view of the vehicle 10, and FIG. 15B is a front view of the vehicle 10. FIG.

As shown in FIGS. 14A, 14B, 15A, and 15B, in this embodiment, an imaging system is mounted, and the first imaging means are arranged on the right side and the left side of the moving object. placed on at least one of the

The cameras 12 and 14 have their optical axes 140 shifted from the center of the imaging plane as shown in FIG. 11(D). A fan-shaped solid line 141 extending from the cameras 12 and 14 indicates the imaging range of the high-resolution area of the cameras 12 and 14 , a fan-shaped dotted line indicates the imaging range of the low-resolution area, and a dashed line indicates the direction of the optical axis 140 .

In FIG. 14A, let θ1h be the absolute value of the angle formed on the XY plane between a straight line passing through the arrangement position of the camera 12 and parallel to the X axis and the optical axis 140 . At this time, the value of θ1h is preferably around 0°, that is, the optical axis is directed perpendicularly to the traveling direction of the vehicle 10, but θ1h may be about 30°. As a result, it is possible to image the forward direction and the backward direction in the traveling direction in the high-resolution area of the optical system 1 .

In FIG. 14B, let θ1v be the angle between a straight line passing through the arrangement position of the camera 12 and parallel to the X axis and the optical axis 140 on the XZ plane. At this time, the value of θ1v is preferably around 0°, that is, the optical axis is directed perpendicularly to the traveling direction of the vehicle 10, but θ1v≦(120°−θv1max) may be sufficient. As a result, the road surface near the traveling vehicle can be imaged in the high resolution area of the optical system 1 .

Also, in the examples of FIGS. 14A and 14B, the optical axis of the optical system 1 of the camera 12 is shifted from the center of the imaging surface toward the vehicle downward direction (road surface direction). That is, the first imaging means is arranged at a position where the optical axis of the first optical system is deviated downward of the moving body with respect to the center of the imaging surface of the first imaging means. This makes it possible to widen the angle of view in the direction of the road surface.

In FIG. 15A, let θ1h1 be the absolute value of the angle formed on the YZ plane by a straight line passing through the arrangement position of the camera 14 and parallel to the Z axis and the optical axis 150 . At this time, it is preferable that the value of θ1h1 is around 0°, that is, the optical axis is directed toward the lower portion of the vehicle 10 (road surface direction, vertical direction), but θ1h1≦30° may be sufficient. As a result, the high-resolution area 151 of the optical system 1 can be used to image the forward direction and the backward direction in the forward direction. Note that 152 is a low resolution area.

In FIG. 15B, let θ1v1 be the angle formed between a straight line passing through the arrangement position of the camera 14 and parallel to the Z-axis and the optical axis 150 on the XZ plane in the right direction. At this time, it is desirable that the value of θ1v1 is around 0°, that is, the optical axis is directed toward the bottom of the vehicle 10 (road surface direction, vertical direction), but the optical axis may be tilted by increasing the value of θ1v1. As a result, the high-resolution area 151 of the optical system 1 can capture an image of a far side of the vehicle.

Also, in the example of FIG. 15, the optical axis 150 of the optical system 1 of the camera 14 is shifted from the center of the imaging plane in the direction away from the vehicle body (the direction away from the side of the vehicle 10). That is, in the first imaging means, the optical axis of the first optical system is deviated from the center of the imaging surface of the first imaging means in the direction away from the main body of the moving body. As a result, the angle of view to the far side of the vehicle can be widened.

In the above description, an example in which the positions of the cameras 12 and 14 on the right side and the left side are respectively changed has been described, but the same positions may be used, or only one camera may be used. Also, an example in which two cameras with optical systems 1 on both sides and two cameras with optical systems 2 on the front and rear sides have been described, but two cameras with optical systems 1 and 2 are used. should have

In addition, it may be combined with a general projection type fisheye camera such as equidistant projection. Moreover, although the preferred shift positions of the optical axis and the imaging plane have been described, they do not have to be shifted.

Moreover, although the arrangement of the camera having the optical system 1 and the optical system 2 has been described, the present invention is not limited to this. The high-resolution areas of the optical system 1 and the optical system 2 need only be placed in the system's attention area, and the camera with the optical system 2 is placed in front or behind the vehicle, and the camera with the optical system 1 is placed on the side of the vehicle. It should be placed on the right side. Moreover, it is desirable that the high resolution areas of the optical system 1 and the optical system 2 are arranged so as to overlap each other so that the front and rear can be imaged in each high resolution area.

The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention. They are not excluded from the scope of the invention. It should be noted that embodiments of the present invention include the following combinations.

(Configuration 1) A first optical image forming a first optical image having a low resolution area corresponding to an angle of view less than a first angle of view and a high resolution area corresponding to an angle of view equal to or greater than the first angle of view a first image capturing means for capturing the first optical image formed by the first optical system to generate first image data; and a first image data modified from the first image data and image processing means for generating deformed image data.

(Arrangement 2) The image processing system according to Arrangement 1, wherein the image processing means performs coordinate transformation of an image to generate a virtual viewpoint image from a virtual viewpoint.

(Structure 3) y1(θ1) is the projection characteristic representing the relationship between the half angle of view θ1 of the first optical system and the image height y1 on the image plane, and θ1max is the maximum half-image of the first optical system. When an angle, f1 is the focal length of the first optical system, and A is a predetermined constant, the following equation 1
3. The image processing system according to configuration 1 or 2, characterized in that it is configured to satisfy:

(Configuration 4) having a second imaging means different from the first imaging means,
The image processing means transforms the second image data captured and generated by the second imaging means and the first image data, respectively, and then synthesizes them to generate a synthesized image. 4. The image processing system according to any one of 1 to 3.

(Arrangement 5) The image processing system according to Arrangement 4, wherein the imaging area of the first imaging means and the imaging area of the second imaging means are arranged so as to partially overlap.

(Arrangement 6) A second optical system for forming a second optical image with respect to the second imaging means, the second optical image corresponding to an angle of view less than the second angle of view 6. The image processing system according to configuration 4 or 5, further comprising a high resolution area and a low resolution area corresponding to an angle of view equal to or larger than the second angle of view.

(Structure 7) The focal length of the second optical system is f2, the half angle of view is θ2, the image height on the image plane is y2, and the projection characteristic representing the relationship between the image height y2 and the half angle of view θ2 is y2 (θ2 ) when
7. The image processing system according to configuration 6, wherein y2([theta]2) in the high resolution area is larger than f2*[theta]2 and is different from the projection characteristic in the low resolution area.

(Structure 8) y2(θ2) is the projection characteristic representing the relationship between the half angle of view θ2 of the second optical system and the image height y2 on the image plane, and θ2max is the maximum half-image of the second optical system. Equation 2 below, where f2 is the focal length of the second optical system and B is a predetermined constant.
8. The image processing system according to configuration 7, characterized in that it is configured to satisfy:

(Configuration 9) The image processing system according to any one of Configurations 1 to 8, wherein the first imaging means is arranged on at least one of the right side and the left side with respect to the moving direction of the moving object. mobile body.

(Structure 10) The image processing system according to any one of Structures 4 to 8, wherein the second imaging means is arranged on at least one of the front side and the rear side with respect to the moving direction of the moving object. mobile body.

(Structure 11) The moving body according to structure 10, wherein the first imaging means is arranged on at least one of the right side and the left side with respect to the moving direction of the moving body.

(Arrangement 12) The moving body according to Arrangement 9 or 10, further comprising display means for displaying deformed image data deformed by the image processing means.

(Arrangement 13) The image processing means determines whether or not to generate a synthesized image by synthesizing the first image data and the second image data after transforming them according to the moving state of the moving body. 13. The mobile body according to any one of configurations 10 to 12, characterized in that the moving object is controlled.

(Arrangement 14) When the moving speed of the moving body is less than a predetermined speed, the image processing means transforms the first image data and the second image data and combines them to generate a composite image. 14. The mobile body according to configuration 13, characterized by:

(Arrangement 15) The image processing means receives the second image data from the second imaging means for imaging the moving direction of the moving object when the moving speed of the moving object is equal to or higher than the predetermined speed. 15. The moving body according to configuration 14, wherein the moving object is processed and displayed.

(Structure 16) A first optical image forming a first optical image having a low resolution area corresponding to an angle of view less than a first angle of view and a high resolution area corresponding to an angle of view equal to or greater than the first angle of view a first imaging means for imaging the first optical image formed by the first optical system to generate first image data; and a second imaging means different from the first imaging means and a second optical system for forming a second optical image with respect to the second imaging means, wherein the second optical image is an image having an angle less than a second angle of view. and a low resolution area corresponding to an angle of view equal to or larger than the second angle of view.

(Arrangement 17) The imaging system according to Arrangement 16, wherein the imaging area of the first imaging means and the imaging area of the second imaging means are arranged so as to partially overlap.

(Configuration 18) A configuration characterized in that an imaging range of the high-resolution area of the first imaging means and an imaging range of the high-resolution area of the second imaging means are arranged so as to partially overlap. 18. The imaging system according to 17.

(Arrangement 19) Arrangements 16 to 18, wherein the first imaging means is arranged such that the optical axis of the first optical system is shifted from the center of the imaging surface of the first imaging means. The imaging system according to any one of .

(Arrangement 20) Arrangements 16 to 19, wherein the second imaging means is arranged such that the optical axis of the second optical system is shifted from the center of the imaging surface of the second imaging means. The imaging system according to any one of .

(Configuration 21) A movement characterized by mounting the imaging system according to any one of configurations 16 to 20, and arranging the first imaging means on at least one of the right side and the left side of a moving object. body.

(Configuration 22) A movement characterized by mounting the imaging system according to any one of configurations 16 to 21 and arranging the second imaging means on at least one of the front side and the rear side of the mobile body. body.

(Configuration 23) The imaging system according to any one of configurations 16 to 21 is mounted, and the second imaging means is arranged on at least one of the front side and the rear side of the moving body, and the front of the moving body. A moving body, wherein the direction is included in the high resolution area of the second imaging means.

(Configuration 24) The imaging system according to any one of Configurations 16 to 21 is mounted, and the first imaging means is such that the optical axis of the first optical system is the imaging plane of the first imaging means. with respect to the center of the moving body in the downward direction of the moving body or in the direction away from the main body of the moving body.

(Configuration 25) The imaging system according to any one of Configurations 16 to 21 is mounted, and the second imaging means is such that the optical axis of the second optical system is the imaging plane of the second imaging means. , wherein the moving body is displaced downward from the center of the moving body.

(Method) A first method of forming a first optical image having a low resolution area corresponding to an angle of view less than a first angle of view and a high resolution area corresponding to an angle of view equal to or greater than the first angle of view. An image processing method using an image processing system having an optical system and first imaging means for receiving the first optical image formed by the first optical system, wherein the first optical An image processing method comprising: a first imaging step of capturing an image to generate first image data; and an image processing step of generating deformed image data obtained by deforming the first image data. .

(Program 1) forming a first optical image having a low resolution area corresponding to an angle of view smaller than the first angle of view and a high resolution area corresponding to an angle of view equal to or greater than the first angle of view; and a first imaging means for receiving the first optical image formed by the first optical system; A computer program for performing a first imaging step of generating first image data and an image processing step of generating deformed image data obtained by deforming the first image data.

(Program 2) Any one image processing system of configurations 1 to 8, or any one mobile body of configurations 9 to 15, or any one imaging system of configurations 16 to 20, or any one of configurations 21 to 25 or a computer program for controlling each means of one moving body by a computer.

It should be noted that a computer program that realizes part or all of the control in this embodiment may be supplied to an image processing system, an imaging system, a mobile object, etc. via a network or various storage media. good. Then, the computer (or CPU, MPU, etc.) in the image processing system, imaging system, mobile body, etc. may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

10: Vehicle (moving object)
100: Image processing system (image processing means)
12, 14: side cameras (first imaging means)
12c, 14c: optical system 1 (first optical system)
21a: image transformation unit

Claims (28)

a first optical system that forms a first optical image having a low resolution area corresponding to an angle of view less than a first angle of view and a high resolution area corresponding to an angle of view greater than or equal to the first angle of view; ,
a first imaging means for imaging the first optical image formed by the first optical system to generate first image data;
image processing means for generating first modified image data obtained by modifying the first image data;
An image processing system comprising: 2. The image processing system according to claim 1, wherein said image processing means transforms the coordinates of an image to generate a virtual viewpoint image from a virtual viewpoint. y1(θ1) is the projection characteristic representing the relationship between the half angle of view θ1 of the first optical system and the image height y1 on the image plane, θ1max is the maximum half angle of view of the first optical system, and f1 is When the focal length of the first optical system, A, is a predetermined constant, the following equation 1
2. The image processing system according to claim 1, wherein the image processing system is configured to satisfy: Having a second imaging means different from the first imaging means,
3. The image processing means transforms the second image data captured and generated by the second imaging means and the first image data, respectively, and then synthesizes them to generate a composite image. 2. The image processing system according to 1. 5. The image processing system according to claim 4, wherein an imaging area of said first imaging means and an imaging area of said second imaging means are arranged so as to partially overlap. Having a second optical system that forms a second optical image with respect to the second imaging means,
The second optical image has a high resolution area corresponding to an angle of view less than the second angle of view and a low resolution area corresponding to an angle of view greater than or equal to the second angle of view. Item 5. The image processing system according to item 4. When the focal length of the second optical system is f2, the half angle of view is θ2, the image height on the image plane is y2, and the projection characteristic representing the relationship between the image height y2 and the half angle of view θ2 is y2(θ2) ,
7. The image processing system according to claim 6, wherein y2([theta]2) in said high resolution area is larger than f2*[theta]2 and is different from said projection characteristic in said low resolution area. y2(θ2) is the projection characteristic representing the relationship between the half angle of view θ2 of the second optical system and the image height y2 on the image plane, θ2max is the maximum half angle of view of the second optical system, and f2 is When the focal length of the second optical system, B, is a predetermined constant, the following equation 2
8. The image processing system according to claim 7, wherein the image processing system is configured to satisfy: 2. A moving object, wherein said first imaging means of the image processing system according to claim 1 is arranged on at least one of the right side and the left side with respect to the moving direction of the moving object. 5. A moving body, wherein said second imaging means of the image processing system according to claim 4 is arranged on at least one of the front side and the rear side with respect to the moving direction of the moving body. 11. The moving body according to claim 10, wherein the first imaging means is arranged on at least one of the right side and the left side with respect to the moving direction of the moving body. 10. The moving body according to claim 9, further comprising display means for displaying deformed image data deformed by said image processing means. The image processing means controls whether or not to generate a synthesized image by synthesizing the first image data and the second image data after transforming them according to the moving state of the moving object. 11. The moving object according to claim 10. The image processing means deforms the first image data and the second image data and combines them to generate a composite image when the moving speed of the moving object is less than a predetermined speed. 14. The moving body according to claim 13. The image processing means processes and displays the second image data from the second image pickup means for picking up an image of the moving direction of the moving object when the moving speed of the moving object is equal to or higher than the predetermined speed. 15. The moving body according to claim 14, characterized in that: a first optical system that forms a first optical image having a low resolution area corresponding to an angle of view less than a first angle of view and a high resolution area corresponding to an angle of view greater than or equal to the first angle of view; ,
a first imaging means for imaging the first optical image formed by the first optical system to generate first image data;
Having a second imaging means different from the first imaging means,
Having a second optical system that forms a second optical image with respect to the second imaging means,
The second optical image is characterized by having a high resolution area corresponding to an angle of view less than the second angle of view and a low resolution area corresponding to an angle of view greater than or equal to the second angle of view. imaging system. 17. The imaging system according to claim 16, wherein the imaging area of said first imaging means and the imaging area of said second imaging means are arranged so as to partially overlap. 18. The apparatus according to claim 17, wherein an imaging range of the high-resolution area of the first imaging means and an imaging range of the high-resolution area of the second imaging means are arranged so as to partially overlap. imaging system. 17. The imaging system according to claim 16, wherein the first imaging means is arranged such that the optical axis of the first optical system is shifted from the center of the imaging surface of the first imaging means. . 17. The imaging system according to claim 16, wherein the second imaging means is arranged such that the optical axis of the second optical system is shifted from the center of the imaging plane of the second imaging means. . 17. A mobile body comprising the imaging system according to claim 16, wherein said first imaging means is arranged on at least one of right and left sides of said mobile body. 17. A mobile body comprising the imaging system according to claim 16, and wherein the second imaging means is arranged on at least one of the front side and the rear side of the mobile body. Equipped with the imaging system according to claim 16,
The second imaging means is arranged on at least one of the front side and the rear side of the moving body, and the front direction of the moving body is included in the high resolution area of the second imaging means. mobile body. Equipped with the imaging system according to claim 16,
In the first imaging means, the optical axis of the first optical system is deviated from the center of the imaging surface of the first imaging means in the downward direction of the moving body or in the direction away from the main body of the moving body. A moving body characterized by: Equipped with the imaging system according to claim 16,
The moving body, wherein the optical axis of the second optical system of the second imaging means is deviated in the downward direction of the moving body with respect to the center of the imaging surface of the second imaging means. a first optical system that forms a first optical image having a low resolution area corresponding to an angle of view less than a first angle of view and a high resolution area corresponding to an angle of view greater than or equal to the first angle of view; ,
An image processing method using an image processing system having first imaging means for receiving the first optical image formed by the first optical system,
a first imaging step of imaging the first optical image to generate first image data;
and an image processing step of generating deformed image data obtained by deforming the first image data. a first optical system that forms a first optical image having a low resolution area corresponding to an angle of view less than a first angle of view and a high resolution area corresponding to an angle of view greater than or equal to the first angle of view; ,
A computer of an image processing system having a first imaging means for receiving the first optical image formed by the first optical system,
a first imaging step of imaging the first optical image to generate first image data;
and an image processing step of generating deformed image data obtained by deforming the first image data. The image processing system of any one of claims 1 to 8, or the moving body of any one of claims 9 to 15, or the imaging system of any one of claims 16 to 20, or any one of claims 21 to 25. or a computer program for controlling each means of one moving body by a computer.

Images