JP2021103365A - Image processing device - Google Patents

Abstract

To provide an image processing device etc. which can draw a course line while suppressing an increase in memory capacity of a display device.SOLUTION: An image processing device has: a first conversion unit which performs a first strain correction on each pixel in a first image region of an input image to create an image, and performs a correction on each pixel in the image on which the first strain correction is performed to output it as an image; a second conversion unit which performs a second strain correction on each pixel in a second image region of the input image to output it as an image; a third conversion unit which performs a third strain correction on each pixel in a third image region of the input image to output it as an image; and a synthesis unit which generates and outputs an output image by the image from the first conversion unit, the image from the second conversion unit, and the image from the third conversion unit. In the output image, the image from the third conversion unit is arranged between the image from the first conversion unit and the image from the second conversion unit. The image from the first conversion unit is the image having only a distortion aberration, or having no distortion.SELECTED DRAWING: Figure 19

Description

The present invention relates to an image processing apparatus.

For example, there is an image processing device that converts and outputs an image of a subject captured by an in-vehicle camera using a fisheye lens so as to correct distortion (for example, Patent Document 1).

In such an image processing device, the entire input image is image-converted so that the linear subject toward the optical axis of the fisheye lens and the linear subject in the vertical direction are substantially linear in the output image. Further, in the image processing device, a subject that is straight in the horizontal direction is substantially linear in the central portion of the output image and the left and right portions that are connected to the central portion on the left and right, and the central portion of the output image. The entire input image is image-converted so as to bend near the boundary between the left side portion and the right side portion. As a result, the image processing device provides an output image in which a wide-angle image that does not give a sense of discomfort to the user is easy to see.

Further, there is known a technique of superimposing and displaying a traveling route in a direction in which a vehicle travels in a display device that inputs an output image output from an image processing device as described above (for example, Patent Document 2).

Japanese Unexamined Patent Publication No. 2008-31890 Japanese Unexamined Patent Publication No. 11-334470

In such a display device, the image of the advance line is generally converted into an image so as to match the distortion of the fisheye lens or the image deformed by other image conversion processes. The course on which the vehicle actually travels is superimposed and displayed on the image output by the device. Therefore, the image conversion process for the above-mentioned advance route data is the same image conversion process as the image conversion process for the image captured by the in-vehicle camera performed in the image processing device.

More specifically, on the display device side, for example, for each output coordinate (X, Y) of the output image of the display device, a table that associates the input coordinates (x, y) with the input of the advance route data stored in advance. Is used to perform image conversion processing that generates (transferred) the brightness and color information of the pixels of the output coordinates (X, Y) by referring to (transferring) the brightness and color information of the pixels of the input coordinates (x, y). Can be considered. In such processing, generally, conversion data in which the input coordinates corresponding to the coordinates are converted into a table is stored in advance for the output coordinates, and the conversion processing is performed according to the table.

However, such tableized conversion data requires input coordinates (x, y) for the number of pixels of the output image, and the capacity increases in line with the recent trend toward higher pixel counts in display devices, resulting in an increase in memory cost. There was a problem of inviting.

In view of such a problem, it is an object of the present invention to provide an image processing device or the like capable of superimposing advance routes while suppressing an increase in the memory of the display device.

The image processing apparatus of one aspect converts the input image into an image and outputs the output image. The image processing apparatus includes a first conversion unit, a second conversion unit, a third conversion unit, and a composition unit. The first conversion unit performs the first distortion correction for each pixel in the first image region of the input image to obtain an image, and corrects each pixel in the image to which the first distortion correction is applied to obtain the image. Output as. The second conversion unit performs a second distortion correction for each pixel in the second image area of the input image and outputs the image as an image. The third conversion unit performs a third distortion correction for each pixel in the third image region of the input image and outputs the image as an image. The compositing unit generates and outputs an output image from the image output from the first conversion unit, the image output from the second conversion unit, and the image output from the third conversion unit. In the output image, the image output from the third conversion unit is arranged between the image output from the first conversion unit and the image output from the second conversion unit, and the first conversion is performed. The image output from the unit is an image having only distortion or no distortion.

According to the present invention, it is possible to draw a traveling route while suppressing an increase in the memory capacity of the display device.

FIG. 1 is an explanatory diagram showing an example of the hardware configuration of the image display system of the prerequisite technology. FIG. 2 is a block diagram showing an example of the functional configuration of the imaging device of the prerequisite technology. FIG. 3 is an explanatory diagram showing an example of an image in which an image captured by the imaging device is displayed on the display device. FIG. 4 is an explanatory diagram showing an example of the table configuration of the first conversion table. FIG. 5 is an explanatory diagram showing an example of the relationship between the real image height and the ideal image height. FIG. 6 is an explanatory diagram showing an example of an image before and after the first distortion correction of the first conversion unit. FIG. 7A is an explanatory diagram showing an example of an image conversion model. FIG. 7B is an explanatory diagram showing an example of the table configuration of the second conversion table. FIG. 8 is an explanatory diagram showing an example of an image before and after the second distortion correction of the second conversion unit. FIG. 9A is an explanatory diagram showing an example of the image after the first distortion correction. FIG. 9B is an explanatory diagram showing the first image region and the image region other than the first image region in the image after the first distortion correction in light and dark. FIG. 9C is an explanatory diagram showing the first image region, the second image region, and the third image region of the prerequisite technology in light and dark. FIG. 10 is an explanatory diagram showing an example of the table configuration of the ratio table. FIG. 11 is an explanatory diagram showing an example of a calculation method when calculating the third input coordinates for correcting the third distortion. FIG. 12 is an explanatory diagram showing an example of the table configuration of the third conversion table. FIG. 13 is an explanatory diagram showing an example of an output image. FIG. 14 is an explanatory diagram showing an example of an input image (original image) captured by the imaging device of the prerequisite technology. FIG. 15 is an explanatory diagram showing an example of the image after the first distortion correction. FIG. 16 is an explanatory diagram showing an example of the image after the second distortion correction. FIG. 17 is an explanatory diagram showing the first image region, the second image region, and the third image region of the prerequisite technology in light and dark. FIG. 18 is an explanatory diagram showing an example of an output image having poor visibility in the prerequisite technique. FIG. 19 is a block diagram showing an example of the functional configuration of the image pickup apparatus of the first embodiment. FIG. 20 is an explanatory diagram showing an example of the image after the first distortion correction. FIG. 21 is an explanatory diagram showing the image region of the first α, the image region of the second α, and the image region of the third α of the first embodiment in light and dark. FIG. 22 is an explanatory diagram showing an example of an output image when the image is captured by the imaging device. FIG. 23 is an explanatory diagram showing an example of an image including the image region of the first β after the distortion correction, which is different from the first distortion correction of the second embodiment. FIG. 24 is an explanatory diagram showing the image region of the first β after the distortion correction different from the first distortion correction, the image region of the second α, and the image region of the third α in light and dark. FIG. 25 is an explanatory diagram showing an example of an output image including the image region of the first α after distortion correction different from that of the first distortion correction.

Hereinafter, examples of the image processing apparatus disclosed in the present application will be described in detail with reference to the drawings. The disclosed technology is not limited by each embodiment. In addition, the examples shown below may be appropriately combined as long as they do not cause a contradiction.

FIG. 1 is an explanatory diagram showing an example of the hardware configuration of the image display system 1 of the prerequisite technology. The image display system 1 shown in FIG. 1 includes an image pickup device 2, a navigation device 3, and a display device 4. The image pickup apparatus 2 includes a lens 11, an image pickup element 12, an image processing processor 13, a ROM (Read Only Memory) 14, and a RAM (Radom Access Memory) 15. The image pickup device 2 is, for example, a camera device arranged near the license plate at the rear of the vehicle. The lens 11 is, for example, a fisheye lens. The image sensor 12 is a CMOS (Complementary Metal Oxide Semiconductor) sensor that captures an image of a specific range behind the vehicle through the lens 11. The image processing processor 13 is an image processing device that executes image processing inside the image pickup device 2. The ROM 14 stores information such as various programs. The RAM 15 stores various information.

The navigation device 3 outputs guidance information such as map information to the display device, draws the advance route 50 on the output image from the image pickup device 2, and outputs the output image on which the advance route 50 is drawn to the display device 4. The display device 4 is, for example, a monitor device arranged on the front panel of the driver's seat and displaying, for example, an output image or map information from the navigation device 3. The navigation device 3 is set to draw the route 50 whose image has been converted so as to match the output image after distortion correction of the image.

FIG. 2 is an explanatory diagram showing an example of the configuration of the imaging device 2 of the prerequisite technology. The image pickup apparatus 2 shown in FIG. 2 includes a control unit 20 and a storage unit 30. The control unit 20 corresponds to the image processor 13 in FIG. 1, and the storage unit 30 corresponds to the ROM 14 in FIG. The control unit 20 includes a first conversion unit 21, a second conversion unit 22, a calculation unit 23, a third conversion unit 24, and a synthesis unit 25. The storage unit 30 has a first conversion table 31, a second conversion table 32, a ratio table 33, and a third conversion table 34.

FIG. 3 is an explanatory diagram showing an example in which an image captured by the image pickup device 2 is displayed on the display device 4. The input image 40 shown in FIG. 3 has a first image area 41, a second image area 42, and a third image area 43. The image processor 13 identifies the first image area 41, the second image area 42, and the third image area 43 in the input image. That is, the image processing processor 13 identifies each pixel value in the input image by the input coordinates, and each pixel value in the first image area 41 is the first input coordinate, and each pixel in the second image area 42. The values are identified by the second input coordinates and the third input coordinates of each pixel value in the third image area 43.

The first image region 41 surrounded by the broken line L1 corresponds to, for example, the image region of the central portion in the input image, and for example, the distortion is corrected so as to have only distortion or no distortion ( This is an image area for drawing a traveling line that changes according to a steering angle such as steering wheel rotation, which is a target area to be subjected to (first distortion correction). The second image region 42 surrounded by the alternate long and short dash line L2 and the dotted line L3 displays, for example, a straight line object as shown in Patent Document 1 described above as a straight line and does not give a sense of discomfort to the user (second). This is the image area to be corrected for distortion. This part has no distortion. The third image region 43 surrounded by the broken line L1 and the alternate long and short dash line L2 is an image region for performing a third distortion correction that weighted averages the first distortion correction and the second distortion correction. The weighted average will be described later.

The first conversion unit 21 is a conversion unit that performs the first distortion correction for each pixel in the first image area 41 of the input image 40 and outputs the output. The pixels in the first image area 41 can be said to be, for example, the pixel values of the first input coordinates. The first conversion unit 21 refers to, for example, the first conversion table 31 and outputs the pixel value of the first input coordinate in the output image after correcting the distortion in the first image area 41 in the input image 40. Output as the pixel value of the output coordinates. FIG. 4 is an explanatory diagram showing an example of the table configuration of the first conversion table 31. The first conversion table 31 is a table that manages the first input coordinates of the pixel values corrected for distortion in the input image in association with each output coordinate of the output image. The first conversion unit 21 outputs the pixel values for each output coordinate in the first image area 41 of the output image to the synthesis unit 25.

Here, an operation in which the navigation device 3 converts the traveling route 50 into an image so as to match the output image after distortion correction of the image will be described. For example, assuming that the image in the first image region 41 has distortion, the navigation device 3 performs image conversion on the data of the route 50 so as to have the same aberration as the above-mentioned distortion. When there is distortion, the real image height and the ideal image height are different. FIG. 5 is an explanatory diagram showing an example of the relationship between the real image height and the ideal image height. The navigation device 3 converts the pixel value of the polar coordinates (R, φ) of the ideal image height into the polar coordinates (r, φ) of the real image height. This conversion is a conversion that moves an arbitrary pixel on the line L of FIG. 5 to another place on the same line L, and this conversion table inputs an input image for each output coordinate (X, Y) of the output image. The size of the data is smaller than that of the conversion table associated with the coordinates (x, y). The image in the first image area 41 may be undistorted. At this time, in the conversion from the pixel value of the polar coordinates (R, φ) in the image to the pixel value of the polar coordinates (r, φ), the above-mentioned pixel movement does not occur, so that the conversion table becomes unnecessary.

The second conversion unit 22 is a conversion unit that applies and outputs a second distortion correction for each pixel in the second image region 42 of the input image 40. The pixels in the second image area 42 can be said to be, for example, the pixel values of the second input coordinates. The second conversion unit 22 refers to, for example, the second conversion table 32, and calculates the pixel value of the second input coordinates in the output image after correcting the distortion in the second image area 42 in the input image 40. It is output as the pixel value of the second output coordinate. The second conversion table 32 is a table that manages the second input coordinates of the pixel values corrected for the second distortion in association with each other for each of the second input coordinates in the second image area 42. The second conversion unit 22 outputs the pixel values for each output coordinate in the second image area 42 of the output image to the synthesis unit 25.

The calculation unit 23 is in the first image area 41 closest to the output coordinates in the third image area 43 for each output coordinate of the pixel value corrected for distortion in the third image area 43 in the input image 41. The ratio of the first distance to the output coordinates of the above and the second distance from the output coordinates in the third image area 43 to the output coordinates in the nearest second image area 42 is calculated. Further, the calculation unit 23 stores the ratio as a ratio in the ratio table 33 for each output coordinate. The calculation unit 23 takes the output obtained by performing the first distortion correction for each pixel of the input image 41 and the output obtained by performing the second distortion correction as inputs, and outputs them by weighted averaging. .. For example, the calculation unit 23 refers to the ratio table 33 and corrects the first input coordinate and the second input coordinate according to the ratio for each output coordinate (first distance: second distance). The calculation unit 23 calculates the third input coordinate of the output image by adding the first input coordinate after the ratio correction and the second input coordinate after the ratio correction.

The calculation unit 23 stores the third input coordinates in the third conversion table 34 for each output coordinate. The third conversion unit 24 is a conversion unit that applies and outputs a third distortion correction for each pixel in the third image region 43 of the input image. The pixels in the third image area 43 can be said to be, for example, the pixel values of the third input coordinates. The third conversion unit 24 refers to, for example, the third conversion table 34, and sets the pixel value of the third input coordinate in which the distortion in the third image region 43 in the input image 40 is corrected as the third image. It is output as a pixel value of the output coordinates in the area 43. The third conversion unit 24 outputs the pixel values for each output coordinate in the third image area 43 to the synthesis unit 25.

FIG. 6 is an explanatory diagram showing an example of images before and after the first distortion correction of the first conversion unit 21. The first conversion unit 21 refers to the first conversion table 31 and outputs the pixel value of the first input coordinate corrected for distortion as the pixel value of the output coordinate for each output coordinate. The first conversion unit 21 outputs the pixel values of the output coordinates in the first image area 41 to the synthesis unit 25. As a result, the first conversion unit 21 can output the image before the first distortion correction as the image after the first distortion correction as shown in FIG. The first conversion unit 21 uses, for example, the pixel value of the input coordinate β1 in the input image as the pixel value of the output coordinate α1 in the output image, and the pixel value of the input coordinate β3 in the input image as the output coordinate in the output image. It is output as a pixel value of α3. For convenience of explanation, in the example of FIG. 6, the case where the pixel values of all the first input coordinates in the input image are output as the pixel values of the output coordinates is illustrated. However, in reality, the first conversion unit 21 uses the pixel value of the first input coordinate corrected for the distortion in the first image area 41 in the input image as the output coordinate in the first image area 41. It is output to the compositing unit 25 as a pixel value.

FIG. 7A is an explanatory diagram showing an example of an image conversion model. The image conversion model shown in FIG. 7A includes an input image 51 formed of a circular plane, a hemispherical virtual object surface 52 on which a subject is projected corresponding to the lens 11, a correction surface 53 for correcting distortion, and image conversion. The display surface 54 to be displayed is arranged in order. The virtual optical axis z corresponding to the optical axis of the lens 11 passes through the center of these surfaces at a right angle from the optical axis origin 0 located at the center of the input image 51. The correction surface 53 is composed of a central surface 53A arranged in the center and a left side surface 53B and a right side surface 53C arranged on the left and right sides of the central surface 53A, respectively. The left side surface 53B and the right side surface 53C are the origins of the optical axis. It is constructed by bending toward. The image on the virtual object surface 52 projected on the correction surface 53 in the virtual optical axis z direction is projected onto the display surface 54 by an orthographic projection method, and the projected image of the display surface 54 is used as an output image.

In the image conversion model, the pixel position of the display surface 54 is converted to the pixel position of the correction surface 53, the pixel position of the correction surface 53 is converted to the polar coordinates of the hemisphere of the virtual object surface 52, and finally the pixels of the input image 51. Convert to position. In the image conversion model, the pixel position of the input image 51 is converted for each pixel position of the display surface 54, and the pixel position of the input image 51 is stored for each pixel position of the display surface 54. That is, the second conversion table 32 manages the second input coordinates of the pixel value corrected for distortion for each output coordinate.

FIG. 7B is an explanatory diagram showing an example of the table configuration of the second conversion table 32. The second conversion table 32 shown in FIG. 7B manages the output coordinates and the second input coordinates in association with each other. The output coordinates are the coordinates for outputting the pixel value of the output image. The second input coordinate is a coordinate for inputting a pixel value corrected for distortion for each output coordinate. The principle for calculating the input coordinates of the pixel value corrected for the second distortion for each output coordinate in the second conversion table 32 is based on the principle of Patent Document 1 (Japanese Patent Laid-Open No. 2008-31890). ) Proposed. Although the technique shown in Patent Document 1 is applied here, the technique is not limited to this, and a known technique for correcting distortion caused by a wide-angle lens may be used.

FIG. 8 is an explanatory diagram showing an example of images before and after the second distortion correction of the second conversion unit 22. The second conversion unit 22 refers to the second conversion table 32, and outputs the pixel value of the second input coordinate corrected for the second distortion as the pixel value of the output coordinate for each output coordinate. For example, as shown in FIG. 8, the second conversion unit 22 uses the pixel value of the input coordinate β5 in the input image as the pixel value of the output coordinate α5 in the output image, and sets the pixel value of the input coordinate β6 in the input image as the pixel value. Is output as the pixel value of the output coordinate α6 in the output image. The second conversion unit 22 outputs the pixel values of the output coordinates in the second image area 42 to the synthesis unit 25. As a result, the second conversion unit 22 outputs the image before the second distortion correction as the image after the second distortion correction. For convenience of explanation, in the example of FIG. 8, a case where all the pixel values of the second input coordinates in the input image are output as the pixel values of the output coordinates is illustrated. However, the second conversion unit 22 actually sets the pixel value of the second input coordinate in the second image region 42 after correcting the second distortion in the second image region 42 in the input image. It is output to the compositing unit 25 as a pixel value of the output coordinates.

FIG. 9A is an explanatory diagram showing an example of the output image after the first distortion correction. There is a first image region 41 in the central portion of the output image shown in FIG. 9A. FIG. 9B is an explanatory diagram showing the first image region 41 in the output image after the first distortion correction and the image regions other than the first image region 41 in light and dark. The first image area 41 in the output image can be represented by a white background as shown in FIG. 9B. The image areas other than the first image area 41 in the output image are the second image area 42 and the third image area 43, and can be represented by a black background as shown in FIG. 9B. FIG. 9C is an explanatory diagram showing the first image area 41, the second image area 42, and the third image area 43 in the output image in light and dark. In the output image shown in FIG. 9C, the first image area 41 is a white background, the second image area 42 is a black background, and the third image area 43 is the boundary between the first image area 41 and the second image area 42. Can be expressed by gradation. A white background indicates that it is the first image area 41 because the brightness is 100%, and a black character indicates that it is the second image area 42 because the brightness is 0%. A portion of less than 100% and more than 0%, which is black, indicates a third image region 43. This lightness and darkness is the ratio of the weighted average in the third distortion correction, which is a new output coordinate by weighted averaging the output coordinates output by the first distortion correction and the image and the output coordinates output by the second distortion correction. Shown.

Here, the weighted average described above will be described in detail. The ratio of each output coordinate in the third image area 43 is the first distance from the output coordinate in the third image area 43 to the output coordinate in the nearest first image area 41 and the third image area 43. It is a ratio according to the ratio from the output coordinates in the image area 43 to the second output coordinates in the nearest second image area 42. For example, when the ratio of the first distance to the second distance to the output coordinates is 5: 5, the calculation unit 23 calculates 50% as the ratio of the output coordinates. Further, for example, when the ratio of the first distance to the second distance to the output coordinates is 7: 3, the calculation unit 23 calculates 70% as the ratio of the output coordinates. Further, for example, when the ratio of the first distance to the second distance to the output coordinates is 3: 7, the calculation unit 23 calculates 30% as the ratio of the output coordinates. The ratio X of the output coordinates in the third image area 43 is 0% <X <100%.

Further, in the case of the output coordinates in the first image area 41, the calculation unit 23 has a ratio of the first distance to the second distance to the output coordinates of 10: 0, so that the ratio of the output coordinates is 100%. Is calculated. The ratio of the output coordinates in the first image area 41 is 100%. Further, in the case of the output coordinates in the second image area 42, the calculation unit 23 has a ratio of the first distance to the second distance to the output coordinates of 0:10, so that the ratio of the output coordinates is 0%. Is calculated. The ratio of the output coordinates in the second image area 42 is 0%. Then, when the calculation unit 23 calculates the ratio for each output coordinate, the calculation unit 23 stores the ratio for each output coordinate in the ratio table 33.

FIG. 10 is an explanatory diagram showing an example of the table configuration of the ratio table 33. The ratio table 33 shown in FIG. 10 manages the ratios for each output coordinate in the output image in association with each other. The ratio is the ratio of the weighted average of the first input coordinate and the second input coordinate for each output coordinate used when calculating the third input coordinate of the pixel value corrected for the third distortion.

The calculation unit 23 calculates, for example, the first input coordinates of the pixel value corrected for the first distortion by f (x, y). The calculation unit 23 calculates, for example, the second input coordinates of the pixel value corrected for the second distortion by g (x, y). FIG. 11 is an explanatory diagram showing an example of a calculation method when calculating the third input coordinates for correcting the third distortion. FIG. 12 is an explanatory diagram showing an example of the table configuration of the third conversion table 34. The calculation unit 23 performs a weighted average processing. More specifically, as shown in FIG. 11, when the ratio of the output coordinates is n%, the ratio of the first input coordinates is corrected by the first input coordinates f (x, y) × n%. Further, the calculation unit 23 proportion-corrects the second input coordinate with the second input coordinate g (x, y) × (100% −n%). Further, the calculation unit 23 adds the first input coordinate after the ratio correction and the second input coordinate after the ratio correction to calculate the third input coordinate for each output coordinate. That is, the calculation unit 23 calculates the third input coordinate for each output coordinate by (Formula 1) of f (x, y) × n% + g (x, y) × (100% −n%). The calculation unit 23 refers to the ratio corresponding to the output coordinates in the ratio table 33, and for example, when the ratio of the output coordinates is 50%, the output coordinates are set to 50% of the first input coordinates corresponding to the output coordinates. Add 50% of the corresponding second input coordinates. Then, the calculation unit 23 calculates the added value as the third input coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34 as shown in FIG.

For example, when the ratio of the output coordinates is 70%, the calculation unit 23 adds 70% of the first input coordinates corresponding to the output coordinates and 30% of the second input coordinates corresponding to the output coordinates. Then, the calculation unit 23 calculates the added value as the third input coordinate of the output coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34 as shown in FIG. ..

For example, when the ratio of the output coordinates is 30%, the calculation unit 23 adds 30% of the first input coordinates corresponding to the output coordinates and 70% of the second input coordinates corresponding to the output coordinates. Then, the calculation unit 23 calculates the added value as the third input coordinate of the output coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34 as shown in FIG. ..

The third conversion unit 24 refers to the third conversion table 34, and sets the pixel value of the third input coordinate in the third image area 43 after correcting the third distortion in the third image area 43. It is output to the compositing unit 25 as a pixel value of the output coordinates.

That is, the first conversion unit 21 synthesizes the pixel value of the first input coordinate corrected for the distortion in the first image area 41 in the input image as the pixel value of the output coordinate in the first image area 41. Output to unit 25. The second conversion unit 22 uses the pixel value of the second input coordinate corrected for the distortion in the second image area 42 in the input image as the pixel value of the output coordinate in the second image area 42 as the composition unit 25. Output to. The third conversion unit 24 uses the pixel value of the third input coordinate corrected for distortion in the third image area 43 in the input image as the pixel value of the output coordinate in the third image area 43 as the composition unit 25. Output to.

The compositing unit 25 includes the pixel values of the output coordinates in the first image area 41 from the first conversion unit 21 and the pixel values of the output coordinates in the second image area 42 from the second conversion unit 22. , The output image obtained by combining the pixel values of the output coordinates in the third image area 43 from the third conversion unit 24 is output to the navigation device 3. FIG. 13 is an explanatory diagram showing an example of an output image. As shown in FIG. 13, the output image has a first image region 41 corrected for the first distortion, a second image region 42 corrected for the second distortion, and a third image corrected for the third distortion. It is an image including the image area 43 of 3. In the output image, for example, a third image area 43 is arranged between the first image area 41 and the second image area 42.

Then, the navigation device 3 draws the advance route 50 in the area on the first image area 41 in the output image from the compositing unit 25, and as shown in FIG. 13, displays the drawn output image on the display device 4. Output. The traveling route 50 has, for example, a width of about the width of a vehicle and a depth of 2 meters.

Next, the operation of the image display system 1 of the prerequisite technology will be described. For convenience of explanation, it is assumed that the contents of the first conversion table 31, the second conversion table 32, the third conversion table 34, and the ratio table 33 are stored in advance.

The first conversion unit 21 referred to the first conversion table 31 and corrected the first distortion for each output coordinate in the first image region 41 in the input image captured by the image sensor 12. The pixel value of the first input coordinate is output to the compositing unit 25 as the pixel value of the output coordinate in the first image area 41. The second conversion unit 22 referred to the second conversion table 32 and corrected the second distortion for each output coordinate in the second image region 42 in the input image captured by the image sensor 12. The pixel value of the second input coordinate is output to the compositing unit 25 as the pixel value of the output coordinate in the second image area 42. The third conversion unit 24 referred to the third conversion table 34 and corrected the third distortion for each output coordinate in the third image region 43 in the input image captured by the image sensor 12. The pixel value of the third input coordinate is output to the compositing unit 25 as the pixel value of the output coordinate in the third image area 43.

The compositing unit 25 includes the pixel values of the output coordinates in the first image area 41 from the first conversion unit 21 and the pixel values of the output coordinates in the second image area 42 from the second conversion unit 22. , The pixel value of the output coordinates in the third image area 43 from the third conversion unit 24 is combined to generate an output image. The compositing unit 25 outputs the generated output image to the navigation device 3. The navigation device 3 draws the advance line 50 in the area of the first image area 41 in the output image from the compositing unit 25, and outputs the output image in which the advance line 50 is drawn to the display device 4. In generating the advance route 50, the same correction process as the first distortion correction applied to the first image area 41 is performed on the original data of the advance route 50. As a result, the user can superimpose and display on the output image how the vehicle travels on the output image of the route 50.

As described above, the navigation device 3 superimposes the forward route obtained by performing the same image conversion on the forward route data in the image portion of the first distortion correction having only distortion aberration or no distortion as described above. .. As described above, since the conversion table for performing this image conversion can be stored in a memory having a small capacity, the memory capacity can be reduced. The cost increase due to the increase in memory capacity is suppressed.

The display device 4 executes image conversion processing on the data of the traveling route 50 in which the vehicle actually travels so as to match the image of the distortion of the lens 11 of the image pickup device 2 actually mounted on the vehicle. The traveling route 50 is drawn and displayed on the image area 41 of 1. Since the lens 11 of the image pickup device 2 is, for example, a fisheye lens, the image captured by the image pickup device 2 is an image of distortion in a field of view having an angle of view of 180 degrees or more horizontally. However, the navigation device 3 corresponds to an image of distortion aberration in a field of view of about 140 degrees horizontally captured by a conventional lens instead of a fisheye lens so that the advance line 50 can be drawn. That is, the first image region 41 on which the advance line 50 is drawn is an image of distortion aberration in a field of view having an angle of view of about 140 degrees horizontally. Therefore, the image of the distortion aberration differs depending on the angle of view difference between the image captured by the fisheye lens and the image that enables the navigation device 3 to draw the advance line 50.

FIG. 14 is an explanatory diagram showing an example of an input image (original image) captured by the imaging device 2 of the prerequisite technology. The image shown in FIG. 14 is an example of an image captured by the fisheye lens which is the lens 11 of the image pickup apparatus 2, and is an image including a predetermined image region S for drawing the advance route 50. The predetermined image area S shown in FIG. 14 is a range actually drawn on the ground for convenience of explanation, and includes a line having a depth of 50 cm, a line having a depth of 100 cm, and a line having a depth of 200 cm from the rear end of the vehicle. It shall include a line indicating the vehicle width.

FIG. 15 is an explanatory diagram showing an example of the image after the first distortion correction. The image shown in FIG. 15 is an image in which the image processing of the first distortion correction is executed by the first conversion unit 21 for each pixel in the original image shown in FIG. FIG. 16 is an explanatory diagram showing an example of the image after the second distortion correction. The image shown in FIG. 16 is an image in which the image processing of the second distortion correction is executed by the second conversion unit 22 for each pixel in the original image shown in FIG.

FIG. 17 is an explanatory diagram showing the first image region 41, the second image region 42, and the third image region 43 of the prerequisite technology in light and dark. In the image shown in FIG. 17, the first image area 41 is a white background, the second image area 42 is a black background, and the third image area 43, which is the boundary between the first image area 41 and the second image area 42, is Expressed in gradation. The white background indicates that it is the first image region 41 because the brightness is 100%. The black background indicates that it is the second image region 42 because the brightness is 0%. A portion where the light and darkness is less than 100% white and more than 0% black indicates that the third image region 43 is present. This lightness and darkness is the ratio of the weighted average in the third distortion correction, which is a new output coordinate by weighted averaging the output coordinates output by the first distortion correction and the image and the output coordinates output by the second distortion correction. Shown. The image pickup apparatus 2 will generate a third conversion table 34 from the ratio table 33 according to the ratio of light and dark shown in FIG. Then, the image pickup apparatus 2 refers to the third conversion table 34 and executes distortion correction for each pixel in the first image area 41, the second image area 42, and the third image area 43.

FIG. 18 is an explanatory diagram showing an example of an output image having poor visibility in the prerequisite technique. The output image shown in FIG. 18 is the pixel value of the output coordinates in the first image area 41 from the first conversion unit 21 and the output coordinates in the second image area 42 from the second conversion unit 22. This is an output image obtained by synthesizing the pixel values and the pixel values of the output coordinates in the third image area 43 from the third conversion unit 24.

However, in the image pickup apparatus 2, the regions other than the first image region 41 (second image regions 42 and third) while leaving the distortion of the conventional lens in the first image region 41 for drawing the traveling path 50. It is assumed that the image conversion process is executed in the image area 43) of. In this case, the first image area 41 is an image of distortion aberration in a field of view of about 140 degrees horizontally so that the navigation device 3 can draw the advance line 50. On the other hand, the image other than the first image region 41 is an image of distortion aberration in a field of view of 180 degrees or more horizontally. An angle of view difference occurs between the first image area 41 and the area other than the first image area 41. Therefore, an extreme change in distortion occurs in the vicinity of the boundary line between the first image region 41 and the region other than the first image region 41 (second image region 42 and third image region 43). That is, the straight line near the vanishing point in the first image area 41 is not a straight line but a distorted state. As a result, the visibility of the image is reduced due to the extreme change in distortion near the boundary line.

Therefore, for example, in order to provide an image display system 1 capable of improving image visibility, an embodiment thereof will be described below as Example 1. The same configurations as those of the prerequisite technology are designated by the same reference numerals, and the description of the overlapping configurations and operations will be omitted.

FIG. 19 is an explanatory diagram showing an example of the configuration of the image pickup apparatus 2A of the first embodiment. The control unit 20 in the image pickup apparatus 2A shown in FIG. 19 includes a first conversion unit 21, a second conversion unit 22, a calculation unit 23, a third conversion unit 24, and a synthesis unit 25. The storage unit 30 in the image pickup apparatus 2A has a first conversion table 31, a distortion table 31A, a second conversion table 32, a ratio table 33, and a third conversion table 34.

The first conversion table 31 is, for example, a conversion table that corrects the first image area 41 in the original image and manages the input coordinates and the output coordinates for each pixel value in the corrected image area 41α of the first α. is there. The corrected image region 41α of the first α reduces the range of the first image region 41 from the original image shown in FIG. 14 and moves upward, for example, 82.5% laterally as shown in FIG. This is a corrected image that has been reduced to 92.8% vertically and moved upward by 20 dots. The corrected image has a first α image region 41α, a second α image region 42α, and a third α image region 43α. As a result, the visibility of the image near the boundary line between the image region 41α of the first α and the image region other than the image region 41α of the first α is improved. That is, the image region 41α of the first α corrects the range of the first image region 41 from the original image so that the visibility of the image near the boundary line is improved.

The ratio table 33 is a table in which the ratio of each pixel in the image region 41α of the first α, the image region 42α of the second α, and the image region 43α of the third α is determined. The ratio is the ratio of the weighted average of the first input coordinate and the second input coordinate for each output coordinate used when calculating the third input coordinate of the pixel value corrected for the third distortion. .. That is, the ratio table 33 defines the ratio of the weighted average of the first input coordinates and the second input coordinates for each output coordinate in the image including the image region 41α of the first α.

The calculation unit 23 refers to the first conversion table 31, and calculates, for example, the first input coordinates of the pixel value corrected for the first distortion by f (x, y). The calculation unit 23 refers to the second conversion table 32, and calculates, for example, the second input coordinates of the pixel value corrected for the second distortion by g (x, y). The calculation unit 23 refers to the ratio table 33 and calculates, for example, the third input coordinates of the pixel value corrected for the third distortion by (Equation formula 1). The calculation unit 23 performs a weighted average processing. As shown in FIG. 15, the calculation unit 23 refers to the ratio table 33, and when the ratio of the output coordinates is n%, the first input coordinates f (x, y) × n% are the first input coordinates. Percentage correction. Further, the calculation unit 23 refers to the ratio table 33 and corrects the ratio of the second input coordinate with the second input coordinate g (x, y) × (100% −n%). Further, the calculation unit 23 adds the first input coordinate after the ratio correction and the second input coordinate after the ratio correction to calculate the third input coordinate for each output coordinate. That is, the calculation unit 23 refers to the ratio table 33 and uses (formula 1) of f (x, y) × n% + g (x, y) × (100% −n%) to obtain the third input coordinate. Calculated for each output coordinate. The calculation unit 23 refers to the ratio corresponding to the output coordinates in the ratio table 33, and for example, when the ratio of the output coordinates is 50%, the output coordinates are set to 50% of the first input coordinates corresponding to the output coordinates. Add 50% of the corresponding second input coordinates. Then, the calculation unit 23 calculates the added value as the third input coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34.

FIG. 20 is an explanatory diagram showing an example of the image after the first distortion correction. The image shown in FIG. 20 is an image including the image region 41α of the first α corrected for the first image region 41 on the image after the first distortion correction of the original image captured by the image pickup apparatus 2A. The first conversion unit 21 executes the first distortion correction process from the original image captured by the image pickup apparatus 2A. Further, the first conversion unit 21 corrects the first image region 41 after the first distortion correction from the original image after the first distortion correction to obtain the first image region 41α. The image region 41α of the first α is, for example, an image in which the range of the first image region 41 is reduced to 82.5% in the horizontal direction and 92.8% in the vertical direction and moved upward by 20 dots. The reduction ratio and the moving direction of the first image region 41 are arbitrarily adjusted at the time of design so as to reduce the visual discontinuity of the boundary portion with the image region 41α of the first α. The reduction ratio of the first image area 41 may be 100% or more, and the moving direction is not limited to the upward direction, but in a predetermined direction determined at the time of design, such as a horizontal direction, a downward direction, or an oblique direction. All you need is.

FIG. 21 is an explanatory diagram showing the image region 41α of the first α, the image region 42α of the second α, and the image region 43α of the third α in light and dark according to the first embodiment. In the image shown in FIG. 21, the first α image region 41α is a white background, the second α image region 42α is a black background, and the third α image region 43α, which is the boundary between the first α image region 41α and the second α image region 42α, is Expressed in gradation. The white background indicates that the image region 41α is the first α because the brightness is 100%. The black background indicates that the image region 42α is the second α because the brightness is 0%. The portion where the light and darkness is less than 100% white and exceeds 0% black indicates that the image region 43α of the third α. This lightness and darkness is the ratio of the weighted average in the third distortion correction to obtain a new output coordinate by weighted averaging the output coordinates output by the first distortion correction and the image and the output coordinates output by the second distortion correction. Is shown.

The first conversion unit 21 refers to the first conversion table 31, and sets the pixel value of the first input coordinate in the image region 41α of the first α in the input image as the image of the first α. It is output to the compositing unit 25 as a pixel value of the output coordinates in the region 41α. The second conversion unit 22 refers to the second conversion table 32, and sets the pixel value of the second input coordinates in the image region 42α of the second α in the input image as the image of the second α. It is output to the compositing unit 25 as a pixel value of the output coordinates in the region 42α. The third conversion unit 24 refers to the third conversion table 34 and sets the pixel value of the third input coordinate in the image region 43α of the third α in the input image as the third α. It is output to the compositing unit 25 as a pixel value of the output coordinates in the image area 43α.

The compositing unit 25 includes the pixel values of the output coordinates in the image region 41α of the first α from the first conversion unit 21 and the pixel values of the output coordinates in the image region 42α of the second α from the second conversion unit 22. , The output image obtained by combining the pixel values of the output coordinates in the image region 43α of the third α from the third conversion unit 24 is output to the navigation device 3.

FIG. 22 is an explanatory diagram showing an example of an output image when the image pickup device 2A captures the image. As shown in FIG. 22, the output image has a first distortion-corrected image region 41α, a second distortion-corrected second α image region 42α, and a third distortion-corrected third image region. It is an image including the image region 43α of 3α. In the output image, for example, the image region 43α of the third α is arranged between the image region 41A of the first α and the image region 42α of the second α. Even when the input image is acquired, the compositing unit 25 can smoothly output an output image having excellent visibility. The straight line near the vanishing point in the image region 41α of the first α in the image shown in FIG. 22 is in a substantially straight straight line state as compared with FIG. As a result, there is little change in distortion near the boundary line, and the visibility of the image is improved. Then, the navigation device 3 draws the advance line 50 in the range on the image region 41α of the first α in the output image from the compositing unit 25 even in the captured image, and as shown in FIG. 22, the output image after drawing. Is output to the display device 4. Although the navigation device 3 is different from the lens image having the actual distortion, the screen coordinates are used, for example, by using correction amounts such as 82.5% in the horizontal direction, 92.8% in the vertical direction, and 20 dots in the upward direction. Is modified to draw the advance line 50 on the image area 41α of the first α.

In the image pickup apparatus 2A of the first embodiment, the image region 42α of the second α and the image region 43α of the third α are specified based on the image region 41α of the first α after the correction of the range of the first image region 41. Further, the image pickup apparatus 2A is a ratio table 33 showing the ratio of the first input coordinates and the second input coordinates for each pixel in the image region 41α of the first α, the image region 42α of the second α, and the image region 43α of the third α. To generate. Further, the image pickup apparatus 2A generates a third conversion table 34 based on the ratio table 33. Then, the image pickup apparatus 2A refers to the third conversion table 34 and executes the image conversion process of distortion correction. As a result, the image pickup apparatus 2A promptly corrects the first distortion from the captured input image even when the angle of view difference occurs between the first image area 41 and the image areas other than the first image area 41. The second distortion correction and the third distortion correction can be smoothly realized. That is, it is possible to minimize the portion that corrects only the distortion of the lens itself, which is insufficient for easy-to-see distortion correction, and it is possible to maintain good visibility by increasing the area that looks natural.

That is, the image pickup apparatus 2A generates the third conversion table 34 based on the corrected image region 41α of the first α after correcting the first image region 41. As a result, even when the angle of view difference occurs, the straight line near the vanishing point of the image near the boundary line between the image region 41α of the first α and the image region other than the image region 41α of the first α becomes substantially straight. Therefore, since the visibility near the boundary line is improved, the output image can be output with good visibility. Moreover, since the display device 4 does not require a table for associating the output coordinates with the input coordinates, the memory capacity can be reduced and the cost increase of the product can be suppressed.

In the image pickup apparatus 2A of the first embodiment, the case of moving upward is illustrated as an example of correcting the range of the first image region 41, but the case is not limited to the upward direction, and for example, the vertical direction or It can be changed in the left-right direction as appropriate. Further, in the image pickup apparatus 2A, the case where the range of the first image region 41 is reduced and moved is illustrated, but only one of the reduction and the movement may be executed, which can be changed as appropriate. Further, in the image pickup apparatus 2A, the range of the first image region 41 may be expanded or expanded and moved, and can be changed as appropriate. Further, in the image pickup apparatus 2A, the range of the first image area 41 is laterally extended so that the visibility of the image near the boundary line between the first image area 41 and the image area other than the first image area 41 is improved. It may be corrected so as to expand in the direction and decrease in the vertical direction, and can be changed as appropriate.

In the image pickup apparatus 2A, the case where the range of the first image region 41 is corrected has been illustrated. For example, the first image region is obtained by applying the fourth distortion correction that leaves a distortion aberration different from that of the first distortion correction. 41 may be replaced, and the embodiment thereof will be described below as Example 2. By assigning the same reference numerals to the same configurations as in the first embodiment, the description of the overlapping configurations and operations will be omitted.

The first conversion unit 21 generates the image region 41β of the first β, which is an image subjected to the fourth distortion correction that leaves a distortion aberration different from that of the first distortion correction of the first image region 41 of the original image. FIG. 23 is an explanatory diagram showing an example of an image including the image region 41β of the first β after the distortion correction, which is different from the first distortion correction of the second embodiment. Further, the first conversion unit 21 corrects the image region 41β of the first β, for example, by enlarging it by 1.5 times in the vertical direction and moving it by 137 dots in the upward direction. The first conversion unit 21 replaces the first image region 41 with the corrected image region 41β of the first β. The image shown in FIG. 23 is an image including the corrected image region 41β of the first β.

FIG. 24 is an explanatory diagram showing the image region 41β of the first β after the distortion correction different from the first distortion correction, the image region 42α of the second α, and the image region 43α of the third α in light and dark. In the image shown in FIG. 24, the first β image region 41β is a white background, the second α image region 42α is a black background, and the third α image region 43α, which is the boundary between the first β image region 41β and the second α image region 42α, is Expressed in gradation. The white background indicates that the image region is 41β of the first β because the brightness is 100%. The black background indicates that the image region 42α is the second α because the brightness is 0%. The portion where the light and darkness is less than 100% white and exceeds 0% black indicates that the image region 43α of the third α. This light and darkness uses the output coordinates output by the first distortion correction, the image, and the output coordinates output by the second distortion correction as inputs, and the inputs are weighted and averaged to obtain new output coordinates. It shows the percentage of the weighted average in the correction.

The first conversion unit 21 outputs the pixel value of the input coordinate in the image region 41β of the first β subjected to the fourth distortion correction to the synthesis unit 25 as the pixel value of the output coordinate in the image region 41β of the first β. .. The second conversion unit 22 refers to the second conversion table 32, and sets the pixel value of the second input coordinates in the image region 42α of the second α in the input image as the image of the second α. It is output to the compositing unit 25 as a pixel value of the output coordinates in the region 42α. The third conversion unit 24 refers to the third conversion table 34 and sets the pixel value of the third input coordinate in the image region 43α of the third α in the input image as the third α. It is output to the compositing unit 25 as a pixel value of the output coordinates in the image area 43α.

The compositing unit 25 includes the pixel values of the output coordinates in the image region 41β of the first β from the first conversion unit 21 and the pixel values of the output coordinates in the image region 42α of the second α from the second conversion unit 22. , The output image obtained by combining the pixel values of the output coordinates in the image region 43α of the third α from the third conversion unit 24 is output to the navigation device 3.

FIG. 25 is an explanatory diagram showing an example of an output image when the image pickup device 2A captures the image. As shown in FIG. 25, the output image includes an image region 41β of the first β, an image region 42α of the second α corrected for the second distortion, and an image region 43α of the third α corrected for the third distortion. It is an image including. In the output image, for example, the image region 43α of the third α is arranged between the image region 41β of the first β and the image region 42α of the second α. The compositing unit 25 can smoothly output an output image having excellent visibility. The straight line near the vanishing point in the image region 41β of the first β in the image shown in FIG. 25 is in a substantially straight straight line state as compared with FIG. As a result, there is little change in distortion near the boundary line, and the visibility of the image is improved. Then, the navigation device 3 draws the advance line 50 on the image region 41β of the first β in the output image from the compositing unit 25 even in the captured image, and displays the output image after drawing as shown in FIG. 25. Output to device 4.

In the image pickup apparatus 2A of the second embodiment, the image region 41β of the first β of the fourth distortion correction different from the first distortion correction is generated, and the first image region 41 is replaced with the image region 41β of the first β. Further, the image pickup apparatus 2A identifies the image region 42α of the second α and the image region 43α of the third α based on the image region 41β of the first β. Further, the image pickup apparatus 2A is a ratio table 33 showing the ratio of the first input coordinates and the second input coordinates for each pixel in the image region 41β of the first β, the image region 42α of the second α, and the image region 43α of the third α. To generate. Further, the image pickup apparatus 2A generates a third conversion table 34 based on the ratio table 33. Then, the image pickup apparatus 2A refers to the third conversion table 34 and executes the image conversion process of distortion correction. As a result, the image pickup apparatus 2A promptly corrects the first distortion from the captured input image even when the angle of view difference occurs between the first image area 41 and the image areas other than the first image area 41. The second distortion correction and the third distortion correction can be smoothly realized. That is, it is possible to minimize the portion that corrects only the distortion of the lens itself, which is insufficient for easy-to-see distortion correction, and it is possible to maintain good visibility by increasing the area that looks natural.

That is, the image pickup apparatus 2A generates the third conversion table 34 based on the image region 41β of the first β of the fourth distortion correction different from the first distortion correction. As a result, even when the angle of view difference occurs, the straight line near the vanishing point of the image near the boundary line between the image region 41β of the first β and the image region other than the image region 41β of the first β becomes substantially straight. Therefore, since the visibility near the boundary line is improved, it is possible to output an output image with good visibility. Moreover, since the display device 4 does not require a table for associating the output coordinates with the input coordinates, the memory capacity can be reduced and the cost increase of the product can be suppressed.

The image pickup apparatus 2A can also be applied to, for example, the case of recognizing that the vehicle is parallel parked and corresponding to the advance route 50 drawn at a completely different position.

In the image pickup apparatus 2 (2A), the storage unit 30 such as the third conversion table 34 is stored in the RAM 15 in the image pickup apparatus 2 (2A). However, it is not limited to the RAM 15, and may be stored in, for example, a storage device (not shown) that can be externally connected to the image pickup device 2 (2A), and can be changed as appropriate.

The image pickup apparatus 2 (2A) has various tables built in the storage unit 30 shown in FIG. 14, but if the table contents of the third conversion table 34 are stored in advance, the image pickup device 2 (2A) has the table contents in the storage unit 30. Only the conversion table 34 of 3 may be built-in and can be changed as appropriate.

In the image pickup apparatus 2 (2A), the first conversion table 31, the second conversion table 32, the third conversion table 34, and the ratio table 33 were used. However, for each output coordinate, a function f for calculating the first input coordinate of the pixel value corrected with the first distortion, and a second input coordinate of the pixel value corrected with the second distortion are calculated. The function g and the input coordinates of the pixel value corrected for the third distortion may be stored (Formula 1).

In this case, in the image pickup apparatus 2 (2A), the function f, the function g, and the equation 1 do not need to prepare the first conversion table 31, the second conversion table 32, the third conversion table 34, and the ratio table 33. Then, the input coordinates of the pixel values subjected to the first distortion correction, the second distortion correction, and the third distortion correction are calculated. Then, the image pickup apparatus 2 (2A) has the pixel values of the output coordinates in the image region 41α of the first α corrected for the first distortion and the output coordinates in the image region 42α of the second α corrected for the second distortion. An output image including the pixel value and the pixel value of the output coordinates in the image region 43α of the third α corrected for the third distortion is output to the navigation device 3.

In the above embodiment, the input image is stored in the RAM 15 or the like in the image pickup device 2 (2A), but the input image may be stored in an external storage device connected to the image pickup device 2 (2A). It is possible.

In the above embodiment, for example, the second distortion correction has been described as distortion correction according to Patent Document 1, but the distortion is not limited to these distortions, and other correction methods that are easy for the user to see may be adopted.

In the above embodiment, the first distortion correction in the first image area 41 (the image area 41α of the first α), the second distortion correction in the second image area 42 (the image area 42α of the second α), and the second. The image processing processor 13 in the image pickup apparatus 2 (2A) executed the distortion correction process (image processing) for executing the third distortion correction in the image area 43 (image area 43α of the third α) of 3. However, the distortion correction process may be executed on an external device connected to the image pickup device 2 (2A), for example, the navigation device 3, and can be changed as appropriate.

In the above embodiment, the first conversion unit 21 executes the first distortion correction, the second conversion unit 22 performs the second distortion correction, and the third conversion unit 24 executes the third distortion correction. However, the first distortion correction, the second distortion correction, and the third distortion correction may be executed sequentially in no particular order or in parallel, and can be changed as appropriate.

The navigation device 3 of the above embodiment uses the distortion table 31A in the image pickup device 2 (2A) to draw the advance line 50 in the area of the image region 41α of the first α. However, the first conversion table 31 may be separately provided in the navigation device 3, and can be changed as appropriate.

In the above embodiment, the advance line 50 is drawn on the output image including the first image area 41 (the image area 41α of the first α) in which the first distortion is corrected, but the advance line 50 is not limited to the advance line 50. For example, a line such as a fixed route may be drawn and can be changed as appropriate.

2A Imaging device 13 Image processing processor 21 First conversion unit 22 Second conversion unit 23 Calculation unit 24 Third conversion unit 25 Synthesis unit 31 First conversion table 32 Second conversion table 33 Ratio table 34 Third Conversion table

Claims (8)

An image processing device that converts an input image into an image and outputs an output image.
A first distortion correction is applied to each pixel in the first image region of the input image to obtain an image, and each pixel in the image to which the first distortion correction is applied is corrected and output as an image. Conversion part and
A second conversion unit that performs a second distortion correction for each pixel in the second image area of the input image and outputs the image as an image.
A third conversion unit that performs a third distortion correction for each pixel in the third image area of the input image and outputs the image as an image.
An image output from the first conversion unit, an image output from the second conversion unit, and a composition unit that generates and outputs the output image from the image output from the third conversion unit.
Have,
In the output image, the image output from the third conversion unit is arranged between the image output from the first conversion unit and the image output from the second conversion unit. An image processing apparatus characterized in that the image output from the first conversion unit is an image having only distortion or no distortion. The first conversion unit is
The image processing apparatus according to claim 1, wherein the pixels in the image are corrected so that the straight line near the vanishing point in the first image region becomes a straight line. The first conversion unit is
The image processing apparatus according to claim 1 or 2, wherein the image to which the first distortion correction has been applied is corrected so as to be enlarged or reduced. The first conversion unit is
The image processing apparatus according to claim 1 or 2, wherein the image to which the first distortion correction has been applied is corrected so as to move in a predetermined direction. The input image is generated from a subject, a lens that collects light from the subject, and an image sensor that forms an image of the collected light.
The first conversion unit is
The image according to claim 1 or 2, wherein the first image region is corrected so as to replace the image with a fourth distortion correction that leaves a distortion aberration different from that of the first distortion correction of the lens. Processing equipment. The input image is generated from a subject, a lens that collects light from the subject, and an image sensor that forms an image of the collected light.
The second conversion unit is
The subject that is linear toward the optical axis of the lens and the subject that is linear in the vertical direction are substantially linear in the output image, and the subject that is linear in the horizontal direction is the central portion of the output image. , And the left side portion and the right side portion connected to the left and right sides of the center portion are substantially linear, and the distortion is corrected so as to bend near the boundary between the center portion of the output image and the left side portion and the right side portion. The image processing apparatus according to claim 1 or 2. The third conversion unit is
The first or second claim is characterized in that the output obtained by applying the first distortion correction and the output obtained by applying the second distortion correction are weighted averaged and output for each pixel of the input image. The image processing apparatus described. The image processing apparatus according to claim 1 or 2, wherein a route is drawn in the first image area.

Images