JP7081265B2 - Image processing equipment - Google Patents

Description

The present invention relates to an image processing apparatus.

For example, there is an image processing device that converts and outputs an image of a subject captured by an in-vehicle camera using a fisheye lens so as to correct distortion (for example, Patent Document 1).

In such an image processing device, the entire input image is converted so that the linear subject toward the optical axis of the fisheye lens and the linear subject in the vertical direction are substantially linear in the output image. Further, in the image processing device, a subject that is linear in the horizontal direction is placed in a substantially linear shape at the center portion of the output image and the left and right portions that are connected to the center portion to the left and right, and the center portion of the output image. The entire input image is image-converted so as to bend near the boundary between the left side portion and the right side portion. As a result, the image processing device provides an output image in which a wide-angle image that does not give a sense of discomfort to the user is easy to see.

Further, there is known a technique of superimposing a traveling route in a direction in which a vehicle travels in a display device using an output image output from an image processing device as an input (for example, Patent Document 2).

Japanese Unexamined Patent Publication No. 2008-31890 Japanese Unexamined Patent Publication No. 11-334470

In such a display device, generally, the image of the route is converted into an image so as to match the image deformed by the curvature aberration of the fisheye lens or other image conversion processing. The course on which the vehicle actually travels is superimposed and displayed on the image output by the device. Therefore, the image conversion process for the above-mentioned route data is the same image conversion process as the image conversion process for the image captured by the in-vehicle camera performed in the image processing device.

More specifically, on the display device side, for example, for each output coordinate (X, Y) of the output image of the display device, a table that associates the input coordinates (x, y) with the input of the advance route data stored in advance. Is used to perform image conversion processing to refer to (transfer) the brightness and color information of the pixels of the output coordinates (X, Y) and the brightness and color information of the pixels of the input coordinates (x, y). Can be considered. In such processing, in general, conversion data in which the input coordinates corresponding to the coordinates are tableized is stored in advance for the output coordinates, and the conversion processing is performed according to the table.

However, such tableized conversion data requires input coordinates (x, y) for the number of pixels of the output image, and the capacity increases in line with the recent trend toward higher pixel counts in display devices, resulting in an increase in memory cost. There was a problem of inviting.

In view of such a problem, it is an object of the present invention to provide an image processing device or the like capable of superimposing a route while suppressing an increase in the memory of a display device.

The image processing device of one embodiment is an image processing device that converts an input image into an image and outputs an output image, and is a first conversion unit, a second conversion unit, a third conversion unit, and a composition unit. And have. The first conversion unit outputs an image to which the first distortion correction is applied for each pixel in the first image region of the input image. The second conversion unit outputs an image to which the second distortion correction is applied for each pixel in the second image region of the input image. The third conversion unit outputs an image to which the third distortion correction is applied for each pixel in the third image region of the input image. The compositing unit generates and outputs the output image from the image output from the first conversion unit, the image output from the second conversion unit, and the image output from the third conversion unit. In the output image, the image output from the third conversion unit is arranged between the image output from the first conversion unit and the image output from the second conversion unit, and the first conversion unit is used. The image output from the conversion unit of is an image having only distortion or no distortion.

According to the present invention, it is possible to reduce the capacity of the table while maintaining an easy-to-see output image that does not give a sense of discomfort to the user.

FIG. 1 is an explanatory diagram showing an example of the hardware configuration of the image display system of the first embodiment. FIG. 2 is a block diagram showing an example of the functional configuration of the image pickup apparatus of the first embodiment. FIG. 3 is an explanatory diagram showing an example of an input image captured by the image pickup apparatus. FIG. 4 is an explanatory diagram showing an example of the table configuration of the first conversion table. FIG. 5 is an explanatory diagram showing an example of an image before and after the first distortion correction of the first conversion unit. FIG. 6A is an explanatory diagram showing an example of an image transformation model. FIG. 6B is an explanatory diagram showing an example of the table configuration of the second conversion table. FIG. 7 is an explanatory diagram showing an example of an image before and after the second distortion correction of the second conversion unit. FIG. 8A is an explanatory diagram showing an example of an output image. FIG. 8B is an explanatory diagram showing the first image region in the output image and the image region other than the first image region in terms of luminance. FIG. 8C is an explanatory diagram represented by a first image region, a second image region, a third image region, and brightness in the output image. FIG. 9 is an explanatory diagram showing an example of the table configuration of the ratio table. FIG. 10 is an explanatory diagram showing an example of a calculation method when calculating the third input coordinates for correcting the third distortion. FIG. 11 is an explanatory diagram showing an example of the table configuration of the third conversion table. FIG. 12 is an explanatory diagram showing an example of an output image. FIG. 13 is an explanatory diagram showing an example of the relationship between the real image height and the ideal image height.

Hereinafter, examples of the image processing apparatus disclosed in the present application will be described in detail with reference to the drawings. The disclosed technique is not limited by each embodiment. In addition, the examples shown below may be appropriately combined as long as they do not cause a contradiction.

FIG. 1 is an explanatory diagram showing an example of the hardware configuration of the image display system 1 of the first embodiment. The image display system 1 shown in FIG. 1 includes an image pickup device 2, a navigation device 3, and a display device 4. The image pickup apparatus 2 includes a lens 11, an image pickup element 12, an image processing processor 13, a ROM (Read Only Memory) 14, and a RAM (Radom Access Memory) 15. The image pickup device 2 is, for example, a camera device arranged near the license plate at the rear of the vehicle. The lens 11 is, for example, a fisheye lens. The image sensor 12 is a CMOS sensor that captures an image of a specific range behind the vehicle through the lens 11. The image processing processor 13 is an image processing device that executes image processing inside the image pickup device 2. The ROM 14 stores information such as various programs. The RAM 15 stores various information.

The navigation device 3 outputs guidance information such as map information to the display device, draws the advance route 50 on the output image from the image pickup device 2, and outputs the output image on which the advance route 50 is drawn to the display device 4. The display device 4 is, for example, a monitor device arranged on the front panel of the driver's seat and displaying and outputting an output image and map information from the navigation device 3, for example. The navigation device 3 is set to draw a route 50 that has been image-converted so as to match the output image after distortion correction of the image.

FIG. 2 is an explanatory diagram showing an example of the configuration of the image pickup apparatus 2 of the first embodiment. The image pickup apparatus 2 shown in FIG. 2 includes a control unit 20 and a storage unit 30. The control unit 20 corresponds to the image processor 13 in FIG. 1, and the storage unit 30 corresponds to the ROM 14 in FIG. The control unit 20 includes a first conversion unit 21, a second conversion unit 22, a calculation unit 23, a third conversion unit 24, and a synthesis unit 25. The storage unit 30 has a first conversion table 31, a second conversion table 32, a ratio table 33, and a third conversion table 34.

FIG. 3 is an explanatory diagram showing an example in which an image captured by the image pickup device 2 is displayed on the display device 4. The input image 40 shown in FIG. 3 has a first image area 41, a second image area 42, and a third image area 43. The image processor 13 identifies the first image area 41, the second image area 42, and the third image area 43 in the input image. That is, the image processing processor 13 identifies each pixel value in the input image by the input coordinates, and each pixel value in the first image area 41 is the first input coordinate, and each pixel in the second image area 42. The values are identified by the second input coordinates, the third input coordinates of each pixel value in the third image area 43.

The first image region 41 surrounded by the broken line L1 corresponds to, for example, the image region of the central portion in the input image, and the distortion is corrected so as to have only distortion or no distortion (for example). It is a target area to be subjected to (first distortion correction), and is an image area for drawing a traveling line that changes according to a steering angle. The second image region 42 surrounded by the alternate long and short dash line L2 is, for example, a distortion correction (second) that displays a straight line object as shown in the above-mentioned patent document 1 as a straight line and does not give a sense of discomfort to the user. This is the image area to be corrected for distortion. This part has no distortion. The third image region 43 surrounded by the broken line L1 and the alternate long and short dash line L2 is an image region for performing a third strain correction by weighted averaging the first strain correction and the second strain correction. The weighted average will be described later.

The first conversion unit 21 is a conversion unit that performs the first distortion correction for each pixel in the first image region 41 of the input image 40 and outputs it. The pixels in the first image area 41 can be said to be, for example, the pixel values of the first input coordinates. The first conversion unit 21 refers to, for example, the first conversion table 31 and outputs the pixel values of the first input coordinates in the output image after correcting the distortion in the first image area 41 in the input image 40. It is output as the pixel value of the output coordinates. The first conversion table 31 is a table that manages the first input coordinates of the pixel values corrected for distortion in the input image in association with each output coordinate of the output image. The first conversion unit 21 outputs the pixel values for each output coordinate in the first image area 41 of the output image to the synthesis unit 25.

Here, the operation of the navigation device 3 to convert the route 50 so as to match the output image after distortion correction of the image will be described. For example, assuming that the image in the first image region 41 has distortion, the navigation device 3 performs image conversion on the route data so as to have the same aberration as the above-mentioned distortion. When there is distortion, the real image height and the ideal image height are different. FIG. 13 is an explanatory diagram showing an example of the relationship between the real image height and the ideal image height. The navigation device 3 converts the pixel value of the polar coordinates (R, φ) of the ideal image height into the polar coordinates (r, φ) of the real image height. This conversion is a conversion that moves an arbitrary pixel on the line L of FIG. 13 to another place on the same line L, and this conversion table inputs an input image for each output coordinate (X, Y) of the output image. The size of the data is smaller than that of the conversion table associated with the coordinates (x, y). The image in the first image area 41 may be distortion-free. At this time, in the conversion from the pixel value of the polar coordinates (R, φ) in the image to the pixel value of the polar coordinates (r, φ), the above-mentioned pixel movement does not occur, so that the conversion table becomes unnecessary.

The second conversion unit 22 is a conversion unit that performs and outputs a second distortion correction for each pixel in the second image region 42 of the input image 40. The pixels in the second image area 42 can be said to be, for example, the pixel values of the second input coordinates. The second conversion unit 22 refers to, for example, the second conversion table 32, and obtains the pixel values of the second input coordinates in the output image after correcting the distortion in the second image area 42 in the input image 40. It is output as the pixel value of the second output coordinate. The second conversion table is a table that manages the second input coordinates of the pixel value corrected for the second distortion in association with each other for each second input coordinate in the second image area 42. The second conversion unit 22 outputs the pixel value for each output coordinate in the second image area 42 of the output image to the synthesis unit 25.

The calculation unit 23 is in the first image area 41 closest to the output coordinates in the third image area 43 for each output coordinate of the pixel value corrected for the distortion in the third image area 43 in the input image 41. The ratio of the first distance to the output coordinates of the above and the second distance from the output coordinates in the third image area 43 to the output coordinates in the nearest second image area 42 is calculated. Further, the calculation unit 23 stores the ratio as a ratio in the ratio table 33 for each output coordinate. The calculation unit 23 weights and averages the output obtained by performing the first strain correction for each pixel of the input image 41 and the output obtained by performing the second strain correction. For example, the calculation unit 23 refers to the ratio table 33 and corrects the first input coordinate and the second input coordinate according to the ratio for each output coordinate (first distance: second distance). The calculation unit 23 calculates the third input coordinate of the output image by adding the first input coordinate after the ratio correction and the second input coordinate after the ratio correction.

The calculation unit 23 stores the third input coordinates in the third conversion table 34 for each output coordinate. The third conversion unit 24 is a conversion unit that applies and outputs a third distortion correction for each pixel in the third image region 43 of the input image. The pixels in the third image region 43 can be said to be, for example, the pixel values of the third input coordinates. The third conversion unit 24 refers to, for example, the third conversion table 34, and uses the pixel value of the third input coordinate corrected for the distortion in the third image region 43 in the input image 40 as the third image. It is output as a pixel value of the output coordinates in the area 43. The third conversion unit 24 outputs the pixel value for each output coordinate in the third image area 43 to the composition unit 25.

FIG. 5 is an explanatory diagram showing an example of an image before and after the first distortion correction of the first conversion unit 21. The first conversion unit 21 refers to the first conversion table 31, and outputs the pixel value of the first input coordinate corrected for distortion as the pixel value of the output coordinate for each output coordinate. The first conversion unit 21 outputs the pixel values of the output coordinates in the first image area 41 to the composition unit 25. As a result, the first conversion unit 21 can output the image before the first distortion correction as the image after the first distortion correction as shown in FIG. The first conversion unit 21 uses, for example, the pixel value of the input coordinate β1 in the input image as the pixel value of the output coordinate α1 in the output image, and the pixel value of the input coordinate β3 in the input image as the output coordinate in the output image. It is output as a pixel value of α3. For convenience of explanation, in the example of FIG. 5, the case where the pixel values of all the first input coordinates in the input image are output as the pixel values of the output coordinates is illustrated. However, in reality, the first conversion unit 21 uses the pixel value of the first input coordinate corrected for the distortion in the first image area 41 in the input image as the output coordinate in the first image area 41. It is output to the compositing unit 25 as a pixel value.

FIG. 6A is an explanatory diagram showing an example of an image transformation model. The image conversion model shown in FIG. 6A includes an input image 51 formed of a circular plane, a hemispherical virtual object surface 52 corresponding to the lens 11 on which a subject is projected, a correction surface 53 for correcting distortion, and an image conversion. The display surface 54 to be displayed is arranged side by side in order. The virtual optical axis z corresponding to the optical axis of the lens 11 passes through the center of these surfaces at a right angle from the optical axis origin 0 located at the center of the input image 51. The correction surface 53 is composed of a central surface 53A arranged in the center, a left side surface 53B and a right side surface 53C arranged on the left and right sides of the central surface 53A, respectively, and the left side surface 53B and the right side surface 53C are optical axis origins. It is configured to bend toward. The image on the virtual object surface 52 projected on the correction surface 53 in the virtual optical axis z direction is projected onto the display surface 54 by an orthogonal projection method, and the projected image of the display surface 54 is used as an output image.

In the image conversion model, the pixel position of the display surface 54 is converted into the pixel position of the correction surface 53, the pixel position of the correction surface 53 is converted into the polar coordinates of the hemisphere of the virtual object surface 52, and finally the pixels of the input image 51. Convert to position. In the image conversion model, the pixel position of the input image 51 is converted for each pixel position of the display surface 54, and the pixel position of the input image 51 is stored for each pixel position of the display surface 54. That is, the second conversion table 32 manages the second input coordinates of the pixel values corrected for distortion for each output coordinate.

FIG. 6B is an explanatory diagram showing an example of the table configuration of the second conversion table 32. The second conversion table 32 shown in FIG. 6B manages the output coordinates and the second input coordinates in association with each other. The output coordinates are the coordinates for outputting the pixel value of the output image. The second input coordinate is a coordinate for inputting a pixel value corrected for distortion for each output coordinate. It should be noted that the principle for calculating the input coordinates of the pixel value corrected for the second distortion for each output coordinate in the second conversion table 32 is that the applicant applies to Patent Document 1 (Japanese Unexamined Patent Publication No. 2008-334470). ) Proposes. Although the technique shown in Patent Document 1 is applied here, the technique is not limited to this, and a known technique for correcting distortion caused by a wide-angle lens may be used.

FIG. 7 is an explanatory diagram showing an example of images before and after the second distortion correction of the second conversion unit 22. The second conversion unit 22 refers to the second conversion table 32, and outputs the pixel value of the second input coordinate corrected for the second distortion as the pixel value of the output coordinate for each output coordinate. For example, as shown in FIG. 7, the second conversion unit 22 uses the pixel value of the input coordinate β5 in the input image as the pixel value of the output coordinate α5 in the output image, and sets the pixel value of the input coordinate β6 in the input image as the pixel value. Is output as the pixel value of the output coordinate α6 in the output image. The second conversion unit 22 outputs the pixel values of the output coordinates in the second image area 42 to the composition unit 25. As a result, the second conversion unit 22 outputs the image before the second distortion correction as the image after the second distortion correction. For convenience of explanation, in the example of FIG. 7, a case where the pixel values of all the second input coordinates in the input image are output as the pixel values of the output coordinates is illustrated. However, the second conversion unit 22 actually sets the pixel value of the second input coordinate in the second image area 42 after correcting the second distortion in the second image area 42 in the input image. It is output to the synthesis unit 25 as a pixel value of the output coordinates.

FIG. 8A is an explanatory diagram showing an example of an output image. In the central portion of the output image shown in FIG. 8A, there is a first image region 41. FIG. 8B is an explanatory diagram showing the first image area 41 in the output image and the image areas other than the first image area 41 in light and dark. The first image area 41 in the output image can be represented by a white background as shown in FIG. 8B. The image areas other than the first image area 41 in the output image are the second image area 42 and the third image area 43, and can be represented by a black background as shown in FIG. 8B. FIG. 8C is an explanatory diagram showing the first image region 41, the second image region 42, and the third image region 43 in the output image in light and dark. In the output image shown in FIG. 8C, the first image area 41 is a white background, the second image area 42 is a black background, and the third image area 43 is the boundary between the first image area 41 and the second image area 42. Can be expressed by gradation. A white background indicates that it is the first image area 41 because the brightness is 100%, and a black character indicates that it is the second image area 42 because the brightness is 0%, and the light and dark are white. A portion of less than 100% and more than 0%, which is black, indicates a third image region 43. This lightness and darkness is the ratio of the weighted average in the third strain correction, which is a new output coordinate by weighted averaging the output coordinates output by the first distortion correction, the image, and the output coordinates output by the second distortion correction. Shows.

Here, the above-mentioned weighted average will be described in detail. The ratio of each output coordinate in the third image area 43 is the first distance from the output coordinate in the third image area 43 to the output coordinate in the nearest first image area 41 and the third image area 43. It is a ratio according to the ratio from the output coordinate in the image area 43 to the second output coordinate in the nearest second image area 42. For example, when the ratio of the first distance to the second distance to the output coordinates is 5: 5, the calculation unit 23 calculates 50% as the ratio of the output coordinates. Further, for example, when the ratio of the first distance to the second distance to the output coordinates is 7: 3, the calculation unit 23 calculates 70% as the ratio of the output coordinates. Further, for example, when the ratio of the first distance to the second distance to the output coordinates is 3: 7, the calculation unit 23 calculates 30% as the ratio of the output coordinates. The ratio X of the output coordinates in the third image area 43 is 0% <X <100%.

Further, in the case of the output coordinates in the first image area 41, the calculation unit 23 has a ratio of the first distance to the second distance to the output coordinates of 10: 0, so that the ratio of the output coordinates is 100%. Is calculated. The ratio of the output coordinates in the first image area 41 is 100%. Further, in the case of the output coordinates in the second image area 42, the calculation unit 23 has a ratio of the first distance to the second distance to the output coordinates of 0:10, so that the ratio of the output coordinates is 0%. Is calculated. The ratio of the output coordinates in the second image area 42 is 0%. Then, when the calculation unit 23 calculates the ratio for each output coordinate, the calculation unit 23 stores the ratio for each output coordinate in the ratio table 33.

FIG. 9 is an explanatory diagram showing an example of the table configuration of the ratio table 33. The ratio table 33 shown in FIG. 9 manages the ratio in association with each output coordinate in the output image. The ratio is the ratio of the weighted average of the first input coordinate and the second input coordinate for each output coordinate used when calculating the third input coordinate of the pixel value corrected with the third distortion.

The calculation unit 23 calculates, for example, the first input coordinates of the pixel value corrected for the first distortion by f (x, y). The calculation unit 23 calculates, for example, the second input coordinates of the pixel value corrected for the second distortion by g (x, y). FIG. 10 is an explanatory diagram showing an example of a calculation method when calculating the third input coordinates for correcting the third distortion. FIG. 11 is an explanatory diagram showing an example of the table configuration of the third conversion table 34. The calculation unit 23 performs weighted average processing. More specifically, as shown in FIG. 10, when the ratio of the output coordinates is n%, the ratio of the first input coordinate is corrected by the first input coordinate f (x, y) × n%. Further, the calculation unit 23 proportionally corrects the second input coordinate with the second input coordinate g (x, y) × (100% −n%). Further, the calculation unit 23 adds the first input coordinate after the ratio correction and the second input coordinate after the ratio correction to calculate the third input coordinate for each output coordinate. That is, the calculation unit 23 calculates the third input coordinate for each output coordinate by (formula 1) of f (x, y) × n% + g (x, y) × (100% −n%). The calculation unit 23 refers to the ratio corresponding to the output coordinates in the ratio table 33, and for example, when the ratio of the output coordinates is 50%, 50% of the first input coordinates corresponding to the output coordinates and the output coordinates are used. Add 50% of the corresponding second input coordinates. Then, the calculation unit 23 calculates the added value as the third input coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34 as shown in FIG.

For example, when the ratio of the output coordinates is 70%, the calculation unit 23 adds 70% of the first input coordinates corresponding to the output coordinates and 30% of the second input coordinates corresponding to the output coordinates. Then, the calculation unit 23 calculates the added value as the third input coordinate of the output coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34 as shown in FIG. ..

For example, when the ratio of the output coordinates is 30%, the calculation unit 23 adds 30% of the first input coordinates corresponding to the output coordinates and 70% of the second input coordinates corresponding to the output coordinates. Then, the calculation unit 23 calculates the added value as the third input coordinate of the output coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34 as shown in FIG. ..

The third conversion unit 24 refers to the third conversion table 34, and sets the pixel value of the third input coordinate in the third image area 43 after correcting the third distortion in the third image area 43. It is output to the compositing unit 25 as a pixel value of the output coordinates.

That is, the first conversion unit 21 synthesizes the pixel value of the first input coordinate corrected for the distortion in the first image area 41 in the input image as the pixel value of the output coordinate in the first image area 41. Output to unit 25. The second conversion unit 22 uses the pixel value of the second input coordinate corrected for the distortion in the second image area 42 in the input image as the pixel value of the output coordinate in the second image area 42 as the composition unit 25. Output to. The third conversion unit 24 uses the pixel value of the third input coordinate corrected for the distortion in the third image area 43 in the input image as the pixel value of the output coordinate in the third image area 43 as the composition unit 25. Output to.

The compositing unit 25 includes the pixel values of the output coordinates in the first image area 41 from the first conversion unit 21 and the pixel values of the output coordinates in the second image area 42 from the second conversion unit 22. , The output image obtained by synthesizing the pixel values of the output coordinates in the third image area 43 from the third conversion unit 24 is output to the navigation device 3. FIG. 12 is an explanatory diagram showing an example of an output image. As shown in FIG. 12, the output image has a first image area 41 corrected for the first distortion, a second image area 42 corrected for the second distortion, and a third image area corrected for the third distortion. It is an image including the image area 43 of 3. In the output image, for example, a third image area 43 is arranged between the first image area 41 and the second image area 42.

Then, the navigation device 3 draws the route 50 in the area on the first image area 41 in the output image from the compositing unit 25, and as shown in FIG. 12, displays the drawn output image on the display device 4. Output. The traveling line 50 has, for example, a width of about the width of a vehicle and a depth of 2 meters.

Next, the operation of the image display system 1 of this embodiment will be described. For convenience of explanation, it is assumed that the contents of the first conversion table 31, the second conversion table 32, the third conversion table 34, and the ratio table 33 are stored in advance.

The first conversion unit 21 referred to the first conversion table 31 and corrected the first distortion for each output coordinate in the first image region 41 in the input image captured by the image pickup element 12. The pixel value of the first input coordinate is output to the compositing unit 25 as the pixel value of the output coordinate in the first image area 41. The second conversion unit 22 referred to the second conversion table 32, and corrected the second distortion for each output coordinate in the second image region 42 in the input image captured by the image sensor 12. The pixel value of the second input coordinate is output to the compositing unit 25 as the pixel value of the output coordinate in the second image area 42. The third conversion unit 24 referred to the third conversion table 34, and corrected the third distortion for each output coordinate in the third image region 43 in the input image captured by the image sensor 12. The pixel value of the third input coordinate is output to the compositing unit 25 as the pixel value of the output coordinate in the third image area 43.

The compositing unit 25 includes the pixel values of the output coordinates in the first image area 41 from the first conversion unit 21 and the pixel values of the output coordinates in the second image area 42 from the second conversion unit 22. , The pixel value of the output coordinate in the third image area 43 from the third conversion unit 24 is combined to generate an output image. The compositing unit 25 outputs the generated output image to the navigation device 3. The navigation device 3 draws the advance line 50 in the area of the first image area 41 in the output image from the compositing unit 25, and outputs the output image on which the advance line 50 is drawn to the display device 4. In generating the traveling line 50, the same correction processing as the first distortion correction applied to the first image area 41 is performed on the original data of the traveling line 50. As a result, the user can superimpose and display on the output image how the vehicle travels on the output image of the route 50.

As described above, the navigation device 3 superimposes the forward route obtained by performing the same image conversion on the forward route data in the image portion of the first distortion correction having only the distortion aberration or no distortion as described above. .. As described above, since the conversion table for performing this image conversion can be stored in a memory having a small capacity, the memory capacity can be reduced. The cost increase due to the increase in memory capacity is suppressed.

In the image pickup apparatus 2, the first conversion table 31, the second conversion table 32, the ratio table 33, and the third conversion table 34 are stored in the RAM 15 in the image pickup apparatus 2. However, the present invention is not limited to the RAM 15, and may be stored in, for example, a storage device (not shown) that can be externally connected to the image pickup device 2, and can be appropriately changed.

The image pickup apparatus 2 has a built-in first conversion table 31, a second conversion table 32, and a ratio table 33, but if the table contents of the third conversion table 34 are stored in advance, the third conversion table 3 Only the conversion table 34 may be built-in and can be changed as appropriate.

In the image pickup apparatus 2, the first conversion table 31, the second conversion table 32, the third conversion table 34, and the ratio table 33 were used. However, for each output coordinate, a function f for calculating the first input coordinate of the pixel value corrected for the first distortion, and a function f for calculating the second input coordinate of the pixel value corrected for the second distortion. The function g and the input coordinates of the pixel value corrected for the third distortion may be stored (formula 1).

In this case, in the image pickup apparatus 2, the function f, the function g, and the formula 1 can be used without preparing the first conversion table 31, the second conversion table 32, the third conversion table 34, and the ratio table 33. The input coordinates of the pixel value subjected to the distortion correction of 1, the second distortion correction, and the third distortion correction are calculated. Then, the image pickup apparatus 2A has a pixel value of the output coordinates in the first image region 41 corrected for the first distortion and a pixel value of the output coordinates in the second image region 42 corrected for the second distortion. , The output image including the pixel value of the output coordinates in the third image area 43 corrected for the third distortion is output to the navigation device 3.

In the above embodiment, the input image is stored in the RAM 14 or the like in the image pickup device 2, but the input image may be stored in an external storage device connected to the image pickup device 2, and can be changed as appropriate.

In the above embodiment, for example, the second strain correction has been described as strain correction according to Patent Document 1, but the strain is not limited to these strains, and other correction methods that are easy for the user to see may be adopted.

In the above embodiment, the distortion that executes the first distortion correction in the first image area 41, the second distortion correction in the second image area 42, and the third distortion correction in the third image area 43. The correction process (image process) was executed by the image processor 13 in the image pickup apparatus 2. However, the distortion correction process may be executed on an external device connected to the image pickup device 2, for example, the navigation device 3, and can be changed as appropriate.

In the above embodiment, the first conversion unit 21 performs the first strain correction, the second conversion unit 22 performs the second strain correction, and the third conversion unit 24 performs the third strain correction. However, the first strain correction, the second strain correction, and the third strain correction may be executed sequentially in no particular order or in parallel, and can be changed as appropriate.

In the navigation device 3 of the above embodiment, the advance line 50 is drawn in the area of the first image region 41 using the distortion table 31A in the image pickup device 2, but the first conversion table 31 is drawn in the navigation device 3. May be provided separately and can be changed as appropriate.

In the above embodiment, the advance line 50 is drawn on the output image including the first image area 41 in which the first distortion is corrected, but the route 50 is not limited to the advance line 50, and for example, a fixed advance line or the like is used. A line may be drawn and can be changed as appropriate.

2 Image processing device 13 Image processing processor 21 First conversion unit 21A First conversion unit 22 Second conversion unit 22A Second conversion unit 23 Calculation unit 23A Calculation unit 24 Third conversion unit 24A Third conversion unit 25 Synthetic part 25A Synthetic part

Claims (4)

An image processing device that converts an input image into an image and outputs an output image.
A first conversion unit that performs a first distortion correction for each pixel in the first image area of the input image and outputs the image as an image.
A second conversion unit that performs a second distortion correction for each pixel in the second image area of the input image and outputs the image as an image.
A third conversion unit that performs a third distortion correction for each pixel in the third image area of the input image and outputs the image as an image.
An image output from the first conversion unit, an image output from the second conversion unit, and a composition unit that generates and outputs the output image from the image output from the third conversion unit.
Have,
In the output image, the image output from the third conversion unit is arranged between the image output from the first conversion unit and the image output from the second conversion unit, and the second conversion unit is used. An image processing device characterized in that the image output from the conversion unit 1 is an image having only distortion aberration or no distortion. The input image is generated from a subject, a lens that collects light from the subject, and an image pickup element that forms an image of the collected light.
The second conversion unit is
The subject having a straight line toward the optical axis of the lens and the subject having a straight line in the vertical direction are substantially linear in the output image, and the subject having a straight line in the horizontal direction is the central portion of the output image. , And the left side part and the right side part that are connected to the left and right sides of the center part are almost linear, and the distortion is corrected so as to bend near the boundary between the center part of the output image and the left side part and the right side part. The image processing apparatus according to claim 1. The third conversion unit is
The first aspect of claim 1, wherein the output obtained by applying the first strain correction and the output obtained by applying the second strain correction are weighted averaged and output for each pixel of the input image. Image processing device. The image processing apparatus according to claim 1, wherein a route is drawn in the first image area.

Images