JP7423973B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing device.

For example, there is an image processing device that converts an image of a subject captured by a vehicle-mounted camera using a fisheye lens so as to correct curvature aberration and outputs the image (for example, Patent Document 1).

In such an image processing device, the entire input image is image-converted so that a straight object toward the optical axis of the fisheye lens and a vertically straight object become substantially straight in the output image. In addition, in the image processing device, a straight object in the horizontal direction is formed into a substantially straight line by the center part of the output image, and the left part and right part which are connected to the left and right of the center part, and the center part of the output image, The entire input image is transformed so that it is curved near the boundary between the left side portion and the right side portion. As a result, the image processing device provides an output image that is a wide-angle image that does not give the user a sense of discomfort and is easy to see.

Furthermore, a technique is known in which a course line is displayed superimposed on the direction in which a vehicle is traveling on a display device that receives an output image output from an image processing device as described above (for example, Patent Document 2).

Japanese Patent Application Publication No. 2008-311890 Japanese Patent Application Publication No. 11-334470

Such display devices generally perform image processing on the course line data by performing image conversion to match the image deformed by the curvature aberration of the fisheye lens or other image conversion processing. The actual route of the vehicle is superimposed and displayed on the image output by the device. Therefore, the image conversion process for the course line data described above is similar to the image conversion process for images captured by an on-vehicle camera performed within the image processing device.

More specifically, on the display device side, for example, for each output coordinate (X, Y) of the output image of the display device, a table is created that associates input coordinates (x, y) with pre-stored course line data as input. to perform an image conversion process that generates the brightness and color information of the pixel at the output coordinates (X, Y) by referencing (transferring) the brightness and color information of the pixel at the input coordinates (x, y). is possible. In such processing, generally, conversion data in which the input coordinates corresponding to the output coordinates are converted into a table is stored in advance, and the conversion process is performed according to the table.

However, such conversion data converted into a table requires input coordinates (x, y) for the number of pixels in the output image, and as the number of pixels in display devices has increased in recent years, the capacity has increased and memory costs have increased. There was the problem of inviting

In view of these problems, it is an object of the present invention to provide an image processing device and the like that can superimpose course lines while suppressing an increase in memory of a display device.

An image processing apparatus according to one aspect is an image processing apparatus that converts an input image and outputs an output image, and includes a first conversion section, a second conversion section, a third conversion section, and a composition section. and has. The first conversion unit outputs an image in which a first distortion correction is applied to each pixel in a first image area of the input image. The second conversion unit outputs an image in which the second distortion correction is applied to each pixel in the second image area of the input image. The third conversion unit outputs an image obtained by performing third distortion correction on each pixel in a third image area of the input image. The composition section generates and outputs the output image from the image output from the first conversion section, the image output from the second conversion section, and the image output from the third conversion section. The first image area is an image area that includes all the predetermined image areas for drawing the course line for each of the input images. The output image is such that the image output from the third conversion unit is placed between the image output from the first conversion unit and the image output from the second conversion unit, and the image output from the third conversion unit is arranged between the image output from the first conversion unit and the image output from the second conversion unit, The image output from the converter has only distortion or no distortion.

According to the present invention, a course line can be drawn while suppressing an increase in the memory capacity of a display device.

FIG. 1 is an explanatory diagram showing an example of the hardware configuration of an image display system based on the base technology. FIG. 2 is a block diagram illustrating an example of the functional configuration of an imaging device based on the base technology. FIG. 3 is an explanatory diagram showing an example of an input image captured by an imaging device. FIG. 4 is an explanatory diagram showing an example of the table configuration of the first conversion table. FIG. 5 is an explanatory diagram showing an example of the relationship between the real image height and the ideal image height, and shows an example of images before and after the first distortion correction by the first conversion unit. FIG. 6 is an explanatory diagram showing an example of images before and after the first distortion correction by the first conversion unit. FIG. 7A is an explanatory diagram showing an example of an image conversion model. FIG. 7B is an explanatory diagram showing an example of the table configuration of the second conversion table. FIG. 8 is an explanatory diagram showing an example of images before and after the second distortion correction by the second conversion unit. FIG. 9A is an explanatory diagram showing an example of an output image after the first distortion correction. FIG. 9B is an explanatory diagram in which the first image area in the output image after the first distortion correction and image areas other than the first image area are expressed in brightness and darkness. FIG. 9C is an explanatory diagram showing the first image area, the second image area, and the third image area in the output image using brightness and darkness. FIG. 10 is an explanatory diagram showing an example of the table configuration of the ratio table. FIG. 11 is an explanatory diagram showing an example of a calculation method when calculating the third input coordinates for correcting the third distortion. FIG. 12 is an explanatory diagram showing an example of the table configuration of the third conversion table. FIG. 13 is an explanatory diagram showing an example of an output image. FIG. 14 is a block diagram illustrating an example of the functional configuration of the imaging device according to the first embodiment. FIG. 15A is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device at the 1-1 mounting position. FIG. 15B is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device at the 1-2 mounting position. FIG. 16A is an explanatory diagram showing an example of the 1-1 predetermined image area in the first distortion-corrected image captured by the imaging device at the 1-1 mounting position. FIG. 16B is an explanatory diagram showing an example of the 1-2 predetermined image area in the first distortion-corrected image captured by the imaging device at the 1-2 mounting position. FIG. 17A shows the 1-1 predetermined image area and the image area other than the 1-1 predetermined image area in the image after the first distortion correction at the 1-1 mounting position in terms of brightness and darkness. It is an explanatory diagram. FIG. 17B shows the 1-2 predetermined image area and the image area other than the 1-2 predetermined image area in the image after the first distortion correction at the 1-2 mounting position in terms of brightness and darkness. It is an explanatory diagram. FIG. 18 is an explanatory diagram in which the first image area and image areas other than the first image area are expressed in brightness and darkness. FIG. 19 is an explanatory diagram in which the first image area, the second image area, and the third image area of Example 1 are expressed in brightness and darkness. FIG. 20A is an explanatory diagram showing an example of an output image captured by the imaging device at the 1-1 mounting position. FIG. 20B is an explanatory diagram showing an example of an output image captured by the imaging device at the 1-2 mounting position. FIG. 21A is an explanatory diagram illustrating an example of an image after the first distortion correction of the original image captured by the imaging device at the 1-1 tilt angle position of the second embodiment. FIG. 21B is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device at the 1-2nd tilt angle position. FIG. 21C is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device at the 1st to 3rd tilt angle positions. FIG. 21D is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device at the 1-4th tilt angle position. FIG. 22A is an explanatory diagram showing an example of the 1-1 predetermined image area in the first distortion-corrected image captured by the imaging device at the 1-1 tilt angle position. FIG. 22B is an explanatory diagram showing an example of the 1-2 predetermined image area in the first distortion-corrected image captured by the imaging device at the 1-2 tilt angle position. FIG. 22C is an explanatory diagram showing an example of the 1-3rd predetermined image area in the first distortion-corrected image captured by the imaging device at the 1-3rd tilt angle position. FIG. 22D is an explanatory diagram showing an example of the 1-4th predetermined image area in the first distortion-corrected image captured by the imaging device at the 1-4th tilt angle position. FIG. 23A shows the 1-1 predetermined image area and the image area other than the 1-1 predetermined image area in the image after the first distortion correction at the 1-1 tilt angle position in terms of brightness and darkness. FIG. FIG. 23B shows the 1-2 predetermined image area and the image area other than the 1-2 predetermined image area in the image after the first distortion correction at the 1-2 tilt angle position in terms of brightness and darkness. FIG. FIG. 23C shows the 1-3 predetermined image area and the image area other than the 1-3 predetermined image area in the image after the first distortion correction at the 1-3 tilt angle position in terms of brightness and darkness. FIG. FIG. 23D shows the 1-4th predetermined image area and the image area other than the 1-4th predetermined image area in the image after the first distortion correction at the 1-4th tilt angle position in terms of brightness and darkness. FIG. FIG. 24 is an explanatory diagram in which the first image area and image areas other than the first image area are expressed in brightness and darkness. FIG. 25 is an explanatory diagram in which the first image area, the second image area, and the third image area of Example 2 are expressed in brightness and darkness. FIG. 26A is an explanatory diagram showing an example of an output image captured by the imaging device at the 1-1 tilt angle position. FIG. 26B is an explanatory diagram showing an example of an output image captured by the imaging device at the 1-2nd tilt angle position. FIG. 26C is an explanatory diagram showing an example of an output image captured by the imaging device at the 1st to 3rd tilt angle positions. FIG. 26D is an explanatory diagram showing an example of an output image captured by the imaging device at the 1st to 4th tilt angle positions. FIG. 27 is an explanatory diagram showing an example of the relationship between the vehicle rear end position and the vehicle trajectory when the vehicle is reversed by rotating the steering wheel counterclockwise at five different steering angles. FIG. 28A is an explanatory diagram showing an example of an image after the first distortion correction of the original image at the 1-1st handle rotation angle in Example 3. FIG. 28B is an explanatory diagram showing an example of an image after the first distortion correction of the original image at the 1-2nd handle rotation angle. FIG. 28C is an explanatory diagram showing an example of an image after the first distortion correction of the original image at the 1st to 3rd steering wheel rotation angles. FIG. 28D is an explanatory diagram showing an example of an image after the first distortion correction of the original image at the 1st to 4th steering wheel rotation angles. FIG. 28E is an explanatory diagram showing an example of an image after the first distortion correction of the original image at the 1st to 5th handle rotation angles. FIG. 29A is an explanatory diagram showing an example of the 1-1 predetermined image area in the image after the first distortion correction at the 1-1 handle rotation angle. FIG. 29B is an explanatory diagram showing an example of the 1-2 predetermined image area in the image after the first distortion correction at the 1-2 handle rotation angle. FIG. 29C is an explanatory diagram showing an example of the 1-3 predetermined image area in the image after the first distortion correction at the 1-3 handle rotation angle. FIG. 29D is an explanatory diagram showing an example of the 1-4th predetermined image area in the image after the first distortion correction at the 1-4th handle rotation angle. FIG. 29E is an explanatory diagram showing an example of the 1-5th predetermined image area in the image after the first distortion correction at the 1-5th steering wheel rotation angle. FIG. 30A shows the 1-1 predetermined image area and the image area other than the 1-1 predetermined image area in the image after the first distortion correction at the 1-1 handle rotation angle in terms of brightness and darkness. FIG. FIG. 30B shows the 1-2 predetermined image area in the image after the first distortion correction at the 1-2 handle rotation angle and the image area other than the 1-2 predetermined image area in brightness and darkness. FIG. FIG. 30C represents the 1-3 predetermined image area and the image area other than the 1-3 predetermined image area in the image after the first distortion correction at the 1-3 handle rotation angle in brightness and darkness. FIG. FIG. 30D shows the 1-4 predetermined image area and the image area other than the 1-4 predetermined image area in the image after the first distortion correction at the 1-4 handle rotation angle in brightness and darkness. FIG. FIG. 30E shows the 1-5 predetermined image area and the image area other than the 1-5 predetermined image area in the image after the first distortion correction at the 1-5 handle rotation angle in terms of brightness and darkness. FIG. FIG. 31 is an explanatory diagram in which the first image area and image areas other than the first image area are expressed in brightness and darkness. FIG. 32 is an explanatory diagram in which the first image area, the second image area, and the third image area of Example 3 are expressed in brightness and darkness. FIG. 33A is an explanatory diagram showing an example of an output image at the 1-1st handle rotation angle. FIG. 33B is an explanatory diagram showing an example of an output image at the 1-2nd handle rotation angle. FIG. 33C is an explanatory diagram showing an example of an output image at the 1st to 3rd steering wheel rotation angles. FIG. 33D is an explanatory diagram showing an example of an output image at the 1st to 4th steering wheel rotation angles. FIG. 33E is an explanatory diagram showing an example of an output image at the 1st to 5th steering wheel rotation angles.

Embodiments of the image processing apparatus disclosed in the present application will be described in detail below based on the drawings. Note that the disclosed technology is not limited by each example. Furthermore, the embodiments shown below may be combined as appropriate to the extent that no contradiction occurs.

FIG. 1 is an explanatory diagram showing an example of the hardware configuration of an image display system 1 as a base technology. The image display system 1 shown in FIG. 1 includes an imaging device 2, a navigation device 3, and a display device 4. The imaging device 2 includes a lens 11 , an imaging element 12 , an image processing processor 13 , a ROM (Read Only Memory) 14 , and a RAM (Radom Access Memory) 15 . The imaging device 2 is, for example, a camera device placed near the license plate at the rear of the vehicle. Lens 11 is, for example, a fisheye lens. The image sensor 12 is a CMOS (Complementary Metal Oxide Semiconductor) sensor that captures an image of a specific range behind the vehicle through the lens 11. The image processing processor 13 is an image processing device that executes image processing inside the imaging device 2. The ROM 14 stores information such as various programs. The RAM 15 stores various information.

The navigation device 3 outputs guide information such as map information to the display device, draws a course line 50 on the output image from the imaging device 2, and outputs the output image on which the course line 50 is drawn to the display device 4. The display device 4 is, for example, a monitor device that is disposed on the front panel of the driver's seat and displays output images and map information from the navigation device 3, for example. Note that the navigation device 3 is set to draw the course line 50 that has been image-converted so as to match the output image after image distortion correction.

FIG. 2 is an explanatory diagram showing an example of the configuration of the imaging device 2 of the base technology. The imaging device 2 shown in FIG. 2 includes a control section 20 and a storage section 30. The control section 20 corresponds to the image processing processor 13 in FIG. 1, and the storage section 30 corresponds to the ROM 14 in FIG. The control section 20 includes a first conversion section 21 , a second conversion section 22 , a calculation section 23 , a third conversion section 24 , and a synthesis section 25 . The storage unit 30 includes a first conversion table 31, a second conversion table 32, a ratio table 33, and a third conversion table 34.

FIG. 3 is an explanatory diagram showing an example of displaying an image captured by the imaging device 2 on the display device 4. The input image 40 shown in FIG. 3 has a first image area 41, a second image area 42, and a third image area 43. The image processor 13 identifies a first image region 41, a second image region 42 and a third image region 43 within the input image. That is, the image processing processor 13 identifies each pixel value in the input image by the input coordinates, and identifies each pixel value in the first image area 41 by the first input coordinates and each pixel value in the second image area 42. The values are identified by the second input coordinates and the third input coordinates of each pixel value in the third image area 43 .

The first image area 41 surrounded by the broken line L1 corresponds to, for example, an image area at the center of the input image, and the first image area 41 is corrected for distortion so that it has only distortion aberration or no distortion. This is a target area for (first distortion correction) and is an image area for drawing a course line that changes depending on the steering angle such as steering wheel rotation. The second image area 42 surrounded by the one-dot chain line L2 and the dotted line L3 is, for example, a distortion correction (second image area) that displays a straight object as a straight line and does not give the user a sense of discomfort, as shown in the above-mentioned Patent Document 1. (distortion correction). This part has no distortion. The third image area 43 surrounded by the broken line L1 and the dashed-dotted line L2 is an image area where third distortion correction is performed, which is a weighted average of the first distortion correction and the second distortion correction. The weighted average will be described later.

The first converting unit 21 is a converting unit that performs first distortion correction on each pixel in the first image area 41 of the input image 40 and outputs the result. Note that the pixels in the first image area 41 can also be said to be the pixel value of the first input coordinates, for example. The first conversion unit 21 refers to the first conversion table 31, for example, and converts the pixel value of the first input coordinate, which has been corrected for distortion in the first image area 41 in the input image 40, into the output image. Output as pixel value of output coordinates. FIG. 4 is an explanatory diagram showing an example of the table configuration of the first conversion table 31. The first conversion table 31 is a table that manages the first input coordinates of pixel values whose distortions in the input image have been corrected in association with each output coordinate of the output image. The first converter 21 outputs pixel values for each output coordinate within the first image area 41 of the output image to the synthesizer 25.

Here, a description will be given of an operation in which the navigation device 3 converts the course line 50 into an image so that it matches the output image after image distortion correction. For example, if the image of the first image area 41 has a distortion aberration, the navigation device 3 performs image conversion on the data of the course line 50 so that it has the same aberration as the aforementioned distortion aberration. When there is distortion, the real image height and the ideal image height differ. FIG. 5 is an explanatory diagram showing an example of the relationship between the real image height and the ideal image height. The navigation device 3 converts the pixel value of the polar coordinates (R, φ) of the ideal image height into the polar coordinates (r, φ) of the real image height. This conversion is a conversion that moves an arbitrary pixel on the line L in FIG. The data size is smaller than the conversion table that associates coordinates (x, y). Note that the image in the first image area 41 may be undistorted. At this time, in the conversion from the pixel value of the polar coordinates (R, φ) in the image to the pixel value of the polar coordinates (r, φ), the above-mentioned pixel movement does not occur, so a conversion table is not required.

The second converting unit 22 is a converting unit that performs second distortion correction on each pixel in the second image area 42 of the input image 40 and outputs the result. Note that the pixels in the second image area 42 can be said to be, for example, the pixel values of the second input coordinates. For example, the second conversion unit 22 refers to the second conversion table 32 and converts the pixel values of the second input coordinates, which have been corrected for distortion in the second image area 42 in the input image 40, into the output image. Output as a pixel value at the second output coordinate. The second conversion table 32 is a table that manages second input coordinates of pixel values whose second distortion has been corrected in association with each second input coordinate within the second image area 42 . The second conversion unit 22 outputs pixel values for each output coordinate within the second image area 42 of the output image to the synthesis unit 25.

The calculation unit 23 calculates, for each output coordinate of the distortion-corrected pixel value in the third image area 43 in the input image 41, a value in the first image area 41 closest to the output coordinate in the third image area 43. The ratio between the first distance to the output coordinates of and the second distance from the output coordinates in the third image area 43 to the nearest output coordinates in the second image area 42 is calculated. Further, the calculation unit 23 stores the ratio as a percentage in the ratio table 33 for each output coordinate. The calculation unit 23 inputs the output obtained by applying the first distortion correction to each pixel of the input image 41 and the output obtained by applying the second distortion correction to each pixel, and outputs a weighted average of them. . For example, the calculation unit 23 refers to the ratio table 33 and corrects the first input coordinate and the second input coordinate according to the ratio for each output coordinate (first distance: second distance). The calculation unit 23 calculates the third input coordinates of the output image by adding the first input coordinates after the ratio correction and the second input coordinates after the ratio correction.

The calculation unit 23 stores the third input coordinates in the third conversion table 34 for each output coordinate. The third converter 24 is a converter that performs third distortion correction on each pixel in the third image area 43 of the input image and outputs the result. Note that the pixels in the third image area 43 can be said to be, for example, the pixel values of the third input coordinates. For example, the third conversion unit 24 refers to the third conversion table 34 and converts the pixel values of the third input coordinates that have been corrected for distortion in the third image area 43 in the input image 40 into the third image. It is output as a pixel value at the output coordinates within the area 43. The third converter 24 outputs pixel values for each output coordinate within the third image area 43 to the synthesizer 25.

FIG. 6 is an explanatory diagram showing an example of images before and after the first distortion correction by the first conversion unit 21. The first conversion unit 21 refers to the first conversion table 31 and outputs the distortion-corrected pixel value of the first input coordinate as the pixel value of the output coordinate for each output coordinate. The first converter 21 outputs the pixel value of the output coordinate within the first image area 41 to the synthesizer 25. As a result, the first conversion unit 21 can output the image before the first distortion correction as the image after the first distortion correction, as shown in FIG. For example, the first conversion unit 21 uses the pixel value of the input coordinate β1 in the input image as the pixel value of the output coordinate α1 in the output image, and the pixel value of the input coordinate β3 in the input image as the pixel value of the input coordinate β3 in the output image. Output as a pixel value of α3. For convenience of explanation, the example of FIG. 6 illustrates a case where all pixel values of the first input coordinates in the input image are output as pixel values of the output coordinates. However, in reality, the first conversion unit 21 converts the pixel value of the first input coordinate, which has been corrected for distortion in the first image area 41 in the input image, into the pixel value of the output coordinate in the first image area 41. This is output to the synthesis unit 25 as a pixel value.

FIG. 7A is an explanatory diagram showing an example of an image conversion model. The image conversion model shown in FIG. 7A includes an input image 51 made of a circular plane, a hemispherical virtual object surface 52 on which a subject is projected corresponding to the lens 11, a correction surface 53 for correcting curvature aberration, and image conversion. display surfaces 54 are arranged in order. A virtual optical axis z corresponding to the optical axis of the lens 11 passes through the centers of these surfaces at right angles from the optical axis origin 0 located at the center of the input image 51. The correction surface 53 is composed of a central surface 53A located at the center, and a left side surface 53B and a right side surface 53C located on the left and right sides of the central surface 53A, respectively, and the left side surface 53B and right side surface 53C are located at the optical axis origin. It is constructed by bending towards. The image on the virtual object surface 52 projected onto the correction surface 53 in the virtual optical axis z direction is projected onto the display surface 54 by orthogonal projection, and the projected image on the display surface 54 is used as an output image.

In the image conversion model, pixel positions on the display surface 54 are converted to pixel positions on the correction surface 53, pixel positions on the correction surface 53 are converted to polar coordinates on the hemispherical surface of the virtual object surface 52, and finally pixels of the input image 51 are Convert to position. In the image conversion model, the pixel positions of the input image 51 are converted for each pixel position on the display surface 54, and the pixel positions of the input image 51 are stored for each pixel position on the display surface 54. That is, the second conversion table 32 manages the second input coordinates of the pixel values whose distortion has been corrected for each output coordinate.

FIG. 7B is an explanatory diagram showing an example of the table configuration of the second conversion table 32. The second conversion table 32 shown in FIG. 7B manages output coordinates and second input coordinates in association with each other. The output coordinates are the coordinates at which the pixel values of the output image are output. The second input coordinates are coordinates into which pixel values whose distortions have been corrected are input for each output coordinate. The principle of calculating the input coordinates of the pixel values corrected for the second distortion for each output coordinate in the second conversion table 32 was developed by the present applicant as described in Patent Document 1 (Japanese Patent Laid-Open No. 2008-334470). ) is proposed. Note that although the technique shown in Patent Document 1 is applied here, the present invention is not limited thereto, and a known technique for correcting distortion caused by a wide-angle lens may be used.

FIG. 8 is an explanatory diagram showing an example of an image before and after the second distortion correction by the second conversion unit 22. The second conversion unit 22 refers to the second conversion table 32 and outputs, for each output coordinate, the pixel value of the second input coordinate with the second distortion corrected as the pixel value of the output coordinate. For example, as shown in FIG. 8, the second conversion unit 22 converts the pixel value at the input coordinate β5 in the input image into the pixel value at the output coordinate α5 in the output image, and converts the pixel value at the input coordinate β6 in the input image into the pixel value at the input coordinate β6 in the input image. is output as a pixel value at output coordinate α6 in the output image. The second converter 22 outputs the pixel value of each output coordinate within the second image area 42 to the synthesizer 25. As a result, the second conversion unit 22 outputs the image before the second distortion correction as the image after the second distortion correction. For convenience of explanation, the example in FIG. 8 illustrates a case where all pixel values of the second input coordinates in the input image are output as pixel values of the output coordinates. However, in reality, the second conversion unit 22 converts the pixel value of the second input coordinate, which has been corrected for the second distortion in the second image area 42 in the input image, into the second image area 42 . This is output to the synthesis unit 25 as a pixel value of output coordinates.

FIG. 9A is an explanatory diagram showing an example of an output image after the first distortion correction. A first image area 41 is located in the central portion of the output image shown in FIG. 9A. FIG. 9B is an explanatory diagram in which the first image area 41 in the output image after the first distortion correction and image areas other than the first image area 41 are expressed in brightness and darkness. The first image area 41 in the output image can be represented as a white background, as shown in FIG. 9B. Image areas other than the first image area 41 in the output image are the second image area 42 and the third image area 43, and can be represented by a black background as shown in FIG. 9B. FIG. 9C is an explanatory diagram showing the first image area 41, the second image area 42, and the third image area 43 in the output image using brightness and darkness. In the output image shown in FIG. 9C, the first image area 41 is a white background, the second image area 42 is a black background, and the third image area 43 is a boundary between the first image area 41 and the second image area 42. can be expressed using gradation. A white background has a brightness of 100%, indicating that it is the first image area 41, and a black background has a brightness of 0%, indicating that it is the second image area 42, and the brightness is white. A portion that is less than a certain 100% and more than 0%, which is black, is a third image area 43. This brightness is determined by the ratio of the weighted average in the third distortion correction, which is a weighted average of the output coordinates output by the first distortion correction, the image, and the output coordinates output by the second distortion correction to obtain new output coordinates. It shows.

Here, the above-mentioned weighted average will be explained in detail. The ratio for each output coordinate in the third image area 43 is calculated based on the first distance from the output coordinate in the third image area 43 to the nearest output coordinate in the first image area 41, and the ratio for each output coordinate in the third image area 43. This is a ratio according to the ratio of the second distance from the output coordinate in the image area 43 to the second output coordinate in the nearest second image area 42 . For example, when the ratio of the first distance to the second distance with respect to the output coordinates is 5:5, the calculation unit 23 calculates 50% as the ratio of the output coordinates. Further, for example, when the ratio of the first distance to the second distance with respect to the output coordinates is 7:3, the calculation unit 23 calculates 70% as the ratio of the output coordinates. Further, for example, when the ratio of the first distance to the second distance with respect to the output coordinates is 3:7, the calculation unit 23 calculates 30% as the ratio of the output coordinates. Note that the ratio X of the output coordinates within the third image area 43 is 0%<X<100%.

In addition, in the case of output coordinates within the first image area 41, the calculation unit 23 calculates that the ratio of the first distance to the second distance with respect to the output coordinates is 10:0, so that the ratio of the output coordinates is 100%. Calculate. Note that the ratio of output coordinates within the first image area 41 is 100%. Further, in the case of output coordinates within the second image area 42, the calculation unit 23 calculates that the ratio of the first distance to the second distance with respect to the output coordinates is 0:10, so the ratio of the output coordinates is 0%. Calculate. Note that the ratio of output coordinates within the second image area 42 is 0%. Then, when calculating the ratio for each output coordinate, the calculation unit 23 stores the ratio for each output coordinate in the ratio table 33.

FIG. 10 is an explanatory diagram showing an example of the table configuration of the ratio table 33. The ratio table 33 shown in FIG. 10 manages a ratio in association with each output coordinate in the output image. The ratio is a weighted average ratio of the first input coordinate and the second input coordinate for each output coordinate, which is used when calculating the third input coordinate of the pixel value corrected for the third distortion.

The calculation unit 23 calculates, for example, the first input coordinates of the pixel value with the first distortion corrected by f(x, y). The calculation unit 23 calculates, for example, the second input coordinate of the pixel value with the second distortion corrected by g(x, y). FIG. 11 is an explanatory diagram showing an example of a calculation method when calculating the third input coordinates for correcting the third distortion. FIG. 12 is an explanatory diagram showing an example of the table configuration of the third conversion table 34. The calculation unit 23 performs weighted average processing. More specifically, as shown in FIG. 11, when the ratio of the output coordinates is n%, the first input coordinates are corrected by the ratio of the first input coordinates f(x,y)×n%. Further, the calculation unit 23 corrects the second input coordinates by the second input coordinates g(x,y)×(100%−n%). Further, the calculation unit 23 calculates third input coordinates for each output coordinate by adding the first input coordinates after the ratio correction and the second input coordinates after the ratio correction. That is, the calculation unit 23 calculates the third input coordinates for each output coordinate using (Equation 1) of f(x,y)×n%+g(x,y)×(100%−n%). The calculation unit 23 refers to the ratio corresponding to the output coordinate in the ratio table 33, and, for example, if the ratio of the output coordinate is 50%, the calculation unit 23 calculates 50% of the first input coordinate corresponding to the output coordinate and the output coordinate. 50% of the corresponding second input coordinates are added. Then, the calculation unit 23 calculates the added value as the third input coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34, as shown in FIG.

For example, when the ratio of output coordinates is 70%, the calculation unit 23 adds 70% of the first input coordinates corresponding to the output coordinates and 30% of the second input coordinates corresponding to the output coordinates. Then, the calculation unit 23 calculates the added value as the third input coordinate of the output coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34 as shown in FIG. .

For example, when the ratio of output coordinates is 30%, the calculation unit 23 adds 30% of the first input coordinates corresponding to the output coordinates and 70% of the second input coordinates corresponding to the output coordinates. Then, the calculation unit 23 calculates the added value as the third input coordinate of the output coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34 as shown in FIG. .

The third conversion unit 24 refers to the third conversion table 34 and converts the pixel value of the third input coordinate in the third image area 43 into which the third distortion is corrected. It is output to the synthesis unit 25 as a pixel value of the output coordinates.

In other words, the first conversion unit 21 synthesizes the pixel value of the first input coordinate, which has been corrected for distortion in the first image area 41 in the input image, as the pixel value of the output coordinate in the first image area 41. output to section 25. The second conversion unit 22 converts the pixel value of the second input coordinate, which has been corrected for distortion in the second image area 42 in the input image, into the pixel value of the output coordinate in the second image area 42, and converts the pixel value to the synthesis unit 25. Output to. The third conversion unit 24 converts the pixel value of the third input coordinate, which has been corrected for distortion in the third image area 43 in the input image, into the pixel value of the output coordinate in the third image area 43, and converts the pixel value to the synthesis unit 25. Output to.

The combining unit 25 combines pixel values of output coordinates in the first image area 41 from the first conversion unit 21 and pixel values of output coordinates in the second image area 42 from the second conversion unit 22. , and the pixel values of the output coordinates in the third image area 43 from the third conversion unit 24 are combined and output to the navigation device 3. FIG. 13 is an explanatory diagram showing an example of an output image. Note that, as shown in FIG. 13, the output image includes a first image area 41 with the first distortion corrected, a second image area 42 with the second distortion corrected, and a second image area 42 with the third distortion corrected. This is an image including an image area 43 of 3. In the output image, for example, a third image area 43 is arranged between the first image area 41 and the second image area 42.

Then, the navigation device 3 draws a course line 50 in an area above the first image region 41 in the output image from the composition unit 25, and displays the drawn output image on the display device 4, as shown in FIG. Output. Note that the route line 50 has a width approximately equal to the width of a vehicle and a depth of 2 meters, for example.

Next, the operation of the image display system 1, which is the underlying technology, will be explained. For convenience of explanation, it is assumed that the contents of the first conversion table 31, second conversion table 32, third conversion table 34, and ratio table 33 are stored in advance.

The first conversion unit 21 refers to the first conversion table 31 and corrects the first distortion for each output coordinate within the first image area 41 within the input image captured by the image sensor 12. The pixel value at the first input coordinate is output to the synthesis unit 25 as the pixel value at the output coordinate within the first image area 41 . The second conversion unit 22 refers to the second conversion table 32 and corrects the second distortion for each output coordinate within the second image area 42 within the input image captured by the image sensor 12. The pixel value at the second input coordinates is output to the synthesis unit 25 as the pixel value at the output coordinates within the second image area 42 . The third conversion unit 24 refers to the third conversion table 34 and corrects the third distortion for each output coordinate within the third image area 43 within the input image captured by the image sensor 12. The pixel value at the third input coordinate is output to the synthesis unit 25 as the pixel value at the output coordinate within the third image area 43 .

The combining unit 25 combines pixel values of output coordinates in the first image area 41 from the first conversion unit 21 and pixel values of output coordinates in the second image area 42 from the second conversion unit 22. , and the pixel values of the output coordinates in the third image area 43 from the third conversion unit 24 are combined to generate an output image. The synthesis unit 25 outputs the generated output image to the navigation device 3. The navigation device 3 draws a course line 50 in the first image region 41 in the output image from the composition unit 25 and outputs an output image in which the course line 50 is drawn to the display device 4 . In generating the course line 50, the same correction process as the first distortion correction applied to the first image area 41 is performed on the original data of the course line 50. As a result, the user can display the course line 50 superimposed on the output image, showing how the vehicle will travel on the output image.

As described above, the navigation device 3 superimposes a course line that has been subjected to similar image conversion on the course line data in the first distortion-corrected image portion that has only distortion or no distortion. . As mentioned above, the conversion table for performing this image conversion can be stored in a memory with a small capacity, so that the memory capacity can be reduced. Cost increase due to increase in memory capacity is suppressed.

However, the mounting position of the imaging device 2 with respect to the vehicle is not constant depending on the vehicle type, for example, and if the mounting position is different, the imaging direction in which the imaging device 2 takes images differs, and therefore the imaging range of the input image also differs. In the input image captured by the imaging device 2, the range of the first image area 41 in which the course line 50 is drawn by the navigation device 3 also changes depending on the mounting position. Therefore, the display device 4 requires a table for each mounting position that associates the output coordinates and input coordinates of images that are different for each mounting position, which increases the memory capacity and increases the cost of the product.

Further, in the input image captured by the imaging device 2, the range of the first image area 41 in which the course line 50 is drawn by the navigation device 3 changes according to changes in vehicle information such as the rotation angle of the steering wheel of the vehicle. Therefore, the display device 4 requires a table that associates the output coordinates and input coordinates of different images for each rotation angle of the handle, which increases the memory capacity and increases the cost of the product.

Therefore, in order to provide an image display system 1 that can adapt to changes in the range within the first image area 41 in which the course line 50 is drawn, an embodiment thereof will be described below as Example 1. Note that the same configurations as the base technology are given the same reference numerals, and explanations of the overlapping configurations and operations will be omitted.

FIG. 14 is an explanatory diagram showing an example of the configuration of the imaging device 2A according to the first embodiment. The control unit 20 in the imaging device 2A shown in FIG. 14 includes a first conversion unit 21, a second conversion unit 22, a calculation unit 23, a third conversion unit 24, and a synthesis unit 25. The storage unit 30 in the imaging device 2A includes a first conversion table 31, a distortion table 31A, a second conversion table 32, a ratio table 33, and a third conversion table 34.

FIG. 15A is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-1 mounting position. Note that the 1-1 mounting position is, for example, the mounting position of the imaging device 2A mounted at the rear of the vehicle at a height of 100 cm, in the center of the left and right sides, and at a tilt angle of 40 degrees. The image shown in FIG. 15A is an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-1 mounting position, and is an image that includes the 1-1 predetermined image area S1. . Note that the 1-1 predetermined image area S1 shown in FIG. 15A is an area actually drawn on the ground for convenience of explanation, and includes a line at a depth of 50 cm from the rear end of the vehicle, a line at a depth of 100 cm from the rear end of the vehicle, and a line at a depth of 100 cm. It shall include a 200cm line and a line indicating the vehicle width.

FIG. 15B is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-2 mounting position. Note that the 1-2 mounting position is, for example, the mounting position of the imaging device 2A mounted at the rear of the vehicle with a height of 80 cm, a leftward position of 5 cm, and a tilt angle of 40 degrees. The image shown in FIG. 15B is also an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-2 mounting position, and is an image that includes the 1-2 predetermined image area S2. . Note that the 1-2 predetermined image area S2 shown in FIG. 15B is an area actually drawn on the ground for convenience of explanation, and includes a line at a depth of 50 cm from the rear end of the vehicle, a line at a depth of 100 cm, and a line at a depth of 100 cm from the rear end of the vehicle. It shall include a 200cm line and a line indicating the vehicle width. Since the image shown in FIG. 15A and the image shown in FIG. 15B have different imaging ranges, the ranges of the 1-1 predetermined image area S1 and the 1-2 predetermined image area S2 are different.

FIG. 16A is an explanatory diagram showing an example of the 1-1 predetermined image area S1 in the first distortion-corrected image captured by the imaging device 2A at the 1-1 mounting position. The image shown in FIG. 16A is an image after the first distortion correction taken by the imaging device 2A at the 1-1 mounting position. The 1-1 predetermined image area S1 can be specified in the image by enclosing the central part of the image shown in FIG. 16A, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle. The 1-1 predetermined image area S1 becomes a part of the first image area 41α in the first distortion-corrected image captured by the imaging device 2A at the 1-1 attachment position.

FIG. 16B is an explanatory diagram showing an example of the 1-2 predetermined image area S2 in the first distortion-corrected image captured by the imaging device 2A at the 1-2 mounting position. The image shown in FIG. 16B is an image after the first distortion correction taken by the imaging device 2A at the 1-2 mounting position. By enclosing the central portion of the image shown in FIG. 16B, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle, the 1-2nd predetermined image area S2 can be specified within the image. The 1-2 predetermined image area S2 becomes a part of the first image area 41α in the first distortion-corrected image captured by the imaging device 2A at the 1-2 mounting position.

FIG. 17A shows the 1-1 predetermined image area S1 in the image after the first distortion correction at the 1-1 mounting position and the image areas other than the 1-1 predetermined image area S1 in brightness and darkness. It is an explanatory diagram expressed. The 1-1 predetermined image area S1 in the image is expressed as a white background, as shown in FIG. 17A. Image areas other than the 1-1 predetermined image area S1 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 17A.

FIG. 17B shows the brightness and darkness of the 1-2 predetermined image area S2 in the image after the first distortion correction at the 1-2 mounting position and the image areas other than the 1-2 predetermined image area S2. It is an explanatory diagram expressed. The 1-2nd predetermined image area S2 in the image is expressed as a white background, as shown in FIG. 17B. Image areas other than the 1-2 predetermined image area S2 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 17B.

FIG. 18 is an explanatory diagram in which the first image area 41α and image areas other than the first image area 41α are expressed in brightness and darkness. The first image area 41α shown in FIG. 18 includes all of the 1-1 predetermined image area S1 shown in FIG. 17A and the 1-2 predetermined image area S2 shown in FIG. 17B, as indicated by the white background. This is the image area. Image areas other than the first image area 41α are a second image area 42α and a third image area 43α, as shown in black.

FIG. 19 is an explanatory diagram in which the first image area 41α, the second image area 42α, and the third image area 43α of the first embodiment are expressed in brightness and darkness. The first image area 41α shown in FIG. 19 is an image area that includes both the 1-1st predetermined image area S1 and the 1-2nd predetermined image area S2. In the image shown in FIG. 19, a first image area 41α is a white background, a second image area 42α is a black background, and a third image area is a boundary between the first image area 41α and the second image area 42α. 43α is expressed with gradation. A white background with a luminance of 100% indicates the first image area 41α. A black background with a luminance of 0% indicates the second image area 42α. A portion where the brightness is less than 100% white and more than 0% black indicates the third image area 43α. The part expressed by this light-dark gradation is obtained by weighting the average of the brightness of the pixel at the output coordinate output by the first distortion correction and the brightness of the pixel at the output coordinate output by the second distortion correction. The weighted average ratio in the third distortion correction is shown. Note that the detailed method of weighted averaging will be described later.

The first conversion unit 21 performs first distortion correction on each pixel within the first image area 41α of the input image 40 and outputs the resultant image. Note that the pixels within the first image area 41α can also be said to be the pixel value of the first input coordinates, for example. The first conversion unit 21 refers to the first conversion table 31, for example, and converts the pixel value of the first input coordinates corrected for the distortion in the first image area 41α in the input image 40 into the pixel value in the output image. It is output to the synthesis unit 25 as a pixel value of the output coordinates. The first conversion table 31 is a table that manages the first input coordinates of pixel values whose distortions in the input image have been corrected in association with each output coordinate of the output image.

The second conversion unit 22 performs the second distortion correction on each pixel within the second image area 42α of the input image 40 and outputs the result. Note that the pixels within the second image area 42α can be said to be, for example, the pixel values of the second input coordinates. The second conversion unit 22 refers to the second conversion table 32, for example, and converts the pixel value of the second input coordinate, which has been corrected for distortion in the second image area 42α in the input image 40, into the output image. The pixel value is outputted to the synthesis unit 25 as a pixel value at the second output coordinate. The second conversion table 32 is a table that manages second input coordinates of pixel values whose second distortion has been corrected in association with each second input coordinate within the second image area 42 .

The calculation unit 23 calculates, for each output coordinate of the distortion-corrected pixel value in the third image area 43α in the input image 41, the output coordinate in the first image area 41α nearest to the output coordinate in the third image area 43α. The ratio between the first distance to the output coordinates of and the second distance from the output coordinates in the third image area 43α to the nearest output coordinates in the second image area 42α is calculated. Further, the calculation unit 23 stores the ratio as a percentage in the ratio table 33 for each output coordinate.

As shown in FIG. 19, the ratio table 33 manages weighted average ratios for each pixel in the first image area 41α, the second image area 42α, and the third image area 43α. The first image area 41α is, for example, a range that includes all of the 1-1 to 1-n predetermined image areas S1 to Sn in the input image captured for each of the n mounting positions.

The calculation unit 23 inputs the output obtained by applying the first distortion correction to each pixel of the input image 41 and the output obtained by applying the second distortion correction to each pixel, and outputs a weighted average of them. . For example, the calculation unit 23 refers to the ratio table 33 and corrects the first input coordinate and the second input coordinate according to the ratio for each output coordinate (first distance: second distance). The calculation unit 23 calculates the third input coordinates of the output image by adding the first input coordinates after the ratio correction and the second input coordinates after the ratio correction.

The calculation unit 23 calculates, for example, the first input coordinate of the pixel value with the first distortion corrected by f(x,y). The calculation unit 23 calculates, for example, the second input coordinate of the pixel value with the second distortion corrected by g(x,y). The calculation unit 23 calculates, for example, the third input coordinate of the pixel value with the third distortion corrected using (Equation 1). The calculation unit 23 performs weighted average processing. The calculation unit 23 refers to the ratio table 33, and when the ratio of the output coordinates is n%, the calculation unit 23 corrects the ratio of the first input coordinates using the first input coordinates f(x, y)×n%. Further, the calculation unit 23 refers to the ratio table 33 and corrects the second input coordinates by the second input coordinates g(x,y)×(100%−n%). Further, the calculation unit 23 calculates third input coordinates for each output coordinate by adding the first input coordinates after the ratio correction and the second input coordinates after the ratio correction. In other words, the calculation unit 23 refers to the ratio table 33 and calculates the third input coordinate using (Formula 1) of f(x, y) x n% + g(x, y) x (100% - n%). Calculate each output coordinate. The calculation unit 23 refers to the ratio corresponding to the output coordinate in the ratio table 33, and, for example, if the ratio of the output coordinate is 50%, the calculation unit 23 calculates 50% of the first input coordinate corresponding to the output coordinate and the output coordinate. and 50% of the corresponding second input coordinates. Then, the calculation unit 23 calculates the added value as the third input coordinate, and stores the third input coordinate corresponding to the output coordinate in the third conversion table 34.

The calculation unit 23 stores the third input coordinates in the third conversion table 34 for each output coordinate. The third conversion unit 24 performs third distortion correction on each pixel within the third image area 43α of the input image and outputs the resultant image. Note that the pixels within the third image area 43α can be said to be, for example, the pixel value of the third input coordinate. For example, the third conversion unit 24 refers to the third conversion table 34 and converts the pixel values of the third input coordinates that have been corrected for distortion in the third image area 43α in the input image 40 into the third image. It is output as a pixel value at the output coordinates within the area 43α. The third converter 24 outputs pixel values for each output coordinate within the third image area 43 to the synthesizer 25.

In other words, the first conversion unit 21 converts the pixel value of the first input coordinate corrected for the first distortion in the first image area 41α in the input image into the pixel value of the output coordinate in the first image area 41α. It is output to the synthesis unit 25 as a value. The second conversion unit 22 converts the pixel value of the second input coordinate, which has been corrected for the second distortion within the second image area 42α in the input image, into the pixel value of the output coordinate within the second image area 42α. It is output to the synthesis section 25. The third conversion unit 24 refers to the third conversion table 34 and converts the pixel value of the third input coordinate corrected for the third distortion in the third image area 43α in the input image into the third conversion table 34. It is output to the synthesis unit 25 as a pixel value at the output coordinates within the image area 43α.

The combining unit 25 combines pixel values of output coordinates within the first image area 41α from the first conversion unit 21 and pixel values of output coordinates within the second image area 42α from the second conversion unit 22. , and the pixel values of the output coordinates in the third image area 43α from the third conversion unit 24 are combined and output to the navigation device 3.

FIG. 20A is an explanatory diagram showing an example of an output image captured by the imaging device 2A at the 1-1 mounting position. Note that, as shown in FIG. 20A, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a second image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41A and the second image area 42α. The synthesizing unit 25 can smoothly output an output image with excellent visibility even when the input image of the 1-1 mounting position is acquired. Then, the navigation device 3 draws a course line 50 in the area above the first image region 41α in the output image from the composition unit 25 even in the image captured at the 1-1 mounting position, as shown in FIG. 20A. The output image after drawing is output to the display device 4 as shown in FIG.

Further, FIG. 20B is an explanatory diagram showing an example of an output image when imaged at the 1-2 mounting position. Note that, as shown in FIG. 20B, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a third image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41α and the second image area 42α. The synthesizing unit 25 can smoothly output an output image with excellent visibility even when the input image of the 1-2 mounting position is acquired. Then, the navigation device 3 draws a course line 50 in the area above the first image region 41α in the output image from the composition unit 25 even in the image captured at the 1-2 mounting position, as shown in FIG. 20B. The output image after drawing is output to the display device 4 as shown in FIG.

In the imaging device 2A of the first embodiment, the first image area 41α is a range that includes all of the predetermined image areas S1 to Sn that are different for each of the n mounting positions, and the second image area 41α is based on the first image area 41α. The image area 42α and the third image area 43α are identified. Furthermore, the imaging device 2A creates a ratio table 33 that indicates the ratio of the first input coordinate and the second input coordinate for each pixel in the first image area 41α, the second image area 42α, and the third image area 43α. generate. Further, the imaging device 2A generates a third conversion table 34 based on the ratio table 33. The imaging device 2A then refers to the third conversion table 34 and executes image conversion processing for distortion correction. As a result, even if the mounting position of the imaging device 2A is between the 1-1st mounting position and the 1-nth mounting position, the first distortion correction, the first The second distortion correction and the third distortion correction can be smoothly realized. In other words, it is possible to minimize the portion that corrects only the distortion of the lens itself, which is insufficient to correct distortion for easy viewing, and by increasing the area that looks natural, it is possible to maintain good visibility.

That is, the imaging device 2A converts the third conversion table 34 based on the first image area 41α formed from the predetermined image areas S1 to Sn from the 1-1st acquisition position to the 1-nth acquisition position. generate. As a result, even when an input image of a mounting position within the range from the 1-1 mounting position to the 1-n mounting position is acquired, the third conversion table 34 is referred to and the change in the mounting position is adapted. It is possible to output an output image with good visibility. Moreover, in the display device 4, there is no need to provide a table for associating output coordinates with input coordinates for each mounting position, so the memory capacity can be reduced and an increase in product cost can be suppressed.

In the imaging device 2A of the first embodiment, for example, a case is illustrated in which the range of the first image area 41α is defined by the predetermined image areas S1 and S2 at two mounting positions. However, it is not limited to two attachment positions, and the first image area 41α may be the predetermined image areas S1 to Sn of n attachment positions, and can be changed as appropriate.

In the imaging device 2A of the first embodiment, the range of the first image area 41α is exemplified by the predetermined image regions S1 to Sn that differ for each mounting position, but even in the case of the same mounting position, the lens of the imaging device 2A The range of the n predetermined image areas S11 to S1n differs for each of the 11 tilt angles. Note that the tilt angle is the vertical tilt angle of the lens 11 of the imaging device 2A. Therefore, the first image area 41α may be comprised of predetermined image areas S11 to Sn that differ for each tilt angle, and an embodiment thereof will be described below as Example 2. Note that the same components as those in the first embodiment are given the same reference numerals, and explanations of the overlapping components and operations will be omitted.

FIG. 21A is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-1 tilt angle position of the second embodiment. The 1-1st tilt angle position is, for example, the mounting position of the imaging device 2A attached to the rear of the vehicle at a height of 100 cm, in the center of the left and right, and with a tilt angle of 0 degrees. The image shown in FIG. 21A is an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-1st tilt angle position, and includes the 1-1st predetermined image area S11. For convenience of explanation, the 1-1 predetermined image area S11 shown in FIG. 21A is an area actually drawn on the ground, including a line at a depth of 50 cm from the rear end of the vehicle, a line at a depth of 100 cm, and a line at a depth of 200 cm. line and a line indicating the vehicle width.

FIG. 21B is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-2nd tilt angle position. The 1-2nd tilt angle position is, for example, the mounting position of the imaging device 2A attached to the rear of the vehicle at a height of 100 cm, at the center of the left and right, and at a tilt angle of 20 degrees. The image shown in FIG. 21B is also an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-2 tilt angle position, and includes the 1-2 predetermined image area S12. be. Note that the 1-2 predetermined image area S12 shown in FIG. 21B is an area actually drawn on the ground for convenience of explanation, and includes a line at a depth of 50 cm from the rear end of the vehicle, a line at a depth of 100 cm, and a line at a depth of 100 cm. It shall include a 200cm line and a line indicating the vehicle width.

FIG. 21C is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-3rd tilt angle position. The 1-3rd tilt angle position is, for example, the mounting position of the imaging device 2A attached to the rear of the vehicle at a height of 100 cm, in the center of the left and right, and at a tilt angle of 40 degrees. The image shown in FIG. 21C is also an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-3rd tilt angle position, and includes the 1-3rd predetermined image area S13. be. For convenience of explanation, the 1-3rd predetermined image area S13 shown in FIG. It shall include a 200cm line and a line indicating the vehicle width.

FIG. 21D is an explanatory diagram showing an example of an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-4th tilt angle position. The 1-4th tilt angle position is, for example, the mounting position of the imaging device 2A attached to the rear of the vehicle at a height of 100 cm, in the center of the left and right, and at a tilt angle of 60 degrees. The image shown in FIG. 21D is also an image after the first distortion correction of the original image captured by the imaging device 2A at the 1-4th tilt angle position, and includes the 1-4th predetermined image area S14. be. For convenience of explanation, the 1-4th predetermined image area S14 shown in FIG. It shall include a 200cm line and a line indicating the vehicle width. When the tilt angles are different even at the same mounting position, the ranges of the 1-1 to 1-4 predetermined image areas S11 to S14 are different as shown in FIGS. 21A to 21D.

FIG. 22A is an explanatory diagram showing an example of the 1-1 predetermined image area S11 in the first distortion-corrected image captured by the imaging device 2A at the 1-1 tilt angle position. The image shown in FIG. 22A is an image after the first distortion correction taken by the imaging device 2A at the 1-1st tilt angle position. The 1-1 predetermined image area S11 can be specified in the image by enclosing the central portion of the image shown in FIG. 22A, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle. The 1-1 predetermined image area S11 becomes a part of the first image area 41α in the first distortion-corrected image captured by the imaging device 2A at the 1-1 tilt angle position.

FIG. 22B is an explanatory diagram showing an example of the 1-2 predetermined image area S12 in the first distortion-corrected image captured by the imaging device 2A at the 1-2 tilt angle position. The image shown in FIG. 22B is an image after the first distortion correction taken by the imaging device 2A at the 1-2nd tilt angle position. By enclosing the central portion of the image shown in FIG. 22B, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle, the 1-2nd predetermined image area S12 can be specified within the image. The 1-2 predetermined image area S12 becomes a part of the first image area 41α in the first distortion-corrected image captured by the imaging device 2A at the 1-2 tilt angle position.

FIG. 22C is an explanatory diagram showing an example of the 1-3rd predetermined image area S13 in the first distortion-corrected image captured by the imaging device 2A at the 1-3rd tilt angle position. The image shown in FIG. 22C is an image after the first distortion correction taken by the imaging device 2A at the 1st to 3rd tilt angle positions. The first to third predetermined image areas S13 can be specified in the image by enclosing the central part of the image shown in FIG. 22C, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle. The 1-3rd predetermined image area S13 becomes a part of the first image area 41α in the first distortion-corrected image captured by the imaging device 2A at the 1-3rd tilt angle position.

FIG. 22D is an explanatory diagram showing an example of the 1-4th predetermined image area S14 in the first distortion-corrected image captured by the imaging device 2A at the 1-4th tilt angle position. The image shown in FIG. 22D is an image after the first distortion correction taken by the imaging device 2A at the 1st to 4th tilt angle positions. By enclosing the central part of the image shown in FIG. 22D, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle, the 1st to 4th predetermined image areas S14 can be identified within the image. The 1-4th predetermined image area S14 becomes a part of the first image area 41α in the first distortion-corrected image captured by the imaging device 2A at the 1-4th tilt angle position.

FIG. 23A shows the brightness and darkness of the 1-1 predetermined image area S11 in the image after the first distortion correction at the 1-1 tilt angle position and the image areas other than the 1-1 predetermined image area S11. It is an explanatory diagram expressed in . The 1-1 predetermined image area S11 in the image is expressed as a white background, as shown in FIG. 23A. Image areas other than the 1-1 predetermined image area S11 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 23A.

FIG. 23B shows the brightness and darkness of the 1-2 predetermined image area S12 in the image after the first distortion correction at the 1-2 tilt angle position and the image area other than the 1-2 predetermined image area S12. It is an explanatory diagram expressed in . The 1-2nd predetermined image area S12 in the image is expressed as a white background, as shown in FIG. 23B. Image areas other than the 1-2 predetermined image area S12 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 23B.

FIG. 23C shows the brightness and darkness of the 1-3 predetermined image area S13 in the image after the first distortion correction at the 1-3 tilt angle position and the image area other than the 1-3 predetermined image area S13. It is an explanatory diagram expressed in . The 1-3rd predetermined image area S13 in the image is expressed as a white background, as shown in FIG. 23C. Image areas other than the 1-3rd predetermined image area S13 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 23C.

FIG. 23D shows the brightness and darkness of the 1-4 predetermined image area S14 in the image after the first distortion correction at the 1-4 tilt angle position and the image area other than the 1-4 predetermined image area S14. It is an explanatory diagram expressed in . The 1st-4th predetermined image area S14 in the image is expressed as a white background, as shown in FIG. 23D. Image areas other than the 1-4th predetermined image area S14 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 23D.

FIG. 24 is an explanatory diagram in which the first image area 41α and image areas other than the first image area 41α are expressed in brightness and darkness. The first image area 41α shown in FIG. 24 is an image area that includes all of the 1-1 to 1-4 predetermined image areas S11 to S14 shown in FIGS. 23A to 23B, as indicated by the white background. Image areas other than the first image area 41α are a second image area 42α and a third image area 43α, as shown in black.

FIG. 25 is an explanatory diagram in which the first image area 41α, the second image area 42α, and the third image area 43α of the second embodiment are expressed in brightness and darkness. The first image area 41α shown in FIG. 25 is an image area that includes all of the 1-1 to 1-4 predetermined image areas S11 to S14. In the image shown in FIG. 25, a first image area 41α is a white background, a second image area 42α is a black background, and a third image area is a boundary between the first image area 41α and the second image area 42α. 43α can be expressed as a gradation. A white background with a luminance of 100% indicates the first image area 41α. A black background with a luminance of 0% indicates the second image area 42α. A portion where the brightness is less than 100% white and more than 0% black indicates the third image area 43α. This brightness is determined by the ratio of the weighted average in the third distortion correction, which is a weighted average of the output coordinates output by the first distortion correction, the image, and the output coordinates output by the second distortion correction to obtain new output coordinates. It shows.

As shown in FIG. 25, the ratio table 33 manages weighted average ratios for each pixel in the first image area 41α, the second image area 42α, and the third image area 43α. The first image area 41α is, for example, a range that includes all of the 1-1 to 1-n predetermined image areas S1 to Sn in the input image captured for each of the n mounting positions. That is, the ratio table 33 manages the ratio of the weighted average of the first input coordinate and the second input coordinate for each output coordinate in the image including the first image area 41α.

The first conversion unit 21 converts the pixel value of the first input coordinate in which the first distortion in the first image area 41α in the input image is corrected to the pixel value of the output coordinate in the first image area 41α. It is output to the synthesis section 25. The second conversion unit 22 converts the pixel value of the second input coordinate, which has been corrected for the second distortion within the second image area 42α in the input image, into the pixel value of the output coordinate within the second image area 42α. It is output to the synthesis section 25. The third conversion unit 24 refers to the third conversion table 34 and converts the pixel values of the third input coordinates corrected for the third distortion of the third image area 43α in the input image into the third image. It is output to the synthesis unit 25 as a pixel value at the output coordinates within the region 43α.

The combining unit 25 combines pixel values of output coordinates within the first image area 41α from the first conversion unit 21 and pixel values of output coordinates within the second image area 42α from the second conversion unit 22. , and the pixel values of the output coordinates in the third image area 43α from the third conversion unit 24 are combined and output to the navigation device 3.

FIG. 26A is an explanatory diagram showing an example of an output image captured by the imaging device 2A at the 1-1st tilt angle position. Note that, as shown in FIG. 26A, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a third image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41α and the second image area 42α. The synthesizing unit 25 can smoothly output an output image with excellent visibility even when the input image at the 1-1 tilt angle position is acquired. Then, the navigation device 3 draws the course line 50 in the area above the first image region 41α in the output image from the combining unit 25 even in the image captured at the 1-1 tilt angle position, and As shown, the output image after drawing is output to the display device 4.

Further, FIG. 26B is an explanatory diagram showing an example of an output image when imaged by the imaging device 2A at the 1-2nd tilt angle position. Note that, as shown in FIG. 26B, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a third image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41α and the second image area 42α. Even when the input image at the 1-2nd tilt angle position is acquired, the combining unit 25 can smoothly output an output image with excellent visibility. Then, the navigation device 3 draws the course line 50 in the area above the first image region 41α in the output image from the composition unit 25 even in the image captured at the 1-2nd tilt angle position, and As shown, the output image after drawing is output to the display device 4.

Further, FIG. 26C is an explanatory diagram showing an example of an output image when an image is captured by the imaging device 2A at the 1st to 3rd tilt angle position. Note that, as shown in FIG. 26C, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a third image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41α and the second image area 42α. The synthesizing unit 25 can smoothly output an output image with excellent visibility even when the input image at the 1st to 3rd tilt angle positions is acquired. Then, the navigation device 3 draws the course line 50 in the area above the first image region 41α in the output image from the composition unit 25 even in the image captured at the 1st to 3rd tilt angle position, and draws the course line 50 in the area above the first image region 41α in the output image from the composition unit 25, and as shown in FIG. 26C. As shown, the output image after drawing is output to the display device 4.

Further, FIG. 26D is an explanatory diagram showing an example of an output image when imaged by the imaging device 2A at the 1st to 4th tilt angle positions. Note that, as shown in FIG. 26D, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a third image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41α and the second image area 42α. The synthesizing unit 25 can smoothly output an output image with excellent visibility even when the input image at the 1-4th tilt angle position is acquired. Then, the navigation device 3 draws the course line 50 in the area above the first image area 41α in the output image from the composition unit 25 even in the image captured at the 1-4th tilt angle position, and As shown, the output image after drawing is output to the display device 4.

In the imaging device 2A of the second embodiment, the range of the first image area 41α is defined as a range that includes all of the predetermined image areas S11 to Sn that differ for each of the n tilt angle positions, and the first image area 41α is the basic image area. Then, the second image area 42α and the third image area 43α are specified. Furthermore, the imaging device 2A creates a ratio table 33 that indicates the ratio of the first input coordinate and the second input coordinate for each pixel in the first image area 41α, the second image area 42α, and the third image area 43α. generate. Further, the imaging device 2A generates a third conversion table 34 based on the ratio table 33. The imaging device 2A then refers to the third conversion table 34 and executes image conversion processing for distortion correction. As a result, even if the tilt angle position of the imaging device 2A is between the 1-1st tilt angle position and the 1-nth tilt angle position, the input image captured at the tilt angle position is immediately changed to the first tilt angle position. Distortion correction, second distortion correction, and third distortion correction can be smoothly realized. In other words, it is possible to minimize the portion that corrects only the distortion of the lens itself, which is insufficient to correct distortion for easy viewing, and by increasing the area that looks natural, it is possible to maintain good visibility.

That is, in the imaging device 2A, based on the first image area 41α that includes all the predetermined image areas S11 to Sn from the 1-1st tilt angle position to the 1-nth tilt angle position, A conversion table 34 is generated. As a result, even when an input image with a tilt angle position within the range from the 1-1st tilt angle position to the 1-nth tilt angle position is acquired, the tilt angle position is changed with reference to the third conversion table 34. It is possible to output an output image with good visibility that adapts to changes in color. Moreover, in the display device 4, there is no need to provide a table for associating output coordinates and input coordinates for each tilt angle position, so the memory capacity can be reduced and an increase in product cost can be suppressed.

In the imaging device 2A of the second embodiment, the tilt angles are 0 degrees, 20 degrees, 40 degrees, and 60 degrees, but the invention is not limited to these. It's fine and can be changed as appropriate.

In the imaging device 2A of the second embodiment, a case has been exemplified in which the range of the first image area 41α is determined from the predetermined image area Sn, which differs for each tilt angle position. However, it goes without saying that the present invention is not limited to the tilt angle, and may be applied to, for example, the pan angle, roll angle, etc., and similar effects can be obtained. Note that the pan angle is, for example, a tilt angle in the left-right direction, and the roll angle is, for example, a tilt angle in the rotation direction.

Furthermore, in the imaging device 2A of Examples 1 and 2, the case where the range of the first image area 41α is defined by the predetermined image areas S1 (S11) to Sn, which differ depending on the mounting position and the tilt angle position, has been exemplified. However, the range of the first image area 41α may be determined by the predetermined image areas S1 (S11) to Sn in accordance with vehicle information such as the rotation angle of the vehicle steering wheel. explain.

FIG. 27 is an explanatory diagram showing an example of the relationship between the vehicle rear end position and the vehicle trajectory when the vehicle is reversed by rotating the steering wheel counterclockwise at five different steering angles. FIG. 27 shows the changes in the right side trajectory of the vehicle rear end, the left-right center trajectory of the vehicle rear end, and the left side trajectory of the vehicle rear end when the center position of the vehicle rear end is retreated by 50 cm, 100 cm, and 200 cm. There is.

FIG. 28A is an explanatory diagram showing an example of an image after the first distortion correction of the original image at the 1-1st handle rotation angle in Example 3. The 1-1 steering wheel rotation angle is, for example, when the vehicle steering wheel is rotated two times to the left with the imaging device 2A attached to the rear of the vehicle at a height of 100 cm, center left and right, and a tilt angle of 0 degrees. . The image shown in FIG. 28A is an image after the first distortion correction of the original image at the 1-1st handle rotation angle, and includes the 1-1st predetermined image area S21. Note that the 1-1 predetermined image area S21 shown in FIG. 28A is an area actually drawn on the ground for convenience of explanation, and includes a line at a depth of 50 cm from the rear end of the vehicle, a line at a depth of 100 cm from the rear end of the vehicle, and a line at a depth of 100 cm. It shall include a 200cm line and a line indicating the vehicle width.

FIG. 28B is an explanatory diagram showing an example of an image after the first distortion correction of the original image at the 1-2nd handle rotation angle. The 1-2 steering wheel rotation angle is, for example, when the vehicle steering wheel is rotated one turn to the left with the imaging device 2A attached to the rear of the vehicle at a height of 100 cm, center left and right, and a tilt angle of 0 degrees. . The image shown in FIG. 28B is also an image after the first distortion correction of the original image at the 1-2nd handle rotation angle, and includes the 1-2nd predetermined image area S22. Note that the 1-2 predetermined image area S22 shown in FIG. 28B is an area actually drawn on the ground for convenience of explanation, and includes a line at a depth of 50 cm from the rear end of the vehicle, a line at a depth of 100 cm from the rear end of the vehicle, and a line at a depth of 100 cm. It shall include a 200cm line and a line indicating the vehicle width.

FIG. 28C is an explanatory diagram showing an example of an image after the first distortion correction of the original image at the 1st to 3rd steering wheel rotation angles. The 1-3rd steering wheel rotation angle is, for example, a state in which the vehicle steering wheel is rotated to 0 with the imaging device 2A attached to the rear of the vehicle at a height of 100 cm, in the center left and right, and a tilt angle of 0 degrees. The image shown in FIG. 28C is also an image after the first distortion correction of the original image at the 1-3rd handle rotation angle, and includes the 1-3rd predetermined image area S23. For convenience of explanation, the 1-3rd predetermined image area S23 shown in FIG. It shall include a 200cm line and a line indicating the vehicle width.

FIG. 28D is an explanatory diagram showing an example of an image after the first distortion correction of the original image at the 1st to 4th steering wheel rotation angles. The 1st-4th steering wheel rotation angle is, for example, when the vehicle steering wheel is rotated one turn to the right with the imaging device 2A attached to the rear of the vehicle at a height of 100 cm, center left and right, and a tilt angle of 0 degrees. . The image shown in FIG. 28D is also an image after the first distortion correction of the original image at the 1-4th handle rotation angle, and includes the 1-4th predetermined image area S24. For convenience of explanation, the 1-4th predetermined image area S24 shown in FIG. It shall include a 200cm line and a line indicating the vehicle width.

FIG. 28E is an explanatory diagram showing an example of an image after the first distortion correction of the original image at the 1st to 5th handle rotation angles. The first to fifth steering wheel rotation angles are, for example, when the vehicle steering wheel is rotated two times to the right with the imaging device 2A attached to the rear of the vehicle at a height of 100 cm, center left and right, and a tilt angle of 0 degrees. . The image shown in FIG. 28E is also an image after the first distortion correction of the original image at the 1-5th handle rotation angle, and includes the 1-5th predetermined image area S25. For convenience of explanation, the 1-5th predetermined image area S25 shown in FIG. It shall include a 200cm line and a line indicating the vehicle width. When the steering wheel rotation angle changes, the ranges of the 1-1 to 1-5 predetermined image areas S21 to S25 differ as shown in FIGS. 29A to 29E.

FIG. 29A is an explanatory diagram showing an example of the 1-1 predetermined image area S21 in the image after the first distortion correction at the 1-1 handle rotation angle. The image shown in FIG. 29A is an image after the first distortion correction at the 1-1st handle rotation angle. The 1-1 predetermined image area S21 can be specified in the image by enclosing the central part of the image shown in FIG. 29A, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle. The 1-1 predetermined image area S21 becomes a part of the first image area 41α in the image after the first distortion correction at the 1-1 handle rotation angle.

FIG. 29B is an explanatory diagram showing an example of the 1-2 predetermined image area S22 in the image after the first distortion correction at the 1-2 handle rotation angle. The image shown in FIG. 29B is an image after the first distortion correction at the 1-2nd handle rotation angle. By enclosing the central part of the image shown in FIG. 29B, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle, the 1-2nd predetermined image area S22 can be specified within the image. The 1-2 predetermined image area S22 becomes a part of the first image area 41α in the image after the first distortion correction at the 1-2 handle rotation angle.

FIG. 29C is an explanatory diagram showing an example of the 1-3 predetermined image area S23 in the image after the first distortion correction at the 1-3 handle rotation angle. The image shown in FIG. 29C is an image after the first distortion correction at the 1st to 3rd handle rotation angles. The first to third predetermined image areas S23 can be specified in the image by enclosing the central part of the image shown in FIG. 29C, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle. The 1-3rd predetermined image area S23 becomes a part of the first image area 41α in the image after the first distortion correction at the 1-3rd handle rotation angle.

FIG. 29D is an explanatory diagram showing an example of the 1-4th predetermined image area S24 in the image after the first distortion correction at the 1-4th steering wheel rotation angle. The image shown in FIG. 29D is an image after the first distortion correction at the 1st to 4th handle rotation angles. By enclosing the central portion of the image shown in FIG. 29D, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle, the 1st to 4th predetermined image areas S24 can be identified within the image. The 1-4th predetermined image area S24 becomes a part of the first image area 41α in the image after the first distortion correction at the 1-4th handle rotation angle.

FIG. 29E is an explanatory diagram showing an example of the 1-5th predetermined image area S25 in the image after the first distortion correction at the 1-5th steering wheel rotation angle. The image shown in FIG. 29E is an image after the first distortion correction at the 1st to 5th handle rotation angles. By enclosing the central portion of the image shown in FIG. 29E, for example, a range of 200 cm depth x vehicle width from the rear end of the vehicle, the 1st to 5th predetermined image areas S25 can be identified within the image. The 1-5th predetermined image area S25 becomes a part of the first image area 41α in the image after the first distortion correction at the 1-5th handle rotation angle.

FIG. 30A shows the brightness and darkness of the 1-1 predetermined image area S21 in the image after the first distortion correction at the 1-1 handle rotation angle and the image areas other than the 1-1 predetermined image area S21. It is an explanatory diagram expressed in . The 1-1 predetermined image area S21 in the image is expressed as a white background, as shown in FIG. 30A. Image areas other than the 1-1 predetermined image area S21 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 30A.

FIG. 30B shows the brightness and darkness of the 1-2 predetermined image area S22 in the image after the first distortion correction at the 1-2 handle rotation angle and the image areas other than the 1-2 predetermined image area S22. It is an explanatory diagram expressed in . The 1-2nd predetermined image area S22 in the image is expressed as a white background, as shown in FIG. 30B. Image areas other than the 1-2 predetermined image area S22 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 30B.

FIG. 30C shows the brightness and darkness of the 1-3 predetermined image area S23 in the image after the first distortion correction at the 1-3 handle rotation angle and the image areas other than the 1-3 predetermined image area S23. It is an explanatory diagram expressed in . The 1-3rd predetermined image area S23 in the image is expressed as a white background, as shown in FIG. 30C. Image areas other than the 1-3rd predetermined image area S23 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 30C.

FIG. 30D shows the brightness and darkness of the 1-4 predetermined image area S24 in the image after the first distortion correction at the 1-4 handle rotation angle and the image area other than the 1-4 predetermined image area S24. It is an explanatory diagram expressed in . The 1st-4th predetermined image area S24 in the image is expressed as a white background, as shown in FIG. 30D. Image areas other than the 1-4th predetermined image area S24 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 30D.

FIG. 30E shows the brightness and darkness of the 1-5 predetermined image area S25 in the image after the first distortion correction at the 1-5 handle rotation angle and the image area other than the 1-5 predetermined image area S25. It is an explanatory diagram expressed in . The 1st to 5th predetermined image areas S25 in the image are expressed as a white background, as shown in FIG. 30E. Image areas other than the 1-5th predetermined image area S25 in the image are part of the second image area 42α and the third image area 43α, and are expressed in black as shown in FIG. 30E.

FIG. 31 is an explanatory diagram in which the first image area 41α and image areas other than the first image area 41α are expressed in brightness and darkness. The first image area 41α shown in FIG. 31 is an image area that includes all of the 1-1 to 1-5 predetermined image areas S21 to S25 shown in FIGS. 30A to 30E, as indicated by the white background. Image areas other than the first image area 41α are a second image area 42α and a third image area 43α, as shown in black.

FIG. 32 is an explanatory diagram in which the first image area 41α, the second image area 42α, and the third image area 43α of the third embodiment are expressed in brightness and darkness. The first image area 41α shown in FIG. 32 is an image area that includes all of the 1-1 to 1-5 predetermined image areas S21 to S25. In the image shown in FIG. 32, a first image area 41α is a white background, a second image area 42α is a black background, and a third image area is a boundary between the first image area 41α and the second image area 42α. 43α can be expressed as a gradation. The white background has a brightness of 100%, and thus indicates the first image area 41α. The black background has a luminance of 0%, and thus indicates the second image area 42α. A portion where the contrast is less than 100% white and more than 0% black indicates the third image area 43. This brightness is determined by the ratio of the weighted average in the third distortion correction, which is a weighted average of the output coordinates output by the first distortion correction, the image, and the output coordinates output by the second distortion correction to obtain new output coordinates. It shows.

As shown in FIG. 32, the ratio table 33 manages weighted average ratios for each pixel in the first image area 41α, the second image area 42α, and the third image area 43α. The first image area 41α is, for example, a range that includes all of the 1-1 to 1-n predetermined image areas S1 to Sn in the input image captured for each of the n mounting positions. That is, the ratio table 33 manages the ratio of the weighted average of the first input coordinate and the second input coordinate for each output coordinate in the image including the first image area 41α.

The first conversion unit 21 converts the pixel value of the first input coordinate in which the first distortion in the first image area 41α in the input image is corrected to the pixel value of the output coordinate in the first image area 41α. It is output to the synthesis section 25. The second conversion unit 22 converts the pixel value of the second input coordinate, which has been corrected for the second distortion within the second image area 42α in the input image, into the pixel value of the output coordinate within the second image area 42α. It is output to the synthesis section 25. The third conversion unit 24 refers to the third conversion table 34 and converts the pixel value of the third input coordinate corrected for the third distortion in the third image area 43α in the input image into the third conversion table 34. It is output to the synthesis unit 25 as a pixel value at the output coordinates within the image area 43α.

The combining unit 25 combines pixel values of output coordinates within the first image area 41α from the first conversion unit 21 and pixel values of output coordinates within the second image area 42α from the second conversion unit 22. , and the pixel values of the output coordinates in the third image area 43α from the third conversion unit 24 are combined and output to the navigation device 3.

FIG. 33A is an explanatory diagram showing an example of an output image at the 1-1st handle rotation angle. Note that, as shown in FIG. 33A, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a second image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41α and the second image area 42α. The synthesizing unit 25 can smoothly output an output image with excellent visibility even when the predetermined image area S21 changes according to the 1-1st handle rotation angle position. Then, even if the predetermined image area S21 changes according to the 1-1 steering wheel rotation angle position, the navigation device 3 adds a course line to the area above the first image area 41α in the output image from the combining unit 25. Draw 50. Then, the navigation device 3 outputs the rendered output image to the display device 4, as shown in FIG. 33A.

Further, FIG. 33B is an explanatory diagram showing an example of an output image at the 1-2nd handle rotation angle. Note that, as shown in FIG. 33B, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a third image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41α and the second image area 42α. The combining unit 25 can smoothly output an output image with excellent visibility even when the 1-2 predetermined image area S22 changes according to the 1-2 handle rotation angle position. Then, even if the 1-2 predetermined image area S22 is changed according to the 1-2 steering wheel rotation angle position, the navigation device 3 is configured to move A course line 50 is drawn in the area. Then, the navigation device 3 outputs the rendered output image to the display device 4, as shown in FIG. 33B.

Further, FIG. 33C is an explanatory diagram showing an example of an output image at the 1st to 3rd steering wheel rotation angles. Note that, as shown in FIG. 33C, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a third image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41α and the second image area 42α. The synthesizing unit 25 can smoothly output an output image with excellent visibility even when the 1-3rd predetermined image area S23 changes according to the 1-3rd handle rotation angle position. Then, even if the 1-3rd predetermined image area S23 is changed according to the 1-3rd steering wheel rotation angle position, the navigation device 3 is configured to A course line 50 is drawn in the area. Then, the navigation device 3 outputs the rendered output image to the display device 4, as shown in FIG. 33C.

Further, FIG. 33D is an explanatory diagram showing an example of an output image at the 1-4th steering wheel rotation angle. Note that, as shown in FIG. 33D, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a second image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41α and the second image area 42α. The synthesizing unit 25 can smoothly output an output image with excellent visibility even when the 1-4th predetermined image area S24 changes according to the 1-4th handle rotation angle position. Then, even if the 1-4th predetermined image area S24 is changed according to the 1-4th steering wheel rotation angle position, the navigation device 3 can A course line 50 is drawn in the area. Then, the navigation device 3 outputs the rendered output image to the display device 4, as shown in FIG. 33D.

Further, FIG. 33E is an explanatory diagram showing an example of an output image at the 1st to 5th steering wheel rotation angles. Note that, as shown in FIG. 33E, the output image includes a first image area 41α with the first distortion corrected, a second image area 42α with the second distortion corrected, and a second image area 42α with the third distortion corrected. This is an image including an image area 43α of 3. In the output image, for example, a third image area 43α is arranged between the first image area 41α and the second image area 42α. The combining unit 25 can smoothly output an output image with excellent visibility even when the 1-5th predetermined image area S25 changes according to the 1-5th handle rotation angle position. Then, even if the 1st-5th predetermined image area S25 changes according to the 1st-5th steering wheel rotation angle position, the navigation device 3 can A course line 50 is drawn in the area. Then, the navigation device 3 outputs the rendered output image to the display device 4, as shown in FIG. 33E.

In the imaging device 2A of the third embodiment, the first image area 41α is a range that includes all of the predetermined image areas S21 to Sn that vary depending on the rotational angular position of the handle, and the first image area 41α is The second image area 42α and the third image area 43α are specified. Furthermore, the imaging device 2A creates a ratio table 33 that indicates the ratio of the first input coordinate and the second input coordinate for each pixel in the first image area 41α, the second image area 42α, and the third image area 43α. generate. Further, the imaging device 2A generates a third conversion table 34 based on the ratio table 33. The imaging device 2A then refers to the third conversion table 34 and executes image conversion processing for distortion correction. As a result, even if the handle rotation angle is between the 1-1st handle rotation angle and the 1-nth handle rotation angle, the first distortion is immediately removed from the image of the predetermined image area changed by the handle rotation angle. Correction, second distortion correction, and third distortion correction can be smoothly realized. In other words, it is possible to minimize the portion that corrects only the distortion of the lens itself, which is insufficient to correct distortion for easy viewing, and by increasing the area that looks natural, it is possible to maintain good visibility.

That is, in the imaging device 2A, the third conversion table 34 generate. As a result, even if the predetermined image area changes according to the handle rotation angle within the range from the 1-1st handle rotation angle to the 1-nth handle rotation angle, the third conversion table 34 is referred to. It is possible to output an output image with good visibility that adapts to changes in the steering wheel rotation angle. Moreover, in the display device 4, there is no need to provide a table for associating output coordinates and input coordinates for each steering wheel rotation angle, so the memory capacity can be reduced and an increase in product cost can be suppressed.

In the imaging device 2A of the third embodiment, the first image area 41α is determined by five predetermined rotation angles of the handle: 2 rotations to the left, 1 rotation to the left, 0 rotations, 1 rotation to the right, and 2 rotations to the right. The image area is illustrated. However, the image area is not limited to the predetermined image area of five handle rotation angles, and may be a predetermined image area of n handle rotation angles of 4 or less or 6 or more, and can be changed as appropriate.

In the imaging device 2A of the third embodiment, the first image area 41α is defined by a predetermined image area that changes according to changes in the rotation angle of the handle. However, the present invention is not limited to the rotation angle of the steering wheel, and can be applied to, for example, a case where the length or width of a predetermined image area is changed depending on the vehicle speed, and can be changed as appropriate. Although a predetermined image area corresponding to a change in the rotation angle of the steering wheel is illustrated as the vehicle information, the predetermined image area may also be a predetermined image area corresponding to a change in the rotation angle of the tires of the vehicle, and can be changed as appropriate.

The imaging device 2A has a first image area that includes, for example, a first image area 41α that takes into account the mounting position shown in FIG. 18, and a first image area 41α that takes into account the tilt angle position shown in FIG. 41α may be generated, and can be changed as appropriate.

The imaging device 2A also includes, for example, a first image area 41α that takes into account the tilt angle shown in FIG. 24 and a first image area 41α that takes into account the handle rotation angle shown in FIG. An image area 41α may be generated and can be changed as appropriate.

The imaging device 2A also includes, for example, a first image area 41α that takes into account the mounting position shown in FIG. 18, and a first image area 41α that takes into account the handle rotation angle shown in FIG. An image area 41α may be generated and can be changed as appropriate.

Further, the imaging device 2A has, for example, a first image area 41α considering the mounting position shown in FIG. 18, a first image area 41α considering the tilt angle shown in FIG. 24, and a handle rotation angle shown in FIG. A first image area 41α that includes all of the first image area 41α in consideration of the above may be generated, and can be changed as appropriate.

For example, the imaging device 2A can be applied to a case where it recognizes that parallel parking is in progress and corresponds to the course line 50 drawn at a completely different position.

In the imaging device 2 (2A), the storage unit 30 such as the third conversion table 34 is stored in the RAM 15 within the imaging device 2 (2A). However, the information is not limited to the RAM 15, and may be stored in, for example, a storage device (not shown) that can be externally connected to the imaging device 2 (2A), and can be changed as appropriate.

In the imaging device 2 (2A), a first conversion table 31, a second conversion table 32, a ratio table 33, and a third conversion table 34 are stored in the RAM 15 in the imaging device 2 (2A). However, the information is not limited to the RAM 15, and may be stored in, for example, a storage device (not shown) that can be externally connected to the imaging device 2 (2A), and can be changed as appropriate.

The imaging device 2 (2A) has a built-in first conversion table 31, second conversion table 32, and ratio table 33, but if the table contents of the third conversion table 34 are stored in advance, It is also possible to incorporate only the third conversion table 34, and it can be changed as appropriate.

The imaging device 2 (2A) uses a first conversion table 31, a second conversion table 32, a third conversion table 34, and a ratio table 33. However, for each output coordinate, there is a function f for calculating the first input coordinate of the pixel value corrected for the first distortion, and a function f for calculating the second input coordinate of the pixel value corrected for the second distortion. A function g, which calculates the input coordinates of the pixel value corrected for the third distortion (Equation 1), may be stored.

In this case, the imaging device 2 (2A) can use the function f, the function g, and the formula 1 without preparing the first conversion table 31, the second conversion table 32, the third conversion table 34, and the ratio table 33. Then, the input coordinates of the pixel values subjected to the first distortion correction, the second distortion correction, and the third distortion correction are calculated. The imaging device 2 (2A) then calculates the pixel values of the output coordinates in the first image area 41 that have been corrected for the first distortion, and the pixel values of the output coordinates in the second image area 42 that have corrected the second distortion. An output image including the pixel values and the pixel values at the output coordinates in the third image area 43 after correcting the third distortion is output to the navigation device 3 .

In the above embodiment, the input image is stored in the RAM 15 or the like in the imaging device 2 (2A), but the input image may be stored in an external storage device connected to the imaging device 2 (2A), and may be changed as appropriate. It is possible.

In the above embodiment, for example, the second distortion correction is described as distortion correction according to Patent Document 1, but the invention is not limited to these distortions, and other correction methods that are easy for the user to see may be adopted.

In the above embodiment, the first distortion correction in the first image area 41 (first image area 41α), the second distortion correction in the second image area 42 (second image area 42α), the second distortion correction in the second image area 42 (second image area 42α), The image processing processor 13 in the imaging device 2 executed distortion correction processing (image processing) for performing third distortion correction in the image area 43 (third image area 43α) of No. 3. However, the distortion correction process may be executed in an external device connected to the imaging device 2, for example, the navigation device 3, and can be changed as appropriate.

In the above embodiment, the first conversion section 21 performed the first distortion correction, the second conversion section 22 performed the second distortion correction, and the third conversion section 24 performed the third distortion correction. However, these first distortion correction, second distortion correction, and third distortion correction may be performed sequentially in random order, or may be performed in parallel, and can be changed as appropriate.

The navigation device 3 of the above embodiment draws the course line 50 in the area of the first image region 41 (first image region 41α) using the distortion table 31A in the imaging device 2; The first conversion table 31 may be provided separately, and can be changed as appropriate.

In the above embodiment, the course line 50 is drawn on the output image including the first image area 41 (first image area 41α) in which the first distortion has been corrected, but the course line 50 is not limited to the course line 50. For example, a line such as a fixed course line may be drawn, and can be changed as appropriate.

2A Imaging device 13 Image processing processor 21 First conversion unit 22 Second conversion unit 23 Calculation unit 24 Third conversion unit 25 Combination unit 31 First conversion table 32 Second conversion table 33 Ratio table 34 Third conversion unit conversion table

Claims (7)

An image processing device that converts an input image and outputs an output image,
a first conversion unit that applies a first distortion correction to each pixel in a first image region of the input image and outputs it as an image;
a second conversion unit that applies a second distortion correction to each pixel in a second image region of the input image and outputs it as an image;
a third conversion unit that applies third distortion correction to each pixel in a third image region of the input image and outputs it as an image;
a combining unit that generates and outputs the output image from the image output from the first conversion unit, the image output from the second conversion unit, and the image output from the third conversion unit;
has
The first image area is
an image area that includes all predetermined image areas for drawing a course line for each of the input images of different imaging ranges ;
The output image is such that the image output from the third conversion unit is placed between the image output from the first conversion unit and the image output from the second conversion unit, and the image output from the third conversion unit is arranged between the image output from the first conversion unit and the image output from the second conversion unit, An image processing device characterized in that the image output from the first conversion unit has only distortion aberration or has no distortion. The first image area is
2. The image processing device according to claim 1, wherein the image area includes all of the predetermined image areas of the input images of the different imaging ranges captured by the imaging device from different imaging directions. 3. The image processing device according to claim 2 , wherein the image pickup device images input images in the different image pickup ranges from the different image pickup directions by changing the mounting position of the image pickup device attached to the vehicle. The first image area is
2. The image processing apparatus according to claim 1, wherein the image area includes all of the predetermined image areas that vary depending on vehicle information of the input image. The vehicle information is
5. The image processing apparatus according to claim 4 , wherein the amount of change varies in accordance with a change in a rotation angle of a steering wheel of a vehicle or a change in a rotation angle of a tire of the vehicle. The input image is generated from a subject, a lens that collects light from the subject, and an image sensor that forms an image of the collected light,
The second conversion unit is
The object that is linear toward the optical axis of the lens and the object that is linear in the vertical direction are approximately linear in the output image, and the object that is linear in the horizontal direction is in the center of the output image. , and the distortion is corrected so that the left and right portions that are connected to the left and right sides of the central portion are substantially straight, and are bent near the boundary between the central portion and the left and right portions of the output image. The image processing device according to claim 1. The third conversion unit is
A claim characterized in that an output obtained by applying the first distortion correction to each pixel of the input image and an output obtained by applying the second distortion correction are input, and a weighted average of them is output. Item 1. The image processing device according to item 1.

Images