WO2020129715A1

WO2020129715A1 - Image correction device, image correction method, and program

Info

Publication number: WO2020129715A1
Application number: PCT/JP2019/047939
Authority: WO
Inventors: 香織片岡; 慎吾安藤; 淳嵯峨田
Original assignee: 日本電信電話株式会社
Priority date: 2018-12-18
Filing date: 2019-12-06
Publication date: 2020-06-25
Also published as: JP2020099009A

Abstract

The present invention addresses the issue of appropriately correcting line image displacement in line parallel images having line images that have been captured by a line scan camera and are lined up in chronological order.　An image generation unit 22 lines up line images 60 and generates a two-dimensional line parallel image 62. A horizontal straight line detection unit 28 detects horizontal straight lines included in each of a plurality of two-dimensional images 66. An inter- camera correspondence unit 30 correlates a line image 30 and a vertical line image 67 that passes through the center of a two-dimensional image 66. A horizontal straight line correspondence unit 34 correlates horizontal straight lines between a plurality of two-dimensional images 66. A label application unit 36 maps horizontal straight lines to each of the line images 60, on the basis of the correlation between the line images 60 and the vertical line image 67, and applies a label to a region including a horizontal straight line in the line parallel image 62. The image correction unit 38 corrects the position of the line image 60 in the vertical direction in the line parallel image 62, on the basis of the region to which the label was applied.

Description

IMAGE CORRECTION DEVICE, IMAGE CORRECTION METHOD, AND PROGRAM

The present disclosure relates to an image correction device, an image correction method, and a program.

The line scan camera capable of acquiring tens of thousands of line images per second is not only used for inspection of abnormalities of products flowing on the belt conveyor in factories, but also mounted on vehicles and arranged in a line. By arranging the image sensors in the direction perpendicular to the ground (direction connecting the ground and the sky) and shooting while running, it has been noted that a high-resolution outdoor image can be acquired while running (Non-Patent Document 1). 1).

By using the line scan camera in this way, it is possible to increase the line image acquisition rate and acquire high resolution images while traveling in the vehicle in general.

However, although two-dimensional images (line parallel images) are created by arranging the line images in the running direction in time series, there may be a case where a deviation occurs between the line images due to vibration of the vehicle when the line images are acquired. is there.

Therefore, conventionally, by estimating the horizontal line included in the line image (line parallel image) and moving the arranged line images up and down by one line so that the estimated horizontal line becomes a straight line, the line in the line parallel image is There is a method for correcting image shift (Non-Patent Document 2).

However, in the conventional method of estimating the horizon, if it is known that the captured line image always contains a guideline such as a horizontal straight line such as a cable, or if the displacement due to vibration is not specified, There was a problem that the estimation of the horizon failed and the correction process failed.

The present disclosure has been made in view of the above points, and in a line parallel image in which line images captured by a line scan camera are arranged in time series, it is possible to appropriately correct the deviation of the line images. It is an object to provide a correction device, an image correction method, and a program.

In order to achieve the above object, an image correction apparatus according to a first aspect of the present disclosure is a line image that is captured by a line scan camera in which image pickup elements are arranged in one row and that extends in a direction perpendicular to a moving direction. Are arranged in time series, and an image generation unit that generates a two-dimensional line-parallel image and a two-dimensional camera in which image pickup elements are arranged in a matrix form a line scan when the line scan camera captures the line image. A horizontal straight line detection unit that detects a horizontal straight line extending in the horizontal direction included in each of a plurality of two-dimensional images obtained by photographing the same direction as the camera, the line image, and the light of the two-dimensional image. An inter-camera associating unit that associates with a vertical line image that passes through the axis center, a horizontal straight line associating unit that associates the horizontal straight lines between the plurality of two-dimensional images, and the line image. Based on the association with the vertical line image, the horizontal straight line is mapped to each of the line images, in the line parallel image, a labeling unit for giving a label to a region including the horizontal straight line, and the label. An image correction unit that corrects the position of the line image in the vertical direction in the line parallel image based on the added region.

An image correction apparatus according to a second aspect of the present disclosure is the image correction apparatus according to the first aspect, wherein the line scan camera and the two-dimensional camera are provided in a vehicle, and the line image is a tire of the vehicle. From the rotary encoder installed in the, based on the signal output each time the vehicle moves a predetermined distance, is an image captured by the line scan camera, the image generation unit, the line image in chronological order Side by side, the line parallel image with the horizontal axis being parallel projection is generated.

An image correction apparatus according to a third aspect of the present disclosure is the image correction apparatus according to the first aspect or the second aspect, wherein the horizontal straight line detection unit includes a pair of sides parallel to the vertical direction and the horizontal direction. At least one side of the pair of sides parallel to the horizontal direction of the rectangle formed by the pair of sides parallel to the horizontal line is the horizontal straight line.

An image correction apparatus according to a fourth aspect of the present disclosure is the image correction apparatus according to any one of the first to third aspects, in which the labeled unit includes the line image and the line parallel image. A horizontal straight line position calculation unit that derives a positional relationship between the two-dimensional images based on the correspondence and calculates coordinates in the real space of the horizontal straight lines that are associated between the two-dimensional images.

In order to achieve the above-mentioned object, an image correction method according to a fifth aspect of the present disclosure is directed to an image generation unit that is perpendicular to a moving direction taken by a line scan camera in which image pickup devices are arranged in one row. Generating a two-dimensional line parallel image by arranging the line images extending in the direction in time series, and the horizontal line detecting unit, the two-dimensional camera in which the image pickup elements are arranged in a matrix, the line scan camera, the line image The step of detecting a horizontal straight line extending in the horizontal direction included in each of a plurality of two-dimensional images obtained by shooting in the same direction as the line scan camera, A step of associating the line image with a vertical line image that passes through the optical axis center of the two-dimensional image; and a horizontal straight line correspondence attaching unit that associates the horizontal straight lines between the plurality of two-dimensional images. The step of performing the labeling, the labeling unit, based on the correspondence between the line image and the vertical line image, the horizontal straight line is mapped to each of the line images, the horizontal straight line in the line parallel image. And a step of correcting the position of the line image in the line parallel image in the vertical direction by the image correction section, based on the labeled area.

In order to achieve the above object, the program according to the sixth aspect of the present disclosure is a program for causing a computer to function as each unit of the image correction apparatus according to any one of the first to fourth aspects. is there.

According to the present disclosure, it is possible to appropriately correct the deviation of line images in a line parallel image in which line images captured by a line scan camera are arranged in time series.

It is a block diagram which shows the structure of an example of the image correction apparatus of embodiment. It is a figure for explaining photography of a line image and a two-dimensional image. It is a figure for explaining photography of a line image. FIG. 6 is a diagram for explaining a shift generated in a line parallel image generated by the image generation unit of the embodiment. It is a figure for demonstrating matching of a line image and a two-dimensional image by the correspondence part between cameras of an embodiment, and a base line length. FIG. 6 is a diagram illustrating correction of a line image shift by the image correction unit of the embodiment. 6 is a flowchart showing an example of an image correction processing routine in the image correction apparatus of the embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

<Structure of the image correction device of the present embodiment>
FIG. 1 is a block diagram showing the configuration of an example of the image correction apparatus 10 of this embodiment. A form in which the image correction device of the present disclosure is applied to a line parallel image generated by arranging line images captured by a line scan camera mounted on a vehicle in time series will be described.

As shown in FIG. 1, the image correction apparatus 10 according to the present embodiment includes an image generation unit 22, a horizontal straight line detection unit 28, an inter-camera correspondence unit 30, a horizontal straight line correspondence unit 34, a labeling unit 36, and an image correction unit. 38. As an example, the image correction apparatus 10 of the present embodiment includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read) that stores a program for executing an image correction processing routine described below and various data. Only Memory), and can be configured with a computer including. Specifically, the CPU that executes the above-described program causes the image generation unit 22, the horizontal straight line detection unit 28, the inter-camera correspondence unit 30, the horizontal straight line correspondence unit 34, and the labeled unit of the image correction apparatus 10 shown in FIG. It functions as the unit 36 and the image correction unit 38. Further, as shown in FIG. 1, the image correction apparatus 10 includes a line image storage unit 20, a line parallel image storage unit 24, a two-dimensional image storage unit 26, and a two-dimensional inter-image baseline length storage unit 32.

First, as shown in FIGS. 2A and 2B, in the present embodiment, a line image and a two-dimensional image are acquired by each of the line scan camera 2 and the two-dimensional camera 4 provided on the side surface of the vehicle 50. The line scan camera 2 is a camera in which the image pickup elements are arranged in one row, and is also called a “line sensor camera” or the like. In the present embodiment, the image pickup elements of the line scan camera 2 are arranged in a state of being aligned in a vertical direction B intersecting the moving direction A of the vehicle 50, and the line scan camera 2 is used to drive a line image while the vehicle 50 is traveling. 60 are photographed at regular intervals.

As a specific example, in the present embodiment, a device such as a rotary encoder that outputs a pulse signal each time the vehicle 50 moves a predetermined distance is provided on the tire or the like of the vehicle 50. The line scan camera 2 sequentially captures line images 60 extending in a vertical direction B intersecting the moving direction of the vehicle 50, that is, extending vertically to the ground, based on the pulse signals output from the device, and the captured line images. The image data representing 60 is output. Note that the present invention is not limited to this embodiment, and the line image 60 may be acquired while the vehicle 50 is moving at a constant speed. In this case as well, the line image 60 whose horizontal axis is parallel projection may be acquired. it can.

Image data representing each of the plurality of line images 60 taken by the line scan camera 2 is input to the image correction apparatus 10. Image data representing each of the plurality of line images 60 input to the image correction apparatus 10 is stored in the line image storage unit 20.

Also, the two-dimensional camera 4 is a camera in which image pickup elements are arranged in a matrix, and is a camera used for taking general photographic images. The two-dimensional camera 4 of the present embodiment is provided on the side surface of the vehicle 50 along with the line scan camera 2, and when the line scan camera 2 captures an image while the vehicle 50 is traveling, the line scan camera 2 is used. Shoot in the same direction as. As a specific example, the two-dimensional camera 4 of the present embodiment captures a moving image while the line scan camera 2 captures a plurality of line images 60. It should be noted that shooting of a moving image by the two-dimensional camera 4 can be performed inexpensively and easily, and there is little burden even if shooting is performed simultaneously with the line scan camera 2 for blur correction, which will be described later.

Image data representing the line image 60 is input to the image generation unit 22 from the line image storage unit 20. The image generation unit 22 arranges the image data representing the line image 60 in chronological order according to the moving direction A of the vehicle 50 to generate a two-dimensional line parallel image (line parallel image 62, see FIG. 3 ). In the present embodiment, a two-dimensional image obtained by arranging a plurality of line images 60 in time series is referred to as “line parallel image”. Therefore, the image of each line (one line in the vertical direction B) of the line parallel image corresponds to the line image 60 captured by the line scan camera 2.

FIG. 3 shows an example of the line parallel image 62 generated by the image generation unit 22. As shown in FIG. 3, the line parallel image 62 is an image in which the line images 60 are arranged one column at a time. Therefore, due to the vibration of the running vehicle 50 or the like, the line image 60 is vertically aligned with the line image 60. Misalignment may occur. The image correction apparatus 10 of the present embodiment corrects the shift of the line image 60 in the vertical direction B in the line parallel image 62.

The image data representing the line parallel image 62 generated by the image generation unit 22 is output to the line parallel image storage unit 24 and stored in the line parallel image storage unit 24.

On the other hand, the image data of the moving image captured by the two-dimensional camera 4 is input to the image correction device 10. In the present embodiment, “moving image” refers to capturing still images, displaying captured still images one after another at high speed, and recognizing them as moving images. Therefore, moving image data is a collection of a plurality of still image data. In the present embodiment, each of the plurality of still image data included in the moving image data is referred to as two-dimensional image data, and the image represented by each of the two-dimensional image data is referred to as “two-dimensional image”. A plurality of two-dimensional image data that forms a moving image and is input to the image correction apparatus 10 is stored in the two-dimensional image storage unit 26.

Two-dimensional image data representing a two-dimensional image is input from the two-dimensional image storage unit 26 to the horizontal straight line detection unit 28. The horizontal straight line detection unit 28 detects a horizontal straight line serving as an index for correcting the deviation of the line image 60 from each of the plurality of two-dimensional images represented by each of the input two-dimensional image data. In the present embodiment, the “horizontal straight line” is a straight line extending in a horizontal direction C (see FIG. 2B, FIG. 3, etc.) orthogonal to the vertical direction B, in other words, parallel to the moving direction A of the vehicle 50. It is a straight line.

It is desirable that the straight line, which is an index for blur correction of the line image 60, is equidistant from the running trajectory of the vehicle 50. When simply detecting a horizontal straight line, it is difficult to determine whether it is a straight line due to the design of the object to be photographed or the like.

Therefore, the horizontal straight line detection unit 28 of this embodiment detects a rectangle from a two-dimensional image.

When photographing outdoors, the building itself to be photographed often has a rectangular shape, and the fact that there are many straight lines forming a rectangle such as a door and a signboard is utilized, and the horizontal straight line detection unit 28 has no vanishing point. That is, only the straight lines estimated to form a rectangle positioned parallel to the moving direction A of the vehicle 50 are selected. As a result, in the image correction apparatus 10 of the present embodiment, it is possible to improve the detection accuracy of the horizontal straight line that serves as an index.

As the process of detecting a rectangle from a two-dimensional image, for example, the process described in Reference 1 may be applied. In the processing, of the detected rectangles, when the inclination of one pair of two sides facing each other is the moving direction A of the vehicle 50 as the horizontal axis, another pair of two sides parallel to and facing the horizontal axis. A rectangle whose inclination is parallel to the vertical direction B (the vertical axis direction of the image) is detected.

[Reference 3] Computer Vision Technical Review and Future Outlook p.151

As described above, in the present embodiment, the rectangle is detected from the two-dimensional image, and at least one side of the pair of sides parallel to the horizontal direction of the rectangle is detected as the horizontal straight line, so that the vehicle 50 can be stably operated. It is possible to select a group of horizontal straight lines existing in parallel to the traveling route of, and it is possible to accurately correct the deviation of the line image 60.

Information indicating the horizontal straight line detected by the horizontal straight line detection unit 28 is output to the horizontal straight line correspondence unit 34.

Image data representing a line-parallel image is input from the line-parallel image storage unit 24 to the inter-camera correspondence unit 30. Image data representing a two-dimensional image is input to the inter-camera correspondence unit 30 from the horizontal straight line detection unit 28.

For each of the two-dimensional images, the inter-camera correspondence unit 30 includes a line image 60 and one line along the vertical direction B passing through the optical axis center of the two-dimensional image, which is obtained from the two-dimensional image. An image (hereinafter, referred to as “vertical line image”) is associated.

The line image 60 taken by the line scan camera 2 is an image in which the horizontal axis is parallel projection. On the other hand, the two-dimensional image captured by the two-dimensional camera 4 is an image by perspective projection. Therefore, the line image 60 and the entire two-dimensional image are different in the way in which the subject is photographed, and it is difficult to associate the same area. However, looking at only the vertical line image passing through the optical axis center in the two-dimensional image, the same image as the line image 60 is captured. Therefore, the inter-camera correspondence unit 30 collates a vertical line image passing through the optical axis center of the two-dimensional image with the image of one line of the line parallel image 62 (line image 60) and associates them. For example, the method described in Reference Document 2 may be used for the matching. For the vertical line image, a correlation value between the vertical line image and the line image to be collated may be derived and the line image having a high correlation may be selected. In order to suppress the difference in resolution between the two-dimensional image and the line image 60, the line parallel image 62 (line image 60) may be reduced or a low-pass filter may be used.

[Reference 2] Image Processing Image Processing Standard Textbook p.253-254

By correlating the line image 60 by the line scan camera 2 and the vertical line image passing through the optical axis center of the two-dimensional image, the position where the two-dimensional image is captured can be known. This is because, as shown in FIG. 4, in the line parallel image 62 (line image 60), the horizontal axis is parallel projection, and the x coordinate, which is the coordinate on the horizontal axis, directly corresponds to the coordinate in the real space. Is. By photographing position of the two-dimensional image is seen, that the two-dimensional image 66 (FIG. 4, 2-dimensional image 66 _1, 66 ₂ reference) base length H between is known, calculates the three-dimensional position of the imaged object in a simple Is possible.

For example, in the example shown in FIG. 4, a vertical line image 67 ₁ in the two-dimensional image 66 _1, and a vertical line image 67 ₂ in the two-dimensional image 66 _2, associated to the line parallel image 62. Then, the absolute value of the difference between the x-coordinates of the vertical line image 67 ₁ and the vertical line image 67 ₂ becomes the base line length H.

Information representing the baseline length H derived by the inter-camera correspondence unit 30 is stored in the two-dimensional inter-image baseline length storage unit 32.

Information indicating the baseline length H is input from the two-dimensional inter-image baseline length storage unit 32 to the horizontal straight line correspondence unit 34. Further, the horizontal straight line correspondence unit 34 is input with information representing the horizontal straight line detected by the horizontal straight line detection unit 28.

The horizontal straight line association unit 34 associates a plurality of two-dimensional images 66 with each other. Specifically, the horizontal straight line association unit 34 of the present embodiment associates the rectangles extracted in each two-dimensional image 66. The method of Reference 2 above may be used for the matching. In the search range, the y-coordinate, which is the height on the vertical axis of the two-dimensional image 66, can be limited to a region having the same y-coordinate as one rectangle.

The labeling result of the horizontal straight line mapping unit 34 is input to the labeling unit 36. Image data representing the line parallel image 62 is input to the labeling unit 36 from the line parallel image storage unit 24.

Further, as shown in FIG. 1, the labeling unit 36 of the present embodiment includes a horizontal straight line position calculation unit 37 that calculates the position of the horizontal straight line in the line parallel image 62.

If the rectangles can be associated, the three-dimensional position of the horizontal line on the rectangle in the real space can be calculated by performing stereoscopic vision (see Reference 3). The labeling unit 36 determines the coordinates of the line image 60 included in the line parallel image 62 in the x-axis direction when the moving direction of the vehicle 50 is the x-axis, and the three-dimensional position of the rectangular horizontal straight line in the real space. On the basis of and, the horizontal straight line is mapped to the line image 60 from the three-dimensional position in the real space of the horizontal straight line in the vertical line image associated with the line image 60, and each line image in the line parallel image 62 is mapped. A region including 60 mapping positions is labeled as a region where a horizontal straight line should be detected.

[Reference 3] Image Processing Image Processing Standard Textbook p.271-273

When the vertical direction to the ground is the y-axis, the y-axis direction of the line parallel image 62 is perspective projection, and therefore the focal length of the line scan camera 2 is f and the height in the real space is set according to the distance Z from the travel route. When the height is Y, the y coordinate is represented by a value obtained by dividing the value obtained by multiplying the focal length f and the height Y by the distance Z. Specifically, the horizontal straight line position calculation unit 37 uses the height Y and the distance Z of the three-dimensional position of the horizontal straight line in the vertical line image associated with the line image 60 in the real space as follows ( The y coordinate of the mapping position of the line image 60 is calculated by the equation 1).
y=(f×Y)/Z (1)

The line parallel image 62 to which the label is added from the labeling unit 36 is input to the image correction unit 38. Specifically, the image correction unit 38 is input with image data representing the line parallel image 62 and information representing the labeled area in the line parallel image 62.

The image correction unit 38 detects the contour of the imaging target in the labeled area of the line parallel image 62 by using the process shown in Reference 4, and the line is adjusted so that the contour line segment becomes a straight line. The image 60 is shifted up and down for correction. Specifically, as shown in FIG. 5, with respect to the area 70 to which the label of the line parallel image 62 is given, adjacent pixels are shifted by several pixels as indicated by an arrow S while the line image 60 is shifted in the vertical direction B (up and down). It is also possible to calculate the image correlation value with the gap as in Reference 2 above and correct the shift amount of the line image 60 so that the correlation becomes high.

[Reference 4] OpenCV Programming Book, Mainichi Communications, p.146-155

The image correction unit 38 outputs image data representing the line parallel image 62 in which the shift of the line image 60 is corrected in this way.

<Operation of the image correction device 10 of the present embodiment>
Next, the operation of the image correction device 10 of the present embodiment will be described with reference to the drawings. FIG. 6 is a flowchart showing an example of an image correction processing routine executed in the image correction apparatus 10 of this embodiment.

In the image correction processing routine shown in FIG. 6, for example, the image data representing the line image 60 captured by the line scan camera 2 is stored in the line image storage unit 20 and the two-dimensional image captured by the two-dimensional camera 4 is used. The image data representing 66 is executed at an arbitrary timing such as the timing when the image data representing 66 is stored in the two-dimensional image storage unit 26 or the timing when the execution instruction of the image correction processing routine is received from the outside of the image correction apparatus 10. In the image correction apparatus 10 of the present embodiment, the image data representing the line image 60 is stored in the line image storage unit 20, and the image data representing the two-dimensional image 66 is stored in the two-dimensional image before the image correction processing routine is executed. The state is stored in the unit 26.

In step S100 of FIG. 6, the image generation unit 22 arranges the image data representing the line image 60 stored in the line image storage unit 20 in chronological order to generate the line parallel image 62, as described above.

In the next step S102, the horizontal straight line detection unit 28 has no vanishing point from each of the plurality of two-dimensional images represented by each of the input two-dimensional image data, that is, the movement of the vehicle 50, as described above. A straight line estimated to form a rectangle positioned parallel to the direction A is detected as a horizontal straight line serving as an index for deviation correction.

In the next step S104, the inter-camera associating unit 30 associates the line images 60 with the vertical line image 67 passing through the optical axis center of the two-dimensional image 66 as described above, thereby associating the cameras. I do. Further, the inter-camera correspondence unit 30 derives the base line length H from each vertical line image 67 of the two two-dimensional images 66.

In the next step S106, the horizontal straight line association unit 34 associates a plurality of two-dimensional images 66 with each other by associating the rectangles extracted in each two-dimensional image 66 as described above.

In the next step S108, the labeling unit 36 calculates the three-dimensional position in the real space of the extracted horizontal straight line on the rectangle by the horizontal straight line position calculation unit 37 as described above. Further, the labeling unit 36 maps a horizontal straight line on the line parallel image 62 based on the coordinate in the x-axis direction, which is the moving direction of the vehicle 50, and labels the area where the horizontal straight line should be detected.

In the next step S110, the image correction unit 38 shifts the line image 60 up and down so that the contour line segment of the photographing target becomes a straight line in the labeled area of the line parallel image 62, as described above. Make a correction. When the image data representing the line parallel image 62 in which the deviation of the line image 60 is corrected by the image correction unit 38 is output, the main image correction processing routine ends.

As described above, the image correction device 10 according to the present embodiment includes the image generation unit 22, the horizontal straight line detection unit 28, the inter-camera correspondence unit 30, the horizontal straight line correspondence unit 34, the labeling unit 36, and the image correction unit 38. Equipped with. The image generation unit 22 arranges the line images 60, which are taken by the line scan camera 2 in which the image pickup elements are arranged in one row and extend in the vertical direction B with respect to the moving direction A, in a time series order to form a two-dimensional line parallel image. 62 is generated. The horizontal straight line detection unit 28 is obtained by the two-dimensional camera 4 in which the image pickup elements are arranged in a matrix, when the line scan camera 2 captures the line image 60, in the same direction as the line scan camera 2. A horizontal straight line extending in the horizontal direction C included in each of the plurality of two-dimensional images 66 is detected. The inter-camera associating unit 30 associates the line image 60 with the vertical line image 67 passing through the optical axis center of the two-dimensional image 66. The horizontal straight line association unit 34 associates the horizontal straight lines among the plurality of two-dimensional images 66. The labeling unit 36 maps a horizontal straight line to each of the line images 60 based on the correspondence between the line image 60 and the vertical line image 67, and assigns a label to a region including the horizontal straight line in the line parallel image 62. To do. The image correction unit 38 corrects the position of the line image 60 in the vertical direction B in the line parallel image 62 based on the labeled area.

As described above, in the image correction apparatus 10 of the present embodiment, the information of the two-dimensional image 66 captured by the two-dimensional camera 4 and the information of the line image 60 captured by the line scan camera 2 are integrated to perform the blur correction. The horizontal straight line that serves as an index is accurately detected, and the blurring of the line parallel image 62 is corrected.

Since the two-dimensional image 66 taken by the two-dimensional camera 4 is an image by perspective projection, when the moving direction A of the vehicle 50 is the x-axis, a horizontal straight line is detected from a region other than the region equidistant from the x-axis. There is a possibility that it will end up. An error will occur if the deviation is corrected with reference to straight lines having different distances from the x-axis, that is, different depths. On the other hand, in the image correction apparatus 10 according to the present embodiment, the inter-camera correspondence unit 30 detects a horizontal straight line existing on the equidistant surface, so that the difference in the appearance (the appearance of the imaging target) can be obtained. The two-dimensional image 66 and the line parallel image 62 (line image 60) can be collated with high accuracy while suppressing.

The inter-camera associating unit 30 detects a rectangle and detects only those in which the inclinations of two straight lines forming the rectangle are the same in the horizontal direction C and the vertical direction B, respectively, and are parallel to the horizontal direction C. One of the sides of the set is used as an index for correcting the deviation. In the image correction apparatus 10 of the present embodiment, by detecting the group of straight lines in the horizontal direction C having no vanishing point in this way, it is possible to accurately shift the horizontal straight lines existing at the same distance from the traveling route of the vehicle 50. It becomes possible to use it as an index for correction.

Further, according to the image correction apparatus 10 of the present embodiment, the position of the vertical line image 67 passing through the optical axis center of the two-dimensional image 66 in the real space can be known. Therefore, the horizontal straight line position calculation unit 37 can calculate the position of the horizontal straight line in the real space only by associating the horizontal straight lines between the two-dimensional images 66. The image correction apparatus 10 can detect the horizontal straight line that is an index for correcting the deviation by re-projecting the horizontal straight line to which the position in the real space is added to the line parallel image 62.

Therefore, according to the image correction apparatus 10 of the present embodiment, it is possible to appropriately correct the deviation of the line image 60 in the line parallel image 62 in which the line images 60 taken by the line scan camera 2 are arranged in time series. ..

Further, according to the image correction apparatus 10 of the present embodiment, the line parallel image 62 is corrected over a long distance without previously capturing an object serving as an index at the same time or designating a region where blurring exists by a user or the like. Can be automatically performed in a short time.

In addition, in the present embodiment, the form in which the program for executing the image correction process is pre-installed has been described, but the program may be stored in a computer-readable recording medium and provided. It is also possible to provide it via a network.

2 line scan camera 4 2D camera 10 image correction device 22 image generation unit 28 horizontal straight line detection unit 30 inter-camera correspondence unit 34 horizontal straight line correspondence unit 36 label unit 37 horizontal straight line position calculation unit 38 image correction unit

Claims

An image generation unit that arranges line images, which are taken by a line scan camera in which image pickup elements are arranged in one row and extend in a direction perpendicular to a moving direction, in time series to generate a two-dimensional line parallel image,
The two-dimensional cameras in which the image pickup devices are arranged in a matrix form a plurality of two-dimensional images obtained by photographing the line image in the same direction as the line scan camera when the line scan camera captures the line image. A horizontal straight line detection unit included, which detects a horizontal straight line extending in the horizontal direction,
An inter-camera associating unit that associates the line image with a vertical line image passing through the optical axis center of the two-dimensional image;
A horizontal straight line correspondence attaching unit that associates the horizontal straight lines between a plurality of the two-dimensional images,
A labeling unit that maps the horizontal straight line to each of the line images based on the correspondence between the line image and the vertical line image, and gives a label to a region including the horizontal straight line in the line parallel image. When,
An image correction unit that corrects the position of the line image in the line parallel image in the vertical direction based on the labeled area;
Image correction device equipped with.
The line scan camera and the two-dimensional camera are provided in a vehicle,
The line image, from a rotary encoder installed on the tire of the vehicle, based on a signal output each time the vehicle moves a predetermined distance, is an image taken by the line scan camera,
The image generation unit arranges the line images in chronological order to generate the line parallel image in which the horizontal axis is parallel projection.
The image correction device according to claim 1.
The horizontal straight line detection unit is a rectangle formed by a set of sides parallel to the vertical direction and a set of sides parallel to the horizontal direction, and at least one side of the set of sides parallel to the horizontal direction. Is the horizontal straight line,
The image correction device according to claim 1.
The labeling unit derives a positional relationship between the two-dimensional images on the basis of the correspondence between the line images and the line parallel images, and realizes the realization of the horizontal straight lines associated between the two-dimensional images. A horizontal straight line position calculation unit for calculating coordinates in space,
The image correction device according to any one of claims 1 to 3.
A step of arranging, by the image generation unit, line images, which are taken by a line scan camera in which the image pickup elements are arranged in one row and extending in a direction perpendicular to the moving direction, in time series to generate a two-dimensional line parallel image; ,
The two-dimensional cameras in which the image pickup elements are arranged in a matrix by the horizontal straight line detection unit are obtained by photographing in the same direction as the line scan camera when the line scan camera captures the line image. Detecting a horizontal straight line included in each of the two-dimensional images and extending in the horizontal direction;
A step of associating the line image with a vertical line image passing through the optical axis center of the two-dimensional image by the inter-camera associating unit;
A step of associating the horizontal straight lines between the plurality of two-dimensional images by the horizontal straight line correspondence attaching unit;
By the labeling unit, based on the correspondence between the line image and the vertical line image, the horizontal straight line is mapped to each of the line images, and in the line parallel image, a label is attached to a region including the horizontal straight line. The steps to grant,
A step of correcting the position of the line image in the line parallel image in the vertical direction based on the area to which the label is attached by the image correction unit;
Image correction method including.
A program for causing a computer to function as each unit of the image correction apparatus according to any one of claims 1 to 4.