JP2014082622A - On-vehicle camera attachment error detection device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To determine whether an on-vehicle camera attachment position is a normal position to enable the performance of calibration or not in an image processing device for performing calibration to identify an effective area from a captured image by an on-vehicle camera.SOLUTION: An image processing device includes storage means for storing position information indicating a display range within a captured image of a vehicle feature point that appears in a captured image when an on-vehicle camera attachment position is within a permission range to enable calibration. In the image processing device, display range superimposition means (S120-S140) superimposes the display range of a vehicle feature point on a captured image by the on-vehicle camera on the basis of position information stored in the storage means, and image output means (S150) outputs a captured image on which the display range of a feature point is superimposed by the display range superimposition means to an external display device.

Description

  The present invention relates to an image processing apparatus that performs calibration for specifying an effective area suitable for image display or image recognition from images captured by an in-vehicle camera, and a position where the in-vehicle camera can be calibrated It is related with the attachment error detection apparatus of the vehicle-mounted camera suitable for determining whether it is.

  2. Description of the Related Art Conventionally, there has been known an image processing apparatus that performs calibration for specifying an effective region suitable for image display such as a bird's-eye view image, image recognition of a predetermined object, and the like from captured images around a vehicle by an in-vehicle camera. It has been.

  In this type of image processing apparatus, for example, two markers provided on the left and right fenders of the vehicle are detected from the captured image by calibration, and each marker is set to a specific position set in advance. An effective area in the captured image is set (see, for example, Patent Document 1).

  For this reason, an image processing apparatus having such a calibration function can extract an image suitable for image display and image recognition from a captured image even if the mounting position of the in-vehicle camera is deviated from the design position. .

JP 2001-233139 A

  By the way, in the image processing apparatus, the effective area can be specified by the calibration when the marker exists within the allowable range in the captured image. That is, when the marker is not within the allowable range, the effective area protrudes outside the captured image, and thus the effective area cannot be extracted from the captured image.

  Therefore, calibration can be performed without problems if the mounting position of the in-vehicle camera is within a predetermined error range, but if the mounting position of the in-vehicle camera deviates greatly beyond the error range, calibration should be performed normally. I can't.

If the calibration cannot be normally performed as described above, it is necessary to readjust the mounting position of the in-vehicle camera and perform calibration again.
Therefore, in this case, it takes time to complete the position adjustment and calibration of the in-vehicle camera, and there is a problem that the productivity of the vehicle is lowered.

  In addition to the mounting error of the in-vehicle camera, for example, abnormalities in the transmission system of the captured image from the in-vehicle camera to the image processing device can be considered as a cause that the calibration cannot be performed normally.

  For this reason, if calibration cannot be performed normally, the effective area may not be extracted from the captured image even if the mounting position of the in-vehicle camera is readjusted and recalibrated. There is also a problem that it takes time to specify.

  On the other hand, in order to prevent this problem, it is conceivable to widen the position range (that is, the above-described allowable range) of the marker that can specify the effective area from the captured image by widening the imaging range by the in-vehicle camera.

However, such a countermeasure has a problem that the time required for image processing for calibration becomes long and the productivity of the vehicle is lowered.
The present invention has been made in view of these problems, and in an image processing apparatus that executes calibration for specifying an effective area from images captured by a vehicle-mounted camera, the mounting position of the vehicle-mounted camera can be calibrated. It is an object of the present invention to make it possible to determine whether or not it is a normal position.

  In the in-vehicle camera mounting error detecting device of the present invention made to achieve the above object, the in-vehicle camera and an image captured by the in-vehicle camera when the mounting position of the in-vehicle camera is within an allowable range where calibration can be performed. Storage means for storing position information representing the display range of the feature points of the vehicle reflected in the captured image.

  In the first invention of the present application (Claim 1), the display range superimposing unit superimposes the display range of the feature point of the vehicle on the image captured by the in-vehicle camera based on the position information stored in the storage unit, and the image output unit However, the captured image on which the display range of the feature points is superimposed is output to an external display device.

  For this reason, according to the mounting error detection device for the in-vehicle camera of the first invention of this application, the user displays the output image from the image output means on the display device and confirms the display screen, thereby It can be very easily determined whether or not the attachment position is within an allowable range in which calibration can be performed.

  In other words, the user checks the display screen of the display device, and if the feature point of the vehicle is within the display range, the mounting position of the in-vehicle camera is within the allowable range, and if the feature point of the vehicle is not within the display range. It can be determined that the mounting position of the in-vehicle camera is not within the allowable range.

  Therefore, if the mounting error detecting device for the in-vehicle camera according to the first invention of the present application is used, it is determined whether or not the mounting position of the in-vehicle camera is a normal position where the calibration can be performed before the in-vehicle camera is calibrated. Judgment can be made.

  Therefore, according to the first invention of the present application, when the mounting position of the in-vehicle camera is not within the allowable range, it is possible to detect that fact and adjust the mounting position of the in-vehicle camera without performing calibration. it can.

  Therefore, according to the first invention of the present application, when the mounting position of the in-vehicle camera is not within the allowable range, it is possible to shorten the time until the calibration is finally completed. Can be improved.

  In addition, when the calibration of the in-vehicle camera is performed and a normal calibration result cannot be obtained, the mounting position of the in-vehicle camera is within an allowable range using the in-vehicle camera mounting error detection device of the first invention of the present application. It may be determined whether or not.

  In this way, when a normal calibration result cannot be obtained, the cause is due to an installation error of the in-vehicle camera or something else (for example, a captured image transmission system). You can easily identify whether you are coming out.

On the other hand, in the second invention of the present application (Claim 2), the feature point detecting means detects the feature point of the vehicle from the image captured by the in-vehicle camera.
Then, the position determination means determines whether or not the position of the feature point in the captured image detected by the feature point detection means is within the feature point display range obtained based on the position information stored in the storage means. Determine.

When the position determination means determines that the position of the feature point in the captured image does not exist within the display range, the defect notification means notifies the attachment failure of the in-vehicle camera.
That is, in the in-vehicle camera mounting error detection device of the second invention of the present application, whether or not the position of the feature point in the captured image exists within the display range, that is, within the allowable range in which the in-vehicle camera can perform calibration. It is automatically determined whether or not there is, and the fact is notified at the time of abnormality.

  For this reason, according to the second invention of the present application, before carrying out the calibration of the in-vehicle camera, it is determined whether or not the mounting position of the in-vehicle camera is normal. If there is, it is possible to prohibit the calibration and notify the abnormality.

  Therefore, according to the second invention of the present application, the determination as to whether or not the mounting position of the in-vehicle camera is normal and the calibration of the in-vehicle camera can be sequentially performed sequentially for each vehicle. As a result, the productivity of the vehicle can be improved.

It is a block diagram showing the structure of the image processing system of 1st Embodiment. It is explanatory drawing showing the attachment position of the vehicle-mounted camera to a vehicle. It is a flowchart showing a determination image output process. It is explanatory drawing showing the display screen of the display apparatus by determination image output processing. It is a flowchart showing the calibre judgment process of 2nd Embodiment. It is a flowchart showing the calibre judgment process of 3rd Embodiment. FIG. 7 is an explanatory diagram illustrating a calibration operation that is performed in accordance with the calibre determination process of FIG. 6.

Embodiments to which the present invention is applied will be described below with reference to the drawings.
The present invention is not construed as being limited in any way by the following embodiments. An aspect in which a part of the configuration of the following embodiment is omitted as long as the problem can be solved is also an embodiment of the present invention. In addition, an aspect configured by appropriately combining the following embodiments is also an embodiment of the present invention. Moreover, all the aspects which can be considered in the limit which does not deviate from the essence of the invention specified only by the wording described in the claims are embodiments of the present invention. Further, the reference numerals used in the description of the following embodiments are also used in the claims as appropriate, but they are used for the purpose of facilitating the understanding of the invention according to each claim, and the invention according to each claim. It is not intended to limit the technical scope of
[First Embodiment]
As shown in FIG. 1, the image processing system according to the embodiment includes an in-vehicle camera 10, an image processing device 20, and a display device 30.

The in-vehicle camera 10 is a camera having an image sensor such as a CCD or a CMOS, and is installed at the upper rear portion of the vehicle 2 to photograph the rear periphery of the vehicle 2 as shown in FIG.
The in-vehicle camera 10 then outputs the captured image behind the vehicle to the image processing device 20 at a predetermined frequency (for example, 60 frames per second).

In the present embodiment, the in-vehicle camera 10 is attached to the vehicle 2 as a feature point of the vehicle 2 so that the bumper 4 behind the vehicle is reflected in the captured image.
Next, the display device 30 is configured by a liquid crystal display, an organic EL display, or the like, and displays an image processed by the image processing device 20 based on an image captured by the in-vehicle camera 10.

The image processing apparatus 20 includes a captured image storage unit 22, an operation unit 24, an image processing data storage unit 26, and a control unit 28.
The captured image storage unit 22 includes a storage device such as a DRAM, and stores captured images sequentially output from the in-vehicle camera 10 for a predetermined time (for example, the past 10 seconds).

  The operation unit 24 is for a user such as a driver to input various operation instructions to the control unit 28, and is installed on the touch panel provided on the display surface of the display device 30 or around the display device 30. It consists of a mechanical key switch and the like.

  The image processing data storage unit 26 includes a nonvolatile storage device such as a flash memory, and is a device for storing a program executed by the control unit 28 and data necessary for various image processes including the calibration described above. .

  In particular, in the present embodiment, in the image processing data storage unit 26, in order to be able to recognize an attachment error of the in-vehicle camera 10 to the vehicle 2 from an image captured by the in-vehicle camera 10, bumper data, an attachment design value And the maximum mounting error is stored.

Here, the bumper data is shape data necessary to recognize the bumper 4 from the captured image.
Further, the mounting design value is data representing the ideal position (see FIG. 4) of the bumper 4 displayed in the captured image when the vehicle-mounted camera 10 is mounted at the design reference position.

  Further, the maximum mounting error is data representing an allowable error range of the bumper position in which calibration can be normally executed even if the position of the bumper 4 is deviated from the ideal position in the captured image.

  Next, the control unit 28 is configured by a microcomputer including a CPU, a RAM, a ROM, an I / O, and the like, and reads a program from the image processing data storage unit 26 and executes various processes.

Hereinafter, the operation of the image processing apparatus 20 will be described.
In the following description, a process for displaying an image captured by the in-vehicle camera 10 on the display device 30 for driving support of the vehicle 2 (e.g., converting a captured image into a bird's-eye view image to assist garage entry and predicting wheels). The processing for synthesizing the locus and the like and displaying it on the display device) and the processing for calibration for specifying the image area used for the image display are well known and will not be described. A determination image output process, which is a simple process, will be described.

  This determination image output process is a process executed according to a command from the operation unit 24 (or an external device (not shown)) before or after the calibration of the in-vehicle camera 10 is performed.

As shown in FIG. 3, when this process is started, first, a captured image is captured from the in-vehicle camera 10 via the captured image storage unit 22 in S110 (S represents a step).
Next, in S120, the bumper data, the mounting design value, and the maximum mounting error are acquired from the image processing data storage unit 26.

  In S130, based on the bumper data, the mounting design value, and the maximum mounting error acquired in S120, the calibration is normally performed even if the position of the bumper 4 is deviated from the ideal position in the image captured by the in-vehicle camera 10. An allowable display range that can be implemented is calculated.

  As shown in FIG. 4, the permissible display range is such that the position of the bumper 4 shifted by the maximum mounting error around the ideal position of the bumper 4 in the captured image obtained from the mounting design value is the maximum pitch. Calculated as a shift position and a minimum pitch shift position. Then, the upper and lower limit positions of the allowable display range of the bumper 4 are set by superimposing a bumper shape based on the bumper data on each calculated displacement position.

  The maximum pitch shift position corresponds to the maximum shift amount when the in-vehicle camera 10 is tilted outward from the reference position, and the maximum pitch shift position is determined by the in-vehicle camera 10 from the reference position to the vehicle 2. It corresponds to the maximum amount of deviation when tilted to the side.

  Therefore, if the bumper 4 is within the allowable display range between the minimum pitch shift position and the maximum pitch shift position in the image captured by the in-vehicle camera 10, the calibration can be performed normally. Otherwise, the calibration is performed. Can not be implemented normally.

  Next, when the allowable display range of the bumper 4 is calculated in S130, the process proceeds to S140, and a line or a pattern for identifying and displaying the calculated allowable display range is superimposed on an image captured by the in-vehicle camera 10.

In subsequent S150, the captured image in which the allowable display range is identified and displayed by the process of S140 is output to the display device 30, and the determination image output process is terminated.
As described above, according to the image processing apparatus 20 of the present embodiment, if the execution command for the determination image output process is input via the operation unit 24, the control unit 28 converts the captured image by the in-vehicle camera 10 into the captured image. An image in which the allowable display range of the bumper 4 is identified and displayed is output to the display device 30.

  For this reason, the user determines whether or not the mounting position of the in-vehicle camera 10 on the vehicle 2 is within an allowable error range in which calibration can be normally performed by checking the display screen of the display device 30. can do.

  Therefore, the user determines whether the mounting position of the in-vehicle camera 10 is within the allowable error range by causing the control unit 28 to execute the determination image output process before performing the calibration. If it is not within the allowable error range, the calibration can be stopped.

  Therefore, according to this embodiment, when the mounting position of the in-vehicle camera 10 is not within the allowable error range, it is possible to shorten the time until the calibration is finally completed. Productivity can be improved.

In addition, calibration of the in-vehicle camera 10 may be performed, and when a normal calibration result cannot be obtained, the control unit 28 may execute determination image output processing.
In this case, since it is possible to determine whether or not the cause that the calibration cannot be normally performed is due to the mounting error of the in-vehicle camera 10, it is possible to quickly implement a measure for performing the calibration normally. it can.
[Second Embodiment]
Next, an image processing system according to a second embodiment to which the present invention is applied will be described.

  The image processing system of the present embodiment has the same configuration as that of the image processing system of the first embodiment shown in FIGS. 1 and 2, and is different from the first embodiment in the control unit 28 in FIG. 5 is executed in place of the determination image output process shown in FIG.

Therefore, in this embodiment, this calibre determination process will be described.
In this calibre determination process, the controller 28 itself automatically determines whether or not the position of the in-vehicle camera 10 is within an allowable error range in which calibration (hereinafter simply referred to as calibre) can be normally performed. It is a thing.

  As shown in FIG. 5, in the calibre determination process, first, in S200, it is determined whether or not the operation mode of the image processing apparatus 20 is a calibre mode for performing calibration. The calibre determination process ends.

Next, when it is determined in S200 that the operation mode of the image processing apparatus 20 is the calibre mode, the process proceeds to S210, and a captured image is captured from the in-vehicle camera 10.
In subsequent S220, the bumper data, the mounting design value, and the maximum mounting error are acquired from the image processing data storage unit 26. In S230, the bumper 4 in the captured image is acquired based on the acquired data. The allowable display range is calculated.

In addition, the process of S210-S230 is performed similarly to the process of S110-S130 of 1st Embodiment.
Next, in S240, the bumper 4 is detected from the captured image based on the bumper data, and in S250, the detected bumper 4 (or at least a part of the bumper 4) is displayed as the allowable display calculated in S230. Determine if it is within range.

  If it is determined in S250 that the bumper 4 (or at least a part of the bumper 4) is within the allowable display range in the image captured by the in-vehicle camera 10, the process proceeds to S260, and calibration processing is performed.

  In subsequent S270, the calibration result (calibration result) is stored in the image processing data storage unit 26, and the display device 30 displays that the calibration is completed, and the calibration determination process ends.

  On the other hand, if it is determined in S250 that the bumper 4 (or at least a part of the bumper 4) is not within the allowable display range, the process proceeds to S280, and the position of the in-vehicle camera 10 is not within the allowable error range (in other words, Then, the in-vehicle camera 10 is poorly attached) is displayed on the display device 30, and the calibre determination process is terminated.

  As described above, in the present embodiment, the control unit 28 automatically determines whether or not the position of the in-vehicle camera 10 is within an allowable error range in which calibration can be normally performed. If the position is within the allowable error range, calibration is performed.

Therefore, according to the present embodiment, the determination as to whether or not the mounting position of the in-vehicle camera 10 is normal and the calibration of the in-vehicle camera 10 can be sequentially performed sequentially for each vehicle 2. Thus, the productivity of the vehicle 2 can be improved.
[Third Embodiment]
Next, an image processing system according to a third embodiment to which the present invention is applied will be described.

  The image processing system according to the present embodiment is basically configured in the same manner as the image processing system according to the second embodiment, and is different from the second embodiment in that a calibration determination process executed by the control unit 28 is performed. Some are different.

  That is, in this embodiment, as shown in FIG. 6, in the calibre determination process, when it is determined in S250 that the position of the bumper 4 is within the allowable display range, the process of S255 is executed as an estimation unit. Thereafter, the process proceeds to the calibration process of S260.

  In S255, based on the amount of deviation between the position of the bumper 4 detected in the captured image and the ideal position of the bumper 4 in the captured image, the amount of deviation from the reference position of the in-vehicle camera 10 in the vehicle 2 is extended. Estimates the actual mounting position of the in-vehicle camera 10.

  As a result, according to the image processing system of the present embodiment, when executing the calibration process of S260, the target for calibration (for example, the right and left described in Patent Document 1 above) is included in the captured image. When searching for the marker provided on the fender), the search area can be narrowed, and the time required for calibration can be further shortened.

  That is, in the calibration process of S260, when a target used to identify an effective area suitable for displaying a rear image of the vehicle 2 is detected from images picked up by the in-vehicle camera 10, conventionally, FIG. As shown, it was necessary to search for the target within the search range A in consideration of the mounting error of the in-vehicle camera 10.

  However, in this embodiment, since the mounting position of the in-vehicle camera 10 is estimated in S255, a search range for detecting a target in the calibration process of S260 is searched from the search range A based on the mounting position. The range B can be narrowed down.

Therefore, according to the present embodiment, the time required for calibration of the in-vehicle camera 10 can be shortened, and thus the productivity of the vehicle 2 can be increased.
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, a various aspect can be taken.

For example, in the above-described embodiment, the in-vehicle camera 10 is described as being disposed behind the vehicle 2 and used to capture a peripheral image behind the vehicle.
However, the in-vehicle camera 10 may be disposed in front of the vehicle 2 and used to capture a peripheral image in front of the vehicle, or may be disposed on the left and right sides of the vehicle 2 to surround the vehicle side. It may be used to capture an image.

  When the in-vehicle camera 10 is disposed in front of the vehicle 2, the bumper disposed in front of the vehicle 2 or the tip of the hood of the vehicle 2 is detected as a feature point of the vehicle 2. That's fine.

In addition, when the in-vehicle camera 10 is disposed on the side of the vehicle 2, an edge portion obtained by viewing the side door of the vehicle 2 from the information may be detected as a feature point of the vehicle 2.
In the description of the above embodiment, the on-vehicle camera 10 is calibrated using a marker or the like provided on the left and right fenders as a target. However, the target may be arranged around the vehicle. Good.

Further, the calibration is not limited to the case where the vehicle 2 is stopped, and may be performed when the vehicle 2 is traveling.
In each of the above embodiments, the control unit 28 configuring the image processing apparatus 20 has been described as being capable of executing either the determination image output process or the calibre determination process. The determination image output process and the calibre determination process may be selectively performed in accordance with a command from 24 or the like.

  DESCRIPTION OF SYMBOLS 2 ... Vehicle, 4 ... Bumper, 10 ... Car-mounted camera, 20 ... Image processing apparatus, 22 ... Captured image storage part, 24 ... Operation part, 26 ... Image processing data storage part, 28 ... Control part, 30 ... Display apparatus.

Claims (3)

In an image processing apparatus that performs calibration for specifying an effective area from images captured by a vehicle-mounted camera, it is determined whether or not the mounting position of the vehicle-mounted camera is a position where the calibration can be performed. An on-vehicle camera mounting error detection device used,
An in-vehicle camera (10) attached to the vehicle so as to image the surroundings of the vehicle;
When the mounting position of the in-vehicle camera is within an allowable range in which the calibration can be performed, positional information representing a display range in the captured image of the feature point of the vehicle reflected in the captured image by the in-vehicle camera Stored storage means (26);
Display range superimposing means (28, S120 to S140) for superimposing the display range of the feature points on the image captured by the in-vehicle camera based on the position information stored in the storage means;
Image output means (28, S150) for outputting a captured image on which the display range of the feature points is superimposed by the display range superimposing means to an external display device;
An on-vehicle camera mounting error detection device characterized by comprising: It is provided in an image processing apparatus that performs calibration for specifying an effective area from images captured by a vehicle-mounted camera, and determines whether the mounting position of the vehicle-mounted camera is a position where the calibration can be performed. An in-vehicle camera mounting error detection device used for
An in-vehicle camera (10) attached to the vehicle so as to image the surroundings of the vehicle;
When the mounting position of the in-vehicle camera is within an allowable range in which the calibration can be performed, positional information representing a display range in the captured image of the feature point of the vehicle reflected in the captured image by the in-vehicle camera Stored storage means (26);
Feature point detection means for detecting a feature point of the vehicle from images captured by the in-vehicle camera (28, S240);
Determining whether or not the position of the feature point in the captured image detected by the feature point detection means is within the display range of the feature point obtained based on the position information stored in the storage means Position determining means (28, S220, S230, S250) to perform,
If the position determination means determines that the position of the feature point in the captured image does not exist within the display range of the feature point, the attachment failure notification means (28, 28) for notifying the attachment failure of the in-vehicle camera. S280),
An on-vehicle camera mounting error detection device characterized by comprising: The storage means stores position information representing a reference position of the feature point of the vehicle reflected in an image captured by the in-vehicle camera,
When the position determination unit determines that the position of the feature point in the captured image is within the display range of the feature point, the position of the feature point in the captured image is stored in the storage unit. The in-vehicle camera mounting error detection device according to claim 2, further comprising estimation means (28, S255) for estimating the mounting position of the in-vehicle camera based on a deviation from the reference position.

Images