JP5201356B2 - On-vehicle camera device calibration support method, calibration support device, and on-vehicle camera device - Google Patents

Description

  The present invention relates to a calibration support method, a calibration support apparatus, and an in-vehicle camera apparatus including the calibration support method that enable a user to easily perform calibration of an in-vehicle camera apparatus used by being installed in a car.

  Conventionally, an in-vehicle camera is known that is installed in the rear part of a vehicle to take a picture of the rear periphery of the vehicle, and displays the captured image on a display device installed in the vehicle in order to support the backward movement of the vehicle during parking. It has been.

  Conventionally, as a method for calibrating such an in-vehicle camera, an index such as a target bar or a reference marker installed on the ground is imaged by the in-vehicle camera and displayed on a display device. A method has been proposed in which camera orientation parameters and installation information are calculated by overlaying markers, etc., and image processing is performed based on that to adjust the captured image to absorb installation errors of in-vehicle cameras in software. (For example, Patent Document 1 and Patent Document 2).

  The calibration method in the prior art that captures the target bar or reference marker installed on the ground with the in-vehicle camera, performs image processing, and softly absorbs the installation error of the in-vehicle camera is the target bar or reference In addition to requiring an index such as a marker, there is a problem that the circuit scale and processing increase, resulting in high costs.

  In order to reduce the cost, the apparatus is not provided with a calibration function that softly absorbs the installation error of the in-vehicle camera, and the user may perform calibration manually. Considering the use of in-vehicle cameras, the installation accuracy does not need to be extremely strict, and even if a user performs calibration manually, there is usually no problem. The problem with performing calibration manually is that it is troublesome and time consuming because of trial and error.

  An object of the present invention is to provide a calibration support method, a calibration support apparatus, and a low-cost in-vehicle camera apparatus that can easily calibrate an in-vehicle camera without being troublesome for a user.

  The present invention obtains image data of a road surface including a parking frame and a similar sign taken with an in-vehicle camera, generates inverted image data obtained by horizontally inverting the image data, and generates image data and inverted image data. It is important to calculate an evaluation value for evaluating the mounting error in the horizontal direction of the in-vehicle camera based on the parking frame in the car and a similar sign, and to notify the user of the calibration operation based on the evaluation value. Features.

  Further, the present invention is characterized in that image data of a road surface including a parking frame and a similar sign is obtained by cutting out a lower half of an image photographed by an in-vehicle camera.

  In the present invention, the image data is binarized and converted into image data A. The image data A is inverted horizontally to generate inverted image data, which is converted into image data B. The image data A and the image data The evaluation value is calculated by paying attention to the white area in the data B.

  Further, the present invention generates image data C obtained by superimposing image data A and image data B, and uses the image data C to calculate a white area overlapping ratio and a white area positional relationship. It is a feature.

  In addition, the present invention is characterized in that a calibration operation is notified to a user using a display device and / or an audio output device.

  The present invention also stores message information for notifying the user of the calibration operation in a plurality of patterns in advance in the storage device, and reads out the message information of a desired pattern from the storage device based on the evaluation value and notifies it. It is characterized by doing.

  According to the present invention, a vehicle-mounted camera can be calibrated easily and with a desired accuracy without bothering the user. Therefore, the in-vehicle camera device does not need to have a function of absorbing the installation error in software, and can achieve cost reduction.

1 is an overall configuration diagram of an embodiment of an in-vehicle camera device of the present invention. It is a flowchart of one Example of the calibration assistance process part in FIG. It is a figure explaining the application conditions of this invention. It is a figure which shows the 1st example of the image data A and the image data B which reversed this horizontally. It is a figure which shows the 2nd example of the image data A and the image data B which reversed this horizontally. It is a figure which shows the 3rd example of the image data A and the image data B which reversed this horizontally. FIG. 5 is a diagram showing image data C obtained by superimposing image data A and image data B in FIG. 4. FIG. 6 is a diagram illustrating image data C obtained by superimposing image data A and image data B in FIG. 5. It is a figure which shows the image data C which overlap | superposed the image data A and the image data B of FIG. It is a figure which shows an example of a calibration operation message. It is the figure which added the positional relationship of the evaluation value to FIG. It is the figure which added the positional relationship of the evaluation value to FIG.

  FIG. 1 is an overall configuration diagram of an embodiment of an in-vehicle camera device to which a calibration support function of the present invention is applied. In FIG. 1, the imaging device 10, the processing device 20, and the storage device 30 are generally configured in one housing and are installed at the rear of the vehicle. The display device 50 is installed in the front portion of the vehicle, the audio output device 60 is installed in the rear of the vehicle, and the like, and is connected to the processing device 20 by a cable or the like. The processing device 20 and the operation unit 40 are connected by a cable or wirelessly. The operation unit 40 may also serve as a remote controller for the navigation device. The display device 50 may also serve as a display unit of the navigation device. The audio output device may also serve as a speaker of the audio device.

  The imaging device 10 captures the rear of a car and acquires captured image data. Specifically, the imaging device 10 includes a wide-angle lens optical system, an imaging device, an auto gain control (AGC) circuit, an analog / digital converter, and the like. The imaging device is composed of a CCD sensor, a CMOS sensor, or the like, and converts an optical image collected by the optical system into an electric signal (analog signal) having strength according to the light intensity. The analog signal output from the image sensor is gain-adjusted by an AGC circuit and then converted to a digital signal (captured image data) by an analog / digital converter. In general, an image sensor is provided with a color filter such as a Bayer array, and data output from the imaging device 10 is captured image data such as a Bayer array. Note that the imaging apparatus 10 may be configured to output monochrome captured image data.

  The processing device 20 includes an FPGA, an ASIC, a DSP, and the like, and includes an image processing unit 210, a calibration support processing unit 220, and a control unit 230 that controls operations of these processing units. In addition, the processing device 20 includes a RAM as a working memory used in the processing units 210 and 220 and the control unit 230, but is omitted in FIG.

  The image processing unit 210 performs Bayer interpolation processing, magnification chromatic aberration correction processing, color space conversion processing (RGB / YCC conversion), MTF correction processing, distortion aberration correction processing, and gamma correction processing on the captured image data output from the imaging device 10. The predetermined image processing such as is performed. The captured image data that has been subjected to predetermined processing by the image processing unit 210 is converted into an NTSC video signal and then displayed on the display device 50. The NTSC encoder may be provided with either the processing device 20 or the display device 50. Since the configuration of the image processing unit 210 is basically the same as the conventional one, further detailed description is omitted.

  The calibration support processing unit 220 receives the captured image data output from the imaging device 10 during the calibration operation, generates image data obtained by horizontally inverting the captured image data, and captures the captured image data, the inverted image data, and the like. Are compared to obtain an evaluation value for evaluating the camera mounting state (left-right inclination), and the calibration operation is notified to the user based on the evaluation value. Specifically, the evaluation value is obtained by comparing the positional relationship of the parking frame on the road in the reverse image data with the captured image data. Notification of the calibration operation to the user is performed by the display device 50 and / or the audio output device 60. The calibration support processing unit 220 is a main feature of the present invention, and specific processing will be described later.

  In FIG. 1, the captured image data output from the imaging device 10 is directly input to the calibration support processing unit 220. However, the captured image data in the middle of the image processing unit 210 (for example, captured image data after Bayer interpolation processing). Or the like, or captured image data after image processing output from the image processing unit 210 may be input. In short, it is only necessary to recognize a white line or the like as a reference such as a parking frame on the road used for calculating the evaluation value.

  The storage device 30 is a rewritable nonvolatile memory such as a PROM, and stores programs, various parameters, and other information necessary for processing and control in the processing units 210 and 220 and the control unit 230 of the processing device 20. . As will be described later, the storage device 30 stores in advance message information of a calibration operation that the calibration support processing unit 220 notifies the user of based on the evaluation value.

  The operation unit 40 is a user interface that allows the user to instruct the control unit 230 to perform various operations. The control unit 230 activates the calibration support processing unit 220 in response to an instruction from the operation unit 40. As described above, when the vehicle is equipped with a navigation device, the remote controller can also serve as the operation unit 40.

  FIG. 2 shows a process flowchart of an embodiment of the calibration support processing unit (calibration support apparatus) 220 which is the main configuration of the present invention. Hereinafter, the processing of the calibration support processing unit 220 will be described in detail with reference to FIG.

  Here, as shown in FIG. 3, it is assumed that the in-vehicle camera is mounted on the center axis of the car, and the car is parked so that the center axis of the car and the center of the parking frame on the road substantially coincide. Of course, the parking frame need not be a vehicle width line. The calibration support target is the adjustment of the left-right direction of the in-vehicle camera, and the adjustment of the depression angle is excluded. In general, the depression angle of the in-vehicle camera is related to the height of attachment, and can be arbitrarily determined unless it is extreme.

  When the calibration support processing unit 220 is activated from the control unit 230, the calibration support processing unit 220 inputs captured image data output from the imaging device 10 (step 1001). As described above, instead of directly inputting the captured image data output from the imaging device 10, image data in the middle of the image processing unit 210 or image data output from the image processing unit 210 may be input. .

  Next, the calibration support processing unit 220 cuts out the lower half of the input captured image data. That is, the road surface image portion including the parking frame is cut out (step 1002). Then, the cut-out road surface image portion is binarized to obtain image data A (step 1003). In the image data A, the parking frame area is basically expressed in white, and the other areas are expressed in black. Note that, depending on the height and depression angle at which the vehicle-mounted camera is attached, step 1002 may be omitted, and the input captured image data may be directly binarized and used as image data A.

  Next, the calibration support processing unit 220 generates inverted image data obtained by horizontally inverting the image data A, which is a binarized image of the captured image of the road surface including the parking frame, and sets the image data B as image data B (step 1004). . Since this process can be realized by simple coordinate conversion, detailed description thereof is omitted.

  FIGS. 4 to 6 are diagrams schematically showing an example of the image data A and the image data B obtained by inverting the left and right. Here, FIG. 4 is an example when the horizontal direction of the in-vehicle camera is almost normal. FIG. 5 shows an example when the in-vehicle camera is facing left, and FIG. 6 is an example when the camera is facing right.

  The calibration support processing unit 220 generates image data C by superimposing the image data A and the image data B obtained by reversing the image data A (step 1005). Specifically, for example, “2” is assigned to the white area (parking frame area) of image data A, and “1” is assigned to the white area of image data B, and each pixel of image data A and image data B is assigned. Is added to generate image data C. The black areas of the image data A and the image data B are “0”. Therefore, in the image data C, all the pixels are configured with values of 0 to 3.

  Conversely, “2” may be assigned to the white area of the image data B, and “1” may be assigned to the white area of the image data A. Further, the value to be assigned may not be “2” or “1”. That is, it is sufficient that the white areas of the image data A and the image data B can be distinguished, and any value can be used. Similarly, the black area may not be “0”.

  FIG. 7 schematically shows the image data C when the image data A and the image data B in FIGS. 4A and 4B are overlapped. Here, a and a 'correspond to the white area of the image data A, and b and b' correspond to the white area of the image data B. The pixel value of each pixel in the overlapping portion of the white area of the image data A and the image data B is “3”, and the pixel value of each pixel corresponding to the white area of the image data A is “2”. The pixel value of each pixel corresponding to the white area of the image data B is “1”.

  FIG. 8 schematically shows the image data C when the image data A and the image data B in FIGS. 5A and 5B are superimposed. Again, a and a 'correspond to the white area of the image data A, and b and b' correspond to the white area of the image data B. In FIG. 8, there is no overlapping portion of the white areas of the image data A and the image data B, the pixel value of each pixel of the white areas a and a ′ of the image data A is “2”, and the white areas b and b of the image data B The pixel value of each pixel 'is “1”.

  FIG. 9 schematically shows the image data C when the image data A and the image data B in FIGS. 6A and 6B are superimposed. Again, a and a 'correspond to the white area of the image data A, and b and b' correspond to the white area of the pixel data B. The difference from FIG. 8 is basically that the positional relationship between a and a 'and b and b' is reversed. That is, by superimposing the image data A and the image data B and comparing the positional relationship between the white areas (parking frame areas) of the two, the in-vehicle camera is facing in the left or right direction, and to what extent it is facing. Can be discriminated.

  The calibration support processing unit 220 uses the image data C obtained by superimposing the image data A and the image data B, and uses the mounting error of the in-vehicle camera (whether the in-vehicle camera is facing in the left or right direction, how much is facing, etc. ) Is calculated with the evaluation value to be evaluated (step 1006), and the calibration pattern is determined based on the evaluation value (step 1007). Here, there are three types of calibration patterns shown in FIG. FIG. 10 is merely an example, and it is possible to further subdivide the degree of rotation. Hereinafter, an embodiment of steps 1006 and 1007 will be described.

The calibration support processing unit 220 first uses the image data C as the evaluation value A,
A = total number of pixels with pixel value 3 / (total number of pixels with pixel value 2 + total number of pixels with pixel value 3)
Calculate When the evaluation value A is equal to or greater than a predetermined threshold (percent), the pattern 1 is determined. That is, the direction of the in-vehicle camera is normal (attachment complete). FIG. 7 shows this case.

If the pattern is not 1, the calibration support processing unit 220 calculates the average value of the X coordinate of the pixel having the pixel value 2 as C = the average value of the X coordinate of the pixel having the pixel value 1 as the evaluation values B and C. Specifically, for the image pattern C, the average coordinates of all the pixels having a pixel value of 2 are obtained (center of gravity), and the X coordinate value is set as the evaluation value B. Similarly, for the image pattern C, the average coordinate of all the pixels having a pixel value of 1 is obtained (center of gravity), and the X coordinate value is set as the evaluation value C. The origin of the coordinate system may be, for example, the upper left corner or the center of the screen.

  Then, the magnitude relationship between the evaluation value B and the evaluation value C is compared. If B> C, the pattern 2 is determined. That is, it is determined that the in-vehicle camera is facing the left direction. FIG. 8 shows this case. FIG. 11 shows the positional relationship between the evaluation values B and C in this case.

  Conversely, if C> B, the pattern 3 is determined. That is, it is determined that the in-vehicle camera is facing the right direction. FIG. 9 shows this case. FIG. 12 shows the positional relationship between the evaluation values B and C in this case.

  FIGS. 8 and 9 show a case where the inclination is large and the white area of the image data A and the horizontally inverted image data B do not overlap at all. Even in the case of overlapping, the evaluation values B and C are obtained in the same manner, and the direction of inclination can be determined by comparing the magnitude relationship. Further, if the difference between the evaluation values B and C is obtained, it is possible to determine how much it is inclined.

  The evaluation values A, B, and C may be calculated by paying attention to the left and right halves of the screen. When the left-right inclination of the in-vehicle camera is small, the same effect can be obtained even with this.

  The calibration support processing unit 220 notifies the user of the calibration operation based on the determined pattern (step 1008). Notification is performed using the display device 50 and the audio output device 60. Specifically, for example, a table as shown in FIG. 10 is prepared in the storage device 30 in advance, and a corresponding message is read based on the patterns 1, 2, and 3, and displayed on the display device 50 for notification. Alternatively, the voice output device 60 notifies the user by voice. As described above, FIG. 10 is merely an example, and can be further divided. For example, pattern 2 is subdivided into (1) the camera is facing the left greatly, rotate it to the right, (2) the camera is facing a little to the left, rotate it slightly to the right, etc. be able to. This is easily possible by obtaining the difference between the evaluation values B and C.

  Note that the calibration support processing unit 220 may send the determined pattern to the control unit 230, and the control unit 230 may read out the corresponding message from the storage device 30 and output it to the display device 50 or the audio device 60.

  As mentioned above, although one embodiment of the present invention has been described, of course, the present invention is not limited to the configuration of FIG. 1 and the processing of FIG. It can be changed.

DESCRIPTION OF SYMBOLS 10 Imaging device 20 Processing apparatus 30 Storage apparatus 40 Operation part 50 Display apparatus 60 Audio | voice output apparatus 210 Image processing part 220 Calibration support processing part 230 Control part

JP 2001-245326 A JP 2004-200819 A

Claims (13)

A calibration support method for supporting calibration of a vehicle-mounted camera by a user,
Obtain image data of the road surface, including parking frames and similar signs, taken with the in-vehicle camera,
From the image data, generate inverted image data obtained by horizontally inverting the image data,
Based on the parking frame in the image data and the reverse image data and a similar sign to it, an evaluation value for evaluating the mounting error in the horizontal direction of the in-vehicle camera is calculated,
Based on the evaluation value, a calibration operation is notified to the user.
A calibration support method characterized by the above.   The calibration support method according to claim 1, wherein image data of a road surface including a parking frame and a similar sign is obtained by cutting out a lower half of an image photographed by an in-vehicle camera. The image data is binarized and used as image data A,
The image data A is reversed left and right to generate inverted image data, which is set as image data B,
The evaluation value is calculated by paying attention to the white area in the image data A and the image data B.
The calibration support method according to claim 1 or 2, characterized in that Generating image data C obtained by superimposing the image data A and the image data B;
Using the image data C, the white area overlap ratio and the white area positional relationship are calculated.
The calibration support method according to claim 3. Notifying the user of the calibration operation using the display device and / or the audio output device,
The calibration support method according to any one of claims 1 to 4, wherein: Message information for notifying the user of the calibration operation is stored in a plurality of patterns in advance in the storage unit, and based on the evaluation value, the message information of a desired pattern is read from the storage unit and notified.
The calibration support method according to any one of claims 1 to 5, wherein: A calibration support device for supporting calibration of a vehicle-mounted camera by a user,
Means for acquiring image data of a road surface including a parking frame or a similar sign taken by an in-vehicle camera;
Means for generating, from the image data, inverted image data obtained by horizontally inverting the image data;
Means for calculating an evaluation value for evaluating a mounting error in a horizontal direction of the in-vehicle camera, based on a parking frame in the image data and the reverse image data or a sign similar thereto;
Means for notifying the user of the calibration operation based on the evaluation value;
A calibration support apparatus characterized by comprising:   The said image data acquisition means acquires the image data of the road surface containing a parking frame and its similar mark by cutting out the lower half of the image image | photographed with the vehicle-mounted camera. Calibration support device. The means for acquiring the image data binarizes the image data and sets it as image data A,
The means for generating the inverted image data generates the inverted image data by horizontally inverting the image data A, which is set as the image data B,
The means for calculating the evaluation value calculates the evaluation value by paying attention to white areas in the image data A and the image data B.
The calibration support apparatus according to claim 7 or 8, characterized in that Means for generating image data C obtained by superimposing the image data A and the image data B;
The means for calculating the evaluation value uses the image data C to calculate a white area overlapping ratio and a white area positional relationship.
The calibration support apparatus according to claim 9. The means for notifying the user of the calibration operation is notified using a display device and / or an audio output device.
The calibration support device according to claim 7, wherein the calibration support device is a calibration support device. A plurality of patterns of message information for notifying the user of the calibration operation, storage means for storing in advance,
The means for notifying the user of the calibration operation reads out and notifies the message information of a desired pattern from the storage means based on the evaluation value.
The calibration support apparatus according to any one of claims 7 to 11, wherein An in-vehicle camera device that is attached to a car, photographs the surroundings of the car, processes the captured image, and displays the image on a display device,
An in-vehicle camera device comprising the calibration support device according to any one of claims 7 to 12.

Images