JP2006313498A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus having a function for efficiently testing an image processing function. <P>SOLUTION: The image processing apparatus provided with image processing parts 22 and 24 to be operated by a distance calculation routine for obtaining correlation of a pair of stereoscopic images, deriving a parallax to a corresponding point common to the both images, and calculating a distance to the corresponding point based on the parallax, is provided with a storage means for preliminarily storing a processing result to be acquired from the image processing parts 22 and 24 when applying the distance calculation routine for the known pair of stereoscopic images. When testing the image processing parts 22 and 24, the processing results acquired from the image processing parts 22 and 24 is compared with the processing results stored in the storage means with respect to a test image same as the known pair of stereoscopic images, thereby abnormality of the image processing parts 22 and 24 are determined. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus having a function of inspecting an image processing function.

Conventionally, a monitoring device that calculates and monitors distance data to an object existing in a monitoring range and performs fail-safe when the reliability of the calculated distance data is reduced is known ( For example, see Patent Document 1). The monitoring apparatus sets a region that is a partial region of the monitoring range and has a specific tendency with respect to the distance to the object in the region as a monitoring region, and also uses the vertical distance data in the monitoring region.・ Detects the dispersion value (variation) of the horizontal component. When the detected variance value is small, it is determined that the reliability of the calculated distance data is high, and when it is large, it is determined that the reliability of the calculated distance data is low.
JP 2001-43496 A

  By the way, when starting a system for image processing, it is necessary to start by determining whether each component realizing the system is functioning normally. However, the number of each component increases, and the function As it becomes more complicated, it takes a lot of time to check each function individually. In this regard, also in the above-described conventional technology, since the above-described complicated check method is performed, the same problem may occur.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus having a function of efficiently inspecting an image processing function.

In order to solve the above problems, according to one aspect of the present invention,
A distance calculation routine for obtaining a correlation between a pair of stereo images, deriving a parallax with respect to a corresponding point common to both images, and calculating a distance to the corresponding point based on the parallax;
In an image processing apparatus having an image processing unit that operates according to the distance calculation routine,
Storage means for storing in advance a processing result to be obtained from the image processing unit when the distance calculation routine is applied to a known set of stereo images;
Processing results obtained from the image processing unit and processing stored in the storage means when the distance calculation routine is applied to the same test image as the known set of stereo images during the inspection of the image processing unit An image processing apparatus is provided that performs an abnormality determination of the image processing unit by comparing with a result.

  In this aspect, it is possible to obtain the processing result by the image processing unit in advance by using a known set of stereo images. Therefore, when the image processing unit is inspected, it is possible to perform a through check of the image processing unit by comparing the processing result obtained in advance with the processing result of the same test image as the stereo image. It is possible to reduce the normal check time while ensuring the reliability. In addition, when inspecting the image processing unit, the distance calculation routine that is normally used can be applied as it is to the abnormality determination inspection, so that specialized processing and extra processing for inspection are almost unnecessary, and inspection is performed. Can also reduce the processing load.

  The processing result to be compared includes the total number of parallax points on the parallax image, the total number of parallax points on a line in a predetermined direction on the parallax image, and the total number of parallax points having a predetermined range of distance data on the parallax image. It is good to be.

  According to the present invention, the image processing function can be efficiently inspected.

  The best mode for carrying out the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of an image processing system using the image processing apparatus of the present invention. The image processing system includes a camera ECU (Electric Control Unit) 10 and a stereo ECU 20. Each ECU comprises a plurality of circuit elements such as a central processing unit (CPU), a ROM for storing programs, a RAM for temporarily storing data, an input interface, and an output interface as a unit. Also called a computer. Note that the function of the image processing apparatus of the present invention is realized by components in the stereo ECU 20.

  The camera ECU 10 includes a camera module 17, a camera CPU 13, an image output unit 14, and the like that constitute a stereo camera such as the imaging elements 11 and 12 and the lenses 15 and 16. The image sensors 11 and 12 are constituted by, for example, a CCD (Charge Coupled Device). The camera CPU 13 controls the exposure and the like on the camera side based on a control signal from the stereo ECU 20. The camera CPU 13 transmits images captured by the imaging elements 11 and 12 to the stereo ECU 20 as image signals via the image output unit 14 that is an output interface. Note that the number of image pickup devices is not limited to two, and may be greater than that.

  The stereo ECU 20 includes an image input unit 21 that is an input interface, a geometric conversion LSI 22 that is a geometric conversion processing unit, an image processing LSI 24 that is an image recognition processing unit, and an SV-CPU 23 that supervises each processing unit.

  The image signal output from the image output unit 14 of the camera ECU 10 is sent to the image input unit 21 that is an input interface of the stereo ECU 20. The image output unit 14 and the image input unit 21 are predetermined digital transmission system interfaces.

  Upon receiving the image signal, the image input unit 21 sends image data of the image signal to the geometric transformation LSI 22. The geometric conversion LSI 22 uses the calibration data to generate hardware internal error factors (lens distortion, light, etc.) from the captured image of the camera module 17 used for stereo calculation processing by the imaging elements 11 and 12 and the lenses 15 and 16. A known process for removing the influence of an axial deviation, a focal distance deviation, an image sensor distortion, etc., and aligning the epipolar line with the image horizontal line is performed. The geometric conversion LSI 22 converts an input image based on a geometric conversion LUT (Look Up Table) stored in the memory 25.

  Since the above-mentioned internal error factors occur, the camera module 17 is shipped from the factory after calibration. Therefore, the geometric conversion LSI 22 uses the predetermined calibration calculation formula and the predetermined correction data for the calibration data (that is, calibration data) performed in advance regardless of the camera ECU 10 including any camera module 17 connected thereto. By applying it to the map (that is, based on the geometric transformation LUT), the above-mentioned internal error factors can be corrected and an appropriate geometric transformation process can be performed. The calibration data includes, for example, focal length, image sensor interval, vertical / horizontal image size, corrected focal length, vertical / horizontal optical axis deviation value, and lens distortion amount (or lens distortion coefficient). . Further, a serial number may be included so that the camera module 17 can be specified.

  The image processing LSI 24 performs image recognition processing based on the geometrically converted image data from the geometric conversion LSI 22. The image recognition process is a process performed by an image processing program that detects a three-dimensional object from a captured image using, for example, a stereo viewing technique. The image processing program and the image to be processed are recorded in the memory 26, and the image processing LSI 24 reads them and performs image recognition processing.

  In the image recognition processing, for example, a correlation between a pair of images captured by the imaging elements 11 and 12 arranged on the left and right is obtained, and a distance to the object is calculated in the manner of triangulation based on the parallax with respect to the same object. is there.

  That is, the image processing LSI 24 extracts a portion in which the same imaging object is captured from a set of stereo images captured by the imaging elements 11 and 12, and the same point of the imaging object is detected between the pair of stereo images. Are calculated, and the distance to the imaging target is calculated by obtaining the shift amount (parallax) of the associated point (corresponding point). When the imaging object is in front, when the image by the imaging element 11 and the image by the imaging element 12 are overlapped, the imaging object is shifted in the horizontal direction. Then, the most overlapping position is obtained while shifting one image pixel by pixel. The number of pixels shifted at this time is n. Assuming that the focal length of the lens is f, the distance between the optical axes is m, and the pixel pitch is d, the distance L to the object to be imaged is expressed by a relational expression “L = (f · m) / (n · d)”. To establish. This (n · d) is parallax.

  The SV-CPU 23 is a CPU that supervises each processing unit. The image processing LSI 24 may be used as well. The SV-CPU 23 transmits and instructs a camera control value (for example, shutter speed information that is a parameter for exposure adjustment) calculated based on the processed image of the image processing LSI 24 to the camera CPU 13 in the camera ECU 10.

  By mounting such a camera ECU 10 or stereo ECU 20 on a vehicle, it can be used for control using image recognition information such as obstacles on the road. For example, the SV-CPU 23 may transmit the result to another ECU that requires the image recognition processing result via the in-vehicle LAN. Other ECUs include, for example, an ECU that controls a collision avoidance / collision reduction system, an ECU that controls a lane keeping support system and a lane departure warning system, an inter-vehicle ECU, a brake ECU, and the like.

  The processing operation when the present invention is applied to the above-described image processing system will be described.

  FIG. 2 is an example of a processing sequence performed when the stereo ECU 20 is activated. In step 1, the SV-CPU 23 activated by turning on the power performs initialization of the memory, setting of an interrupt vector, and the like. In step 2, peripheral ICs such as the EEPROM 28 and a CAN driver (LAN connection interface IC) are initialized. In step 3, the reset of the geometric conversion LSI 22 and the image processing LSI 24 is released, and these LSIs are started up. In step 4, the SV-CPU 23 confirms the RUN pulse (the pulse generated after the reset is released and the PLL lockup is completed) from the geometric conversion LSI 22 and the image processing LSI 24. Perform a startup check. If there is an abnormality in the RUN pulse, the abnormality is notified to a predetermined ECU that manages the abnormality via the LAN. In step 5, the initial setting of the register of the geometric transformation LSI 22 and the writing check are performed. In step 6, initial setting of the register of the image processing LSI 24 and writing check are performed. In step 7, the SV-CPU 23 confirms that the ECC / parity error notification of the memories 25, 26 and 27 is normally performed via the geometric conversion LSI 22 and the image processing LSI 24. If it is normal, the SV-CPU 23 releases the reset of the memories 25, 26, and 27. If there is an abnormality, a memory abnormality is notified to a predetermined ECU that manages the abnormality via the LAN. In step 8, a communication check between the camera ECU 10 (camera CPU 13) and the stereo ECU 20 (SV-CPU 23) is performed. If it is normal, the SV-CPU 23 instructs the camera CPU 13 to start image transmission. In the case of an abnormality, the camera communication abnormality is notified to the DSS 1. In step 9, it is confirmed whether or not the geometric conversion LSI 22 can normally receive the image signal via the image output unit 14 in the camera ECU 10 and the image input unit 21 in the stereo ECU 20. If there is an abnormality, a predetermined ECU that manages the abnormality is notified of an abnormality in image communication via the LAN. In step 10, when the image signal from the camera ECU 10 stops, the geometric conversion LSI 22 determines an error, and it is confirmed that the SV-CPU 23 is normally interrupted. If an abnormality is detected, an abnormality of the error determination function is notified to a predetermined ECU that manages the abnormality via the LAN. In step 11, the image processing function is checked by a test pattern (test image) transmitted from the camera ECU 10. Details of step 11 will be described later. After the check by the test pattern is completed, in step 12, the stereo ECU 20 captures calibration data of the camera ECU 10 (camera module 17). As described above, the stereo ECU 20 needs calibration data of the camera ECU 10 (camera module 17) in order to properly perform its own image processing. Therefore, when the camera ECU 10 and the stereo ECU 20 are connected for the first time, the stereo ECU 20 captures the serial number of the camera module 17 and its calibration data. Both the fetched data are stored in the EEPROM 28. After the storage, in the next communication (for example, at the next start-up), only the serial number is confirmed, and if it is the same as the first communication, the calibration data stored in the EEPROM 28 is used. In step 13, the image processing LSI 24 loads an image processing program from the memory. In step 14, the geometric transformation LUT is finally set. In step 15, the geometric conversion processing operation of the geometric conversion LSI 22 can be started, and in step 16, the image recognition processing operation of the image processing LSI 24 can be started.

  Now, the check operation of the image processing function performed by the test pattern transmitted from the camera ECU 10 in step 11 will be described. FIG. 3 is a detailed sequence of the test pattern check in step 11 of FIG. In step 20, the geometric conversion LUT in the memory 25 is set. Here, in order to generalize the check routine, the LUT for geometric transformation is set to 1: 1 (input image before geometric transformation = output geometric transformation image). In step 21, the test pattern check program is loaded into the program area of the image processing LSI 24. In step 22, the SV-CPU 23 instructs the camera CPU 13 in the camera ECU 10 to start transmitting a test pattern, and the camera CPU 13 that has received the instruction promptly transmits the test pattern. In step 23, the SV-CPU 23 instructs (sets in the register) the geometric transformation LSI 22 to start the geometric transformation process. In step 24, the SV-CPU 23 instructs the image processing LSI 24 to start a test pattern check program. In step 25, first, the image processing LSI 24 processes the test pattern received from the camera ECU 10 using a test pattern check program. Then, the processing result is compared with the correct value stored in advance in the EEPROM 28 or other nonvolatile memory. If the processing result by the test pattern check program matches the correct answer value, it is determined as normal and the process proceeds to the next process (step 12 in FIG. 2). If they do not match, it is determined that there is an abnormality, and an abnormality in the image processing function is notified to a predetermined ECU that manages the abnormality via the LAN.

  Here, the correct answer value is a processing result to be obtained when the image processing unit including the geometric conversion LSI 22 and the image processing LSI 24 processes the test pattern. Since the test pattern is literally a known pattern for the test, a reference value (that is, correct value) for determining whether there is an abnormality in the image processing functions of the geometric conversion LSI 22 and the image processing LSI 24 is calculated in advance by design. It is possible to do.

  Next, a simple specific example of the test pattern and the test pattern check program will be described. FIG. 4 is a diagram showing the test pattern and the processing contents of the test pattern by the test pattern check program. The test pattern is an image having three rod-like patterns (A-A ′, B-B ′, C-C ′) as shown in FIGS. 4A and 4B, for example. FIG. 4A shows a pattern for the left imaging element among the two imaging elements arranged on the left and right. FIG. 4B shows a pattern for the right image sensor out of the two image sensors arranged on the left and right. The three rod-like patterns are arranged so that their parallaxes are different. For these two images, the test pattern check program performs a three-dimensional object detection process with the same logic as a normal image process (three-dimensional object detection process). By this three-dimensional object detection processing, the three bar-shaped patterns are detected as three three-dimensional objects, and as a result, the height and position of each bar-shaped pattern in the space are measured (FIG. 4C). By comparing the measured height and position with those correct values, it is possible to determine whether or not the image has been processed normally.

  Now, a method for inspecting that the stereo ECU 20 (particularly, the geometric transformation LSI 22 and the image processing LSI 24) is operating normally by evaluating the image processing result of the test pattern will be described in more detail. This method calculates the distance between the pair of images taken by the imaging elements 11 and 12 arranged on the left and right sides, and calculates the distance to the object in the manner of triangulation based on the parallax with respect to the same object. Use routines.

  FIG. 5 is a diagram illustrating an inspection flow of the image recognition processing function of the stereo ECU 20 based on the test pattern. In step 30, the image processing LSI 24 activates a test pattern check program, calls the above distance calculation routine, and performs image recognition processing of the test pattern. When the distance calculation routine is executed, a parallax image is generated as a result of the processing. If any of condition 1 (step 31), condition 2 (step 32), and condition 3 (step 33), which are determination conditions using this parallax image, is satisfied, the image recognition processing function of the stereo ECU 20 is normal. To be judged. If any one of the conditions does not hold, it is determined that the image recognition processing function of the stereo ECU 20 is abnormal.

  These three conditions will be described with reference to FIGS. The parallax images shown in FIGS. 6 to 8 are the same image. As a test pattern, a known real image showing a road, a pedestrian crossing, and a background is used. A black part (black point) on the parallax image is a “parallax point” having distance data calculated based on the parallax. On the other hand, the white part on the parallax image has no distance data because there is no object and no parallax.

  FIG. 6 is a diagram for explaining condition 1. In condition 1, it is confirmed whether or not “the total number of parallax points on the parallax image by the current image recognition process” matches “the total number of parallax points on the known parallax image (correct value)”. That is, if the total number of parallax points does not match 12345 (example), it is determined that the image recognition processing function of the stereo ECU 20 is abnormal.

  FIG. 7 is a diagram for explaining condition 2. In condition 2, “the total number of parallax points for each identical X coordinate on the parallax image by the current image recognition process” matches “the total number of parallax points for each identical X coordinate on the known parallax image (correct value)”. (The total number of parallax points on the line drawn on the parallax image of FIG. 7 is compared for the current processing result and the correct value, and is implemented for all X coordinates). That is, if the total number of parallax points on any of the same X coordinate directions does not match, it is determined that the image recognition processing function of the stereo ECU 20 is abnormal.

  FIG. 8 is a diagram for explaining condition 3. In condition 3, “the total number of parallax points for each predetermined parallax area on the parallax image by the current recognition process” matches “the total number of parallax points for each predetermined parallax area on the known parallax image (correct value)” It is confirmed whether or not. The predetermined parallax region is a region obtained by dividing the distance data of a black dot into a certain distance range. That is, for the current processing result and the expected value, a comparison of the total number of parallax points having distance data in the range of 0 m to 10 m, a comparison of the total number of black points having distance data in the range of 10 m to 20 m, and so on. Thus, all ranges (that is, all parallax regions) are compared. If the total number of parallax points in any range does not match, it is determined that the image recognition processing function of the stereo ECU 20 is abnormal. In addition, what is necessary is just to determine arbitrarily the division range of a parallax area | region according to a specification etc.

  As described above, by applying the distance calculation routine to the known test pattern, it is determined whether or not the image recognition processing function of the stereo ECU 20 is abnormal. As a result, the distance calculation routine used in normal image recognition processing (not during inspection) can be directly applied to the abnormality determination inspection, so that specialized processing and extra processing for inspection are almost unnecessary. In addition, the processing load on the LSI for inspection can be reduced. In addition, since the result of the image recognition process can be assumed in advance by using a known test pattern, the assumed process result can be stored in advance as a correct answer value. By comparing the processing results using the same test pattern and the correct answer values, it is possible to efficiently perform an image processing recognition function that is complicated and originally required to be subdivided and checked by through check. The check time can be reduced while ensuring the reliability of the check.

1 is a diagram illustrating an example of a configuration of an image processing system using an image processing apparatus of the present invention. It is an example of the processing sequence which stereo ECU20 performs at the time of starting. It is a detailed sequence of the test pattern check of step 11 of FIG. It is a figure which shows the test pattern and the processing content of the test pattern by the test pattern check program. It is a figure which shows the check flow of the image recognition system by a test pattern. It is a figure which shows the comparison regarding the total number of the parallax points on the parallax image by a recognition process. It is a figure which shows the comparison regarding the total number of the parallax points for every same X coordinate on the parallax image by a recognition process. It is a figure which shows the comparison regarding the total number of the parallax points for every predetermined parallax area on the parallax image by recognition processing.

Explanation of symbols

10 Camera ECU
11, 12 Image sensor
13 Camera CPU
14 Image output unit 15, 16 Lens 17 Camera module 20 Stereo ECU
21 Image Input Unit 22 Geometric Transformation LSI
23 SV-CPU
24 Image processing LSI
25, 26, 27 Memory 28 EEPROM

Claims (4)

A distance calculation routine for obtaining a correlation between a pair of stereo images, deriving a parallax with respect to a corresponding point common to both images, and calculating a distance to the corresponding point based on the parallax;
In an image processing apparatus having an image processing unit that operates according to the distance calculation routine,
Storage means for storing in advance a processing result to be obtained from the image processing unit when the distance calculation routine is applied to a known set of stereo images;
Processing results obtained from the image processing unit and processing stored in the storage means when the distance calculation routine is applied to the same test image as the known set of stereo images during the inspection of the image processing unit An image processing apparatus that performs abnormality determination of the image processing unit by comparing with a result.   The image processing apparatus according to claim 1, wherein the processing result to be compared is a total number of parallax points on the parallax image.   The image processing apparatus according to claim 1, wherein the processing result to be compared is a total number of parallax points on a line in a predetermined direction on the parallax image.   The image processing apparatus according to claim 1, wherein the processing result to be compared is a total number of parallax points having a predetermined range of distance data on the parallax image.

Images