JP2006155428A - Apparatus for monitoring area around vehicle - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To rapidly and accurately achieve pattern matching when determining the parallax of a target based on a first and a second image. <P>SOLUTION: Images of aiming targets 134a-134c placed in a right position PR are picked up using a right infrared camera 16R to obtain a right gray-scale image 54. Images of aiming targets 134a-134c placed in a left position PL are picked up using a left infrared camera 16L to obtain a left gray-scale image 58. The average value yra of the y-coordinates of the aiming targets 134a-134c in the cut area 162R of a preset right image is determined and the average value yla of the y-coordinates of the aiming targets 134a-134c in the cut area 162L of a left image is determined. The difference Δya between the average value yra and the average value yla is calculated and the cut area 162L is set to move in y-direction by the difference Δya. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a vehicle periphery monitoring device that detects a peripheral object from a first image obtained by a first imaging means and a second image obtained by a second imaging means, and in particular, the obtained first image and The present invention relates to a vehicle periphery monitoring device that obtains parallax of an object by performing pattern matching on a second image and detects the distance of the object.

  For example, two imaging means are mounted on the vehicle, the same object is imaged using these imaging means, the distance from the vehicle to the object is measured using the parallax between the captured images, Or the vehicle periphery monitoring apparatus which measured the real space position of the target object with respect to a vehicle, and alert | reported the presence or absence of the front obstacle to a driver is developed (refer patent document 1).

  After the object is detected on the two images obtained by these imaging means, the parallax of the object is obtained by performing pattern matching, and the distance of the object is detected based on the parallax.

JP 2003-216937 A

  In order to accurately measure the position and distance of the object, the mounting angle of the imaging means must be accurately determined. In particular, when the object is located far away, a large error occurs in the position and distance measurement due to the influence of a slight difference in the mounting angle of the imaging means. For this reason, it is good to image the target arrange | positioned in a known position with an imaging means, and to perform the aiming for calculating | requiring the exact attachment angle of an imaging means based on the target image on the obtained image.

  In this case, an appropriate cutout area is set for each of the two images, the object extraction process, the pattern matching process using the correlation calculation, and the parallax detection process are performed from the images of these cutout areas, and the distance between the objects Is calculated. The cutout areas set in the two images are calculated from known parameters such as the height of the imaging means and the height of the object, and can be set independently.

  By the way, in practice, there is a slight distortion of the vehicle itself and an error in manufacturing the support member of the image pickup means, so the height and direction of the two image pickup means may be slightly different. If set, there is a concern that consistency between the two cannot be ensured. In such a case, in the pattern matching performed by extracting the image of the target object extracted from one image as the reference image and moving it on the other image as the comparison image, the search time for matching is established. There is a concern of increasing or matching to other objects.

  The present invention has been made in consideration of such a problem, and sets an appropriate cutout region from each of the first image and the second image, and performs pattern matching when obtaining the parallax of the object from the cutout region. An object of the present invention is to provide a vehicle periphery monitoring device that can be quickly and accurately performed.

  A vehicle periphery monitoring device according to the present invention is a vehicle periphery monitoring device that detects the position of a peripheral object from images of a plurality of imaging means mounted on a vehicle, and for each image of the plurality of imaging means. An image area setting means for setting a cutout area to be used for object detection, an object extraction means for performing an object detection process on each cutout area set by the image area setting means, and an object extraction means And a modified image region setting unit that obtains a corrected image region by correcting at least one of the cutout regions set by the image region setting unit based on an extraction result (the invention according to claim 1). .

  Thereby, since each cut-out image is set based on the object actually extracted, both are associated and the consistency is ensured. Therefore, in pattern matching performed by extracting an image of an object extracted from one of the images and moving it on another image, it is possible to reduce the time until matching is achieved and / or to match another object. Therefore, it is possible to realize a quick and accurate process.

  The vehicle periphery monitoring device according to the present invention is a vehicle periphery monitoring device that detects the position of a peripheral object from the first image obtained by the first imaging means and the second image obtained by the second imaging means. , Object extraction means for extracting the object imaged by the first imaging means and the second imaging means from the first image and the second image, respectively, the first imaging means and the second imaging means. An image area setting for setting a first cutout area for the first image as an area for detecting the object imaged by the pattern matching and setting a second cutout area for the second image And the image region setting means is based on the position of the object extracted by the object extraction means, and the first imaging means and the second imaging means. And sets the first cut-out area and the second cut-out region as the vertical coordinate is coincident to the virtual plane including the optical axis (the second aspect of the invention).

  In this way, since the first clipped image and the second clipped image are set based on the actually extracted object, both are associated with each other to ensure consistency. Therefore, in the pattern matching performed by extracting the image of the target object extracted from one image and moving it on the other image, the time until matching is achieved and / or the possibility of matching with another object may be reduced. Therefore, it is possible to realize a quick and accurate process.

  In this case, three or more imaging units may be provided. For example, one imaging unit may be used as the first imaging unit, and the rest may be used as the second imaging unit.

  Further, the first imaging unit images a first target as the object, the second imaging unit images a second target arranged at the same height as the first target as the object, and the image When the region setting means sets the first cutout region and the second cutout region based on a difference between the vertical coordinates of the first target and the second target in the first image and the second image. Good (invention of claim 3). Thus, by using the vertical difference on the images of the first target and the second target having the same height, the first cutout area and the second cutout area can be set quickly and easily. In this case, the first target and the second target may be the same.

  Since the vehicle periphery monitoring apparatus according to the present invention sets a cut-out image based on an actually captured object, the images are associated with each other to ensure consistency. Therefore, in the pattern matching performed by extracting the image of the target object extracted from the image as the reference image and moving it on the other image as the comparison image, the time until the matching is established is shortened, and other objects There is almost no possibility of matching, and it is possible to realize a quick and accurate process.

  Hereinafter, an embodiment of the vehicle periphery monitoring device according to the present invention will be described with reference to FIGS.

  As shown in FIG. 1, a night vision system (vehicle periphery monitoring device) 10 according to the present embodiment is a system mounted on a vehicle 12, and includes an ECU (Electric Control Unit) 14 that is a main control unit, A pair of left and right infrared cameras (first imaging means, second imaging means) 16L, 16R, a HUD (Head Up Display) 18 for displaying a detection result image, a speaker 20 for generating an alarm sound, etc., and a traveling speed A vehicle speed sensor 22 for detecting, a yaw rate sensor 24 for detecting a yaw rate during traveling, a solar radiation sensor 26, a headlight switch 28, a main switch 30 for starting or stopping the system, and a connection part 32 for connecting to an external computer system Have For the connection between these devices constituting the night vision system 10, an in-vehicle communication network that is also used for other systems may be used.

  The infrared cameras 16R and 16L are arranged in parallel at the right end and left end of the horizontally long grill hole at the lower part of the bumper, and are directed forward at the left and right symmetrical positions, respectively, and are spaced by an inter-camera distance (hereinafter also referred to as a baseline length) B. Is provided. The infrared cameras 16R and 16L detect far-infrared rays to pick up infrared images that become brighter at higher temperatures and supply them to the ECU 14, respectively.

  The HUD 18 is provided so as not to obstruct the front view at the front position of the driver above the instrument panel. The HUD 18 is stored in the instrument panel in the system-off state, is determined to be night from the information of the solar radiation sensor 26, and the headlight (or fog light) is turned on from the information of the headlight switch 28. Pops up when the main switch 30 is turned on. The video display unit of the HUD 18 is composed of a concave mirror, and reflects and projects an image from the inside of the instrument panel. The night vision system 10 may be automatically activated in conjunction with the auto light function regardless of the operation of the main switch 30. The display brightness of the HUD 18 may be adjustable by an appropriate switch operation.

  The ECU 14 detects the heat source object from the parallax between the two images based on the stereo infrared images obtained from the infrared cameras 16R and 16L, and displays it as a white silhouette on the HUD 18. Further, when a pedestrian is specified from among the heat source objects, a sound is emitted from the speaker 20 for alerting, and at the same time, the pedestrian projected on the HUD 18 is highlighted and surrounded by a conspicuous color frame. This alerting function predicts the time required to reach the position of the pedestrian in a predetermined speed range, and alerts at a timing at which a sufficient avoidance operation can be performed.

  Infrared cameras 16R and 16L are subjected to aiming setting, which is an adjustment process described later, at the time of manufacturing in a manufacturing facility or during periodic inspection in order to accurately determine the position, distance, and shape of a distant heat source object.

  As shown in FIG. 2, the ECU 14 converts the infrared image captured by the infrared cameras 16R and 16L from A / D to generate a grayscale image, and binarizes the grayscale image based on a threshold value. A binarization processing unit 42 that generates a digitized image, an image storage unit 44 that stores a grayscale image and a binarized image, and an aiming mode execution unit 48 that performs aiming and stores the processing result in the camera parameter storage unit 46. A normal mode execution unit 50 that performs normal image processing while controlling the HUD 18 and the speaker 20 while referring to a sensor such as the vehicle speed sensor 22 and the camera parameter storage unit 46, and an external computer system via the connection unit 32. A mode selection unit 52 for selecting either the aiming mode or the normal mode based on the instruction A.

  As shown in FIG. 3, the image storage unit 44 includes a right grayscale image 54 and a right binarized image 56 based on an infrared image captured by the right infrared camera 16R, and an infrared image captured by the left infrared camera 16L. A left grayscale image 58 based on is stored. Each of these images is a horizontally long digital image obtained by imaging the front of the vehicle 12. In the right grayscale image 54 and the left grayscale image 58, the luminance of each pixel is represented by a predetermined gradation (for example, 256 gradations), and in the right binarized image 56, the luminance of each pixel is represented by 0 or 1. This is a binary image. In practice, multiple images can be stored simultaneously.

  Returning to FIG. 2, the aiming mode execution unit 48 includes a manufactory mode unit 70 that performs aiming using an aiming target control device 100 (see FIG. 4) as an external computer system in a manufactory that manufactures the vehicle 12, and a service aiming setting device. 120 (see FIG. 5) has a service mode unit 72 that performs aiming at a service factory or the like, and one of the factory mode unit 70 and the service mode unit 72 is selected based on an instruction from these external computer systems Executed.

  The aiming mode execution unit 48 is stored in a parameter input unit 74 that inputs predetermined parameters from an external computer system at the start of aiming, an initial setting unit 76 that performs initial settings necessary for aiming, and an image storage unit 44. A template matching unit 78 for matching the left and right grayscale images 54 and 58 using a template, and a brightness adjustment LUT for adjusting the brightness of the image signals obtained by imaging with the infrared cameras 16R and 16L. Brightness adjustment LUT setting unit 80 to be set, left and right camera image distortion correction unit 82 for correcting image distortion caused by solid differences such as focal lengths and pixel pitches of infrared cameras 16R and 16L, and infrared cameras 16R and 16L Left and right camera mounting angle calculation unit 84 for calculating the mounting angle (pan angle, pitch angle) of The left and right camera image cutout coordinate calculation unit 86 for calculating cutout coordinates serving as a reference for cutting out the calculation range from the image, and the object image resulting from the optical axes of the infrared cameras 16R and 16L being not set in parallel. A parallax offset value calculation unit 88 that calculates a parallax offset value that is an error included in the parallax.

  Note that the parallax offset value calculation unit 88 calculates the actual parallax on the images of the objects captured by the infrared cameras 16R and 16L, and each image captured by the infrared cameras 16R and 16L for each pan. It functions as a parallax correction value calculation unit that cuts out according to a corner and calculates a parallax offset value in order to further improve distance measurement accuracy.

  The initial setting unit 76 selects a template from six templates TP1, TP2, TP3, TP4, TP5, and TP6 (typically also shown as templates TP) prepared according to the distance to the object. 94. Further, the ECU 14 has a model storage unit 96 that stores a perspective transformation model for obtaining the position of the object as a mathematical expression, and the aiming mode execution unit 48 and the normal mode execution unit 50 use the perspective transformation model to target the object. The position of is calculated. The model storage unit 96 stores a short distance model when the object is a short distance and a long distance model when the object is a long distance.

  The ECU 14 includes a central processing unit (CPU) as a main control unit, a random access memory (RAM) and a read only memory (ROM) as storage units, and the above-described functional units are programmed by the CPU. This is realized by executing software processing in cooperation with a storage unit or the like.

  As shown in FIG. 4, the aiming target control device 100 includes a positioning device 102 for the vehicle 12, a gate 104 provided at a known distance Zf in front of the infrared cameras 16R and 16L in a state where the vehicle 12 is positioned, A main control device 106 that communicates with the ECU 14 via the connection unit 32 and controls the gate 104 is provided. The gate 104 includes two support columns 108 provided at intervals slightly wider than the vehicle width, and a horizontally long target plate 110 supported by the support columns 108 at the left and right ends. The target plate 110 is a main controller 106. It is possible to move up and down along the column 108 under the action of. On the target plate 110, eight aiming targets 112a, 112b, 112c, 112d, 112e, 112f, 112g, and 112h (typically also indicated as aiming targets 112) as objects to be imaged are arranged horizontally in order from the left. The aiming targets 112a to 112h are heat sources.

  The left aiming targets 112a to 112d are arranged at a relatively small distance d (d <B) to form the left target group 114. Similarly, the right aiming targets 112e to 112h are arranged at a distance d to form the right target group 116. The distance between the aiming target 112d, which is the right end of the left target group 114, and the aiming target 112e, which is the left end of the right target group 116, is equal to the base line length B, and the aiming targets 112d and 112e are arranged in front of the infrared cameras 16R and 16L. The

  The aiming targets 112a to 112h are not limited to a heat source such as a heating element. For example, as shown in FIG. 5, small metal plates (aluminum plates or the like) 118a to 118h as heat reflecting plates may be used. By using such a metal plate as a target, heat (infrared rays) generated and radiated by the vehicle 12 is reflected and can be imaged by the infrared cameras 16R and 16L. In this case, it is not necessary to switch the target on and off, and there is no power consumption. In addition, it is preferable to use a material with low heat reflectance as the target plate 110 because the contrast between the metal plates 118a to 118h and the target plate 110 becomes clear on the captured image.

  As shown in FIG. 6, the service aiming setting device 120 includes a position mark 122 indicating the wheel position of the vehicle 12 at the time of aiming setting, and the infrared cameras 16R and 16L in a state where the vehicle 12 is positioned based on the position mark 122. A headlight tester 124 provided at a position Z ahead of a predetermined distance (hereinafter also referred to as an object distance), and a main controller 126 that communicates with the ECU 14 via the connection portion 32. The headlight tester 124 is movable along the rail 128 in parallel with the vehicle width direction, and the lifting platform 130 can be moved up and down. The lifting platform 130 is provided with a target plate 132. The target plate 132 has three aiming targets 134a, 134b, and 134c (typical imaging objects) that are three heat sources arranged horizontally at intervals d (d <B). Are also shown as aiming targets 134). The aiming target 134 is the same as or substantially the same as the aiming target 112.

  Next, a general perspective transformation model M and a short distance model and a long distance model stored in the model storage unit 96 will be described.

  As shown in FIG. 7, the general perspective transformation model M is a model that shows a state where an object W having a height Y is imaged by a lens 138 having a focal length f to obtain an image having a height y. That is, the light that has entered the lens center O from the point Pa that is one end of the object W travels straight, and the light that has entered parallel to the optical axis C is refracted at the point O ′ of the lens 138. Light is collected at point Pb. The light that has entered the lens center O from the point Pa ′ on the optical axis C, which is the other end of the object W, travels straight on the optical axis C to the point Pb ′. In this way, the object W forms an image w having a height y through the lens 138 at the position of the imaging distance F on the opposite side. As is clear from FIG. 7, the image w is inverted with respect to the object W.

By the way, the similarity between the triangle composed of the three points Pa, Pa ′, O and the triangle composed of the three points Pb, Pb ′, O, and the triangle composed of the three points O, O ′, Pc and the three points Pb, Pb ′, Pc. The following expressions (1) and (2) are derived from the similarity relationship of the triangles consisting of: However, the point Pc is a point at the position of the focal length f on the optical axis C in the direction of the image w when viewed from the lens 138.
y / Y = F / Z (1)
y / Y = (F−f) / f (2)

When y and Y are deleted from the equations (1) and (2), the following equation (3) is obtained.
F = fZ / (Z−f) (3)

  In the aiming mode, since the object distance Z is a known distance, the imaging distance F is set by the equation (3) in steps S8 and S33 described later. Actually, the expression (3) may be included in the expressions (7-1) to (7-4) described later.

When the object W is sufficiently far from the lens 138 (when Z is f), the expression (3) can be expressed by the following approximate expression (4).
F≈f (= fZ / Z) (4)

  Note that the width X of the object W can also be represented by a perspective transformation model M similar to the height Y, and the image w is represented by the width x with respect to the width X. In addition, the actual image w is formed on the opposite side of the lens 138 with respect to the object W, but the virtual imaging plane S is on the same side as the object W and from the lens 138 to simplify the model. It can be provided at the position of the imaging distance F. As a result, as shown in FIG. 8, the object W and the image w become similar upright images with the lens center O as a reference, and the object W is perspective-transformed into the image plane S.

As is clear from FIG. 8, the width x and height y of the image w on the imaging plane S are expressed by the following equations (5) and (6).
x = F / Z · X / p (5)
y = F / Z · Y / p (6)

  Here, the parameter p is a pixel pitch when the imaging plane S is applied to an actual digital image. Also, when the object W is relatively close, the error cannot be ignored when the expression (4) is applied, so the expression (3) is substituted into the expressions (5) and (6). The perspective transformation model when the object W is relatively close is represented by the following first formula group (7-1) to (7-4).

  The first formula group is included in the short distance model in the model storage unit 96, and is selected by the aiming mode execution unit 48 when the mode selection unit 52 selects the aiming mode in step S3 described later. The position of the aiming target 112 or 134 is used for calculation. That is, a value automatically set as the known distance Zf is used as the object distance Z at the time of manufacturing plant aiming, and the object distance Z input from the main controller 126 is used at the time of service aiming. Since each aiming target 112 or 134 has a known X value as a coordinate value in the vehicle width direction and Y as a coordinate value in the height direction, the image is formed from the equations (7-1) and (7-2). The theoretical coordinates Pb (x, y) of the image w on the surface S are obtained. During service aiming, the height coordinate Y is corrected based on the camera height H (see FIG. 1).

  Thereafter, the mounting angles of the infrared cameras 16R and 16L are obtained by comparing the theoretical coordinates Pb (x, y) with the actually obtained coordinates of the aiming target 112 on the real screen. Further, theoretical coordinates of the aiming target 112 are obtained from the coordinates on the real screen using the equations (7-3) and (7-4), and compared with the known coordinate values X and Y, the infrared camera 16R and A mounting angle of 16L may be obtained.

  Since the first group of equations is set based on the equation (3) indicating the imaging distance F, the theoretical coordinates Pb (x, y) are accurately obtained, and as a result, the infrared cameras 16R and 16L are obtained. Can be obtained with high accuracy.

  On the other hand, when the object W is far away, the expression (4) can be used as an approximate expression, and the expressions (5) and (6) are the following second expression groups (8-1) to (8). -4) It is expressed by the formula.

  This second formula group is included in the long-distance model in the model storage unit 96, and is selected by the normal mode execution unit 50 when the mode selection unit 52 selects the normal mode in step S3 described later. Then, the position of the object is used for calculation. That is, after calculating the object distance Z to the object from the parallax between the two images obtained by the infrared camera 16R or 16L, the equations (8-3) and (8-4) are used from the coordinates on the image. Thus, coordinate values X and Y in the space of the object are obtained. In general, since the object distance Z is sufficiently larger than the focal distance f in the normal mode, the error in the equation (4) is negligible, and the coordinate values X and Y of the object are (8-3). It is obtained with sufficiently high accuracy by the equation and the equation (8-4). In addition, since the equations (8-3) and (8-4) are simpler than the equations (7-3) and (7-4), high speed calculation is possible.

  Even in the normal mode, when the obtained object distance Z is equal to or less than a predetermined threshold, the expressions (7-3) and (7-4) may be used.

  Next, a method for aiming the night vision system 10 using the aiming target control device 100 or the service aiming setting device 120 configured as described above will be described.

  The aiming includes a manufacturing aiming mode in which the aiming target control device 100 is used in the manufacturing facility and a service aiming mode in which the service aiming setting device 120 is used.

  In the factory aiming mode, the positioning device 102 positions the vehicle 12 and connects the main control device 106 to the connection portion 32 of the vehicle 12. The main control device 106 sends an instruction to the ECU 14 to execute the manufacturing aiming using the aiming target control device 100. The aiming targets 112 a to 112 h are adjusted so as to have a specified height according to the type of the vehicle 12.

  In the service aiming mode, positioning is performed by aligning each wheel of the vehicle 12 with the position mark 122, and the main control device 126 is connected to the connection portion 32 of the vehicle 12. Main controller 126 sends an instruction to ECU 14 to execute service aiming using service aiming setting device 120. The aiming targets 134a to 134c are adjusted to have a specified height.

  The process shown in FIGS. 9 and 11 to 14 is a process executed mainly by the aiming mode execution unit 48 in the ECU 14.

  9. First, in step S1 of FIG. 9, stereo infrared images are input from the infrared cameras 16R and 16L, and A / D conversion is performed in the image input unit 40 (step S2) to obtain gray scale images 54 and 58. The gray scale images 54 and 58 are held in the image storage unit 44 until a predetermined clear instruction or overwrite type is given, and a plurality of sheets can be recorded. The gray scale images 54 and 58 are each binarized by the binarization processing unit 42, and the right binarized image 56 is stored in the image storage unit 44.

  In step S3, the mode selection unit 52 determines whether to execute the aiming mode or the normal mode in accordance with an instruction from the main control device 106 or 126. When executing the normal mode, the process proceeds to step S5, and when executing the aiming mode, the process proceeds to step S4.

  In the normal mode in step S5, under the action of the normal mode execution unit 50, the camera parameters stored in the camera parameter storage unit 46 are referred to, and the HUD 18 and the speaker 20 are controlled and detected as described later. The normal mode for alerting according to is executed.

  In step S5, the mode selection unit 52 determines whether to use the aiming target control device 100 or the service aiming setting device 120. If it is determined that the aiming target control device 100 is used, the process proceeds to step S6. The mill aiming mode is executed under the action of the mode unit 70. If it is determined that the service aiming setting device 120 is used, the process proceeds to step S30 (see 12), and the service aiming mode is executed under the action of the service mode unit 72. Hereinafter, the factory aiming mode and the service aiming mode will be described in order.

  In the factory aiming mode, first, in step S6, the distance from the infrared cameras 16R and 16L to the target plate 110, that is, the object distance Z is set. In this case, the object distance Z is automatically set as a known distance Zf.

  In step S <b> 7, the reference template is selected according to the object distance Z under the action of the template setting unit 94. As shown in FIG. 10, templates TP1, TP2, TP3, TP4, TP5, and TP6 are small image data obtained by imaging one temporary target arranged at a specified distance, and the specified distance is close to the infrared cameras 16R and 16L. They are set as Z1, Z2, Z3, Z4, Z5, and Z6 at equal intervals (or equal ratio intervals, etc.) in order. Templates TP1, TP2, TP3, TP4, TP5, and TP6 are stored in template storage unit 95 in ECU. As the temporary target, the same target as the aiming target 112 is used. In the factory aiming mode, for example, if the known distance Zf is Zf = Z3, the template TP3 is selected as the reference template.

  In step S8, the imaging distance F is set based on the perspective transformation model M (see FIG. 7).

  Note that the processing in steps S6 to S8 is performed under the action of the initial setting unit 76, and is executed only once for the first time when aiming is executed while referring to a predetermined execution frequency parameter.

  In step S9 (object extraction means), template matching processing is performed based on the reference template selected in step S7. That is, the grayscale images 54 and 58 of the aiming target 112 captured by the infrared cameras 16R and 16L are correlated with the template TP, and the coordinates of the grayscale image that minimizes the correlation calculation result are calculated and stored. To do. (Step S10, target coordinate calculation means). For the correlation calculation, for example, an SAD value (Sum of Absolute Difference) that is the sum of absolute value errors for each pixel is used.

  In step S11, it is checked whether or not the number of acquired grayscale images 54 and 58 has reached a predetermined number N, and if it has reached N, the process proceeds to step S12, where it is less than N. Returning to step S1, the acquisition of the grayscale images 54 and 58 is continued, and the target coordinate calculation storage process is repeated.

  In step S12, a process of calculating an average value Pave of the calculated N sets of target coordinates is performed. If the target coordinate calculation is normally calculated, the process proceeds to step S14. If not, the process proceeds to step S3. A branch process of returning (step S13) is performed.

  In step S14 of FIG. 11, a luminance adjustment LUT setting process is performed. In this process, for example, the levels of the luminance signals of the aiming targets 112 detected by the infrared cameras 16R and 16L are compared with each other so that the template matching by the correlation calculation can be executed reliably. A brightness adjustment LUT is set to adjust so that the brightness signal from the infrared camera 16R to be always larger than the brightness signal from the infrared camera 16L. When the brightness adjustment LUT setting process is normally executed, the process proceeds to step S16 (step S15).

  In step S16, image distortion correction value calculation processing for correcting image distortion caused by individual differences such as focal lengths and pixel pitches of the infrared cameras 16R and 16L is performed. If the video distortion correction value is normally calculated, the process proceeds to step S18 (step S17).

  In step S18, a pan angle and a pitch angle (also referred to as tilt angle), which are left and right camera mounting angles, are calculated. If the left and right camera mounting angles are normally calculated, the process proceeds to step S20 (step S19). ).

  Here, the pan angle means that the optical axes of the two infrared cameras 16R and 16L are directed in the same height direction, and therefore the rotation direction in the camera array virtual plane including both optical axes of the infrared cameras 16R and 16L. It can be said to be an angle and a direction in which parallax occurs, and corresponds to the x direction on the obtained image (see FIG. 8). Further, the pitch angle can be referred to as a vertical elevation angle with respect to the camera array virtual plane, and corresponds to the y direction on the obtained image.

  In step S20, a standard cutout coordinate calculation process is performed to cut out a range used for image processing from each image captured by the infrared cameras 16R and 16L. If the cutout coordinates are normally calculated, step S20 is performed. The process moves to S22 (step S21).

  In step S22, a process of calculating a parallax offset value that is an error included in the parallax of the object image due to the fact that the optical axes of the infrared cameras 16R and 16L are not set in parallel is performed. When the parallax offset is calculated, the process proceeds to step S24 (step S23).

  In step S24, the brightness adjustment LUT, image distortion correction value, pan angle and pitch angle, cut-out coordinates, and parallax offset value obtained in steps S14, S16, S18, S20, and S22 are stored in the camera parameter storage unit 46. If it is correctly stored, the mill aiming mode (or service aiming mode) is terminated. At this time, the ECU 14 sends a signal to the main control device 106 or 126 to the effect that the aiming has been completed. Thereafter, when the normal mode is executed, a predetermined restart process may be performed. If the determination result in each branch process in steps S17, S19, S21, S23, and S25 is negative, the process returns to step S3 as in the case of a negative result in the branch process in step S13. .

  Next, processing in the service aiming mode will be described. In the service aiming mode, the processing of steps S1 to S3 (see FIG. 9) is executed in the same manner as the manufacturing aiming, and the branch processing from the next step S4 to step S30 is performed. 12 to 14 are executed.

  In step S30 in FIG. 12, the distance from the infrared cameras 16R and 16L to the target plate 132, that is, the object distance Z is input and set. In this case, the object distance Z is input from the main controller 126 as a numerical value in the range from Z1 to Z7 (> Z6) m and supplied to the ECU 14. The input of the object distance Z may be input based on an easily measurable location (for example, the position mark 122), and a predetermined offset amount may be automatically subtracted. Further, the object distance Z may be automatically detected and input by using a laser distance detecting device or the like as the object distance detecting means.

  In step S31, the heights of the infrared cameras 16R and 16L are confirmed, and the camera height H (see FIG. 1) is input.

  In step S32, under the action of the template setting unit 94, one of the templates TP1 to TP6 is selected as a reference template according to the object distance Z. As described above, the templates TP1 to TP6 have the specified distances Z1 to Z6m when the temporary target is imaged. In step S32, a template whose corresponding specified distance is equal to or smaller than the object distance Z and closest to the object distance Z is selected. Specifically, when the object distance Z is Z1 to Z2 ′, Z2 to Z3 ′, Z3 to Z4 ′, Z4 to Z5 ′, Z5 to Z6 ′, Z6 to Z7, the templates TP1 to TP6 correspond in order. To be selected. Here, Z2 ', Z3', Z4 ', Z5' and Z6 'are values smaller than Z2, Z3, Z4, Z5 and Z6 by the minimum input unit, respectively. More specifically, when the object distance Z is between Z4 and Z5 ', the template TP4 is selected.

  In step S33, the image forming distance F is set as in step S8. Note that the processing of steps S30 to S33 is performed under the action of the initial setting unit 76, and is executed only once for the first time when aiming is executed while referring to a predetermined execution frequency parameter.

  In step S34, the position of the target plate 132 is confirmed. That is, in the service aiming mode, the target plate 132 is arranged in the order of the center position PC, the left position PL, and the right position PR (see FIG. 6) to perform the aiming. Therefore, when step S34 is first executed, the main controller 126 In response to this, a confirmation instruction signal is sent so as to set the target plate 132 at the center position PC. In response to this signal, the main controller 126 displays a message such as “Place the target at the center position PC and press the“ Y ”key” on the monitor screen. In response to this message, the coordinator moves the headlight tester 124 along the rail 128 manually or by a predetermined driving device, places the target plate 132 at the center position PC, and presses the “Y” key.

  Step S34 is repeatedly executed, and after every N times, “Place the target at the left position PL and press the“ Y ”key.”, “Place the target at the right position PR and“ Y ”. "Please press the key." Is displayed, prompts the adjuster to place the target plate 132 at the left position PL and the right position PR, and causes the "Y" key to be input after the placement. At the first time after the target plate 132 is placed at the center position Pc, the left position PL, and the right position PR, it is confirmed that the “Y” key has been pressed, and the process proceeds to step S35. No special processing is performed.

  In step S35, a branching process is performed according to the position of the target plate 132 at that time. That is, when the target plate 132 is disposed at the center position PC (first to Nth times), the process proceeds to step S36, and when it is disposed at the left position PL, the process proceeds to step S41 and is disposed at the right position PR. Sometimes, the process proceeds to step S46.

  In step S36, template matching processing is performed in the same manner as in step S9.

  In step S37, as in step S10, the target coordinates of the aiming target 134 are calculated and stored.

  In step S38, as in step S11, the number of acquired grayscale images 54 and 58 is confirmed, and if it is N or more, the process proceeds to step S39, and if it is less than N, the process returns to step S1. (S3 to S8 or S30 to S35 are skipped after the second time.)

  In step S39, a process for calculating the average value Pave of the target coordinates is performed for the center position PC as in step S12. If the target coordinates are normally calculated, the process proceeds to step S1. Then, a branching process (step S40) of returning to step S3 is performed.

  Similarly, the target plate 132 is arranged at the left position PL, and steps S41 to S45 shown in FIG. 13 are executed.

  Further, the target plate 132 is disposed at the right position PR, and steps S46 to S50 shown in FIG. 14 are executed.

  If it is determined in the last step S50 that the target coordinates have been normally calculated, the process proceeds to step S14 (see FIG. 11), and thereafter, the same processing as in the factory aiming mode is performed, and each camera parameter is stored in the camera parameter. Stored in the unit 46.

  Next, step S36, which is a process of performing template matching using the selected reference template (hereinafter referred to as template TPs), will be described with reference to FIG.

  As shown in FIG. 15, the selected template TPs is moved in the vertical and horizontal directions by a small amount in a predetermined order on the right grayscale image 54, pattern matching is performed at each position, and matching with the background image is performed. Check whether it is true or false. The matching is determined based on the SAD value.

  In this case, the size of the temporary target image 140 in the template TPs is equal to or larger than the size of the images of the aiming targets 134a to 134c on the image. This is because in step S32 described above, the template whose distance is equal to or smaller than the object distance Z and that is closest to the object distance Z is selected.

  As indicated by a two-dot chain line TP ′ in FIG. 15, when the template TPs moves to the position of the aiming target 134c, the image 140 and the aiming target 134c substantially coincide with each other, and the high luminance part and the low luminance part overlap. The SAD value is a sufficiently small value. Thereby, it is determined that matching has been established, and the target coordinates of the aiming target 134c are specified from the center point of the template TPs at that time. That is, the center point coordinate Pb (x, y) (see FIG. 8) in the range of the two-dot chain line TP ′ is stored in the predetermined storage unit as Pt [i] in the array format (step S10). Here, the parameter i is a counter indicating the number of processing times, and is incremented from i = 1 to i = N according to the number of right grayscale images 54. In practice, target coordinates are obtained for each of the aiming targets 134a to 134c, but are typically shown as one target coordinate Pt [i] for ease of explanation.

  The calculation of the average value Pave in step S12 is obtained by the following equation (9).

  By performing such averaging processing, even if the infrared cameras 16R and 16L or the imaging environment (fluctuations in temperature, etc.) are somewhat unstable, the average value Pave cancels the error, and the target coordinates are obtained accurately. be able to.

  By the way, the place where service aiming is performed is assumed to be an inspection place of a general service factory, and it is difficult to completely eliminate unnecessary heat sources that emit or reflect infrared rays other than the aiming targets 134a to 134c. Other heat sources may be captured as relatively weak levels or small images. For example, another heat source (for example, distant illumination) 142 exists around the aiming target 134c on the right grayscale image 54, and the heat source 142 may be imaged smaller than the aiming target 134c on the image. Even in such a case, since the size of the image 140 of the template TPs is equal to or larger than the size of the image of the aiming target 134c, the SAD value is considerably large when the template TPs moves to the position of the heat source 142. It becomes a value and is clearly distinguished from the aiming target 134c.

  For the aiming targets 134a and 134b, the target coordinates Pt [i] are specified by the same process, and the average value Pave is obtained. The aiming targets 134a to 134c can be identified by a relative positional relationship.

  FIG. 15 shows an example of the right grayscale image 54 when the headlight tester 124 is arranged at the center position PC. However, when the headlight tester 124 is arranged at the right position PR or the left position PL. Alternatively, the same processing is performed on the left grayscale image 58. A similar process is also performed for step S9 which is a pattern matching process in the factory aiming mode.

  Next, the processing from step S18, which is a mounting angle calculation process for the infrared cameras 16R and 16L, to step S20, which is a left and right camera image cutout coordinate calculation process, is described with reference to FIG. While explaining. For convenience of explanation, steps S18, S19, and S20 are not distinguished in FIG.

  First, in step S101 in FIG. 16, the reference coordinate value P0 indicating the spatial coordinates of the target is read from a predetermined storage unit (coordinate reference value storage means).

  In step S102, the reference coordinate value P0 is converted using equations (7-1) and (7-2), which are short-distance models, to obtain the reference coordinate P0 '(x0, y0) on the image. In the aiming mode, a short-distance model is selected in advance, and the coordinates of a relatively short-distance target can be accurately perspective-transformed on the image.

  Further, by storing the reference coordinate value P0 as a value in the real space, for example, when the position of the target is changed, the coordinates actually measured when inputting from the main control devices 106 and 126 are input as they are. This is preferable. The input actual measurement value is accurately converted based on the short distance model.

  In step S103 (attachment angle calculation means), the reference coordinate value P0 ′ on the image is compared with the average value Pave obtained in step S12 to obtain the pan angle direction difference Δx and the pitch angle direction difference Δy. .

  That is, the differences Δx and Δy indicate the mounting angle as an error with respect to the design reference value of the pan angle and the pitch angle of the infrared camera 16R according to the reference coordinate value P0 ′, respectively. Specifically, when the design reference value is defined as 0 °, the difference Δx is 2 ° and the difference Δy is 1 °, and the pan angle is 2 ° and the pitch angle is 1 °. It is done.

  In step S104, as shown in FIG. 17, a cutout area 162R obtained by moving the reference area 160R on the right grayscale image 54 by the difference Δx in the x direction and the difference Δy in the y direction is obtained. The reference area 160R is set based on the reference coordinate value P0, and is a reference area used for image processing when the infrared camera 16R is accurately pointing forward. By using the cutout area 162R for image processing, the same effect as that obtained when the infrared camera 16R is mechanically adjusted so that the direction of the infrared camera 16R is accurately directed to the front can be obtained.

  As the cutout area 162R, the coordinates of the end point Q1 obtained by moving the end point Q0 of the reference area 160R by the difference Δx in the horizontal direction and the difference Δy in the vertical direction may be typically recorded. Although a detailed description is omitted, the same processing is performed on the left grayscale image 58 obtained from the left infrared camera 16L.

  Next, steps S105 to S108, which are pitch alignment adjustments between the cutout area 162R and the cutout area 162L set independently on the left and right, are performed. The pitch alignment adjustment is a process for associating the cutout areas 162R and 162L and ensuring the consistency in the pitch direction based on the actually captured object. In the following description, the right grayscale image 54 is an image when the target plate 132 is disposed at the right position PR, and the left grayscale image 58 is an image when the target plate 132 is disposed at the left position PL. And Further, as described above, the target plate 132 is set to a constant height at both the right position PR and the left position PL.

  First, in step S105, as shown in FIG. 18, yr1, yr2, and yr3, which are y coordinate values of the aiming targets 134a, 134b, and 134c in the cutout area 162R extracted from the right grayscale image 54, are obtained. Further, an average value yra (= (yr1 + yr2 + yr3) / 3) of these values is obtained.

  In step S106, the y coordinate values yl1, yl2, and yl3 of the aiming targets 134a, 134b, and 134c in the cutout area 162L extracted from the left grayscale image 58 are obtained, and the average value yla ( = (Yl1 + yl2 + yl3) / 3) is obtained.

  In step S107, a difference Δya of the average value yla in the left cutout area 162L with respect to the average value yra in the right cutout area 162R is obtained as Δya ← yra−yla.

  In step S108, a corrected image area is set by moving the left cutout area 162L by the difference Δya in the y direction (that is, the pitch direction). In the example shown in FIG. 18, since yra <yla, the difference Δya becomes a negative value, and the cutout area 162L moves downward in the image and the end point Q2 moves to the end point Q2 ′. The moved cutout area 162L typically stores the coordinate value of the end point Q2 '.

  According to such pitch alignment adjustment, the left cutout area 162L is moved based on the actually imaged aiming targets 134a to 134c so that the pitch angle is aligned with the right cutout area 162R. Distortion and camera stay manufacturing errors can be compensated. Accordingly, as shown in FIG. 19, when the cutout areas 162R and 162L are arranged side by side, the image of the target object when the same target object is imaged is a relative y coordinate value in the cutout areas 162R and 162L. (The value of yra in FIG. 19) matches.

  As a result, quick and reliable pattern matching is possible in the normal mode. That is, as shown in FIG. 20, a small image including an object image 170 extracted by a predetermined method in the right cutout area 162R as the reference image is cut out as a template TPr, and the left cutout area as the comparison image It arrange | positions in the same position in 162L. Next, template matching is performed by correlation calculation such as SAD while moving the template TPr in the cutout area 162L toward the right, which is the positive x direction. At this time, since the y-coordinates of the image 170 in the cutout area 162L and the corresponding image 172 of the same object coincide with each other, the template TPr moves by an appropriate distance according to the parallax, and reaches the broken line position in FIG. When it reaches, it will match the image 172. That is, since the template TPr is matched with little movement in the y direction, there is no need to excessively expand the search area in the y direction, and simple and quick processing is possible. At this time, the SAD value becomes very small, and the determination of matching is surely made. In addition, the template TPr does not match another image 174 that has a different y coordinate and is similar to the image 172, so that reliable matching is realized.

  Note that the processing shown in FIG. 16 may be performed based on an image obtained in a state where the position of the target plate 132 is arranged at the center position PC. The pitch alignment adjustment may be based on any of the cutout areas 162R and 162L, or both of the cutout areas 162R and 162L may be moved based on a predetermined reference. Furthermore, since the aiming targets 112a to 112h of the aiming target control apparatus 100 are set to the same height, the pitch alignment adjustment can be performed even in the manufacturing aiming mode. In this case, the average value yra may be obtained from the right target group 116 in the right grayscale image 54 and the average value yla may be obtained from the left target group 114 in the left grayscale image 58.

  The pitch alignment adjustment is not limited to the aiming targets 112 and 134 that are imaging objects for inspection, and may be performed by imaging a general appropriate heat source. In this case, even if the distance and height of the heat source are unknown, the pitch alignment adjustment can be performed by imaging the same heat source with the infrared cameras 16R and 16L.

  Next, a procedure of processing for detecting an actual object in the normal mode performed after aiming is described. This normal mode is repeatedly executed every minute time under the action of the normal mode execution unit 50.

  First, in step S201 in FIG. 21, stereo infrared images are input from the infrared cameras 16R and 16L, and the right grayscale image 54 and the right binarized image 56, left by the action of the image input unit 40 and the binarization processing unit 42. A gray scale image 58 is stored in the image storage unit 44.

  Thereafter, an object extraction process is performed based on the right binarized image 56 (step S202), parallaxes in the right grayscale image 54 and the left grayscale image 58 are obtained, and the distance from the parallax to each object is determined. Calculate (step S203, object distance detection means). When performing pattern matching, since matching is performed on the cutout area 162R in the right grayscale image 54 and the cutout area 162L in the left grayscale image 58, rapid processing is possible, and reliable matching can be performed.

  Next, the relative position of the object is calculated with reference to the vehicle 12 (step S204), and after performing a predetermined correction based on the behavior of the vehicle 12, the movement vector of each object is calculated (step S205). At this time, the ECU 14 can accurately and quickly detect the position of the actual object based on the equations (8-1) to (8-4) as the long-distance model recorded in the model storage unit 96.

  Further, road structures and other vehicles are identified and excluded while referring to the movement vector or the like (step S206), and pedestrians are determined based on the shape of the object (step S207).

  Thereafter, the right grayscale image 54 is displayed on the HUD 18, and when it is determined that the pedestrian detected in step S207 exists within the predetermined range, an emphasis frame is displayed around the image indicating the pedestrian. A sound is emitted from the speaker 20 and an alerting output is performed (step S209).

  As described above, in the night vision system 10 according to the present embodiment, the right cutout area 162R and the left cutout area 162L are set based on the aiming target 112 (or 134) actually captured and extracted. Both are related to ensure consistency. Therefore, in the pattern matching performed while extracting the image of the object extracted in the right cutout area 162R as the reference image and moving it on the left cutout area 162L as the comparison image, the time until matching is established is shortened, In addition, there is almost no possibility of matching with other objects, and it is possible to realize a quick and accurate process.

  Note that the number of infrared cameras as imaging means is not limited to two, and three or more infrared cameras may be provided. In this case, for example, one infrared camera may be used for the reference image, and the rest may be used for the comparison image.

  The vehicle periphery monitoring device according to the present invention is not limited to the above-described embodiment, and can of course adopt various configurations without departing from the gist of the present invention.

It is the block block diagram which showed schematic structure of the night vision system which concerns on this Embodiment with the vehicle. It is a block block diagram of ECU. It is a figure which shows the structure of the memory | storage data of an image memory | storage part. It is a perspective view which shows an aiming target control apparatus and a vehicle. It is a perspective view which concerns on the modification of a target board. It is a perspective view which shows a service aiming setting apparatus and a vehicle. It is a model figure of a general perspective transformation model. It is a model figure which shows a mode that an image is seen through to an image plane by a perspective transformation model. It is a flowchart (the 1) which shows the procedure which performs aiming setting. It is a figure which shows the content of the template memory | storage part. It is a flowchart (the 2) which shows the procedure which performs aiming setting. It is a flowchart (the 3) which shows the procedure which performs aiming setting. It is a flowchart (the 4) which shows the procedure which performs aiming setting. It is a flowchart (the 5) which shows the procedure which performs aiming setting. It is a conceptual diagram which shows the template matching process in aiming mode. It is a flowchart which shows the procedure of an attachment angle calculation process and a right-and-left camera image cut-out coordinate calculation process. It is a conceptual diagram which shows the process which sets a cutting-out area | region from a reference | standard area | region. It is a conceptual diagram which shows the pitch alignment adjustment process performed with respect to the cut-out area | region on either side. It is a figure which shows the positional relationship of the right and left cut-out area | region where the pitch alignment adjustment was made. It is a conceptual diagram which shows the template matching process in normal mode. It is a flowchart which shows the process sequence of normal mode.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Night vision system 12 ... Vehicle 14 ... ECU 16R, 16L ... Infrared camera 18 ... HUD 20 ... Speaker 44 ... Image memory | storage part 46 ... Camera parameter memory | storage part 48 ... Aiming mode execution part 50 ... Normal mode execution part 52 ... Mode selection 54: Right grayscale image 58 ... Left grayscale image 70 ... Manufacturing mode unit 72 ... Service mode unit 78 ... Template matching unit 94 ... Template setting unit 96 ... Model storage unit 100 ... Aiming target control devices 112, 112a to 112h, 134, 134a to 134c ... aiming target 120 ... service aiming setting device 124 ... headlight tester 138 ... lens 160R ... reference area 162R, 162L ... cut-out area B ... baseline length C ... optical axis F ... imaging distance f ... focal length Separation M ... perspective transformation model P0 ... reference coordinate value PC ... center position PL ... left position PR ... right position S ... imaging plane TP, TP1 to TP6, TPr, TPs, TP '... template yla, yr ... average value Z ... Object distance

Claims (3)

In a vehicle periphery monitoring device for detecting the position of a peripheral object from images of a plurality of imaging means mounted on a vehicle,
Image area setting means for setting a cutout area used for object detection for each image of the plurality of imaging means;
An object extraction means for performing an object detection process on each cut-out area set by the image area setting means;
Modified image region setting means for obtaining a corrected image region by correcting at least one of the cutout regions set by the image region setting unit based on the extraction result of the object extracting unit;
A vehicle periphery monitoring device comprising: In the vehicle periphery monitoring device for detecting the position of a surrounding object from the first image obtained by the first imaging means and the second image obtained by the second imaging means,
Object extraction means for extracting the object imaged by the first image pickup means and the second image pickup means from the first image and the second image, respectively;
A first cutout area is set for the first image as an area for detecting the object imaged by the first imaging means and the second imaging means by pattern matching, and for the second image Image area setting means for setting the second cutout area,
Have
The image area setting means has a coordinate in a vertical direction with respect to a virtual plane including an optical axis of the first imaging means and the second imaging means based on the position of the object extracted by the object extraction means. The vehicle periphery monitoring device is characterized in that the first cutout area and the second cutout area are set as described above. In the vehicle periphery monitoring device according to claim 2,
The first imaging unit images a first target as the object, the second imaging unit images a second target arranged at the same height as the first target as the object, and sets the image area. The means sets the first cut-out area and the second cut-out area based on a difference between the vertical coordinates of the first target and the second target in the first image and the second image. A vehicle periphery monitoring device.

Images