JP2014211694A - Image processing apparatus, distance measurement apparatus, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect an object with stable detection accuracy, and to hold a smaller amount of image data to be used for the detection processing in a memory.SOLUTION: An image processing apparatus includes: a parallax image generation unit 201 and a U map generation unit 202 for generating U map information which indicates a frequency distribution of a parallax value in each of column areas obtained by horizontally dividing a captured image into a plurality of sections, on the basis of parallax image data obtained from left and right luminance image data captured by two imaging units; and a clip position calculation unit 204 and an image clipping unit 206 for clipping an image data part of a detection object candidate area from unclipped image data to be processed (the whole parallax image data), after specifying the detection object candidate area including an object image area including an image of the object, on the basis of generated parallax histogram information, and storing the clipped image data part as image data to be processed, in a memory (image storage unit) 130.

Description

  The present invention relates to a parallax calculation target location on a reference image that is one of a plurality of captured images obtained by imaging with a plurality of imaging means, and a corresponding location on a comparison image that is another captured image of the plurality of captured images. The present invention relates to an image processing device, a distance measurement device, and an image processing program that calculate parallax.

  This type of image processing apparatus is a distance measuring device such as a moving body control apparatus that performs movement control of a moving body such as a vehicle, a ship, an aircraft, or an industrial robot, and an information providing apparatus that provides useful information to a driver of the moving body. Widely used for processing. If a specific example is given, what is used for a driver support system for supporting driving of a driver (driver) of a vehicle is known, for example. Specific examples include an Autonomous Emergency Braking System (AEBS) that automatically brakes when there is a danger of a collision, or approaches the preceding vehicle by driving at the set vehicle speed without stepping on the accelerator. ACC (Adaptive Cruise Control) that automatically adjusts the vehicle speed and keeps the distance from the preceding vehicle.

  In such a driver assistance system, an automatic brake function and an alarm function for avoiding the collision of the own vehicle with an obstacle or reducing an impact at the time of the collision, and maintaining a distance from the preceding vehicle are maintained. Various functions are realized, such as a function for adjusting the speed of the own vehicle to support the vehicle and a function for supporting the prevention of deviation from the traveling lane in which the host vehicle is traveling. For this purpose, an image captured around the host vehicle is analyzed, and various detection objects existing around the host vehicle (for example, other vehicles, pedestrians, road surface components such as lanes and manhole covers, utility poles and guardrails) It is important to accurately detect the type and distance of roadside structures such as roadside structures.

  Patent Document 1 discloses an environment recognition device that identifies the type of an object from the brightness and height position of the object (detection object) present in the detection area (imaging area). In this environment recognition apparatus, first, a specific object map is created by assigning an identification number for each specific object to a target part having a luminance included in a predetermined luminance range for each specific object. Thereafter, the height position of the target part from the road surface is acquired for each target part to which the identification number is assigned. The height position of each target region is calculated from disparity information derived based on two image data obtained from two imaging devices. Then, if the height position is included in the height position range of the specific object specified by the identification number in the specific object table, the target part is provisionally determined to be the specific object.

  After this provisional determination, the difference between the target part and the horizontal distance is within a predetermined range with an arbitrary target part as a base point, and the difference between the target part and the relative distance (distance between the host vehicle and the target part) is predetermined. The target parts that are tentatively determined to correspond to the same specific object within the range (with the same identification number attached) are grouped and set as the target object. Then, the width of the object determined in this way is calculated, and if the width is included in the width range of the specific object in the specific object table, the object is determined to be the specific object. .

  The environment recognition apparatus described in Patent Document 1 performs a series of processes from input of a luminance image (captured image) to identification of an object, a central processing unit (CPU), a ROM storing a program, a work area, and the like. This is executed by a central control unit composed of a semiconductor integrated circuit including a memory such as a RAM. In such a configuration, the entire luminance image data to be input is normally held in a memory such as a RAM, and various arithmetic processes (image processing) using the luminance image data are executed by the CPU. Therefore, a large capacity memory is required.

  However, when there is a restriction on the memory capacity that can be used to hold the image data, such as when an object (detection target) is detected by a mounting component on a board such as an in-vehicle camera, the entire image data is Sometimes it cannot be kept in memory. In addition, in many cases, it is desirable that the amount of image data held in the memory is small from the viewpoint of shortening the processing time.

  On the other hand, as a method of reducing the memory holding amount of unprocessed image data (processing target image data) held in the memory before performing image processing executed by the CPU, it is determined in advance from the entire image data. A method may be considered in which only a part is cut out and stored in a memory as processing target image data. For example, since the image portion on which a detection target such as another vehicle is projected is determined to some extent in the captured image, only the image data portion corresponding to the image portion is cut out and stored in the memory. However, in this method, when a detection target object is projected in a portion other than the image portion, this cannot be detected, and in this case, the detection accuracy is lowered.

  Further, in the above method, if the data of the image portion excluding only the image portion where the detection target object is never projected is cut out and held in the memory, the situation where the detection target object cannot be detected is reduced. Is done. However, in this case, the effect of reducing the amount of data held in the memory is small. Further, depending on the type and number of detection objects, an image portion on which the detection objects can be projected may be distributed over a wide range on the captured image. In such a case, it is difficult to obtain an effect of reducing the amount of data held in the memory because the image portion on which the detection target object is projected cannot be narrowed down in advance.

  The present invention has been made in view of the above problems, and an object of the present invention is to reduce the memory holding amount of processing target image data used for the detection processing while realizing stable detection accuracy of the detection target. It is to provide an image processing apparatus, a distance measuring apparatus, and an image processing program that can be used.

  In order to achieve the above object, the present invention projects a detection target existing in the imaging area using image data obtained by imaging the imaging area or another image data generated from the image data. A detection object image area specifying unit that executes a process of specifying a detection object image area, and a process of storing unprocessed image data before the detection object image area specifying unit executes the process as processing object image data In the image processing apparatus, the detection target image area specifying unit executes the processing using the processing target image data stored in the processing target image data storage unit. Data amount reduction without using the processing target image data storage means for image data obtained by imaging a region or other image data generated from the image data. A data amount reduction processing unit that executes processing and stores the unprocessed image data obtained thereby in the processing target image data storage unit, and the data amount reduction processing unit captures images by a plurality of imaging units. Disparity histogram information generating means for generating disparity histogram information indicating the frequency distribution of disparity values in each region obtained by dividing a captured image into a plurality of specific directions based on disparity information obtained from a plurality of obtained captured image data And the detected image candidate area including the detected object image area on which the detected object is displayed based on the parallax histogram information, and then the captured image data or image data generated from the captured image data An image obtained by cutting out the image data portion of the detection target candidate region from the image data to be processed before being cut out Characterized in that it comprises a candidate area extracting process means for storing the over data portion to the processing target image data storage means as a processing target image data.

  According to the present invention, it is possible to obtain an excellent effect that it is possible to reduce the memory holding amount of the processing target image data used for the detection processing while realizing stable detection accuracy of the detection target.

It is a mimetic diagram showing a schematic structure of an in-vehicle device control system in an embodiment. It is a hardware block diagram which shows schematic structure of the imaging unit and image analysis unit which comprise the same vehicle equipment control system. It is a functional block diagram in connection with the object detection process of embodiment. It is explanatory drawing which shows the luminance image image | photographed with the left and right cameras, and the parallax image produced | generated from these. It is a figure explaining the parallax when image | photographing with the left and right cameras. It is explanatory drawing which shows an example of the parallax value distribution of a parallax image. It is explanatory drawing which shows the vertical direction parallax histogram (U map) which shows the parallax value frequency distribution for every column area | region of the parallax image shown in FIG. It is a flowchart which shows the flow of the same object detection process. (A) is an image example which shows an example of a reference | standard image (luminance image), (b) is explanatory drawing which shows the outline | summary of the U map corresponding to Fig.9 (a). It is a flowchart which shows the flow of the process in a cut-out position calculation part. It is a flowchart which shows the flow of the mask data generation process which the cutout position calculation part performs about each line (parallax value d) on a U map. (A) is an example of an image which shows an example of a U map schematically, (b) is explanatory drawing which shows the mask data corresponding to Fig.12 (a). It is a flowchart which shows the flow of the image clipping process which an image clipping part performs. (A)-(c) is explanatory drawing which shows an example of the correspondence of mask data, the luminance image data before clipping, and the luminance image data after clipping stored in the memory 130. It is a table | surface which shows the content of the table data corresponding to the example shown in FIG.

Hereinafter, an embodiment in which an image processing device according to the present invention is applied to an object detection device as a distance measurement device used in an in-vehicle device control system as a vehicle system will be described.
Note that the image processing apparatus according to the present invention is not limited to the in-vehicle device control system, and can be applied to, for example, other systems equipped with a distance measuring apparatus that measures a distance to a subject based on a captured image. As specific examples, it can be considered to be used in an emergency braking control system for automobiles, an automatic cruise equipment system, an operation control system for industrial robots, and the like.

FIG. 1 is a schematic diagram illustrating a schematic configuration of an in-vehicle device control system according to the present embodiment.
The in-vehicle device control system of the present embodiment is provided with an image pickup unit 101 as an image pickup unit that picks up an image of a forward area in the traveling direction of the host vehicle 100 as a moving body. For example, the imaging unit 101 is installed in the vicinity of a room mirror (not shown) of the windshield 105 of the host vehicle 100. Various data such as captured image data obtained by imaging by the imaging unit 101 is input to an image analysis unit 102 as image processing means. The image analysis unit 102 analyzes the data transmitted from the imaging unit 101 and calculates, for example, the position, direction, and distance of another vehicle that exists in front of the host vehicle 100. In the detection of other vehicles, a detection object on the road surface is detected as a vehicle based on the parallax information.

  The calculation result of the image analysis unit 102 is also sent to the vehicle travel control unit 106. For example, the vehicle travel control unit 106 may notify the driver of the host vehicle 100 of a warning based on the detection result of the preceding vehicle when the distance between the preceding vehicle and the preceding vehicle becomes narrower or wider. Also, driving support control such as controlling the steering wheel and brake of the host vehicle is performed.

  The calculation result of the image analysis unit 102 is also sent to the headlamp control unit 103. For example, the headlamp control unit 103 performs control such as switching the beam direction of the bed lamp 104 to a high beam or a low beam based on the detection result of the oncoming vehicle.

FIG. 2 is a hardware block diagram illustrating a schematic configuration of the imaging unit 101 and the image analysis unit 102.
The imaging unit 101 is a stereo camera including two imaging units 110A and 110B, and the configuration of the two imaging units 110A and 110B is the same. Each of the imaging units 110A and 110B includes an imaging lens, a sensor substrate including an image sensor in which the imaging elements are two-dimensionally arranged, and an analog electric signal output from the sensor substrate (light reception received by each light receiving element on the image sensor). A signal processing unit that generates and outputs captured image data obtained by converting (quantity) into a digital electrical signal.

  In addition, the imaging unit 101 includes a processing hardware unit 120 including an FPGA (Field-Programmable Gate Array) or the like. The processing hardware unit 120 obtains a parallax image from left and right luminance image data output from the imaging units 110A and 110B, and a corresponding image portion between the left and right captured images respectively captured by the imaging units 110A and 110B. Processing for calculating the parallax value is performed. The parallax value referred to here corresponds to the same point in the imaging region, with one of the captured images captured by the imaging units 110A and 110B (imaging unit 110A) as a reference image and the other (imaging unit 110B) as a comparison image. The positional deviation amount of the image portion on the comparison image with respect to the image portion on the reference image is calculated as the parallax value of the image portion. By using the principle of triangulation, the distance to the same point in the imaging area corresponding to the image portion can be calculated from the parallax value.

  On the other hand, the image analysis unit 102 includes a memory 130 that stores various data output from the imaging unit 101 and a CPU (Central Processing Unit) 140. The CPU 140 executes detection processing of the detection target using various data stored in the memory 130.

Next, object detection processing, which is a characteristic part of the present invention, will be described.
FIG. 3 is a functional block diagram related to the object detection process of the present embodiment.
In the figure, the object detection apparatus of the present embodiment includes a parallax image generation unit 201 that generates parallax image data from left and right captured images (luminance images) acquired from two imaging units 110A and 110B, and a parallax on a parallax image. U map generation unit 202 that generates U map information that is vertical direction parallax histogram information as parallax histogram information from data, U map storage unit 203 that stores the generated U map information, and a luminance image based on the U map information A cut-out position calculation unit 204 that calculates a cut-out position on a pre-cut-out processing target image such as a (captured image) or a parallax image (an image generated from the captured image), and cut-out position information (cut-out mask) that indicates the calculated cut-out position Data) and a cut-out pre-processing target image using the cut-out mask data A part of the image data is cut out from the data, and the image cut-out unit 206 that stores the cut-out image data as processing target image data in the memory 130 (image storage unit); An image processing unit including a CPU 140 and a result output unit 141 that outputs a result of detection processing.

  In the present embodiment, as shown in FIG. 3, the processing target image data cut out after the left and right captured images (luminance images) are input from the two imaging units 110 </ b> A and 110 </ b> B are stored in the memory for speeding up the processing. A series of processing up to storage in 130 is performed by hardware processing by the processing hardware unit 120 made of FPGA or the like without using the memory 130, and detection processing using the processing target image data stored in the memory 130. This processing is performed by software processing using the CPU 140. In other words, in the present embodiment, the pixel value data of the left and right captured images (luminance images) input in a predetermined order from the two imaging units 110A and 110B as the process before the image processing using the CPU 140. A hardware process is executed for a process that can easily execute a sequential process with respect to (position information and luminance value on an image). Even in such sequential processing, it may be necessary to temporarily store the processed data. However, even in this case, the unprocessed data is usually stored sequentially in such a manner that the processed data portion is overwritten. Therefore, the necessary recording area may be a storage area that can store one line of image data or several lines of image data.

  The parallax image generation unit 201 functions as a parallax information generation unit, and as illustrated in FIG. 4, on the reference image using image data of left and right captured images (luminance images) input from the two imaging units 110A and 110B. The parallax value for each pixel is calculated, and parallax image data (parallax information) that can generate a parallax image having the parallax value as a pixel value is generated. Therefore, the parallax image data includes a horizontal position on the parallax image (same as the horizontal position on the reference image) x, a vertical position on the parallax image (same as the vertical position on the reference image) y, 3D coordinate information (x, y, d) having a parallax value d is included. If it demonstrates with reference to FIG. 5, as for the imaging position in the right-and-left image with respect to the O point on the ranging object 301, the distance from an imaging center will be (DELTA) 1 and (DELTA) 2, respectively. The parallax value d at this time can be defined as Δ = Δ1 + Δ2.

  The U map generation unit 202 functions as parallax histogram information generation means, and vertical direction parallax histogram information indicating the frequency distribution of the parallax values d in each column area obtained by dividing a parallax image (captured image) into a plurality of horizontal directions. U map information is generated. In this embodiment, the width of each column region is one pixel. To explain with a specific example, when parallax image data having a parallax value distribution as shown in FIG. 6 is input, the U map generation unit 202 performs the parallax value frequency distribution for each column area as shown in FIG. Is calculated and output.

  The U map information includes three-dimensional coordinates (x, y, d) corresponding to each pixel included in the parallax image data, a horizontal position x on the parallax image on the X axis, a parallax value d on the Y axis, and Z Converted to three-dimensional coordinate information (x, d, f) having the axis f and the frequency f of the parallax value d in the column region corresponding to the image horizontal position x, or this three-dimensional coordinate information (x , D, f) is two-dimensional information (x, d). In this embodiment, U map information composed of three-dimensional coordinate information (x, d, f) is used, and this U map information is stored in the U map storage unit 203. By using such U map information, for example, a two-dimensional coordinate system having a parallax value d on the vertical axis and an image horizontal position x on the horizontal axis has a frequency f exceeding a predetermined frequency threshold. A vertical parallax histogram image (U map) in which pixels corresponding to the parallax value are distributed can be created. Note that U map information including two-dimensional information (x, d) limited to information having a frequency f exceeding a predetermined frequency threshold may be used from the three-dimensional coordinate information (x, y, d).

  The parameter setting unit 207 sets parameters for generating U map information in the U map generation unit 202 and parameters for processing in the cutout position calculation unit 204. As a parameter of the U map generation unit 202, for example, when only an object existing at a distance of 40m to 60m ahead of the host vehicle 100 is a detection target, only a parallax value range corresponding to a distance of 40m to 60m is extracted. For example, a parallax threshold value for the purpose. By limiting the parallax value range in this way, it is possible to reduce the amount of processing when generating U map information. The parameters of the cutout position calculation unit 204 include, for example, a frequency threshold T for a pixel value (frequency f) on the U map, which will be described later, and the number of consecutive appearances of pixels having a frequency f exceeding the frequency threshold T. A lower threshold MIN_W is exemplified.

  The cutout position calculation unit 204 performs processing for specifying a detection target object candidate area including a detection target object image area on which the detection target object is projected, based on the U map information. In the present embodiment, among the parallax values d having the frequency f exceeding the predetermined frequency threshold, the parallax values d within the predetermined parallax value proximity range are supported, and the interval Δl in the horizontal direction of the image is within the predetermined range. A group of pixels is assumed to be a detection object image area that displays one detection object, and includes the detection object image area and an image area that exists in the same image left and right position on the candidate area processing target image. The range is specified as a detection object candidate region. The position information of the detection target object candidate area is sent to the image cutout unit 206 as cutout mask data.

  The image cutout unit 206 uses the cutout mask data sent from the cutout position calculation unit 204 to obtain the image data portion of the detection target candidate region from the image data (luminance image data and parallax image data) of the entire captured image. Process to cut out. Then, the image data cut out by this processing is stored in the image storage unit (memory 130).

  The CPU 140 serving as an image processing unit functions as a detection target image region specifying unit, and uses the image data portion of the detection target candidate region stored in the image storage unit (memory 130) as processing target image data. Object detection processing is performed. This processing result is output from the result output unit 141.

Next, the flow of object detection processing in this embodiment will be described.
FIG. 8 is a flowchart showing the flow of object detection processing in the present embodiment.
First, when input of left and right luminance image data is received from each of the imaging units 110A and 110B of the imaging unit 101 (S1), the parallax image generation unit 201 first uses the input left and right luminance image data to generate a reference image. A parallax value for each of the upper pixels is calculated, and parallax image data capable of generating a parallax image having the parallax value as a pixel value is generated (S2).

  As the parallax image data generation processing, for example, the following processing can be employed. First, for a certain row of reference image data, a block composed of a plurality of pixels (for example, 5 pixels × 5 pixels) centering on one target pixel is defined. On the other hand, in the same row as the target pixel in the comparison image data, a block having the same size as the block of the defined reference image data is shifted by one pixel in the horizontal line direction (image left-right direction), and the pixel of the block defined in the reference image data Correlation values indicating the correlation between the feature amount indicating the feature of the value and the feature amount indicating the feature of the pixel value of each block in the comparison image data are respectively calculated. Then, based on the calculated correlation value, a matching process is performed for selecting a block of comparison image data that is most correlated with a block of reference image data among the blocks of the comparison image data. Thereafter, a positional deviation amount between the target pixel of the block of the reference image data and the corresponding pixel of the block of the comparison image data selected by the matching process is calculated as the parallax value d. The parallax image data can be obtained by performing such processing for calculating the parallax value d for the entire area of the reference image data or a specific area.

  As the feature amount of the block used for the matching process, for example, the value (luminance value) of each pixel in the block can be used, and as the correlation value, for example, the value (luminance) of each pixel in the block of the reference image data Value) and the sum of absolute values of the differences between the values (luminance values) of the pixels in the block of comparison image data corresponding to these pixels, respectively. In this case, it can be said that the block having the smallest sum is most correlated.

  The parallax image data generated in this way is sent to the U map generation unit 202. The U map generation unit 202 performs processing for converting the parallax image data (x, y, d) into U map information (x, d, f) to generate U map information (S3). To explain using the U map explanatory diagram shown in FIG. 7, for example, the group of pixels having the parallax value d = 4 on the parallax image data shown in FIG. 6 exists at the same distance from the vehicle 100. This is a pixel that shows the object to be played. Since these pixels (pixels having the same parallax value) are arranged at the same vertical position on the U map shown in FIG. 7, a pixel displaying one object having a certain width among these pixels. About, it shows the feature that it is distributed in the shape of a horizontal line on the U map. Further, among these pixels, a pixel showing one object having a certain height shows a certain frequency or more on the U map. Therefore, a group of pixels having a certain frequency and distributed in a horizontal line on the U map is said to be a pixel group showing one object having a certain width and height. be able to.

  In the present embodiment, using this feature, a detection target object candidate region including a detection target image region that projects each detection target in the imaging region (real space) is cut out (S4, S5), A detection object image area is specified from the detection object candidate area (S6). Then, the position information (position information on the luminance image or the parallax image) of the detection object image area specified in this way is output as a processing result (S7).

  Note that, by simply converting the parallax image data (x, y, d) into the U map information (x, d, f), the U map as shown in FIG. I can't get it. Therefore, in practice, a U map as shown in FIG. 9B is obtained using a smoothing process, a Hough transform process, or the like. Here, the smoothing process is a process of filling discontinuous portions by filling in the space between pixels (pixels having a certain frequency) within a predetermined range (within a predetermined number of pixels). is there.

FIG. 10 is a flowchart showing the flow of processing in the cutout position calculation unit 204.
In the present embodiment, mask data generation processing (S20) is executed for each row (disparity value d) on the U map (S11 to S13). When the mask data generation processing is completed for all rows (parallax value d), the cutout mask data generated thereby is stored in the cutout position storage unit 205 (S14). The clipping mask data generated here is information indicating the position in the left-right direction of the detection target candidate region cut out from the image data (luminance image data and parallax image data) of the captured image. The cutout mask data is used as information indicating which data portion is cut out or not cut out from the image data (luminance image data and parallax image data) of the captured image in the image cutout unit 206.

  Specifically, the clipping mask data of the present embodiment is composed of data having a size (number of elements) for one line of an image. For example, when the overall image size is 1280 pixels (left and right direction, X direction) × 960 pixels (up and down direction, Y direction), the size (number of elements) of the clipping mask data is 1280. Each element of the mask data has a value of either 0 or 1, and if it is 1, the pixel belonging to the image row (row extending in the vertical direction of the image) at the element position (X coordinate) is a target to be cut out. If 0, the pixel belonging to the image sequence at the element position (X coordinate) is not cut out.

FIG. 11 is a flowchart showing a flow of mask data generation processing (S20) performed by the cutout position calculation unit 204 for each row (parallax value d) on the U map.
In the following description, the frequency f (pixel value on the U map) of the pixel located at the horizontal position (X coordinate) x on the U map in the row is U (x). In addition, the count value of the number of consecutive appearances of a pixel having a pixel value U (x) greater than a predetermined frequency threshold T is W, and the X coordinate that is the starting point of the consecutive pixels is Sx. WIDTH is the size of the U map in the left-right direction (the number of pixels in the X direction), and is 1280 in this embodiment. Moreover, MIN_W is a lower limit threshold value for the number of consecutive appearances of pixels having a pixel value U (x) exceeding the threshold value T, and is 10 in this embodiment.

  In the mask data generation process in the present embodiment, first, the X coordinate position x, the count value W of the continuous appearance number, and the start point coordinate Sx of the continuous appearance number of the pixel to be processed are reset to 0, which is an initial value (S21). Thereafter, data of the pixel value U (x) of the pixel whose X coordinate is x is input from the U map storage unit 203 (S22), and whether or not the pixel value U (x) is larger than the frequency threshold value T is determined. Judgment is made (S23). In this determination, when it is determined that the pixel value U (x) is larger than the frequency threshold T (Yes in S23), first, it is confirmed whether the count value W of the continuous appearance number is 0 (S24). At this time, if W = 0 (Yes in S24), after setting the starting point coordinates Sx = x of the continuous appearance number (S25), W = 1 is set (S26). If W is not 0 (No in S24), a process of adding 1 to the count value W of the current number of consecutive appearances is performed (S26). Thereafter, after adding 1 to x (S28), the process returns to the data input (S22) of the next pixel value U (x).

  If the pixel value U (x) input from the U map storage unit 203 is smaller than the frequency threshold T (No in S23), is the current continuous appearance count value W greater than the lower limit threshold MIN_W of the continuous appearance count? It is determined whether or not (S29). At this time, if W> MIN_W is not satisfied (No in S29), after resetting the count value W of the continuous appearance number and the start point coordinate Sx of the continuous appearance number to the initial value 0 (S31), a process of adding 1 to x is performed. Perform (S28), and return to the data input (S22) of the next pixel value U (x). On the other hand, if W> MIN_W (Yes in S29), the value of the mask data in the range from the X coordinate position indicated by the start point coordinate Sx of the consecutive appearance number to the X coordinate position indicated by (Sx + W-1) is set to 1. Processing is performed (S30). After that, the count value W of the continuous appearance number and the start point coordinate Sx of the continuous appearance number are reset to the initial value 0 (S31), and 1 is added to x (S28), and the next pixel value U (x ) To return to data input (S22).

  The above processing is repeated until x = (WIDTH-1) is reached (S27). When x = (WIDTH-1) is reached (Yes in S27), the mask data generation processing for the row is terminated. When such mask data generation processing has been completed up to the last row, in any row, a group of pixels in which the pixel value U (x) exceeding the threshold value T exceeds the lower limit threshold value MIN_W of the number of consecutive appearances. Mask data in which the value corresponding to the existing X coordinate range is 1 is generated. An example of the correspondence between the clipping mask data generated in this way and the U map is as shown in FIG.

Next, a specific example of the process of cutting out the image data portion of the detection target candidate region from the image data (luminance image data and parallax image data) of the captured image using the cut-out mask data will be described.
In the following description, the case where the image data portion of the detection target candidate region is cut out from the luminance image data of the reference image will be described. However, the image of the detection target candidate region from the parallax image data generated by the parallax image generation unit 201. The same applies to the case where the data portion is cut out.

  In the following description, i is a pixel number on the luminance image data, and starts from the pixel located at the left end of the uppermost row of the image, proceeds toward the right side of the image, and reaches the right end of the image when it reaches the right end of the next row. By moving, it is indicated by a sequential number whose end point is the pixel located at the right end of the bottom row of the image, and its initial value is zero. Therefore, the final value of the pixel number i is (PXEL-1) when the number of pixels of the luminance image data is PXEL. In the present embodiment, since the image size of the luminance image data is 1280 pixels × 960 pixels, PXEL = 1280 × 960. D (i) is pixel value data of pixel number i in the luminance image data. X is the X coordinate position on the luminance image data, and is the same as the X coordinate position on the U map. MSK (x) is the value of the mask data at the X coordinate position x. Further, m is a relative address from the write start position (memory address) of the memory 130 (image storage unit) that stores the cut-out image data. MEM (m) is a memory area having a relative address m. W is a count value of consecutive numbers in which the value of the mask data MSK (x) is 1. Further, k is an index number that associates the luminance image data SRC (k) before clipping with the luminance image data DST (k) after clipping. Q is a discriminator for discriminating whether or not it is the first row.

FIG. 13 is a flowchart showing a flow of image cutout processing performed by the image cutout unit 206.
In the image cutout process in the present embodiment, first, the pixel number i, X coordinate position x, relative address m, continuous count value W, and index number k of the pixel to be processed are reset to 0, which is an initial value (S41). Thereafter, a mask data value MSK (x) whose X coordinate is x is input from the cut-out position storage unit 205 (S42), and it is determined whether or not the MSK (x) is 1 (S43). At this time, if MSK (x) = 1 (Yes in S43), it is first determined whether or not the continuous count value W is 0 and the discriminator Q is 0 (S44). In this determination, if W = 0 and Q = 0 (Yes in S44), the X coordinate position SRC (k) .. of the cutout start position on the luminance image data SRC (k) before cutout. The current x value is written to Sx, and the luminance image data DST (k) after extraction is written to the memory 130 at the writing start position DST (k). The current value of m is written into Sx (S45). Thereafter, W = 1 is set (S46), and D (i) is written in the memory area MEM (m) of the memory 130 (S48). Thereafter, after adding 1 to the relative address m, the pixel number i, and the X coordinate position x (S48 to S50), the process returns to the data input (S42) of the next mask data value MSK (x).

  Thereafter, while the mask data value MSK (x) to be input is 1 (Yes in S43), D (i) is sequentially written in the memory area MEM (m) of the memory 130 (S44 to S50). . When the value MSK (x) of the input mask data becomes 0 (No in S43), it is determined whether or not the continuous count value W is 1 or more and the discriminator Q is 0 (S52). . In this determination, if W ≧ 1 and Q = 0 (Yes in S52), the X coordinate position SRC (k) .. of the cutout end position on the luminance image data SRC (k) before cutout. The current value of x is written to Ex, and the luminance image data DST (k) after extraction is written to the memory 130 at the write end position DST (k). The current value of m is written to Ex (S53). Thereby, the cutout of the first (k = 0) image data portion is completed.

  Thereafter, a process of adding 1 to the index number k is performed (S54). When W = 0 is set (S55), a process of adding 1 to the pixel number i and the X coordinate position x is performed (S49, S50). ), The process returns to the data input (S42) of the next mask data value MSK (x). Then, while the input mask data value MSK (x) is 0 (No in S43), D (i) is not written in the memory area MEM (m) of the memory 130.

  The above processing is repeated until x = (WIDTH) is reached (S51), and when x = (WIDTH) is reached (Yes in S51), the image data for the first row is cut out. Thereafter, the X coordinate position x is reset to 0, and 1 is set to the discriminator Q (S57). As a result, the image data is cut out again from x = 0 for the next row. At this time, by setting 1 to the discriminator Q, the X coordinate position SRC (k). Sx and memory write start position SRC (k). Sx and the X coordinate position SRC (k). Ex and memory write end position SRC (k). Ex is not written (S44, S52). This is because these data have already been acquired in the first row.

  In this manner, image data is sequentially cut out for each row, and when the pixel number i reaches the final value PXEL (Yes in S56), the image cut-out process is terminated.

FIGS. 14A to 14C are explanatory diagrams illustrating an example of a correspondence relationship between mask data, luminance image data before clipping, and luminance image data after clipping stored in the memory 130. FIG.
As can be seen from the comparison between FIG. 14B and FIG. 14C, the luminance image data after the cut-out is not cut out than the luminance image data before the cut-out (the image data of the entire captured image). The amount of data is small by the amount of data. The luminance image data after the clipping is stored in the memory 130 as processing target image data used for detection target object detection processing performed by the CPU 140 serving as an image processing unit. Therefore, compared with the case where the luminance image data before clipping (image data of the entire captured image) is stored as the processing target image data in the memory 130, the memory holding amount of the processing target image data can be reduced.

  X coordinate position SRC (k). Sx and memory write start position SRC (k). Sx and the X coordinate position SRC (k). Ex and memory write end position SRC (k). Ex is stored in the cut-out position storage unit 205 as table data. The contents of the table data corresponding to the example shown in FIG. 14 are, for example, as shown in FIG. This table data is called up and used by the CPU 140 when the CPU 140 as the image processing unit performs detection processing of the detection target. By referring to the table data, the CPU 140 can associate which image data in which memory area is the image data at which position on the captured image (luminance image or parallax image).

  In the present embodiment, the image data portion of the detection target candidate area including the detection target image area specified based on the U map information obtained from the captured image is cut out and stored in the memory 130. Therefore, it is possible to select and cut out only an image area where there is a high possibility that the detection target object is projected. Although the situation of the imaging area changes every moment, according to the present embodiment, even if the situation of the imaging area changes every moment, stable detection accuracy can be realized, and the memory of the processing target image data by clipping A holding amount reducing effect can also be obtained.

  As a method of reducing the memory holding amount of the processing target image data, a method of cutting out only a predetermined portion from the entire image data can be considered. However, as described above, in this method, other than the image portion is considered. When a detection object is projected on the screen, this cannot be detected, and stable detection accuracy cannot be realized. Further, if the range of image data to be cut out is increased in order to obtain stable detection accuracy, the effect of reducing the memory retention amount of the processing target image data cannot be sufficiently obtained. Compared with this method, the present embodiment can sufficiently achieve the effect of reducing the memory retention amount of the processing target image data while realizing stable detection accuracy.

  As described above, when the cut-out image data is stored in the memory 130 as the processing target image data, the CPU 140 executes a detection target object detection process using the processing target image data. In this detection processing, for example, parallax image data cut out in the memory 130 is used as processing target image data, a group of pixels having a parallax value within a predetermined approximate range (isolated region) is specified, and each isolated region is identified. The image width and image height are calculated. Thereafter, an average value (parallax average value) of pixel values on the parallax image in each isolated area is calculated, and the isolated area is calculated from the distance calculated from the average value and the image width and image height of the isolated area. Calculate the actual size (actual width and actual height) of the object shown on the screen. When the actual size of the object corresponding to each isolated region is calculated in this way, the type of the detection target corresponding to the actual size is set in advance using table information (correspondence between the type of detection target and the actual size). Based on the information).

  In addition to the detection processing using such parallax image data, the CPU 140 also performs detection processing using, for example, luminance image data cut out in the memory 130 as processing target image data. For example, processing is performed such as extracting an edge portion having a large luminance change from the luminance image data cut out in the memory 130 and specifying an object image region on which one object is projected. By combining the result of the detection process using the luminance image data and the result of the detection process using the parallax image data, information such as the type, position, and distance of the detection target displayed on the captured image is detected.

What has been described above is merely an example, and the present invention has a specific effect for each of the following modes.
(Aspect A)
A detection target for projecting a detection target existing in the imaging area using image data such as luminance image data obtained by imaging the imaging area or other image data such as parallax image data generated from the image data A detection object image area specifying unit such as a CPU (image processing unit) 140 that executes a process for specifying an object image area, and unprocessed image data before the detection object image area specifying unit executes the process Processing target image data storage means such as a memory 130 (image storage unit) that stores the target image data, and the detection target image area specifying means is a processing target stored in the processing target image data storage means. In an image processing apparatus that executes the processing using image data, captured image data obtained by imaging an imaging region or an image generated from the captured image data An image of the processing hardware unit 120 or the like that performs preprocessing on the data without using the processing target image data storage unit and stores the unprocessed image data obtained thereby in the processing target image data storage unit Data pre-processing means, the image data pre-processing means imaging based on parallax information obtained from a plurality of captured image data obtained by imaging by a plurality of imaging means such as two imaging units 110A, 110B A parallax image generation unit 201, a U map generation unit 202, and the like that generate parallax histogram information such as U map information indicating the frequency distribution of parallax values in each region obtained by dividing an image into a plurality of specific directions (image left and right directions) And a detection object image area in which the detection object is projected based on the parallax histogram information. After specifying the target object candidate area, the image data portion of the detection target candidate area is cut out from the picked-up image data or the pre-cut-out processing target image data that is image data generated from the picked-up image data, and cut out And a candidate area cutout processing unit such as a cutout position calculation unit 204 and an image cutout unit 206 that store the data portion as processing target image data in the processing target image data storage unit.
In this aspect, the image data pre-processing means cuts out the image data portion of the detection target candidate area from the pre-cut processing target image data, and stores the cut-out image data portion as the processing target image data in the processing target image data storage means. . Therefore, the memory retention amount of unprocessed image data (processing target image data) held in the processing target image data storage unit before performing the processing executed by the detection target image area specifying unit is the pre-cutting processing target. This is less than when image data is held as processing target image data.
Here, in this aspect, after specifying the detection target object candidate area including the detection target image area based on the parallax histogram information, the image data portion of the detection target candidate area is cut out from the pre-cut processing target image data. If the parallax histogram information is used, it is possible to quickly and accurately specify a candidate area of the detection target image area that projects one object. This point is described in Japanese Patent Application No. 2013-44434 (hereinafter referred to as “prior application”) proposed by the present applicant. In summary, in the apparatus proposed in the prior application, the disparity values within a predetermined disparity value proximity range among disparity values having a frequency exceeding a predetermined frequency threshold in each column area obtained by dividing a captured image into a plurality of left and right directions. A group of pixels corresponding to the parallax value and satisfying the condition that the interval in the horizontal direction of the image is within a predetermined range is extracted. A pixel corresponding to a parallax value having a frequency exceeding a predetermined frequency threshold indicates a pixel having a relatively large number of pixels corresponding to the same parallax value in the column region. In addition, the collection of pixels corresponding to the parallax values within a predetermined parallax value proximity range is a distance (distance from the imaging unit) at which each location displayed on those pixels corresponds to the parallax value proximity range. Is a group of pixels that are close to each other within the range of, but in other words, a group of pixels that project a part having the same distance from the imaging means. Accordingly, among the disparity values having a frequency exceeding the predetermined frequency threshold in each column area, the distance from the imaging unit is about the same as the collection of pixels corresponding to the disparity values within the predetermined disparity value proximity range. In addition, it can be said that it is a collection of pixels displaying a portion having a certain height. Further, when the condition that the image horizontal spacing is within a predetermined range is added, such a group of pixels is a surface having the same distance from the imaging means and a constant height and a constant width. It can be said that this is a group of pixels (a group of pixels) displaying the image. Since such a surface can be considered as a surface on one object, by extracting a group of pixels as described above, individual objects existing in the imaging region can be distinguished and detected with high accuracy. can do. The parallax histogram information used in the prior application is vertical direction parallax histogram information (U map information) as in the case of the present embodiment. For example, each parallax histogram information obtained by dividing a captured image into a plurality of parts in the vertical direction is provided. The horizontal direction parallax histogram information (V map information) indicating the frequency distribution of the parallax values in the horizontal region can be considered in the same manner as the vertical direction parallax histogram information (U map information) by switching the vertical and horizontal directions of the image.
According to this aspect, the detection target candidate region including the detection target image region that can be identified with high accuracy from the parallax histogram information is cut out as described above, and the image data portion is used as unprocessed image data (processing target image data). Saved in the processing target image data storage means. Since it is possible to appropriately cut out and save the image area that shows the detection target existing in the imaging area where the situation changes every moment, the situation where the image area that shows the detection target existing in the imaging area is not cut out There are few, and stable detection accuracy is realizable. In addition, the image data of the image area where the detection target object is not projected is unnecessarily stored in the processing target image data storage means as processing target image data (unprocessed image data) processed by the detection target image area specifying means. The situation of preservation is also reduced. Therefore, it is possible to obtain an effect of reducing the memory retention amount of unprocessed image data (processing target image data) held in the processing target image data storage unit before performing the processing executed by the detection target image area specifying unit. It is done.

(Aspect B)
In the aspect A, the disparity histogram information generated by the disparity histogram information generating unit is a vertical direction such as U map information indicating the frequency distribution of the disparity values in each column area obtained by dividing a captured image into a plurality of left and right directions. It is parallax histogram information.
According to this, it becomes possible to quickly specify the candidate area of the detection target object image area in which one object is projected based on the parallax histogram information.

(Aspect C)
In the aspect A or B, the parallax histogram information generation unit generates the parallax histogram information for a part of the captured image in a direction orthogonal to the specific direction.
According to this, parallax histogram information can be generated by narrowing down to an image portion excluding an image portion where a detection target object cannot exist, and thus it is possible to reduce the load of the parallax histogram information generation processing.

(Aspect D)
In any one of the above aspects A to C, the candidate region cutout processing unit is configured to select a disparity within a predetermined disparity value proximity range among disparity values having a frequency exceeding a predetermined frequency threshold based on the disparity histogram information. A group of pixels corresponding to a value and having an interval in the specific direction within a predetermined range is set as the detection target image region, and a detection target candidate region including the detection target image region is specified. It is characterized by.
According to this, it becomes possible to specify with high accuracy the candidate area of the detection target image area that projects one object based on the parallax histogram information.

(Aspect E)
In any one of the above aspects A to D, the candidate area cutout processing means is a cutout mask data MSK that indicates the position of the detection target candidate area on the precutout process target image based on the parallax histogram information. (X) or the like is generated, and the image data portion of the detection target object candidate region is extracted from the pre-cut processing target image data according to the generated candidate region position information.
According to this, the image data of the specified detection target candidate region can be extracted from the pre-processing target image data by simple processing.

(Aspect F)
In any one of the above aspects A to E, the candidate area cutout processing unit is present at the same position in the specific direction as the detection target image area and the detection target image area on the precutting target image. A detection object candidate area including an image area is specified.
According to this, it is possible to quickly extract the image data of the detection target candidate region specified based on the parallax histogram information from the pre-processing target image data.

(Aspect G)
In any one of the above aspects A to F, the candidate area cut-out processing unit is configured to store a data storage position DST (k). Sx, DST (k). Ex, and the position SRC (k) .K on the pre-cut processing target image of the detection target object candidate area corresponding to the post-cut processing target image data. Sx, SRC (k). Correspondence information such as table data for associating with Ex is generated.
According to this, when the detection object image area specifying unit performs the process of specifying the detection object image area, by referring to this correspondence information, which storage area in the processing object image data storage unit is held. It becomes easy to associate which position the image data is on the captured image.

(Aspect H)
In the above aspect G, the detection object image area specifying unit extracts a detection object image data portion corresponding to the detection object image area from the processing object image data stored in the processing object image data storage unit. Based on the correspondence information, position information indicating a position of the detection target image area corresponding to the extracted detection target image data portion on the pre-cut processing target image is output.
According to this, it becomes possible to perform various processes specifying the position on the captured image on which the detection target object is projected, using this position information.

(Aspect I)
An image for specifying a detection object image area in which a detection object existing in the imaging area is projected using a plurality of imaging means for imaging the imaging area and captured image data obtained by imaging with the imaging means Image processing means for performing processing, and a plurality of captured image data obtained by imaging the distance to the detection target object displayed in the detection target image area specified by the image processing means by the plurality of imaging means A distance measuring device having a distance calculating unit that calculates based on parallax information obtained from the image processing unit, wherein the image processing device according to any one of claims 1 to 8 is used as the image processing unit. To do.
According to this, it is possible to detect the detection target with stable detection accuracy while calculating the memory holding amount of the processing target image data, and calculate the distance of the detection target.

(Aspect J)
Using the image data obtained by imaging the imaging region or another image data generated from the image data, a process of specifying a detection target image region that displays the detection target existing in the imaging region is executed. A detection target image region specifying unit; and a processing target image data storage unit that stores unprocessed image data before the detection target image region specifying unit executes the processing as processing target image data. In the image processing program for causing the computer of the image processing apparatus to execute the processing using the processing target image data stored in the processing target image data storage unit, the target image region specifying unit is an imaging region. The processing target image data storage means is used for image data obtained by imaging an image or another image data generated from the image data. The computer is caused to function as a data amount reduction processing unit that executes the data amount reduction processing and stores the unprocessed image data obtained thereby in the processing target image data storage unit. The processing means calculates the frequency distribution of the parallax values in each region obtained by dividing the captured image into a plurality of specific directions based on the parallax information obtained from the plurality of captured image data obtained by imaging with the plurality of imaging means. Parallax histogram information generating means for generating parallax histogram information to be shown, and after detecting a detection target object candidate area including a detection target object image area on which the detection target object is projected based on the parallax histogram information, the captured image Detection from pre-cutting target image data that is data or image data generated from the captured image data Cut out image data portion of the elephant candidate region, characterized in that it comprises a candidate area extracting process means the image data portion as a processing target image data is stored in the processing target image data storage means cut.
According to this, it is possible to reduce the memory holding amount of the processing target image data used for the detection processing while realizing stable detection accuracy of the detection target.
This program can be distributed or obtained in a state of being recorded on a recording medium such as a CD-ROM. It is also possible to distribute and obtain signals by placing this program and distributing or receiving signals transmitted by a predetermined transmission device via transmission media such as public telephone lines, dedicated lines, and other communication networks. Is possible. At the time of distribution, it is sufficient that at least a part of the computer program is transmitted in the transmission medium. That is, it is not necessary for all data constituting the computer program to exist on the transmission medium at one time. The signal carrying the program is a computer data signal embodied on a predetermined carrier wave including the computer program. Further, the transmission method for transmitting a computer program from a predetermined transmission device includes a case where data constituting the program is transmitted continuously and a case where it is transmitted intermittently.

DESCRIPTION OF SYMBOLS 100 Own vehicle 101 Imaging unit 102 Image analysis unit 103 Headlamp control unit 106 Vehicle travel control unit 110A, 110B Imaging part 120 Processing hardware part 130 Memory 140 CPU
201 parallax image generation unit 202 U map generation unit 203 U map storage unit 204 cutout position calculation unit 205 cutout position storage unit 206 image cutout unit 207 parameter setting unit

JP 2012-226689 A

Claims (10)

Using the image data obtained by imaging the imaging region or another image data generated from the image data, a process of specifying a detection target image region that displays the detection target existing in the imaging region is executed. A detection object image region specifying means;
Processing target image data storage means for storing unprocessed image data before the detection target image area specifying means executes the processing as processing target image data;
In the image processing apparatus for executing the processing using the processing target image data stored in the processing target image data storage unit, the detection target image area specifying unit is
Data amount reduction processing is executed without using the processing target image data storage means on image data obtained by imaging the imaging region or other image data generated from the image data, and Data amount reduction processing means for storing processed image data in the processing target image data storage means;
The data amount reduction processing means is a parallax value in each region obtained by dividing a captured image into a plurality of specific directions based on parallax information obtained from a plurality of captured image data obtained by imaging with a plurality of imaging means. Parallax histogram information generating means for generating parallax histogram information indicating the frequency distribution of
After the detection object candidate area including the detection object image area in which the detection object is projected based on the parallax histogram information is specified, the captured image data or image data generated from the captured image data. A candidate region cutout processing unit that cuts out the image data portion of the detection target candidate region from the precutting processing target image data and stores the cut out image data portion in the processing target image data storage unit as processing target image data. An image processing apparatus. The image processing apparatus according to claim 1.
The parallax histogram information generated by the parallax histogram information generating unit is vertical parallax histogram information indicating a frequency distribution of parallax values in each column area obtained by dividing a captured image into a plurality of horizontal directions. Image processing device. The image processing apparatus according to claim 1 or 2,
The parallax histogram information generation means generates the parallax histogram information for a part of the captured image in a direction orthogonal to the specific direction. The image processing apparatus according to any one of claims 1 to 3,
The candidate area cut-out processing unit corresponds to a disparity value within a predetermined disparity value proximity range among disparity values having a frequency exceeding a predetermined frequency threshold based on the disparity histogram information, and the specific direction An image processing apparatus characterized in that a group of pixels whose intervals are within a predetermined range is set as the detection object image area, and a detection object candidate area including the detection object image area is specified. The image processing apparatus according to any one of claims 1 to 4,
The candidate area cut-out processing unit generates candidate area position information indicating the position of the detection target object candidate area on the pre-cut-out processing target image based on the parallax histogram information, and before cutting out according to the generated candidate area position information. An image processing apparatus that extracts an image data portion of a detection target object candidate region from processing target image data. The image processing apparatus according to any one of claims 1 to 5,
The candidate region cutout processing unit specifies a detection target candidate region including the detection target image region and an image region existing at the same position in the specific direction as the detection target image region on the precut processing target image. An image processing apparatus. The image processing apparatus according to any one of claims 1 to 6,
The candidate area cut-out processing means includes a data storage position of post-cut processing target image data in the processing target image data storage means, and a pre-cut processing target image of a detection target candidate area corresponding to the post-cut processing target image data An image processing apparatus that generates correspondence information that correlates an upper position. The image processing apparatus according to claim 7.
The detection object image area specifying means extracts a detection object image data portion corresponding to the detection object image area from the processing object image data stored in the processing object image data storage means, and An image processing apparatus that outputs position information indicating a position on a pre-cut processing target image of a detection target image area corresponding to an extracted detection target image data portion. A plurality of imaging means for imaging the imaging region;
Image processing means for performing image processing for specifying a detection object image area that projects a detection object existing in the imaging area, using captured image data obtained by imaging by the imaging means;
Based on the parallax information obtained from the plurality of captured image data obtained by imaging the distance to the detection target displayed in the detection target image area specified by the image processing unit by the plurality of imaging units. In a distance measuring device having a distance calculating means for calculating,
9. A distance measuring apparatus using the image processing apparatus according to claim 1 as the image processing means. Using the image data obtained by imaging the imaging region or another image data generated from the image data, a process of specifying a detection target image region that displays the detection target existing in the imaging region is executed. A detection object image region specifying means;
Processing target image data storage means for storing unprocessed image data before the detection target image area specifying means executes the processing as processing target image data;
In the image processing program for causing the computer of the image processing apparatus to execute the processing using the processing target image data stored in the processing target image data storage unit, the detection target object image area specifying unit,
Data amount reduction processing is executed without using the processing target image data storage means on image data obtained by imaging the imaging region or other image data generated from the image data, and The computer is caused to function as a data amount reduction processing unit for storing processed image data in the processing target image data storage unit,
The data amount reduction processing means is a parallax value in each region obtained by dividing a captured image into a plurality of specific directions based on parallax information obtained from a plurality of captured image data obtained by imaging with a plurality of imaging means. Parallax histogram information generating means for generating parallax histogram information indicating the frequency distribution of
After the detection object candidate area including the detection object image area in which the detection object is projected based on the parallax histogram information is specified, the captured image data or image data generated from the captured image data. A candidate region cutout processing unit that cuts out the image data portion of the detection target candidate region from the precutting processing target image data and stores the cut out image data portion in the processing target image data storage unit as processing target image data. An image processing program characterized by the above.