JP2018088234A - Information processing device, imaging device, apparatus control system, movable body, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device that can continue accurate tracking.SOLUTION: An information processing device (apparatus control system 1) comprises: an acquisition part that acquires information with which the position in the vertical direction, position in the horizontal direction, and position in the depth direction of an object are associated; a judging part 142 that judges the type of the object on the basis of the information acquired by the acquisition part; and a determination part 143 that determines the position of the object with a method according to the type of the object judged by the judging part; and a tracking part 144 that tracks the object on the basis of the position of the object determined by the determination part.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an information processing device, an imaging device, a device control system, a moving body, an information processing method, and a program.

  Conventionally, in the safety of automobiles, body structures of automobiles have been developed from the viewpoint of how to protect pedestrians and protect passengers when they collide with pedestrians and automobiles. However, in recent years, with the development of information processing technology and image processing technology, technology for detecting people, cars, etc. at high speed has been developed. Automobiles that apply these technologies to automatically apply a brake before a collision to prevent the collision are already on the market.

  In order to automatically apply a brake, it is necessary to measure the distance to an object such as a person or another vehicle. For this reason, measurement using an image of a stereo camera has been put into practical use.

  In the measurement using the image of the stereo camera, an object such as a vehicle ahead of the host vehicle is detected from the parallax image of a certain frame, and then the object is tracked (tracked) in the parallax images of the subsequent frames. A technique is known (see, for example, Patent Document 1).

  However, in the prior art, for example, when a person suddenly spreads his / her hand, the position of an object such as a person detected from the previous frame and the person who has spread his / her hand detected from the current frame The difference from the position of the object becomes relatively large. In this case, it may become impossible to track the same object.

  Therefore, an object is to provide a technique capable of continuing highly accurate tracking.

  In the information processing apparatus, based on the information acquired by the acquisition unit, the acquisition unit that acquires information in which the vertical position, the horizontal position, and the depth direction position of the object are associated with each other. Based on the determination unit that determines the type of the object, the determination unit that determines the position of the object by a method according to the type of the object determined by the determination unit, and the position of the object determined by the determination unit And a tracking unit that tracks the object.

  According to the disclosed technique, tracking with high accuracy can be continued.

It is a figure which shows the structure of the apparatus control system which concerns on embodiment. It is a figure which shows the structure of the imaging unit and image analysis unit which concern on embodiment. It is a figure for demonstrating the principle which calculates a distance from a parallax value by utilizing the principle of triangulation. It is a figure which shows an example of the functional block diagram of an apparatus control system. It is a figure for demonstrating the parallax image data and the V map produced | generated from the parallax image data. It is a figure which shows the image example of the picked-up image as a reference | standard image imaged with one imaging part, and the V map corresponding to the picked-up image. It is a figure which shows the example of an image which represented typically an example of the reference | standard image. It is a figure which shows U map corresponding to an example of an image. It is a figure which shows the real U map corresponding to U map. It is a figure for demonstrating the method of calculating | requiring the value of the horizontal axis of a real U map from the value of the horizontal axis of a U map. It is a flowchart which shows an example of an isolated area | region detection process. It is a figure which shows the real frequency U map which set the rectangular area | region which the isolated area detected by the isolated area detection part inscribed. It is a figure which shows the parallax image which set the scanning range corresponding to a rectangular area. It is a figure which shows the parallax image which searched the scanning range and set the object area | region. It is a flowchart which shows the flow of the process performed by the corresponding area detection part and object area extraction part of a parallax image. It is a figure which shows an example of the table data for classifying an object type. It is a figure which shows an example of a three-dimensional position determination process. It is a figure explaining the method of calculating the position of objects, such as a vehicle. It is a figure explaining the method to calculate the position of the object which is a pedestrian, a motorcycle, or a bicycle.

  Hereinafter, a device control system having the image processing apparatus according to the embodiment will be described.

<Configuration of device control system>
FIG. 1 is a diagram illustrating a configuration of a device control system according to the embodiment.

  The device control system 1 is mounted on a host vehicle 100 such as an automobile that is a moving body, and includes an imaging unit 101, an image analysis unit 102, a display monitor 103, and a vehicle travel control unit 104. Then, the imaging unit 101 detects and tracks an object in front of the host vehicle from a plurality of captured image data (frames) in the forward direction of the host vehicle (imaging region) in which the front of the moving body is captured, and the tracking result Is used to control mobile objects and various in-vehicle devices. The control of the moving body includes, for example, warning notification, control of the handle of the own vehicle 100 (own moving body), or braking of the own vehicle 100 (own moving body).

  The imaging unit 101 is installed near a room mirror (not shown) of the windshield 105 of the host vehicle 100, for example. Various data such as captured image data obtained by imaging by the imaging unit 101 is input to an image analysis unit 102 as image processing means.

  The image analysis unit 102 analyzes the data transmitted from the imaging unit 101, and is on the traveling road surface in front of the own vehicle with respect to the road surface portion on which the own vehicle 100 is traveling (the road surface portion located directly below the own vehicle). The relative height (position information) at each point is detected, and the three-dimensional shape of the traveling road surface in front of the host vehicle is grasped. Also, recognition objects such as other vehicles in front of the host vehicle, pedestrians, and various obstacles are recognized.

  The analysis result of the image analysis unit 102 is sent to the display monitor 103 and the vehicle travel control unit 104. The display monitor 103 displays captured image data and analysis results obtained by the imaging unit 101. The display monitor 103 may not be provided. For example, the vehicle travel control unit 104 notifies the driver of the host vehicle 100 of a warning based on the recognition result of the recognition target objects such as other vehicles in front of the host vehicle, pedestrians, and various obstacles by the image analysis unit 102. Or driving support control such as controlling the steering wheel and brake of the host vehicle.

<Configuration of Imaging Unit 101 and Image Analysis Unit 102>
FIG. 2 is a diagram illustrating configurations of the imaging unit 101 and the image analysis unit 102 according to the embodiment.

  The imaging unit 101 is configured by a stereo camera including two imaging units 110a and 110b as imaging means, and the two imaging units 110a and 110b are the same. The imaging units 110a and 110b respectively include imaging lenses 111a and 111b, sensor substrates 114a and 114b including image sensors 113a and 113b in which light receiving elements are two-dimensionally arranged, and analogs output from the sensor substrates 114a and 114b. It comprises signal processing units 115a and 115b that generate and output captured image data obtained by converting electrical signals (electrical signals corresponding to the amount of light received by the light receiving elements on the image sensors 113a and 113b) into digital electrical signals. ing. Luminance image data and parallax image data are output from the imaging unit 101.

  In addition, the imaging unit 101 includes a processing hardware unit 120 including an FPGA (Field-Programmable Gate Array) or the like. In order to obtain a parallax image from the luminance image data output from each of the imaging units 110a and 110b, the processing hardware unit 120 obtains the parallax value of the corresponding image portion between the captured images captured by the imaging units 110a and 110b. A parallax calculation unit 121 is provided as parallax image information generation means for calculation.

  The parallax value referred to here is a comparison image for an image portion on the reference image corresponding to the same point in the imaging region, with one of the captured images captured by the imaging units 110a and 110b as a reference image and the other as a comparison image. The positional deviation amount of the upper image part is calculated as the parallax value of the image part. By using the principle of triangulation, the distance to the same point in the imaging area corresponding to the image portion can be calculated from the parallax value.

  FIG. 3 is a diagram for explaining the principle of calculating the distance from the parallax value by using the principle of triangulation. In the figure, f is the focal length of each of the imaging lenses 111a and 111b, and D is the distance between the optical axes. Z is a distance from the imaging lenses 111a and 111b to the subject 301 (a distance in a direction parallel to the optical axis). In this figure, the imaging positions in the left and right images with respect to the point O on the subject 301 are Δ1 and Δ2 from the imaging center, respectively. The parallax value d at this time can be defined as d = Δ1 + Δ2.

  Returning to the description of FIG. The image analysis unit 102 includes an image processing board and the like. The storage unit 122 includes a RAM, a ROM, and the like that store luminance image data and parallax image data output from the imaging unit 101; A CPU (Central Processing Unit) 123 that executes a computer program for performing parallax calculation control, a data I / F (interface) 124, and a serial I / F 125 are provided.

  The FPGA that constitutes the processing hardware unit 120 performs parallax images by performing processing that requires real-time processing on image data, such as gamma correction, distortion correction (parallelization of left and right captured images), and parallax calculation by block matching. Are generated and written to the RAM of the image analysis unit 102. The CPU of the image analysis unit 102 is responsible for the control of the image sensor controller of each of the imaging units 110A and 110B and the overall control of the image processing board, the detection processing of the three-dimensional shape of the road surface, the guardrail and other various objects (objects). A program for executing the detection process is loaded from the ROM, the luminance image data and the parallax image data stored in the RAM are input, and various processes are executed. The processing results are read from the data I / F 124 and the serial I / F 125. Output to the outside. When executing such processing, the vehicle I / F 124 is used to input vehicle operation information such as the vehicle speed, acceleration (mainly acceleration generated in the longitudinal direction of the vehicle), steering angle, yaw rate, etc. It can also be used as a processing parameter. Data output to the outside is used as input data for controlling various devices of the host vehicle 100 (brake control, vehicle speed control, warning control, etc.).

  Note that the imaging unit 101 and the image analysis unit 102 may be configured as the imaging device 2 that is an integrated device.

<Object detection processing>
Next, with reference to FIG. 4, the function of performing the object detection process realized by the processing hardware unit 120 and the image analysis unit 102 in FIG. 2 will be described. FIG. 4 is a diagram illustrating an example of a functional block diagram of the device control system 1. Hereinafter, the object detection process in this embodiment will be described.

  Luminance image data is output from the two imaging units 110a and 110b constituting the stereo camera. At this time, when the imaging units 110a and 110b are in color, color luminance conversion for obtaining a luminance signal (Y) from the RGB signals is performed using, for example, the following equation [1].

Y = 0.3R + 0.59G + 0.11B (1)
<< Parallax image generation processing >>
Next, in the parallax image generation unit 132 configured by the parallax calculation unit 121, parallax image data (parallax image information. “The position in the vertical direction, the position in the horizontal direction, and the position in the depth direction of the detection target correspond to each other. An example of “attached information”) is generated. In the parallax image generation processing, first, the luminance image data of one imaging unit 110a of the two imaging units 110a and 110b is set as reference image data, and the luminance image data of the other imaging unit 110b is set as comparison image data. The parallax between them is calculated to generate and output parallax image data. The parallax image data indicates a parallax image in which pixel values corresponding to the parallax value d calculated for each image portion on the reference image data are represented as pixel values of the respective image portions.

<< V map generation process >>
Next, in the V map generation unit 134, parallax image data is acquired from the parallax image generation unit 132, and V map generation processing for generating a V map is executed. Each piece of parallax pixel data included in the parallax image data is indicated by a set (x, y, d) of an x-direction position, a y-direction position, and a parallax value d. This is converted into three-dimensional coordinate information (d, y, f) in which d is set on the X-axis, y is set on the Y-axis, and frequency f is set on the Z-axis, or this three-dimensional coordinate information (d, y, f) 3D coordinate information (d, y, f) limited to information exceeding a predetermined frequency threshold is generated as parallax histogram information. The parallax histogram information of this embodiment is composed of three-dimensional coordinate information (d, y, f), and this three-dimensional histogram information distributed in an XY two-dimensional coordinate system is represented by a V map (parallax histogram map). V-disparity map).

  More specifically, the V map generation unit 134 calculates a parallax value frequency distribution for each row region of parallax image data obtained by dividing an image into a plurality of vertical directions. Information indicating the parallax value frequency distribution is parallax histogram information.

  FIG. 5 is a diagram for explaining parallax image data and a V map generated from the parallax image data. Here, FIG. 5A is a diagram illustrating an example of the parallax value distribution of the parallax image, and FIG. 5B is a diagram illustrating a V map illustrating the parallax value frequency distribution for each row of the parallax image of FIG. 5A.

  When parallax image data having a parallax value distribution as shown in FIG. 5A is input, the V map generation unit 134 calculates a parallax value frequency distribution that is a distribution of the number of data of each parallax value for each row, This is output as parallax histogram information. The parallax value frequency distribution information of each row obtained in this way is represented on a two-dimensional orthogonal coordinate system in which the y-axis position on the parallax image (the vertical position of the captured image) is taken on the Y-axis and the parallax value is taken on the X-axis. By expressing this, a V map as shown in FIG. 5B can be obtained. This V map can also be expressed as an image in which pixels having pixel values corresponding to the frequency f are distributed on the two-dimensional orthogonal coordinate system.

  FIG. 6 is a diagram illustrating an example of a captured image as a reference image captured by one imaging unit and a V map corresponding to the captured image. Here, FIG. 6A is a captured image, and FIG. 6B is a V map. That is, the V map shown in FIG. 6B is generated from the captured image as shown in FIG. 6A.

  In the image example shown in FIG. 6A, a road surface 401 on which the host vehicle is traveling, a preceding vehicle 402 that exists in front of the host vehicle, and a utility pole 403 that exists outside the road are displayed. In addition, the V map illustrated in FIG. 6B includes a road surface 501, a preceding vehicle 502, and a utility pole 503 corresponding to the image example.

《Road surface shape detection processing》
Next, in this embodiment, road surface shape detection in which the road surface shape detection unit 135 detects the three-dimensional shape of the front road surface of the vehicle 100 from the V map information (parallax histogram information) generated by the V map generation unit 134. Processing is executed.

  In the image example shown in FIG. 6A, the road surface in which the front road surface of the host vehicle 100 is relatively flat, that is, the front road surface of the host vehicle 100 extends in front of the host vehicle 100 in a plane parallel to the road surface portion directly below the host vehicle 100. This corresponds to a case where the virtual reference road surface (virtual reference movement surface) obtained in this way is matched. In this case, in the lower part of the V map corresponding to the lower part of the image, the high-frequency points (road surface 501) are distributed in a substantially straight line having an inclination such that the parallax value d decreases toward the upper side of the image. Pixels exhibiting such a distribution are pixels that are present at almost the same distance in each row on the parallax image, have the highest occupation ratio, and project a detection object whose distance continuously increases toward the top of the image. It can be said that.

  Since the imaging unit 110A captures an area in front of the host vehicle, as shown in FIG. 6B, the parallax value d of the road surface decreases as the captured image 110A moves upward. Also, in the same row (horizontal line), the pixels that project the road surface have substantially the same parallax value d. Therefore, the high-frequency points (road surface 501) distributed substantially linearly on the V map correspond to the characteristics of the pixels displaying the road surface (moving surface). Therefore, it is possible to estimate that the pixels of the points distributed on or near the approximate straight line obtained by linearly approximating high-frequency points on the V map are the pixels displaying the road surface with high accuracy. Further, the distance to the road surface portion projected on each pixel can be obtained with high accuracy from the parallax value d of the corresponding point on the approximate straight line. In addition, since the height of a road surface is calculated | required by estimation of a road surface, the height of the object on the said road surface can be calculated | required. This can be calculated by a known method. For example, a linear expression representing the estimated road surface is obtained, and the corresponding y coordinate y0 when the parallax value d = 0 is set as the height of the road surface. For example, when the parallax value is d and the y coordinate is y ′, the height from the road surface when y′−y0 is the parallax value d is indicated. The height H from the road surface of the above-described coordinates (d, y ′) can be obtained by an arithmetic expression H = (z × (y′−y0)) / f. In this arithmetic expression, “z” is a distance calculated from the parallax value d (z = BF / (d−offset)), and “f” is a focal length of the imaging units 10a and 10b (y′−y0). The value converted to the same unit as. Here, BF is a value obtained by multiplying the base line length B of the imaging units 10a and 10b and the focal length f, and offset is a parallax when an object at infinity is photographed.

<< U map generation process >>
Next, the U map generation unit 137 performs frequency U map generation processing and height U map generation processing as U map generation processing for generating a U map (U-disparity map).

  In the frequency U map generation processing, a set (x, y, d) of the x-direction position, the y-direction position, and the parallax value d in each parallax pixel data included in the parallax image data is x on the X axis and d on the Y axis. , The frequency is set on the Z axis, and XY two-dimensional histogram information is created. This is called a frequency U map. In the U map generation unit 137 of this embodiment, a frequency U map is created only for a point (x, y, d) of a parallax image whose height H from the road surface is in a predetermined height range (for example, 20 cm to 3 m). . In this case, an object existing within the predetermined height range from the road surface can be appropriately extracted.

  In the height U map generation processing, a set (x, y, d) of the x-direction position, the y-direction position, and the parallax value d in each piece of parallax pixel data included in the parallax image data is represented by x, Y on the X axis. X-Y two-dimensional histogram information is created by setting the axis d and the Z-axis height from the road surface. This is called a height U map. The height value at this time is the highest from the road surface.

  FIG. 7 is an image example schematically showing an example of a reference image imaged by the imaging unit 110a, and FIG. 8 is a U map corresponding to the image example of FIG. Here, FIG. 8A is a frequency U map, and FIG. 8B is a height U map.

  In the image example shown in FIG. 7, guard rails 413 and 414 exist on the left and right sides of the road surface, and one preceding vehicle 411 and one oncoming vehicle 412 exist as other vehicles. At this time, in the frequency U map, as shown in FIG. 8A, the high-frequency points corresponding to the left and right guard rails 413 and 414 are substantially linear 603 and 604 extending upward from the left and right ends toward the center. Distributed. On the other hand, high-frequency points corresponding to the preceding vehicle 411 and the oncoming vehicle 412 are distributed between the left and right guard rails in the state 601 and 602 of line segments extending substantially parallel to the X-axis direction. In addition, in a situation where the side portions of these vehicles other than the rear portion of the preceding vehicle 411 or the front portion of the oncoming vehicle 412 are projected, the parallax is within the image area displaying the same other vehicle. Arise. In such a case, as shown in FIG. 8A, the high-frequency point corresponding to the other vehicle is a state in which a line segment extending in parallel with the substantially X-axis direction and a line segment inclined with respect to the approximately X-axis direction are connected. The distribution of.

  In the height U map, the points with the highest height from the road surface in the left and right guard rails 413, 414, the preceding vehicle 411, and the oncoming vehicle 412 are distributed in the same manner as the frequency U map. Here, the heights of the point distribution 701 corresponding to the preceding vehicle and the point distribution 702 corresponding to the oncoming vehicle are higher than the point distributions 703 and 704 corresponding to the guardrail. Thereby, the height information of the object in the height U map can be used for object detection.

<< Real U map generation process >>
Next, the real U map generation unit 138 will be described. The real U map generation unit 138 performs real frequency U map generation processing and real height U map generation processing as U map generation processing for generating a real U map (an example of “distribution data”). Run.

  The real U map is obtained by converting the horizontal axis in the U map from a pixel unit of an image to a unit corresponding to an actual distance, and converting the parallax value on the vertical axis into a thinned parallax having a thinning rate corresponding to the distance. Note that the real U map here can also be referred to as a bird's-eye view image (bird's-eye view image) obtained from a bird's-eye view of real space.

  In the real frequency U map generation process, the real U map generation unit 138 determines a set (x, y, d) of the x direction position, the y direction position, and the disparity value d in each piece of disparity pixel data included in the disparity image data. X-Y two-dimensional histogram information is created by setting an actual distance in the horizontal direction on the X axis, thinning parallax on the Y axis, and frequency on the Z axis. Note that the real U map generation unit 138 of the present embodiment is similar to the U map generation unit 137 only for the point (x, y, d) of the parallax image in which the height H from the road surface is within a predetermined height range. Create a real frequency U map. Note that the real U map generation unit 138 may generate a real U map based on the U map generated by the U map generation unit 137.

  FIG. 9 is a diagram showing a real U map (hereinafter, real frequency U map) corresponding to the frequency U map shown in FIG. 8A. As shown in the figure, the left and right guardrails are represented by vertical linear patterns 803 and 804, and the preceding vehicle and the oncoming vehicle are also represented by patterns 801 and 802 that are close to the actual shape.

  The thinning parallax on the vertical axis is not thinned for a long distance (here, 50 m or more), thinned to 1/2 for a medium distance (20 m or more and less than 50 m), and 1 / for a short distance (10 m or more and less than 20 m). 3 is thinned out, and the short distance (10 m or more and less than 20 m) is thinned out to 1/8.

  That is, the farther away, the smaller the amount to be thinned out. The reason for this is that because the object appears small in the distance, the parallax data is small and the distance resolution is small, so the thinning is reduced.On the other hand, in the short distance, the object is large, so the parallax data is large and the distance resolution is large. To increase.

  An example of a method for converting the horizontal axis from the pixel unit of the image to an actual distance and an example of a method for obtaining (X, d) of the real U map from (x, d) of the U map will be described with reference to FIG.

  A width of 10 m on each side, that is, 20 m when viewed from the camera is set as the object detection range. If the width of one pixel in the horizontal direction of the real U map is 10 cm, the horizontal size of the real U map is 200 pixels.

  The focal length of the camera is f, the lateral position of the sensor from the camera center is p, the distance from the camera to the subject is Z, and the lateral position from the camera center to the subject is X. When the pixel size of the sensor is s, the relationship between x and p is expressed by “x = p / s”. Further, there is a relationship of “Z = Bf / d” from the characteristics of the stereo camera.

  From the figure, since there is a relationship of “x = p * Z / f”, it can be expressed by “X = sxB / d”. X is an actual distance, but since the width of one pixel in the horizontal direction on the real U map is 10 cm, the position of X on the real U map can be easily calculated.

  A real U map (hereinafter, real height U map) corresponding to the height U map shown in FIG. 8B can be created in the same procedure.

  The real U map has a merit that the processing can be performed at high speed because the vertical and horizontal lengths can be made smaller than the U map. In addition, since the horizontal direction is independent of distance, the same object can be detected with the same width in both the distant and the vicinity, and the subsequent peripheral area removal and processing branching to horizontal separation and vertical separation are determined. There is also an advantage that (width threshold processing) is simplified.

  The vertical length in the U map is determined by how many meters the shortest measurable distance is. That is, since “d = Bf / Z”, the maximum value of d is determined according to the shortest measurable Z. The parallax value d is usually calculated in units of pixels in order to handle a stereo image. However, since the parallax value includes a small number, a parallax value obtained by multiplying the parallax value by a predetermined value and rounding off a decimal part is used.

When the shortest measurable Z is halved, d is doubled, so the U map data is huge. Therefore, when creating a real U map, the data is compressed by thinning out pixels as the distance is short, and the amount of data is reduced compared to the U map.
Therefore, object detection by labeling can be performed at high speed.

《Isolated area detection》
Next, an isolated area detection process performed by the isolated area detection unit 139 will be described. FIG. 11 is a flowchart illustrating an example of the isolated region detection process. The isolated region detection unit 139 first smoothes the information of the real frequency U map generated by the real U map generation unit 138 (step S111).

  This is because it is easy to detect an effective isolated region by averaging the frequency values. That is, the disparity value has dispersion due to calculation errors and the like, and the disparity value is not calculated for all pixels. Therefore, the real U map is different from the schematic diagram shown in FIG. Is included. Therefore, the real U map is smoothed in order to remove noise and to easily separate an object to be detected. Like smoothing of an image, this may be considered as noise by applying a smoothing filter (for example, a simple average of 3 × 3 pixels) to a real U map frequency value (real frequency U map). The frequency decreases, and in the object portion, the frequency becomes higher than that of the surrounding area, and there is an effect of facilitating the subsequent isolated region detection processing.

  Next, a threshold value for binarization is set (step S112). Initially, the smoothed real U map is binarized using a small value (= 0) (step S113). Thereafter, labeling of coordinates having a value is performed to detect an isolated region (step S114).

  In these two steps, an isolated region (referred to as an island) having a higher frequency than the surroundings is detected in the real frequency U map. For detection, the real frequency U map is first binarized (step S113). First, binarization is performed with a threshold value of 0. This is a countermeasure against the fact that there are islands that are isolated or connected to other islands because of the height of the object, its shape, separation from the road surface parallax, and the like. In other words, binarizing the real frequency U map from a small threshold first detects an isolated island of an appropriate size, and then increases the threshold to separate connected islands and isolate It is possible to detect an island of an appropriate size.

  Labeling is used as a method for detecting an island after binarization. Coordinates that are black after binarization (coordinates whose frequency value is higher than the binarization threshold) are labeled based on their connectivity, and regions with the same label are defined as islands.

  The size of each of the detected isolated areas is determined (step S115). In this case, since the detection target is a pedestrian to a large automobile, it is determined whether or not the width of the isolated region is within the size range. If the size is large (step S115: YES), the binarization threshold is incremented by 1 (step S112), and binarization is performed only within the isolated region of the real frequency U map (step S113). Then, labeling is performed, a smaller isolated area is detected (step S114), and the size is determined (step S115).

  The labeling process is repeatedly performed from the above threshold setting, and an isolated region having a desired size is detected. If an isolated region having a desired size is detected (step S115: NO), the peripheral region is then removed (step S116). This is a distant object, the accuracy of road surface detection is poor, the road surface parallax is introduced into the real U map, and when the object and road surface parallax are detected together, the height of the left and right, the vicinity Is a process of deleting an area close to the road surface (peripheral part in the isolated area). If the removal area exists (step S117: YES), labeling is performed again to reset the isolated area (step S114).

<< Detection of corresponding area of parallax image and extraction of object area >>
Next, the corresponding region detection unit 140 and the object region extraction unit 141 for parallax images will be described. 12 is a diagram showing a real frequency U map in which a rectangular area inscribed by an isolated area detected by the isolated area detection unit is set. FIG. 13 is a parallax in which a scanning range corresponding to the rectangular area in FIG. 12 is set. FIG. 14 is a diagram illustrating a parallax image in which an object area is set by searching the scanning range in FIG. 13.

  As shown in FIG. 12, the first detection island 811 and the first detection island 811 are defined as rectangular areas inscribed in the first vehicle 801 and the second vehicle 802 as the isolated areas. When the second detection island 812 is set, the width of the rectangular area (the length in the X-axis direction on the U map) corresponds to the width of the identification object (object) corresponding to the isolated area. Further, the height of the set rectangular area corresponds to the depth (length in the traveling direction of the host vehicle) of the identification object (object) corresponding to the isolated area. On the other hand, the height of the identification object (object) corresponding to each isolated region is unknown at this stage. The corresponding region detection unit 140 of the parallax image detects a corresponding region on the parallax image corresponding to the isolated region in order to obtain the height of the object corresponding to the isolated region related to the object candidate region.

  The corresponding region detection unit 140 for the parallax image is based on the isolated region information output from the isolated region detection unit 139, and the position, width, and minimum of the first detection island 811 and the second detection island 812 island detected from the real U map. From the parallax, the x-direction range (xmin, xmax) of the first detection island corresponding region scanning range 481 and the second detection island corresponding region scanning range 482, which are the ranges to be detected in the parallax image shown in FIG. Further, in the parallax image, the height and position of the object (ymin = "y indicating the height of the road surface obtained from the maximum parallax dmax" from y coordinate corresponding to the maximum height from the road surface at the time of the maximum parallax dmax ")" Can be determined).

  Next, in order to detect the exact position of the object, the set scanning range is scanned, and pixels having a parallax with values of the range of the rectangular depth (minimum parallax dmin, maximum parallax dmax) detected by the isolated region detection unit 139 Are extracted as candidate pixels. In the extracted candidate pixel group, a line having a predetermined ratio or more in the horizontal direction with respect to the detection width is set as an object candidate line.

  Next, scanning is performed in the vertical direction, and when there are other object candidate lines having a predetermined density or more around an object candidate line of interest, the object candidate line of interest is determined as an object line. .

  Next, the object area extraction unit 141 searches the object line in the parallax image search area to determine the lowermost and uppermost ends of the object line, and as shown in FIG. 462 is determined as regions 451 and 452 of the objects (first vehicle and second vehicle) in the parallax image.

  FIG. 15 is a flowchart illustrating a flow of processing performed by the corresponding region detection unit 140 and the object region extraction unit 141 for parallax images. First, a search range in the x-axis direction for the parallax image is set from the position, width, and minimum parallax of the island in the real U map (step S161).

  Next, the maximum search value ymax in the y-axis direction with respect to the parallax image is set from the relationship between the maximum parallax dmax of the island and the road surface height (step S162). Next, by obtaining and setting the minimum search value ymin in the y-axis direction for the parallax image from the maximum height of the island in the real height U map and ymax and dmax set in step S172, the y-axis for the parallax image is set. A direction search range is set (step S163).

  Next, the parallax image is searched in the set search range, and pixels within the range of the minimum parallax dmin and the maximum parallax dmax of the island are extracted and set as object candidate pixels (step S164). When the object candidate pixels are at a certain ratio in the horizontal direction, the line is extracted as an object candidate line (step S165).

  The density of the object candidate line is calculated, and if the density is greater than a predetermined value, the line is determined as the object line (step S166). Finally, a circumscribed rectangle of the object line group is detected as an object area in the parallax image (step S167).

  Thereby, an identification target (object, object) can be recognized.

<Object type classification>
Next, the object type classification unit 142 will be described.

  From the height (yomax-yomin) of the object area extracted by the object area extraction unit 141, the identification object (object) displayed in the image area corresponding to the object area is calculated by the following equation [2]. The actual height Ho can be calculated. However, “zo” is the distance between the object corresponding to the object area calculated from the minimum parallax value d in the object area and the host vehicle, and “f” is the focal length of the camera (yomax−yomin) The value converted to the same unit as.

Ho = zo × (yomax−yomin) / f (2)
Similarly, from the width (xomax−xomin) of the object area extracted by the object area extraction unit 141, the identification object (object) displayed in the image area corresponding to the object area is calculated from the following equation [3]. The actual width Wo can be calculated.

Wo = zo × (xomax−xomin) / f (3)
The depth Do of the identification target object (object) displayed in the image area corresponding to the object area is expressed by the following equation [4] from the maximum parallax dmax and the minimum parallax dmin in the isolated area corresponding to the object area. ] Can be calculated.

Do = BF × {(1 / (dmin−offset) −1 / (dmax−offset)}} Equation [4]
The object type classification unit 142 classifies the object type from information on the height, width, and depth of the object corresponding to the object area that can be calculated in this way. The table shown in FIG. 16 shows an example of table data for classifying object types. In the example of FIG. 16, for example, if the width is less than 1100 mm, the height is less than 250 mm, and the depth exceeds 1000 mm, it is determined as “motorcycle, bicycle”. If the width is less than 1100 mm, the height is less than 250 mm, and the depth is 1000 mm or less, it is determined as a “pedestrian”. According to this, it becomes possible to distinguish and recognize whether the identification object (object) existing in front of the host vehicle is a pedestrian, a bicycle or a motorcycle, a small car, a truck, or the like. The above-described method is an example, and various methods can be used in the present invention as long as the object type classification and the object position can be specified.

<< 3D position determination >>
Next, processing of the three-dimensional position determining unit 143 will be described with reference to FIG. The three-dimensional position determination unit 143 determines a relative three-dimensional position of the identification target (object) with respect to the host vehicle 100.

  FIG. 17 is a diagram illustrating an example of the three-dimensional position determination process. In FIG. 17, a process for one object (target object) among the objects extracted by the object area extraction unit 141 will be described. Therefore, the three-dimensional position determination unit 143 performs the process of FIG. 17 on each object extracted by the object region extraction unit 141.

  In step S201, the three-dimensional position determination unit 143 determines whether or not the object type of the target object is a type that is not a rigid body (an example of “first type”). The rigid body is an object that does not deform, and the type of the rigid body is, for example, “small car”, “ordinary car”, “truck”, and the like. The types that are not rigid bodies include, for example, “pedestrians”, “motorcycles, bicycles”, and the like. Although the motorcycle or bicycle itself is a rigid body, since the driver of the motorcycle or bicycle is not a rigid body, it is determined as a type that is not a rigid body.

  If the object type is a type of rigid body (an example of “second type”) (NO in step S201), the three-dimensional position determination unit 143 calculates the center position of the target object (step S202). The three-dimensional position determination unit 143 determines the three-dimensional position of the object based on the distance to the object corresponding to the detected object region and the distance on the image between the image center of the parallax image and the center of the object region on the parallax image. The center position in the coordinates is calculated by the following formula, for example.

  When the center coordinates of the object region on the parallax image are (region_centerX, region_centerY) and the image center coordinates of the parallax image are (image_centerX, image_centerY), the horizontal direction relative to the imaging units 110a and 110b of the identification object (object) The center position Xo in the direction and the center Yo position in the height direction can be calculated from the following equations [5] and [6].

Xo = Z * (region_centerX-image_centerX) / f (5)
Yo = Z × (region_centerY−image_centerY) / f (6)
Subsequently, the three-dimensional position determination unit 143 determines the position of the center of the target object as the position of the target object (step S203), and ends the process.

  FIG. 18 is a diagram illustrating a method for calculating the position of an object such as a vehicle. When the object type is not “pedestrian” or “motorcycle, bicycle”, the three-dimensional position determination unit 143 sets, for example, the positions 901a, 901b, and 901c of the center of the object region as the positions of the target objects. Thereby, the influence of noise or the like can be reduced as compared with a method in which the positions 900a, 900b, and 900c of the center of gravity of the target object, which will be described later, are set as the positions of the target object.

  On the other hand, if the object type is “pedestrian” or “motorcycle, bicycle” (YES in step S201), the three-dimensional position determination unit 143 calculates the position of the center of gravity of the target object (step S204). Thereby, the horizontal position of the target object can be determined by a method according to the object type. That is, when the object type is a type that is not a rigid body, the position of the center of gravity in the horizontal direction of the object is determined as the horizontal position of the object. When the object type is a type that is a rigid body, the positions of the centers of both ends in the horizontal direction of the object are determined as the horizontal positions of the object.

  FIG. 19 is a diagram illustrating a method of calculating the position of an object that is a pedestrian, motorcycle, or bicycle. FIG. 19A shows the position 910a of the center of gravity when the pedestrian is closing his arm. FIG. 19B shows the position 910b of the center of gravity when the pedestrian is opening his arm.

  In step S204, the three-dimensional position determination unit 143 searches the parallax image for a corresponding region of the target object (for example, the first detection island corresponding region scanning range 481 or the second detection island corresponding region scanning range 482 in FIG. 13). Range. Then, in the search range, the number of pixels indicating the parallax of the object is summed for each position in the horizontal direction (x direction). For example, the number of pixels having the parallax value d within the maximum parallax value from the minimum parallax value of the object is summed for each position in the horizontal direction (x direction). Then, the three-dimensional position determining unit 143 sets the horizontal position having the largest total number as the horizontal position of the target object. Note that the vertical position of the target object may be obtained in the same manner as the position of the center of gravity in the horizontal direction, or may be the center position in the vertical direction.

  Subsequently, the three-dimensional position determination unit 143 sets the position of the center of gravity of the target object as the position of the target object (step S205), and ends the process.

<Modification of the method for calculating the position of the center of gravity>
In step S204, the position of the center of gravity may be calculated as follows, for example.

  The three-dimensional position determination unit 143 may set the horizontal position where the height of the pixel indicating the parallax of the object is the highest in the above-described search range as the position of the horizontal center of gravity of the target object. This is because a pedestrian, a bicycle, or the like is considered to have a human head at the highest position.

  Alternatively, the three-dimensional position determination unit 143 may recognize the person's head by recognizing the reference image, and set the position of the recognized person's head in the horizontal direction as the position of the center of gravity in the horizontal direction of the target object. As a human head recognition method, various known methods can be applied.

  Alternatively, the three-dimensional position determining unit 143 may calculate the target based on the isolated region on the real U map corresponding to the object region extracted by the object region extracting unit 141 instead of calculating based on the search range in the parallax image. The position of the center of gravity of the object may be calculated. In this case, for example, in the isolated region, the three-dimensional position determination unit 143 calculates the total value of the parallax frequencies along the Y axis for each position of each X coordinate. Then, the three-dimensional position determination unit 143 sets the X coordinate having the largest total value as the position of the center of gravity in the horizontal direction of the target object.

《Object tracking》
Next, the object tracking unit 144 will be described. The object tracking unit 144 executes processing for tracking (tracking) an object (object) detected from a parallax image of a previous (past) frame.

  The object tracking unit 144 predicts the position of each object with respect to the parallax image of the current frame based on the position of each object determined by the three-dimensional position determination unit 143 based on the parallax images of a plurality of previous frames. . Specifically, the object tracking unit 144 specifies the relative moving speed and moving direction between the object and the host vehicle 100 using the positions of the objects in the previous plurality of frames, and the moving speed and moving direction. Based on the above, the position of each object with respect to the parallax image of the current frame is predicted. A known technique can be applied to the object tracking process. However, when the object tracking process is performed using the moving speed and the moving direction, if the position of the object such as a non-rigid person who has detected the object fluctuates between frames, for example, by spreading the hand, the object will be accompanied accordingly. Is considered to have moved, and erroneous object tracking processing occurs. This is prevented by adopting the position of the center of gravity as described above. On the other hand, the influence of noise or the like can be reduced by adopting a position based on the object area for a rigid object.

  Then, the object tracking unit 144 continues to track the object based on the similarity between the parallax image of the object area in the parallax image of the previous frame and the parallax image of the area in the parallax image of the current frame with respect to the predicted position. To do. Thereby, the same object is grasped as the same object in a plurality of frames.

  The object tracking unit 144 erroneously determines that the parallax of the object is not sufficiently measured due to, for example, reflection of sunlight or darkness, or that the pedestrian and the object adjacent to the pedestrian are one object. If so, the position of the object is estimated. In this case, the object tracking unit 144 estimates the object area in the parallax image of the current frame based on the relative moving speed and moving direction of the object with respect to the host vehicle 100 in a plurality of previous frames. Then, when the object is a non-rigid object such as a person, the object tracking unit 144 calculates the position of the center of gravity in order to obtain the position of the center of gravity. The position and the ratio of the position of the left end and the position of the center of gravity are calculated. Then, the object tracking unit 144 estimates a position obtained by dividing the horizontal direction of the object region estimated in the parallax image of the current frame by the ratio as the position of the center of gravity of the object in the parallax image of the current frame.

  For example, when the average of the ratio of the right end position and the center of gravity in the horizontal direction and the ratio of the left end position and the position of the center of gravity of the object area of the object in a plurality of previous frames is 2: 1, the parallax image of the current frame The position obtained by dividing the horizontal direction of the object area estimated in step 2 into 2 is estimated as the position of the center of gravity of the object in the parallax image of the current frame.

<Summary>
Conventionally, when a person such as a pedestrian or a bicycle driver suddenly spreads his / her hand in the lateral direction, the position of the center of both ends in the lateral direction of the object may change abruptly, which may make tracking impossible.
Also, for example, when predicting and tracking the position of the object in the current frame based on the position of the object in a plurality of previous frames, a person such as a pedestrian or a bicycle driver suddenly moves the hand in the lateral direction. If the hand is spread, the position where the hand is spread may be erroneously recognized as the movement direction of the person, and the position prediction accuracy may be lowered.

  According to the above-described embodiment, the position of the object in the parallax image is determined according to the type of the object. Thereby, highly accurate tracking can be continued.

  Since the distance value (distance value) and the parallax value can be handled equivalently, in the present embodiment, the parallax image is used as an example of the distance image, but the present invention is not limited to this. For example, a distance image may be generated by integrating distance information generated using a detection device such as a millimeter wave radar or a laser radar with a parallax image generated using a stereo camera. Further, the detection accuracy may be further enhanced by using a stereo camera and a detection device such as a millimeter wave radar or a laser radar in combination with the detection result of the object by the stereo camera described above.

  The system configuration in the above-described embodiment is an example, and it goes without saying that there are various system configuration examples depending on applications and purposes.

  For example, the functional unit that performs processing of at least a part of the functional units of the processing hardware unit 120 and the image analysis unit 102 may be realized by cloud computing including one or more computers.

  Moreover, although the above-mentioned embodiment demonstrated the example with which the apparatus control system 1 was mounted in the motor vehicle as the own vehicle 100, it is not limited to this. For example, it may be mounted on a vehicle such as a motorcycle, bicycle, wheelchair, or agricultural cultivator as an example of another vehicle. Moreover, it is good also as what is mounted not only on the vehicle as an example of a mobile body but on mobile bodies, such as a robot.

  The functional units of the processing hardware unit 120 and the image analysis unit 102 may be realized by hardware, or may be realized by executing a program stored in the storage device by the CPU. Good. The program may be recorded and distributed on a computer-readable recording medium by a file in an installable or executable format. Examples of the recording medium include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc, and the like. Further, this program may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program may be configured to be provided or distributed via a network such as the Internet.

1 device control system 100 own vehicle 101 imaging unit 102 image analysis unit (an example of “information processing apparatus”)
103 display monitor 104 vehicle travel control unit (an example of “control unit”)
110a, 110b Imaging unit 120 Processing hardware unit 132 Parallax image generation unit (an example of “generation unit”)
134 V map generation part (an example of "acquisition part")
135 Road surface shape detection unit 137 U map generation unit 138 Real U map generation unit 139 Isolated region detection unit 140 Parallax image corresponding region detection unit 141 Object region extraction unit 142 Object type classification unit (an example of “determination unit”)
143 3D position determination unit (an example of “determination unit”)
144 Object tracking part (an example of a "tracking part")
2 Imaging device

JP-A-10-283462

Claims (9)

An acquisition unit that acquires information in which a vertical position, a horizontal position, and a depth direction position of an object are associated;
A determination unit that determines a type of the object based on the information acquired by the acquisition unit;
A determination unit that determines the position of the object by a method according to the type of the object determined by the determination unit;
A tracking unit that tracks the object based on the position of the object determined by the determining unit;
An information processing apparatus comprising: The determining unit determines the position of the object by a first method using the position of the center of gravity of the object when the type of the object determined by the determining unit is the first type; The information processing apparatus according to claim 1, wherein when the type of the object determined by the determination unit is a second type, the position of the object is determined by a second method different from the first method. The determination unit, when the type of the object determined by the determination unit is a second type, determines the positions of the centers of both ends of the object in the lateral direction as the positions of the object. The information processing apparatus described. The said determination part determines the position of the gravity center in the said horizontal direction of the said object based on the number of the information of the position of the depth direction corresponding to each position of the said horizontal direction of the said object in the said information. The information processing apparatus described. A plurality of imaging units;
A generating unit that generates the information based on a plurality of images captured by the plurality of imaging units;
An information processing apparatus according to any one of claims 1 to 4,
An imaging apparatus comprising: An imaging device according to claim 5;
Based on the data of the object being tracked by the tracking unit, a control unit that controls the moving body,
With
The plurality of imaging units are mounted on the moving body, and are device control systems that image the front of the moving body. The apparatus control system according to claim 6 is provided.
A moving body controlled by the control unit. Computer
Obtaining information in which the vertical position, the horizontal position, and the depth position of the object are associated with each other;
Determining the type of the object based on the acquired information;
Determining the position of the object by a method according to the determined type of the object;
Tracking the object based on the determined position of the object;
An information processing method is executed. On the computer,
Obtaining information in which the vertical position, the horizontal position, and the depth position of the object are associated with each other;
Determining the type of the object based on the acquired information;
Determining the position of the object by a method according to the determined type of the object;
Tracking the object based on the determined position of the object;
A program that executes

Images