JP6756101B2 - Object recognition device - Google Patents

Description

The present invention relates to an object recognition device.

Conventionally, the following Patent Document 1 is known as a document relating to a technique for recognizing an object from an image captured around a vehicle. In Patent Document 1, when the captured image includes a road marking such as a diamond-shaped mark (a mark indicating that there is a pedestrian crossing in front) drawn on the road, the width of the road marking stipulated by law is used. A device is shown that calculates the distance from the vehicle to the road markings based on the width of the road markings in the captured image. In this device, the position of the object included in the captured image is measured by using the calculated distance from the vehicle to the road marking.

Japanese Unexamined Patent Publication No. 2012-159470

By the way, in the above-mentioned conventional device, the width of the road marking on the captured image is used in order to obtain the distance from the vehicle to the road marking. Therefore, when converting the width on the captured image to the distance, an error for each pixel of the width (for example, an error of about 10 cm at a distance of 15 m and an error of about 50 cm at a distance of 30 m) is included in the calculation result. As a result, the accuracy of measuring the position of the object may decrease.

Therefore, in the present technical field, it is desired to provide an object recognition device capable of accurately calculating the position of an object from an captured image.

One aspect of the present invention is an object recognition device that recognizes an object around a vehicle based on an image obtained by imaging the surroundings of the vehicle, and is a vehicle position recognition that recognizes the position of the vehicle on a map. A storage unit that stores the position information of the landmark on the map, and the position coordinates of the landmark on the captured image based on the captured image, the position of the vehicle on the map, and the position information of the landmark on the map. The landmark recognition unit that recognizes the image, the object recognition unit that recognizes the position coordinates of the object on the captured image based on the captured image, the position coordinates of the object on the captured image, the position coordinates of the landmark on the captured image, It also includes an object position calculation unit that calculates the position of an object on the map by using the positional relationship between the landmark and the object on the same captured image based on the position information of the landmark on the map.

As described above, according to the present invention, the position of the object can be calculated accurately from the captured image.

It is a block diagram which shows the object recognition apparatus which concerns on this embodiment. It is a figure for demonstrating the recognition of the landmark included in the captured image. It is a flowchart which shows the object position calculation process of an object recognition apparatus. It is a flowchart which shows the recognition process of the position coordinate of a landmark using a white line point. (A) It is a top view which shows the white line point on a map. (B) It is a figure which shows the white line point projected on the captured image. It is a flowchart which shows the calculation process of the position of an object when a landmark is a white line. (A) It is a figure for demonstrating how to obtain the position coordinates of intersection points L, R on a captured image. (B) It is a top view for demonstrating how to obtain the positions of intersections L and R on a map. (A) It is a graph for demonstrating how to obtain the x-axis coordinate of the intersection L. (B) It is a graph for demonstrating how to obtain the y-axis coordinate of the intersection L. (A) It is a figure for demonstrating an example of how to obtain the position of an object using three landmarks. (B) It is a figure for demonstrating another example of how to obtain the position of an object using three landmarks.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

The object recognition device 100 according to the present embodiment shown in FIG. 1 is a device mounted on a vehicle such as a passenger car and recognizes an object based on an captured image obtained by imaging the surroundings of the vehicle. An object is an obstacle that may collide with a vehicle. Objects include structures such as other vehicles (including parked vehicles), pedestrians, bicycles, and utility poles. The object recognition device 100 calculates the position of the object from the captured image around the vehicle.

[Configuration of object recognition device]
As shown in FIG. 1, the object recognition device 100 includes an ECU [Engine Control Unit] 10. The ECU 10 is an electronic control unit having a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], a CAN [Controller Area Network] communication circuit, and the like. The ECU 10 realizes various functions by loading the program stored in the ROM into the RAM via the CAN communication circuit and executing the program loaded in the RAM in the CPU. The ECU 10 may be composed of a plurality of electronic control units. A monocular camera 1, a map information storage unit 2, a landmark information storage unit 3, and a GPS reception unit 4 are connected to the ECU 10.

The monocular camera 1 is an imaging device (for example, a CCD camera or a CMOS camera) that images the outside of a vehicle. The monocular camera 1 is arranged behind the windshield of the vehicle and images the front (traveling direction) of the vehicle at a predetermined angle of view. The monocular camera 1 transmits an image captured in front of the vehicle to the ECU 10. The monocular camera 1 may be provided on the back surface of the vehicle, the left side surface of the vehicle, and the right side surface of the vehicle. In this case, the monocular camera 1 images the rear of the vehicle, the left side of the vehicle, and the right side of the vehicle.

The map information storage unit 2 is a database provided with map information. The map information storage unit 2 is formed in an HDD [Hard Disk Drive] mounted on the vehicle. The map information includes road position information, road shape information (for example, curve, type of straight line portion, curvature of curve, etc.), position information of intersections and branch points, position information of buildings, and the like. The map information storage unit 2 may be stored in a computer of a facility such as an information processing center capable of communicating with the vehicle.

The landmark information storage unit 3 is a database containing landmark information. The landmark information storage unit 3 is formed in an HDD mounted on the vehicle. A landmark is a fixed position on the road surface (including on a road surface other than the vehicle lane) and serves as a reference for calculating the position of an object. Landmarks include road signs and road markings. Road signs include guide signs, warning signs, regulatory signs, instruction signs, and the like. Road markings include regulatory markings and instruction markings. Regulation signs include a turning prohibition mark, a maximum speed mark, and the like. The instruction signs include a white line (roadway center line, roadway outside line, lane boundary line, etc.), a diamond mark indicating that there is a pedestrian crossing in front, a triangle mark indicating that there is a priority road ahead, a traveling direction mark, etc. is there.

The landmark information storage unit 3 stores the location information of the landmark on the map. That is, the landmark information storage unit 3 stores the location information of the landmark associated with the map information stored in the map information storage unit 2. Further, the landmark information storage unit 3 stores the image information of the landmark for recognizing the landmark from the captured image. The landmark image information is information on the characteristics of the image used for pattern recognition, which will be described later. The image information of the landmark includes the shape of the road sign, the shape of the road marking, and the like.

The landmark information storage unit 3 stores the location information of the landmark and the image information of the landmark in association with each other. The landmark information storage unit 3 may be stored in a computer of a facility such as an information processing center capable of communicating with the vehicle. Further, the map information storage unit 2 may store the landmark position information and the landmark image information in addition to the map information.

The GPS receiving unit 4 is mounted on the vehicle and functions as a position measuring unit for measuring the position of the vehicle. The GPS receiving unit 4 measures the position of the vehicle (for example, the latitude and longitude of the vehicle M) by receiving signals from three or more GPS satellites. The GPS receiving unit 4 transmits the measured vehicle position information to the ECU 10.

Next, the functional configuration of the ECU 10 will be described. The ECU 10 has a vehicle position recognition unit 11, a landmark recognition unit 12, an object recognition unit 13, and an object position calculation unit 14.

The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on the map information of the map information storage unit 2 and the position information of the GPS receiving unit 4. The vehicle position recognition unit 11 recognizes the direction of the vehicle on the map based on the time change of the position information of the GPS receiving unit 4. The vehicle position recognition unit 11 does not necessarily have to recognize the direction of the vehicle.

The landmark recognition unit 12 recognizes the landmark included in the captured image captured by the monocular camera 1. The landmark recognition unit 12 is based on the landmark position information and the landmark image information stored in the landmark information storage unit 3 and the position and orientation of the vehicle on the map recognized by the vehicle position recognition unit 11. And recognize the landmarks included in the captured image.

Here, FIG. 2 is a diagram for explaining the recognition of landmarks included in the captured image. FIG. 2 shows an captured image, white lines W1 and W2, a diamond mark (road marking) D, a road sign Rs, and a root Rt of the road sign Rs. Further, the uv Cartesian coordinate system is shown with the horizontal direction of the captured image as the u-axis direction and the vertical direction of the captured image as the v-axis direction. In the present embodiment, the u-axis direction of the captured image corresponds to the vehicle width direction (left-right direction) of the vehicle.

In the captured image shown in FIG. 2, the landmark recognition unit 12 uses the monocular camera 1 from among the stored landmarks based on the position and orientation of the vehicle on the map and the position information of the landmarks on the map. Extract landmarks located in the imaging direction of. The landmark recognition unit 12 recognizes the landmark included in the captured image by performing pattern recognition based on the image information of the landmark stored in the landmark information storage unit 3 with respect to the extracted landmark.

In pattern recognition, the characteristics of the landmark image to be recognized (included in the image information) are learned in advance, and the characteristics of the partial image cut out from the captured image are compared with the characteristics of the learned landmark image. The part cut out when it is close is regarded as a landmark. This pattern recognition is also used for object recognition from captured images.

Of the landmarks, the road markings drawn on the road surface may be recognized by using edge extraction. The road marking is recognized from the edge information on the road surface and the image information of the road marking by extracting the part where the brightness of the pixel (pixel) suddenly changes in the u-axis direction (vehicle width direction) of the captured image as an edge. be able to.

The landmark recognition unit 12 recognizes the white lines W1 and W2, the diamond-shaped mark D, and the road sign Rs shown in FIG. 2 as landmarks by pattern recognition or edge extraction. The landmark recognition unit 12 recognizes the position coordinates (u-axis coordinates, v-axis coordinates) on the captured image of the recognized landmark. The position coordinates of the diamond mark D, which is a road marking, can be the coordinates of the center of the diamond mark D on the captured image. The position coordinates of the road sign Rs can be the coordinates of the root Rt of the road sign Rs on the captured image. The position coordinates of the white lines W1 and W2 will be described later. The landmark recognition unit 12 recognizes the same landmark by associating the position coordinates of the landmark on the captured image with the position information of the landmark on the map.

The object recognition unit 13 recognizes an object included in the captured image based on the captured image captured by the monocular camera 1. The object recognition unit 13 recognizes an object by using the pattern recognition and edge extraction described above. The object recognition unit 13 can recognize an object included in the captured image by using a well-known method. The object recognition unit 13 recognizes the position coordinates (u-axis coordinates, v-axis coordinates) of the object on the captured image. The position coordinates of the object can be the coordinates of the center of the lower side (lower end) of the object on the captured image.

The object position calculation unit 14 uses the position coordinates of the object on the captured image recognized by the object recognition unit 13, the position coordinates of the landmark on the captured image recognized by the landmark recognition unit 12, and the position information of the landmark on the map. Based on this, the position of the object on the map is calculated. The object position calculation unit 14 calculates the position of the object on the map by using the positional relationship between the landmark and the object on the captured image. The calculation of the position of the object will be described in detail later.

[Object position calculation process of object recognition device]
Hereinafter, the object position calculation process of the object recognition device 100 according to the present embodiment will be described. FIG. 3 is a flowchart showing an object position calculation process of the object recognition device 100. The flowchart shown in FIG. 3 is started when the engine of the vehicle is started by the driver.

As shown in FIG. 3, the ECU 10 of the object recognition device 100 determines in S10 whether or not the object recognition unit 13 has recognized the object included in the captured image. The object recognition unit 13 recognizes an object such as another vehicle by a well-known method based on the captured image of the surroundings of the vehicle captured by the monocular camera 1.

When the object is not recognized (S10: NO), the ECU 10 ends the current process. Then, after a lapse of a certain period of time, the process is repeated from S10 again. When the object included in the captured image is recognized (S10: YES), the ECU 10 shifts to S12. In S12, the ECU 10 recognizes the position coordinates (u-axis coordinates, v-axis coordinates) of the object on the captured image by the object recognition unit 13. After that, it shifts to S14.

In S14, the ECU 10 determines whether or not the landmark recognition unit 12 has recognized the landmark included in the captured image. Based on the landmark position information and image information stored in the landmark information storage unit 3 and the position and orientation of the vehicle on the map recognized by the vehicle position recognition unit 11, the landmark included in the captured image is stored. recognize.

When the landmark is not recognized (S14: NO), the ECU 10 ends the current process. Then, after a lapse of a certain period of time, the process is repeated from S10 again. When the landmark included in the captured image is recognized (S14: YES), the ECU 10 shifts to S16. In S16, the ECU 10 recognizes the position coordinates of the landmark included in the captured image on the captured image by the landmark recognition unit 12. The landmark recognition unit 12 recognizes the position coordinates of the landmark on the captured image in association with the position information of the landmark on the map. After that, it shifts to S18.

In S18, the ECU 10 calculates the position of the object on the map in the object position calculation unit 14. The object position calculation unit 14 calculates the position of the object on the map based on the position coordinates of the object on the captured image, the position coordinates of the landmark on the captured image, and the position information of the landmark on the map. When the ECU 10 calculates the position of the object on the map, the ECU 10 ends this process. Then, after a lapse of a certain period of time, the process is repeated from S10 again.

[Object position calculation process when landmark is white line]
Next, the processing when the landmark is a white line will be described. Here, the processing after S16 in FIG. 3 will be described with reference to FIG. FIG. 4 is a flowchart showing a landmark position coordinate recognition process using white line points. White line points are points plotted on the white line on the map at regular intervals. The white line points correspond to the position information of the white lines on the map. The flowchart shown in FIG. 4 is started when the landmark recognition unit 12 recognizes a white line as a landmark included in the captured image.

As shown in FIG. 4, the ECU 10 projects a white line point on the captured image by the landmark recognition unit 12 as S20. Here, FIG. 5A is a plan view showing white line points on the map. FIG. 5A shows the vehicle M, the two white lines w1 and w2 forming the traveling lane in which the vehicle M travels, and the white line points a1 to a6. The white lines w1 and w2 shown in FIG. 5A are white lines in the map information stored in the map information storage unit 2. The white line points a1 to a3 are points corresponding to the position information of the white line w1. The white line points a4 to a6 are points corresponding to the position information of the white line w2. Further, in FIG. 5A, the xy orthogonal coordinate system is shown with the traveling direction (forward direction) of the vehicle as the x-axis direction and the vehicle width direction (left-right direction) as the y-axis direction.

As shown in FIG. 5A, when the landmark recognition unit 12 recognizes a white line as a landmark, the white line W1 stored in the landmark information storage unit 3 is based on the position and orientation of the vehicle on the map. , W2 recognizes a plurality of white line points (including white line points a1 to a6) corresponding to the position information of W2. The white line points have position information (x-axis coordinates, y-axis coordinates) on the map, respectively.

The landmark recognition unit 12 projects the recognized white line point on the map onto the captured image. The landmark recognition unit 12 projects a white line point seen from the monocular camera 1 on the captured image according to the parameters of the monocular camera 1 by a well-known method. The parameters of the monocular camera 1 are the mounting height, roll, pitch, and yaw of the monocular camera 1. When these parameters change, the appearance of the white line point seen from the monocular camera 1 changes.

When the parameters of the monocular camera 1 have already been changed in S26 described later, the landmark recognition unit 12 projects a white line point on the captured image using the changed parameters of the monocular camera 1. When the landmark recognition unit 12 has never looped the flowchart shown in FIG. 4 (when it has never reached RETURN), the landmark recognition unit 12 projects a white line point with the parameter of the monocular camera 1 as an initial value (preset value). I do. When the landmark recognition unit 12 projects a white line point on the captured image, the landmark recognition unit 12 shifts to S22.

In S22, the landmark recognition unit 12 calculates the deviation between the white line points a1 to a6 projected on the captured image and the white lines W1 and W2 of the captured image. Here, FIG. 5B is a diagram showing white line points projected on the captured image. FIG. 5B shows the difference d between the white lines W1 and W2 on the captured image, the projected white line points a1 to a6, and the white line W1 and the white line point a3. The difference d is the difference (pixel difference) between the white line W1 and the white line point a3 in the u-axis direction (horizontal direction) of the captured image. When the landmark recognition unit 12 calculates the deviation of all the white line points, the landmark recognition unit 12 shifts to S24.

In S24, the landmark recognition unit 12 determines whether or not the end condition is satisfied. The end condition is satisfied when the deviation between all the white line points calculated in S22 and the white line W1 or the white line W2 is equal to or less than a preset threshold value. Further, the termination condition is satisfied when the parameter change processing of the monocular camera 1 in S26, which will be described later, is executed for all the patterns prepared in advance. The termination condition may be satisfied when the processing of S20 to S24 is repeated until the preset upper limit number of times is reached.

When the landmark recognition unit 12 determines that the end condition is not satisfied (S24: NO), the landmark recognition unit 12 shifts to S26. When the landmark recognition unit 12 determines that the end condition is satisfied (S24: YES), the landmark recognition unit 12 shifts to S28.

In S26, the landmark recognition unit 12 changes the parameters of the monocular camera 1 used in the next S20. The landmark recognition unit 12 changes the parameters of the monocular camera 1 in a preset pattern. The landmark recognition unit 12 may increase any of the roll, pitch, and yaw by a predetermined angle each time S26 is executed. In addition, the landmark recognition unit 12 may change the parameter of the monocular camera 1 according to the magnitude of the deviation between the white line point and the white line calculated in S22. When the parameter of the monocular camera 1 is changed, the landmark recognition unit 12 ends the current process. After that, the process is repeated from S20 again.

In S28, the landmark recognition unit 12 associates the position coordinates of the white line points on the captured image with the position information of the white line points on the map. When the flowchart shown in FIG. 4 is repeated a plurality of times, the landmark recognition unit 12 has the position coordinates of the white line points on the captured image when the average value of the deviations between all the white line points is minimized. Is adopted. The landmark recognition unit 12 associates the position coordinates of the white line points on the captured image with the minimum deviation with the position information of the white line points on the corresponding map. When the landmark recognition unit 12 makes an association, the landmark recognition unit 12 ends the current process. After that, the ECU 10 starts the process shown in FIG.

FIG. 6 is a flowchart showing a calculation process of the position of an object when the landmark is a white line. The flowchart shown in FIG. 6 corresponds to S18 in FIG. The flowchart shown in FIG. 6 is started when S28 shown in FIG. 5 is completed.

As shown in FIG. 6, in S30, the ECU 10 calculates the position coordinates of the intersections L and R located on the left and right sides of the object on the captured image by the object position calculation unit 14. Here, FIG. 7A is a diagram for explaining how to obtain the position coordinates of the intersections L and R on the captured image. FIG. 7A shows an object N, a center C of the lower side of the object N, a straight line E extending the lower side of the object N in the u-axis direction (horizontal direction), and a straight line segment K1 connecting the white line point a1 and the white line point a2. , The linear line segment K2 connecting the white line point a4 and the white line point a5, the intersection L of the straight line E and the line segment K1, and the intersection R of the straight line E and the line segment K2 are shown. The object N is, for example, a preceding vehicle traveling in front of the vehicle M. In this case, the lower side center C corresponds to the position of the center of the rear wheel of the preceding vehicle in the vehicle width direction.

As shown in FIG. 7A, the position coordinates of the white line point a4 are (u4, v4), the position coordinates of the white line point a5 are (u5, v5), and the position coordinates of the intersection R are (uR, vR). Similarly, the position coordinates of the white line point a1 are (u1, v1), the position coordinates of the white line point a2 are (u2, v2), and the position coordinates of the intersection R are (uL, vL). Further, let the position coordinates of the lower side center C of the object N be (uC, vC). The object recognition unit 13 has already recognized the position coordinates (uC, vC) of the lower side center C of the object N in S12 of FIG.

As shown in FIG. 7A, the lower center C, the intersection L, and the intersection R coincide with each other with respect to the v-axis coordinates of the captured image. Therefore, the v-axis coordinates of the intersection L and the intersection R are equal to the v-axis coordinates of the lower center C, and can be obtained from the relationship of vC = vL = vR. On the other hand, in the u-axis direction of the captured image, the white line points a1 and a2 and the intersection point L are located on the line segment K1, and the white line points a4 and a5 and the intersection point R are located on the line segment K2. It can be obtained by linear interpolation using equations (1) and (2).
uL = (vL-v1) / (v2-v1) x u2 + (vL-v2) / (v1-v2) x u1 ... (1)
uR = (vR-v4) / (v5-v4) x u5 + (vR-v5) / (v4-v5) x u4 ... (2)

From the above equations (1) and (2), the u-axis coordinates of the intersection L and the intersection R can be obtained. In this way, the object position calculation unit 14 calculates the position coordinates (uL, vL) of the intersection L and the position coordinates (uR, vR) of the intersection R. After that, the object position calculation unit 14 shifts to S32 shown in FIG.

In S32, the object position calculation unit 14 calculates the position of the intersection L on the map (xL, yL) and the position of the intersection R on the map (xR, yR). Here, FIG. 7B is a plan view for explaining how to obtain the positions of the intersections L and R on the map. In FIG. 7B, the center in the width direction of the rear wheel of the object N, which is the preceding vehicle, corresponds to the center C of the lower side in the captured image. Since the y-axis direction (horizontal direction) on the map and the u-axis direction on the captured image are the same, the straight line E is represented as a straight line extending in the y-axis direction through the lower side center C. Further, as shown in FIG. 7B, the intersection point L is located on the same line segment K1 as the white line point a1 and the white line point a2 on the map. Similarly, on the map, the intersection R is located on the same line segment K2 as the white line point a4 and the white line point a5.

FIG. 8A is a graph for explaining how to obtain the x-axis coordinates of the intersection L. In the graph of FIG. 8A, the vertical axis is the v-axis and the horizontal axis is the x-axis. As shown in FIG. 8A, a curve x = f (v) passing through the white line points a1 to a3 based on the position information of the white line points a1 to a3 (position information stored in the landmark information storage unit 3). ) Can be expressed by a cubic expression of v. Lagrange interpolation, Newton polynomial, and the like can be used for this cubic equation. By substituting the v-axis coordinates (vL) of the intersection L into this cubic equation, the x-axis coordinates (xL) of the intersection L can be obtained. The x-axis coordinate (xR) of the intersection R is equal to the x-axis coordinate (xL) of the intersection L.

FIG. 8B is a graph for explaining how to obtain the y-axis coordinates of the intersection L. In the graph of FIG. 8B, the vertical axis is the y-axis and the horizontal axis is the u-axis. In this case, as shown in FIG. 8B, the white line points a1 to a3 and the intersection point L have a substantially linear relationship. Therefore, the y-axis coordinates (yL) of the intersection L can be obtained by linear interpolation using the following equation (3).
yL = (uL-u2) / (u1-u2) x y1 + (u2-u1) / (u2-u1) x y2 ... (3)

Similarly, the y-axis coordinates (yR) of the intersection R can be obtained by linear interpolation using the following equation (4).
yR = (uR-u5) / (u4-u5) x y4 + (uR-u4) / (u5-u4) x y5 ... (4)

In this way, the object position calculation unit 14 calculates the position of the intersection L on the map (xL, yL) and the position of the intersection R on the map (xR, yR). After that, the object position calculation unit 14 shifts to S34 shown in FIG.

In S34, the object position calculation unit 14 calculates the position of the object N on the map. The object position calculation unit 14 includes the position coordinates (uC, vC) of the lower side center C of the object N on the captured image, the position coordinates (uL, vL) of the intersection L on the captured image, and the position coordinates of the intersection R on the captured image. The position of the lower center C of the object N on the map is calculated based on (uR, vR), the position of the intersection L on the map (xL, yL), and the position of the intersection R on the map (xR, yR). To do.

Here, considering a two-dimensional plane including the x-axis direction on the map and the u-axis direction on the captured image, the straight line E connecting the intersection L and the intersection R is substantially a straight line on the map. Therefore, the x-axis coordinates of the lower side center C of the object N can be obtained by linear interpolation using the following equation (5).
xC = (uC-uR) / (uL-uR) x xL + (uC-uL) / (uR-uL) x xR ... (5)

Similarly, considering a two-dimensional plane including the y-axis direction on the map and the u-axis direction on the captured image, the y-axis coordinates of the lower side center C of the object N are obtained by linear interpolation using the following equation (6). Can be sought.
yC = (uC-uR) / (uL-uR) x yL + (uC-uL) / (uR-uL) x yR ... (6)

In this way, the object position calculation unit 14 can calculate the position (xC, yC) of the lower side center C of the object N on the map.

[Action and effect of object recognition device]
According to the object recognition device 100 according to the present embodiment described above, the position coordinates of the object N and the position coordinates of the landmark on the captured image are recognized, and the position information of the landmark on the map stored in advance is used. Therefore, the position of the object N on the map can be calculated. Therefore, according to this object recognition device 100, since the position information of the landmark on the map stored in advance is used, the distance between the vehicle and the landmark can be determined from the size (width) and position of the landmark on the captured image. Unlike the conventional device that calculates the position of an object using the obtained distance, the calculation accuracy of the position does not decrease as the landmark and the object move in the vertical direction on the captured image, and the object on the map is calculated from the captured image. The position of N can be calculated accurately.

Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment. The present invention can be carried out in various forms in which various modifications and improvements have been made based on the knowledge of those skilled in the art, including the above-described embodiment. For example, the image captured by the monocular camera 1 capturing the front of the vehicle M has been described as an example, but the present invention can also be applied to captured images of the rear of the vehicle, the right side of the vehicle, and the left side of the vehicle. is there.

[Example when the landmark is a road sign or road marking]
Hereinafter, an example in which the landmark is a road sign or a road marking will be described. FIG. 9A is a diagram for explaining an example of how to obtain the position of an object using three landmarks. Here, a case where three landmarks are arranged so as to surround the object N on the captured image will be described. As shown in FIG. 9A, when a road sign or a road marking is used as a landmark instead of a white line, the position coordinates of the landmark can be shown as a point (for example, the root of the road sign, the center of the road marking). ..

In FIG. 9A, points b1 to b3 of the landmark, object N, the center C of the lower side of the object N, a straight line E extending the lower side of the object N, a straight line segment K10 connecting the points b1 and b2, and a point. The linear line segment K11 connecting b2 and the point b3, the intersection L of the straight line E and the line segment K10, and the intersection R of the straight line E and the line segment K11 are shown. The landmark points b1 to b3 are recognized by the landmark recognition unit 12 as the position coordinates (u-axis coordinates, v-axis coordinates) on the captured image. Further, the position information (x-axis coordinates, y-axis coordinates) on the map of the landmark points b1 to b3 is stored in the landmark information storage unit 3. Further, the position coordinates (uC, vC) on the captured image are recognized by the object recognition unit 13 at the lower side center C of the object N.

In the situation shown in FIG. 9A, the object position calculation unit 14 uses linear interpolation to obtain the position coordinates (uL, vL) of the intersection L on the captured image by the same procedure as when the landmark is a white line. calculate. In the above equation (1), the object position calculation unit 14 adopts the position coordinates of the point b1 instead of the position coordinates of the white line point a1, and adopts the position coordinates of the point b2 instead of the position coordinates of the white line point a2. Therefore, the u-axis coordinates (uL) of the intersection L can be obtained. The v-axis coordinate (vL) of the intersection L is the same as the v-axis coordinate (vC) of the lower side center C of the object N. Similarly, the object position calculation unit 14 calculates the position coordinates (uR, vR) of the intersection R on the captured image.

Next, the object position calculation unit 14 calculates the position (xL, yL) of the intersection L on the map by the same procedure as when the landmark is a white line. In the above equation (3), the object position calculation unit 14 adopts the position information of the point b1 instead of the position information of the white line point a1, and adopts the position information of the point b2 instead of the position information of the white line point a2. Therefore, the y-axis coordinates (yL) of the intersection L can be calculated. Further, the object position calculation unit 14 calculates the x-axis coordinates (xL) of the intersection L by the same linear interpolation. By replacing y with x and u with v in the above equation (3), the x-axis coordinates (xL) of the intersection L can be calculated. Similarly, the object position calculation unit 14 calculates the position (xR, yR) of the intersection R on the map.

After that, the object position calculation unit 14 uses the same procedure as when the landmark is a white line, based on the position of the intersection L on the map (xL, yL) and the position of the intersection R on the map (xR, yR). The position (xC, yC) of the lower side center C of the object N on the map is calculated.

FIG. 9B is a diagram for explaining another example of how to obtain the position of an object using three landmarks. 9 (b) shows the landmark points b11 to b13, the object N, the lower side center C of the object N, the straight line E extending the lower side of the object N, the straight line segment K20 connecting the points b11 and b12, and the points. The linear line segment K21 connecting b12 and the point b13, the intersection L of the straight line E and the line segment K20, and the intersection R of the straight line E and the line segment K21 are shown. Since the intersection L and the intersection R can be obtained even in the situation shown in FIG. 9B, the position (xC, yC) of the lower side center C of the object N on the map can be calculated by the above procedure.

In addition, the ECU 10 may store in advance map data in which the position coordinates on the captured image and the position (spatial position) on the map with respect to the vehicle M are associated with each other. In this case, the object position calculation unit 14 calculates the position of the object on the map from the position coordinates of the object on the captured image and the position of the vehicle on the map. At this time, the object position calculation unit 14 uses the position coordinates of the landmark on the captured image and the position information of the landmark on the map to position the position coordinates of the object on the captured image and the position of the landmark on the captured image. By correcting the calculation result of the position of the object on the map from the relationship of the coordinates, the position of the object can be calculated with high accuracy.

1 ... Monocular camera, 2 ... Map information storage unit, 3 ... Landmark information storage unit, 4 ... GPS receiver unit, 11 ... Vehicle position recognition unit, 12 ... Landmark recognition unit, 13 ... Object recognition unit, 14 ... Object position Calculation unit, 100 ... object recognition device, a1-a6 ... white line point, C ... lower side center, d ... difference, E ... straight line, L, R ... intersection, M ... vehicle, N ... object, w1, w2, W1, W2 …white line.

Claims (1)

An object recognition device that recognizes objects around the vehicle based on the captured image obtained by imaging the surroundings of the vehicle.
A vehicle position recognition unit that recognizes the position of the vehicle on the map,
A storage unit that stores the location information of landmarks on the map,
A landmark recognition unit that recognizes the position coordinates of the landmark on the captured image based on the captured image, the position of the vehicle on the map, and the position information of the landmark on the map.
An object recognition unit that recognizes the position coordinates of the object on the captured image based on the captured image.
Based on the position coordinates of the object on the captured image, the position coordinates of the landmark on the captured image, and the position information of the landmark on the map , the landmark and the object on the same captured image. An object position calculation unit that calculates the position of the object on the map using the positional relationship of
An object recognition device equipped with.

Images