JP2021148730A - Position estimation method, position estimation device, and program - Google Patents

Abstract

To provide a position estimation method and the like capable of accurately estimating the position of an object even when using image data captured by a monocular camera.SOLUTION: A position estimation method includes the steps of: acquiring image data including an object from a monocular camera 20(S101); converting first coordinates of the object in the image data into second coordinates in the real world (S103); calculating the size of the object based on the second coordinates (S104); determining whether or not the size is included in a predetermined range according to the object (S105); and when the size is not included in the specified range (No in S105), converting again the first coordinates into third coordinates in the real world based on representative values within the specified range (S108).SELECTED DRAWING: Figure 8

Description

The present disclosure relates to position estimation methods, position estimation devices, and programs.

In recent years, the number of vehicles equipped with collision damage reduction brakes has increased in order to prevent accidents while driving, and it is expected that the number will increase in the future. In order to realize such a collision damage reduction brake, an object detection device that detects an object around a vehicle by using image data captured by an in-vehicle camera or the like is known. Since the traveling of the vehicle is controlled based on the result of the object detection device detecting the object, it is desired that the detection accuracy of the object detection device is high.

Patent Document 1 discloses a three-dimensional object recognition device that estimates the position of a three-dimensional object from a plurality of image data (two-dimensional image data) obtained by imaging an object having a known three-dimensional shape. The three-dimensional object recognition device holds the three-dimensional shape as wire frame data, and generates a virtual real image by pasting the captured real image on the wire frame data.

Japanese Patent No. 3456029

However, in the technique of Patent Document 1, when the position of the object is estimated based on the image data captured by the monocular camera, the position of the object may not be estimated accurately.

Therefore, the present disclosure relates to a position estimation method, a position estimation device, and a program capable of accurately estimating the position of an object even when using image data captured by a monocular camera.

In the position estimation method according to one aspect of the present disclosure, image data including an object is acquired from a monocular camera, the first coordinates of the object in the image data are converted into second coordinates in the real world, and the first coordinate is converted. When the size of the object is calculated based on the two coordinates, it is determined whether or not the size is included in the predetermined range according to the object, and the size is not included in the predetermined range. , The first coordinate is converted back to the third coordinate in the real world based on the representative value within the predetermined range.

The position estimation device according to one aspect of the present disclosure is an acquisition unit that acquires image data including an object from a monocular camera, and converts the first coordinates of the object in the image data into second coordinates in the real world. A first conversion unit, a calculation unit that calculates the size of the object based on the second coordinates, and a determination unit that determines whether or not the size is included in a predetermined range corresponding to the object. And a second conversion unit that reconverts the first coordinate to the third coordinate based on the representative value within the predetermined range when the size is not included in the predetermined range.

The program according to one aspect of the present disclosure is a program for causing a computer to execute the above-mentioned position estimation method.

According to the position estimation method or the like according to one aspect of the present disclosure, the position of the object can be accurately estimated even when the image data captured by the monocular camera is used.

FIG. 1A is a diagram for explaining the estimation of the position of an object in the position estimation device according to the comparative example. FIG. 1B is a second diagram for explaining the estimation of the position of the object in the position estimation device according to the comparative example. FIG. 2 is a block diagram showing a functional configuration of a vehicle including the position estimation device according to the first embodiment. FIG. 3 is a diagram showing a detection result of an object by the detection unit according to the first embodiment. FIG. 4 is a diagram for explaining the coordinate conversion of the first conversion unit according to the first embodiment. FIG. 5 is a diagram for explaining a predetermined range according to the first embodiment. FIG. 6 is a diagram showing an estimation result of the size of the object in the past frame according to the first embodiment. FIG. 7 is a diagram showing an example of position information output by the position estimation device according to the first embodiment. FIG. 8 is a flowchart showing the operation of the position estimation device according to the first embodiment. FIG. 9 is a table including standard values of the sizes of the objects stored in the storage unit according to the second embodiment. FIG. 10 is a flowchart showing the operation of the position estimation device according to the second embodiment.

(Background to this disclosure)
In recent years, various studies have been conducted on an object detection device that detects an object around a vehicle by using image data captured by an in-vehicle camera or the like. For example, studies are being conducted to estimate the position of an object based on image data captured by a monocular camera (see FIG. 3 described later). By estimating the position of the object using a monocular camera, the position of the object can be estimated even if the vehicle does not have a plurality of cameras. That is, the position of the object can be estimated at a lower cost.

The estimation of the position of the object based on the image data captured by the monocular camera will be described with reference to FIGS. 1A and 1B. FIG. 1A is a diagram for explaining the estimation of the position of an object in the position estimation device according to the comparative example. FIG. 1A shows a case where a pedestrian U in contact with a road L (ground) is in front of a vehicle 10 equipped with a monocular camera 20. Further, the vehicle 10 is in contact with the road L. It can be said that FIG. 1A shows a case where the pedestrian U is in contact with the same plane as the plane in which the vehicle 10 is in contact. Pedestrian U is an example of an object.

As shown in FIG. 1A, the monocular camera 20 of the vehicle 10 images a pedestrian U in front of the vehicle 10. The position estimation device (not shown) included in the vehicle 10 estimates the position of the pedestrian U based on the image data captured by the monocular camera 20. Here, the position estimation device estimates the position of the pedestrian U on the premise that the pedestrian U shown in the captured image data is on the same plane as the plane in which the vehicle 10 is in contact. Specifically, the position estimation device converts the coordinates on the image data of the pedestrian U into the coordinates in the real world on the assumption that the pedestrian U and the vehicle 10 are on the same plane. Such a premise is used when estimating the position using the monocular camera 20. The coordinates on the image data are the coordinates of the camera coordinate system, and the coordinates in the real world are the coordinates of the Cartesian coordinate system.

In the case of FIG. 1A, since the state in which the pedestrian U is in contact with the road L and the premise of the coordinate conversion in the position estimation device match, the position estimation device accurately estimates the position of the pedestrian U. can do. The position estimation device can estimate that the pedestrian U is at the position P1. In this case, the pedestrian U in contact with the road L is imaged on the image plane of the monocular camera 20. The image plane is an imaging surface of the monocular camera 20.

Next, a case where the pedestrian U and the vehicle 10 are not on the same plane will be described. FIG. 1B is a second diagram for explaining the estimation of the position of the object in the position estimation device according to the comparative example. FIG. 1B shows a case where a pedestrian U who is not in contact with the road L (ground) is in front of the vehicle 10 equipped with the monocular camera 20. FIG. 1B shows a case where the pedestrian U is jumping.

In the case of FIG. 1B, since the state in which the pedestrian U is not in contact with the road L and the premise of the coordinate conversion in the position estimation device do not match, the position estimation device accurately estimates the position of the pedestrian U. Can not do it. That is, the position estimation device erroneously estimates the position of the pedestrian U. The position estimation device estimates the position of the pedestrian U on the assumption that the pedestrian U and the vehicle 10 are on the same plane even when the pedestrian U and the vehicle 10 are not on the same plane. Is. The position estimation device estimates that the position of the pedestrian U actually at the position P1 is the position P2 farther than the position P1 (see the pedestrian M shown in FIG. 1B). Further, in this case, the position estimation device estimates the size of the pedestrian U to be larger than the original size (see the pedestrian M shown in FIG. 1B).

Further, even if the pedestrian U is in contact with the road L, if the tire of the vehicle 10 is riding on an object such as a manhole, the pedestrian U in contact with the road L and the vehicle 10 riding on the object It will not be on the same plane. Therefore, the position estimation device accurately estimates the position of the pedestrian U even when the tire of the vehicle 10 is riding on an object such as a manhole, as in the case where the pedestrian U is not in contact with the road L. Can not do it.

As described above, in the position estimation device that estimates the position of the pedestrian U using the image data of the monocular camera 20, when the relative positional relationship between the pedestrian U and the own vehicle changes, the pedestrian with respect to the own vehicle. The estimation of the position of U may be incorrect. The position estimation device is used when, for example, the object is performing irregular movements such as jumping, or when the tire of the vehicle 10 is riding on the object or the like is performing irregular behavior. It may not be possible to accurately estimate the position of an object. Similarly, in Patent Document 1 described above, the estimation of the position of the pedestrian U may be erroneous. Therefore, the inventor of the present application has diligently studied a position estimation method and the like capable of accurately estimating the position of an object even when using image data captured by a monocular camera, and will be described below. The position estimation method was devised.

In the position estimation method according to one aspect of the present disclosure, image data including an object is acquired from a monocular camera, the first coordinates of the object in the image data are converted into second coordinates in the real world, and the first coordinate is converted. When the size of the object is calculated based on the two coordinates, it is determined whether or not the size is included in the predetermined range according to the object, and the size is not included in the predetermined range. , The first coordinate is converted back to the third coordinate in the real world based on the representative value within the predetermined range.

When performing coordinate conversion based on the image data captured by the monocular camera, the first coordinates are based on the image data captured when the object is performing an action such as jumping or the vehicle is riding on an object. Is converted to the second coordinate, but an accurate second coordinate cannot be obtained. The second coordinate is farther than it should be. That is, the exact position of the object cannot be estimated. Further, when the position of the object is not accurate in this way, the size of the object cannot be estimated accurately.

Therefore, in the position estimation method according to one aspect of the present disclosure, when the size is not included in the predetermined range, the coordinate conversion is performed again based on the representative value. Since the third coordinate is a coordinate that has been coordinate-transformed based on a representative value corresponding to the object, it can be a coordinate closer to the real-world coordinate of the object than the second coordinate. Therefore, even when the image data captured by the monocular camera is used, the position of the object can be accurately estimated.

Further, for example, the representative value may be a standard value preset according to the object.

As a result, the first coordinate is converted to the third coordinate based on the standard value corresponding to the object, so that the coordinate is converted using a constant standard value regardless of the object, as compared with the case where the coordinate is converted using a constant standard value regardless of the object. The position can be estimated accurately.

Further, for example, the standard value is set for each class of the object, and when the size is not included in the predetermined range, the standard value corresponding to the class to which the object belongs is represented. It may be acquired as a value, and the first coordinate may be converted back to the third coordinate based on the acquired representative value.

As a result, the first coordinate can be converted to the third coordinate using the standard value for each label, so that the position of the object can be estimated more accurately.

Further, for example, the representative value may be determined based on the time series data of the size of the object based on the image data acquired in the past.

As a result, since the time series data of the object is used, the representative value can be set more appropriately. Therefore, the position of the object can be estimated more accurately.

Further, for example, when the size is included in the predetermined range, the size of the object may be updated based on the size in the current frame and the size of the object in the past frame.

This makes it possible to obtain a more accurate size of the object. For example, the position of the object can be estimated more accurately by performing the coordinate transformation again using the size.

Further, the position estimation device according to one aspect of the present disclosure uses an acquisition unit that acquires image data including an object from a monocular camera, and the first coordinates of the object in the image data as second coordinates in the real world. A first conversion unit to be converted, a calculation unit that calculates the size of the object based on the second coordinates, and a determination as to whether or not the size is included in a predetermined range corresponding to the object. A determination unit and a second conversion unit that reconverts the first coordinate to the third coordinate based on a representative value within the predetermined range when the size is not included in the predetermined range are provided. Further, the program according to one aspect of the present disclosure is a program for causing a computer to execute the above-mentioned position estimation method.

As a result, the same effect as that of the above-mentioned position estimation method is obtained.

It should be noted that these general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a non-temporary recording medium such as a computer-readable CD-ROM, and the system, the method, the integrated. It may be realized by any combination of circuits, computer programs or recording media. The program may be stored in the recording medium in advance, or may be supplied to the recording medium via a wide area communication network including the Internet or the like.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that all of the embodiments described below show comprehensive or specific examples. Numerical values, shapes, components, arrangement positions and connection forms of components, steps, step order, and the like shown in the following embodiments are examples, and are not intended to limit the present disclosure. For example, a numerical value is not an expression that expresses only a strict meaning, but an expression that includes a substantially equivalent range, for example, a difference of about several percent. Further, among the components in the following embodiments, the components not described in the independent claims will be described as arbitrary components. Further, each figure is a schematic view and is not necessarily exactly illustrated. Further, in each figure, the same components are designated by the same reference numerals.

(Embodiment 1)
Hereinafter, a vehicle provided with the position estimation device according to the present embodiment will be described with reference to FIGS. 2 to 8.

[1-1. Vehicle configuration]
First, the configuration of the vehicle according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing a functional configuration of a vehicle 10 including the position estimation device 30 according to the present embodiment.

As shown in FIG. 2, the vehicle 10 includes a monocular camera 20 and a position estimation device 30.

The monocular camera 20 is mounted on the vehicle 10 and images the surroundings of the vehicle 10. The monocular camera 20 is, for example, a small vehicle-mounted camera (vehicle-mounted monocular camera) mounted near the center position of the vehicle width in front of the vehicle 10. The monocular camera 20 may be mounted on the ceiling near the windshield in the vehicle, for example. In this case, the imaging range of the monocular camera 20 is a range including a part of the bonnet in front of the vehicle 10. Further, the monocular camera 20 may be attached to the tip of the bonnet, for example. Further, the monocular camera 20 may be attached so as to be able to take an image of the rear or side of the vehicle 10.

The monocular camera 20 is not particularly limited, and a known monocular camera can be used. The monocular camera 20 is, for example, a general visible light camera that captures light having a wavelength in the visible light region, but may be a camera capable of acquiring infrared light information. Further, the monocular camera 20 may capture images at a wide angle, for example. The monocular camera 20 may be, for example, a fisheye camera having a fisheye lens. Further, the monocular camera 20 may be a monochrome camera that captures a monochrome image, or may be a color camera that captures a color image.

The monocular camera 20 outputs the captured image data to the position estimation device 30. The monocular camera 20 is an example of an imaging device. The image data is two-dimensional image data.

The position estimation device 30 estimates the position of the object based on the image data acquired from the monocular camera 20. The position estimation device 30 is a three-dimensional position estimation device that estimates the three-dimensional position of an object in the real world based on image data. The position estimation device 30 includes a detection unit 40, a control unit 50, and a storage unit 60.

The detection unit 40 detects an object to be detected based on the image data acquired from the monocular camera 20. Hereinafter, an example in which the object to be detected by the detection unit 40 is a person (for example, a pedestrian U) will be described, but the object is not limited to the person. The detection unit 40 functions as an acquisition unit that acquires image data including the pedestrian U from the monocular camera 20.

The detection unit 40 uses image data as input and detects a person using a trained model trained to output a detection frame that detects a person appearing in the image data. The trained model is generated, for example, by machine learning with supervised data. The trained model is generated by using machine learning using the image data of a person as training data and the detection frame (coordinates of the detection frame) of the person in the image data as correct answer data. Further, the trained model is generated by deep learning using, for example, a neural network. That is, it can be said that the detection unit 40 is a deep learning person detection unit. The detection unit 40 outputs a detection frame list, which is a list of detected detection frames, to the control unit 50. The detection frame list is an example of the detection result.

The detection frame list includes the number of detection frames detected by the detection unit 40, the coordinate information on the image data of each detection frame, and the reliability of each detection frame. The reliability can be, for example, a value from 0 to 1. The detection unit 40 is learned so that it can detect a pedestrian U who is walking or jumping. The detection unit 40 is learned so that it can detect each of a pedestrian U in contact with the road L and a pedestrian U not in contact with the road L. Further, the detection frame list may include the class of each detection frame. The class indicates the classification of the object that can be detected by the detection unit 40. The class is appropriately determined according to the object on which the position estimation device 30 is mounted. When the position estimation device 30 is mounted on the vehicle 10, the class is "person", "car", etc., and the class of the pedestrian U is, for example, "person". The object is a moving body such as a vehicle 10, but it may be a device used in a fixed manner or a device used in a stationary manner.

The number of detection frames included in the detection frame list may be zero. In this case, the detection frame list contains information indicating that the number of detection frames is zero.

Here, the detection frame will be described with reference to FIG. FIG. 3 is a diagram showing a detection result of an object by the detection unit 40 according to the present embodiment.

As shown in FIG. 3, the detection unit 40 outputs a detection frame list including the coordinate information of the detection frame F of the pedestrian U to the control unit 50 based on the image data D acquired from the monocular camera 20. When the pedestrian U is in contact with the road L, the detection unit 40 outputs a detection frame list including the coordinate information of the detection frame F surrounding the pedestrian U in contact with the road L to the control unit 50. Further, when the pedestrian U is not in contact with the road L, the detection unit 40 obtains the coordinate information of the detection frame F surrounding the pedestrian U (for example, the jumping pedestrian U) that is not in contact with the road L. The detection frame list including the detection frame is output to the control unit 50. The coordinate information includes, for example, the coordinates Pb1 (u1, v1) and Pb2 (u2, v2) of the diagonal points of the detection frame F.

With reference to FIG. 2 again, the control unit 50 estimates the position of the object based on the detection frame list. The control unit 50 according to the present embodiment determines whether or not the pedestrian U is in contact with the road L, and estimates the position of the pedestrian U based on the determination result. The control unit 50 includes a first conversion unit 51, a calculation unit 52, a determination unit 53, and a second conversion unit 54.

When the first conversion unit 51 acquires the detection frame list from the detection unit 40, the first conversion unit 51 converts the coordinates of each detection frame included in the detection frame list. Specifically, the first conversion unit 51 converts the coordinates of each detection frame from the coordinates on the image data to the coordinates in the real world (real space). It can be said that the first conversion unit 51 converts the coordinates of the pedestrian U in the camera coordinate system into the coordinates of the pedestrian U in the Cartesian coordinate system. The coordinates on the image data are an example of the first coordinates, and the coordinates in the real world obtained by the coordinate conversion by the first conversion unit 51 are an example of the second coordinates.

The method by which the first conversion unit 51 performs coordinate conversion is not particularly limited, but is performed by, for example, the following method. FIG. 4 is a diagram for explaining the coordinate conversion of the first conversion unit 51 according to the present embodiment. The coordinates Pa (X, Y, Z) shown in FIG. 4 are the coordinates of the pedestrian U in the real world, and indicate three-dimensional points (three-dimensional coordinates) in the Cartesian coordinate system. Further, the x-axis, y-axis and z-axis are orthogonal to each other, forming a three-dimensional Cartesian coordinate system. The z-axis is an axis in a direction parallel to the optical axis of the monocular camera 20, and each of the x-axis and the y-axis is an axis orthogonal to the z-axis. The intersection of the xy plane and the optical axis is the origin in the Cartesian coordinate system.

Further, the Xc axis, the Yc axis, and the Zc axis are orthogonal to each other and form a camera coordinate system. The Zc axis is an axis in a direction parallel to the optical axis of the monocular camera 20, and each of the Xc axis and the Yc axis is an axis orthogonal to the Zc axis. The intersection of the XcYc plane and the optical axis is the origin Fc in the camera coordinate system.

The coordinates of the point where the straight line connecting the coordinate Pa and the origin Fc of the monocular camera 20 and the image plane intersect are defined as the coordinates Pb (uc, vc). The coordinates Pb (uc, vc) are the coordinates in the camera coordinate system.

At this time, the coordinates Pb (uc, vc) can be calculated by the following equations 1 and 2.

uc = (X × fx) / Z + cx (Equation 1)
vc = (Y × fy) / Z + cy (Equation 2)

Here, fx and fy are focal lengths expressed in pixel units, fx is a focal length in the x-axis direction, and fy is a focal length in the y-axis direction. Further, cx and cy are image centers (principal points). The center of the image is the point where the optical axis of the monocular camera 20 and the image plane intersect.

Using the coordinates Pb1 (u1, v1) of the upper left corner of the rectangular frame and the coordinates Pb2 (u2, v2) of the lower right corner on the image data shown in FIG. 3, the image data of the rectangular frame can be used. The area size can be calculated by the following formula 3.

Simg = | u1-u2 | × | v1-v2 | (Equation 3)

The coordinates in the real world corresponding to the coordinates Pb1 on the image data are the coordinates Pa1 (X1, Y1, Z1), and the coordinates in the real world corresponding to the coordinates Pb2 on the image data are the coordinates Pa2 (X2, Y2, Assuming that Z2) and the depth Z1 = Z2 = Z, the area quad of the detection frame in the real world is calculated by the following equation 4.

World = | X1-X2 | × | Y1-Y2 |
= | Z (u1-cx) / fx-Z (c2-cx) / fx | ×
| Z (v1-cy) / fy-Z (v2-cy) / fy |
= (Z ² x Sigma) / (fx x fy) (Equation 4)

The area Round can be calculated assuming that the width is about 50 cm and the height (height) is about 170 cm when the object is a person (for example, an adult person). Therefore, the value of the depth Z, which is the remaining unknown in Equation 4, can be calculated. The depth Z indicates the distance from the image plane of the monocular camera 20 to the object in the direction parallel to the optical axis. Then, the coordinates Pa1 and Pa2 can be calculated based on the value of the depth Z and the coordinates Pb1 and Pb2.

Although the example of calculating the depth Z value using the area of the detection frame has been described above, it is also possible to calculate the depth Z value using either the height or the width of the detection frame. For example, in the case of the vehicle 10, the width of the object differs depending on whether the object is viewed from the front or from the side, but the height is approximately the same. Therefore, in the following, a case where the value of the depth Z is calculated using the height will be described. The height Hworld of the detection frame in the real world is calculated by the following equation 5 assuming that the height of the detection frame on the image data is Himg.

Hworld = | Y1-Y2 | = | Z (v1-cy) / fy-Z (v2-cy) / fy | = (Z × Himg) / fy (Equation 5)

Here, the height Himg of the detection frame on the image data is calculated by the following formula 6.

Himg = | v1-v2 | (Equation 6)

As shown in Equations 5 and 6, if the height Hworld of the detection frame in the real world is known, the value of the depth Z can be calculated.

With reference to FIG. 2 again, the calculation unit 52 calculates the size of the pedestrian U based on the coordinates converted by the first conversion unit 51, that is, the coordinates in the real world. The calculation unit 52 calculates, for example, the height of the pedestrian U. The size of the pedestrian U may be the width of the pedestrian U or the area.

The determination unit 53 determines whether or not the size of the pedestrian U in the real world is included in the predetermined range corresponding to the pedestrian U. In the present embodiment, the determination unit 53 determines a predetermined range based on the time-series data of the size of the pedestrian U in the real world based on the image data acquired from the monocular camera 20 in the past. That is, the predetermined range is determined for each pedestrian U (for each object). It can be said that the determination unit 53 determines whether or not the accurate position of the pedestrian U has been acquired based on the size of the pedestrian U. The time-series data includes the size of the pedestrian U in each of the plurality of frames, which is calculated based on the image data acquired in the plurality of frames before the current frame. The time series data (for example, see FIG. 7 described later) is stored in the storage unit 60. FIG. 5 is a diagram for explaining a predetermined range according to the present embodiment. The vertical axis of FIG. 5 indicates the size, and the horizontal axis indicates the time.

As shown in FIG. 5, the determination unit 53 determines a predetermined range based on the size of the pedestrian U in the past several frames. The determination unit 53 determines a predetermined range based on, for example, the average value and standard deviation of the sizes of the pedestrian U. For example, assuming that the average value of the sizes of the pedestrians U in the past several frames is Savg and the standard deviation is σ, the determination unit 53 determines a predetermined range based on the following equations 7 and 8.

Upper limit = Savg + k × σ (Equation 7)
Lower limit = Savg-k x σ (Equation 8)

In this case, the determination unit 53 determines a range equal to or greater than the lower limit value calculated by the equation 8 and less than or equal to the upper limit value calculated by the equation 7 as a predetermined range. Note that k is a preset constant.

With reference to FIG. 2 again, when the size of the pedestrian U in the current frame is within the predetermined range, the determination unit 53 determines that the pedestrian U is in contact with the road L in the current frame. It can be said that the determination unit 53 determines that the state in which the pedestrian U is in contact with the road L matches the premise of the coordinate conversion in the position estimation device 30. Further, when the size of the pedestrian U in the current frame is out of the predetermined range, the determination unit 53 determines that the pedestrian U is not in contact with the road L in the current frame. When the size of the pedestrian U in the current frame is out of the predetermined range, the state in which the pedestrian U is not in contact with the road L and the premise of the coordinate conversion in the position estimation device 30 do not match. The coordinates after coordinate conversion by the first conversion unit 51 in the current frame, that is, the coordinates in the real world may not be accurate values. The determination unit 53 outputs the determination result to the second conversion unit 54.

The second conversion unit 54 determines that the size of the pedestrian U in the current frame is not included in the predetermined range by the determination unit 53, that is, the state in which the pedestrian U is not in contact with the road L. If the premise of the coordinate conversion in the position estimation device 30 does not match, the coordinates of the pedestrian U in the image data are converted back to the coordinates of the pedestrian U in the real world based on the values within the predetermined range. The second conversion unit 54 reconverts the coordinates of the detection frame from the detection unit 40 to the coordinates of the detection frame in the real world based on the representative value. The second conversion unit 54 reconverts the pedestrian U into the coordinates of the detection frame in the real world, assuming that the size of the pedestrian U in the real world is the value of the representative value. In the present embodiment, the representative value is, for example, a value between the upper limit value calculated by the formula 7 and the lower limit value calculated by the formula 8, and is, for example, in the real space of several frames in the past. Average size, median, mode, etc. That is, the representative value is determined based on the time series data of the size of the object in the real world based on the image data acquired in the past. Further, the coordinates in the real world obtained by the coordinate conversion by the second conversion unit 54 are an example of the third coordinates.

Further, the second conversion unit 54 may convert the coordinates of the detection frame in the real world into the coordinates when the vehicle 10 is viewed from a bird's-eye view. It can be said that the second conversion unit 54 converts the viewpoint from the viewpoint of the monocular camera 20 to the viewpoint based on the bird's-eye view. The second conversion unit 54 may calculate the two-dimensional coordinates when the bird's-eye view is taken as the coordinates when the bird's-eye view is taken.

The storage unit 60 is a storage device that stores various information for processing performed by each processing unit of the position estimation device 30. The storage unit 60 stores, for example, the size information of the pedestrian U in the past frame, the information shown in each equation 1 to 8, various coefficients, and the like. The storage unit 60 is realized by, for example, a semiconductor memory or the like.

Here, the estimation result of the size of the pedestrian U in the past frame stored in the storage unit 60 will be described with reference to FIG. FIG. 6 is a diagram showing the estimation result of the size of the object in the past frame according to the present embodiment.

As shown in FIG. 6, the estimation result is a table in which the time and the size of the pedestrian U are associated with each other. The time interval is not particularly limited. Further, the size of the pedestrian U is a value including a measurement error and the like, and variations occur for each frame. In FIG. 6, the table includes the size of the pedestrian U at the present (eg, 8:10) in the last 4 frames (eg, 8: 06-8: 9).

The control unit 50 stores in the storage unit 60 the size of the pedestrian U calculated by the calculation unit 52 in the real world as a table shown in FIG. For example, when the size of the pedestrian U does not change with time, the control unit 50 determines that the size is the size when the pedestrian U is in contact with the road L, and stores the size in the storage unit 60. Remember. For example, when the size of the pedestrian U takes a similar value in several frames, the control unit 50 stores the size as the size of the pedestrian U in the storage unit 60. The similar value may be, for example, a value within a range set based on an error that may occur due to the performance (for example, resolution) of the monocular camera 20, the processing by the control unit 50, or the like. , The value may be within a preset range.

The position information output from the position estimation device 30 as described above will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of position information output by the position estimation device 30 according to the present embodiment. The position information includes the position estimation result of the pedestrian U with respect to the vehicle 10 estimated by the position estimation device 30. Specifically, the position information includes the result of the coordinate conversion of the first conversion unit 51 or the second conversion unit 54. Although FIG. 6 describes an example in which the position information is shown in the bird's-eye view, the position information is not limited to being shown in the bird's-eye view. Further, in FIG. 7, it is assumed that the monocular camera 20 is arranged at the tip of the bonnet of the vehicle 10.

As shown in FIG. 7, the position information includes the position of the pedestrian U with respect to the monocular camera 20. When the monocular camera 20 is mounted on the vehicle 10, the position information includes the position of the pedestrian U with respect to the vehicle 10. FIG. 7 shows an example in which the position estimation device 30 estimates that the pedestrian U is located 7 m in front of the vehicle 10 and 8 m on the left side of the vehicle 10.

The position information may further include the speed of the pedestrian U calculated based on the time series data of the position of the pedestrian U.

[1-2. Operation of position estimator]
Subsequently, the operation of the position estimation device 30 according to the present embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing the operation of the position estimation device 30 according to the present embodiment.

As shown in FIG. 8, the detection unit 40 of the position estimation device 30 acquires the image data captured by the monocular camera 20 (S101). The detection unit 40 acquires image data including an object from, for example, the monocular camera 20. When the detection unit 40 acquires the image data, the detection unit 40 executes a process of detecting whether the image data includes an object (pedestrian U in the present embodiment). Specifically, the detection unit 40 inputs image data to the trained model and acquires the coordinate information and reliability of the detection frame which is the output of the trained model. Then, the detection unit 40 outputs a detection frame list including the coordinate information of the detection frame and the reliability to the control unit 50.

Next, when the control unit 50 acquires the detection frame list from the detection unit 40, the control unit 50 determines whether or not the detection unit 40 has detected the object (S102). The control unit 50 determines Yes in step S102 when the number of detection frames included in the detection frame list is 1 or more, and determines No in step S102 when the number of detection frames is 0.

Next, when the number of detection frames included in the detection frame list is 1 or more (Yes in S102), the first conversion unit 51 converts the coordinates on the image data of each detection frame into the coordinates in the real world. (S103). The first conversion unit 51 performs coordinate conversion on the assumption that the object to be detected is in contact with the road L. The first conversion unit 51 outputs the coordinates (coordinates of the object) of the detection frame whose coordinates have been converted to the calculation unit 52. The first conversion unit 51 may perform coordinate conversion only on the detection frames whose reliability of the detection frames included in the detection frame list is equal to or higher than a predetermined value, or may perform coordinate conversion on all the detection frames.

Next, the calculation unit 52 calculates the size of the object based on the coordinates of the detection frame whose coordinates have been converted by the first conversion unit 51 (S104). The calculation unit 52 calculates the size of the object in the real world, for example, based on the coordinates of the diagonal position of the detection frame. The calculation unit 52 outputs the size of the object in the real world to the determination unit 53. In the example of FIG. 3, the calculation unit 52 may calculate the size in the real world corresponding to the difference between the coordinates v1 and v2 as the size (height) of the object.

Next, the determination unit 53 determines whether or not the calculated size is within a predetermined range (S105). The determination unit 53 determines whether or not the size is within a predetermined range according to the object. The determination unit 53 determines a predetermined range based on the time series data of the size of the object in the past frame, and determines in step S105 based on the determined predetermined range. The determination unit 53 determines a predetermined range based on, for example, the equations shown in the equations 7 and 8. In the example of FIG. 5, the determination unit 53 determines that the size is within the predetermined range because the size of the current frame is included in the predetermined range. The determination unit 53 outputs the determination result to the second conversion unit 54.

When the second conversion unit 54 acquires the determination result indicating that the size is within the predetermined range from the determination unit 53 (Yes in S105), the second conversion unit 54 updates the size of the object (S106). The second conversion unit 54 calculates, for example, based on the size of the object of several frames in the past and the size of the object of the current frame calculated by the calculation unit 52, that is, the coordinates converted by the first conversion unit 51. The size of the object is updated based on the size of the object. The second conversion unit 54 may update the size of the object by, for example, adding the size of the object of the current frame to the size of the object of several frames in the past. The second conversion unit 54 uses the size of the current frame to update the estimation result for the determination unit 53 to determine the predetermined range in the next frame. The predetermined range determined by the determination unit 53 may change dynamically. Further, the second conversion unit 54 calculates, for example, the average value of the sizes of the objects from the sizes of the objects in the past several frames and the sizes of the objects in the current frame. You may update the size of.

Next, in the case of Yes in step S105, the second conversion unit 54 outputs position information including the coordinates of the object whose coordinates have been converted by the first conversion unit 51 in the real world as the position of the object (S107). .. That is, the second conversion unit 54 outputs the second coordinates as the position of the object. The second output unit 54 also functions as an output unit that outputs position information.

When the second conversion unit 54 acquires a determination result indicating that the size is out of the predetermined range from the determination unit 53 (No in S105), the second conversion unit 54 performs coordinate conversion based on the size of the object in the past frame (No). S108). The second conversion unit 54 uses the size of the object of the past frame (for example, the estimation result shown in FIG. 6) without using the size of the object of the current frame, and the detection frame from the detection unit 40. Reconvert the coordinates of each detection frame in the list. The second conversion unit 54 sets the position where the size of the object in the real world becomes a representative value (for example, average value, mode value, etc.) based on the size of the object in the past frame. Coordinates are converted as the size in the real world. The second conversion unit 54, for example, substitutes a representative value into the height Hworld shown in Equation 5 to perform coordinate conversion. In this way, when the size is not included in the predetermined range, the second conversion unit 54 reconverts the coordinates of the detection frame from the detection unit 40 into the coordinates in the real world based on the representative value within the predetermined range. ..

As a result, the position estimation device 30 can accurately obtain an estimated value of the size of the object in the real world even when the object is not in contact with the road L, so that the target can be obtained. It is possible to prevent an object from being presumed to be far away.

Next, the second conversion unit 54 outputs the position information including the coordinates of the object whose coordinates have been converted by the second conversion unit 54 in the real world as the position of the object (S107). In other words, if No in step S105, the second conversion unit 54 outputs the position information that does not include the coordinates converted by the first conversion unit 51. The second conversion unit 54 outputs the third coordinate as the position of the object.

Further, when the number of detection frames included in the detection frame list is 0 (No in S102), the control unit 50 ends the process of estimating the position of the object.

The timing at which the position estimation device 30 executes the process shown in FIG. 8 is not particularly limited, but is repeatedly executed, for example, while the vehicle 10 is traveling.

As described above, the position estimation device 30 according to the present embodiment is based on the size of the object in the past several frames, and whether or not there is an abnormality in the size of the object in the current frame, that is, suddenly. Determine if the size has not changed. Then, when the size of the object of the current frame is normal, the position estimation device 30 generates position information based on the result of the coordinate conversion of the first conversion unit 51. Further, in the position estimation device 30, when the size of the object of the current frame is abnormal, the pedestrian U is performing an irregular movement such as jumping, or the vehicle 10 is riding on the object. Since the behavior is irregular and the result of the coordinate conversion of the first conversion unit 51 may not be correct, the coordinate conversion is performed again based on the detection frame list of the detection unit 40. At this time, the second conversion unit 54 performs coordinate conversion using a value based on the size of the pedestrian U in the past several frames (for example, the average value of the sizes) as the size of the pedestrian U in the current frame. ..

As a result, the position estimation device 30 may perform irregular movements such as a pedestrian U jumping or an irregular movement such as a vehicle 10 riding on an object. Since the coordinate conversion is performed again using the value based on the size of the pedestrian U in the past several frames, the position of the object can be estimated accurately. That is, even when the position estimation device 30 is performing irregular movements such as jumping or the vehicle 10 is riding on an object, the position estimation device 30 is performing irregular movements such as jumping. It is possible to prevent the pedestrian U from erroneously estimating the position.

(Embodiment 2)
[2-1. Position estimation device configuration]
First, the configuration of the position estimation device according to the present embodiment will be described with reference to FIG. FIG. 9 is a table including standard values of the sizes of the objects stored in the storage unit according to the present embodiment. The position estimation device according to the present embodiment is different from the position estimation device 30 according to the first embodiment in that the information mainly stored in the storage unit and the determination process of the determination unit are different. That is, the functional configuration of the position estimation device according to the present embodiment may be the same as that of the position estimation device 30 according to the first embodiment. Therefore, hereinafter, the functional configuration of the position estimation device according to the present embodiment is assumed to be the same as the functional configuration of the position estimation device 30 according to the first embodiment, and the position estimation device 30 according to the first embodiment is used. The same reference numerals are given, and the description thereof will be omitted. Further, in the following, the size will be described as indicating the height, but the size is not limited to this. The size may be width or area.

As shown in FIG. 9, the storage unit 60 according to the present embodiment stores a table to which a class, a predetermined range, and a standard value (for example, a height) are associated with each other. The number of classes is not particularly limited.

The predetermined range is a range of size (height) for the determination unit 53 to use for determination. That is, the predetermined range is a range in which it can be determined whether or not the object is in contact with the road L, and is set for each class.

The standard value indicates the standard size (height) in the class. The standard value is a numerical value within a predetermined range and is set in advance for each class. The standard value is, for example, an average value, a median value, or a mode value in a predetermined range, but is not limited thereto. The standard value is an example of a representative value within a predetermined range. It can be said that the representative value is a standard value set in advance according to the object.

The determination unit 53 determines whether or not the size of the object is included in the predetermined range corresponding to the object by using the class and the predetermined range included in the above table. When the object is a pedestrian U, the determination unit 53 determines whether or not the size of the pedestrian U based on the image data is included in 1.4 to 2.0 m, which is a predetermined range corresponding to the pedestrian U. To judge.

When the size of the object is out of the predetermined range according to the object, the second conversion unit 54 performs coordinate conversion of the coordinates of the object based on the classes and standard values included in the above table. fix. When the size of the object is out of the predetermined range corresponding to the object, the second conversion unit 54 can be said to reconvert the coordinates of the object based on the fixed value according to the class.

[2-2. Operation of position estimator]
Subsequently, the operation of the position estimation device 30 according to the present embodiment will be described with reference to FIG. FIG. 10 is a flowchart showing the operation of the position estimation device 30 according to the present embodiment. The same or similar operation as the position estimation device 30 according to the first embodiment is designated by the same reference numerals as the operation of the position estimation device 30 according to the first embodiment, and the description thereof will be omitted. The operation of the position estimation device 30 according to the present embodiment is different in that step S208 is executed instead of step S108 shown in FIG. 8, and this point will be mainly described.

As shown in FIG. 10, the determination unit 53 determines whether or not the size is within a predetermined range (S105). The determination unit 53 determines a predetermined range based on the table stored in the storage unit 60 (see, for example, FIG. 9), and determines in step S105 based on the determined predetermined range. For example, the determination unit 53 acquires the class of the detection frame detected by the detection unit 40 from the detection frame list, and acquires a predetermined range corresponding to the acquired class based on the table shown in FIG. When a plurality of detection frames are included in the detection frame list, a predetermined range is acquired in each of the detection frames. Obtaining a predetermined range from a table is an example of determining a predetermined range.

The determination unit 53 determines in step S105 using a predetermined range determined based on FIG. 9. When a plurality of detection frames are included in the detection frame list, the determination unit 53 determines in step S105 using a predetermined range corresponding to the detection frame in each of the detection frames.

When the second conversion unit 54 acquires a determination result indicating that the size is out of the predetermined range from the determination unit 53 (No in S105), the second conversion unit 54 reconverts the coordinates based on the representative value corresponding to the object (No). S208). Specifically, the second conversion unit 54 uses the representative value of the size corresponding to the object (for example, the standard value shown in FIG. 9), and the coordinates of each detection frame in the detection frame list from the detection unit 40. Is reconverted to the coordinates of the real world. The second conversion unit 54 coordinates the position where the size of the object in the real world is a representative value as the position of the object in the real world. The second conversion unit 54 performs coordinate conversion by, for example, substituting a representative value into the height Hworld shown in Equation 5.

In this way, when the size calculated by the calculation unit 52 is not included in the predetermined range, the second conversion unit 54 acquires and acquires a standard value of the size corresponding to the class to which the object belongs as a representative value. Based on the representative value, the coordinates of the detection frame of the detection unit 40 are reconverted to the coordinates of the real world.

As a result, the position estimation device 30 can acquire an estimated value of the size of the object in the real world based on the representative value even when the object is not in contact with the road L, for example. Therefore, it is possible to prevent the object from being presumed to be far away.

As described above, the position estimation device 30 according to the present embodiment has no abnormality in the size of the object of the current frame based on the size range (predetermined range) set for each class of the object. Whether or not, that is, whether or not the size has changed suddenly is determined. Then, when the size of the object of the current frame is normal, the position estimation device 30 generates position information based on the result of the coordinate conversion of the first conversion unit 51, and the size of the object of the current frame is increased. If it is abnormal, the pedestrian U is performing irregular movements such as jumping, or the vehicle 10 is riding on an object or other irregular behavior, and the coordinates of the first conversion unit 51 are converted. Since there is a possibility that the result of the above is not correct, the coordinate conversion is performed again based on the detection frame list of the detection unit 40. At this time, the second conversion unit 54 performs coordinate conversion using a standard value (an example of a representative value) indicating a standard size for each class of the object as the size of the pedestrian U in the current frame.

As a result, the position estimation device 30 may perform irregular movements such as a pedestrian U jumping or an irregular movement such as a vehicle 10 riding on an object. Since the coordinate conversion is performed again using the standard size of each class of the object, the position of the object can be estimated accurately. That is, even when the position estimation device 30 is performing irregular movements such as jumping or the vehicle 10 is riding on an object, the position estimation device 30 is performing irregular movements such as jumping. It is possible to prevent the pedestrian U from erroneously estimating the position.

(Other embodiments)
Although the position estimation method and the like according to one or more aspects have been described above based on the embodiment, the present disclosure is not limited to this embodiment. As long as the purpose of the present disclosure is not deviated, various modifications that can be conceived by those skilled in the art are applied to the present embodiment, and a form constructed by combining components in different embodiments may also be included in the present disclosure. ..

For example, in the above-described embodiment and the like, the detection unit has described an example in which the object detection is performed using the trained model, but any known technique may be used as the object detection method in the image data. The detection unit may detect an object by pattern matching, for example.

Further, the position estimation device according to the above embodiment and the like may be realized by a plurality of devices. When the position estimation device is realized by a plurality of devices, each component of the position estimation device may be distributed to the plurality of devices in any way. Further, at least one of the components included in the position estimation device may be realized by the server device. For example, at least one of the processing units such as the first conversion unit, the calculation unit, the determination unit, and the second conversion unit may be realized by the server device. When the position estimation device is realized by a plurality of devices including the server device, the communication method between the devices included in the position estimation device is not particularly limited, and may be wireless communication or wired communication. You may. Further, wireless communication and wired communication may be combined between the devices.

Further, the order in which each step in the flowchart is executed is for exemplifying the present disclosure in detail, and may be an order other than the above. Further, a part of the above steps may be executed simultaneously with other steps (parallel), or a part of the above steps may not be executed.

Further, the division of the functional block in the block diagram is an example, and a plurality of functional blocks can be realized as one functional block, one functional block can be divided into a plurality of functional blocks, and some functions can be transferred to other functional blocks. You may. Further, the functions of a plurality of functional blocks having similar functions may be processed by a single hardware or software in parallel or in a time division manner.

Further, a part or all of the components included in the position estimation device in the above embodiment may be composed of one system LSI (Large Scale Integration: large-scale integrated circuit).

The system LSI is an ultra-multifunctional LSI manufactured by integrating a plurality of processing units on one chip, and specifically, a microprocessor, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. It is a computer system composed of. A computer program is stored in the ROM. The system LSI achieves its function by operating the microprocessor according to the computer program.

Further, one aspect of the present disclosure may be a computer program that causes a computer to execute each characteristic step included in the position estimation method shown in FIG. 8 or FIG. For example, the program may be a program to be executed by a computer. Also, one aspect of the present disclosure may be a computer-readable, non-temporary recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or distributed. For example, by installing the distributed program on a device having another processor and causing the processor to execute the program, it is possible to cause the device to perform each of the above processes.

The present disclosure is useful for a position estimation device that estimates the position of an object using image data captured by a monocular camera.

10 Vehicle 20 Monocular camera 30 Position estimation device 40 Detection unit 50 Control unit 51 First conversion unit 52 Calculation unit 53 Judgment unit 54 Second conversion unit 60 Storage unit D Image data F Detection frame Himg, Hworld Height L Road M, U Pedestrian (object)
P1, P2 Position Pa, Pb, Pb1, Pb2 Coordinates Simg, Lead Area

Claims (7)

Obtain image data including an object from a monocular camera and
The first coordinate of the object in the image data is converted into the second coordinate in the real world.
The size of the object is calculated based on the second coordinates.
It is determined whether or not the size is included in a predetermined range according to the object, and the size is determined.
A position estimation method in which when the size is not included in the predetermined range, the first coordinate is converted back into a third coordinate in the real world based on a representative value within the predetermined range. The position estimation method according to claim 1, wherein the representative value is a standard value preset according to the object. The standard value is set for each class of the object, and is set.
If the size is not within the predetermined range
The position estimation according to claim 2, wherein the standard value corresponding to the class to which the object belongs is acquired as the representative value, and the first coordinate is converted back to the third coordinate based on the acquired representative value. Method. The position estimation method according to claim 1, wherein the representative value is determined based on time-series data of the size of the object based on image data acquired in the past. When the size is included in the predetermined range, any one of claims 1 to 4 for updating the size of the object based on the size in the current frame and the size of the object in the past frame. The position estimation method according to item 1. An acquisition unit that acquires image data including an object from a monocular camera,
A first conversion unit that converts the first coordinate of the object in the image data to the second coordinate in the real world,
A calculation unit that calculates the size of the object based on the second coordinates,
A determination unit for determining whether or not the size is included in a predetermined range according to the object,
A position estimation device including a second conversion unit that reconverts the first coordinate to a third coordinate based on a representative value within the predetermined range when the size is not included in the predetermined range. A program for causing a computer to execute the position estimation method according to any one of claims 1 to 5.

Images