JP2020173504A - Position estimation system, position estimation device, position estimation method, and program - Google Patents

Abstract

To provide a position estimation system capable of estimating a position of a target object partially hidden by another object in estimating the position of the target object based on an image obtained by imaging.SOLUTION: A position estimation system comprises: an imaging apparatus; a height calculation part which calculates a height of a target object based on image recognition processing executed for captured images obtained by capturing with the imaging apparatus; and a position calculation part which calculates a position of the target object on a projection plane of the imaging apparatus corresponding to a ground surface based on the height calculated by the height calculation part.SELECTED DRAWING: Figure 3

Description

The present invention relates to a position estimation system, a position estimation device, a position estimation method, and a program.

For example, there is known a technique for estimating a three-dimensional shape based on images taken by an imaging device before and after movement (see, for example, Patent Document 1). In the technique of Patent Document 1, for example, the road surface is used as a reference surface, the positions of the feature points of the object existing on the road surface are estimated, and the three-dimensional shape is estimated from the set of the estimated feature point positions. It has been done. That is, in the technique of Patent Document 1, the position of the object is estimated.

Japanese Unexamined Patent Publication No. 2010-181919

However, in the position estimation technique according to Patent Document 1 described above, when an object in a state of being partially hidden by another object is used as an object, the distance of the object cannot be calculated. Position estimation becomes difficult.

The present invention has been made in view of such circumstances, and when estimating the position of an object based on an image obtained by imaging, the position of an object whose part is partially hidden by another object is also estimated. The purpose is to be able to do it.

One aspect of the present invention that solves the above-mentioned problems is a height that calculates the height of an object based on an image pickup device and an image recognition process executed on the image captured by the image pickup device. A position including a calculation unit and a position calculation unit that calculates the position of the object on the projection surface of the image pickup device set corresponding to the ground surface based on the height calculated by the height calculation unit. It is an estimation system.

Further, one aspect of the present invention includes a height calculation unit that calculates the height of an object based on an image recognition process executed on an image captured by an imaging device, and the height calculation unit. It is a position estimation device including a position calculation unit for calculating the position of the object on the projection surface of the image pickup device set corresponding to the ground surface based on the height calculated by.

Further, one aspect of the present invention includes a height calculation step of calculating the height of an object based on an image recognition process executed on an image captured by an imaging device, and the height calculation step. This is a position estimation method including a position calculation step for calculating the position of the object on the projection surface of the image pickup apparatus set corresponding to the ground surface based on the height calculated by.

Further, one aspect of the present invention is a height calculation unit that calculates the height of an object based on an image recognition process executed on an image captured by an imaging device, and the height calculation unit. This is a program for functioning as a position calculation unit for calculating the position of the object on the projection surface of the image pickup device set corresponding to the ground surface based on the calculated height.

As described above, according to the present invention, when estimating the position of an object based on an image obtained by imaging, it is possible to estimate the position of an object whose part is partially hidden by another object. The effect is obtained.

It is a figure which shows the configuration example of the position estimation system in this embodiment. It is a figure explaining the method example of the position estimation of the object in this embodiment. It is a figure explaining the method example of the position estimation of the object in this embodiment. It is a figure explaining the method example of the position estimation of the object in this embodiment. It is a flowchart which shows the example of the processing procedure executed by the position estimation apparatus in this embodiment.

FIG. 1 shows a configuration example of the position estimation system 1 of the present embodiment. The position estimation system of the present embodiment may be applied to, for example, a surveillance system including a surveillance camera installed fixedly, or may be provided by a moving body such as a vehicle or a robot.
The position estimation system 1 in the figure includes an image pickup device 100, a position estimation device 200, and an information output device 300.

The imaging device 100 performs imaging. The image pickup apparatus 100 of the present embodiment is not something like a stereo camera, for example, but may be a monocular device, for example.

The position estimation device 200 is a device that estimates the position of an object included as a subject in the captured image in the real space based on the image (captured image) obtained by imaging with the imaging device 100.
The position estimation device 200 includes an input / output interface unit 201, an information processing unit 202, and a storage unit 203 as functional units.

The input / output interface unit 201 is a portion that inputs / outputs information between the position estimation device 200 and the outside. The captured image output by the imaging device 100 is input by the input / output interface unit 201 and passed to the information processing unit 202.
Further, information such as images and sounds output by the information processing unit 202 is output from the input / output interface unit 201 to the information output device 300.

The information processing unit 202 executes information processing related to the position estimation of the object. The function as the information processing unit 202 is realized by the CPU (Central Processing Unit) provided in the position estimation device 200 executing a program. The information processing unit 202 in the figure includes a projection surface setting unit 221, a height calculation unit 222, a position calculation unit 223, and a position conversion unit 224.
The projection plane setting unit 221 sets the projection plane corresponding to the image pickup apparatus 100. Specifically, the projection plane setting unit 221 sets a projection plane (ground projection plane) corresponding to the ground surface as the projection plane.
The height calculation unit 222 calculates the height of the object based on the image recognition process executed on the image captured by the image pickup device 100.
The position calculation unit 223 calculates the position of the object on the ground projection surface based on the height calculated by the height calculation unit 222.
The position conversion unit 224 converts the position calculated by the position calculation unit 223 into a position in the real space.

The storage unit 203 stores various types of information corresponding to the position estimation device 200.

The information output device 300 outputs the information output from the position estimation device 200. The information output device 300 may be a display device, an audio output device, or a printing device.

An example of a position estimation method by the position estimation system 1 of the present embodiment will be described with reference to FIGS. 2 to 4.
FIG. 2 shows an example of an captured image obtained by imaging with the imaging device 100. In the captured image of the figure, the extraction result of the subject by the image recognition process is shown. Here, for convenience of simplicity, the case where the conditions such as the position and the imaging direction of the image pickup apparatus 100 are fixed and known will be given as an example.
The captured image P1 in the figure includes six subjects, subjects OB1 to OB6. When the subjects OB1 to OB6 are not particularly distinguished, they are described as subject OB.
The height calculation unit 222 of the position estimation device 200 performs image recognition processing on the captured image P1, and then, for example, an AI (Artificial Intelligence) framework such as a neural network (or a machine learning framework or the like). The attribute for each subject OB is recognized by the process using.
As one of the simple examples, the height calculation unit 222 recognizes that the subjects OB1, OB2, and OB3 are adults, the subject OB3 is a child, and the subject OB5 is a passenger car. Is recognized as a vehicle corresponding to the above, and the subject OB6 is recognized as a vehicle as a truck.
Then, the height calculation unit 222 determines that the subject OB as a person is, for example, a woman or a man based on the feature amount according to the face, clothes, hairstyle, accessories and belongings worn. You may recognize that there is. Further, the subject OB as a person may be made to recognize which ethnic group it corresponds to, such as Asian, European, or African, based on the feature amount of the face or the like.
In addition, the height calculation unit 222 further recognizes the subject OB as a vehicle in categories such as minivans and sedans for passenger cars, and small, medium, and large for trucks. Good. Further, regarding the subject OB as a vehicle, the vehicle type, model number, and the like may be recognized.
Then, the height calculation unit 222 calculates the height corresponding to the recognized attribute. For example, the height calculation unit 222 refers to the data of the height table in which the height is associated with each attribute, acquires the height associated with the recognized attribute, and sets the acquired height as the height. It may be the calculation result of. Alternatively, the height calculation unit 222 may be made to derive the height of the recognized attribute along with the recognition of the attribute by machine learning.
In this way, the height calculation unit 222 can calculate the height of each subject OB extracted in the captured image of the figure.

The object for which the height is calculated is not limited to people and vehicles. The object may be, for example, an object, a robot, an animal, a landmark, or the like. The position of the object whose height is calculated in this embodiment can be estimated. That is, in the present embodiment, the object in the captured image that is the target at a vacant position is not limited to a person or a vehicle.

FIG. 3 shows the relationship between the projection surface of the image pickup apparatus 100 and the subject OB included in the captured image P1.
In the figure, the projection center V of the image pickup apparatus 100 is fixedly defined. Further, in the figure, the point (position) on the ground surface projected vertically from the projection center V is the vertical projection point V _h . Here, since the position of the image pickup apparatus 100 is fixed, the distance VV _h is invariant (constant).

Further, in the figure, the projection plane abcd is shown. The projection surface abcd is a projection surface of a two-dimensional image obtained by a perspective projection model having a projection center V determined corresponding to the image pickup apparatus 100. Such a projection plane is equivalent to a two-dimensional image obtained by imaging with a monocular camera.
Further, in the figure, the projection plane ABCD corresponding to the ground surface is also shown. In the following description, the projection surface ABCD corresponding to the ground surface is also described as the ground projection surface ABCD to distinguish it from the projection surface abcd not corresponding to the ground surface.
Some one coordinate _{m i} in the projection plane abcd shown in FIG. _{(X image, y image, 1} ) , in the ground projection plane ABCD, the coordinates _{M hi (x h, y h} , -H) corresponding .. H is the height at which the image pickup apparatus 100 is installed.
The relationship between the coordinate m _i and the coordinate M _hi may be represented by Equation 1 below.

In Equation 1, A is a conversion matrix, which is obtained from the internal parameters (image center, focal length, effective pixel size) and external parameters (posture (height, imaging direction, etc.) of the imaging device 100).

FIG. 4 schematically shows a calculation method for the position of an object by the position calculation unit 223.
Here, the height H of the projection center V of the imaging apparatus 100 is known, the height of the object OB _i as i-th is calculated by the height calculation unit 222 that is H _i.
Further, the position calculation unit 223 shows an example of the indicated position pt _i (an example of coordinates of a specific position of the object, which is defined as a human head in the figure) corresponding to the height of the object OB _i . ) Specify the two-dimensional coordinates in the captured image. Position calculating unit 223 specifies the coordinates m _i of the projection plane abcd corresponding to two-dimensional coordinates in the imaging identified. Position calculating unit 223, in response to the coordinate _{m i} of the projection plane abcd is specified by the equation 1, it is possible to determine the coordinates _{M hi} on the ground projection plane ABCD uniquely.
Then, the position calculating unit 223, during the period from the vertical projection point _{V h} to the coordinates _{M hi,} indicated position pt _i of the object OB _i are present on a line segment connecting the projection center V and the coordinate _{M hi} state The position at the time of is specified as the position Mi of the object OB _{i on} the ground projection plane ABCD.
That is, the position _{M i} of the object OB _i, can be expressed by Equation 2 below.

Accordingly, the position calculation unit 223, a position _{M i} of the object OB _i, for example, the height H of the imaging apparatus 100, the height _{H i} calculated for the object OB _i, the object on the projection plane abcd can be calculated by equation 3 below using the coordinate _{m i} of the position indicated pt _i of OB _i.

Further, in the above description, the case where the position of the image pickup apparatus 100 is fixed is taken as an example, but when the image pickup apparatus 100 is provided on a moving body or the like to move, the position calculation unit 223 is described below. it may calculate the position M _i of the object OB _i as. That is, the position calculation unit 223 defines the projection center V as the coordinates (Xo (t), Yo (t), 0) that change due to translation according to the time t, and then uses the following equation 4 to define the object. it may calculate the position _{M i} of OB _i.

The position conversion unit 224, the coordinates of the position M _i of the object OB _i calculated by the position calculating unit 223 as described above, for example, by converting the coordinates on the map, the position in the real space (the coordinates ).

In this way, in the present embodiment, the position estimation device 200 can estimate the position of the object by using the subject in the captured image obtained by the image pickup device 100 as the object. For example, the position estimation device 200 can output the estimated position by the information output device 300 by displaying, printing, or the like so that the estimated position is shown on the map.

In the position estimation system 1 of the present embodiment in which the position of the object is estimated as described above, for example, in the captured image, a subject whose part (particularly the subject portion corresponding to the ground surface) is hidden by the subject in front is targeted. Position estimation can be performed even when the object is used. Then, in the present embodiment, the same algorithm can be applied to the position of the object whose part is not hidden and the object whose part is hidden in the captured image. That is, in the position estimation system 1 of the present embodiment, the position of an object whose part is partially hidden can be estimated with the same accuracy as that of the object whose part is not hidden.
Further, in the position estimation method by the position estimation system 1 of the present embodiment, it is not necessary to obtain a plurality of captured images corresponding to the difference in parallax as in a stereo camera, for example. That is, as the position estimation system 1 of the present embodiment, as the image pickup device 100 for imaging an object, a monocular device can be adopted as described above. As a result, for example, it is possible to reduce the cost and reduce the size of the hardware constituting the position estimation system 1.

The hardware configuration of the position estimation system 1 is not limited to the configuration of the three devices as the image pickup device 100, the position estimation device 200, and the information output device 300 shown in FIG. For example, a predetermined two or more of the image pickup device 100, the position estimation device 200, and the information output device 300 may be integrated as one hardware. Further, any device among the image pickup device 100, the position estimation device 200, and the information output device 300 may be further divided into a plurality of hardware.

The flowchart of FIG. 5 shows an example of a processing procedure executed by the position estimation device 200.
Step S101: In the position estimation device 200, the projection plane setting unit 221 sets the coordinates of the projection center V. The coordinates of the projection center V may be three-dimensional coordinates including the x-axis coordinates, the y-axis coordinates, and the z-axis coordinates corresponding to the height H. Setting the coordinates of the projection center V corresponds to determining the position of the image pickup apparatus 100 and the image pickup direction.
In this case, when the position and the imaging direction of the imaging device 100 are fixed, the coordinates of one projection center V are obtained in advance based on the known position and imaging direction of the imaging device 100. The coordinates of the projection center V may be stored in the storage unit 203. The projection plane setting unit 221 may acquire the coordinates of the projection center V stored in the storage unit 203 as the step S101.
Alternatively, when the imaging device 100 is provided on the moving body and moves with the passage of time, the projection plane setting unit 221 may execute the following processing as the step S101. That is, the projection plane setting unit 221 derives the position and imaging direction of the imaging device 100 based on, for example, its own position (which may include the direction) positioned by the moving body. The projection plane setting unit 221 may calculate the coordinates of the projection center V according to the position of the derived imaging device 100 and the imaging direction.

Step S102: The information processing unit 202 acquires an image captured by the image capturing device 100 via the input / output interface unit 201.

Step S103: The projection plane setting unit 221 sets the ground projection plane ABCD having the projection center V set in step S101 as the origin. By setting the ground projection plane ABCD, the coordinates of 1 on the ground projection plane ABCD corresponding to the two-dimensional coordinates of 1 in the captured image can be uniquely specified.

Step S104: height calculation unit 222 calculates the height _{H i} of the object OB _i to be position estimation. As the process of step S104, the height calculation unit 222 extracts the subject as the object OB _i from the captured image by image recognition and machine learning for the captured image as described above, and the extracted object OB _i recognizes the attributes, it may be adapted to calculate the height H _i in accordance with the recognized attributes.
Step S105: position calculating unit 223 calculates the position _{M i} on the ground projection plane ABCD of the object OB _i. Position calculating unit 223, as described in FIG. 4, by utilizing the height H of the projection center V, the height _{H i} of the object OB _i, and an indication position pt _i of the object OB _i, it may calculate the position _{M i.}
Step S106: position conversion unit 224 converts the position _{M i} on the ground projection plane ABCD of the object OB _i calculated in the step S105, the position in the real space (coordinates). Position M _i calculated in the step S105, for example 3, as illustrated in FIG. 4, a specified position in the virtual ground projection plane ABCD that is set to the projection center V as the origin, the physical space It has not been determined which position in the. Therefore, the position conversion unit 224 performs processing for converting the position M _i to a position in the real space. That is, the position conversion unit 224 specifies which position in the real space the position Mi corresponds to.
For this purpose, as an example, the storage unit 203 stores map information in which coordinates are set in the real space. The position conversion unit 224 associates the projection center V with one coordinate in the map information, for example, based on the position of the image pickup apparatus 100 positioned in the real space. Position conversion unit 224, for example, in the map information, specifies the coordinate position M _i corresponding projection center V as a reference. In this way, the position M _i is converted to a position in the real space.

The program for realizing the function of the position estimation device 200 described above is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by the computer system and executed. The processing as the position estimation device 200 may be performed. Here, "loading a computer system a program recorded on a recording medium and executing it" includes installing the program in the computer system. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer system" may include a plurality of computer devices connected via a network including a communication line. The recording medium also includes an internal or external recording medium that can be accessed from the distribution server to distribute the program.

1 Position estimation system, 100 Imaging device, 200 Position estimation device, 201 Input / output interface unit, 202 Information processing unit, 203 Storage unit, 221 Projection surface setting unit, 222 Height calculation unit, 223 Position calculation unit, 224 Position conversion unit , 300 Information output device

Claims (6)

Imaging device and
A height calculation unit that calculates the height of an object based on an image recognition process performed on an image captured by the image pickup device.
A position estimation system including a position calculation unit that calculates the position of the object on the projection surface of the image pickup apparatus set corresponding to the ground surface based on the height calculated by the height calculation unit. The position calculation unit
Calculated by the height calculation unit, the coordinates of the projection surface determined corresponding to the coordinates of the specific position of the object in the captured image, the height of the projection center corresponding to the projection surface from the ground surface, and the height calculation unit. The position estimation system according to claim 1, wherein the position of the object on the projection plane is calculated based on the height of the object. The position estimation system according to claim 1 or 2, wherein the imaging device is monocular. A height calculation unit that calculates the height of an object based on an image recognition process executed on the captured image obtained by imaging with an imaging device.
A position estimation device including a position calculation unit that calculates the position of the object on the projection surface of the image pickup device set corresponding to the ground surface based on the height calculated by the height calculation unit. A height calculation step of calculating the height of an object based on an image recognition process executed on the captured image obtained by imaging with an imaging device, and
A position estimation method including a position calculation step for calculating the position of the object on the projection surface of the image pickup apparatus set corresponding to the ground surface based on the height calculated by the height calculation step. Computer,
A height calculation unit that calculates the height of an object based on an image recognition process executed on the captured image obtained by imaging with an imaging device.
A program for functioning as a position calculation unit for calculating the position of the object on the projection surface of the image pickup device set corresponding to the ground surface based on the height calculated by the height calculation unit.

Images