JP6991955B2 - Object position identification device, object position identification method, and object position identification program - Google Patents

Description

The present invention relates to an object position specifying device, an object position specifying method, and an object position specifying program.

Conventionally, for the purpose of marketing, monitoring, etc., a technique has been developed in which an image captured by an image pickup device installed in a predetermined area such as a store is analyzed to identify the position of a person existing in the predetermined area.

Patent Document 1 discloses an image analysis device that identifies a person's position by analyzing an captured image obtained by capturing a predetermined area with an omnidirectional camera. In the image analysis apparatus described in Patent Document 1, when a three-dimensional model of a person is arranged at one or more positions in a predetermined area, the correspondence relationship between the projected area of the three-dimensional model on the captured image and the arranged position. Is specified, and the corresponding person position is specified by matching the area of the person detected from the captured image with the projected area. This method has a feature that it can be applied not only to an image captured diagonally downward but also to an image captured directly downward.

In Non-Patent Document 1, both the whole body region and the head region of the person are detected from the captured image, and the relative positional relationship between the detected region indicated by the whole body and the region indicating the head is used to describe the person. A method for suppressing false positives is disclosed.

JP-A-2017-117244

Roberto Henschel, Laura Leal-Taixe, Daniel Cremers, Bodo Rosenhahn “Fusion of Head and Full-Body Detectors for Multi-Object Tracking”, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp.1428-1437 (2018) )

An image taken directly below has a characteristic that the appearance of the person changes greatly depending on the position of the person to be imaged. FIG. 12 is a diagram showing an example of how a person to be imaged is reflected at a plurality of positions in a plurality of areas. In the captured image captured in the direct downward direction, as shown in FIG. 12, the person may be tilted or only the top of the head of the person may be captured depending on the position of the person to be imaged. Therefore, there is a problem that the accuracy of detecting a person is low in the captured image captured in the direct downward direction.

On the other hand, in the method disclosed in Non-Patent Document 1, it is necessary to uniquely specify in advance the ideal positional relationship between the region showing the whole body of the person to be imaged and the region showing the head. FIG. 13 is a diagram showing an ideal positional relationship between a region showing the whole body of a person to be imaged and a region showing the head. The region surrounded by the solid line shown in FIG. 13 indicates the region corresponding to the whole body, and the region surrounded by the wavy line indicates the region corresponding to the head.

On the other hand, FIG. 14 is a diagram showing an example of the positional relationship between the whole body and the head, which is assumed in the captured image captured in the direction directly below. In FIG. 14, as in FIG. 13, the region surrounded by the solid line indicates the region corresponding to the whole body, and the region surrounded by the wavy line indicates the region corresponding to the head. As shown in FIG. 14, in the captured image captured in the direct downward direction, the positional relationship between the region showing the whole body and the region showing the head changes depending on the position of the person, so that the technique of Non-Patent Document 1 cannot be applied. There's a problem.

Therefore, the present invention has been made in view of these points, and is an object position specifying device, an object position specifying method, which can improve the object position specifying accuracy regardless of the type of the image pickup device and the image pickup angle. And to provide an object position identification program.

The object position specifying device according to the first aspect of the present invention is a first object which is a first region showing an acquisition unit for acquiring an image captured by an image pickup device for capturing a predetermined area and an object reflected in the captured image. An object area specifying portion that specifies an area, an object that is reflected in the captured image, and a second object area that is a second area different from the first area, and a three-dimensional model that imitates the object are provided in the predetermined area. Of the projected regions that are the regions of the three-dimensional model reflected in the captured image when arranged at each of the plurality of positions, the plurality of first model regions corresponding to the first region and the second region are supported. A model that identifies a second model region corresponding to a first model region having a relatively short first distance, which indicates a distance to the specified first object region, based on a correspondence relationship with a plurality of second model regions. When the second distance indicating the distance between the area specifying unit, the specified second object area, and the specified second model area is equal to or less than the first threshold value, the position of the object is specified. It is equipped with a position identification unit.

The model area specifying unit is one or more second model areas corresponding to one or more first model areas having a relatively small first distance to the specified first object area based on the correspondence relationship. The second distance, which is relatively short among the second distances corresponding to each of the specified one or more second model regions, is equal to or less than the first threshold value. In some cases, the position of the object may be specified.

The object position specifying device is the first model area of the projection area which is the area of the three-dimensional model reflected in the captured image when the three-dimensional model is arranged at each of a plurality of positions in the predetermined area. The model area specifying unit further includes a correspondence relationship specifying unit that specifies a correspondence relationship with the second model area, and the model area specifying unit has the first object region specified based on the specified correspondence relationship. A second model region corresponding to a first model region in which the distance of 1 is relatively short may be specified.

The correspondence relationship specifying unit identifies the correspondence relationship corresponding to the three-dimensional model corresponding to each of the plurality of sizes that the object can take, and the model area specifying unit specifies the correspondence relationship based on the specified correspondence relationship. A plurality of second model regions corresponding to a plurality of first model regions having a relatively short first distance to the specified first object region may be specified.

The correspondence relationship specifying unit identifies the correspondence relationship corresponding to the three-dimensional model corresponding to each of the plurality of shapes that the object can take, and the model area specifying unit specifies the correspondence relationship based on the specified correspondence relationship. A plurality of second model regions corresponding to a plurality of first model regions having a relatively short first distance to the specified first object region may be specified.
The correspondence relationship specifying unit may integrate the correspondence relationship corresponding to a plurality of first model regions at relatively close positions.

The model area specifying unit calculates the overlap rate between the specified first object area and the first model area, and the larger the overlap rate, the shorter the first distance. The distance of 1 may be calculated.

The model area specifying unit specifies one or more of the first model areas having a position relatively close to the position indicating the first object area, and specifies among the specified one or more of the first model areas. A second model area corresponding to the first model area having a relatively short first distance from the first object area may be specified.

The position specifying unit calculates the overlap rate between the specified second object area and the specified second model area, and the larger the overlap rate is, the shorter the second distance is. The second distance may be calculated.

The object area specifying unit identifies one or more of the first object areas and one or more of the second object areas based on the captured image, and the position specifying unit is one or more of the second objects. For each of the regions, a third distance indicating the distance from one or more of the first object regions is specified, and the relatively short third distance of the one or more second object regions is the second. The position of the object corresponding to the second object region whose second distance from the second model region is equal to or less than the threshold value and which is equal to or less than the first threshold value may be specified.

The position specifying unit specifies a third distance indicating the distance from the one or more first object areas for each of the one or more second object areas, and among the one or more second object areas. The position of the object corresponding to the second object region where the relatively short third distance exceeds the second threshold value and the second distance to the second model region is equal to or less than the first threshold value. May be specified.

The position specifying unit has specified and specified a large ratio of the ratio of the second object area and the ratio of the first object area to the union of the second object area and the first object area. The third distance may be calculated so that the larger the ratio is, the shorter the third distance is.

The position specifying unit is set in the second model area when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value. The position of the object in the predetermined area may be specified based on the arrangement position of the corresponding three-dimensional model.

The position specifying unit refers to the second object area when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value. The position of the object in the captured image may be specified based on at least one of the position in the captured image and the position in the captured image of the second model region.

The object position specifying method according to the second aspect of the present invention includes a step of acquiring an image captured by an image pickup device that captures a predetermined area, which is executed by a computer, and a first region showing an object reflected in the captured image. A step of specifying a certain first object area and a second object area which is a second area different from the first area and shows an object reflected in the captured image, and a three-dimensional model imitating the object are set in the predetermined area. Of the projected regions that are the regions of the three-dimensional model reflected in the captured image when arranged at each of the plurality of positions, the plurality of first model regions corresponding to the first region and the second region are supported. Based on the correspondence with the plurality of second model regions, the second model region corresponding to the first model region having a relatively short first distance indicating the distance to the specified first object region is specified. A step and a step of specifying the position of the object when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value. , Equipped with.

The object position specifying program according to the third aspect of the present invention is a first region showing an object reflected in the captured image, an acquisition unit that acquires a captured image captured by an imaging device that captures a predetermined area of a computer. An object area specifying unit that specifies one object area and a second object area that indicates an object reflected in the captured image and is a second area different from the first area, and a three-dimensional model imitating the object in the predetermined area. Of the projected regions that are the regions of the three-dimensional model reflected in the captured image when arranged at each of the plurality of positions, the plurality of first model regions corresponding to the first region and the second region are supported. Based on the correspondence with the plurality of second model regions, the second model region corresponding to the first model region having a relatively short first distance indicating the distance to the specified first object region is specified. When the model area specifying unit and the second distance indicating the distance between the specified second object area and the specified second model area are equal to or less than the first threshold value, the position of the object is determined. It functions as a specified position specifying part.

According to the present invention, there is an effect that the accuracy of specifying the position of an object can be improved regardless of the type of the image pickup apparatus and the image pickup angle.

It is a figure which shows the outline of the object position specifying apparatus which concerns on this embodiment. It is a figure which shows the structure of the object position specifying apparatus which concerns on this embodiment. It is a flowchart which shows the flow of the process in the object position specifying apparatus which concerns on this embodiment. It is a figure which shows the relationship between the object at each of the plurality of positions of the predetermined area which concerns on this embodiment, and the 1st object area and the 2nd object area of the said object. It is a figure which shows an example of the result of specifying the 1st object area and the 2nd object area using a detection model. It is a figure which shows an example of the 3D model which concerns on this embodiment. It is a figure which shows the example which specifies the 1st model area and the 2nd model area which concerns on this embodiment. It is a figure which shows an example of a plurality of 3D models of different sizes which concerns on this embodiment. It is a figure which shows an example of the 1st model region which has the same center position. It is a figure explaining the example that the model area specifying part which concerns on this embodiment specifies the 2nd model area. It is a figure which shows the example which specified the 2nd model area with respect to the result which specified the 1st object area and the 2nd object area shown in FIG. It is a figure which shows an example of the image condition of the person to be imaged at a plurality of positions of a plurality of areas. It is a figure which shows the ideal positional relationship between the region which shows the whole body and the region which shows a head of a person to be imaged, which is assumed in the image taken in the diagonally downward direction. It is a figure which shows the example of the positional relationship between the whole body and the head assumed in the image taken in the direct downward direction.

[Overview of object position identification device 1]
FIG. 1 is a diagram showing an outline of the object position specifying device 1 according to the present embodiment. The object position specifying device 1 is a computer that specifies the position of an object reflected in an image captured by the image pickup device. The image pickup device is, for example, an omnidirectional camera installed on the ceiling of a predetermined area and taking an image in the direction directly below. In the present embodiment, the image pickup device is an omnidirectional camera that captures an image in the direct downward direction, but the present invention is not limited to this. The image pickup device may be a normal camera different from the omnidirectional camera, and the image pickup angle of the image pickup device may be any angle.

The object position specifying device 1 acquires a captured image of a predetermined area captured by the imaging device ((1) in FIG. 1). The predetermined area is, for example, a space inside a store. The object is, for example, a clerk or a customer who acts in a predetermined area.

The object position specifying device 1 identifies a first object area, which is a first area showing an object reflected in a captured image, and a second object area, which is a second area different from the first area and shows an object reflected in a captured image. ((2) in FIG. 1). In FIG. 1, the rectangular area of the broken line in the captured image indicates the first object area, and the rectangular area of the solid line indicates the second object area.

The object position specifying device 1 is out of the projection area of the three-dimensional model displayed in the captured image when the three-dimensional model corresponding to the object is virtually arranged at each of a plurality of positions in the three-dimensional space indicating a predetermined area. Based on the correspondence between the plurality of first model areas corresponding to the first area and the plurality of second model areas corresponding to the second area, the first object area at a position relatively close to the specified first object area. A second model area corresponding to one model area is specified ((3) in FIG. 1). Here, as the correspondence relationship, there is a case where the relationship between one first model area and one second model area is shown, and a case where one first model area and a plurality of second model areas are shown. Suppose there is.

The object position specifying device 1 specifies the position of the object corresponding to the second object area when the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value (FIG. 1 (4)).

By doing so, the object position specifying device 1 is based on the correspondence between the first model area and the second model area, which show an appropriate positional relationship between the first area and the second area of the object. The position of the object can be specified when the positional relationship between the first object area specified in the captured image and the second object area is appropriate.

Further, the correspondence relationship between the first model area and the second model area is defined based on the projection area of the three-dimensional model with respect to the captured image captured by the image pickup apparatus. Therefore, the object position specifying device 1 can maintain the accuracy of specifying the position of the object regardless of the type of the image pickup device and the image pickup angle.

[Configuration of object position specifying device 1]
Subsequently, the configuration of the object position specifying device 1 will be described. FIG. 2 is a diagram showing a configuration of an object position specifying device 1 according to the present embodiment. As shown in FIG. 2, the object position specifying device 1 includes a storage unit 11 and a control unit 12.

The storage unit 11 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The storage unit 11 stores a program executed by the control unit 12. For example, the storage unit 11 stores an object position specifying program that causes the object position specifying device 1 to function as an acquisition unit 121, an object area specifying unit 122, a correspondence relationship specifying unit 123, a model area specifying unit 124, and a position specifying unit 125. is doing.

The control unit 12 is, for example, a CPU (Central Processing Unit). By executing the object position specifying program stored in the storage unit 11, the control unit 12 serves as the acquisition unit 121, the object area identification unit 122, the correspondence identification unit 123, the model area identification unit 124, and the position identification unit 125. Function.

Hereinafter, with reference to FIG. 3 showing the flow of processing in the object position specifying device 1, the details of the acquisition unit 121, the object area specifying unit 122, the correspondence relationship specifying unit 123, the model area specifying unit 124, and the position specifying unit 125 will be described. explain. FIG. 3 is a flowchart showing a processing flow in the object position specifying device 1 according to the present embodiment.

The acquisition unit 121 acquires an image captured by an image pickup device that captures a predetermined area (S1). For example, the captured image of a predetermined area captured by the imaging device is stored in advance in the storage unit 11. The acquisition unit 121 acquires the captured image stored in the storage unit 11.

The object area specifying unit 122 indicates a first object area, which is a first area showing an object reflected in the captured image acquired by the acquisition unit 121, and an object reflected in the captured image acquired by the acquisition unit 121. A second object area, which is a different second area, is specified (S2). The object area specifying unit 122 identifies one or more of the first object area and the second object area from the captured image acquired by the acquisition unit 121.

Here, the first region is, for example, a region showing the head of a person as an object, and the second region is a region showing the whole body of the person. The size of the second region is a size that includes the first region. For example, the object area specifying unit 122 is four-dimensionally composed of information (u, v) indicating the center position of a rectangle indicating the first area and information (w, h) indicating the width and height of the rectangle. The first object area is specified by specifying the vector (u, v, w, h) of. Similarly, the object area specifying unit 122 identifies a four-dimensional vector composed of information indicating the center position of the rectangle indicating the second area and information indicating the width and height of the rectangle. 2 Specify the object area. In the following description, the four-dimensional vector indicating the rectangle is also referred to as rectangle information. Hereinafter, the case where each area is represented by a rectangle will be described as an example, but the object area specifying unit 122 may specify each area by any other shape.

The object area specifying unit 122 provides machine learning (for example, Support Vector Machine) and deep learning (for example, Convolutional Neural) learned by adding correct rectangular information to each of the first object area and the second object area in the captured image. By using the detection model of Network), the first object area and the second object area included in the captured image are specified. FIG. 4 is a diagram showing the relationship between an object at each of a plurality of positions in a predetermined area according to the present embodiment and a first object area and a second object area of the object. As shown in FIG. 4, the object area specifying unit 122 can perform learning in advance using the information obtained by manually adding rectangular information to the captured image to create a detection model.

The object area specifying unit 122 may use two detection models in which the first object area and the second object area are individually learned, but from the viewpoint of inference time, the first object area is the first class. It is desirable to train the second object area as the second class and use a single detection model capable of detecting the first object area and the second object area at once.

FIG. 5 is a diagram showing an example of the result of specifying the first object area and the second object area using the detection model. It is confirmed that the captured image shown in FIG. 5 shows the three objects P1 to P3, and that the four first object areas H1 to H4 and the five second object areas F1 to F5 are detected. can. The first object areas H3 and H4 and the second object areas F4 and F5 are false positives. On the other hand, the first object area corresponding to the object P3 is undetected (False Negative).

For example, when the object is detected based on the detection of the second object region by using the conventional method disclosed in Patent Document 1, the precision in the example of FIG. 5 is 3/5 = 60%. The detection rate (Recall) is 3/3 = 100%. The object position specifying device according to the present embodiment also aims to improve the conformance rate.

When the three-dimensional model imitating an object is arranged at each of a plurality of positions in a predetermined area, the correspondence identification unit 123 is set in the first area of the projection area which is the area of the three-dimensional model reflected in the captured image. The correspondence relationship between the corresponding first model area and the second model area corresponding to the second area is specified (S3).

FIG. 6 is a diagram showing an example of a three-dimensional model according to the present embodiment. For example, assuming that only a person of a specific height as an object exists in a predetermined area and all the persons are upright, the height of the 3D model to be used is set to a constant value (for example, 1.7 m). .. The shape of the three-dimensional model may be a simple one, and as shown in FIG. 6, the head may be formed of a sphere and the body portion may be formed of a cylinder. Further, assuming that the height of the person is 1.7 m and the height of the person is 7, as shown in FIG. 6, the diameter of the sphere corresponding to the head may be 0.2 m.

The correspondence relationship specifying unit 123 calculates the projection area of the three-dimensional model included in the image captured by the image pickup apparatus when the three-dimensional model is virtually arranged at a position on a predetermined area. For example, the storage unit 11 stores point cloud information indicating a point cloud constituting the three-dimensional model, and an image pickup device showing a correspondence relationship between one point of the image captured by the image pickup device and one point in the three-dimensional space. Memorize the parameters related to the calibration of. Here, each of the plurality of points constituting the point cloud constituting the three-dimensional model is represented by the coordinates in the three-dimensional space. Further, the parameters include external parameters determined by the installation position and installation direction of the image pickup device, and internal parameters related to image distortion of the image pickup device.

The correspondence relationship specifying unit 123 acquires the point cloud information and the parameter from the storage unit 11, and when the three-dimensional model is virtually arranged at a position on a predetermined area, the point cloud information indicates the point cloud. For at least a part, the projection point on the image captured by the image pickup device is calculated based on the parameter. Then, the correspondence relationship specifying unit 123 specifies the circumscribing rectangle of the polygonal region that approximates the contour of the region including the point group of the projection points corresponding to the first region (head) as the first model region, and the second model region. The circumscribing rectangle of the polygonal region that approximates the contour of the region including the point group of the projection points corresponding to the region (whole body) is specified as the second model region.

FIG. 7 is a diagram showing an example of specifying a first model area and a second model area according to the present embodiment. As shown in FIG. 7, the correspondence relationship specifying unit 123 is, for example, a first model when a three-dimensional model is virtually arranged at a position (X, Y) on one plane (Z = 0) of a predetermined area. Rectangular information (u_h, v_h, w_h, h_h) indicating the position on the captured image indicating the region Hm, and rectangular information (u_f, v_f, w_f, h_f) indicating the position on the captured image indicating the second model region Fm. Can be confirmed that has been specified. Here, u_h, v_h, and u_f, v_f are information indicating the center coordinates of the circumscribed rectangle. Further, w_h and w_f are information indicating the width of the circumscribed rectangle. Further, h_h and h_f are information indicating the height of the circumscribed rectangle.

The correspondence relationship specifying unit 123 generates the following eight-dimensional vector as information indicating the correspondence relationship between the rectangular information indicating the first model area and the rectangular information indicating the second model area.
(U_h, v_h, w_h, h_h, u_f, v_f, w_f, h_f)

The correspondence relationship specifying unit 123 identifies a plurality of 8-dimensional vectors showing the relationship between the first model area and the second model area when the three-dimensional model is virtually arranged at a plurality of positions in a predetermined area. The correspondence between the first model area and the plurality of second model areas is specified. The correspondence relationship specifying unit 123 stores the specified plurality of 8-dimensional vectors as model information in the storage unit 11.

In the present embodiment, the correspondence relationship specifying unit 123 specifies the correspondence relationship after the processing of the object area specifying unit 122, but the present invention is not limited to this. The object area specifying unit 122 may specify in advance the correspondence between the first model area and the second model area, and store the model information in the storage unit 11. Further, the correspondence relationship specifying unit 123 refers to an 8-dimensional vector indicated by model information in order to reduce the calculation cost when the model area specifying unit 124, which will be described later, specifies the second model area corresponding to the first object area. As the data to be generated, the first two-dimensional data to be used as a query at the time of reference may be indexed by a kd (k-dimensional) tree.

In addition, there may be cases where people of various heights exist in a predetermined area, or the posture of the person may change. In this case, assuming that the size and shape of the object in the predetermined area are constant, there is a problem that the accuracy of specifying the position of the object is lowered. Therefore, the correspondence relationship specifying unit 123 may specify the correspondence relationship between the first model area and the second model area corresponding to the three-dimensional model corresponding to each of the plurality of sizes and the plurality of shapes that the object can take. ..

FIG. 8 is a diagram showing an example of a plurality of three-dimensional models having different sizes according to the present embodiment. In this embodiment, an example of a plurality of three-dimensional models having different sizes will be described, but detailed description will be omitted because the same can be applied to three-dimensional models having different shapes. In the example shown in FIG. 8, the height range of the person as an object is assumed to be 1.3 m to 1.7 m, and three three-dimensional models are defined at intervals of 0.2 m. In this case, the correspondence identification unit 123 corresponds to each of the three three-dimensional models for one position in the predetermined area, and is a rectangle indicating the position on the captured image indicating the first model area as shown below. The information and the rectangular information indicating the position on the captured image indicating the second model area are specified.

The height of the object is 1.7m
Rectangle information of the first model area: (u1_h, v1_h, w1_h, h1_h)
Rectangle information of the second model area: (u1_f, v1_f, w1_f, h1_f)

The height of the object is 1.5m
Rectangle information of the first model area: (u2_h, v2_h, w2_h, h2_h)
Rectangle information of the second model area: (u2_f, v2_f, w2_f, h2_f)

The height of the object is 1.3m
Rectangle information of the first model area: (u3_h, v3_h, w3_h, h3_h)
Rectangle information of the second model area: (u3_f, v3_f, w3_f, h3_f)

Then, the correspondence relationship specifying unit 123 can generate the following three model information for one position in the predetermined area.
(U1_h, v1_h, w1_h, h1_h, u1_f, v1_f, w1_f, h1_f)
(U2_h, v2_h, w2_h, h2_h, u2_f, v2_f, w2_f, h2_f)
(U3_h, v3_h, w3_h, h3_h, u3_f, v3_f, w3_f, h3_f)

The larger the height range of the 3D model and the narrower the height interval of the plurality of 3D models, the more accurately the position of the object can be specified by the position specifying unit 125 described later. be. On the other hand, there is a problem that the capacity of the specified model information increases, and a problem that the calculation cost for specifying the second model area corresponding to the first object area in the model area specifying unit 124 described later increases. be.

Therefore, the correspondence relationship specifying unit 123 may integrate the correspondence relationship between the first model area and the second model area corresponding to the plurality of first model areas at relatively close positions. For example, the correspondence relationship specifying unit 123 refers to the rectangular information corresponding to the plurality of first model regions, identifies a plurality of first model regions having relatively close center positions, and identifies the plurality of first model regions of the plurality of first model regions. Integrate multiple correspondences corresponding to each.

FIG. 9 is a diagram showing an example of a first model region having the same center position. The example shown in FIG. 9 shows an example in which the center positions of the first model regions Hm1 to Hm3 of the three three-dimensional models having different arrangement positions and different sizes are the same. Here, since the center positions of the three first model regions Hm1 to Hm3 are the same, the information (u1_h, v1_h), (u2_h, v2_h), (u3_h,) indicating the center positions of the first model regions Hm1 to Hm3 are the same. In v3_h), the following relationship holds.
u1_h = u2_h = u3_h
v1_h = v2_h = v3_h

In this case, the correspondence relationship specifying unit 123 stores, for example, a 20-dimensional vector configured as follows in the storage unit 11 as model information indicating the correspondence relationship having the same center position. By doing so, the calculation cost can be suppressed when the model information is referred to, and the storage amount can also be reduced.
(U1_h, v1_h, w1_h, h1_h, u1_f, v1_f, w1_f, h1_f,
w2_h, h2_h, u2_f, v2_f, w2_f, h2_f,
w3_h, h3_h, u3_f, v3_f, w3_f, h3_f)

The model area specifying unit 124 is a plurality of first projection areas corresponding to the first area among the projection areas which are the areas of the three-dimensional model reflected in the captured image when the three-dimensional model is arranged at each of the plurality of positions in the predetermined area. Based on the correspondence between the one model area and the plurality of second model areas corresponding to the second area, the first distance indicating the distance to the first object area specified by the object area specifying unit 122 is relatively short. The second model area corresponding to the first model area is specified (S4).

Specifically, the model area specifying unit 124 is a first distance indicating a distance to the specified first object area based on the model information specified by the correspondence specifying unit 123 and stored in the storage unit 11. Specifies a second model region corresponding to a relatively short first model region.

The model area specifying unit 124 calculates the overlap rate between the specified first object area and the first model area, and sets the first distance so that the larger the overlap rate is, the shorter the first distance is. calculate. For example, the model area specifying unit 124 calculates a value obtained by subtracting the overlap rate from 1 as the first distance. Here, the duplication rate is IoU (Intersection over Union). When the specified first object area is A and the first model area is B, the overlap rate is represented by (A∩B) / (A∪B). The model area specifying unit 124 refers to the model information stored in the storage unit 11 and specifies a second model area corresponding to the first model area in which the first distance from the first object area is relatively short. ..

The model area specifying unit 124 may specify a plurality of second model areas corresponding to the first model area in which the first distance indicating the distance to the specified first object area is relatively short. By doing so, the model area specifying unit 124 can specify a plurality of second model areas corresponding to the three-dimensional models having different sizes and postures with respect to the specified first object area.

Further, the model area specifying unit 124 may refer to the model information using the kd tree and specify the first model area having a relatively short distance from the center position of the specified first object area. Here, although the first model area can be specified at high speed by using the kd tree, the specified first model area does not necessarily have a large overlap rate with the first object area. Therefore, the model area specifying unit 124 first identifies one or more first model areas having a position indicating the first object area and a relatively close center position, and then one or more specified first model areas. Of these, the first model area with a relatively short first distance may be specified by calculating the overlap rate with the specified first object area.

FIG. 10 is a diagram illustrating an example in which the model area specifying unit 124 according to the present embodiment specifies a second model area. As shown in FIG. 10, for example, the model area specifying unit 124 has specified the four first model areas shown in FIG. 10 as the first model area having a relatively short distance from the center position of the first object area. And. In this case, the model area specifying unit 124 calculates the first distance between the first object area and each of the four first model areas. For example, when the first distance between the first object area and the first model area whose rectangular information is (u3_h, v3_h, w3_h, h3_h) is the shortest, the model area specifying unit 124 is set in the first model area. As the associated second model area, the second model area whose rectangular information is (u3_f, v3_f, w3_f, h3_f) is specified.

The model area specifying unit 124 specifies a second model area for each of the plurality of first object areas specified by the object area specifying unit 122. FIG. 11 is a diagram showing an example in which the second model area is specified with respect to the result of specifying the first object area and the second object area shown in FIG. As shown in FIG. 11, it can be confirmed that the second model areas Fm1 to Fm4 are specified with respect to the first object areas H1 to H4. In addition, since the figure becomes complicated, FIG. 11 does not show the first model region Hm specified in the specific process of the second model region.

In the position specifying unit 125, when the second distance indicating the distance between the second object area specified by the object area specifying unit 122 and the second model area specified by the model area specifying unit 124 is equal to or less than the first threshold value. Specify the position of the object in (S5).

Specifically, the position specifying unit 125 is a second distance from the second model area specified by the model area specifying unit 124 for each of the one or more second object areas specified by the object area specifying unit 122. Specifies a second model region where is less than or equal to the first threshold. Then, the position specifying unit 125 specifies the position of the object indicated by the second object area with respect to the second object area for which the second model area can be specified. When a plurality of second model regions are specified, the position specifying unit 125 has a second distance that is relatively short among the second distances corresponding to each of the specified one or more second model regions. When it is equal to or less than the threshold value of 1, the position of the object corresponding to the second object area is specified.

The position specifying unit 125 calculates the overlap rate between the specified second object area and the second model area, and calculates the second distance so that the larger the overlap rate is, the shorter the second distance is. do. For example, the position specifying unit 125 calculates a value obtained by subtracting the overlap rate from 1 as the second distance. Here, the duplication rate is the IoU of the second object area and the second model area.

In the example shown in FIG. 11, the second model region having the smallest second distance with respect to each of the second object regions F1, F2, F3, F4 and F5 is Fm1, Fm2, Fm3, respectively, and is not applicable (not applicable). There is no overlap), Fm4. In the example shown in FIG. 11, the position specifying unit 125 identifies the second object areas F1 and F2 as the second object area in which the second distance from the second model area is equal to or less than the first threshold value.

The position specifying unit 125 corresponds to the second model area when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value. Based on the placement position of the three-dimensional model, the position of the object corresponding to the second object area in the predetermined area is specified.

For example, the position correspondence information (u_f,) is associated with the rectangular information indicating the second model area and the position (X, Y) on the predetermined area where the three-dimensional model corresponding to the second model area is arranged. It is stored in advance in the storage unit 11 as v_f, w_f, h_f, X, Y). The positions (X, Y) on the predetermined area may be associated with the model information indicating the correspondence between the first model area and the second model area.

The position specifying unit 125 refers to the position correspondence relation information when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value. The position (X, Y) on the predetermined area associated with the second model area is specified as the position on the predetermined area of the object corresponding to the specified second object area.

Further, the position specifying unit 125 is a position specifying unit 125 of the second object area when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value. Specify the position of the object corresponding to the specified second object region in the captured image based on at least one of the rectangular information indicating the position in the captured image and the rectangular information indicating the position in the captured image of the second model region. do. By doing so, the object position specifying device 1 can also specify the position of the object in the captured image.

As described above, the precision in the examples of FIGS. 5 and 11 when the object position specifying device 1 is used is 2/2 = 100%, and the conventional method described in Patent Document 1 is used. If there is, the precision rate exceeds 60%. On the other hand, the second object area F3, which could be detected by the conventional method, is not detected by the above processing, so that the detection rate (Recall) is lower than that of the conventional method, which is 2/3 = 66%. Become.

Therefore, the object position specifying device 1 may accept the setting of whether or not to give priority to the conformance rate via the input unit (not shown). Here, the setting that gives priority to the precision rate is called "matching rate priority mode", and the setting that raises the detection rate without giving priority to the matching rate is called "detection rate matching rate balanced mode".

When set to the "adaptation rate priority mode", the position specifying unit 125 specifies a third distance indicating the distance from the one or more first object areas for each of the one or more second object areas. Of the one or more second object regions, the relatively short third distance is equal to or less than the second threshold value, and the second distance from the second model region is equal to or less than the first threshold value in the second object region. Identify the position of the corresponding object.

Further, when the "detection rate matching rate balanced mode" is set, the position specifying unit 125 is, in addition to the processing corresponding to the "matching rate priority mode", relative to one or more second object areas. The position of the object corresponding to the second object area where the short third distance exceeds the second threshold value and the second distance from the second model area is equal to or less than the first threshold value is specified.

Here, the position specifying unit 125 identifies and identifies a large ratio of the ratio of the second object area and the ratio of the first object area to the union of the second object area and the first object area. The third distance is calculated so that the larger the ratio, the shorter the third distance. When the first object area is A and the second object area is B, the position specifying unit 125 sets the third distance to, for example, 1-max (A / (A∪B), B / (A∪B)). ). Here, max () is a function that selects the larger numerical value among the two numerical values passed internally. By doing so, when the first object area A is included in the second object area B or the second object area B is included in the first object area A, the third distance is specified as zero. The object.

In the example shown in FIG. 11, F3 and F4 are specified as the second object region in which the third distance from the first object region exceeds the second threshold value (for example, 0.6). Further, with respect to the second object areas F1, F2 and F5, H1, H2 and H4 are specified as the first object areas having the smallest third distance, respectively.

Further, among the second object areas F1, F2 and F5, F1 and F2 are specified as the second object area in which the second distance from the second model area is equal to or less than the first threshold value (for example, 0.6). The object. The position specifying unit 125 is combined with the second object areas F1 and F2 that specify the position of the object when the “detection rate matching rate balanced mode” is set and the “matching rate priority mode” is set. The positions of the objects corresponding to the second object areas F3 and F4 are also specified. For example, the position specifying unit 125 refers to the model information stored in the storage unit 11, identifies a second model area in which the second distance is equal to or less than the first threshold value, and positions the second model area. The position of the object corresponding to the second object areas F3 and F4 is specified based on. As a result, the detection rate (Recall) is improved from 66% in the precision rate priority mode to 3/3 = 100%. On the other hand, the precision rate drops from 100% to 3/4 = 75%. Both the "matching rate priority mode" and the "detection rate matching rate balanced mode" of the object position specifying device 1 according to the present embodiment can improve the matching rate as compared with the conventional method.

[Effects in this embodiment]
As described above, the object position specifying device 1 according to the present embodiment has a first object area, which is a first area showing an object reflected in a captured image in a predetermined area, and a second area, which is different from the first area. 2 Specify the object area. The object position specifying device 1 is a first region corresponding to the first region of the projection region of the three-dimensional model displayed in the captured image when the three-dimensional model imitating an object is arranged at each of a plurality of positions in a predetermined area. Based on the correspondence between the model area and the second model area corresponding to the second area, the second model area corresponding to the first model area having a relatively short distance from the specified first object area is specified. Then, the object position specifying device 1 specifies the position of the object when the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value. By doing so, the object position specifying device 1 can maintain the accuracy of specifying the position of the object regardless of the type of the image pickup device and the image pickup angle.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. For example, in the above embodiment, the second region is a region showing the whole body of a person as an object, but the region is not limited to this. The second region may be a region showing a human torso.

Further, in the above embodiment, the object position specifying device 1 compares the second object area specified from the captured image with the second model area specified based on the first object area specified from the captured image, and is appropriate. By specifying the second object area having a positional relationship, it is decided to suppress the over-detection of the object, but the present invention is not limited to this. For example, the object position specifying device 1 may specify the first model area corresponding to the second object area by referring to the second object area specified from the captured image and the model information. Then, the object position specifying device 1 compares the specified first object area with the specified first model area, and identifies the first object area having an appropriate positional relationship to suppress over-detection of the object. You may do it.

Further, in particular, the specific embodiment of the distribution / integration of the apparatus is not limited to those shown above, and all or a part thereof may be arbitrary according to various additions or the like, or according to the functional load. It can be functionally or physically distributed / integrated in units.

1 ... Object position specifying device, 11 ... Storage unit, 12 ... Control unit, 121 ... Acquisition unit, 122 ... Object area specifying unit, 123 ... Correspondence relationship specifying unit, 124.・ ・ Model area identification part, 125 ・ ・ ・ Position identification part

Claims (16)

An acquisition unit that acquires an image captured by an image pickup device that captures a predetermined area, and an acquisition unit.
An object area specifying unit that specifies a first object area, which is a first area showing an object reflected in the captured image, and a second object area, which is a second area different from the first area and showing an object reflected in the captured image. When,
A plurality of projection regions corresponding to the first region among the projection regions that are the regions of the three-dimensional model reflected in the captured image when the three-dimensional model imitating the object is arranged at each of a plurality of positions in the predetermined area. The first distance indicating the distance to the specified first object area is relatively short based on the correspondence between the first model area of the above and the plurality of second model areas corresponding to the second area. A model area specifying unit that specifies a second model area corresponding to one model area, and
A position specifying unit that specifies the position of the object when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value.
An object locating device. The model area specifying unit is one or more second model areas corresponding to one or more first model areas having a relatively small first distance to the specified first object area based on the correspondence relationship. Identify and
When the relatively short second distance of the second distances corresponding to each of the specified one or more second model regions is equal to or less than the first threshold value, the position specifying unit is used. Identify the position of the object,
The object position specifying device according to claim 1. Of the projection areas that are the areas of the three-dimensional model that appear in the captured image when the three-dimensional model is placed at each of a plurality of positions in the predetermined area, the first model area and the second model area. Further equipped with a correspondence relationship identification part that identifies the correspondence relationship with
The model area specifying unit identifies a second model area corresponding to the first model area in which the first distance to the specified first object area is relatively short, based on the specified correspondence relationship. ,
The object position specifying device according to claim 1 or 2. The correspondence identification unit identifies the correspondence corresponding to the three-dimensional model corresponding to each of the plurality of sizes that the object can take.
The model area specifying unit is a plurality of second models corresponding to a plurality of first model areas having a relatively short first distance to the specified first object area based on the specified correspondence relationship. Identify the area,
The object position specifying device according to claim 3. The correspondence relationship specifying unit identifies the correspondence relationship corresponding to the three-dimensional model corresponding to each of the plurality of shapes that the object can take.
The model area specifying unit is a plurality of second models corresponding to a plurality of first model areas having a relatively short first distance to the specified first object area based on the specified correspondence relationship. Identify the area,
The object position specifying device according to claim 3 or 4. The correspondence identification unit integrates the correspondences corresponding to a plurality of first model regions at relatively close positions.
The object position specifying device according to any one of claims 3 to 5. The model area specifying unit calculates the overlap rate between the specified first object area and the first model area, and the larger the overlap rate, the shorter the first distance. Calculate the distance of 1
The object position specifying device according to any one of claims 1 to 6. The model area specifying unit specifies one or more of the first model areas having a position relatively close to the position indicating the first object area, and specifies among the specified one or more of the first model areas. The second model area corresponding to the first model area having a relatively short first distance from the first object area is specified.
The object position specifying device according to claim 7. The position specifying unit calculates the overlap rate between the specified second object area and the specified second model area, and the larger the overlap rate is, the shorter the second distance is. Calculate the second distance,
The object position specifying device according to any one of claims 1 to 8. The object area specifying unit identifies one or more of the first object areas and one or more of the second object areas based on the captured image.
The position specifying unit specifies a third distance indicating the distance from the one or more first object areas for each of the one or more second object areas, and among the one or more second object areas. The position of the object corresponding to the second object region where the relatively short third distance is equal to or less than the second threshold value and the second distance from the second model region is equal to or less than the first threshold value. To identify,
The object position specifying device according to any one of claims 1 to 9. The position specifying unit specifies a third distance indicating the distance from the one or more first object areas for each of the one or more second object areas, and among the one or more second object areas. The position of the object corresponding to the second object region where the relatively short third distance exceeds the second threshold value and the second distance to the second model region is equal to or less than the first threshold value. To identify,
The object position specifying device according to claim 10. The position specifying unit has specified and specified a large ratio of the ratio of the second object area and the ratio of the first object area to the union of the second object area and the first object area. The third distance is calculated so that the larger the ratio is, the shorter the third distance is.
The object position specifying device according to claim 10 or 11. The position specifying unit is set in the second model area when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value. Identifying the position of the object in the predetermined area based on the placement position of the corresponding three-dimensional model.
The object position specifying device according to any one of claims 1 to 12. The position specifying unit refers to the second object area when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value. The position of the object in the captured image is specified based on at least one of the position in the captured image and the position in the captured image of the second model region.
The object position specifying device according to any one of claims 1 to 12. Computer runs,
A step of acquiring an image captured by an image pickup device that captures a predetermined area, and
A step of specifying a first object area, which is a first area showing an object reflected in the captured image, and a second object area, which is a second area different from the first area showing an object reflected in the captured image.
A plurality of projection regions corresponding to the first region among the projection regions that are the regions of the three-dimensional model reflected in the captured image when the three-dimensional model imitating the object is arranged at each of a plurality of positions in the predetermined area. Based on the correspondence between the first model region of the above and the plurality of second model regions corresponding to the second region, the first distance indicating the distance to the specified first object region is relatively short. Steps to identify the second model area corresponding to one model area,
A step of specifying the position of the object when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value.
How to locate an object. Computer,
An acquisition unit that acquires an image captured by an image pickup device that captures a predetermined area.
An object area specifying unit that specifies a first object area, which is a first area showing an object reflected in the captured image, and a second object area, which is a second area different from the first area and showing an object reflected in the captured image. ,
A plurality of projection regions corresponding to the first region among the projection regions that are the regions of the three-dimensional model reflected in the captured image when the three-dimensional model imitating the object is arranged at each of a plurality of positions in the predetermined area. The first distance indicating the distance to the specified first object area is relatively short based on the correspondence between the first model area of the above and the plurality of second model areas corresponding to the second area. A model area specifying unit that specifies a second model area corresponding to one model area, and
A position specifying unit that specifies the position of the object when the second distance indicating the distance between the specified second object area and the specified second model area is equal to or less than the first threshold value.
An object position identification program that functions as.

Images