JP6995714B2 - Specific device, specific method and specific program - Google Patents

Description

The present invention relates to a specific device, a specific method, and a specific program for specifying the orientation of an object.

Conventionally, when analyzing the behavior of a clerk or a customer who stays in a predetermined area such as in a store for the purpose of marketing, the position of an object indicating a person is specified from a captured image obtained by capturing the predetermined area, and a person flow line is generated. Technology is used. Further, in order to enable advanced behavioral analysis, a technique for estimating the orientation of a person from an image of an object showing a person using machine learning or deep learning has been proposed (see, for example, Non-Patent Document 1). ).

Chen, Cheng, and Jean-Marc Odobez. "We are not contortionists: Coupled adaptive learning for head and body orientation estimation in surveillance video." Computer Vision and Pattern Recognition (CVPR), 2012.

In order to construct an image classifier by machine learning or deep learning, a large amount of learning data in which the orientation information of the object is given as teacher data (teacher label) is required for the image of the object. FIG. 11 is a diagram showing an example of learning data. For example, if four classes of front, left side, back side, and right side are defined as the orientation of the object to be reflected in the image, a large number of images having one of the class labels as a teacher label are required as shown in FIG. It becomes. On the other hand, when the orientation information is manually given to the image of each object, the worker needs to specify the orientation of the object included in the image. Therefore, the work of manually assigning the orientation information requires a great deal of labor.

Therefore, the present invention has been made in view of these points, and an object of the present invention is to provide a specific device, a specific method, and a specific program capable of efficiently specifying the orientation of an object reflected in an image.

The specific device according to the first aspect of the present invention includes an acquisition unit that acquires an image captured by an image pickup device that captures a predetermined area, and an object that identifies a predetermined object included in the first image acquired by the acquisition unit. The predetermined unit, the first position indicating the position of the predetermined object at the first time corresponding to the first image acquired by the acquisition unit, and the predetermined position at a second time different from the first time. The orientation of the predetermined object in the first image based on the object position specifying unit that specifies the second position indicating the position of the object and the first position and the second position specified by the object position specifying unit. It is provided with an object orientation specifying part for specifying.

The acquisition unit acquires movement line information indicating the position of the predetermined object at each time, and the object position identification unit obtains the predetermined object at the first time based on the first image acquired by the acquisition unit. The first position indicating the position of the object is specified, and the second position indicating the position of the predetermined object at the second time, which is a time different from the first time, is specified based on the movement line information. You may.

The object position specifying unit identifies the first position indicating the position of the predetermined object at the first time based on the first image acquired by the acquisition unit, and the second image acquired by the acquisition unit. Based on the second image captured at the time, the second position indicating the position of the predetermined object at the second time may be specified.

The object position specifying unit may specify the position of an object separated from the first position by a predetermined distance or more as the second position.
The object orientation specifying portion is the predetermined object orientation specifying portion based on an angle formed by a moving direction of the predetermined object determined based on the first position and the second position and a reference direction defined in the first image. You may specify the orientation of the object.

The object orientation specifying unit may specify the direction from the first position toward the position of the reference point defined in the first image as the reference direction.
The object orientation specifying portion refers to a projection point on the plane of an intersection of a plane that is a virtual extension of the image plane of the image pickup device and a perpendicular line drawn from the optical center of the image pickup device to the predetermined area. It may be used as a reference point.

Each of the plurality of angle ranges is associated with the orientation of one object, and the object orientation identification unit is associated with the orientation of the predetermined object to the angle range including the angle. The orientation of the object may be specified.

The object position specifying unit specifies the position of the predetermined object at each of three or more times related to the first image acquired by the acquisition unit, and the object orientation specifying unit has three or more specified objects. For each of the times, the movement direction of the object is specified based on the position of the predetermined object at the time, and the movement direction of the object is specified based on the movement direction of the object at each of the three or more specified times. You may specify the direction of.

Each of the plurality of angle ranges is associated with the orientation of one object, and the object orientation identification unit includes an angle corresponding to each of the three or more specified times. The orientation of one object associated with the angle range may be specified and the orientation of the predetermined object may be specified based on the proportion including the orientation of the identified object.

The specifying device includes an object area specifying unit that specifies an area in the first image of the predetermined object, an image indicated by the area specified by the object area specifying unit, and the orientation specified by the object orientation specifying unit. It may be further provided with an extraction unit for extracting the orientation information indicating the above as learning data in machine learning.

The object orientation specifying portion is the predetermined object orientation specifying portion based on an angle formed by a moving direction of the predetermined object determined based on the first position and the second position and a reference direction defined in the first image. The orientation of the object may be specified, and the extraction unit may control not to extract the training data when the angle is outside the predetermined range.
The extraction unit may control not to extract training data when the distance between the first position and the reference point on the image defined in the first image is less than a predetermined value.

The specific method according to the second aspect of the present invention includes a step of acquiring an image captured by an image pickup device that captures a predetermined area, which is executed by a computer, and a predetermined image included in the first image acquired in the acquisition step. The first position indicating the position of the predetermined object in the first time corresponding to the first image acquired in the step of specifying the object, and the time different from the first time. Specifying the orientation of the predetermined object in the first image based on the step of specifying the second position indicating the position of the predetermined object at the second time and the specified first position and the second position. With steps to do.

The specific program according to the third aspect of the present invention identifies a computer as an acquisition unit that acquires an image captured by an image pickup device that captures a predetermined area, and a predetermined object included in the first image acquired by the acquisition unit. The object specifying unit, the first position indicating the position of the predetermined object at the first time corresponding to the first image acquired by the acquisition unit, and the second time at a time different from the first time. An object position specifying unit that specifies a second position indicating the position of a predetermined object, and the predetermined object in the first image based on the first position and the second position specified by the object position specifying unit. It functions as an object orientation specification part that specifies the orientation of.

According to the present invention, there is an effect that the orientation of the object reflected in the image can be efficiently specified.

It is a figure which shows the outline of the learning data generation apparatus which concerns on this embodiment. It is a figure which shows the structure of the learning data generation apparatus which concerns on this embodiment. It is a figure which shows the example which specifies the 2nd position when the predetermined distance is a predetermined number of pixels in an image coordinate system. It is a figure explaining an example which specifies the orientation of an object when the 1st image which the image pickup apparatus has taken is an omnidirectional image. It is a figure explaining the example which specifies the orientation of an object when the 1st image which the image pickup apparatus has taken is a normal image. It is a figure which shows the example which specifies the reference point. It is a figure which shows the example which specified the orientation of the object with respect to the object which is presumed to have taken the front of the object. It is a figure which shows the example which the orientation of the object is specified with respect to the object which is presumed to have imaged the back surface of the object. It is a figure which shows the example which specified the position of the object at each of three or more times related to the 1st image. It is a flowchart which shows the flow of the process from the acquisition of the first image to the extraction of the learning data by the learning data generation apparatus which concerns on this embodiment. It is a figure which shows the example of the training data.

[Overview of learning data generation device 1]
FIG. 1 is a diagram showing an outline of the learning data generation device 1 according to the present embodiment. The learning data generation device 1 is used in machine learning in which an image of a predetermined object included in an image captured in a predetermined area captured by an image pickup device (not shown) is associated with a direction in which the predetermined object appears in the image. It is a computer that generates training data, and functions as a specific device that specifies the orientation of a predetermined object.

Here, the predetermined area is, for example, the floor of a store. In the predetermined area, a person such as a clerk exists as a predetermined object to be analyzed for movement. In the following description, a predetermined object is simply referred to as an object.

As shown in FIG. 1, the learning data generation device 1 acquires a captured image of a predetermined area captured by the imaging device ((1) in FIG. 1) and identifies an object included in the captured image ((1) in FIG. 1 (FIG. 1). 2)). The learning data generation device 1 specifies the position of the object at the first time corresponding to the captured image ((3) in FIG. 1), and also specifies the position of the object at the second time different from the first time (FIG. 1). (4)).

The learning data generation device 1 specifies the orientation of the object based on the relationship between the position of the specified object at the first time, the position of the object at the second time, and the reference direction in the first image (FIG. 1). (5)). By doing so, the learning data generation device 1 specifies the moving direction of the object shown in the captured image, and determines the direction in which the object is shown in the image based on the relationship between the moving direction and the reference direction. It can be identified with high accuracy and efficiency.

After that, the learning data generation device 1 extracts an image of the object appearing in the captured image and orientation information indicating the direction in which the specified object appears in the image as learning data in machine learning ((6) in FIG. 1). )). By doing so, the learning data generation device 1 can generate the learning data at a higher speed than in the case of manually generating the learning data.

[Configuration of learning data generator 1]
Subsequently, the configuration of the learning data generation device 1 will be described. FIG. 2 is a diagram showing a configuration of a learning data generation device 1 according to the present embodiment. As shown in FIG. 2, the learning data generation device 1 includes a storage unit 11 and a control unit 12.

The storage unit 11 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The storage unit 11 stores a program executed by the control unit 12. For example, the storage unit 11 causes the learning data generation device 1 to function as an acquisition unit 121, an object identification unit 122, an object position identification unit 123, an object area identification unit 124, an object orientation identification unit 125, and an extraction unit 126. Remembers the generator. The learning data generation program includes an object orientation specifying program that causes the learning data generation device 1 to function as an acquisition unit 121, an object identification unit 122, an object position identification unit 123, an object area identification unit 124, and an object orientation identification unit 125. May be good.

Further, the storage unit 11 stores one or more images captured by the image pickup device that captures a predetermined area. The image pickup device is, for example, a camera installed on the ceiling or the like of a predetermined area, and generates an image by taking an image in a direction directly below or diagonally downward from above the predetermined area. Each of the plurality of captured images is associated with the capture time of the image. Examples of the captured image include an omnidirectional image generated by an omnidirectional camera installed on a ceiling or the like in a predetermined area, and an image generated by a general camera. In the present embodiment, an image captured by a general camera is referred to as a normal image.

Further, the storage unit 11 stores one or more flow line information indicating the position of the object in the predetermined area for each time, which is generated based on one or more captured images of the predetermined area. The flow line information is information in which the time is associated with the position information indicating the position of the object in a predetermined area at the time. It is assumed that the time included in the flow line information corresponds to any of the imaging times associated with each of the one or more captured images in which the predetermined area is captured. It is assumed that the flow line information includes position information corresponding to each of at least two times.

Here, the position information included in the flow line information is assumed to indicate, for example, a local position (for example, an image coordinate system) which is a position in a captured image. When the correspondence between the local position and the global position (for example, the world coordinate system) is stored in the storage unit 11, the position information included in the flow line information is, for example, the global position which is the position of a predetermined area. It may indicate.

The control unit 12 is, for example, a CPU (Central Processing Unit). By executing the learning data generation program stored in the storage unit 11, the control unit 12 includes an acquisition unit 121, an object identification unit 122, an object position identification unit 123, an object area identification unit 124, an object orientation identification unit 125, and an object orientation identification unit 125. It functions as an extraction unit 126.

The acquisition unit 121 acquires an image captured by an image pickup device that captures a predetermined area. Further, the acquisition unit 121 acquires the flow line information stored in the storage unit 11. Here, the image acquired by the acquisition unit 121 is referred to as a first image.

The object specifying unit 122 identifies an object included in the first image acquired by the acquisition unit 121. Specifically, the object specifying unit 122 identifies the flow line information including the time corresponding to the imaging time associated with the first image among the flow line information acquired by the acquisition unit 121, thereby specifying the acquisition unit. The object included in the first image acquired by 121 is specified.

Note that the flow line information may be associated with attribute information indicating the attributes of an object such as a clerk or a customer. The object identification unit 122 may accept the attribute of the object from the user of the learning data generation device 1, for example. Then, the object specifying unit 122 may specify only the object corresponding to the received attribute by specifying the flow line information having the received attribute.

The object position specifying unit 123 includes the position of the object at the first time, which is the imaging time corresponding to the first image acquired by the acquisition unit 121, and the position of the object at the second time, which is a time different from the first time. To identify. Here, it is assumed that the second time is a time before the first time. Further, the position of the object at the first time is also referred to as the first position, and the position of the object at the second time is also referred to as the second position.

The object position specifying unit 123 specifies the first position based on the first image acquired by the acquisition unit 121. For example, the object position specifying unit 123 refers to the flow line information including the time corresponding to the imaging time associated with the first image, and specifies the position information associated with the time, thereby specifying the first image. The first position corresponding to is specified.

The object position specifying unit 123 identifies a second position indicating the position of a predetermined object at the second time based on the flow line information including the time corresponding to the imaging time associated with the first image. Here, the object position specifying unit 123 may specify the position of the object separated from the first position by a predetermined distance or more as the second position.

The predetermined distance is based on, for example, the distance in the image coordinate system which is the coordinate system of the captured image (for example, in pixel units) or the distance in the world coordinate system which is the coordinate system of the predetermined area (for example, in meters). It is decided. FIG. 3 is a diagram showing an example of specifying a second position when a predetermined distance is set to a predetermined number of pixels (for example, 50 pixels) in the image coordinate system. Here, the imaging time associated with the first image is defined as time t. In the example shown in FIG. 3, at time t-1 and time t-2, which are times one frame before and two frames before time t, the distance to the object position at time t is less than a predetermined pixel, and time t 3 The distance becomes a predetermined pixel or more for the first time at time t-3, which is the time before the frame. Therefore, the object position specifying unit 123 specifies the position corresponding to the time t-3 as the second position.

The movement trajectory of the object may move slightly at the foot position even if it is actually almost stationary. On the other hand, the object position specifying unit 123 specifies the position of the object that is separated from the first position by a predetermined distance or more as the second position, so that the position before the movement when the object actually moves is set to the second position. Can be specified as. As a result, the learning data generation device 1 can improve the accuracy of specifying the moving direction of the object.

The object position specifying unit 123 may specify the first position by performing image analysis of the first image acquired by the acquisition unit 121. Further, the acquisition unit 121 may acquire the captured image captured at the second time, which is a different imaging time from the first time as the imaging time associated with the first image, as the second image. Then, the object position specifying unit 123 may specify the second position based on the second image acquired by the acquisition unit 121. For example, the object position specifying unit 123 may specify the second position by performing image analysis of the second image acquired by the acquisition unit 121. In this case, the object position specifying unit 123 may specify the first position and the second position based on the object area specified by the object area specifying unit 124 described later. For example, the object position specifying unit 123 may use the position of the center point of the object area as the position of the object. Further, the object position specifying unit 123 identifies one point (for example, if the object is a person, the foot position) among the point clouds constituting the three-dimensional model corresponding to the object described later, and on the image of the point. The position of the projection point on may be the position of the object. By doing so, the learning data generation device 1 can specify the first position and the second position without using the flow line information.

The object area specifying unit 124 specifies an object area, which is an area in the first image of the object. For example, the storage unit 11 stores the first image and the object area extracted in advance from the first image in association with each other, and the object area specifying unit 124 stores the object area associated with the first image. By acquiring from 11, the object area is specified.

Further, the object area specifying unit 124 is an image captured by the image pickup apparatus when the three-dimensional model corresponding to the object is virtually arranged at the position of the object in the first image specified by the object position specifying unit 123. The object area may be specified by calculating the area corresponding to the three-dimensional model included in. Specifically, the storage unit 11 stores the point cloud information indicating the point cloud constituting the three-dimensional model and the projection parameters of the image pickup apparatus. Then, the object area specifying unit 124 acquires the point cloud information and the parameter from the storage unit 11, and when the three-dimensional model is virtually arranged, the point cloud information indicates at least a part of the point cloud. Based on the projection parameters, the projection points on the image captured by the image pickup apparatus are calculated, and the contour region of the polygon that approximates the point cloud of the projection points is specified as the object region.

Further, the object area specifying unit 124 may specify a rectangular area including the specified contour area as an object area. The object area specifying unit 124 may specify a rectangular area having a relatively small area of the rectangle as an object area among a plurality of rectangular areas including the specified contour area.

Further, when the image pickup device is an image pickup device that captures an omnidirectional image, the object area identification unit 124 includes a straight line passing through a point corresponding to a rectangular area, a straight line passing through the center of the image captured by the image pickup device, and an omnidirectional image. The rotation component of the rectangular region may be calculated based on the angle formed by the straight line corresponding to the predetermined reference axis with respect to the image. Then, the object area specifying unit 124 may specify a rectangular area having a relatively small area as an object area among the rectangular areas having the calculated rotation component.

The object orientation specifying unit 125 in the first image is based on a first position indicating the position of the object at the first time specified by the object position specifying unit 123 and a second position indicating the position of the object at the second time. Identify the orientation of the object.

Hereinafter, with reference to FIGS. 4 to 6, the details of the process in which the object orientation specifying unit 125 specifies the orientation of the object will be described. FIG. 4 is a diagram illustrating an example of specifying the orientation of an object when the first image captured by the image pickup apparatus is an omnidirectional image. FIG. 5 is a diagram illustrating an example of specifying the orientation of an object when the first image captured by the image pickup apparatus is a normal image. FIG. 6 is a diagram showing an example of specifying a reference point.

The object orientation specifying unit 125 uses the premise that the orientation of the moving object is the same as the moving direction when specifying the orientation of the object. For example, if the object is moving forward, the object is facing forward, and if the object is moving left, the object is facing left.

The object orientation specifying unit 125 specifies the direction from the first position toward the position of the reference point defined in the first image as the reference direction. For example, as shown in FIG. 4, when the image captured by the imaging device is an image captured in the direct downward direction, the reference point Pb is the center position of the image, and the reference direction Db is the reference point from the first position P1. It is the direction toward Pb. When the image captured by the image pickup apparatus is an image captured in the direct downward direction, the object orientation specifying unit 125 specifies the center position of the image and specifies the center position as a reference point. The reference direction Db may be the direction from the second position P2 toward the reference point Pb.

Further, when the image captured by the image pickup apparatus is not an image captured in the direct downward direction, the object orientation specifying unit 125 specifies the reference point Pb as shown in FIG. Specifically, when the image captured by the image pickup device is not an image captured in the direct downward direction, the object orientation specifying unit 125 includes a plane obtained by virtually extending the image plane of the image pickup device as shown in FIG. The projection point on the plane of the intersection with the perpendicular line drawn from the optical center of the image pickup device to the predetermined area is defined as the reference point Pb. When the image captured by the image pickup device is not an image captured in the direct downward direction, the object orientation identification unit 125 includes a plane obtained by virtually extending the image plane of the image pickup device and a perpendicular line drawn from the optical center of the image pickup device to a predetermined area. The projection point of the intersection with and on the plane is specified, and the projection point is specified as a reference point. As shown in FIG. 6, when the installation angle of the image pickup device is slanted and the image plane is not directly below the optical center, the reference point Pb is outside the image plane (imaging plane) as shown in FIG. There is. When the image captured by the image pickup device is an image captured in the direct downward direction, the reference point specified according to the above is the center position of the image.

The reference point information indicating the position of the reference point may be stored in the storage unit 11 in advance. Then, the object orientation specifying unit 125 may specify the reference point by referring to the reference point information stored in the storage unit 11. The object orientation specifying unit 125 specifies the direction from the first position in the first image toward the specified reference point as the reference direction.

As shown in FIG. 5, it is an image of a normal camera, and when the reference point is largely separated from the object position on the image plane, the reference direction can be approximated to the direction directly below. In this case, the reference direction information indicating the reference direction is stored in the storage unit 11 in advance, and the object orientation specifying unit 125 refers to the reference direction information stored in the storage unit 11 to specify the reference direction. You may.

The object orientation specifying unit 125 specifies the moving direction of the object determined based on the first position and the second position. The object orientation specifying unit 125 specifies the moving direction of the object by subtracting the coordinate component indicating the second position from the coordinate component indicating the first position. For example, the object orientation specifying unit 125 specifies the moving direction of the object by subtracting the coordinate component of the second position from the coordinate component of the first position corresponding to the image coordinate system indicating the position on the image. Even if the object orientation specifying unit 125 specifies the moving direction of the object by subtracting the coordinate component of the second position from the coordinate component of the first position corresponding to the world coordinate system indicating the position on the predetermined area. good.

In the present embodiment, the second time is assumed to be a time before the first time, and the object orientation specifying unit 125 subtracts the coordinate component indicating the second position from the coordinate component indicating the first position. By doing so, the moving direction of the object was specified, but it is not limited to this. The second time may be a time after the first time. In this case, the object orientation specifying unit 125 may specify the moving direction of the object by subtracting the coordinate component indicating the first position from the coordinate component indicating the second position.

The object orientation specifying unit 125 specifies the orientation of the specified object in the image based on the angle formed by the moving direction of the specified object and the reference direction defined in the first image. In the example shown in FIGS. 4 and 5, the object orientation specifying unit 125 uses the direction from the second position P2 to the first position P1 as the object movement direction Dm. Then, the object orientation specifying unit 125 uses the angle θ formed by the movement direction Dm and the reference direction Db as the orientation information indicating the orientation of the object. The object orientation specifying unit 125 uses a relatively small angle θ among a plurality of angles formed by the moving direction Dm and the reference direction Db as orientation information indicating the orientation of the object in the image.

In both cases of FIG. 4 and FIG. 5, the object orientation specifying unit 125 automatically obtains object orientation information of θ = about 90 ° with respect to the object area image presumed to be an image of the left side surface of the object. Can be granted.

FIG. 7 is a diagram showing an example in which the orientation of an object is specified with respect to an object that is presumed to be an image of the front of the object. FIG. 8 is a diagram showing an example in which the orientation of the object is specified with respect to the object presumed to be an image of the back surface of the object. In the example shown in FIG. 7, it can be confirmed that θ is about 0 °, and in the example shown in FIG. 8, θ is about 180 °.

As shown in FIGS. 7 and 8, when the reference point exists in the image, the moving direction of the object on the image is the same direction (in FIGS. 7 and 8, the moving direction is the lower right direction). However, the orientations of the objects (front in FIG. 7, back in FIG. 8) are different. On the other hand, the learning data generation device 1 can accurately specify the direction of the object in the image by specifying the angle between the moving direction of the moving body and the reference direction.

The orientation of one object may be associated with each of the plurality of angle ranges. For example, in each range where θ is 0 ° ± α, 90 ° ± α, 180 ° ± α, and 270 ° ± α, four classes of front, left side, back, and right side are associated as object orientations. May be. Then, the object orientation specifying unit 125 may specify the orientation of the object to the orientation (class) of one object associated with the angle range including the angle θ. Further, the object orientation specifying unit 125 may specify the label value of the teacher data corresponding to the orientation of the object as the orientation information indicating the orientation of the object. For example, the label value of the front class is 0, the label value of the left side class is 1, the label value of the back class is 2, the label value of the right side class is 3, and the object orientation specifying unit 125 sets the orientation of the specified object. The corresponding label value may be specified. By assigning a discrete value as the label value of the teacher data, it is possible to learn as a classification problem in machine learning (for example, 4 classes in the above example).

Further, the object position specifying unit 123 may specify the position of the object at each of the three or more times related to the first image acquired by the acquisition unit 121. Then, the object orientation specifying unit 125 specifies the moving direction of the object based on the position of the object at the specified time for each of the specified three or more times, and the object at each of the specified three or more times. You may specify the orientation of the object based on the direction of movement of.

In this case, the object orientation specifying unit 125 specifies the angle between the moving direction of the object corresponding to each of the three or more specified times and the reference direction. FIG. 9 is a diagram showing an example of specifying the position of an object at each of three or more times related to the first image. In the example shown in FIG. 9, in addition to the first position P1 which is the position of the object corresponding to the imaging time (time t1) associated with the first image, the second position corresponding to the time before the imaging time. An example in which P2, the third position P3, and the fourth position P4 are specified is shown. Here, the second position P2, the third position P3, and the fourth position P4 are the positions of the objects corresponding to the times one frame before, two frames, and three frames before the imaging time. The object orientation specifying unit 125 specifies the angle θ1, the angle θ2, and the angle θ3 as the angles formed by the moving directions at the second position P2, the third position P3, and the fourth position P4.

The object orientation specifying unit 125 may use the average value or the median value of the angles θ1, θ2, and θ3 as the orientation information of the object. Further, as described above, the object orientation specifying unit 125 determines the orientation of one object associated with the angle range including the angle corresponding to each of the three or more specified times. It may be specified. Then, the object orientation specifying unit 125 may specify the orientation of the object based on the ratio including the orientation of the identified object.

For example, as described above, when the label value of the front class is 0, the label value of the left side class is 1, the label value of the back class is 2, and the label value of the right side class is 3, the object orientation specific unit 125 However, when the label values of 0, 0, and 1 are specified for the angles θ1, θ2, and θ3, the label value indicating the direction of the object is specified as 0 by a majority decision. By doing so, it is possible to accurately specify the moving direction of the object and accurately specify the orientation of the object based on the moving direction.

The extraction unit 126 extracts the image indicated by the area specified by the object area specifying unit 124 and the orientation information indicating the orientation of the object specified by the object orientation specifying unit 125 as learning data in machine learning, and extracts the learning data. Is stored in the storage unit 11. Here, it is assumed that machine learning includes deep learning. Machine learning includes Support Vector Machine and CNN (Convolutional Neural Network).

When the object orientation specifying unit 125 extracts the angle information indicating the angle θ, that is, the continuous value as the orientation information indicating the orientation of the object, the extraction unit 126 trains the extracted learning data as a regression problem in machine learning. be able to.

When the angle θ is outside the predetermined range, the extraction unit 126 may control so as not to extract the learning data corresponding to the angle θ. For example, in machine learning, when learning as the above-mentioned four-class classification problem, the learning data generation device 1 learns images that are difficult to clearly classify, such as the intermediate direction between the front surface and the left side surface. By omitting from, learning can be carried out efficiently.

Further, the object orientation specifying unit 125 sets a plurality of angles θ1, θ2, and θ3 at 0, 0, and 1, respectively, corresponding to the positions of the objects at each of the three or more times related to the first image (the above-mentioned above. In the case of discretizing with (in the case of a four-class problem), the extraction unit 126 may control so as not to extract the training data when the ratio including the discrete values is less than a predetermined value. For example, when the predetermined value is set to 100%, the extraction unit 126 extracts the training data only when all the discrete values are the same. Even in this way, the learning data generation device 1 can efficiently carry out learning by omitting images that are difficult to clearly classify from the learning data.

Further, the extraction unit 126 learns when the distance between the first position indicating the position of the predetermined object corresponding to the first image and the reference point on the image defined in the first image is less than a predetermined value. You may control not to extract the data. At a position close to the reference point on the first image, the image is taken from directly above the object, which is not suitable for estimating the orientation of the object. On the other hand, the learning data generation device 1 suppresses unnecessary learning because the distance between the first position and the reference point is less than a predetermined value and does not extract an image as if the object was captured from directly above. be able to.

[Process flow in the learning data generation device 1]
Subsequently, the flow of processing in the learning data generation device 1 will be described. FIG. 10 is a flowchart showing a flow of processing from the acquisition of the first image to the extraction of the learning data by the learning data generation device 1 according to the present embodiment. In this flowchart, the flow of processing when the flow line information is used to specify the object position and the direction of the object in the image is classified into any of the four classes will be described.

First, the acquisition unit 121 acquires the first image (S1).
Subsequently, the object specifying unit 122 identifies the object included in the first image by specifying the flow line information corresponding to the first image (S2).

Subsequently, the object position specifying unit 123 sets the first position, which is the position of the object at the first time, which is the imaging time corresponding to the first image, and the first time, based on the specified flow line information. The second position, which is the position of the object at the second time, which is a different time, is specified (S3).

Subsequently, the object orientation specifying unit 125 specifies the moving direction of the object based on the first position and the second position (S4).
Subsequently, the object orientation specifying unit 125 specifies an angle formed by the specified movement direction and the reference direction in the first image (S5), and specifies a class corresponding to the orientation of the object based on the specified angle (S5). S6).

Subsequently, the extraction unit 126 determines whether or not the object orientation specifying unit 125 can specify the class corresponding to the object orientation (S7). When the extraction unit 126 determines that the class can be specified, it shifts the process to S8, and when it determines that the class cannot be specified, the extraction unit 126 ends the process according to this flowchart.

Subsequently, the extraction unit 126 determines whether or not the distance between the first position and the reference point in the first image is equal to or greater than a predetermined value (S8). When the extraction unit 126 determines that the distance between the first position and the reference point is equal to or greater than a predetermined value, the process is transferred to S9, and when it is determined that the distance between the first position and the reference point is less than the predetermined value, the present invention is performed. End the process related to the flowchart. The extraction unit 126 may execute the process related to S8 at any timing during the process from S3 to S8, such as immediately after S3.

Subsequently, the object area specifying unit 124 specifies an object area, which is an area in the first image of the object (S9).
Subsequently, the extraction unit 126 extracts training data in which the specified object area and the class corresponding to the orientation of the object are associated with each other (S10).

In this flowchart, the object position specifying unit 123 specifies the position of the object based on the specified flow line information, but the present invention is not limited to this. The object position specifying unit 123 performs image analysis on at least one of the first image and the second image acquired by the acquisition unit 121, and thereby at least one of the first position and the second position, which is the position of the object. May be specified.

[Effects in this embodiment]
As described above, the learning data generation device 1 according to the present embodiment identifies a predetermined object included in the first image captured by the image pickup device that captures the predetermined area, and the first time corresponding to the first image. The first position indicating the position of the predetermined object in the above position and the second position indicating the position of the predetermined object at the second time, which is a time different from the first time, are specified. The learning data generation device 1 specifies the orientation of a predetermined object in the first image based on the specified first position and second position. By doing so, the learning data generation device 1 can efficiently specify the orientation of the object in the image, as compared with the case where the orientation of the object is manually specified.

Further, the learning data generation device 1 directs the predetermined object based on the angle formed by the moving direction of the predetermined object determined based on the first position and the second position and the reference direction determined in the first image. To identify. By doing so, the learning data generation device 1 can accurately specify the orientation of the object in the image.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments.

For example, in the above-described embodiment, the storage unit 11 stores one or more images captured by the image pickup device and the flow line information, and the acquisition unit 121 stores the image and the flow line information stored in the storage unit 11. Was decided to be acquired, but it is not limited to this. Even if a device different from the learning data generation device 1 stores one or more images and flow line information captured by the image pickup device, and the acquisition unit 121 acquires the images and flow line information stored in the device. good.

Further, in particular, the specific embodiment of the distribution / integration of the apparatus is not limited to those shown above, and all or a part thereof may be arbitrary according to various additions or the like, or according to the functional load. It can be functionally or physically distributed / integrated in units.

1 ... Learning data generation device, 11 ... Storage unit, 12 ... Control unit, 121 ... Acquisition unit, 122 ... Object identification unit, 123 ... Object position identification unit, 124 ... Object area specification part, 125 ... Object orientation specification part, 126 ... Extraction part

Claims (15)

An acquisition unit that acquires an image captured by an imaging device that captures a predetermined area from a position above a predetermined object existing in the predetermined area .
The predetermined object included in the first image acquired by the acquisition unit , and the orientation of the object moving in the real space in the real space is the same as the movement direction of the object in the real space. The object identification part that identifies
At the first position indicating the position corresponding to the coordinate system of the image of the predetermined object at the first time corresponding to the first image acquired by the acquisition unit, and at the second time before the first time. An object position specifying unit that specifies a second position indicating a position corresponding to the coordinate system of the predetermined object, and
The projection point on the plane of the intersection of the plane obtained by virtually extending the image plane which is the plane of the image captured by the image pickup device and the perpendicular line drawn from the optical center of the image pickup device to the predetermined area is described. The direction from the first position to the position of the reference point is specified as the reference direction as the reference point in the coordinate system, and the direction from the second position specified by the object position specifying unit to the first position is the predetermined direction. An object orientation specifying unit that identifies the moving direction of the object and specifies the orientation of the predetermined object in the first image based on the angle formed by the specified moving direction and the reference direction .
Specific device equipped with. An acquisition unit that acquires an image captured by an imaging device that captures a predetermined area from a position above a predetermined object existing in the predetermined area.
The predetermined object included in the first image acquired by the acquisition unit, and the orientation of the object in the real space while moving in the real space is the same as the movement direction of the object in the real space. The object identification part that identifies
At the first position indicating the position corresponding to the coordinate system of the image of the predetermined object at the first time corresponding to the first image acquired by the acquisition unit, and at the second time after the first time. An object position specifying unit that specifies a second position indicating a position corresponding to the coordinate system of the predetermined object, and
The projection point on the plane of the intersection of the plane obtained by virtually extending the image plane which is the plane of the image captured by the image pickup device and the perpendicular line drawn from the optical center of the image pickup device to the predetermined area is described. The direction from the first position to the position of the reference point is specified as the reference direction as the reference point in the coordinate system, and the direction from the first position specified by the object position specifying unit to the second position is the predetermined direction. An object orientation specifying unit that identifies the moving direction of the object and specifies the orientation of the predetermined object in the first image based on the angle formed by the specified moving direction and the reference direction.
Specific device equipped with. The acquisition unit acquires the flow line information indicating the position of the predetermined object at each time, and obtains the flow line information.
The object position specifying unit identifies the first position indicating the position of the predetermined object at the first time based on the first image acquired by the acquisition unit, and based on the flow line information, said the object position specifying unit. A second position indicating the position of the predetermined object at the second time, which is a time different from the first time, is specified.
The specific device according to claim 1 or 2 . The object position specifying unit identifies the first position indicating the position of the predetermined object at the first time based on the first image acquired by the acquisition unit, and the second image acquired by the acquisition unit. Based on the second image captured at the time, the second position indicating the position of the predetermined object at the second time is specified.
The specific device according to claim 1 or 2 . The object position specifying unit specifies the position of an object separated from the first position by a predetermined distance or more as the second position.
The specific device according to any one of claims 1 to 4 . Each of the plurality of angle ranges is associated with the orientation of one said object.
The object orientation specifying unit identifies the orientation of the predetermined object to the orientation of one object associated with the angle range including the angle.
The specific device according to any one of claims 1 to 5 . The object position specifying unit identifies the position of the predetermined object at each of three or more times related to the first image acquired by the acquisition unit.
The object orientation specifying unit specifies the moving direction of the object based on the position of the predetermined object at the specified time for each of the specified three or more times, and each of the specified three or more times. Specify the orientation of the predetermined object based on the moving direction of the object in
The specific device according to any one of claims 1 to 6 . Each of the plurality of angle ranges is associated with the orientation of one said object.
The object orientation specifying unit identifies and identifies the orientation of one object associated with the angle range that includes the angle corresponding to each of the three or more specified times. Specify the orientation of the predetermined object based on the ratio including the orientation of the object.
The specific device according to claim 7 . An object area specifying portion that specifies an area in the first image of the predetermined object, and
It further includes an extraction unit that extracts an image indicated by the area specified by the object area specifying unit and orientation information indicating the orientation specified by the object orientation specifying unit as learning data in machine learning.
The specific device according to any one of claims 1 to 8 . The extraction unit controls not to extract training data when the angle is outside a predetermined range.
The specific device according to claim 9 . The extraction unit controls so as not to extract training data when the distance between the first position and the reference point is less than a predetermined value.
The specific device according to claim 9 or 10 . Computer runs,
A step of acquiring an image captured by an imaging device that captures a predetermined area from a position above a predetermined object existing in the predetermined area .
The predetermined object included in the first image acquired in the acquisition step, and the orientation of the object in the real space while moving in the real space is the same as the movement direction of the object in the real space. And the steps to identify the object in
The first position indicating the position corresponding to the coordinate system of the image of the predetermined object at the first time corresponding to the first image acquired in the acquisition step, and the second position before the first time. A step of specifying a second position indicating a position corresponding to the coordinate system of the predetermined object at time, and a step of specifying the position.
The projection point on the plane of the intersection of the plane obtained by virtually extending the image plane which is the plane of the image captured by the image pickup device and the perpendicular line drawn from the optical center of the image pickup device to the predetermined area is described. A step of specifying a reference point in the coordinate system and a direction from the first position toward the position of the reference point as a reference direction.
The direction from the specified second position to the first position is specified as the movement direction of the predetermined object, and the predetermined direction in the first image is based on the angle formed by the specified movement direction and the reference direction . Steps to determine the orientation of the object,
Specific method to provide. Computer runs,
A step of acquiring an image captured by an imaging device that captures a predetermined area from a position above a predetermined object existing in the predetermined area.
The predetermined object included in the first image acquired in the acquisition step, and the orientation of the object in the real space while moving in the real space is the same as the movement direction of the object in the real space. Steps to identify the object of
The first position indicating the position corresponding to the coordinate system of the image of the predetermined object at the first time corresponding to the first image acquired in the acquisition step, and the second position after the first time. A step of specifying a second position indicating a position corresponding to the coordinate system of the predetermined object at time, and a step of specifying the position.
The projection point on the plane of the intersection of the plane obtained by virtually extending the image plane which is the plane of the image captured by the image pickup device and the perpendicular line drawn from the optical center of the image pickup device to the predetermined area is described. A step of specifying a reference point in the coordinate system and a direction from the first position toward the position of the reference point as a reference direction.
The direction from the specified first position to the second position is specified as the movement direction of the predetermined object, and the predetermined direction in the first image is based on the angle formed by the specified movement direction and the reference direction. Steps to determine the orientation of the object and
Specific method to provide. Computer,
An acquisition unit that acquires an image captured by an imaging device that captures a predetermined area from a position above a predetermined object existing in the predetermined area .
The predetermined object included in the first image acquired by the acquisition unit , and the orientation of the object moving in the real space in the real space is the same as the movement direction of the object in the real space. Object identification part, which identifies
At the first position indicating the position corresponding to the coordinate system of the image of the predetermined object at the first time corresponding to the first image acquired by the acquisition unit, and at the second time before the first time. An object position specifying unit that specifies a second position indicating a position corresponding to the coordinate system of the predetermined object, and
The projection point on the plane of the intersection of the plane obtained by virtually extending the image plane which is the plane of the image captured by the image pickup device and the perpendicular line drawn from the optical center of the image pickup device to the predetermined area is described. The direction from the first position to the position of the reference point is specified as the reference direction as the reference point in the coordinate system, and the direction from the second position specified by the object position specifying unit to the first position is the predetermined direction. An object orientation specifying unit that identifies the moving direction of the object and specifies the orientation of the predetermined object in the first image based on the angle formed by the specified moving direction and the reference direction .
A specific program that functions as. Computer,
An acquisition unit that acquires an image captured by an imaging device that captures a predetermined area from a position above a predetermined object existing in the predetermined area.
The predetermined object included in the first image acquired by the acquisition unit, and the orientation of the object in the real space while moving in the real space is the same as the movement direction of the object in the real space. Object identification part, which identifies
At the first position indicating the position corresponding to the coordinate system of the image of the predetermined object at the first time corresponding to the first image acquired by the acquisition unit, and at the second time after the first time. An object position specifying unit that specifies a second position indicating a position corresponding to the coordinate system of the predetermined object, and
The projection point on the plane of the intersection of the plane obtained by virtually extending the image plane which is the plane of the image captured by the image pickup device and the perpendicular line drawn from the optical center of the image pickup device to the predetermined area is described. The direction from the first position to the position of the reference point is specified as the reference direction as the reference point in the coordinate system, and the direction from the first position specified by the object position specifying unit to the second position is the predetermined direction. An object orientation specifying unit that identifies the moving direction of the object and specifies the orientation of the predetermined object in the first image based on the angle formed by the specified moving direction and the reference direction.
A specific program that functions as.

Images