JP7073179B2 - Image extractor, image extraction method and image extraction program - Google Patents

Description

The present invention relates to an image extraction device, an image extraction method, and an image extraction program that extract an image showing an object used for machine learning from an image captured by the image pickup device.

Conventionally, in analyzing the flow line of a person who acts in a predetermined area such as in a store, an object indicating the person is recognized from an image captured by capturing the predetermined area. Object recognition is performed, for example, by a classifier that classifies whether or not the image indicates an object in response to an image input (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-130583

In order to improve the determination accuracy of an object in a predetermined area, it is necessary to train the classifier in advance using many images showing the object. Therefore, it is required to efficiently collect images showing objects in a predetermined area. Further, it is preferable that the example image showing the correct example of the object contains only a single object.

Therefore, the present invention has been made in view of these points, and an object of the present invention is to provide an image extraction device, an image extraction method, and an image extraction program capable of efficiently collecting images suitable for machine learning. ..

The image extraction device according to the first aspect of the present invention identifies an object specifying unit that specifies one or more objects having a predetermined attribute included in an image captured by an image pickup device that captures a predetermined area, and an object specifying unit in the image. Surrounded by a contour area specifying part that specifies a contour area indicating the contour of the object, a rectangular area specifying part that specifies a rectangular area corresponding to the specified object in the image, and a rectangular area among the rectangular areas. A non-overlapping rectangular area indicating a rectangular area having an overlap rate of a predetermined value or less with the contour area corresponding to another object different from the object is specified, and an image showing the specified non-overlapping rectangular area is obtained by a machine. It is provided with an extraction unit for extracting as a regular example image showing a regular example of the object in learning.

The rectangular area specifying portion may specify the rectangular area based on the contour area of the specified object.
The rectangular area specifying portion may specify a rectangular area from the specified contour area of the object as a rectangle that approximates a point cloud constituting the contour of the object.
The rectangular area specifying portion may specify a rectangular area having a relatively small area of the rectangle as the rectangular area among a plurality of rectangular candidates including a point cloud constituting the contour of the object.

When the image pickup device is an image pickup device that captures an omnidirectional image, the rectangular area identification portion is based on the positional relationship between one point corresponding to the contour area and the center of the image captured by the image pickup device. The rotation component of the rectangular region with respect to the reference axis may be calculated, and among the rectangular regions having the rotation component, the rectangular region having a relatively small area may be specified as the rectangular region corresponding to the object.

The object specifying unit identifies the object by acquiring position information indicating the position of each of the one or more objects in the predetermined area, and the contour area specifying unit is indicated by the position information in the predetermined area. When the three-dimensional model corresponding to the object is virtually arranged at the position, the contour region corresponding to the three-dimensional model included in the image captured by the image pickup apparatus may be specified.

The image extraction device associates each of the plurality of positions in the image captured by the image pickup device with information indicating the contour region when the three-dimensional model is virtually arranged at each of the plurality of positions. A storage unit for storing position-specific area information is provided, and the contour area specifying unit refers to the position-specific area information and obtains a contour area associated with a position indicated by the position information acquired by the object identification unit. It may be specified as the contour area of the three-dimensional model.

The image extraction device includes a storage unit that stores point group information indicating a point group constituting the three-dimensional model and parameters related to calibration of the image pickup device, and the contour region specifying unit is stored from the storage unit. When the point group information and the parameter are acquired and the three-dimensional model is virtually arranged, at least a part of the point group indicated by the point group information is imaged by the image pickup device based on the parameter. The polygonal region that approximates the point group of the projection points may be specified as the contour region of the three-dimensional model by calculating the projection points on the image.

The object specifying unit identifies a first object included in the first image captured by the imaging device, and also identifies one or more second objects included in the first image, and the extracting unit identifies the first object. When the overlap ratio between the rectangular area corresponding to one object and the contour area corresponding to each of the one or more second objects is equal to or less than the predetermined value, the rectangular area corresponding to the first object is used. It may be specified as a non-overlapping rectangular area.

The object specifying unit identifies the first object, which is the object included in the first image captured by the imaging device, and is the object included in the second image captured after the first image. The second object is specified, and when the first image and the second image are overlapped with each other, the extraction unit sets the second object in the contour area or the rectangular area corresponding to the first object. The non-overlapping rectangular area whose overlap rate with the corresponding contour area or the rectangular area is equal to or less than the predetermined value may be specified.

The rectangular area specifying portion is a rectangular area that does not correspond to the object in the image, and is a rectangular area in which the overlap rate with the contour area of all the specified objects is equal to or less than a predetermined value. The extraction unit may extract an image showing the second rectangular region as a negative example image showing a negative example of the object in the machine learning.

The extraction unit inputs an image showing the specified non-overlapping rectangular region to a determination device for determining whether or not the input image indicates the object, and the determination device causes the image to display the object. If it is determined that the image is shown, the image may be extracted as the example image.

The extraction unit calculates the distance between the specified position corresponding to the non-overlapping rectangular region and the position corresponding to the imaging device, and based on the distance, obtains an image showing the non-overlapping rectangular region. It may be determined whether or not to extract as a regular image.

The image pickup device is an image pickup device that captures an omnidirectional image, and the extraction unit calculates the distance between the position corresponding to the specified non-overlapping rectangular region and the center of the image captured by the image pickup device. When the distance is larger than the first distance, an image showing the non-overlapping rectangular region is extracted as the example image, and when the distance is equal to or less than the first distance, the image showing the non-overlapping rectangular region is taken as the above. It may be controlled not to be extracted as a regular image.

The extraction unit calculates the distance between the position corresponding to the specified non-overlapping rectangular region and the center of the image captured by the imaging device, and when the distance is smaller than the second distance, the non-overlapping rectangular region. When the distance is equal to or greater than the second distance, the image showing the non-overlapping rectangular region may be controlled not to be extracted as the regular image.

The image extraction method according to the second aspect of the present invention includes a step of specifying one or more objects of a predetermined attribute included in an image captured by an image pickup device that captures a predetermined area, which is executed by a computer, and the image. , A step of specifying a contour area indicating the contour of the specified object, a step of specifying a rectangular area corresponding to the specified object in the image, and a step of specifying the rectangular area, which is surrounded by the rectangular area. A non-overlapping rectangular area indicating a rectangular area whose overlap rate with the contour area corresponding to another object different from the object is equal to or less than a predetermined value is specified, and an image showing the specified non-overlapping rectangular area is obtained in machine learning. It comprises a step of extracting as a canonical image showing a canonical example of an object.

In the image extraction program according to the third aspect of the present invention, the computer is an object identification unit that identifies one or more objects having a predetermined attribute included in an image captured by an image pickup device that captures a predetermined area, in the image. A contour area specifying part that specifies a contour area indicating the contour of the specified object, a rectangular area specifying part that specifies a rectangular area corresponding to the specified object in the image, and a rectangular area among the rectangular areas. A non-overlapping rectangular area indicating a rectangular area having an overlap rate of a predetermined value or less with the contour area corresponding to another object different from the object surrounded by is specified, and an image showing the specified non-overlapping rectangular area is displayed. , It functions as an extraction unit for extracting as a regular example image showing a regular example of the object in machine learning.

According to the present invention, there is an effect that images suitable for machine learning can be efficiently collected.

It is a figure which shows the outline of the image extraction apparatus which concerns on 1st Embodiment. It is a figure which shows the structure of the image extraction apparatus which concerns on 1st Embodiment. It is a flowchart which shows the flow of processing in the image extraction apparatus which concerns on 1st Embodiment. It is a figure which shows the example which the image which the image pickup time is the oldest is displayed on the display part. It is a figure which shows the contour area of the 3D model arranged at the position of the object specified from the image shown in FIG. It is a figure which shows the rectangular area specified for each of the contour areas of the plurality of 3D models shown in FIG. It is a figure which shows the example which extracts the image corresponding to a non-overlapping rectangular region as a regular image from the image which the image pickup apparatus has taken. It is a figure which shows the example which specified the 2nd rectangular area with respect to the example shown in FIG.

<First Embodiment>
[Overview of image extraction device 1]
FIG. 1 is a diagram showing an outline of the image extraction device 1 according to the first embodiment. The image extraction device 1 is a computer that extracts an image showing an object used for machine learning from an image captured by the image pickup device.

The image extraction device 1 is one or more moving trajectories of a moving image captured by one imaging device that captures a predetermined area and an object of a predetermined attribute moving in the predetermined area generated based on the moving image. The flow line information is acquired ((1) in FIG. 1). Here, the predetermined area is, for example, a space in a store, and the predetermined attribute is, for example, an attribute indicating a target for detecting an object. An object with a predetermined attribute is, for example, a clerk who acts in a predetermined area.

The image extraction device 1 corresponds to the position of one or more objects at a predetermined time based on one or more flow line information, and virtually creates a three-dimensional model corresponding to the object in a three-dimensional space indicating a predetermined area. Arrange ((2) in FIG. 1). The image extraction device 1 specifies the contour area indicating the contour of the three-dimensional model when the three-dimensional model is projected onto the image P captured by the image pickup device as the contour area of the object, and the three-dimensional model is used. The surrounding rectangular area is specified as the rectangular area of the object ((3) in FIG. 1).

The image extraction device 1 specifies, among the specified rectangular areas, a rectangular area that does not overlap with the contour area corresponding to another object different from the object surrounded by the rectangular area, as a non-overlapping rectangular area. For example, in the example shown in FIG. 1, the rectangular area corresponding to the object O1 does not overlap with the contours of the other objects O2 and O3, and is a non-overlapping rectangular area. Further, since the rectangular area corresponding to the object O2 overlaps with the contours of the other objects O3, it is not a non-overlapping rectangular area.

The image extraction device 1 is a machine learning device for determining whether or not an input image is an object for an image corresponding to a non-overlapping rectangular region in an image at a predetermined time among a plurality of images constituting a moving image. It is extracted as a regular image showing a regular example of the object in ((4) of FIG. 1). By doing so, the image extraction device 1 can efficiently collect images suitable for machine learning.

[Configuration of image extraction device 1]
Subsequently, the configuration of the image extraction device 1 will be described. FIG. 2 is a diagram showing the configuration of the image extraction device 1 according to the first embodiment. As shown in FIG. 2, the image extraction device 1 includes an input unit 11, a display unit 12, a storage unit 13, and a control unit 14. The display unit 12 is required when the object specifying unit 142 accepts the user's designated operation on the image, but is not required when it is not accepted.

The input unit 11 is, for example, a mouse or a keyboard. The input unit 11 receives an operation input from the user of the image extraction device 1.
The display unit 12 is composed of, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like. The display unit 12 displays various information according to the control of the control unit 14.

The storage unit 13 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The storage unit 13 stores a program executed by the control unit 14. For example, the storage unit 13 stores an image extraction program that causes the image extraction device 1 to function as an acquisition unit 141, an object identification unit 142, a contour area identification unit 143, a rectangular area identification unit 144, and an extraction unit 145.

The control unit 14 is, for example, a CPU (Central Processing Unit). By executing the image extraction program stored in the storage unit 13, the control unit 14 functions as an acquisition unit 141, an object identification unit 142, a contour area identification unit 143, a rectangular area identification unit 144, and an extraction unit 145.

Hereinafter, the details of the acquisition unit 141, the object identification unit 142, the contour area identification unit 143, the rectangular area identification unit 144, and the extraction unit 145 will be described with reference to the processing flow in the image extraction device 1. FIG. 3 is a flowchart showing a flow of processing according to the image extraction device 1 according to the first embodiment.

First, the acquisition unit 141 acquires a moving image captured by an imaging device that images a predetermined area and one or more flow line information indicating a movement trajectory of an object generated based on the moving image (S1).

The moving image is composed of a plurality of images captured by the image pickup apparatus. Each of the plurality of images is associated with an image capture time. The flow line information is information in which the time is associated with the position information indicating the position of the object in the predetermined area at the time. It is assumed that the time included in the flow line information corresponds to any of the plurality of imaging times associated with the plurality of images included in the moving image. The acquisition unit 141 in the first embodiment does not necessarily have to acquire information associated with a plurality of times, and even if it acquires an image at a specific time and information indicating the position of an object at that time. good.

Subsequently, the object specifying unit 142 identifies an unselected image from the plurality of images included in the moving image. The object specifying unit 142 selects the image having the oldest imaging time among the unselected images (S2).

Subsequently, the object specifying unit 142 determines whether or not the image is selected for the first time in this flowchart (S3). When the object specifying unit 142 determines that the image is selected for the first time, the process is transferred to S4, and when it is determined that the image is not selected for the first time, the process is transferred to S6.

Subsequently, the object specifying unit 142 identifies an object having a predetermined attribute included in the image captured by the image pickup device that captures the predetermined area (S4, S6). For example, the object identification unit 142 identifies a first object included in a first image captured by an image pickup apparatus that captures a predetermined area, and also identifies one or more second objects included in the first image.

Specifically, the object specifying unit 142 identifies one or more objects from the selected image when the determination in S3 is YES, that is, when an image is selected for the first time (S4).

More specifically, the object specifying unit 142 identifies the image selected in S2, that is, the image having the oldest imaging time among the plurality of images included in the moving image captured by the imaging device. The object specifying unit 142 identifies one or more flow line information including the imaging time corresponding to the image. Then, the object specifying unit 142 identifies an object corresponding to each of the specified one or more flow line information.

The object specifying unit 142 may accept the user to select the flow line information for specifying the object. Then, the object specifying unit 142 may specify the object corresponding to the selected flow line information.

Further, the object specifying unit 142 causes the display unit 12 to display the image selected in S2, that is, the image having the earliest imaging time among the plurality of images included in the moving image captured by the imaging device, and the user from the image. The object may be specified by the operation of. In this case, the object identification unit 142 inputs information indicating the position of the object by receiving a setting operation of a rectangular area surrounding each of the one or more objects included in the image displayed on the display unit 12 from the user. Accept. Here, the rectangular area is, for example, an upright rectangular area that is a rectangle whose two sides are parallel to a reference axis predetermined for an image.

FIG. 4 is a diagram showing an example in which the image P having the oldest imaging time is displayed on the display unit 12. FIG. 4A is a diagram showing the oldest image P, and FIG. 4B is a diagram showing an example after the user sets a rectangular area surrounding the object from the image P. It can be confirmed that the objects O1 to O3 are displayed in FIG. 4A. Further, as shown in FIG. 4B, it can be confirmed that the rectangular areas R1 to R3 are set for the objects O1 to O3 by the setting operation of the user.

The object specifying unit 142 has decided to specify an object from the user for the image P having the oldest imaging time, but the present invention is not limited to this. For example, when the object identification unit 142 has the flow line information whose start time is the imaging time corresponding to the selected image, the object identification unit 142 identifies the object corresponding to the flow line information or corresponds to the start time. An image may be displayed on the display unit 12 to accept an object designation from the user.

Here, in the example shown in FIG. 4, it is assumed that the object specifying unit 142 specifies the flow line information corresponding to each of the objects O1 to O3.

Subsequently, the object identification unit 142 obtains position information indicating the position of each of the one or more objects in a predetermined area based on the flow line information corresponding to each of the one or more objects, thereby causing the one or more objects. Specify the position (S5). The object identification unit 142 refers to the flow line information specified in S4, and acquires the position information associated with the same time as the imaging time of the selected image, so that the position of each of the one or more objects is obtained. To identify.

When the determination in S3 is NO, that is, when the image is not selected for the first time, the object identification unit 142 of the object identification unit 142 of one or more objects in a predetermined area based on the specified flow line information, as in S5. One or more objects are specified by acquiring the position information indicating each position (S6).

Subsequently, the contour area specifying unit 143 specifies a contour area indicating the contour of the specified object (S7). Specifically, the contour area specifying unit 143 is imaged by the image pickup device when a three-dimensional model corresponding to the object is virtually arranged at a position on a predetermined area indicated by the position information specified in S5 or S6. The contour area corresponding to the three-dimensional model included in the created image is calculated. Then, the contour area specifying unit 143 specifies the calculated contour area as the contour area of the object.

More specifically, in the storage unit 13, each of the position information indicating a plurality of positions in the image captured by the image pickup apparatus and the contour area when the three-dimensional model is virtually arranged at each of the plurality of positions. The position-specific area information associated with the information indicating the above is stored in advance. Then, the contour area specifying unit 143 refers to the position-specific area information stored in the storage unit 13, and virtually arranges the contour area associated with the position indicated by the position information specified by the object specifying unit 142. It may be specified as the contour area of the three-dimensional model. By doing so, the image extraction device 1 can identify the contour region of the object at high speed without analyzing the selected image.

The contour area specifying unit 143 does not specify the contour area of the three-dimensional model based on the position-specific area information, but specifies the contour area of the three-dimensional model based on the point cloud constituting the three-dimensional model. May be good. In this case, for example, the storage unit 13 stores point cloud information indicating a point cloud constituting the three-dimensional model, and calibrates the image pickup device for correcting distortion of the image captured by the image pickup device. Remember the parameters. Here, each of the plurality of points constituting the point cloud constituting the three-dimensional model is represented by the coordinates in the three-dimensional space. Further, the parameters include external parameters determined by the installation position and installation direction of the image pickup device, and internal parameters related to image distortion of the image pickup device.

Then, when the contour area specifying unit 143 acquires the point cloud information and the parameter from the storage unit 13 and virtually arranges the three-dimensional model at a position on the specified predetermined area, the point cloud information For at least a part of the point cloud indicated by, the projection points on the image captured by the image pickup apparatus are calculated based on the parameters. Then, the contour region specifying unit 143 specifies a polygonal region that approximates the contour of the region including the point cloud of the projection point as the contour region of the three-dimensional model. By doing so, the image extraction device 1 can accurately identify the contour region of the three-dimensional model based on the calculation of the projection points on the image of the plurality of point clouds.

FIG. 5 is a diagram showing a contour region of a three-dimensional model arranged at positions of objects O1 to O3 identified from the image P shown in FIG. The contour region shown in FIG. 5 is not actually drawn on the image, but is calculated by the contour region specifying unit 143. In the example shown in FIG. 5, it can be confirmed that the contour regions OL1 to OL3 are calculated corresponding to each of the objects O1 to O3.

Subsequently, the rectangular area specifying unit 144 identifies the rectangular area corresponding to the object based on the contour of the object specified in S7 (S8). Specifically, the rectangular area specifying unit 144 specifies a rectangular area from the contour area of the specified object as a rectangle that approximates a point cloud constituting the contour of the object.

More specifically, the rectangular area specifying unit 144 specifies a rectangular area having a relatively small area of the rectangle as a rectangular area among a plurality of rectangular candidates including a point cloud constituting the contour of the object. For example, when specifying an upright rectangular area as a rectangular area, a plurality of upright rectangular candidates having different sizes can be considered as an upright rectangle including a point cloud. The rectangular area specifying unit 144 specifies an upright rectangle having the smallest rectangular area among a plurality of candidates for upright rectangles as a rectangular area corresponding to an object. It is not limited to the upright rectangle, and a rotating rectangle that minimizes the area of the rectangle may be specified.

FIG. 6 is a diagram showing a rectangular region specified for each of the contour regions of the plurality of three-dimensional models shown in FIG. The rectangular area shown in FIG. 6 is not actually drawn on the image, but is calculated by the rectangular area specifying unit 144. In the example shown in FIG. 6, it can be confirmed that the rectangular regions UR1 to UR3 are calculated for the contour regions OL1 to OL3.

The rectangular area specifying unit 144 specifies an upright rectangle as a rectangular area corresponding to an object, but the present invention is not limited to this. Since the person, which is an object projected on the omnidirectional image, is projected so as to extend from the center of the image toward the position of the object, the rectangular area specifying portion 144 has two sides parallel to the direction. A rectangular area may be specified.

For example, the rectangular area identification unit 144 is a rectangular area with respect to a reference axis predetermined for the image based on the positional relationship between one point corresponding to the contour area and the center of the omnidirectional image captured by the image pickup device. The rotation component of may be calculated. Then, the rectangular area specifying unit 144 may specify a rectangular area having a relatively small area among the rotating rectangular areas having a rotation component calculated with respect to the reference axis as a rectangular area corresponding to the object. Here, the center of the omnidirectional image is assumed to be, for example, the center point of the image, but the present invention is not limited to this, and may be a point near the center of the image. Further, one point corresponding to the contour region is assumed to be, for example, the center point of the contour region, but the present invention is not limited to this, and may be a point near the center of the contour region. By doing so, the image extraction device 1 can specify the rectangular area so that the foot position of the object (person) corresponds to one side of the rectangular area in the omnidirectional image.

Subsequently, the extraction unit 145 identifies a non-overlapping rectangular area indicating a rectangular area that does not overlap with the other objects among the rectangular areas specified for each of the one or more objects. For example, the extraction unit 145 has a rectangular area corresponding to the first object when the overlap ratio between the rectangular area corresponding to the first object and the contour area corresponding to each of the one or more second objects is equal to or less than a predetermined value. Is specified in the non-overlapping rectangular area. Then, the extraction unit 145 extracts an image showing the specified non-overlapping rectangular region as a regular image showing a regular example of the object in machine learning (S9 to S11).

An example of the process until the extraction unit 145 extracts the regular image will be described below. First, the extraction unit 145 calculates the overlap rate with the contour area corresponding to another object different from the object surrounded by the rectangular area among the rectangular areas specified for each of the one or more objects. (S9). Specifically, the extraction unit 145 calculates the overlap ratio between the rectangular area specified for the object and the contour area corresponding to each of the other objects different from the object.

In the example shown in FIG. 6, the extraction unit 145 calculates the overlap rate of the rectangular region UR1 corresponding to the object O1 with the contour region OL2 corresponding to the object O2, and overlaps with the contour region OL3 corresponding to the object O3. Calculate the rate. Similarly, the extraction unit 145 calculates the overlap rate with the contour area OL1 and the overlap rate with the contour area OL3 for the rectangular area UR2 corresponding to the object O2. Further, the extraction unit 145 calculates the overlap rate with the contour area OL1 and the overlap rate with the contour area OL2 for the rectangular area UR3 corresponding to the object O3.

Subsequently, the extraction unit 145 identifies a non-overlapping rectangular area indicating a rectangular area in which the calculated overlap rate is equal to or less than a predetermined value (S10). For example, in the example shown in FIG. 6, when the predetermined value is 0, the extraction unit 145 specifies the rectangular region UR3 that does not overlap with any contour region as a non-overlapping rectangular region.

Subsequently, the extraction unit 145 extracts an image showing the specified non-overlapping rectangular region as a regular image showing a positive example of the object in the machine learning of the determination device for determining the object (S11). Specifically, the extraction unit 145 extracts an image corresponding to the non-overlapping rectangular region specified in S10 from the image selected in S2 as a regular image, and uses the extracted regular image as a label indicating a regular example. It is associated and stored in the storage unit 13.

FIG. 7 is a diagram showing an example of extracting an image corresponding to a non-overlapping rectangular region as a regular image from an image captured by the image pickup apparatus. The image P shown in FIG. 7 is the same image as the image P shown in FIG. 4 (a). As shown in FIG. 7, it can be confirmed that the image included in the rectangular region UR3 as the non-overlapping rectangular region is extracted from the image P as the regular image PE.

The extraction unit 145 calculates the distance between the position corresponding to the specified non-overlapping rectangular region and the position corresponding to the imaging device, and based on the distance, an image showing the non-overlapping rectangular region is a positive example. It may be determined whether or not to extract as an image.

Specifically, the extraction unit 145 may calculate the distance between the position corresponding to the non-overlapping rectangular region and the center of the image captured by the image pickup apparatus. In this case, the position corresponding to the non-overlapping rectangular area is the center position of the non-overlapping rectangular area. Then, when the calculated distance is larger than the predetermined first distance, the extraction unit 145 extracts an image showing the non-overlapping rectangular region as a regular image, and when the distance is equal to or less than the first distance, the extraction unit 145 extracts the image. By controlling so that the image showing the non-overlapping rectangular region is not extracted as the regular image, the non-overlapping rectangular region near the center of the image may not be extracted as the regular image.

When the image captured by the image pickup device is an omnidirectional image, the image showing the object contained in the non-overlapping rectangular area near the center of the image is the image obtained by capturing the object from a position close to directly above, and the identification accuracy of the object is high. It gets lower. The image extraction device 1 can prevent the accuracy of the classifier from being lowered by preventing the image obtained by capturing the object from a position close to the top as a regular image.

Further, when the distance between the position corresponding to the non-overlapping rectangular region and the center of the image captured by the imaging device is smaller than the predetermined second distance, the extraction unit 145 displays an image indicating the non-overlapping rectangular region. A non-overlapping rectangle that is extracted as a regular image and is too far from the center of the image by controlling so that the image showing the non-overlapping rectangular area is not extracted as the regular image when the distance is the second distance or more. The area may not be extracted as a regular image.

An image showing an object contained in a non-overlapping rectangular area far from the center of the image has a low object identification accuracy because the object is projected small. The image extraction device 1 can suppress a decrease in the accuracy of the classifier by preventing the image of an object that is too far from the center of the image from being extracted as a regular image.

Further, the extraction unit 145 inputs an image showing a non-overlapping rectangular area into a determination device for determining whether or not the input image indicates the object, and the determination device displays an image showing the non-overlapping rectangular area. If it is determined that the object is indicated, the image may be extracted as a regular image. In this case, the determination device may be stored in the storage unit 13 or may be stored in an external device accessible to the image extraction device 1. By extracting the regular image based on the judgment result of the judgment device, the image that is not an object can be excluded from the regular image.

Subsequently, the object specifying unit 142 determines whether or not all the images included in the moving image have been selected (S12). When the object specifying unit 142 determines that all the images have been selected, it ends the process in this flowchart, and when it determines that all the images have not been selected, it shifts the process to S2.

In this flowchart, the rectangular area specifying unit 144 specifies a rectangular area corresponding to the object, and the extracting unit 145 does not overlap as a rectangular area of the rectangular area that does not overlap with the contour area of another object. The rectangular area was extracted as a regular image, but it is not limited to this.

The rectangular area specifying unit 144 is a rectangular area that does not correspond to an object in the image captured by the imaging unit, and the overlap ratio with all of the contour areas of one or more specified objects is a predetermined value or less. The area may be specified. Here, the rectangular area specifying unit 144 may specify a plurality of second rectangular areas for one image. Further, the size of the second rectangular region may be the size of the smallest rectangular region surrounding the contour region of the three-dimensional model when the three-dimensional model is arranged at the position of the second rectangular region.

FIG. 8 is a diagram showing an example in which a second rectangular region is specified with respect to the example shown in FIG. In the example shown in FIG. 8, it can be confirmed that the second rectangular region UR4 is specified in addition to the rectangular regions UR1 to UR3 shown in FIG.

Then, the extraction unit 145 may extract an image showing the second rectangular region as a negative example image showing a negative example of the object in machine learning. By doing so, the image extraction device 1 can automatically collect negative example images as well.

[Effect in the first embodiment]
As described above, the image extraction device 1 according to the first embodiment identifies one or more objects having a predetermined attribute included in the image captured by the image pickup device that captures a predetermined area, and identifies the objects in the image. Specify a contour area that shows the outline of an object and a rectangular area that corresponds to the specified object. The image extraction device 1 is a non-overlapping rectangular area indicating a rectangular area having an overlap rate of a predetermined value or less with a contour area corresponding to another object different from the object surrounded by the specified rectangular area. Is specified, and an image showing the specified non-overlapping rectangular area is extracted as a regular example image showing a regular example of an object in machine learning. By doing so, the image extraction device 1 can efficiently collect images suitable for machine learning.

<Second Embodiment>
[Extract an example image based on multiple images with different imaging times]
Subsequently, the second embodiment will be described. The image extraction device 1 according to the first embodiment extracts an equivalent image as a regular image when the object corresponding to the flow line information has not moved for a long time due to sitting or the like. Therefore, the image extraction device 1 according to the second embodiment is different from the first embodiment in that an equivalent image is not extracted as a regular image based on a plurality of images having different imaging times. The image extraction device 1 according to the second embodiment will be described below. The same parts as those of the first embodiment will be omitted as appropriate.

In the second embodiment, the object identification unit 142 identifies the first object, which is an object included in the first image captured by the imaging device, and the object included in the second image captured after the first image. Identify the second object that is. Here, the second image may be a plurality of images captured after the first image. Further, the first object and the second object may be objects specified from the same flow line information.

The contour area specifying unit 143 specifies a contour area indicating the contour of the first object specified in the first image, and the rectangular area specifying unit 144 specifies a rectangular area corresponding to the first object.

Further, the contour area specifying unit 143 specifies a contour area indicating the contour of the second object specified in the second image, and the rectangular area specifying unit 144 specifies a rectangular area corresponding to the second object.

When the first image and one or more second images are overlapped with each other, the extraction unit 145 has a contour area corresponding to one or more second objects among the contour areas or rectangular areas corresponding to the first object. Alternatively, a rectangular area whose overlap rate with the rectangular area is equal to or less than a predetermined value is specified as a non-overlapping rectangular area. Then, the extraction unit 145 extracts the image showing the non-overlapping rectangular region in the first image as a regular image showing a regular example of the object in machine learning.

[Effect in the second embodiment]
As described above, the image extraction device 1 according to the second embodiment is included in the first image when the first image and the second image captured after the first image are superimposed. Among the contour areas or rectangular areas corresponding to the objects, the non-overlapping rectangular areas whose overlap rate with the contour areas or rectangular areas corresponding to the second object included in the second image is equal to or less than a predetermined value is specified, and the non-overlapping rectangular areas are specified. Is extracted as a regular image showing a regular example of an object in machine learning.

By doing so, the image extraction device 1 suppresses the extraction of the image corresponding to the object as a regular image from the images at a plurality of imaging times corresponding to the objects that do not move over time. can do.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. For example, in the above-described embodiment, the image captured by the image pickup apparatus is an omnidirectional image, but the present invention is not limited to this, and an image other than the omnidirectional image may be used.

Further, in the above-described embodiment, when the image extraction device 1 does not select an image for the first time, the position of the object is specified based on the flow line information, but the present invention is not limited to this. In all the images selected by the image extraction device 1, the position of the object may be specified by the user's specification.

Further, in the above-described embodiment, the description is made assuming that the number of image pickup devices is one, but the present invention is not limited to this, and a plurality of image pickup devices may be provided in a predetermined area. In this case, the image extraction device 1 corresponds to the image shown by the second image pickup device based on one or more flow line information corresponding to one or more objects specified based on the image captured by the first image pickup device. Extract the correct example image of the object.

Specifically, the image extraction device 1 is the one in the image shown by the second image pickup device when one or more three-dimensional models are virtually arranged at positions included in the one or more flow line information. The contour area and the rectangular area corresponding to the above three-dimensional model are specified as the contour area and the rectangular area corresponding to one or more objects. Then, the image extraction device 1 identifies a non-overlapping rectangular area based on the contour area and the rectangular area corresponding to the specified one or more objects, and the image corresponding to the rectangular area in the image shown by the second image pickup device. Is extracted as a regular image.

Further, in particular, the specific embodiment of the distribution / integration of the apparatus is not limited to those shown above, and all or a part thereof may be arbitrary according to various additions or the like, or according to the functional load. It can be functionally or physically distributed / integrated in units.

1 ... Image extraction device, 11 ... Input unit, 12 ... Display unit, 13 ... Storage unit, 14 ... Control unit, 141 ... Acquisition unit, 142 ... Object identification unit , 143 ... Contour area specifying part, 144 ... Rectangular area specifying part, 145 ... Extracting part

Claims (17)

An object identification unit that identifies one or more objects with a predetermined attribute included in an image captured by an image pickup device that captures a predetermined area.
A contour area specifying portion that specifies a contour area indicating the contour of the specified object in the image, and a contour area specifying portion.
A rectangular area specifying portion that specifies a rectangular area corresponding to the specified object in the image, and a rectangular area specifying portion.
Among the rectangular areas, a non-overlapping rectangular area indicating a rectangular area whose overlap rate with the contour area corresponding to another object different from the object surrounded by the rectangular area is equal to or less than a predetermined value is specified and specified. An extraction unit that extracts an image showing the non-overlapping rectangular area as a positive example image showing a positive example of the object in machine learning.
An image extraction device equipped with. The rectangular area specifying portion identifies the rectangular area based on the contour area of the specified object.
The image extraction device according to claim 1. The rectangular area specifying portion specifies a rectangular area from the specified contour area of the object as a rectangle that approximates a point cloud constituting the contour of the object.
The image extraction device according to claim 2. The rectangular area specifying portion specifies a rectangular area having a relatively small area of the rectangular area as the rectangular area among a plurality of rectangular candidates including a point cloud constituting the contour of the object.
The image extraction device according to claim 3. When the image pickup device is an image pickup device that captures an omnidirectional image, the rectangular region specifying portion is the object from a point near a point indicating the center of the omnidirectional image or a point indicating the center of the omnidirectional image. Of a plurality of rectangular regions having two sides parallel to the direction toward the position of, a rectangular region having a relatively small area is specified as a rectangular region corresponding to the object.
The image extraction device according to any one of claims 2 to 4. The object identification unit identifies the object by acquiring position information indicating the position of each of the one or more objects in the predetermined area.
The contour area specifying portion corresponds to the three-dimensional model included in the image captured by the image pickup apparatus when the three-dimensional model corresponding to the object is virtually arranged at the position indicated by the position information in the predetermined area. Identify the contour area to be
The image extraction device according to any one of claims 1 to 5. Stores position-specific area information in which each of a plurality of positions in an image captured by the image pickup apparatus and information indicating the contour area when the three-dimensional model is virtually arranged at each of the plurality of positions are associated with each other. Equipped with a storage unit
The contour area specifying unit refers to the position-specific area information, and identifies the contour area associated with the position indicated by the position information acquired by the object specifying unit as the contour area of the three-dimensional model.
The image extraction device according to claim 6. A storage unit for storing point cloud information indicating a point cloud constituting the three-dimensional model and parameters related to calibration of the image pickup apparatus is provided.
The contour area specifying unit acquires the point cloud information and the parameters from the storage unit, and when the three-dimensional model is virtually arranged, the point cloud information indicates at least a part of the point cloud. Based on the parameter, the projection point on the image captured by the imaging device is calculated, and the polygonal region that approximates the point cloud of the projection point is specified as the contour region of the three-dimensional model.
The image extraction device according to claim 6. The object specifying unit identifies the first object included in the first image captured by the imaging device, and also identifies one or more second objects included in the first image.
When the overlap ratio between the rectangular area corresponding to the first object and the contour area corresponding to each of the one or more second objects is equal to or less than the predetermined value, the extraction unit applies to the first object. Identifying the corresponding rectangular area as the non-overlapping rectangular area,
The image extraction device according to any one of claims 1 to 8. The object specifying unit identifies the first object, which is the object included in the first image captured by the imaging device, and is the object included in the second image captured after the first image. Identify the second object,
When the first image and the second image are overlapped with each other, the extraction unit may use the contour region or the rectangular region corresponding to the first object, the contour region corresponding to the second object, or the rectangle region. Identifying the non-overlapping rectangular area whose overlap rate with the rectangular area is equal to or less than the predetermined value.
The image extraction device according to any one of claims 1 to 8. The rectangular area specifying portion is a rectangular area that does not correspond to the object in the image, and is a rectangular area in which the overlap rate with the contour area of all the specified objects is equal to or less than a predetermined value. Identify and
The extraction unit extracts an image showing the second rectangular region as a negative example image showing a negative example of the object in the machine learning.
The image extraction device according to any one of claims 1 to 10. An object identification unit that identifies one or more objects with a predetermined attribute included in an image captured by an image pickup device that captures a predetermined area.
A contour area specifying portion that specifies a contour area indicating the contour of the specified object in the image, and a contour area specifying portion.
A rectangular area specifying portion that specifies a rectangular area corresponding to the specified object in the image, and a rectangular area specifying portion.
Among the rectangular areas, a non-overlapping rectangular area indicating a rectangular area whose overlap rate with the contour area corresponding to another object different from the object surrounded by the rectangular area is equal to or less than a predetermined value is specified and specified. An image showing the non-overlapping rectangular area is input to a determination device for determining whether or not the input image indicates the object, and the determination device determines that the image indicates the object. And an extraction unit that extracts the image as a regular image showing a regular example of the object in machine learning.
An image extraction device equipped with. An object identification unit that identifies one or more objects with a predetermined attribute included in an image captured by an image pickup device that captures a predetermined area.
A contour area specifying portion that specifies a contour area indicating the contour of the specified object in the image, and a contour area specifying portion.
A rectangular area specifying portion that specifies a rectangular area corresponding to the specified object in the image, and a rectangular area specifying portion.
Among the rectangular areas, a non-overlapping rectangular area indicating a rectangular area having an overlap rate of a predetermined value or less with the contour area corresponding to another object different from the object surrounded by the rectangular area was specified and specified. The distance between the position on the image corresponding to the non-overlapping rectangular region and the position on the image corresponding to the imaging device is calculated, and the image showing the non-overlapping rectangular region is obtained based on the distance. When it is determined whether or not to extract as a regular example image showing a regular example of the object in machine learning and it is determined to extract the image showing the non-overlapping rectangular region as the regular example image, the image is used as the regular example image. The extraction unit to be extracted and
An image extraction device equipped with. The image pickup device is an image pickup device that captures an omnidirectional image.
The extraction unit calculates the distance between the position on the image corresponding to the specified non-overlapping rectangular region and the center of the image captured by the image pickup apparatus, and when the distance is larger than the first distance, the above-mentioned An image showing a non-overlapping rectangular region is extracted as the regular image, and when the distance is equal to or less than the first distance, an image showing the non-overlapping rectangular region is controlled not to be extracted as the regular image.
The image extraction device according to claim 13. When the calculated distance is smaller than the second distance, which is larger than the first distance, the extraction unit extracts an image showing the non-overlapping rectangular region as the example image, and the distance is the second distance or more. In the case of, the image showing the non-overlapping rectangular area is controlled not to be extracted as the positive example image.
The image extraction device according to claim 14 . Computer runs,
A step of identifying one or more objects of a predetermined attribute included in an image captured by an image pickup device that captures a predetermined area, and
A step of specifying a contour area indicating the contour of the specified object in the image, and a step of specifying the contour area.
A step of specifying a rectangular area corresponding to the specified object in the image,
Among the rectangular areas, a non-overlapping rectangular area indicating a rectangular area whose overlap rate with the contour area corresponding to another object different from the object surrounded by the rectangular area is equal to or less than a predetermined value is specified and specified. A step of extracting an image showing the non-overlapping rectangular area as a regular image showing a positive example of the object in machine learning, and a step of extracting the image.
An image extraction method comprising. Computer,
An object identification unit that identifies one or more objects with a predetermined attribute included in an image captured by an image pickup device that captures a predetermined area.
A contour area specifying portion that specifies a contour area indicating the contour of the specified object in the image,
A rectangular area specifying portion that specifies a rectangular area corresponding to the specified object in the image, and a rectangular area specifying portion, and
Among the rectangular areas, a non-overlapping rectangular area indicating a rectangular area whose overlap rate with the contour area corresponding to another object different from the object surrounded by the rectangular area is equal to or less than a predetermined value is specified and specified. An extraction unit that extracts an image showing the non-overlapping rectangular area as a positive example image showing a positive example of the object in machine learning.
An image extraction program that functions as.

Images