JP6930389B2 - Image collectors, programs, and methods - Google Patents

Description

The present invention relates to image acquisition devices, programs, and methods, and more particularly to image collection devices, programs, and methods for efficiently collecting training images used for object identification.

Estimate whether the object reflected in the image corresponds to the object prepared in advance for identifying the object, that is, the object included in the "training image", or does not correspond to any object included in the training image. The object identification technology for identifying the type of object reflected in an image is an important elemental technology for realizing information search and information management of an object using an image as an interface, and is used in a wide range of industrial fields. ..

For example, if the user can identify the product from the image of the product package of the food taken with a smartphone or the like and obtain the detailed product information such as allergy information from the database on the Internet, the description described in the product package Even a user whose native language is not the language can easily obtain detailed product information described in the user's native language from the database.

Therefore, an object identification technique (see, for example, Patent Document 1) in which a training image is learned in advance and an object is identified by using a local feature amount of the object, and a key point indicating a difference between objects even if they are similar objects ("" An object identification technique (see, for example, Non-Patent Document 1) for accurately identifying an object by using (called a discriminative key point) has been proposed.

Japanese Unexamined Patent Publication No. 2015-201123

Written by Noto Watanabe, Go Irie, Takayuki Kurozumi, Tetsuya Fuchibuchi, "Recognition of Specific Objects from Similar Objects Based on Selection of Discriminating Key Points", Journal of the Institute of Image Information and Television Engineers, Vol.71, No.2, pp. J93-J100, 2017

On the other hand, in order to accurately identify an object by the object identification techniques shown in Patent Document 1 and Non-Patent Document 1, it is necessary to prepare a training image close to the imaging conditions of the object to be identified in advance. For example, in order to respond to changes in the appearance of an object in a captured image due to changes in the shooting angle and shooting distance of the object, as shown in FIG. 11, a plurality of objects photographed in various directions and distances. Training images must be prepared in advance.

Further, in the training image of the object, for example, as shown in FIG. 12, information such as a product name and a general noun for uniquely identifying the type of the object included in the training image (hereinafter referred to as "label information") is provided in the training image. Although it is necessary to associate them in advance, if incorrect label information is associated with the training image, the identification accuracy of the object will be lowered.

Further, when an object different from the object represented by the label information associated with the training image is included in the training image together with the object represented by the label information, any object is the object represented by the label information as it is. Since it is not known whether it exists, the identification accuracy of the object will decrease. Therefore, in this case, for example, as shown in FIG. 13, prior work such as acquiring the position information of the object represented by the label information (indicated by the dotted line frame in the example of FIG. 13) from the training image is required. Sometimes.

As described above, since it takes a lot of time and effort to prepare the training image, the cost related to the introduction and maintenance of the object identification technique increases.

In order to solve these problems, methods for efficiently collecting training images have been proposed so far. For example, Non-Patent Document 2 proposes a method of collecting training images corresponding to the searched label information by searching the name of an object (corresponding to label information) on the Internet.

[Non-Patent Document 2]: Michel Rubinstein, Armand Joulin, Johannes Kopf, Ce Liu. "Unsupervised Joint Object Discovery and Segmentation in internet Images" in CVPR, 2013

However, in the method shown in Non-Patent Document 2, for example, an object that is handled only in a store and is not distributed on the Internet, an industrial part used in a special field, or an object that is difficult to be accumulated in a storage device on the Internet. It becomes difficult to acquire the image of the above, and there is a tendency that the amount of the training image is insufficient. Therefore, the identification accuracy of such an object may decrease.

Further, in Non-Patent Document 3, when it is estimated that the object to be estimated is the object represented by the training image as a result of executing the identification of the object using the training image of the specific object, the estimation target is described. A method of collecting training images has been proposed by newly adding an image containing an object to the training image.

[Non-Patent Document 3]: Daisuke Deguchi, Keisuke Michimitsu, Ichiro Ide, Hiroshi Murase, "Higher accuracy of marker detectors using automatic collection of marker images based on retrospective tracking", IEICE Transactions, Vol.J95-D, No.1, pp.76-84, 2012

However, the method shown in Non-Patent Document 3 is based on the premise that no error occurs in the identification of the object using the training image. In reality, an object may be erroneously identified. Therefore, if an object is erroneously identified in the method shown in Non-Patent Document 3, a training image of a different type of object is added to the training image of a specific object. Will be. After that, when the object identification is executed using the training image, it becomes easy to add the training image of the object of a type different from the specific object in a chain reaction. Therefore, when such a training image is used, the object identification accuracy Will decrease.

The present invention has been made to solve the above problems, and is capable of efficiently collecting more training images to which label information for identifying an object is correctly associated. It is intended to provide equipment, programs, and methods.

In order to achieve the above object, the image collecting device according to the first invention is a label of the object from a label specifying device that acquires label information indicating the type of the object given to the object by approaching the object. From each of the label specifying unit that acquires information and specifies the type of the object, and the captured image captured by at least one imaging device that captures the acquisition status of the label information of the object using the label identifying device. The reference frame image at the moment when the label information of the object used for specifying the type of the object is acquired by the label specifying unit is acquired, and the object region including the object is estimated from the acquired reference frame image. The object area estimation unit, the type of the object specified by the label identification unit, the reference frame image used to estimate the object area of the object by the object area estimation unit, and the object area estimation unit estimated. From each of the training image registration unit that associates the object regions of the objects in the reference frame image and registers them in the storage device as training images and the reference frame image of the captured image captured by the imaging device, the label specifying device. The object area estimation unit includes a label identification device estimation unit that estimates the device area including the above, and the object area estimation unit is based on the position of the device area of the label identification device estimated by the label identification device estimation unit. The object area including the object is estimated from.

Further, the image collecting device according to the first invention is an adjacent image taken from each of the captured images in at least one range before and after the reference frame image acquired by the object area estimation unit. The frame images are acquired in chronological order, the object included in the reference frame image is tracked from each of the acquired adjacent frame images, and the object is included from each of the adjacent frame images including the object. The training image registration unit further includes an object tracking unit that estimates an object area, and the training image registration unit includes the type of the object specified by the label specifying unit and the adjacent frame used by the object tracking unit to estimate the object area of the object. The image and the object area of the object in the adjacent frame image estimated by the object tracking unit are associated with each other and registered in the storage device as a training image.

Further, in the image collecting device according to the first invention, the label specifying unit acquires the label information of the object from the label specifying device that reads the label information of the object by using light or radio waves.

The program according to the second invention causes the computer to function as each part of the image collecting device according to any one of claims 1 to 3.

In the image collecting method according to the third invention, the label information of the object is acquired from a label specifying device that acquires the label information indicating the type of the object given to the object by approaching the object, and the object of the object is obtained. It was used to identify the type of the object from each of the step of specifying the type and the photographed image taken by at least one photographing device for photographing the acquisition status of the label information of the object using the label specifying device. A step of acquiring a reference frame image at the moment when the label information of the object is acquired and estimating an object region including the object from the acquired reference frame image, the specified type of the object, and the object of the object. The step of associating the reference frame image used for estimating the object region and the object region of the object in the reference frame image and registering them in the storage device as a training image, and the step of registering the captured image captured by the imaging device in the storage device. The device of the label specifying device estimated in the step of estimating the device area including the label specifying device from each of the reference frame images and estimating the object area including the object from the reference frame image. Based on the position of the region, the object region including the object is estimated from the reference frame image.

Further, in the image collecting method according to the third invention, adjacent frame images, which are images taken in at least one range before and after the reference frame image, are acquired from each of the captured images in chronological order. A step of tracking the object included in the reference frame image from each of the acquired adjacent frame images and estimating an object region including the object from each of the adjacent frame images including the object. A step of associating the specified type of the object, the adjacent frame image used for estimating the object area of the object, and the object area of the object in the adjacent frame image, and registering the object area as a training image in the storage device. And further prepare.

According to the image collecting device, the program, and the method of the present invention, it is possible to efficiently collect more training images to which the label information for identifying the object is correctly associated. ..

It is a figure which shows the configuration example of an image acquisition system. It is a figure which shows the structural example of the image collecting apparatus which concerns on 1st Embodiment. It is a flowchart which shows an example of the flow of the image acquisition routine which concerns on 1st Embodiment. It is a figure which shows an example of the process of estimating an object area from the position of a hand area. It is a figure which shows the structural example of the image collecting apparatus which concerns on 2nd Embodiment. It is a flowchart which shows an example of the flow of the image acquisition routine which concerns on 2nd Embodiment. It is a figure which shows an example of the process of collecting a training image from an adjacent frame image. It is a figure which shows the structural example of the image collecting apparatus which concerns on 3rd Embodiment. It is a flowchart which shows an example of the flow of the image acquisition routine which concerns on 3rd Embodiment. It is a figure which shows an example of the process of estimating an object area from a device area. It is a figure which shows an example of the training image. It is a figure which shows an example of the training image which associated the label information. It is a figure which shows an example of the training image which associated the label information and the position information of an object.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It should be noted that components and processes having the same function shall be given the same code throughout the drawings, and duplicate explanations shall be omitted.

<First Embodiment>
<Configuration of image collection system>
First, the configuration of the image acquisition system 1 according to the present invention will be described with reference to FIG.

The image collection system 1 includes an image collection device 2, a label identification device 4, and a photographing device 6, and collects training images of an object 8.

Label information indicating the type of the object is given to the object 8 in advance, and the user presses, for example, an operation button provided on the label specifying device 4, or approaches the object 8 to the label specifying device 4 to a predetermined distance. The label specifying device 4 acquires the label information given to the object 8.

Specifically, when a bar code indicating label information is given to the object 8, a bar code reader that reads the label information of the object 8 indicated by the bar code by using the reflected light of the irradiated light is the label specifying device 4. Used as. The barcode indicating the label information may be either a one-dimensional barcode or a two-dimensional barcode.

When an electronic tag that stores label information is attached to the object 8, RFID (Radio Frequency IDentification) that communicates with the electronic tag using radio waves and reads the label information of the object 8 stored in the electronic tag. The reader is used as the label identification device 4. The electronic tag may be called an RF tag, an IC tag, a non-contact tag, or a wireless tag, but here, it will be referred to as an "electronic tag".

The label identification device 4 is connected to the image collection device 2, and the label identification device 4 transmits the acquired label information to the image collection device 2.

The photographing device 6 is connected to the image collecting device 2, and photographs a situation in which the user acquires the label information of the object 8 using the label specifying device 4. There is no limitation on the number of photographing devices 6 used in the image acquisition system 1, and it is sufficient that at least one photographing device 6 is included. However, there is a situation in which a user acquires label information of an object 8 by using a label specifying device 4. It is preferable to use a plurality of photographing devices 6 so that the images can be photographed from a plurality of directions. In the example of FIG. 1, three photographing devices 6 are used, and a situation in which a user acquires label information of an object 8 by using a label specifying device 4 is photographed from different directions. The imaging device 6 may be installed so that the distance between the object 8 and each imaging device 6 is the same, but each imaging device 6 may be installed so that the distances are different from each other.

There are no restrictions on the type of the photographing device 6 as long as the images can be photographed in chronological order, and for example, a camera built in a smartphone or a surveillance camera such as a webcam can be used. Further, the image taken by the photographing device 6 may be a black-and-white image or a color image, but since the color image can be taken including the color information of the object 8 as compared with the black-and-white image, the color image can be taken. It is preferable to use a photographing device 6 capable of photographing an image. Further, the resolution of the photographing device 6 is not particularly limited as long as the object 8 can be confirmed and the resolution is such that the user can confirm the situation in which the label information of the object 8 is acquired by using the label specifying device 4. The higher the resolution, the clearer the captured image. Therefore, it is preferable to use the imaging device 6 having a high resolution.

<Structure of the image collecting device according to the first embodiment>
Next, the configuration of the image collecting device 2 will be described. The image collection device 2 according to the embodiment of the present invention is a CPU, a RAM, a ROM that stores an image collection program for executing an image collection routine described later, a ROM that stores various data, and a non-volatile memory that stores data such as thresholds. It can be configured with a computer that includes memory. The image collecting device 2 functionally includes an input unit 10 and a calculation unit 20 as shown in FIG.

The input unit 10 receives the label information acquired by the label specifying device 4, receives the photographed image photographed by each photographing device 6, and notifies the calculation unit 20 of the photographed image.

The calculation unit 20 includes a label identification unit 22, an object area estimation unit 24, a training image registration unit 26, and a training image storage unit 28.

When the label specifying unit 22 receives the label information acquired by the label specifying device 4, the label specifying unit 22 analyzes the received label information and identifies the type of the object 8. Then, the label specifying unit 22 notifies the training image registration unit 26 of the type of the specified object 8. On the other hand, the label specifying unit 22 notifies the object area estimation unit 24 of the processing start command.

When the object area estimation unit 24 receives the processing start command from the label specifying unit 22, the object area estimation unit 24 uses each of the captured images of the photographing device 6 to capture the situation in which the user acquires the label information of the object 8 using the label specifying device 4. The label specifying unit 22 acquires an image (hereinafter, referred to as “reference frame image 30A”) at the moment when the label information of the object 8 used for specifying the type of the object 8 is acquired. Then, the object area estimation unit 24 estimates the object area 32 including the object 8 whose type is specified by the label identification unit 22 from each of the acquired reference frame images 30A.

The object area estimation unit 24 notifies the training image registration unit 26 of the position information of each reference frame image 30A used for estimating the object area 32 and the object area 32 of the object 8 estimated from each reference frame image 30A. do.

When the training image registration unit 26 receives the position information of the reference frame image 30A and the object area 32 from the object area estimation unit 24 and receives the type of the object 8 from the label specifying unit 22, the object specified by the label specifying unit 22 The reference frame image 30A used for estimating the object area 32 by the object area estimation unit 24 and the object area 32 of the object 8 in the reference frame image 30A are associated with each of the reference frame images 30A. Then, the training image registration unit 26 stores each reference frame image 30A associated with the type of the object 8 and the object area 32 in the training image storage unit 28 as a training image, so that the training image storage unit 28 Register the training image in.

The training image storage unit 28 is an example of a storage unit that maintains the stored information even when the power supplied to the training image storage unit 28 is cut off, and for example, a semiconductor memory or a hard disk is used.

<Operation of the image collecting device according to the first embodiment>
Next, the operation of the image collecting device 2 according to the present embodiment will be described. When the input unit 10 receives the label information of the object 8, the image collecting device 2 executes the image collecting routine shown in FIG. The input unit 10 constantly receives the photographed images taken by each photographing device 6, and stores the photographed images in the RAM for each photographing device 6.

First, in step S10, the CPU acquires the label information from the label specifying device 4 received by the input unit 10 and stores the acquired label information in the RAM.

In step S20, the CPU analyzes the label information stored in the RAM in step S10 and identifies the type of the object 8 indicated by the label information. Specifically, the CPU specifies the type of the object 8 indicated by the label information by referring to the type table in which the label information and the type of the object 8 are associated with each other. The type table may be stored in advance in a non-volatile memory mounted on the image collecting device 2, for example, and may be referred to. However, the type table may be referred to by referring to a database for storing the type table connected to the Internet. The type of the object 8 indicated by the label information may be specified.

In step S30, the CPU acquires the captured image of each photographing device 6 received by the input unit 10 from the RAM.

Since the captured image is composed of a plurality of frame images in chronological order, in step S40, the CPU acquires the reference frame image 30A from the captured images for each imaging device 6 acquired in step S30. Specifically, the CPU extracts the image captured when the label information is acquired in step S10 as the reference frame image 30A from the captured image. When time information is added to each frame image included in the captured image, the frame image to which the time information corresponding to the time when the label information was acquired in step S10 is added with reference to the time information. May be acquired as the reference frame image 30A.

In step S50, the CPU estimates an object area 32 including the object 8 whose type is specified in step S20 for each reference frame image 30A acquired in step S40. At this time, a known method can be used as the method for estimating the object region 32.

For example, when the installation location and shooting direction of each shooting device 6 are fixed and the shooting range of the shooting device 6 does not change, only the background image in which the object 8 is not captured is shot in advance for each shooting device 6 to obtain an image. Even if the object area 32 is estimated from the reference frame image 30A by storing it in the non-volatile memory of the collection device 2 in advance and calculating the difference between the reference frame image 30A and the background image for each photographing device 6. good.

Further, for example, when the product is settled at the cash register of a supermarket or a convenience store, the label specifying device 4 is brought close to the object 8 while the user holds the object 8 (in this case, the product), and the price and product name of the object 8 are obtained. When acquiring the label information including the object 8, first, the hand area of the user who grabs the object 8 is extracted from the reference frame image 30A, and the object area 32 including the object 8 is estimated from the position of the extracted hand area. It may be (see, for example, Non-Patent Document 4).

[Non-Patent Document 4]: Xiaolong Zhu, Wei Liu, Xuhui Jia, Kwan-Yee K.Wong. "A Two-stage Detector for Hand Detection in Ego-centric Videos" in WACV, 2016

FIG. 4 is a diagram showing an example of a process of extracting a user's hand region 31 from the reference frame image 30A and estimating an object region 32 from the position of the extracted hand region 31.

The image collecting device 2 extracts the hand region 31 for each reference frame image 30A captured by each imaging device 6.

_{Here, let ^ c i} be the center coordinate of the hand region 31 extracted from the reference frame image 30A taken by the i (in this case, i = 1 to 3) th imaging device 6. Further, since the center coordinates of the object 8 (that is, the center coordinates of the object area 32) when the object 8 is held in the hand are _{located at positions deviated from the center coordinates ^ c i} of the hand area 31, various objects 8 are prepared in advance. using the reference frame image 30A including the hand holding the center coordinates ^ x _^'i and the center coordinate ^ c _i offset value experimentally obtained mean value of the shift amount of the hand region 31 of the object region 32 by storing an amount ^ b _i, the center coordinate ^ x _^'i of the object region 32 in each reference frame image 30A is expressed by equation (1).

(Number 1)
_{^{^ x 'i = ^ c i}} + ^ b i ··· (1)

Further, by using the reference frame image 30A including the hand holding the pre various objects 8, the average value w _i of the width of the object region 32 experimentally obtained, the average value of the height of the object region 32 h by storing as _i, as shown in FIG. 4, the range of the object region 32 in each reference frame image 30A has a width w _i around the coordinates ^ x _^'i, is represented by the height of the rectangle h _i NS.

Two hand regions 31 corresponding to the left and right hands are extracted from the reference frame image 30A. However, the user if determined conditions and grab an object 8 with one hand a predetermined, CPU is when estimating the center coordinates ^ x _^'i of the object region 32, any of the center coordinates of the hand region 31 ^ It is possible to determine whether c _{i should be used.}

Further, the shape of the object region 32 is not limited to a rectangle, and other shapes such as a circle, an ellipse, and a polygon may be used to represent the object region 32.

In step S60, the CPU determines the type of the object 8 included in the object area 32 specified in step S20 and the reference frame image 30A with respect to the reference frame image 30A used for estimating the object area 32 in step S50. position information of the object region 32, i.e., the center coordinates of the object region 32 ^ x _^'i, the width w _i, and a height h _i, associated to each reference frame image 30A. Then, the CPU stores each of the reference frame image 30A, that is, the training image, in which the type of the object 8 and the position information of the object area 32 are associated with each other, in, for example, the non-volatile memory mounted on the image collecting device 2.

There are no restrictions on the storage destination of the training image, and the training image may be stored in a database connected to the Internet, for example.

As a result, the image collection routine shown in FIG. 3 is completed.

As described above, according to the image collecting device 2 according to the present embodiment, the label information of the object 8 is obtained with respect to the reference frame image 30A at the moment when the user acquires the label information of the object 8 with the label specifying device 4. The type of the object 8 and the object area 32 estimated from the reference frame image 30A are associated with each other to form a training image.

As can be seen from the fact that the label identification device 4 that reads label information using light or radio waves, such as a barcode reader or RFID reader, is used as a payment means and an inspection means for an object 8, for example, it can be used in other methods. Compared to this, the reading error of label information is low. Therefore, when the type of the object 8 is specified by using the label information acquired from the label specifying device 4, it is compared with the case where the type of the object 8 is specified by performing image recognition on a photographed image of the object, for example. Therefore, the erroneous recognition rate for the type of the object 8 becomes low.

That is, in the image collecting device 2, the correct type of the object 8 can be associated with the training image as compared with the case where the type of the object 8 is specified by using image recognition.

Further, since the image collecting device 2 uses the reference frame image 30A, which is an image at the moment when the user acquires the label information of the object 8 with the label specifying device 4, as the training image, the object 8 of the type specified by using the label information. Is guaranteed to be the object 8 included in the reference frame image 30A.

Further, by photographing the situation in which the user acquires the label information of the object 8 using the label specifying device 4 with a plurality of photographing devices 6, the photographing is performed in various directions and distances by one task of acquiring the label information. Since a plurality of training images can be obtained, more training images can be efficiently collected.

<Second Embodiment>
The image collecting device 2 according to the first embodiment uses the reference frame image 30A, which is an image at the moment when the user acquires the label information of the object 8 with the label specifying device 4, as a training image. However, among the captured images captured by the photographing device 6, the image including the object 8 is not limited to the reference frame image 30A.

Therefore, in the second embodiment, among the adjacent frame images which are images taken in at least one range before and after the reference frame image 30A with reference to the reference frame image 30A, the adjacent frame image including the object 8 is included. The image collecting device 2A for collecting as a training image will be described.

The configuration of the image acquisition system 1 including the image acquisition device 2A is the same as that shown in FIG.

<Structure of the image collecting device according to the second embodiment>
FIG. 5 is a diagram showing a configuration example of the image collecting device 2A.

The configuration of the image collecting device 2A shown in FIG. 5 is different from the configuration of the image collecting device 2 shown in FIG. 2 in that the object tracking unit 25 is added and the training image registration unit 26 is replaced with the training image registration unit 26A. This is the point.

When the object tracking unit 25 receives the position information of the reference frame image 30A and the object area 32 from the object area estimation unit 24, the object tracking unit 25 sequentially acquires the frame images from the frame images adjacent to the reference frame image 30A in chronological order. An object for determining whether or not the same object 8 as the object 8 included in the object area 32 received from the object area estimation unit 24 is included in each frame image along the series, that is, each of the adjacent frame images. Perform tracking. The object tracking unit 25 estimates the object region 32 with respect to the adjacent frame image determined to include the object 8 by the object tracking.

Then, the object tracking unit 25 adds the object area of the object 8 estimated from the adjacent frame image to the adjacent frame image containing the same object 8 as the object 8 included in the object area 32 received from the object area estimation unit 24. The position information of 32 is added and notified to the training image registration unit 26A. Further, the object tracking unit 25 notifies the training image registration unit 26A of the position information of the reference frame image 30A and the object area 32 in the reference frame image 30A received from the object area estimation unit 24.

When the object tracking unit 25 receives a plurality of reference frame images 30A due to the existence of a plurality of photographing devices 6, the object tracking unit 25 receives adjacent frame images from the captured images captured by the respective photographing devices 6 corresponding to the respective reference frame images 30A. To track the object.

Similar to the training image registration unit 26, the training image registration unit 26A receives the position information of the reference frame image 30A and the object area 32 in the reference frame image 30A from the object tracking unit 25, and selects the type of the object 8 from the label specifying unit 22. Upon receipt, the type of the object 8 specified by the label specifying unit 22, the reference frame image 30A used for estimating the object area 32 by the object area estimation unit 24, and the object area 32 of the object 8 in the reference frame image 30A are used as a reference. Correspondence is made for each frame image 30A.

Further, when the training image registration unit 26A receives the position information of the adjacent frame image and the object region 32 in the adjacent frame image from the object tracking unit 25 and receives the type of the object 8 from the label specifying unit 22, the label specifying unit 22 receives the type of the object 8. The type of the identified object 8, the adjacent frame image determined by the object tracking unit 25 to include the object 8, and the object region 32 of the object 8 in the adjacent frame image determined to include the object 8 are adjacent frame images. Correspond to each. Then, the training image registration unit 26A stores each reference frame image 30A and each adjacent frame image associated with the type of the object 8 and the object area 32 in the training image storage unit 28 as training images.

<Operation of the image collecting device according to the second embodiment>
Next, the operation of the image collecting device 2A according to the present embodiment will be described. When the input unit 10 receives the label information of the object 8, the image collecting device 2A executes the image collecting routine shown in FIG. The input unit 10 constantly receives the photographed images taken by each photographing device 6, and stores the photographed images in the RAM for each photographing device 6. Therefore, the RAM stores the captured image before receiving the label information of the object 8 and the captured image after receiving the label information of the object 8.

The difference between the image collection routine shown in FIG. 6 and the image collection routine of the image collection device 2 shown in FIG. 3 is that steps S52 to S58 are newly added and step S60 is replaced with step S60A.

In step S50, when the object area 32 including the object 8 whose type is specified in step S20 is estimated with respect to the reference frame image 30A, step S52 is executed.

In step S52, the CPU selects any one of the reference frame images 30A from which the object region 32 is estimated, and refers to the captured images including the selected reference frame image 30A in chronological order. Acquire one adjacent frame image adjacent to the frame image 30A. At this time, the CPU may acquire the adjacent frame image along the time series in the past direction taken before the time when the reference frame image 30A was taken, or from the time when the reference frame image 30A was taken. Adjacent frame images may be acquired along the time series in the future direction taken later. Here, as an example, it will be described assuming that adjacent frame images are acquired along a time series in the past direction.

In step S54, the CPU determines whether or not the adjacent frame image acquired in step S52 includes the same object 8 as the object 8 included in the object area 32 of the reference frame image 30A estimated in step S50. I do.

For tracking an object in an adjacent frame image, for example, a known method as shown in Non-Patent Document 5 can be used.

[Non-Patent Document 5]: Michael Isard, Andrew Blake. "CONDENSATION-Conditional Density Propagation for Visual Tracking" International Journal of Computer Vision, Vol29, No.1, pp.5-28, 1998

In Non-Patent Document 5, the tracked object (in this case, the object 8) is expressed as a discrete probability density by a large number of hypothesis groups having likelihoods, and the movement fluctuation and observation of the object 8 are performed using the state transition model. Techniques for achieving robust tracking of noise have been disclosed. Therefore, when the likelihood of the existence of the object 8 in the adjacent frame image is equal to or higher than a predetermined threshold value, it is determined that the object 8 is included in the adjacent frame image, and the likelihood is a predetermined threshold value. If it is less than, it is determined that the object 8 is not included in the adjacent frame image.

When the object 8 is included in the adjacent frame image, the CPU estimates the object area 32 including the object 8 with respect to the adjacent frame image by using the same method as in step S50.

In step S56, the CPU determines whether or not the object region 32 is included in the adjacent frame image acquired in step S52. When the object region 32 is included in the adjacent frame image, the object 8 may also be included in the adjacent frame image along the time series in the past direction adjacent to the adjacent frame image, so the process proceeds to step S52. ..

Then, in step S52, one adjacent frame image along the time series in the past direction adjacent to the adjacent frame image acquired immediately before is acquired from the captured image including the adjacent frame image acquired immediately before. After that, in the determination process of step S56, steps S52 to S56 are repeatedly executed until it is determined that the adjacent frame image acquired in step S52 does not include the object region 32, so that the reference frame image 30A is in the past direction. Among the adjacent frame images in the range traced back to, the adjacent frame image including the object 8 and the object area 32 in the adjacent frame image can be acquired.

On the other hand, if the determination process in step S56 is a negative determination, that is, if it is determined that the adjacent frame image acquired in step S52 does not include the object region 32, the process proceeds to step S58.

In step S58, the CPU determines whether or not there is an unselected reference frame image 30A among the reference frame images 30A in which the object region 32 is estimated. If there is an unselected reference frame image 30A, the process proceeds to step S52.

Then, in step S52, one reference frame image is selected from the unselected reference frame images 30A, and adjacent frames adjacent to the reference frame image 30A are selected in chronological order from the captured images including the selected reference frame image 30A. Get one image. After that, by repeatedly executing steps S52 to S58 until it is determined that the unselected reference frame image 30A does not exist in the determination process of step S58, an object is obtained from each of the captured images captured by the plurality of imaging devices 6. It is possible to acquire the adjacent frame image including 8 and the object area 32 in the adjacent frame image.

In step S60A, the CPU determines the type of the object 8 included in the object area 32 specified in step S20 and the reference frame image 30A with respect to the reference frame image 30A used for estimating the object area 32 in step S50. The position information of the object area 32 is associated with each reference frame image 30A. Further, the CPU determines the type of the object 8 included in the object area 32 specified in step S20 and the position information of the object area 32 in each adjacent frame image for each of the adjacent frame images including the object area 32. Is associated with each adjacent frame image.

Then, the CPU stores each of the reference frame image 30A and the adjacent frame image associated with the type of the object 8 and the position information of the object area 32 as training images in, for example, a non-volatile memory mounted on the image collecting device 2A. do.

As a result, the image collection routine shown in FIG. 6 is completed.

In the image collection routine shown in FIG. 6, the training image is collected using the adjacent frame image in the range retroactive from the reference frame image 30A, or the adjacent frame image in the range from the reference frame image 30A toward the future direction. Although any example of collecting the training image using the above is shown, after collecting the training image using the adjacent frame image in the range retroactive from the reference frame image 30A, the reference frame image 30A is further shown. Training images may be collected using adjacent frame images in the range from to the future. Specifically, after the processing of step S58 of FIG. 6 is completed, the adjacent frame image is acquired in the direction opposite to the time-series direction in which the adjacent frame image was acquired so far, and the adjacent frame images are acquired in steps S52 to S58. The process may be executed again.

In this case, more training images can be efficiently obtained from one captured image as compared with the case of collecting training images from adjacent frame images along either one of the future direction and the past direction with reference to the reference frame image 30A. Can be collected well.

FIG. 7 is a diagram showing an example of collecting training images from adjacent frame images along both the future direction and the past direction with reference to the reference frame image 30A. In FIG. 7, the adjacent frame image along the time series in the past direction from the reference frame image 30A is represented by the “adjacent frame image 30B”, and the adjacent frame image along the time series in the future direction from the reference frame image 30A is represented by the “adjacent frame”. It is represented by "Image 30C". Further, the sign "-n (n = 1 to 3)" following the adjacent frame images 30B and 30C indicates the time series order from the reference frame image 30A, and the reference frame increases as the value of "n" increases. It shows that the time interval from the image 30A is large.

That is, the order of adjacent frame image 30B-3 → adjacent frame image 30B-2 → adjacent frame image 30B-1 → reference frame image 30A → adjacent frame image 30C-1 → adjacent frame image 30C-2 → adjacent frame image 30C-3. Each image has been taken.

In the example of FIG. 7, since the object region 32 does not exist in the adjacent frame image 30B-3 and the adjacent frame image 30C-3, the adjacent frame image 30B-2, the adjacent frame image 30B-1, the reference frame image 30A, and the adjacent frame image 30C-1 and the adjacent frame image 30C-2 will be collected as training images.

As described above, according to the image collecting device 2A according to the present embodiment, not only the reference frame image 30A but also the adjacent frame including the object 8 taken in at least one of the future direction and the past direction of the reference frame image 30A. Images can also be collected as training images. Therefore, as compared with the case where only the reference frame image 30A is collected as the training image, more training images can be efficiently collected from one captured image.

<Third Embodiment>
The image collecting device 2 according to the first embodiment extracts the user's hand region 31 from the reference frame image 30A, and estimates the object region 32 from the position of the extracted hand region 31. However, the object region 32 in the reference frame image 30A can also be estimated using other methods.

Therefore, in the third embodiment, the image collecting device 2B that estimates the object region 32 from the position of the label specifying device 4 included in the reference frame image 30A will be described.

The configuration of the image acquisition system 1 including the image acquisition device 2B is the same as that shown in FIG.

<Structure of the image collecting device according to the third embodiment>
FIG. 8 is a diagram showing a configuration example of the image collecting device 2B.

The configuration of the image collection device 2B shown in FIG. 8 is different from the configuration of the image collection device 2 shown in FIG. 2 in that the label identification device estimation unit 23 is added and the object area estimation unit 24 is added to the object area estimation unit 24B. It is a replaced point.

When the label specifying unit 22 notifies the label specifying device estimation unit 23 of the processing start command, the label specifying device estimation unit 23 acquires the reference frame image 30A from each of the captured images captured by the photographing device 6. Then, the label specifying device estimation unit 23 estimates the device area 34 including the label specifying device 4 from each of the acquired reference frame images 30A.

Then, the label specifying device estimation unit 23 notifies the object area estimation unit 24B of the position information of the device area 34 for each estimated reference frame image 30A.

When the object area estimation unit 24B receives the position information of the device area 34 for each reference frame image 30A from the label specifying device estimation unit 23, the object area 32 including the object 8 is selected from the position information of the device area 34 as the reference frame image 30A. Estimate for each.

Then, the object area estimation unit 24B receives the position information of each reference frame image 30A used for estimating the object area 32 and the object area 32 of the object 8 estimated from each reference frame image 30A in the training image registration unit 26. Notify to.

<Operation of the image collecting device according to the third embodiment>
Next, the operation of the image collecting device 2B according to the present embodiment will be described. When the input unit 10 receives the label information of the object 8, the image collecting device 2B executes the image collecting routine shown in FIG.

The difference between the image collection routine shown in FIG. 9 and the image collection routine of the image collection device 2 shown in FIG. 3 is that step S42 is newly added and step S50 is replaced with step S50A.

In step S40, when the reference frame image 30A is acquired from each captured image, step S42 is executed.

In step S42, the CPU estimates the device area 34 including the label specifying device 4 from each of the reference frame images 30A acquired in step S40. At this time, a known method can be used as the estimation method of the device region 34.

For example, training images of the label specifying device 4 obtained by photographing the label specifying device 4 in various directions and distances are prepared in advance, and the label is specified from each reference frame image 30A by using the object identification technique shown in Patent Document 1. The device area 34 including the device 4 may be estimated.

In step S50A, the CPU estimates the object area 32 including the object 8 whose type is specified in step S20 based on the position information of the device area 34 in each reference frame image 30A estimated in step S42. When estimating the object area 32 from the device area 34, a known estimation method can be used.

For example, when acquiring the label information of the object 8, the user performs an operation of bringing the label specifying device 4 closer to the object 8, so that the object 8 is often present in the vicinity of the label specifying device 4.

Therefore, the relative position of the object 8 with respect to the label specifying device 4 when the user tries to acquire the label information by bringing the label specifying device 4 close to the object 8, and the average size of the object 8 (that is, the object area 32). The size) is statistically obtained in advance for each reference frame image 30A in each photographing device 6 by an experiment. Then, as shown in FIG. 10, the device area 34 is moved based on the relative position of the object 8 obtained in advance, and the device area 34 is enlarged or reduced to the size of the object area 32 obtained in advance. Therefore, the object region 32 can be estimated for each reference frame image 30A in each photographing device 6.

The object area 32 may be estimated by moving, enlarging, and reducing the device area 34 by using the method described in Non-Patent Document 6.

[Non-Patent Document 6]: Carstern Rother, Vladimir Kolmogorov, Andrew Blake. "GrabCut-Interactive Foreground Extraction using Iterated Graph Cuts" in SIGGRAPH, 2014

After that, in step S60 already described, the type of the object 8 included in the object area 32 specified in step S20 and the reference frame with respect to the reference frame image 30A used for estimating the object area 32 in step S50A. The position information of the object area 32 in the image 30A may be associated with each reference frame image 30A to obtain a training image.

As described above, according to the image collecting device 2B according to the present embodiment, instead of estimating the user's hand area 31 from the reference frame image 30A, the device area 34 including the label specifying device 4 is estimated and the estimated device area is estimated. The object region 32 including the object 8 is estimated based on 34.

Since the shape of the hand changes variously but the shape of the label specifying device 4 does not change, the estimation accuracy of the device area 34 may be higher than the estimation accuracy of the hand area 31. Therefore, as the estimation accuracy of the device area 34 improves, the estimation accuracy of the object area 32 can be improved.

Here, an example of estimating the object area 32 after estimating the device area 34 from the reference frame image 30A has been described, but it goes without saying that it can also be applied to the case of estimating the object area 32 from the adjacent frame image.

The present invention is not limited to the above-described embodiments, and various modifications and applications are possible without departing from the gist of the present invention.

For example, in each of the above-described embodiments, an example in which the image acquisition routine is realized by software is shown, but it may be realized by using hardware such as an ASIC (Application Specific Integrated Circuit).

Further, in each of the above-described embodiments, the mode in which the image acquisition program is installed in the ROM has been described, but the present invention is not limited to this. The image acquisition program according to the present invention can also be provided in a form recorded on a computer-readable storage medium. For example, the image acquisition program according to the present invention may be provided in the form of being recorded on an optical disk such as a CD (Compact Disc) -ROM or a DVD (Digital Versatile Disc) -ROM. Further, the image acquisition program according to the present invention may be provided in a form of being recorded in a semiconductor memory such as a USB (Universal Serial Bus) memory and a flash memory. Further, when the image collecting device is connected to a communication line such as the Internet, the image collecting program according to the present invention may be acquired from a terminal device such as a server connected to the communication line.

1 ... Image collection system 2 (2A, 2B) ... Image collection device 4 ... Label identification device 6 ... Imaging device 8 ... Object 10 ... Input unit 20 ... Calculation unit 22 ... Label identification unit 23 ... Label identification device estimation unit 24 (24B) ... Object area estimation unit 25 ... Object tracking unit 26 (26A) ... Training image registration unit 28 ... Training image Storage unit 30A ... Reference frame image 30B (30C) ... Adjacent frame image 31 ... Hand area 32 ... Object area 34 ... Device area

Claims (6)

A label specifying unit that acquires label information of the object from a label specifying device that acquires label information indicating the type of the object given to the object by approaching the object, and specifies the type of the object.
The object used to identify the type of the object in the label specifying unit from each of the captured images taken by at least one photographing device that photographs the acquisition status of the label information of the object using the label specifying device. An object area estimation unit that acquires a reference frame image at the moment when the label information of is acquired and estimates an object area including the object from the acquired reference frame image.
The type of the object specified by the label specifying unit, the reference frame image used for estimating the object area of the object by the object area estimation unit, and the reference frame image estimated by the object area estimation unit. A training image registration unit that associates the object areas of objects with each other and registers them in the storage device as training images.
A label specifying device estimation unit that estimates a device area including the label specifying device from each of the reference frame images of the captured image taken by the shooting device is provided.
The object area estimation unit is an image collecting device that estimates an object area including the object from the reference frame image based on the position of the device area of the label identification device estimated by the label identification device estimation unit. From each of the captured images, adjacent frame images, which are images captured in at least one range before and after the reference frame image acquired by the object area estimation unit, were acquired and acquired in chronological order. Further provided is an object tracking unit that tracks the object included in the reference frame image from each of the adjacent frame images and estimates an object region including the object from each of the adjacent frame images including the object.
The training image registration unit includes the type of the object specified by the label specifying unit, the adjacent frame image used by the object tracking unit to estimate the object area of the object, and the estimated object tracking unit. The image collecting device according to claim 1, wherein the object regions of the objects in the adjacent frame images are associated with each other and registered in the storage device as a training image. The image collecting device according to claim 1 or 2 , wherein the label specifying unit acquires label information of the object from the label specifying device that reads the label information of the object using light or radio waves. A program for causing a computer to function as each part of the image collecting device according to any one of claims 1 to 3. A step of acquiring label information of the object from a label specifying device for acquiring label information indicating the type of the object given to the object by approaching the object and specifying the type of the object.
The label information of the object used for identifying the type of the object is acquired from each of the captured images taken by at least one photographing device for photographing the acquisition status of the label information of the object using the label specifying device. A step of acquiring a reference frame image at the moment when the object is created and estimating an object region including the object from the acquired reference frame image.
A step of associating the specified type of the object, the reference frame image used for estimating the object area of the object, and the object area of the object in the reference frame image, and registering them in the storage device as a training image. When,
A step of estimating the device area including the label specifying device from each of the reference frame images of the captured image captured by the photographing device is included.
In the step of estimating the object area including the object from the reference frame image, the image that estimates the object area including the object from the reference frame image based on the estimated position of the device area of the label specifying device. Collection method. Adjacent frame images, which are images taken in at least one range before and after the reference frame image, are acquired from each of the captured images in chronological order, and the reference is obtained from each of the acquired adjacent frame images. A step of tracking the object included in the frame image and estimating an object region containing the object from each of the adjacent frame images including the object.
The identified type of the object, the adjacent frame image used for estimating the object area of the object, and the object area of the object in the adjacent frame image are associated with each other and registered in the storage device as a training image. Steps and
5. The image collecting method according to claim 5, further comprising.

Images