JP7222519B2 - Object identification system, model learning system, object identification method, model learning method, program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Law 1. Presentation of object identification system, model learning system, object identification method, model learning method, program at meeting Name of meeting: Faculty of Software and Information Sciences, Iwate Prefectural University 2017 Graduation Research Results Presentation Venue: Faculty of Software and Information Sciences Building, Iwate Prefectural University Date: February 9, 2018 2. Publication of object identification system, model learning system, object identification method, model learning method, program Publication: Proceedings of the 80th Annual Conference of Information Processing Society of Japan Publisher: Information Processing Society of Japan Publication date: Heisei March 13, 2018 3. Presentation of object identification system, model learning system, object identification method, model learning method, program at meeting Name of meeting: Information Processing Society of Japan 80th National Convention Venue: Waseda University Nishi-Waseda Campus Date: March 15, 2018 Day

The present invention relates to an object identification system, model learning system, object identification method, model learning method, and program.

2. Description of the Related Art Conventionally, there is known an image recognition technique targeting objects (for example, people, animals, objects, etc.) included in a captured image (for example, Patent Document 1). In the technique described in Patent Document 1, a feature quantity extraction unit that calculates a feature vector including, as an element, a gradient moment that is the product of the gradient of the pixel value of a pixel in an image and the coordinates of the pixel, and a feature quantity extraction unit: and a classification unit that classifies the images based on the similarity of the feature vectors calculated in the above, and classifies the images into "vehicle images" and "non-vehicle images" based on the image features. It is configured.

JP 2015-011552 A

In the prior art, only the class (“vehicle” or “non-vehicle”) to which the object included in the image belongs can be recognized (class recognition). is present, it is difficult to recognize (instance recognition) the object itself included in each captured image.

The present invention has been made in view of the above problems, and aims to provide an object identification system, a model learning system, an object identification method, a model learning method, and a program capable of identifying an object itself included in a captured image. aim.

In order to solve the above problems, firstly, the present invention provides a first acquisition means for acquiring a captured image of any one of a plurality of objects existing in a predetermined space, and a second acquisition means for acquiring environmental information at a position; and when the captured image and the environmental information are acquired, the captured image and the acquired environmental information, and using the captured image and the environmental information as data for learning. and an identification means for identifying which of the plurality of objects the photographed object is based on the learned model based on machine learning (Invention 1) .

According to this invention (Invention 1), when the captured image and the environment information are acquired, the acquired captured image and the environment information, and the learned model based on machine learning using the captured image and the environment information as learning data, Therefore, for example, instead of identifying the class to which the object belongs using only the captured image of the object, it is possible to identify the class of the object. By further using environmental information at the imaging position of the object together with the captured image, it becomes possible to identify the object itself included in the captured image. As a result, the individual recognition performance of the imaged object can be improved.

In the above invention (invention 1), the second acquisition means includes any one of communication devices provided at a plurality of positions in the predetermined space and a terminal device present at a position where any one of the objects is captured. A received signal strength when a radio signal transmitted from one of them is received by the other may be acquired as the environmental information (Invention 2).

According to this invention (invention 2), the received signal strength when the radio signal transmitted from one of the plurality of communication devices and the terminal device is received by the other is calculated as the strength of the received signal when any one of the objects is imaged. It can be acquired as environment information representing the position of the terminal device (that is, the imaging position of any of the objects).

In the above invention (invention 2), assuming that the environment information used as the learning data follows a Gaussian distribution in which the received signal strength corresponding to each communication device follows a different Gaussian distribution for each captured object and for each communication device, A plurality of combinations of received signal strength of each communication device extracted according to a Gaussian distribution of received signal strength of each communication device corresponding to a predetermined object among the plurality of objects may be included (Invention 3).

According to this invention (Invention 3), for example, assuming that a predetermined object among a plurality of objects is imaged, the received signal strength is calculated according to the Gaussian distribution of the received signal strength of each communication device corresponding to the predetermined object. By extracting for each communication device, it is possible to easily generate a plurality of combinations of received signal strengths corresponding to the predetermined object. As a result, it is possible to increase the amount of environmental information included in the learning data (data augmentation), so machine learning can proceed efficiently.

Secondly, the present invention provides an acquisition means for acquiring a captured image of any one of a plurality of objects existing in a predetermined space and environmental information at the position where the one of the objects was captured; learning means for learning a model used to identify which of the plurality of objects the imaged object is by machine learning using the captured image and the environment information as learning data; (Invention 4).

According to this invention (Invention 4), there is provided a model that is used to identify which of a plurality of objects a captured object is by machine learning using captured images and environment information as learning data. can be learned, the object itself included in the captured image can be identified by using this model.

Thirdly, the present invention provides a computer with a step of acquiring a captured image of any one of a plurality of objects existing in a predetermined space, and a step of acquiring environmental information at the position where the one of the objects is captured. and, when the captured image and the environment information are acquired, the acquired captured image and the environment information, and a learned model based on machine learning using the captured image and the environment information as learning data; Based on this, there is provided an object identification method for identifying which of the plurality of objects the photographed object is (Invention 5).

Fourthly, the present invention provides a computer with a step of obtaining, in a computer, a captured image of one of a plurality of objects existing in a predetermined space and environmental information at the position where the one of the objects was captured; a step of learning a model used to identify which of the plurality of objects the imaged object is, using the captured image and the environment information obtained as learning data; (Invention 6).

Fifthly, the present invention provides a computer with a function of acquiring a captured image of any one of a plurality of objects existing in a predetermined space, and a function of acquiring environmental information at the position where the one of the objects is captured. and, when the captured image and the environment information are acquired, the acquired captured image and the environment information, and a learned model based on machine learning using the captured image and the environment information as learning data; Based on this, a program for realizing a function of identifying which of the plurality of objects the photographed object is is provided (Invention 7).

Sixthly, the present invention provides a computer with a function of acquiring a captured image of any one of a plurality of objects existing in a predetermined space and environmental information at the position where the one of the objects was captured; a function of learning a model used to identify which of the plurality of objects the imaged object is, using the captured image and the environment information obtained as learning data. (Invention 8).

According to the object identification system, model learning system, object identification method, model learning method, and program of the present invention, the object itself included in the captured image can be identified.

1 is a diagram schematically showing the basic configuration of an object identification system and a model learning system according to one embodiment of the present invention; FIG. It is a block diagram which shows the structure of a terminal device. It is a block diagram which shows the structure of an identification device. FIG. 3 is a functional block diagram for explaining functions that play major roles in the object identification system and model learning system; It is a figure which shows the structural example of acquisition data. FIG. 4 is a diagram showing the relationship between the distance between a communication device and a terminal device and received signal strength; It is a figure which shows the structural example of the data for learning. (a) to (c) are diagrams showing an example of the structure of a model to be machine-learned. (a) to (c) are diagrams showing an example of the distribution of received signal strength for each imaging target and for each communication device. 4 is a flowchart showing an example of main processing of the model learning system according to one embodiment of the present invention; 4 is a flow chart showing an example of main processing of the object identification system according to one embodiment of the present invention; FIG. 4 is a diagram showing an example of division of functions between an identification device and a learning device for each function of an object identification system and a model learning system;

An embodiment of the present invention will be described in detail below with reference to the accompanying drawings. However, this embodiment is an example, and the present invention is not limited to this.

(1) Basic Configuration of Object Identification System and Model Learning System FIG. 1 is a diagram schematically showing the basic configuration of an object identification system and a model learning system according to an embodiment of the present invention. As shown in FIG. 1, the object identification system according to the present embodiment includes a plurality of objects (in the example of FIG. ) exists in the space S, a plurality of terminal devices 20 (indicated as “a” to “d” in the example of FIG. 1) existing in the space S 4) objects (for example, an object such as a trash can or a fire extinguisher) OB, a captured image captured by using the terminal device 20, and a terminal device when capturing an image of the object OB. When the identification device 30 acquires environmental information (in this embodiment, received signal strength indicator (RSSI)) at the position of the device 20, the identification device 30 acquires the captured image and the environmental information, and the captured image and the environmental information. Based on the learned model based on machine learning used as learning data, which object OB is the imaged object OB among the plurality of objects OB is identified. Here, the terminal device 20 and the identification device 30 are connected to a communication network NW (network) such as the Internet or a LAN (Local Area Network).

Further, in the model learning system according to the present embodiment, when the identification device 30 acquires the captured image and the environment information, the captured object OB is determined by machine learning using the acquired captured image and the environment information as learning data. It learns a model that is used to identify which object OB is among a plurality of objects OB.

In this embodiment, a case where a plurality of objects OB are objects of the same type (class) will be described as an example, but each object OB may be an object of a different type.

Each communication device 10 is provided in a position where it is possible to perform wireless communication with the terminal device 20 using a wireless LAN (eg, Wi-Fi (registered trademark)) within the space S. Each communication device 10 may be, for example, a device that relays wireless communication between two or more terminal devices 20, or may be a device that relays wireless communication between the terminal device 20 and another terminal device (not shown) existing in the space S. It may be a device that relays wireless communication between terminals, or a device that relays communication between the terminal device 20 and another device connected via the communication network NW. Also, each communication device 10 may be a packet capture.

Note that although a case where wireless communication is performed using Wi-Fi (registered trademark) is described as an example here, the communication method is not limited to this case. For example, a wireless communication method such as Bluetooth (registered trademark), ZigBee (registered trademark), UWB, or optical wireless communication (for example, infrared) may be used, or a wired communication method such as USB may be used.

The terminal device 20 is configured to be able to wirelessly communicate with each communication device 10 using a wireless LAN when present in the space S. In addition, in order to perform wireless communication with each communication device 10, the terminal device 20 transmits a radio signal (eg probe request etc.) including its own identification information (eg MAC address etc.) at predetermined intervals (eg several seconds). may be configured to transmit. Furthermore, the terminal device 20 may be configured to measure environmental information (received signal strength of a wireless signal in this embodiment) at the imaging position when any object OB is imaged. The terminal device 20 is, for example, a mobile terminal, a smart phone, a PDA (Personal Digital Assistant), a personal computer, a television receiver having a two-way communication function (including a so-called multifunctional smart television), or the like. , terminal devices operated by individual users.

The identification device 30 is configured to communicate with the terminal device 20 via the communication network NW and acquire the captured image and environmental information from the terminal device 20 . Note that the identification device 30 may not be connected to the terminal device 20 via the communication network NW when it is configured to be able to communicate with the terminal device 20 via a plurality of communication devices 10 . The identification device 30 may be, for example, a general-purpose personal computer.

(2) Configuration of Terminal Device The configuration of the terminal device 20 will be described with reference to FIG. FIG. 2 is a block diagram showing the internal configuration of the terminal device 20. As shown in FIG. As shown in FIG. 2, the terminal device 20 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage device 24, a display processing unit 25, A display unit 26, an input unit 27, an imaging unit 28, and a communication interface unit 29 are provided, and a bus 20a is provided for transmitting control signals or data signals between the units.

When the terminal device 20 is powered on, the CPU 21 loads various programs stored in the ROM 22 or the storage device 24 into the RAM 23 and executes them. In addition, every time an image including one of the plurality of objects OB is captured using the imaging unit 28, the CPU 21 transmits the captured image and environmental information at the imaging position via the communication interface unit 29. is configured to transmit to the terminal device 20.

The storage device 24 is, for example, a non-volatile storage device such as flash memory, SSD (Solid State Drive), magnetic storage device (eg, HDD (Hard Disk Drive), floppy disk (registered trademark), magnetic tape, etc.), optical disk, or the like. Alternatively, it may be a volatile storage device such as a RAM, which stores programs executed by the CPU 21 and data referred to by the CPU 21 .

The display processing unit 25 displays the display data provided from the CPU 21 on the display unit 26 . The display unit 26 is, for example, an LCD (Liquid Crystal Display) monitor including thin film transistors arranged in a matrix on a pixel-by-pixel basis. indicate.

If the terminal device 20 is a button input type communication terminal, the input unit 27 includes a group of buttons including a plurality of instruction input buttons such as a direction instruction button and a decision button for accepting user's operation input, and a numeric keypad. and an interface circuit for recognizing the pressing (operation) input of each button and outputting it to the CPU 21 .

If the terminal device 20 is a touch panel input type communication terminal, the input unit 27 mainly accepts touch panel type input by touching the display screen with a fingertip or a pen. The touch panel input method may be a known method such as a capacitive method.

Further, when the terminal device 20 is a terminal device capable of voice input, the input unit 27 may be configured to include a microphone for voice input, or may be configured to include a voice input via an external microphone. An interface circuit for outputting data to the CPU 21 may be provided.

The imaging unit 28 may be an imaging device (for example, a digital camera, a digital video camera, etc.) that captures moving images and/or still images. It is configured to perform processing and store the captured image in the RAM 23 or the storage device 24, for example. Note that the imaging unit 28 may be built in the terminal device 20 or may be provided outside the terminal device 20 .

The communication interface unit 29 includes an interface circuit for wireless communication with each communication device 10 and an interface circuit for communication via the communication network NW. Further, the communication interface unit 29 is provided with an RSSI circuit for detecting the received signal strength when receiving the radio signal transmitted from each communication device 10 .

Note that the radio signal transmitted from each communication device 10 may include identification information (eg, MAC (Media Access Control) address, etc.) of the communication device 10 that transmitted the radio signal. When the received signal strength of the transmitted radio signal is detected by the RSSI circuit, the detected received signal strength and the identification information of the communication device 10 are associated with each other and stored in the RAM 23 or the storage device 24, for example. may

(3) Configuration of Identification Device The configuration of the identification device 30 will be described with reference to FIG. FIG. 3 is a block diagram showing the internal configuration of the identification device 30. As shown in FIG. As shown in FIG. 3, the identification device 30 includes a CPU 31, a ROM 32, a RAM 33, a storage device 34, a display processing section 35, a display section 36, an input section 37, and a communication interface section 38. A bus 30a is provided for transmitting control signals or data signals between the units.

When the identification device 30 is powered on, the CPU 31 loads various programs stored in the ROM 32 or the storage device 34 into the RAM 33 and executes them. In this embodiment, the CPU 31 reads out and executes a program stored in the ROM 32 or the storage device 34 to obtain first acquisition means 41, second acquisition means 42, identification means 43, acquisition means 44, and learning means, which will be described later. 45 (shown in FIG. 4).

The storage device 34 may be, for example, a nonvolatile storage device such as a flash memory, an SSD, a magnetic storage device (eg, HDD, floppy disk (registered trademark), magnetic tape, etc.), an optical disk, or a volatile storage device such as a RAM. It may be a physical storage device, and stores programs executed by the CPU 31 and data referred to by the CPU 31 . Acquisition data (shown in FIG. 5) and learning data (shown in FIG. 7), which will be described later, are stored in the storage device 34 .

The input unit 37 may be, for example, an information input device such as a mouse or keyboard, may be configured to include a microphone for voice input, or may include a digital camera or digital video camera for image input. or an interface circuit for receiving image data captured by an external digital camera or digital video camera and outputting the data to the CPU 31 .

The communication interface unit 38 includes an interface circuit for communicating via the communication network NW. Details of other units in the identification device 30 may be the same as those in the terminal device 20 .

(4) Outline of Each Function in Object Identification System and Model Learning System Functions implemented in the object identification system and model learning system of the present embodiment will be described with reference to FIG. FIG. 4 is a functional block diagram for explaining functions that play major roles in the object identification system and model learning system of this embodiment. In the functional block diagram of FIG. 4, the first acquisition means 41, the second acquisition means 42 and the identification means 43 correspond to the main components of the object identification system of the present invention, and the acquisition means 44 and the learning means 45 correspond to the main components of the object identification system of the present invention. It corresponds to the main configuration of the model learning system of

The first acquisition means 41 has a function of acquiring a captured image of any one of a plurality of objects OB present in the predetermined space S. FIG.

The functions of the first acquisition means 41 are realized, for example, as follows. First, the CPU 21 of the terminal device 20 receives a predetermined imaging instruction when any object OB among a plurality of objects OB (for example, the object OB of "a") exists within the imaging range of the imaging unit 28. When input using the unit 27, the imaging unit 28 is caused to perform imaging processing. The data of the image captured by the imaging unit 28 (here, the captured image of the object OB of “a”) is stored in the RAM 23 or the storage device 24, for example. Then, the CPU 21 transmits the captured image data stored in, for example, the RAM 23 or the storage device 24 to the identification device 30 via the communication interface unit 29 and the communication network NW.

On the other hand, when the CPU 31 of the identification device 30 receives (acquires) the captured image data via the communication interface unit 38, the received captured image data is stored in the RAM 33 or the storage device 34, for example. Note that when the captured image is composed of a moving image, the CPU 31 may divide the captured image into several frames and store each divided frame as a captured image in the RAM 33 or the storage device 34, for example. good. In this way, it is possible to obtain a captured image of one of the plurality of objects OB present in the space S (here, the object OB of "a").

The second acquisition means 42 has a function of acquiring environment information at a position where any one of the plurality of objects OB is imaged.

In addition, the second acquisition unit 42 receives data transmitted from either one of the communication devices 10 provided at a plurality of positions within the predetermined space S and the terminal device 20 present at the position where any object OB is imaged. The received signal strength when the other side receives the radio signal received by the other side may be acquired as the environment information. As a result, the received signal strength when one of the plurality of communication devices 10 and the terminal device 20 receives a radio signal transmitted from the other is calculated as the received signal strength of the terminal device 20 when any object OB is imaged. It can be acquired as environment information representing the position (that is, the imaging position of any object OB).

Here, the received signal strength obtained by the second obtaining means 42 may be, for example, the value of the received signal strength itself, or may be obtained by substituting the value of the received signal strength into a predetermined formula. It may be a value, or it may be information representing the degree of received signal strength.

The function of the second obtaining means 42 is realized, for example, as follows. Here, the second acquisition unit 42 obtains the strength of the received signal when the terminal device 20 receives the wireless signal transmitted from each communication device 10 by capturing an image of one of the plurality of objects OB. A case of acquiring environment information at a position will be described as an example. First, every time the CPU 21 of the terminal device 20 receives a radio signal transmitted from each communication device 10, the received signal strength value of the radio signal detected by the RSSI circuit is stored in the RAM 23 or the storage device 24, for example. . Further, when causing the imaging unit 28 to perform imaging processing as described above, the CPU 21 selects a plurality of latest received signal strength values among the received signal strength values stored in the RAM 23 or the storage device 24, for example. is extracted for each communication device 10, and the value of the extracted received signal strength is transmitted to the identification device 30 via the communication interface unit 29 and the communication network NW. Here, the CPU 21 may transmit the value of the extracted received signal strength together with the captured image.

On the other hand, when the CPU 31 of the identification device 30 receives (acquires) the value of the received signal intensity corresponding to each communication device 10 via the communication interface unit 38, the captured image acquired based on the function of the first acquisition means 41 The data and the value of the received signal strength corresponding to each communication device 10 are stored in the acquired data shown in FIG. 5, for example, in a state of being associated with each other. Acquired data is data described in a state in which values of received signal strength corresponding to each communication device 10 (communication devices A to D in the example of the drawing) are associated with each captured image. In this way, the received signal strength when the terminal device 20 receives the radio signal transmitted from each communication device 10 is acquired as environmental information at the position where any one of the plurality of objects OB is imaged. be able to.

式（１）中、Ｐ_ｒは受信信号強度（ｄＢｍ）を示し、Ｐ_ｔは電波発信装置（ここでは、通信装置１０）の送信電力（ｄＢｍ）を示し、Ｇ_ｒは端末装置２０の受信アンテナの利得（ｄＢｉ）を示し、Ｇ_ｔは通信装置１０の送信アンテナの利得（ｄＢｉ）を示し、Ｌは自由空間損失（ｄＢｍ）を示している。このＬは式（２）でもとめられ、式（２）中、ｄは通信装置１０と端末装置２０との距離（ｍ）を示し、ｆは電波の周波数（Ｈｚ）を示し、ｃは光速（＝２．９９７９２４５８×１０^８）（ｍ／ｓ）を示している。式（１）及び式（２）によってもとめられる距離と受信信号強度の値との関係は、例えば図６に示す対数関数で表される。図６に示すように、通信装置１０と端末装置２０との距離が短いほど、受信信号強度の値が大きいことがわかる。 The distance between the communication device 10 and the terminal device 20 can be calculated based on the received signal strength when the terminal device 20 receives the radio signal transmitted from the communication device 10 . Specifically, the distance between one communication device 10 and the terminal device 20 can be calculated using, for example, the following equations (1) and (2).
P _r =P _t +G _r +G _t −L (1)

In equation (1), P _r indicates the received signal strength (dBm), P _t indicates the transmission power (dBm) of the radio wave transmitting device (here, the communication device 10), and G _r indicates the receiving antenna of the terminal device 20. _Gt denotes the gain (dBi) of the transmission antenna of the communication device 10, and L denotes the free space loss (dBm). This L is determined by the equation (2), in which d indicates the distance (m) between the communication device 10 and the terminal device 20, f indicates the frequency of radio waves (Hz), and c indicates the speed of light ( =2.99792458×10 ⁸ ) (m/s). The relationship between the distance and the value of the received signal strength determined by equations (1) and (2) is represented by a logarithmic function shown in FIG. 6, for example. As shown in FIG. 6, the shorter the distance between the communication device 10 and the terminal device 20, the greater the value of the received signal strength.

In this way, by obtaining the distance between each of the plurality of communication devices 10 and the terminal device 20 when any object OB is imaged, the distance in the space S when any object OB is imaged can be determined. , the position of the terminal device 20 (that is, the imaging position of any object OB) can be estimated. Note that when the number of communication devices 10 provided in the space S is three or more, the position of the terminal device 20 on a predetermined plane in the space S can be estimated, and the number of communication devices 10 is four. In the case of one or more, the three-dimensional position of the terminal device 20 in the space S can be estimated.

When the captured image and the environment information are acquired, the identification means 43 acquires the captured image and the environment information, and the learned model based on machine learning using the captured image and the environment information as learning data. It has a function of identifying which of a plurality of objects OB the imaged object OB is.

The function of the identification means 43 is realized, for example, as follows. When the CPU 31 of the identification device 30 acquires the captured image and the environment information based on the functions of the first acquisition unit 41 and the second acquisition unit 42, the CPU 31 converts the acquired captured image and the environment information into learning data. By inputting to a learned model (described later) based on machine learning used as , it is identified which object OB the imaged object OB is among a plurality of objects OB. An example of learning data is shown in FIG. The learning data shown in FIG. 7 is a received signal corresponding to environmental information at the imaging position (in the example in FIG. 7, each communication device 10 in the imaging position (communication devices A to D in the example in the figure) at the imaging position for each captured image. intensity value) and the imaged object OB (which is correct data) (one of “a”, “b”, “c” and “d” in the example of FIG. 7) are associated with each other. It is the data described in the state. Thus, as a result of machine learning, a learned model representing the relationship between the captured image and the environment information (here, received signal strength corresponding to each communication device 10) and the captured object OB is constructed.

Note that, when the CPU 31 identifies which object OB (for example, the object OB of "a") the imaged object OB is, for example, the identified object OB (here, the object OB of "a") (For example, it may be displayed on the display unit 36, or may be output from an audio output device such as a speaker), the first acquisition means 41 and The captured image and the environment information acquired based on the function of the second acquisition means 42 and the information related to the identified object OB may be stored in the RAM 33 or the storage device 34 in association with each other.

Acquisition means 44 has a function of acquiring a captured image of any object OB among a plurality of objects OB existing in the predetermined space S, and environmental information at the position where the object was captured.

The function of the acquisition means 44 is realized, for example, as follows. Here, in the case where the acquiring unit 44 acquires the received signal strength when the terminal device 20 receives the radio signal transmitted from each communication device 10 as the environmental information at the position where any object OB is imaged. will be described as an example. The CPU 31 of the identification device 30 receives the captured image data and the received signal strength values corresponding to the communication devices 10 via the communication interface unit 38, similarly to the functions of the first acquisition unit 41 and the second acquisition unit 42. is received (acquired) from the terminal device 20, the data of the received captured image and the value of the received signal strength corresponding to each communication device 10 are stored in the learning data shown in FIG. . Further, the CPU 31 selects which object OB among a plurality of objects OB (“a”, “b”, “c” and “d” in the example of FIG. 7) for each captured image stored in the learning data. is input using the input unit 37, the input correct data is stored in the item of "object" corresponding to the captured image.

The learning means 45 is a model used for identifying which of a plurality of objects OB the imaged object OB is by machine learning using the acquired captured image and environment information as learning data. Equipped with a function to learn

The function of the learning means 45 is realized, for example, as follows. For example, when a predetermined model learning instruction is input using the input unit 37, the CPU 31 of the identification device 30 learns the model using the learning data shown in FIG. Here, the CPU 31 may learn using any one of the three types of models shown in FIGS. 8(a) to 8(c). Each model is described below.

式（３）において、Ｐ_ｉｊはｘ_ｉｊｋと同じ縦横比であって全ての要素が１のテンソルであり、式（３）において、｜｜は各テンソルの連結を表す。θは、確率的勾配降下法を用いて決定される。入力層で連結を行うことから、本実施形態では、このモデルをｍＣＮＮ－ｃ（multimodal Convolutional Neural Network concatenate）と称することとする。 In the model shown in FIG. 8A, first, the input layer shapes a plurality of environmental information (in this case, the received signal strength corresponding to each of the plurality of communication devices 10) to the same aspect ratio as the image. It is considered that this makes it possible to simultaneously extract a plurality of environmental information and image features. The model then learns the tensor after concatenating each environmental information and the image after shaping. Although three images are shown in FIG. 8(a), these are tensors made up of respective pixels of RGB of the captured image. Here, a tensor representing an image is x _ijk (i is the number of vertical pixels, j is the number of horizontal pixels, k is the number of channels), and a tensor representing environmental information is s _l (l is the number (here, the number of terminal devices 20)), X is the tensor obtained by manipulating each of _xijk and _sl , and f and f' are neural networks. Then, the probability f _θ (x _ijk , _sl ) belonging to each object “a”, “b”, “c” and “d”) is calculated by using the following equations (3) and (4) can do.

In equation (3), P _ij is a tensor with the same aspect ratio as x _ijk and all elements are 1, where || represents the concatenation of each tensor. θ is determined using stochastic gradient descent. Since concatenation is performed in the input layer, this model is called mCNN-c (multimodal convolutional neural network concatenate) in this embodiment.

θ及びｗは、確率的勾配降下法を用いて決定される。入力層で重みを付けることから、本実施形態では、このモデルをｍＣＮＮ－ｗ（multimodal Convolutional Neural Network weighted）と称することとする。 In the model shown in FIG. 8(b), the input layer weights a plurality of environmental information (in this case, the received signal strength corresponding to each of the plurality of communication devices 10) and learns the sum with the image. ing. It is considered that this makes it possible to express the degree of influence of each piece of environmental information. If the weight for s _l is w _l , the probability f _θ (x _ijk , s _l ) can be calculated by using equations (5) and (6) below.

θ and w are determined using stochastic gradient descent. Since the input layer is weighted, this model is called mCNN-w (multimodal convolutional neural network weighted) in this embodiment.

θは、確率的勾配降下法を用いて決定される。全結合層の前の連結を行うことから、本実施形態では、このモデルをｍＣＮＮ－ｆ（multimodal Convolutional Neural Network fully-connected）と称することとする。なお、発明者は、上述した３つのモデルのうちｍＣＮＮ－ｆを用いた場合に、少ない学習回数で高精度の識別結果が得られることを見出した。 In general, convolutional neural networks (CNNs) transform the tensor to rank 1 before the output layer and use fully connected layers. The model shown in FIG. 8(c) concatenates a rank-1 tensor after using CNN for the image and a rank-1 tensor obtained by stacking multiple fully-connected layers for the environment information, and Bonding layers are stacked. Thus, it is conceivable that image features can be extracted by CNN, and environmental information features can be extracted by FNN (Forward Neural Network). When the intermediate layers are f'' and f'', the probability f _θ ( x, s) can be calculated by using equation (7) below.

θ is determined using stochastic gradient descent. In this embodiment, we refer to this model as mCNN-f (multimodal Convolutional Neural Network fully-connected), since it performs the concatenation before the fully connected layer. The inventors have found that, among the three models described above, when mCNN-f is used, highly accurate identification results can be obtained with a small number of times of learning.

It should be noted that the environment information used as learning data is assumed to follow a Gaussian distribution in which the received signal intensity corresponding to each communication device 10 is different for each object OB captured and for each communication device 10. A plurality of combinations of the received signal strength of each communication device 10 extracted according to the Gaussian distribution of the received signal strength of each communication device 10 corresponding to a predetermined object OB may be included. As a result, for example, assuming that a predetermined object OB among a plurality of objects OB is imaged, the received signal strength is calculated by the communication device 10 according to the Gaussian distribution of the received signal strength of each communication device 10 corresponding to the predetermined object OB. A plurality of combinations of received signal strengths corresponding to the predetermined object OB can be easily generated by extracting each of them. As a result, it is possible to increase the amount of environmental information included in the learning data (data augmentation), so machine learning can proceed efficiently.

The functions of the acquisition means 44 or the learning means 45 in this case are realized, for example, as follows. The CPU 31 of the identification device 30 performs learning at, for example, each predetermined timing (for example, each time a predetermined time elapses, or each time a predetermined number of captured images and environmental information are stored). Data augmentation processing may be performed on the following environment information (here, received signal strength corresponding to each communication device 10) using the data for the environment.

FIGS. 9A to 9C show an example of distribution of received signal strength corresponding to each communication device 10 when an object OB out of a plurality of objects OB is imaged. FIG. 9(a) shows an example of the distribution of the received signal intensity corresponding to the communication device A when the object OB of "a" is imaged, and FIG. 9(b) shows the distribution of the object OB of "b". 9C shows an example of the distribution of the received signal intensity corresponding to the communication device A when is imaged, and FIG. 9C shows the reception corresponding to the communication device B when the object OB of "a" is imaged An example of signal intensity distribution is shown. These distributions can be generated using training data, for example.

As shown in FIGS. 9A to 9C, each distribution approximates a symmetrical bell-shaped distribution (Gaussian distribution) about the average value of received signal strength. Therefore, the received signal strength corresponding to each communication device 10 is determined for each imaged object OB (“a”, “b”, “c”, and “d” object OB) and for each communication device (communication device A to D) can be assumed to follow different Gaussian distributions. Therefore, the CPU 31 of the identification device 30 performs data augmentation processing of the environment information when a predetermined object OB (for example, the object OB of "a") among the plurality of objects OB is imaged. A plurality of combinations of received signal strengths of the communication devices 10 (communication devices A to D) corresponding to the predetermined object OB may be extracted according to the Gaussian distribution of the received signal strengths of the communication devices A to D). Then, the CPU 31 learns a plurality of combinations of the extracted received signal strengths of the communication devices 10 (communication devices A to D) while associating them with a predetermined object OB (here, the object OB of "a"). may be stored in the data for use.

(5) Flow of Main Processing of Model Learning System of this Embodiment Next, an example of flow of main processing performed by the model learning system of this embodiment will be described with reference to the flowchart of FIG. 10 .

First, the CPU 21 of the terminal device 20 receives a predetermined imaging instruction when any object OB among a plurality of objects OB (for example, the object OB of "a") exists within the imaging range of the imaging unit 28. When input using the unit 27, the imaging unit 28 is caused to perform imaging processing. The data of the image captured by the imaging unit 28 (here, the captured image of the object OB of “a”) is stored in the RAM 23 or the storage device 24, for example. Also, each time the CPU 21 of the terminal device 20 receives a radio signal transmitted from each communication device 10, the received signal strength value of the radio signal detected by the RSSI circuit is stored in the RAM 23 or the storage device 24, for example. . Then, when causing the imaging unit 28 to perform imaging processing, the CPU 21 sends the latest received signal strength value among the received signal strength values stored in the RAM 23 or the storage device 24 to the plurality of communication devices 10, for example. The value of the received signal strength extracted for each time and the captured image data stored in, for example, the RAM 23 or the storage device 24 are transmitted to the identification device 30 via the communication interface unit 29 and the communication network NW.

On the other hand, the CPU 31 of the identification device 30 transmits the captured image data of any object OB and the environment information (here, the received signal strength value corresponding to each communication device 10) at the captured position to the communication interface unit 38. When received (acquired) from the terminal device 20 via (step S100), the data of the received captured image and the value of the received signal strength corresponding to each communication device 10 are stored in the learning data in a state of being associated with each other. do. In addition, when correct data indicating which object OB among a plurality of objects OB was captured is input using the input unit 37 for each captured image stored in the learning data, the CPU 31 receives the input data. The correct answer data is stored in the item of "object" corresponding to the captured image.

Next, the CPU 31 of the identification device 30 identifies which of the plurality of objects OB the imaged object OB is by machine learning using the acquired captured image and environment information as learning data. (step S102). Specifically, for example, when a predetermined model learning instruction is input using the input unit 37, the CPU 31 performs model learning using learning data. Here, the CPU 31 may learn using any one of the three models of mCNN-c, mCNN-w, and mCNN-f described above.

In this way, by machine learning using the captured image and the environment information as learning data, a model used for identifying which of the plurality of objects OB the imaged object OB is is learned. it becomes possible to

(6) Flow of Main Processing of Object Identification System of this Embodiment Next, an example of flow of main processing performed by the object identification system of this embodiment will be described with reference to the flowchart of FIG.

First, when the CPU 31 of the identification device 30 receives (acquires) the data of the captured image transmitted from the terminal device 20 via the communication interface unit 38 (step S200), the received data of the captured image is stored in the RAM 33 or the like. Store in device 34 .

Next, when the CPU 31 of the identification device 30 receives (obtains) the value of the received signal strength corresponding to each communication device 10 transmitted from the terminal device 20 via the communication interface unit 38 (step S202), The captured image data acquired in the process and the received signal intensity value corresponding to each communication device 10 are stored in acquired data in a state of being associated with each other.

Then, when the captured image and the environment information are acquired, the CPU 31 of the identification device 30 acquires the captured image and the environment information acquired in the processing of steps S200 and S202, and the machine using the captured image and the environment information as learning data. Based on the learned model based on the learning, which object OB is the imaged object OB among the plurality of objects OB is identified (step S204).

When the captured image and the environment information are acquired in this way, based on the acquired captured image and the environment information, and the learned model based on machine learning using the captured image and the environment information as learning data, , it becomes possible to identify which object OB the photographed object OB is among a plurality of objects OB.

As described above, according to the object identification system, object identification method, and program of the present embodiment, when a captured image and environment information are acquired, the acquired captured image and environment information and the captured image and environment information are used as learning data. Based on the learned model based on the used machine learning, which object OB is the imaged object OB among a plurality of objects OB is identified, for example, only the captured image of the object OB Instead of identifying the class to which the object OB belongs, it is possible to identify the object OB itself contained in the captured image by further using environmental information at the imaging position of the object OB together with the captured image of the object OB. become. Thereby, the individual recognition performance of the imaged object OB can be improved.

Further, according to the model learning system, the model learning method, and the program of the present embodiment, machine learning using captured images and environment information as learning data causes the imaged object OB to Since it becomes possible to learn a model that is used to identify whether an object is an OB or not, the object OB itself included in the captured image can be identified by using this model.

Modifications of the above-described embodiment will be described below.
(Modification 1)
In the above embodiment, the case where the second acquisition unit 42 acquires the received signal strength when the terminal device 20 receives the radio signal transmitted from each communication device 10 has been described as an example, but this is not the only case. do not have. For example, the second acquisition unit 42 may acquire the received signal strength when each communication device 10 receives the radio signal transmitted from the terminal device 20 . Here, each communication device 10 may be provided with an RSSI circuit that detects the received signal strength when receiving the radio signal transmitted from the terminal device 20 .

In this case, the CPU 21 of the terminal device 20 may request each communication device 10 to transmit the received signal strength value of the radio signal to the terminal device 20 . Then, the CPU 31 of the identification device 30 receives (acquires) the value of the received signal strength of each communication device 10 transmitted from the terminal device 20 via the communication interface unit 38 as a function of the second acquisition means 42, and receives The obtained information may be stored in the acquired data.

Thus, according to the object identification system, the model learning system, the object identification method, the model learning method, and the program according to this modified example, it is possible to exhibit the same effect as the above-described embodiment.

(Modification 2)
In the above embodiment, the second acquisition unit 42 receives the Although the case where the received signal strength when the other party receives the transmitted radio signal is acquired as the environment information has been described as an example, the present invention is not limited to this case. For example, the second acquisition means 42 may acquire temperature, humidity, atmospheric pressure, illuminance, geomagnetism, latitude, longitude, altitude, etc. at the position where any object OB is imaged as environmental information. In this case, the terminal device 20 includes sensors such as a temperature sensor, a humidity sensor, an atmospheric pressure sensor, an illuminance sensor, a geomagnetic sensor, a GPS (Global Positioning System) sensor, etc. that perform measurement when any object OB is imaged. A device may be provided.

The program of the present invention may be stored in a computer-readable storage medium. The storage medium storing this program may be the ROM 22, RAM 23 or storage device 24 of the terminal device 20 shown in FIG. 2, or the ROM 32, RAM 33 or storage device 34 of the identification device 30 shown in may be Alternatively, it may be a CD-ROM or the like that can be read by being inserted into a program reader such as a CD-ROM drive. Further, the storage medium may be magnetic tape, cassette tape, flexible disk, MO/MD/DVD, etc., or semiconductor memory.

The embodiments and modifications described above are described to facilitate understanding of the present invention, and are not described to limit the present invention. Therefore, each element disclosed in the above embodiments and modifications is meant to include all design changes and equivalents that fall within the technical scope of the present invention.

For example, in the above-described embodiment, the case of performing data augmentation of environmental information has been described as an example, but data augmentation of captured images may be performed. In this case, the data amount of the image included in the learning data may be increased by performing rotation, clipping, horizontal reversal, or the like on the captured image.

In the above-described embodiment, the identification device 30 is configured to realize the functions of the first acquisition means 41, the second acquisition means 42, the identification means 43, the acquisition means 44, and the learning means 45. Not limited. For example, a learning device 50 (shown in FIG. 12) configured by a computer or the like (for example, a general-purpose personal computer, a server computer, or the like) communicably connected to the identification device 30 via a communication network such as the Internet or a LAN A learning device 50 may be provided for learning a model used to identify which of the plurality of objects OB the imaged object OB is. In this case, the identification device 30 and the learning device 50 can have substantially the same hardware configuration. can be realized by For example, each function of the functional block diagram shown in FIG. 4 may be arbitrarily shared between the identification device 30 and the learning device 50 as shown in FIGS. 12(a) and 12(b).

The object identification system, the model learning system, the object identification method, the model learning method, and the program of the present invention as described above can identify the object itself included in the captured image, and are suitable for use in, for example, an image recognition system. Since it can be used, its industrial applicability is extremely large.

DESCRIPTION OF SYMBOLS 10...Communication device 20...Terminal device 30...Identification device 41...First acquisition means 42...Second acquisition means 43...Identification means 44...Acquisition means 45...Learning means 50...Learning device OB...Object S...Space

Claims (7)

a first obtaining means for obtaining a captured image of any one of a plurality of objects present in a predetermined space;
a second acquiring means for acquiring environment information at a position where any one of the objects is imaged;
When the captured image and the environment information are acquired, based on the acquired captured image and the environment information, and a trained model based on machine learning using the captured image and the environment information as learning data an identification means for identifying which of the plurality of objects the photographed object is;
with
The second acquisition means acquires wireless signals transmitted from one of communication devices provided at a plurality of positions within the predetermined space and a terminal device present at a position where the object is imaged. An object identification system that acquires a received signal strength when received by the other as the environment information . The environment information used as the learning data is the predetermined object among the plurality of objects, assuming that the received signal strength corresponding to each communication device follows a Gaussian distribution that differs for each captured object and for each communication device. 2. The object identification system of claim 1 , comprising a plurality of combinations of received signal strengths of each communication device sampled according to a Gaussian distribution of received signal strengths of each communication device corresponding to . Acquisition means for acquiring a captured image of any one of a plurality of objects existing in a predetermined space and environmental information at the position where the one of the objects was captured;
learning means for learning a model used for identifying which of the plurality of objects the imaged object is by machine learning using the acquired captured image and the environment information as learning data; and,
with
The acquisition means acquires a radio signal transmitted from one of communication devices provided at a plurality of positions within the predetermined space and a terminal device present at a position where an image of one of the objects is captured, from the other. A model learning system that acquires received signal strength at the time of reception as the environment information . to the computer,
acquiring a captured image of any one of a plurality of objects existing within a predetermined space;
a step of acquiring environmental information at a position where any one of the objects is imaged;
When the captured image and the environment information are acquired, based on the acquired captured image and the environment information, and a trained model based on machine learning using the captured image and the environment information as learning data , identifying which of the plurality of objects the imaged object is;
to execute each step of
The step of acquiring the environment information includes wirelessly transmitted from one of a communication device provided at a plurality of positions in the predetermined space and a terminal device present at a position where the one of the objects is captured. A method for identifying an object , comprising the step of obtaining, as the environment information, a received signal strength when the signal is received by the other party . to the computer,
acquiring a captured image of any one of a plurality of objects present in a predetermined space and environmental information at the position where the one of the objects was captured;
a step of learning a model that is used to identify which of the plurality of objects the imaged object is, using the captured image and the environment information that have been acquired as learning data;
to execute each step of
The step of acquiring the captured image and the environment information includes communication devices provided at a plurality of positions within the predetermined space, or terminal devices present at positions where the object is captured. a model learning method comprising the step of acquiring, as the environment information, a received signal strength when the other receives a radio signal transmitted from the other . to the computer,
a function of acquiring a captured image of any one of a plurality of objects existing within a predetermined space;
a function of acquiring environmental information at a position where any of the objects is imaged;
When the captured image and the environment information are acquired, based on the acquired captured image and the environment information, and a trained model based on machine learning using the captured image and the environment information as learning data , a function of identifying which of the plurality of objects the imaged object is;
A program for realizing
The function of acquiring the environmental information is performed by radio transmission from either one of communication devices provided at a plurality of positions within the predetermined space and a terminal device present at a position where the object is imaged. A program including a function of acquiring a received signal strength when the other party receives the signal as the environment information . to the computer,
a function of obtaining a captured image of any one of a plurality of objects present in a predetermined space and environmental information at the position where the one of the objects was captured;
a function of learning a model that is used to identify which of the plurality of objects the imaged object is, using the captured image and the environment information that have been acquired as learning data;
A program for realizing
The function of acquiring the captured image and the environment information is provided by either communication devices provided at a plurality of positions within the predetermined space, or a terminal device present at a position where any one of the objects is captured. A program including a function of acquiring, as the environment information, a received signal strength when the other receives a radio signal transmitted from the other .

Images