WO2021014781A1

WO2021014781A1 - Counting method, computer program, and counting system

Info

Publication number: WO2021014781A1
Application number: PCT/JP2020/022056
Authority: WO
Inventors: 一喜鳥塚
Original assignee: 一喜鳥塚
Priority date: 2019-07-23
Filing date: 2020-06-04
Publication date: 2021-01-28
Also published as: JP2021018747A; JP6598182B1

Abstract

[Problem] To provide a counting method, computer program, and counting system. [Solution] The present invention is provided with: a first computation unit which, on the basis of measurement data of a load sensor which measures loads of objects mounted in a specific region, computes a total weight of the objects; an identification unit which identifies the types of the objects mounted in the specific region; and a second computation unit which computes the number of the objects mounted in the specific region on the basis of the types of the objects identified by the identification unit, the weight per unit quantity of the objects, and the total weight of the objects computed by the first computation unit.

Description

Counting methods, computer programs, and counting systems

The present invention relates to a counting method, a computer program, and a counting system.

Conventionally, in stores selling foods and daily necessities, a management system using a non-contact data carrier (RFID: Radio Frequency Identification) has been adopted when managing information on items placed on shelves (for example). , Patent Document 1).

Such a management system using RFID is a system that can acquire article information such as expiration date, production area, and inventory quantity by reading the RFID attached to the article. Therefore, it is effective in that the work efficiency when managing the article information can be improved. Further, since the management system using RFID can read the article information from the RFID without touching the article, it is also effective from the viewpoint of the reliability and safety of the article.

Japanese Unexamined Patent Publication No. 2008-26979

However, in the case of a management system using RFID, since the article information can be obtained only by reading the RFID attached to the article, it is necessary to attach the RFID tag to each article. Further, when managing the article information, the work of reading the RFID tag attached to each article one by one using the reading terminal is required. As described above, even a management system using RFID has a problem that it takes a lot of time and effort.

An object of the present invention is to provide a counting method, a computer program, and a counting system capable of counting articles according to their types.

The counting method according to one aspect of the present invention includes a step in which the first calculation unit calculates the total weight of the article based on the measurement data of the load sensor that measures the load of the article placed in the specific area. The specific part specifies the type of the article placed in the specific area, and the second calculation unit determines the type of the article specified by the specific part, the weight per unit number of the article, and the first. It includes a step of calculating the number of articles placed in the specific area based on the total weight of the articles calculated by the calculation unit.

The computer program according to one aspect of the present invention is a first calculation unit that calculates the total weight of the article based on the measurement data of the load sensor that measures the load of the article placed on the specific area of the computer. To the specific part that specifies the type of the article placed in the specific area, the type of the article specified by the specific part, the weight per unit number of the article, and the total weight of the article calculated by the first calculation unit. Based on this, it functions as a second calculation unit that calculates the number of articles placed in the specific area.

The counting system according to one aspect of the present invention includes a first calculation unit that calculates the total weight of the article based on the measurement data of the load sensor that measures the load of the article placed in the specific area, and the specific area. Based on the specific part that specifies the type of the article placed in, the type of the article specified by the specific part, the weight per unit number of the article, and the total weight of the article calculated by the first calculation unit. A second calculation unit for calculating the number of articles placed in the specific area is provided.

According to the present application, articles can be counted according to the type.

It is a schematic diagram explaining the structure of the count system which concerns on this embodiment. It is a block diagram which shows the internal structure of the arithmetic unit. It is a conceptual diagram which shows an example of a weight table. It is a schematic diagram which shows the structural example of the learning model. It is a conceptual diagram which shows an example of the load distribution data obtained from a load sensor. It is a schematic diagram which shows the structural example of the learning model. It is a flowchart explaining the generation procedure of a learning model. It is a flowchart explaining the procedure of counting the number of articles.

Hereinafter, the present invention will be specifically described with reference to the drawings showing the embodiments thereof.
FIG. 1 is a schematic diagram illustrating a configuration of a counting system according to the present embodiment. The counting system according to the present embodiment is a system for counting the number of articles placed on the article shelves S for each type, and the load sensors 130 installed on the shelf boards S1 and S2 of the article shelves S, Various calculations are performed based on the data obtained from the image pickup device 140 that images the article group placed on the article shelf S, the load sensor 130, and the image pickup device 140, and the number of articles included in the article group is calculated for each type. The computing device 10 is provided. In FIG. 1, the articles indicated by reference numerals O1 to O4 indicate that they are different types of articles.

The goods shelf S is, for example, a product shelf installed in a store for displaying various products. The article shelf S includes a plurality of shelf boards S1 and S2 that are horizontally supported by columns. In the present embodiment, the article shelf S is configured to include two shelf boards S1 and S2, but may be configured to include three or more shelf boards. Further, although the article shelf S shown in FIG. 1 has the front surface, the back surface, and the side surface open, at least one of the back surface and the side surface may be closed.

The articles O1 to O4 placed on the article shelf S are arbitrary, but have a weight determined for each type. The weights of the articles O1 to O4 may be different for each type, and may be the same regardless of the type. FIG. 1 shows a state in which four types of articles O1 to O4 are placed on the article shelf S, but the types of articles placed on the article shelf S may be three or less. It may be more than one kind.

The load sensor 130 is a sheet-shaped pressure-sensitive sensor and outputs measurement data (load distribution data) of the load distribution. The load sensor 130 has, for example, a rectangular measurement area (specific area), and includes a plurality of pressure-sensitive elements arranged in a grid pattern in the measurement area. The load sensor 130 outputs a measured value (pressure value) at the position where each pressure sensitive element is arranged as load distribution data. That is, the load distribution data indicates matrix data composed of position information in the measurement region and measurement values measured by each pressure sensitive element. Alternatively, the load distribution data may be display data in which the magnitude of the measured value measured by each pressure sensitive element is represented by shading (or color).

In the measurement area of the load sensor 130, a separator SP for classifying articles O1 to O4 by type is placed together with articles O1 to O4. In other words, the measurement area of the load sensor 130 is divided into a plurality of divided areas by the separator SP, and articles O1 to O4 are placed in each divided area for each type. The material, weight, and shape of the separator SP can be arbitrarily designed. The separator SP is placed at an appropriate place in the measurement area of the load sensor 130, and does not need to be fixed at a fixed place. Further, the number of separator SPs mounted in the measurement area of the load sensor 130 does not have to be one, and may be two or more.

In the example of FIG. 1, the measurement area of the load sensor 130 installed on the upper shelf plate S1 is divided into two division areas R1 and R2 by one separator SP. Of the two divided areas R1 and R2, the article O1 is placed in the divided area R1 on the left side when viewed from the front, and a plurality of articles O2 are placed in the divided area R2 on the right side. Similarly, the measurement area of the load sensor 130 installed on the lower shelf plate S2 is divided into two division areas R3 and R4 by one separator SP. Of these two divided areas R3 and R4, the article O3 is placed in the divided area R3 on the left side when viewed from the front, and the article O4 is placed in the divided area R4 on the right side. In this way, articles O1 to O4 are placed in each of the divided regions R1 to R4 for each type, and a plurality of types of articles are not mixed and placed.

The image pickup device 140 is a digital camera or a digital video camera, and outputs image data obtained by imaging articles O1 to O4. The image pickup apparatus 140 is installed in a place where the types of articles O1 to O4 can be specified. For example, when another article shelf (not shown) is installed on the side facing the article shelf S, the image pickup apparatus 140 may be installed on another shelf facing the article shelf S.

The arithmetic unit 10 is a dedicated or general-purpose computer, and the load distribution data output from the load sensor 130 and the image data output from the image pickup device 140 are input. The arithmetic unit 10 executes arithmetic processing based on the input load distribution data and image data, and calculates the number of articles O1 to O4 for each type.

FIG. 2 is a block diagram showing the internal configuration of the arithmetic unit 10. The arithmetic unit 10 includes an arithmetic unit 11, a storage unit 12, a first connection unit 13, a second connection unit 14, a communication unit 15, an operation unit 16, and a display unit 17.

The calculation unit 11 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The ROM included in the arithmetic unit 11 stores a control program or the like that controls the operation of each hardware unit included in the arithmetic unit 10. The CPU in the arithmetic unit 11 executes the control program stored in the ROM and various computer programs stored in the storage unit 12 described later, and controls the operation of each hardware unit to control the operation of each hardware unit, thereby controlling the load sensor 130 and the imaging device 140. A function of executing arithmetic processing based on the data obtained from the above and calculating the number of articles O1 to O4 placed on the article shelf S for each type is realized. Data used during execution of the calculation is temporarily stored in the RAM included in the calculation unit 11.

The calculation unit 11 is configured to include a CPU, a ROM, and a RAM. Alternatively, the arithmetic unit 11 is one or a plurality of arithmetic circuits including a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a quantum processor, a volatile or non-volatile memory, and the like. It may be. In addition, the calculation unit 11 may have functions such as a clock for outputting date and time information, a timer for measuring the elapsed time from giving the measurement start instruction to giving the measurement end instruction, and a counter for counting the number.

The storage unit 12 includes storage devices such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), and an EEPROM (Electronically Erasable Programmable Read Only Memory). The storage unit 12 stores a computer program and various data executed by the calculation unit 11.

The computer program stored in the storage unit 12 calculates the number of articles O1 to O4 placed on the article shelf S for each type based on the data obtained from the load sensor 130 and the image pickup apparatus 140. Includes an arithmetic processing program 200 for causing the computer to execute.

The computer program stored in the storage unit 12 may be provided by a non-temporary recording medium M in which the computer program is readablely recorded. The recording medium M is, for example, a portable memory such as a CD-ROM, a USB memory, an SD (Secure Digital) card, a micro SD card, or a compact flash (registered trademark). The computer program recorded on the recording medium M is read by a reading device (not shown) and stored in the storage unit 12.

The data stored in the storage unit 12 includes a weight table 201 stored in association with the type of each article and the weight of each article. FIG. 3 is a conceptual diagram showing an example of the weight table 201. In the weight table 201, the type of each article and the weight of each article are stored in association with each other. In the weight table 201 illustrated in FIG. 3, the weights of each of the articles of types A, B, C, D, ..., X are Wagram, Wbgram, Wcgram, Wdgram, ..., Wxgram, respectively. It is shown that. Such a table is generated by receiving information on the type of an article and the weight of each article through, for example, an operation unit 16. When the type of the article is specified, the calculation unit 11 can grasp the weight of each article by referring to the weight table 201. Needless to say, the weights Wagram, Wbgram, Wcgram, Wdgram, ..., Wxgram stored in the weight table 201 are not limited to the weight per article, but are units including the case of one article. It may be the weight per piece (1 piece or more). In this case, the weight per unit number (1 piece or more) and the information that can be converted into the weight per piece of the article are stored in the weight table 201. Therefore, it may be possible to convert it into the weight of each article.

Further, the storage unit 12 may store the learning model 210 used for the process of specifying the types of the articles O1 to O4. The learning model 210 is configured to output data relating to the types of articles O1 to O4 in response to input of image data obtained from the image pickup apparatus 140. The structure of the learning model 210 is defined by the definition information. The definition information includes configuration information that defines the configuration of the learning model 210 (number of layers, number of nodes, etc.), and various parameters such as weights and biases between nodes referred to in the calculation using the learning model 210.

Further, the storage unit 12 may store the learning model 220 used for the process of detecting the position of the separator SP on the load sensor 130. The learning model 220 is configured to output data regarding the position of the separator SP in response to the input of the load distribution data obtained from the load sensor 130. The configuration of the learning model 220 is defined by the definition information. The definition information includes configuration information that defines the configuration of the learning model 220 (number of layers, number of nodes, etc.), and various parameters such as weights and biases between nodes referred to in the calculation using the learning model 220.

The learning

models

210 and 220 may be generated by the arithmetic unit 10. Further, the learning

models

210 and 220 may be generated by an external server device (not shown). In the latter case, the arithmetic unit 10 may communicate with the external server device and download the learning

models

210 and 220 by communication. Further, the learning

models

210 and 220 may be provided by the recording medium M as well as various computer programs.

The first connection unit 13 includes a connection interface for connecting the load sensor 130. The load sensor 130 is a sheet-shaped pressure-sensitive sensor, and outputs load distribution data (load distribution data) measured in the measurement area. The load distribution data includes information on measured values measured corresponding to the positions of the pressure-sensitive elements included in the load sensor 130. The load distribution data in a state where the articles O1 to O4 and the separator SP are placed in the measurement area of the load sensor 130 is input to the first connection portion 13. The input load distribution data may be stored in the storage unit 12 via the calculation unit 11.

The second connection unit 14 includes a connection interface for connecting the image pickup device 140. The image pickup apparatus 140 is a digital camera or a digital video camera, and outputs image data obtained by imaging an image pickup target. In the image data, for example, each pixel is represented by an RGB gradation value. Image data obtained by imaging the articles O1 to O4 placed on the article shelf S is input to the second connection portion 14. The input image data may be stored in the storage unit 12 via the calculation unit 11.

The communication unit 15 includes a communication interface for transmitting and receiving various data. The communication interface included in the communication unit 15 is, for example, a communication interface conforming to the communication standard of LAN (Local Area Network) used in WiFi (registered trademark) and Ethernet (registered trademark). When the data to be transmitted is input from the calculation unit 11, the communication unit 15 transmits the data to be transmitted to the designated destination. Further, when the communication unit 15 receives the data transmitted from the external device, the communication unit 15 outputs the received data to the calculation unit 11.

The operation unit 16 is provided with input interfaces such as various operation buttons, switches, and a touch panel, and receives various operation information and setting information. The calculation unit 11 performs an appropriate calculation based on the operation information input from the operation unit 16, and stores the calculation result in the storage unit 12 as needed.

The display unit 17 is provided with a display panel such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel, and displays information to be notified to an administrator or the like. At this time, the calculation unit 11 generates display data including information to be notified to the administrator and the like, and the generated display data is output to the display unit 17, so that various information is displayed.

In the present embodiment, the arithmetic unit 10 is configured to include the operation unit 16 and the display unit 17, but the operation unit 16 and the display unit 17 are not indispensable configurations and are computers connected to the outside of the arithmetic unit 10. The operation may be received through the device and the information to be notified to the administrator or the like may be output to an external computer.

The arithmetic unit 10 uses the learning model 210 to specify the type of the article placed on the article shelf S. Hereinafter, the learning model 210 used when specifying the type of the article will be described.

FIG. 4 is a schematic diagram showing a configuration example of the learning model 210. The learning model 210 is, for example, a learning model based on CNN (Convolutional Neural Networks), and includes an input layer 211, an intermediate layer 212, and an output layer 213. When an image (captured image) captured by the image pickup device 140 is input, the learning model 210 is learned in advance so as to output data relating to the type of the article.

The captured image captured by the imaging device 140 is input to the input layer 211. The captured image input to the input layer 211 is sent to the intermediate layer 212.

The intermediate layer 212 is composed of, for example, a convolution layer 212a, a pooling layer 212b, and a fully connected layer 212c. A plurality of convolution layers 212a and pooling layers 212b may be provided alternately. The convolution layer 212a and the pooling layer 212b extract the features of the captured image input through the input layer 211 by the calculation using the nodes of each layer. The fully connected layer 212c combines the data from which the feature portion is extracted by the convolution layer 212a and the pooling layer 212b into one node, and outputs the feature variable converted by the activation function. The feature variable is output to the output layer 213 through the fully connected layer 212c.

The output layer 213 includes one or more nodes. The output layer 213 is converted into a probability using a softmax function based on the feature variable input from the fully connected layer 212c of the intermediate layer 212, and indicates which type the article contained in the captured image corresponds to. Output the probability from each node.

For example, the output layer 213 is composed of n nodes (n is an integer of 1 or more) from the first node to the nth node, and the type of the article placed in the division areas R1 to R4 is different from the first node. Probability P1 of "type A", "type B", "type C", and "type D" is output, and the types of articles placed in the division areas R1 to R4 are "types" from the second node, respectively. Probability P2 of "B", "Type C", "Type D", and "Type A" is output, and ..., From the nth node, the types of articles placed in the division areas R1 to R4 are "Type X", respectively. , "Type X", "Type X", "Type X" Probability Pn may be output. The number of nodes constituting the output layer 213 and the calculation result assigned to each node are not limited to the above examples, and can be appropriately designed.

The arithmetic unit 11 of the arithmetic unit 10 can specify the type of the article placed in each area by referring to the arithmetic result obtained from the learning model 210 and, for example, selecting the combination of the types having the highest probability. It is possible.

In the present embodiment, the captured image obtained from the imaging device 140 is input to the input layer 211 of the learning model 210, but the captured image processed in advance is input to the input layer 211 of the learning model 210. May be. For example, the calculation unit 11 masks the other regions of the captured image so as to include only the articles placed in the region where the number of articles is to be counted, and uses the masked captured image as the learning model 210. It may be input to the input layer 211. In this case, the learning model 210 may be trained to output data relating to one type of article.

Although the learning model 210 by CNN is shown in the example of FIG. 3, the machine learning model for constructing the learning model 210 can be arbitrarily set. For example, instead of CNN, a learning model based on R-CNN (Region-based CNN), YOLO (You Only Look Once), SSD (Single Shot Detector), or the like may be set.

The arithmetic unit 10 inputs the load distribution data acquired from the load sensor 130 into the learning model 220, and acquires the calculation result from the learning model 220 to determine the position of the separator SP placed in the measurement area of the load sensor 130. To detect. Hereinafter, the load distribution data obtained from the load sensor 130 will be described, and then the configuration of the learning model 220 will be described.

FIG. 5 is a conceptual diagram showing an example of load distribution data obtained from the load sensor 130. The load sensor 130 includes a plurality of pressure-sensitive elements arranged in a grid pattern, and outputs a measured value (pressure value) corresponding to the position of each pressure-sensitive element. When the load sensor 130 includes m pressure sensors in the horizontal direction (for example, the width direction of the shelf board S1) and n pressure sensors in the vertical direction (for example, the depth direction of the shelf board S1), the positions of the pressure sensitive elements are (X1, It is represented by position coordinates such as Y1), (X2, Y1), ..., (Xm, Y1), (X1, Y2,) ..., (Xm, Yn). Here, m and n are integers of 1 or more. The load sensor 130 outputs a measured value (pressure value) corresponding to each position coordinate. In the example of FIG. 5, a region having a low pressure value (light weight) is indicated by a light shade region, and a region having a high pressure value (heavy weight) is indicated by a dark shade region.

When the separator SP is placed in the measurement area, the load sensor 130 outputs a measured value (shade) according to the weight of the separator SP. The example of FIG. 5 shows how the separator SP is placed in the vertical direction near the center of the measurement area. One of the divided regions divided by the separator SP is a divided region R1, and the other is a divided region R2. When some article is placed in the divided areas R1 and R2, the load sensor 130 outputs a measured value (shade) according to the weight of the article. The example of FIG. 5 shows a state in which a plurality of articles O1 are placed in the division area R1 on the left side of the separator SP, and a plurality of articles O2 are placed in the division area R2 on the right side of the separator SP. However, in the example of FIG. 5, although the symbols of the article O1 and the article O2 are specified for the sake of explanation, the type of the article cannot be specified only from the measured value of the load sensor 130. The type and number of articles will be specified by the processing of the arithmetic unit 10 described later.

FIG. 6 is a schematic diagram showing a configuration example of the learning model 220. The learning model 220 is, for example, a learning model based on CNN, and includes an input layer 221, an intermediate layer 222, and an output layer 223. The learning model 210 is trained in advance so as to output data regarding the position of the separator SP when the load distribution data measured by the load sensor 130 is input.

The load distribution data measured by the load sensor 130 is input to the input layer 221. The load distribution data input to the input layer 221 is sent to the intermediate layer 222.

The intermediate layer 222 is composed of, for example, a convolution layer 222a, a pooling layer 222b, and a fully connected layer 222c. A plurality of convolution layers 222a and pooling layers 222b may be provided alternately. The convolution layer 222a and the pooling layer 222b extract the characteristics of the load distribution data input through the input layer 221 by the calculation using the nodes of each layer. The fully connected layer 222c combines the data whose feature portions are extracted by the convolution layer 222a and the pooling layer 222b into one node, and outputs the feature variable converted by the activation function. The feature variable is output to the output layer 223 through the fully connected layer 222c.

The output layer 223 includes one or more nodes. The output layer 223 converts the characteristic variables input from the fully connected layer 222c of the intermediate layer 222 into probabilities using a softmax function, and outputs data regarding the position of the separator SP from each node. For example, when the load sensor 130 includes m pressure sensors in the horizontal direction and n pressure sensors in the vertical direction, each node of the output layer 223 has the position coordinates (X1, Y1), (X2, Y1) of each pressure sensor. ), ..., (Xi, Yj), ..., (Xm, Yn), the probability corresponding to the separator SP may be output. For example, the output layer 223 is composed of n × m nodes (n and m are integers of 1 or more) from the first node to the n × m node, and the position coordinates (X1, Y1) from the first node. The probability P _{11 in} which the position corresponding to the position corresponds to the separator SP is output, and the probability P _{21 in} which the position corresponding to the position coordinates (X1, Y1) corresponds to the separator SP is output from the second node. From the mn node, the probability P _mn that the position corresponding to the position coordinates (Xm, Yn) corresponds to the separator SP may be output. The number of nodes constituting the output layer 223 and the calculation result assigned to each node are not limited to the above examples, and can be appropriately designed.

The arithmetic unit 11 of the arithmetic unit 10 refers to the arithmetic result obtained from the learning model 220 and selects the position coordinates whose probability is equal to or greater than the threshold value (for example, the probability of corresponding to the separator SP is 90% or more). It is possible to identify the position of.

The arithmetic unit 10 executes a process of generating

learning models

210 and 220 in the learning phase. Hereinafter, the procedure for generating the learning model 210 in the arithmetic unit 10 will be described.

FIG. 7 is a flowchart illustrating the procedure for generating the learning model 210. The arithmetic unit 10 collects image data of the article to be counted and label data indicating the type of the article as a preparatory step for generating the learning model 210, and stores the collected data in the storage unit 12 as teacher data. .. By collecting a sufficient number of image data and label data in this preparatory stage, the estimation accuracy of the type can be improved.

The calculation unit 11 accesses the storage unit 12 and acquires the teacher data used for generating the learning model 210 (step S101). The teacher data includes image data of the article and label data indicating the type of the article included in the image data. In step S101, one set of image data and label data may be acquired out of a large number of image data and label data included as teacher data. At the initial stage of generating the learning model 210, the teacher data prepared by the administrator of the arithmetic unit 10 or the like is set. Further, if the learning progresses, the estimation result by the learning model 210 and the captured image used for the estimation process may be acquired, and the acquired data may be set as teacher data.

Next, the calculation unit 11 inputs the image data included as the teacher data into the learning model 210 (step S102), and acquires the calculation result from the learning model 210 (step S103). At the stage before the start of learning, it is assumed that the definition information describing the learning model 210 is given an initial setting value. In the learning model 210, as described above, a predetermined operation is performed between the nodes constituting each layer.

Next, the calculation unit 11 evaluates the calculation result obtained in step S103 (step S104), and determines whether or not the learning is completed (step S105). Specifically, the calculation unit 11 can evaluate the calculation result by using an error function (also referred to as an objective function, a loss function, or a cost function) based on the calculation result obtained in step S103 and the teacher data. When the error function becomes less than or equal to the threshold value (or more than the threshold value) in the process of optimizing (minimizing or maximizing) the error function by the gradient descent method such as the steepest descent method, the calculation unit 11 states that the learning is completed. to decide. In order to avoid the problem of overfitting, techniques such as cross-validation and early stopping may be adopted to end learning at an appropriate timing.

When it is determined that the learning is not completed (S105: NO), the calculation unit 11 updates the weights and biases between the nodes of the learning model 210 (step S106), and returns the process to step S101. The arithmetic unit 11 may update the weights and biases between the nodes by using the error back propagation method in which the weights and biases between the nodes are sequentially updated from the output layer 213 of the learning model 210 toward the input layer 211. it can.

When it is determined that the learning is completed (S105: YES), the calculation unit 11 stores the learned learning model 210 in the storage unit 12 (step S107), and ends the process according to this flowchart.

In this way, the arithmetic unit 10 can generate the learning model 210 by collecting the image data of the article to be counted and the label data indicating the type of the article and using the collected data as the teacher data. it can. Further, the arithmetic unit 10 can generate the learning model 220 for detecting the position of the separator SP by the same processing procedure as that shown in FIG. 7. In this case, the arithmetic unit 10 collects the load distribution data obtained from the load sensor 130 and the label data indicating the position of the separator SP in the measurement area, and uses the collected data as the teacher data to perform the learning model 220. It should be generated. In the present embodiment, the procedure for generating the learning

models

210 and 220 in the arithmetic unit 10 has been described, but the learning model is also generated in the case where the learning

models

210 and 220 are generated in the external server device by the same processing procedure. 210 and 220 may be generated.

The arithmetic unit 10 can execute the counting process of the number of articles at an appropriate timing in the operation phase after the

learning models

210 and 220 are generated. The counting process of the number of articles may be executed when an execution instruction is given by an administrator or the like, or may be executed at a regular timing. Hereinafter, the counting process of the number of articles by the arithmetic unit 10 will be described.

FIG. 8 is a flowchart illustrating a procedure for counting the number of articles. The arithmetic unit 11 of the arithmetic unit 10 acquires the load distribution data output from the load sensor 130 through the first connection unit 13 (step S121). The load distribution data shows matrix data composed of position information in the measurement region and measurement values measured by each pressure sensitive element.

The calculation unit 11 inputs the acquired load distribution data to the learning model 220 for detecting the position of the separator SP (step S122). The calculation unit 11 gives the acquired load distribution data to the input layer 221 of the learning model 220 to execute the calculation by the learning model 220. The load distribution data given to the input layer 221 of the learning model 220 is sent to the intermediate layer 222. In the intermediate layer 222, an operation using an activation function including weights and biases between nodes is executed. The features of the load distribution data are extracted from the convolution layer 222a and the pooling layer 222b of the intermediate layer 222. The data of the feature portion extracted by the convolution layer 222a and the pooling layer 222b is combined with each node constituting the fully connected layer 222c and converted into a feature variable by the activation function. The converted feature variable is output to the output layer 223 through the fully connected layer 222c. The output layer 223 is converted into a probability by using a softmax function based on the feature variable input from the fully connected layer 222c of the intermediate layer 222, and the probability corresponding to the position coordinates of the pressure sensitive element corresponds to the separator SP. Is output from each node.

The calculation unit 11 acquires the calculation result from the learning model 220, and specifies the division area based on the obtained calculation result (step S123). The calculation unit 11 can detect the position of the separator SP placed in the measurement area based on the calculation result obtained from the learning model 220. The calculation unit 11 may specify the division area based on the detected position coordinates of the separator SP. For example, as shown in FIG. 5, when the separator SP extends in the vertical direction (depth direction of the shelf board S1), the left side of the separator SP (the region where the X coordinate is smaller than the X coordinate of the separator SP) is one divided region. It can be specified as (division area R1), and the right side of the separator SP (the area whose X coordinate is larger than the X coordinate of the separator SP) can be specified as one division area (division area R2).

Next, the calculation unit 11 derives the total weight in the divided region (step S124). The calculation unit 11 can derive the total weight in the divided region by obtaining the sum of the measured values in the divided region among the measured values from the load sensor 130. The calculation unit 11 may derive the total weight for each of the divided regions.

Next, the calculation unit 11 acquires the image data output from the image pickup apparatus 140 through the second connection unit 14 (step S125). The image data is, for example, data in which each pixel is represented by RGB gradation values.

The calculation unit 11 inputs the acquired image data to the learning model 210 for specifying the type of the article (step S126). The calculation unit 11 gives the acquired image data to the input layer 211 of the learning model 210 to execute the calculation by the learning model 210. The image data given to the input layer 211 of the learning model 210 is sent to the intermediate layer 212. In the intermediate layer 212, an operation using an activation function including weights and biases between nodes is executed. Image features are extracted from the convolutional layer 212a and the pooling layer 212b of the intermediate layer 212. The data of the feature portion extracted by the convolution layer 212a and the pooling layer 212b is combined with each node constituting the fully connected layer 212c and converted into a feature variable by the activation function. The converted feature variable is output to the output layer 213 through the fully connected layer 212c. The output layer 213 is converted into a probability using a softmax function based on the feature variable input from the fully connected layer 212c of the intermediate layer 212, and the type of the article contained in each division region is determined. The indicated probability is output from each node.

The calculation unit 11 acquires a calculation result from the learning model 210, and specifies the type of the article based on the obtained calculation result (step S127). The calculation unit 11 can specify the type of the article placed in each division region, for example, by selecting the combination of articles having the highest probability based on the calculation result obtained from the learning model 210.

Next, the calculation unit 11 reads out the weight per article from the weight table 201 with respect to the article whose type is specified in step S127 (step S128). For example, when the type specified in step S127 is "type A", the calculation unit 11 refers to the weight table 201 and sets the weight "Wa" per article stored in association with "type A". You just have to read it. The same applies when other types are specified. In step S128, the weight per article corresponding to the type may be read out in each of the divided regions.

Next, the calculation unit 11 calculates the number of articles based on the total weight in the divided area derived in step S124 and the weight per article placed in the divided area specified in step S128. (Step S129). For example, when the total weight in the divided region derived in step S124 is "WA" and the weight per piece specified in step S128 is "Wa", the calculation unit 11 calculates WA / Wa. Calculate the number in the division area. In step S129, the number of articles may be calculated in each of the divided regions.

The calculation unit 11 stores the calculated number of articles in the storage unit 12. Further, the calculation unit 11 may display the calculated number of articles on the display unit 17, or may notify the terminal device used by the administrator or the like through the communication unit 15.

As described above, in the present embodiment, the type of the article in the divided areas R1 to R4 divided by the separator SP is specified by using the learning model 210, and the number of articles in each divided area R1 to R4 is calculated. Can be done.

In the present embodiment, the divided region is specified based on the load distribution data from the load sensor 130, and then the type is specified based on the image data from the image pickup device 140. However, the type is specified first. After that, the procedure may be to specify the division area.

Further, in the present embodiment, the learning model 210 is used to specify the type of the article placed in each divided region, but the type of the article is specified in the captured image and the type is specified. The type of the article in each division region may be specified based on the position in the image of the article and the position information of the separator SP detected by using the learning model 220. In the latter case, when the image data output from the image pickup apparatus 140 is input, it is preferable to use a learning model trained to output data relating to the type of the article specified in the image. As the learning procedure of the learning model in this case, the same procedure as the learning procedure in the learning model 210 can be used.

Further, in the present embodiment, the divided region is specified based on the load distribution data from the load sensor 130, but the divided region may be specified based on the image data obtained from the image pickup apparatus 140. In the latter case, when the image data output from the image pickup apparatus 140 is input, it is preferable to use a learning model trained to output the data regarding the position of the separator SP. As the learning procedure of the learning model in this case, the same procedure as the learning procedure in the learning model 220 can be used.
Further, when adopting a configuration in which the divided region can be specified without being based on the load distribution data from the load sensor 130, it is not necessary to use the distribution data (load distribution data) as the measurement data of the load sensor 130, and the divided region It suffices to use the data that can calculate the total weight of the articles placed on the load sensor 130.

Further, in the present embodiment, the learning model 220 is used to detect the position of the separator SP, but a position detection sensor that detects the position of the separator SP may be used. As the position detection sensor, an optical sensor, a magnetic sensor, or the like can be used. When an optical sensor is used, a separator is used by measuring the light receiving intensity of the light receiving element by using a light emitting element that emits light toward the separator SP and a light receiving element that receives light reflected by the separator SP. The position of the SP can be detected. When a magnetic sensor is used, the position of the separator SP can be detected by attaching a magnet to the separator SP and detecting the position of the magnet with the magnetic sensor.

Further, in the present embodiment, the learning

models

210 and 220 are generated in the learning phase, but re-learning may be executed at any timing after the start of operation. In this case, the calculation unit 11 may display the estimation results of the learning

models

210 and 220 on the display unit 17 and accept the selection of whether or not the estimation results by the learning

models

210 and 220 are correct. The calculation unit 11 can relearn the learning model 210 by using the image data used when the estimation process is performed and the label data indicating whether or not the estimation result of the type is correct as the teacher data. Further, the calculation unit 11 re-uses the measurement distribution data used when the estimation process is performed and the label data indicating whether or not the estimation result of the position of the separator SP is correct as the teacher data to re-learn the learning model 220. You can learn. Since the re-learning procedure is exactly the same as the learning procedure shown in FIG. 7, the description thereof will be omitted.

[Modified example of the embodiment]
In the above embodiment, the image data obtained by imaging the article group is acquired, and the type of the article included in the article group is specified based on the acquired image data, but the article is not necessarily from the image data. There is no need to specify the type. Hereinafter, other methods for specifying the type of the article will be described. The same reference numerals are given to the same configurations as those described in the above embodiment, and duplicate description will be omitted.

For example, the area to be placed on the load sensor 130 (corresponding to the divided area of the above embodiment) is determined in advance for each type of article. Then, an identifier (hereinafter, also referred to as an ID) is given to each pressure-sensitive element arranged in a grid pattern in the measurement area of the load sensor 130, and the identifier and the pressure-sensitive element having the ID are placed on the identifier. By storing in the storage unit 12 in association with the type of the article, the type of the article placed on the load sensor 130 can be specified based on the ID of the pressure sensitive element that senses the pressure due to the load of the article. can do.

Table 1 is an example of a table in which the ID of the pressure-sensitive element, the divided region on the load sensor 130, and the type of the article are associated with each other. In the example shown in Table 1, the pressure sensitive element ID 1-100 is associated with the division region R1 and the article O1, and the pressure sensitive element ID 101-200 is associated with the division region R2 and the article O2. That is, the information shown in Table 1 is stored in the storage unit 12, and by referring to the information in Table 1, the type of the article placed based on the ID of the pressure sensitive element that senses the pressure due to the load of the article can be determined. Can be identified. When the article is placed on the load sensor 130, it is preferable to provide a visually recognizable partition such as color-coding the divided regions R1 and R2 on the load sensor 130 so that they can be recognized.

According to the above configuration, among a plurality of pressure-sensitive elements arranged in a grid pattern in the measurement area of the load sensor 130, the type of the article is determined based on the ID or position information of the pressure-sensitive element that has detected the placement of the article. It can be specified, and the number of articles placed in the specific area can be calculated based on the type of the specified article, the weight per article, and the total weight of the articles.

In the above description, the information shown in Table 1 in which the pressure sensitive element ID, the division area, and the article type are associated is stored in the storage unit 12, but the information in which the pressure sensitive element ID and the article type are directly associated (division area). Information is not associated with this) is stored in the storage unit 12, and by referring to the information stored in the storage unit 12, the information is placed based on the ID of the pressure-sensitive element that senses the pressure due to the load of the article. The type of the article may be specified. Further, it is preferable that the relationship (association between the pressure sensitive element ID and the article type) stored in the storage unit 12 and shown in Table 1 can be changed so that the divided region of the load sensor 130 can be changed.

Further, in the above embodiment, in the measurement area of the load sensor 130, a separator SP for classifying the articles O1 to O4 by type is placed together with the articles O1 to O4, and the separator SP is placed. The placement position is configured so that data relating to the position of the separator SP is output in response to input of load distribution data obtained from the load sensor 130. However, it is not always necessary to detect the position of the separator SP based on the load distribution data obtained from the load sensor 130. For example, the position of the separator SP may be detected based on the image data output from the image pickup apparatus 140.

In this case, the storage unit 12 may store the learning model used for the process of detecting the position of the separator SP. The learning model is configured to output data regarding the position of the separator SP in response to the input of image data obtained from the image pickup apparatus 140. The structure of the learning model is defined by the definition information. The definition information includes configuration information that defines the configuration of the learning model (number of layers, number of nodes, etc.), and various parameters such as weights and biases between nodes referred to in the calculation using the learning model.

In this modified example as well, a configuration in which the divided region can be specified without being based on the load distribution data from the load sensor 130 may be adopted as in the above embodiment. Similarly, in this case as well, it is not necessary to use the distribution data (load distribution data) as the measurement data of the load sensor 130, and it is sufficient to use the data that can calculate the total weight of the articles placed in the divided region by the load sensor 130. ..

Similarly, also in this modification, the weight table 201 may store not only the weight per article but also the weight per unit number (one or more) including the case of one article. In this case as well, the weight per unit number (one or more) and the information that can be converted into the weight per article can be stored in the weight table 201 and converted into the weight per article. You can do it.

The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the meaning described above, and is intended to include all modifications within the meaning and scope equivalent to the claims.

10 Arithmetic logic unit 11 Arithmetic logic unit 12 Storage unit 13 1st connection unit 14 2nd connection unit 15 Communication unit 16 Operation unit 17 Display unit 130 Load sensor 140 Imaging device

Claims

A first calculation unit that calculates the total weight of the article based on the measurement data of the load sensor that measures the load of the article placed in the specific area.
A specific part that specifies the type of article placed in the specific area, and
The number of articles placed in the specific area is calculated based on the type of the article specified by the specific unit, the weight per unit number of the articles, and the total weight of the articles calculated by the first calculation unit. The second calculation unit and
A counting system for the number of articles.
The load sensor is
It has a plurality of pressure sensitive elements arranged at predetermined intervals, and has a plurality of pressure sensitive elements.
The specific part is
Based on the pressure-sensitive element that has detected the placement of the article among the plurality of pressure-sensitive elements by referring to the information associated with each of the plurality of pressure-sensitive elements and the type of the article placed above. The counting system for the number of articles according to claim 1, which specifies the type of the article.
The specific part is
The counting system for the number of articles according to claim 1, which specifies the type of the article based on the imaging data of the article.
The second calculation unit
The total weight of the articles placed in the division area divided by the separator for classifying the articles by type is calculated.
The number of articles according to claim 1, wherein the number of articles placed in the divided area is calculated based on the total weight of the articles placed in the divided area and the weight per unit number of the articles. Counting system.
A first detection unit that detects the position of the separator based on the measurement data of the load distribution in the state where the separator is placed on the specific region is provided.
The second calculation unit
The counting system for the number of articles according to claim 4, wherein a division area divided by the separator is specified based on the detection result of the first detection unit.
A second detection unit that detects the position of the separator based on the imaged data is provided.
The second calculation unit
The counting system for the number of articles according to claim 4, wherein a division area divided by the separator is specified based on the detection result of the second detection unit.
A step in which the first calculation unit calculates the total weight of the article based on the measurement data of the load sensor that measures the load of the article placed in the specific area.
A step in which the specific part specifies the type of the article placed in the specific area, and
The second calculation unit is placed in the specific area based on the type of the article specified by the specific unit, the weight per unit number of the article, and the total weight of the article calculated by the first calculation unit. The process of calculating the number of articles
How to count the number of articles with.
Computer,
The first calculation unit, which calculates the total weight of the article based on the measurement data of the load sensor that measures the load of the article placed in the specific area.
A specific part that specifies the type of article placed in the specific area,
The number of articles placed in the specific area is calculated based on the type of the article specified by the specific unit, the weight per unit number of the articles, and the total weight of the articles calculated by the first calculation unit. Second calculation unit,
A computer program that acts as.