JP7132044B2 - Refrigerator entry/exit management system and refrigerator - Google Patents

Description

TECHNICAL FIELD Embodiments of the present invention relate to a refrigerator entry/exit management system and a refrigerator.

Refrigerators are known that perform storage/retrieval management according to a predetermined algorithm. Further improvement in accuracy is expected for such warehouse entry/exit management.

JP 2015-81762 A

A problem to be solved by the present invention is to provide a refrigerator entering/leaving management system and a refrigerator capable of improving the accuracy of entering/leaving management.

A refrigerator entry/exit management system according to an embodiment includes a detection unit, an information generation unit, and a correction acceptance unit. The detection unit uses a first learning model trained to output a determination result of entering/leaving when an image photographed by a camera provided in a refrigerator is input, and detects the image photographed by the camera . Based on this, it is determined whether the refrigerator is in or out of the warehouse. The information generating unit causes the display screen of at least one of the refrigerator and the terminal device used by the user of the refrigerator to display the determined content of the detection unit and the image captured by the camera in association with each other. to generate When there is an error in the correspondence relationship between the image displayed on the display screen and the judgment content of the detection unit, the correction reception unit corrects the error based on a user's operation of the refrigerator or the terminal device. accept. The correction acceptance unit accepts correction of the judgment content of the detection unit when there is an error in the judgment content of the detection unit displayed in association with the image displayed on the display screen.

1 is a diagram showing the overall configuration of a warehousing/dispatching management system according to a first embodiment; FIG. The figure which shows the refrigerator of 1st Embodiment. 1 is a block diagram showing the system configuration of a warehousing/dispatching management system according to the first embodiment; FIG. The figure which shows typically the 1st learning model of 1st Embodiment. FIG. 2 is a diagram schematically showing part of a neural network used in the first learning model of the first embodiment; FIG. 4 is a diagram showing an example of input data for the input layer of the first embodiment; FIG. FIG. 5 is a diagram showing an example of input data that is incorrect answer data in the first embodiment; FIG. 5 is a diagram showing an example of teacher data of a second learning model according to the first embodiment; FIG. The figure which shows an example of the warehousing/dispatching management database of 1st Embodiment. FIG. 4 is a diagram showing an example of information displayed on the terminal device according to the first embodiment; FIG. 5 is a diagram showing another example of information displayed on the terminal device according to the first embodiment; The figure which shows an example of the correction reception screen of 1st Embodiment. FIG. 10 is a view showing a display screen when a first correction process is performed according to the first embodiment; FIG. FIG. 10 is a view showing a display screen when a second correction process is performed according to the first embodiment; FIG. FIG. 8 is a diagram showing another example of the correction acceptance screen according to the first embodiment; 4 is a flow chart showing the flow of processing in the execution stage of the warehousing/dispatching management system of the first embodiment; 4 is a flow chart showing the flow of correction processing of the warehousing/dispatching management system of the first embodiment; The figure which shows typically the 1st learning model of the 1st modification of 1st Embodiment. The figure which shows the whole structure of the entering/exiting management system of 2nd Embodiment.

A refrigerator entry/exit management system and a refrigerator according to embodiments will be described below with reference to the drawings. In the following description, the same reference numerals are given to components having the same or similar functions. Duplicate descriptions of these configurations may be omitted. As used herein, "based on XX" means "based on at least XX", and includes the case of being based on another element in addition to XX. Moreover, "based on XX" is not limited to the case of using XX directly, but also includes the case of being based on what has been calculated or processed with respect to XX. "XX" is an arbitrary element (for example, arbitrary information). In this specification, the stage of learning the learning model is called the "learning stage", and the stage of performing processing using the learned learning model is called the "execution stage".

(First embodiment)
[1. overall structure]
A first embodiment will be described with reference to FIGS. 1 to 17. FIG. First, the overall configuration of the warehousing/dispatching management system 1 will be described. FIG. 1 shows the overall configuration of a storage/delivery management system 1. As shown in FIG. The entering/leaving management system 1 includes, for example, a refrigerator 100 installed in each home, a terminal device 200 used by a user of the refrigerator 100, and a server device 300. Server device 300 is, for example, a cloud server, and can communicate with refrigerator 100 and terminal device 200 via network NW. The network NW includes, for example, the Internet, a WAN (Wide Area Network), a LAN (Local Area Network) in each home, and the like. Some or all communications in the network NW may be realized wirelessly. Each device included in the warehousing/dispatching management system 1 will be described in detail below.

[2. refrigerator]
First, refrigerator 100 will be described.
FIG. 2 shows the refrigerator 100 of this embodiment. (a) in FIG. 2 shows a state in which the door 120 of the refrigerator compartment 111A is opened. (b) in FIG. 2 shows a state in which the door 120 of the refrigerator compartment 111A is closed. Refrigerator 100 includes, for example, refrigerator body 110 , multiple doors 120 , open/close detection switches 130 , multiple sensors 140 , display device 150 , operation unit 160 , wireless transmission unit 170 , and control device 180 .

Refrigerator main body 110 is configured by a cabinet having heat insulation properties, and a plurality of storage compartments 111 are formed therein. The storage compartment 111 includes a refrigerator compartment 111A, a vegetable compartment, an ice making compartment, a small freezer compartment, a main freezer compartment, and the like. Refrigerator main body 110 has openings 112 on the front side of storage compartments 111 that allow objects to be taken in and out of storage compartments 111 . Opening 112 is closed by one or more doors 120 .

The open/close detection switch 130 is provided, for example, between the refrigerator main body 110 and the door 120 . The open/close detection switch 130 detects that the door 120 is opened when the door 120 is opened with respect to the opening 112 of the storage chamber 111 . Further, the open/close detection switch 130 detects that the door 120 is closed when the door 120 is closed with respect to the opening 112 of the storage chamber 111 .

A plurality of sensors 140 are provided, for example, in the storage chamber 111 to sense the state of the storage chamber 111 . A plurality of sensors 140 are arranged, for example, around the opening 112 . Sensor 140 is, for example, a camera. For this reason, for the sake of convenience, the term "sensor 140" will be replaced with "camera 140" below. An "image captured by a camera" is an example of a "detection value detected by a sensor." However, the sensor 140 is not limited to a camera, and may be another type of sensor such as an infrared sensor or an ultrasonic sensor. Therefore, "image captured by the camera" in the following description can be read as "information acquired by the sensor" and "detection value detected by the sensor". The "detection value detected by the sensor" is a detection value that includes different detection results depending on, for example, the position (movement) of the user's hand, the shape and posture of the hand (the way the fingertips are opened and the angle), etc. .

In this embodiment, the camera 140 is provided, for example, on the ceiling surface or the side surface of the storage room 111, and photographs objects passing through the opening 112, the user's hand, and the like. The camera 140 starts shooting when the open/close detection switch 130 detects that the door 120 is opened, and shoots at a predetermined sampling period (eg, 5 times per second). The camera 140 finishes photographing when the opening/closing detection switch 130 detects that the door 120 is closed. Note that the camera 140 may be a camera that captures an image of a location other than the opening 112 (for example, an inner pocket of the door 120 or a shelf provided in the storage room 111). Also, only one camera 140 may be provided for refrigerator 100 . In order to simplify the explanation, an example in which the server device 300 determines whether a product is in or out based on an image captured by one camera 140 provided on the ceiling surface of the storage room 111 will be described below.

The display device 150 is provided on the front surface of the door 120, for example. The display device 150 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display, and has a display screen 151 capable of displaying images and characters.

The operation unit 160 is, for example, a touch panel type input device, and is provided so as to overlap the display screen 151 of the display device 150 . The user can perform predetermined input and selection by operating the operation unit 160 . Note that operation unit 160 is not limited to a touch panel type input device, and may be a button or the like provided on refrigerator 100 .

Wireless transmission unit 170 includes a high-frequency circuit, an antenna, and the like, and is wirelessly connectable to network NW. For example, wireless transmission unit 170 is connected to network NW via a wireless LAN router and modem provided in the home where refrigerator 100 is installed.

Control device 180 controls refrigerator 100 as a whole. For example, the control device 180 centrally controls the operations of the open/close detection switch 130 , the plurality of cameras 140 , the display device 150 , the operation section 160 and the wireless transmission section 170 . Each time a new image is captured by camera 140 , control device 180 transmits the newly captured image to server device 300 . Control device 180 associates an image captured by camera 140 with time information when the image was captured (hereinafter referred to as “capturing time information”), and transmits the information to server device 300 .

[3. terminal equipment]
Returning to FIG. 1, the terminal device 200 will be described. The terminal device 200 is, for example, a terminal device owned by the user of the refrigerator 100, such as a mobile terminal device such as a smart phone or a tablet terminal, or a personal computer. The terminal device 200 has a display device 210 and an operation section 220 .

The display device 210 is, for example, a liquid crystal display or an organic EL display, and has a display screen 211 capable of displaying images and characters. The operation unit 220 is, for example, a touch panel type input device, and is provided so as to overlap the display screen 211 of the display device 210 . The user can perform predetermined input and selection by operating the operation unit 220 . Note that the operation unit 220 is not limited to a touch panel type input device, and may be buttons or keys provided on the terminal device 200 .

In refrigerator 100 and terminal device 200 described above, an application program, a web browser, or the like for using entry/exit management system 1 is started, and the processing described below is supported.

[4. Server device]
Next, the server device 300 will be described.
FIG. 3 shows the system configuration of the warehousing/dispatching management system 1. As shown in FIG. The server device 300 includes, for example, an information acquisition unit 310, a detection unit 320, a selection unit 330, an entry/exit management unit 340, an information generation unit 350, an information output unit 360, a correction reception unit 370, a first additional learning unit 380, a second It has an additional learning section 390 and a storage section ST. These functional units (information acquisition unit 310, detection unit 320, selection unit 330, entry/exit management unit 340, information generation unit 350, information output unit 360, correction acceptance unit 370, first additional learning unit 380, and second additional learning Part or all of the unit 390) is implemented by a hardware processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) executing a program (software) stored in the storage unit ST. However, some or all of these functional units are realized by hardware (circuitry; including circuitry) such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). may be realized by cooperation of software and hardware.

The storage unit ST is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a mask ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory), or the like. Consists of one or more. The storage unit ST contains a first learning model 410, a second learning model 420, an entry/exit management database 430 (hereinafter referred to as an "entry/exit management DB 430"), a first test data set 440, a second test data set 440, and a second test data set. A data set 450 and teacher data (not shown) used for learning the first and second learning models 410 and 420 are stored.

First, the information acquisition unit 310 will be described. Information acquisition unit 310 receives information transmitted from refrigerator 100 . For example, the information acquisition unit 310 acquires an image captured by the camera 140 and its capturing time information. The term “acquisition” as used in this specification is not limited to the case of causing refrigerator 100 to transmit information by transmitting a transmission request to refrigerator 100 , but also the case of passively receiving information transmitted from refrigerator 100 . include. Information acquisition unit 310 registers the image and shooting time information acquired from refrigerator 100 in entering/leaving management DB 430 . Information acquisition unit 310 also outputs the image acquired from refrigerator 100 to detection unit 320 .

Next, the detection unit 320 will be described. The detection unit 320 uses the first learning model 410 learned in advance to determine whether the refrigerator 100 is in or out of the warehouse based on the image captured by the camera 140 . Specifically, the first learning model 410 is a neural network that has been trained to output the determination result of entry/exit of the refrigerator 100 when an image captured by the camera 140 provided in the refrigerator 100 is input. In the first learning model 410, an image showing the state inside the refrigerator 100 is associated with the correct information corresponding to the image ("entered", "exited", or "neither entered nor exited"). The dataset is learned as training data. In this embodiment, an example in which the first learning model 410 is constructed by a convolution network (CNN) will be described. An example in which the first learning model 410 is constructed by a recurrent neural network (RNN) will be described later.

FIG. 4 schematically shows a first learning model 410 configured by CNN. The first learning model 410 includes an input layer 411 , multiple hidden layers 412 and an output layer 413 . The multiple hidden layers 412 include a convolution layer 412A, a pooling layer 412B, a ReLU (Rectified Linear Unit) layer (not shown), and the like. The convolution layer 412A performs a convolution operation on the image (filtering on the image). The pooling layer 412B performs processing to reduce the image. The ReLU layer performs a process of zeroing negative values of operation results. The convolutional layer 412A, the pooling layer 412B, and the ReLU layer form one intermediate layer, and this intermediate layer is stacked many times. The final stage of the hidden layer 412 is provided with a fully coupled layer 412C. In FIG. 4, one image is given as input data to the input layer 411, but in this embodiment, two or more images that are consecutive in time series are given to the input layer 411 at the same time. This will be discussed later.

FIG. 5 schematically shows part of the neural network used for the first learning model 410. As shown in FIG. x indicates the value of a unit (node) in each layer, and when an image is input, it becomes a value corresponding to a pixel value in the input layer 411 . The value of each unit is obtained by calculating the value obtained by multiplying the output of the unit in the previous layer by the weight w and adding the result using a predetermined function. These are represented by the following formulas.

where b is the bias term. A ReLU, a sigmoid function, or the like is used as the function f. By repeating the calculation of the above formula (1) from the input layer 411 to the output layer 413, the determination result by the first learning model 410 is obtained. "Learning" means adjusting the weight w, the bias term b, etc. to appropriate values. Learning is performed by stochastic gradient descent or the like. That is, random values are given to the weight w and the bias term b of each layer, and a dataset serving as teacher data is given to the input layer 411 . At the initial stage of learning, the output value output from the output layer 413 is an erroneous value. Weights and the like are updated in reverse order from the output layer 413 to the input layer 411 so that the deviation (error) becomes small. This is called error backpropagation, and the error between the output value and the target value is expressed as a squared error or cross entropy. At this time, if the error is represented by a differentiable function, the amount of adjustment (slope) to reduce the error can be calculated.

Next, input data given to the input layer 411 of the first learning model 410 will be described. In this embodiment, as input data for the input layer 411, two or more (for example, three or more) consecutive images are given simultaneously in time series. That is, as input data for the input layer 411, two or more (for example, three or more) consecutive images are given as one unit. "Two or more consecutive images (two or more detection values) in chronological order" refers to images captured at different times between multiple images (multiple detection values) sequentially captured by a camera (sensor). It means that the detected image (detection value) is not interrupted. "Two or more images consecutive in time series (two or more detection values)" means "Two or more images arranged in chronological order (associated in chronological order) (two or more detection values value)”.

FIG. 6 shows an example of one unit of input data for the input layer 411 (hereinafter referred to as “input data 510”). One unit of input data 510 includes, for example, a plurality of images IM that capture a series of user actions related to entering or exiting refrigerator 100 . In the example shown in FIG. 6, the input data 510 includes five consecutive images IM in time series. In this embodiment, these five images IM are input to the input layer 411 simultaneously. The first learning model 410 is learned using the above-described input data 510 as teacher data, so that the input of two or more images IM that are consecutive in time series is reflected to output the determination result of entering/leaving. have been learned to

In the learning stage, not only correct data but also incorrect data are provided. FIG. 7 shows an example of input data 510 that is incorrect data. The incorrect answer data is, for example, the image IM when neither warehousing nor warehousing is performed. Also, as training data, data such as changing the angle or type of an object to be held, wearing gloves, or holding an object with a magic hand may be given. Furthermore, in order to reduce erroneous detection due to environmental differences, images obtained by processing or synthesizing may also be learned. For example, processing methods for images include clipping of random areas, horizontal reversal, rotation, change in brightness, change in contrast, whitening, and the like.

In the execution stage, the first learning model 410 uses, as two or more time-series images IM, the most recent image IM captured by the camera 140 for a predetermined time period (for example, one second). An image IM is input. When a plurality of images IM captured by camera 140 are input to first learning model 410 , first learning model 410 outputs an output value indicating the determination result of entering/leaving of refrigerator 100 . For example, the first learning model 410 uses probability values indicating "possibility of entry", "possibility of exit", and "possibility of neither entry nor exit" as determination results of entry and exit of the refrigerator 100. to output For example, the first learning model 410 has output values such as 0.6 for “possibility of warehousing”, 0.1 for “possibility of warehousing”, and 0.3 for “possibility of neither warehousing nor warehousing”. Output.

In this case, the detection unit 320 may have a determination unit 320a in addition to the first learning model 410 . For example, when the output value indicating the possibility of warehousing exceeds the warehousing threshold value (for example, 0.8), the determination unit 320a determines that the output value indicates the possibility of warehousing. .8) is exceeded, it is determined as "out". However, the determination result output from the first learning model 410 is not limited to the above example. For example, the first learning model 410 may output a binary (YES/NO) determination result for each of "incoming" and "outgoing".

In the present embodiment, when an image IM captured by the camera 140 is input, the detection unit 320 determines the determination content indicating one of "entered", "exited", and "neither entered nor exited". to output The determination content of the detection unit 320 is output to the selection unit 330 and the storage/delivery management unit 340 in association with, for example, the set of images IM used for the determination.

In the present embodiment, the detection unit 320 determines whether the object is "entered" or "exited", but does not determine what the object entered or exited from the warehouse is. The content of determination by the detection unit 320 is managed in association with the image IM selected by the selection unit 330, which will be described later.

Next, the selection unit 330 will be described. The selection unit 330 automatically selects an image IM suitable for being associated with the determination content of the detection unit 320 from among the plurality of images IM captured by the camera 140 and input to the first learning model 410. . In the present embodiment, the selection unit 330 uses the second learning model 420 that has been learned in advance to select an image associated with the determination content of the detection unit 320 from among the plurality of images IM input to the first learning model 410. Select IM. Specifically, the second learning model 420 is configured by a neural network similar to that of the first learning model 410 . The second learning model 420 is trained so that when an image IM is input, it outputs a determination result regarding suitability as the image IM, which is associated with the determination content of the detection unit 320 .

FIG. 8 shows an example of teacher data used in the learning stage of the second learning model 420. As shown in FIG. The teacher data used for learning the second learning model 420 is, for example, a data set in which an image IM representing the state inside the refrigerator 100 and a score representing the suitability of the image IM are associated with each other. For example, the higher the score is set, the closer the object to be stored or delivered is located in the center of the image IM. In the example shown in FIG. 8, the topmost image IM is given a score of "0.2", the second image IM from the top is given a score of "1.0", and the remaining images IM are given a score of "0.2". A score of .0” is given. The second learning model 420 is trained such that when an image IM of teacher data is input, the output result of the second learning model 420 matches the score given to the input image IM.

In the execution stage, when the image IM captured by the camera 140 is input to the second learning model 420, the value indicating the appropriateness of the image IM associated with the determination content of the detection unit 320 is set to, for example, "0. Output a score between 0 and 1.0.

In this embodiment, the selection unit 330 has a determination unit 330a in addition to the second learning model 420 . For example, when the detection unit 320 determines “entering” or “leaving”, the determination unit 330a outputs the highest score by the second learning model 420 from among the set of images IM used for the determination. The resulting image IM is selected as the image IM associated with the determination content of the detection unit 320 . In addition to the set of images IM used for the determination of "entering" or "leaving", the selection unit 330 selects images IM shot slightly before or after those images IM based on the shooting time information. may be The selection result by the selection unit 330 is associated with the determination content of the detection unit 320 and output to the warehousing/delivery management unit 340 .

Next, the warehousing/dispatching management unit 340 will be described. Entry/exit management unit 340 manages the entry/exit status of refrigerator 100 based on the determination result of detection unit 320 and image IM selected by selection unit 330 .

FIG. 9 shows an example of the storage/delivery management DB 430 managed by the storage/delivery management unit 340 . The warehousing/delivery management DB 430 stores, for example, all images IM that are captured by the camera 140 and that have not been taken for many days. That is, the images IM stored in the warehousing/delivery management DB 430 include not only the images IM selected by the selection unit 330 but also the images IM not selected by the selection unit 330 . These images IM are managed in association with the shooting time information of the images IM.

The warehousing/delivery management unit 340 registers the content of the determination made by the detection unit 320 (information indicating “warehouse entry” or “warehouse delivery”) in the warehousing/dispatch management DB 430 . For example, the entering/leaving management unit 340 associates and manages the determination content of the detection unit 320 and the image IM (selected image) selected by the selection unit 330 corresponding to the determination. In addition, the entering/leaving management unit 340 may manage the content of the determination by the detection unit 320 and the set of images IM used for the determination in association with each other.

In the present embodiment, when the detection unit 320 determines that the product is in storage, the storage/delivery management unit 340 adds a predetermined marking to the image IM selected by the selection unit 330 in response to the determination. The predetermined marking is a marking indicating stock. On the other hand, if the detection unit 320 determines that the product is “out of the warehouse”, the warehousing/leaving management unit 340 deletes the marking attached at the time of warehousing. Specifically, when the detection unit 320 determines “out”, the entry/exit management unit 340 selects the image IM selected by the selection unit 330 in response to the determination, and the marking image IM in the entry/exit management DB 430 . with one or more images IM marked with . Then, if the degree of similarity between the images IM exceeds a predetermined threshold, the warehousing/delivery management unit 340 determines that the object corresponding to the image IM has been delivered, and the marking attached to the image IM. delete. In this way, the entering/leaving management unit 340 manages the inventory status in the refrigerator 100 .

Next, the information generator 350 will be described. The information generation unit 350 displays the determination content of the detection unit 320 and the image IM selected by the selection unit 330 on the display screens 151 and 211 of at least one of the refrigerator 100 and the terminal device 200 used by the user of the refrigerator 100. Generates information to be displayed in association with . The information generation unit 350 generates the above information based on, for example, the warehousing/dispatching management DB 430 . Note that the "determination content of the detection unit 320" referred to in this specification may be content indicating that the object (as a result of warehousing) exists as inventory. That is, the phrase "to display the image IM in association with the determination content of the detection unit 320" in this specification includes the case of displaying the image IM as "inventory". An example in which information generated by the information generation unit 350 is displayed on the display screen 211 of the terminal device 200 will be described below. However, information similar to the content described below may also be displayed on display screen 151 of refrigerator 100 .

FIG. 10 shows an example of information generated by the information generator 350 and displayed on the display screen 211 of the terminal device 200. As shown in FIG. On the display screen 211 of the terminal device 200, for example, information indicating the content of the determination (“entering” or “leaving”) of the detection unit 320 and the image IM selected by the selection unit 330 corresponding to the determination correspond. attached and displayed. Display screen 211 also displays a correction button BT11 for accepting correction of the content displayed on display screen 211 . This function will be described later.

FIG. 11 shows another example of information generated by the information generation unit 350 and displayed on the terminal device 200. FIG. A display screen 211 of the terminal device 200 displays information indicating a list of inventory in the refrigerator 100 . For example, the display screen 211 of the terminal device 200 displays a list of images IM showing the inventory in the refrigerator 100 . The display screen 211 also displays a correction button BT12 for accepting correction of the content displayed on the display screen 211 . This function will be described later.

Next, the information output section 360 will be described. Information output unit 360 transmits the information generated by information generation unit 350 to at least one of refrigerator 100 and terminal device 200 used by the user of refrigerator 100 . The information output unit 360 may transmit the above information each time the detection unit 320 determines whether to enter or leave the warehouse. may be sent.

Next, the correction receiving section 370 will be described. If there is an error in the correspondence relationship between the image IM displayed on the display screen 211 and the determination content of the detection unit 320, the correction reception unit 370 corrects the user's operation to the operation unit 160 of the refrigerator 100 or the operation unit 220 of the terminal device 200. Accept corrections of errors based on An example in which a screen for accepting correction is displayed on the display screen 211 of the terminal device 200 and correction is performed based on the user's operation on the operation unit 220 of the terminal device 200 will be described below. A similar screen may be displayed on display screen 151 of refrigerator 100 , and correction may be performed based on the user's operation on operation unit 160 of refrigerator 100 .

FIG. 12 shows an example of a correction acceptance screen 610 of this embodiment. The correction acceptance unit 370 displays a correction acceptance screen 610 on the display screen 211 when the user operates the correction button BT11 shown in FIG. 10, for example. Correction acceptance screen 610 includes, for example, a first button BT21 for accepting correction of the determination content of detection unit 320, and a second button BT22 for accepting correction of displayed image IM.

In the present embodiment, the correction accepting unit 370 corrects the image IM displayed on the display screen 211, but the determination content of the detection unit 320 displayed on the display screen 211 in association with the image IM is erroneous. When the user operates the first button BT21 (when there is an error in the determination of entering or leaving the warehouse), a first correction process for receiving a correction of the determination content of the detection unit 320 is performed.

FIG. 13 shows the display screen 211 when the first correction process is performed. In the first correction process, the correction receiving unit 370 causes the display screen 211 to display candidates for other determination content of the detection unit 320 . For example, when the first button BT21 is operated in a state in which "entering" is displayed, the button BT31 for correcting the determination content of the detection unit 320 to "exit" and the button BT31 for A button BT32 for correcting "is not set" is displayed. Then, the correction accepting unit 370 accepts correction of the determination content of the detecting unit 320 based on the operation of the buttons BT31 and BT32.

On the other hand, the correction accepting unit 370 corrects the determination content of the detection unit 320 displayed on the display screen 211, but corrects the image IM selected by the selection unit 330 and displayed on the display screen 211 in association with the determination content. is not appropriate (image selection is not appropriate), a second correction process is performed to accept correction of the image IM associated with the determination content of the detection unit 320 .

FIG. 14 shows the display screen 211 when the second correction process is performed. In the second correction process, the correction accepting unit 370 causes the display screen 211 to display one or more other images IM captured by the camera 140 as candidate images. The “another one or more images” are, for example, the remaining four images IM not selected by the selection unit 330 in the set of five images IM used for determination by the detection unit 320 . However, in addition to the above four images IM, the correction accepting unit 370 may display another image IM taken slightly before or after those images IM based on the shooting time information as a candidate image. The correction accepting unit 370 accepts correction of the image IM associated with the determination content of the detecting unit 320 based on the operation of selecting one image IM from the candidate images input by the user.

FIG. 15 shows another example of the correction acceptance screen 620 of this embodiment. The correction accepting unit 370 displays a correction accepting screen 620 on the display screen 211 when the user operates the correction button BT12 shown in FIG. 11, for example.

In this embodiment, when the image IM that should be displayed on the display screen 211 in association with the determination content of the detection unit 320 is not displayed (when the received item is not displayed as inventory), the correction reception unit 370 A third correction process is performed to accept the selection of the image IM associated with the determination content of 320 .

For example, in the third correction process, the correction accepting unit 370 causes the display screen 211 to display one or more images IM captured by the camera 140 as candidate images. The “one or more images” are, for example, images IM captured by the camera 140 in the past. The correction accepting unit 370 accepts selection of an image IM to be added as inventory (an image IM to be marked) based on an operation of selecting one or more images IM from the candidate images input by the user. .

On the other hand, when the image IM that should not be associated with the determination content of the detection unit 320 is associated with the determination content of the detection unit 320 and displayed on the display screen 211 (the delivered object is in inventory ), a fourth correction process is performed to cancel the correspondence between the image IM and the determination content of the detection unit 320 .

For example, in the fourth correction process, the correction accepting unit 370 accepts selection of an image IM to be deleted from inventory (an image IM to be demarked) from the images IM displayed on the display screen 211 as inventory.

Upon receiving the correction content, the correction reception unit 370 outputs the received correction content to the warehousing/delivery management unit 340 . The warehousing/dispatching management unit 340 updates the warehousing/dispatching management DB 430 based on the correction contents received by the correction reception unit 370 .

Next, the first additional learning section 380 will be described. The first additional learning unit 380 additionally learns the first learning model 410 based on the information received by the correction receiving unit 370 when the correction process of receiving the correction of the determination content of the detection unit 320 is performed. For example, the correction receiving unit 370 associates the content of the correction made to the determination content of the detection unit 320 with the set of images IM used for the determination of the detection unit 320 to be corrected, and creates a new teacher. data. Then, the first additional learning unit 380 performs additional learning (relearning) on the first learning model 410 so as to reflect the new teacher data.

In this embodiment, after additionally learning the first learning model 410 , the first additional learning unit 380 confirms the determination accuracy of the additionally learned first learning model 410 . Specifically, the first additional learning unit 380 inputs the first test data set 440 to the additionally trained first learning model 410 . The first test data set 440 is a data set in which the image IM showing the state inside the refrigerator 100 and the correct information on entering and leaving the warehouse corresponding to the image IM are correctly associated with each other. This data set is not used for additional learning. When the accuracy of the determination result of the first learning model 410 when the first verification data set 440 is input satisfies a predetermined condition, the first additional learning unit 380 applies the additionally learned content to the first learning model. Reflected in 410 . “If a predetermined condition is satisfied” is, for example, a case where the determination accuracy of the additionally learned first learning model 410 does not fall below the determination accuracy of the first learning model 410 before the additional learning.

Next, the second additional learning section 390 will be described. Second additional learning unit 390 additionally learns second learning model 420 based on information received by correction receiving unit 370 when correction processing for receiving correction of image IM selected by selecting unit 330 is performed. Let For example, the correction receiving unit 370 associates the image IM with the content of the correction of the score attached to the image IM, and uses it as new teacher data. Then, the second additional learning unit 390 performs additional learning (relearning) on the second learning model 420 so as to reflect the new teacher data.

In the present embodiment, the second additional learning unit 390 confirms the determination accuracy of the additionally learned second learning model 420 after additionally learning the second learning model 420 . Specifically, the second additional learning unit 390 inputs the second test data set 450 to the additionally trained second learning model 420 . Second test data set 450 is a data set in which image IM showing the state inside refrigerator 100 is appropriately associated with a score assigned to the image IM. This data set is not used for learning. Second additional learning unit 390 applies additionally learned content to second learning model 420 when the determination result of second learning model 420 satisfies a predetermined condition when second verification data set 450 is input. To reflect. “When a predetermined condition is satisfied” means, for example, that the determination accuracy (appropriateness of the output score) by the additionally learned second learning model 420 is higher than the determination accuracy by the second learning model 420 before being additionally learned. is not lower than

In this embodiment, the first and second learning models 410, 420 are separated for each user (for each family). In this case, the additional learning by the first and second additional learning units 380 and 390 makes it easier to apply the determination accuracy of the detection unit 320 and the selection unit 330 to the situation and environment specific to the user. On the other hand, the first and second learning models 410, 420 may be shared by multiple users (multiple homes). In this case, a plurality of users can share the result of additional learning according to another user's correction.

Next, the flow of processing in the execution stage of the warehousing/dispatching management system 1 will be described.
FIG. 16 is a flow chart showing the flow of processing in the execution stage of the warehousing/dispatching management system 1. As shown in FIG. First, control device 180 of refrigerator 100 detects that the door of refrigerator 100 has been opened based on the detection result of open/close detection switch 130 (S101). Next, the control device 180 takes an image with the camera 140 (S102), and transmits the captured image IM to the server device 300 each time an image is taken (S103).

When server device 300 receives a new image IM from refrigerator 100, server device 300 discards the old and unused image IM (S104). “Unused” is an image IM that is not managed in association with the detection content of the detection unit 320 . Then, server device 300 stores the new image IM received from refrigerator 100 in storage unit ST (S105).

Every time an image IM is received from refrigerator 100, server device 300 inputs a plurality of latest images IM including the latest image IM received from refrigerator 100 to first learning model 410 (S110). Then, the detection unit 320 of the server device 300 determines whether or not the output value indicating the probability of "stocking" output from the first learning model 410 is equal to or greater than the storage threshold (S111). When the output value indicating the probability of "stocking" is equal to or greater than the storage threshold (S111: YES), the detection unit 320 determines that the product is "stocking" and performs the storage process (S112). Specifically, the detection unit 320 associates the content of the determination indicating that there was a warehousing with the set of images IM used for the determination, and outputs them to the selection unit 330 and the storage/delivery management unit 340 .

In this case, the selection unit 330 inputs to the second learning model 420 a plurality of images IM used for the determination of “warehouse”. Then, the selecting unit 330 associates the image IM output with the highest score by the second learning model 420 from the set of images IM used for the determination of “warehouse” with the determination content of the detecting unit 320. is selected as the image IM to be used. The selection unit 330 associates the determination content of the detection unit 320 with the selected image IM, and outputs the result to the storage/delivery management unit 340 . The warehousing/dispatching management unit 340 gives IM marking to the image selected by the selection unit 330 (S113).

On the other hand, when the output value indicating the probability of "entering" is less than the threshold for "entering" (S111: NO), the detection unit 320 detects that the output value indicating the probability of "exiting" output from the first learning model 410 is less than the threshold for "entering". It is determined whether or not the above is satisfied (S121). When the output value indicating the probability of "departure" is equal to or greater than the delivery threshold value (S121: YES), the detection unit 320 determines "departure" and performs delivery processing (S122). Specifically, detection unit 320 associates the content of determination indicating that there was a delivery with the set of images IM used for the determination, and outputs the result to selection unit 330 and entry/exit management unit 340 .

In this case, the selection unit 330 inputs to the second learning model 420 a plurality of images IM used for the determination of “delivery”. Then, the selecting unit 330 associates the image IM output with the highest score by the second learning model 420 from among the set of images IM used for the determination of “departure” with the determination content of the detecting unit 320. is selected as the image IM to be used. The selection unit 330 associates the determination content of the detection unit 320 with the selected image IM, and outputs the result to the storage/delivery management unit 340 . The warehousing/dispatching management unit 340 compares the image selected by the selection unit 330 with one or more marked images IM in the warehousing/dispatching management DB 430 . Then, the warehousing/dispatching management unit 340 selects an image IM representing the object of “shipping” from among the images IM managed by the warehousing/dispatching management DB 430 based on the degree of similarity of the images IM. is deleted (S123).

Note that the detection unit 320 detects when the output value indicating the probability of “entering” output from the first learning model 410 is less than the threshold for entering the warehouse and the output value indicating the probability of “leaving” is also less than the threshold for leaving the warehouse (S121 : NO), no processing related to entering/exiting is performed.

Control device 180 of refrigerator 100 determines whether the door of refrigerator 100 is open at a predetermined sampling period (S131). When it is determined that the door of refrigerator 100 is open based on the detection result of open/close detection switch 130 (S131: YES), the processes from S102 to S123 are repeated. On the other hand, when control device 180 determines that the door of refrigerator 100 is closed (S131: NO), control device 180 terminates photographing by camera 140 . This completes the processing of this flow.

Next, the flow of correction processing of the warehousing/dispatching management system 1 will be described.
FIG. 17 is a flow chart showing the flow of correction processing of the warehousing/dispatching management system 1. As shown in FIG. This flow shows an example in which the terminal device 200 requests the server device 300 to browse inventory information. Also, in this example, for simplicity of explanation, the case where there is only one error to be corrected is taken up.

First, the server device 300 receives a browsing request from the terminal device 200 (S201). Next, the information generating unit 350 extracts the marked images IM from the incoming/outgoing management DB 430, generates information indicating a list of the extracted images IM as inventory information (S202), and transmits the generated inventory information to the terminal. It is presented to the user by transmission from the device 200 (S203).

Next, the correction accepting unit 370 determines whether or not there is a correction operation by the user (S211). If there is no correction operation by the user (S211: NO), the process of this flow ends. On the other hand, if there is a correction operation by the user (S211: YES), the correction reception unit 370 determines whether the correction operation is a "correction operation regarding warehousing" or a "correction operation regarding warehousing" (S212). ). If the correction operation by the user is "correction operation regarding delivery" (S212: "correction operation for delivery"), the correction reception unit 370 determines that the delivery has not been made and the marking remains. On the other hand, the marking is added to the image IM which has been erroneously determined to be "delivered" even though it has not been delivered and the marking has been deleted (S221). In this case, the process proceeds to S241, which will be described later.

On the other hand, when the correction operation by the user is "correction operation regarding warehousing" (S212: "correction operation for warehousing"), the correction accepting unit 370 does not determine that the correction operation by the user is "warehousing". to the determination of "warehouse" or an operation to replace the image IM displayed in association with the determination of "warehouse" (S231). Specifically, the correction accepting unit 370 determines that the image IM selected by the user to be added to the judgment of "warehouse" is an image IM that has not been judged to be "warehousing" in the past judgment by the detection unit 320. If it is not included in the set of images IM used for the determination of "warehouse", it is recognized as an additional operation of "warehousing". On the other hand, if the image IM selected by the user to be added to "warehouse" is the image IM that was determined to be "warehousing" by the detection unit 320 in the past determination ("warehouse is included in the set of images IM used for the determination of ”), it is recognized as an image IM replacement operation.

If the user's correction operation is an addition operation of "warehousing" (S231: "addition operation"), the correction reception unit 370 determines that the content of the determination made by the detection unit 320 is incorrect, and corrects the image IM selected by the user. A marking is added (S232). In this case, the process proceeds to S241, which will be described later.

On the other hand, when the correction operation by the user is the replacement operation of the image IM (S231: “replacement operation”), the correction accepting unit 370 determines that the selection of the image IM by the selection unit 330 is inappropriate, and determines that the selection of the image IM is not appropriate. The marking is added to the image IM to be added, and the marking of the image IM to be deleted by the replacement operation is deleted (S233). In this case, the process proceeds to S242, which will be described later.

Next, in S<b>241 , the first learning model 410 is additionally learned by the first additional learning section 380 based on the information accepted by the correction accepting section 370 . On the other hand, in S<b>242 , the second learning model 420 is additionally learned by the second additional learning section 390 based on the information accepted by the correction accepting section 370 . This completes the processing of this flow. If there are multiple corrections, those contents are saved as information such as editing history, and additional learning is performed to collectively reflect the multiple corrections.

According to the warehousing/dispatching management system 1 having such a configuration, it is possible to improve the accuracy of warehousing/dispatching management. Here, as a comparative example, a configuration is considered in which storage/delivery management is performed by a predetermined algorithm that does not rely on learning. In this case, it is assumed that the determination will fail in a situation that is not suitable for the algorithm, and the accuracy of incoming/outgoing management will be lowered. Also, it may be difficult to adjust parameters such as thresholds used in the algorithm. Also, it is necessary to improve the algorithm through trial and error in order to improve the judgment accuracy. This can result in inefficient development.

On the other hand, in the present embodiment, entry/exit of the refrigerator 100 is determined using the first learning model 410 learned in advance. According to such a configuration, it is possible to improve the accuracy of entering/leaving management by appropriately performing learning. In addition, work for parameter adjustment and algorithm improvement can be eliminated or reduced, and development can be made more efficient.

In the present embodiment, the detection unit 320 determines entering/leaving of the refrigerator 100 by inputting to the first learning model 410 two or more images IM that are consecutive in time series captured by the camera 140 . According to such a configuration, entry/exit can be determined based on the movement of the user, and determination accuracy can be further improved.

In this embodiment, the camera 140 takes pictures only from when the door 120 of the refrigerator 100 is opened until the door 120 is closed. According to such a configuration, it is possible to save energy and reduce the amount of data that needs to be stored.

In the present embodiment, the entry/exit management system 1 associates the determination content of the detection unit 320 with the image IM captured by the camera 140 on the display screens 151 and 211 of at least one of the refrigerator 100 and the terminal device 200. and an information generation unit 350 that generates information to be displayed. According to such a configuration, it is possible to check the stock status of refrigerator 100 without opening refrigerator 100 (for example, even when away from home), thereby enhancing convenience for the user.

In this embodiment, if there is an error in the correspondence relationship between the image IM displayed on the display screens 151 and 211 and the determination content of the detection unit 320, the entry/exit management system 1 performs an operation on the refrigerator 100 or terminal device 200. and a correction receiving unit 370 for receiving correction of errors. According to such a configuration, based on the information received by the correction receiving unit 370, correct information can be displayed from the next time onward.

In this embodiment, the warehousing/dispatching management system 1 includes a first additional learning section 380 that additionally learns the first learning model 410 based on the information received by the correction receiving section 370 . According to such a configuration, it is possible to further improve the accuracy of determination of the first learning model 410 based on the information received by the correction receiving unit 370 .

In this embodiment, the first additional learning unit 380 performs additional learning when the determination accuracy of the first learning model 410 when the first verification data set 440 is input satisfies a predetermined condition. 1 reflected in the learning model 410 . According to such a configuration, even if the user erroneously corrects the determination content of the detection unit 320, it is possible to prevent the determination accuracy of the first learning model 410 from deteriorating.

In this embodiment, the warehousing/dispatching management system 1 includes a selection unit 330 that selects an image IM suitable for being associated with the determination content of the detection unit 320 from among the plurality of images IM input to the first learning model 410. . According to such a configuration, it is possible to display an image IM suitable for the display screens 151 and 211, thereby enhancing convenience for the user.

In this embodiment, the selection unit 330 selects the image IM to be associated with the determination content of the detection unit 320 using the second learning model 420 learned in advance. According to such a configuration, it is possible to further improve the determination accuracy of the image IM selected by the selection unit 330 .

In this embodiment, the warehousing/dispatching management system 1 includes a second additional learning section 390 that additionally learns the second learning model 420 based on the information received by the correction receiving section 370 . According to such a configuration, it is possible to further improve the determination accuracy of the second learning model 420 .

In this embodiment, the detection unit 320 , the selection unit 330 , the storage unit ST, etc. are provided in the server device 300 . According to such a configuration, the specs of the refrigerator 100, such as the CPU, can be lowered, and the cost and power consumption of the refrigerator 100 can be reduced. Further, if the detection unit 320 and the selection unit 330 are provided in the server device 300, version upgrade (performance improvement) of the detection unit 320 and the selection unit 330 is facilitated.

(First modification)
Next, the 1st modification of 1st Embodiment is demonstrated. The warehousing/dispatching management system 1 of this modified example is different from that of the first embodiment in that the first learning model 410A is configured by RNN. Configurations other than those described below are the same as those of the first embodiment.

FIG. 18 schematically shows the first learning model 410A configured by RNN. Similar to the first learning model 410 of the first embodiment, the first learning model 410 is a neural network trained in advance so as to output an entry/exit determination result when an image IM captured by the camera 140 is input. network. The first learning model 410 includes an input layer 411 , multiple hidden layers 412 (only one is shown in FIG. 18), and an output layer 413 . Hidden layer 412 has a loop to keep a memory of the input data. The hidden layer 412 has, for example, an LSTM (Long Short-Term Memory) structure.

According to such a first learning model 410A, it is possible to learn so as to reflect the input of two or more images IM that are consecutive in time series to output the determination result of entering/leaving. In this embodiment, the latest images IM captured by the camera 140 are sequentially input to the input layer 411 of the first learning model 410A one by one. With such a configuration as well, it is possible to improve the accuracy of incoming/outgoing management, as in the first embodiment.

(Second modification)
Next, the 2nd modification of 1st Embodiment is demonstrated. The entry/exit management system 1 of this modification differs from that of the first embodiment in that the selector 330 is implemented by a predetermined algorithm instead of a neural network. Configurations other than those described below are the same as those of the first embodiment.

In this modification, the selection unit 330 performs predetermined image processing on the image IM captured by the camera 140 to select the determination content of the detection unit 320 from among the plurality of images IM captured by the camera 140. Select the image IM associated with . For example, the selection unit 330 compares the image IM captured by the camera 140 with a template image (an image in which the object and the user's hand are not captured), and the template image and the pixels in the image IM captured by the camera 140 are different. Remove matching parts. Thereby, the selection unit 330 obtains an image in which the object and the hand are extracted.

Next, the selection unit 330 further removes the portion corresponding to the user's hand from the image based on, for example, color information. This gives an image corresponding to the object. The selection unit 330 selects the image IM having the largest image corresponding to the object from among the plurality of images IM input to the selection unit 330 as the image IM to be associated with the determination content of the detection unit 320 . With such a configuration as well, it is possible to improve the accuracy of incoming/outgoing management, as in the first embodiment.

(Second embodiment)
Next, with reference to FIG. 19, a second embodiment will be described. The second embodiment differs from the first embodiment in that the functional units provided in the server device 300 in the first embodiment are provided in the refrigerator 100 . Configurations other than those described below are the same as those of the first embodiment.

FIG. 19 shows the overall configuration of a warehousing/dispatching management system 1A according to the second embodiment. In this embodiment, many functional units of the entering/leaving management system 1A including the detection unit 320 and the selection unit 330 are provided in the refrigerator 100 . In this case, refrigerator 100 and terminal device 200 may communicate directly or may communicate via network NW. With such a configuration as well, it is possible to improve the accuracy of incoming/outgoing management, as in the first embodiment.

Although several embodiments and modifications have been described above, the embodiments are not limited to the above examples. For example, some embodiments and variations described above may be implemented in combination with each other. In this specification, “inputting information acquired by the sensor 140 to the learning model (inputting an image captured by the camera 140)” means that after being acquired by the sensor (camera), a predetermined It also includes the case where preprocessed information or images are input.

According to at least one of the embodiments described above, it is possible to improve the accuracy of entering/leaving management by providing a detection unit that determines entering/leaving of a refrigerator using a learned learning model.

While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, as well as the scope of the invention described in the claims and equivalents thereof.

DESCRIPTION OF SYMBOLS 1, 1A... Entry/exit management system, 100... Refrigerator, 140... Sensor (camera), 150... Display device of refrigerator, 151... Display screen, 200... Terminal device, 210... Display device of terminal device, 211... Display screen, 300... Server device, 320... Detecting unit, 330... Selecting unit, 370... Correction receiving unit, 410, 410A... First learning model, 420... Second learning model.

Claims (9)

Using a first learning model that has been learned to output a determination result of entering/leaving when an image photographed by a camera provided in the refrigerator is input, the refrigerator is entered based on the image photographed by the camera . a detection unit that determines delivery;
An information generation unit for generating information for displaying the determined content of the detection unit and the image captured by the camera in association with each other on the display screen of at least one of the refrigerator and a terminal device used by the user of the refrigerator. When,
a correction reception unit that receives correction of the error based on a user's operation of the refrigerator or the terminal device when there is an error in the correspondence relationship between the image displayed on the display screen and the determination content of the detection unit;
with
The correction accepting unit accepts correction of the judgment content of the detection unit when there is an error in the judgment content of the detection unit displayed in association with the image displayed on the display screen.
Refrigerator entry/exit management system. The first learning model is trained to reflect the input of two or more images that are consecutive in time series to output the determination result of entering and leaving the warehouse,
The detection unit determines whether the refrigerator is in or out of storage by inputting two or more time-sequential images taken by the camera into the first learning model.
2. The system for managing entry and exit of refrigerators according to claim 1 . Using a first learning model that has been learned to output a determination result of entering/leaving when an image photographed by a camera provided in the refrigerator is input, the refrigerator is entered based on the image photographed by the camera. a detection unit that determines delivery;
An information generation unit for generating information for displaying the determined content of the detection unit and the image captured by the camera in association with each other on the display screen of at least one of the refrigerator and a terminal device used by the user of the refrigerator. When,
a correction reception unit that receives correction of the error based on a user's operation of the refrigerator or the terminal device when there is an error in the correspondence relationship between the image displayed on the display screen and the determination content of the detection unit;
a first additional learning unit for additionally learning the first learning model based on the information received by the correction receiving unit;
with
The first additional learning unit inputs a test data set to the additionally trained first learning model, and the accuracy of the determination result of the first learning model when the test data set is input is a predetermined value. Reflecting the additionally learned content in the first learning model when the condition is satisfied;
Refrigerator entry/exit management system. Using a first learning model that has been learned to output a determination result of entering/leaving when an image photographed by a camera provided in the refrigerator is input, the refrigerator is entered based on the image photographed by the camera. a detection unit that determines delivery;
An information generation unit for generating information for displaying the determined content of the detection unit and the image captured by the camera in association with each other on the display screen of at least one of the refrigerator and a terminal device used by the user of the refrigerator. When,
a selection unit that selects an image suitable for being associated with the determination content of the detection unit from among the plurality of images input to the first learning model ;
Refrigerator entry /exit management system. When an image is input, the selection unit uses a second learning model trained to output a determination result regarding suitability as an image associated with the determination content of the detection unit, and selects the first learning model. selecting an image associated with the determination content of the detection unit from among the plurality of input images;
5. The refrigerator entry/exit management system according to claim 4 . The information generation unit generates information to be displayed on a display screen of at least one of the refrigerator and the terminal device in association with the determination content of the detection unit and the image selected by the selection unit.
6. The refrigerator entry/exit management system according to claim 5 . When the image selected by the selection unit and displayed on the display screen in correspondence with the determination content of the detection unit is not appropriate, another one or more images captured by the camera are displayed on the display screen. and further comprising a correction receiving unit that receives correction of the image associated with the determination content of the detection unit,
7. The refrigerator entry/exit management system according to claim 6 . a second additional learning unit for additionally learning the second learning model based on the information received by the correction receiving unit when the correction regarding the image selected by the selecting unit is received by the correction receiving unit; rice field,
8. The refrigerator entry/exit management system according to claim 7 . the camera;
a warehousing/dispatching management system according to any one of claims 1 to 8 ;
refrigerator.

Images