JP7040193B2 - Refrigerator system - Google Patents

Description

The present invention relates to a refrigerator system.

In a refrigerator system, a camera device as an image pickup means for imaging a storage room, a transmission means for transmitting image information captured by the camera device to an external device, and a food material discrimination means for discriminating foodstuffs from the image information captured by the camera device. And, are known (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2015-05630

In a refrigerator system as shown in Patent Document 1, it is desirable to prepare reference data for identifying foodstuffs in order to accurately discriminate (identify) foodstuffs from images taken by a camera. When preparing learned data that learns the correspondence between the characteristic amount of the image of the food and the corresponding food as the reference data for identifying the food, the image of the food and the type or product name of the food correspond correctly. You will need a dataset. However, it is not easy to manually prepare a large number of such data sets.

The present invention has been made to solve such a problem. The purpose is to automatically acquire a data set for learning the feature amount of the image of the food and the correspondence with the food, to improve the accuracy of food identification and to expand the range of foods that can be identified. To get a refrigerator system that can.

The refrigerator system according to the present invention includes a refrigerator in which a storage chamber for storing food is formed, a camera for taking an image of the storage chamber, a feature amount extraction unit for extracting the feature amount of the image taken by the camera, and a feature amount extraction unit. The feature amount extraction unit extracted using the storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the foodstuff and the foodstuff, and the learned data stored in the storage unit. When the foodstuff identification unit that identifies the foodstuff from the characteristic amount and the foodstuff identification unit identify the foodstuff, the characteristic amount related to this identification is re-learned to correspond to the foodstuff and stored in the storage unit. The re-learning unit for updating the learned data and the area specifying unit for specifying the area where the foodstuff exists in the image taken by the camera are provided , and the feature amount extracting unit is specified by the area specifying unit. The feature amount is extracted for each region, the foodstuff identification unit identifies the foodstuff for each region specified by the region identification unit, and the camera captures images of the same region specified by the region identification unit from different multiple viewpoints. The feature amount extraction unit extracts the feature amounts of the multi-viewpoint images taken by the camera from the plurality of viewpoints in the same area, and the re-learning unit uses the foodstuff identification unit as the foodstuff. Is identified, the feature amount of each of the multi-viewpoint images extracted by the feature amount extraction unit is re-learned in association with the food material, and the learned data stored in the storage unit is updated. To.
Alternatively, the refrigerator system according to the present invention includes a refrigerator in which a storage chamber for storing food is formed, a camera that captures an image of the storage chamber, and a feature quantity extraction unit that extracts the feature quantity of the image captured by the camera. And the storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food material and the foodstuff, and the feature amount extraction unit using the learned data stored in the storage unit. When the foodstuff identification unit that identifies the foodstuff from the extracted feature amount and the foodstuff identification unit identify the foodstuff, re-learning is performed so that the feature amount related to this identification corresponds to the foodstuff, and the storage unit is stored. The foodstuff identification unit includes a re-learning unit that updates the stored learned data, and the foodstuff identification unit uses different identification methods depending on whether the foodstuff type is identified or the foodstuff product name is identified.
Alternatively, the refrigerator system according to the present invention includes a refrigerator in which a storage chamber for storing food is formed, a camera that captures an image of the storage chamber, and a feature quantity extraction unit that extracts the feature quantity of the image captured by the camera. And the storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food material and the foodstuff, and the feature amount extraction unit using the learned data stored in the storage unit. When the foodstuff identification unit that identifies the foodstuff from the extracted feature amount and the foodstuff identification unit identify the foodstuff, re-learning is performed so that the feature amount related to this identification corresponds to the foodstuff, and the storage unit is stored. The foodstuff identification unit includes a re-learning unit for updating the stored learned data, depending on whether or not the amount of the learned data stored in the storage unit is equal to or greater than a preset reference amount. And use different identification methods.
Alternatively, the refrigerator system according to the present invention includes a refrigerator in which a storage chamber for storing food is formed, a camera that captures an image of the storage chamber, and a feature quantity extraction unit that extracts the feature quantity of the image captured by the camera. And the storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food material and the foodstuff, and the feature amount extraction unit using the learned data stored in the storage unit. When the foodstuff identification unit that identifies the foodstuff from the extracted feature amount and the foodstuff identification unit identify the foodstuff, re-learning is performed so that the feature amount related to this identification corresponds to the foodstuff, and the storage unit is stored. A re-learning unit for updating the stored learned data and an order history storage unit for storing the order history of the product ordered by the user are provided, and the foodstuff identification unit is stored in the order history storage unit. Identify the ingredients within the range of products in the order history.
Alternatively, the refrigerator system according to the present invention includes a refrigerator in which a storage chamber for storing food is formed, a camera that captures an image of the storage chamber, and a feature quantity extraction unit that extracts the feature quantity of the image captured by the camera. And the storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food material and the foodstuff, and the feature amount extraction unit using the learned data stored in the storage unit. When the foodstuff identification unit that identifies the foodstuff from the extracted feature amount and the foodstuff identification unit identify the foodstuff, re-learning is performed so that the feature amount related to this identification corresponds to the foodstuff, and the storage unit is stored. The refrigerator includes a re-learning unit that updates the stored learned data and a shelf position detecting unit that detects the position of the shelf in the storage room from the image taken by the camera, and the refrigerator detects the shelf position. A camera moving device for moving the camera in the vertical direction according to the position of the shelf detected by the unit is provided.
Alternatively, the refrigerator system according to the present invention includes a refrigerator in which a storage chamber for storing food is formed, a camera that captures an image of the storage chamber, and a feature quantity extraction unit that extracts the feature quantity of the image captured by the camera. And the storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food material and the foodstuff, and the feature amount extraction unit using the learned data stored in the storage unit. When the foodstuff identification unit that identifies the foodstuff from the extracted feature amount and the foodstuff identification unit identify the foodstuff, re-learning is performed so that the feature amount related to this identification corresponds to the foodstuff, and the storage unit is stored. Each item of the re-learning unit that updates the stored learned data, the foodstuff list output unit that outputs the foodstuff identification result by the foodstuff identification unit as a foodstuff list, and the foodstuff list output by the foodstuff list output unit. The re-learning unit includes a correct / incorrect input unit in which the user can input correct / incorrect data, and the re-learning unit performs re-learning using the input contents to the correct / incorrect input unit and stores the learned data in the storage unit. Is updated, and it is assumed that the correct / incorrect input unit has input the correct answer for the item for which the correct / incorrect is not input when a predetermined fixed time has elapsed since the foodstuff list output unit outputs the foodstuff list.

According to the refrigerator system according to the present invention, it is possible to improve the identification accuracy of the foodstuffs stored in the storage chamber and expand the range of the foodstuffs that can be identified.

It is a block diagram which shows the structure of the refrigerator system which concerns on Embodiment 1 of this invention. It is a front view of the refrigerator provided in the refrigerator system which concerns on Embodiment 1 of this invention. It is a vertical sectional view of the refrigerator which concerns on Embodiment 1 of this invention. It is a perspective view of the camera moving device of the refrigerator which concerns on Embodiment 1 of this invention. It is an enlarged perspective view which shows through the main part of the camera moving apparatus which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the control system of the refrigerator which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the control device of FIG. It is a block diagram which shows the structure of the server apparatus provided in the refrigerator system which concerns on Embodiment 1 of this invention. It is a figure which shows an example of the contents of the trained data of the refrigerator system which concerns on Embodiment 1 of this invention. It is a figure explaining the re-learning of the refrigerator system which concerns on Embodiment 1 of this invention. It is a figure explaining the re-learning of the refrigerator system which concerns on Embodiment 1 of this invention. It is a flow figure which shows an example of the image upload operation of the refrigerator which concerns on Embodiment 1 of this invention. It is a flow diagram which shows an example of the operation at the time of image reception of the server apparatus which concerns on Embodiment 1 of this invention. It is a flow diagram which shows an example of the operation at the time of displaying the foodstuff list of the refrigerator system which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the server apparatus provided in the refrigerator system which concerns on Embodiment 2 of this invention. It is a flow diagram which shows an example of the operation which concerns on the shelf position detection of the server apparatus which concerns on Embodiment 2 of this invention. It is a block diagram which shows the structure of the server apparatus provided in the refrigerator system which concerns on Embodiment 3 of this invention. It is a flow diagram which shows an example of the foodstuff identification operation of the server apparatus which concerns on Embodiment 3 of this invention. It is a block diagram which shows the structure of the 2nd server apparatus provided in the refrigerator system which concerns on Embodiment 4 of this invention. It is a flow diagram which shows an example of the operation at the time of order history reception of the server apparatus provided in the refrigerator system which concerns on Embodiment 4 of this invention.

A mode for carrying out the present invention will be described with reference to the accompanying drawings. In each figure, the same or corresponding parts are designated by the same reference numerals, and duplicate description will be appropriately simplified or omitted. The present invention is not limited to the following embodiments, and can be variously modified without departing from the spirit of the present invention.

Embodiment 1.
1 to 14 relate to the first embodiment of the present invention. FIG. 1 is a block diagram showing a configuration of a refrigerator system. FIG. 2 is a front view of the refrigerator provided in the refrigerator system. FIG. 3 is a vertical sectional view of the refrigerator. FIG. 4 is a perspective view of the camera moving device of the refrigerator. FIG. 5 is an enlarged perspective view showing a main part of the camera moving device through a perspective view. FIG. 6 is a block diagram showing the configuration of the control system of the refrigerator. FIG. 7 is a block diagram showing the configuration of the control device of FIG. FIG. 8 is a block diagram showing a configuration of a server device included in the refrigerator system. FIG. 9 is a diagram showing an example of the contents of the trained data of the refrigerator system. 10 and 11 are diagrams illustrating re-learning using a multi-viewpoint image of a refrigerator system. FIG. 12 is a flow chart showing an example of the image upload operation of the refrigerator. FIG. 13 is a flow chart showing an example of the operation of the server device when receiving an image. FIG. 14 is a flow chart showing an example of the operation of the refrigerator system when the food list is displayed.

In each figure, the dimensional relationship and shape of each component may differ from the actual ones. In addition, the positional relationship between the constituent members (for example, the vertical relationship, etc.) in the specification is, in principle, when the refrigerator / freezer is installed in a usable state.

As shown in FIG. 1, the refrigerator system provided with the refrigerator 100 according to the first embodiment of the present invention includes the refrigerator 100 and the server device 200. The refrigerator 100 and the server device 200 are connected so as to be able to communicate with each other.

The refrigerator 100 is installed, for example, in the house of the user of the refrigerator 100. The server device 200 is, for example, a computer installed in the user's house. The server device 200 may be installed outside the user's house.

The number of refrigerators 100 communicably connected to the server device 200 is not limited to one, and may be plural. Further, the server device 200 may be a cloud server including a plurality of servers (server group).

First, the configuration of the refrigerator 100 according to the first embodiment of the present invention will be described with reference to FIGS. 2 to 6. The refrigerator 100 has a heat insulating box 1 as shown in FIG. The front surface (front surface) of the heat insulating box 1 is opened to form a storage space inside. The heat insulating box body 1 has an outer box, an inner box, and a heat insulating material. The outer box is made of steel. The inner box is made of resin. The inner box is placed inside the outer box. The heat insulating material is, for example, urethane foam or the like, and is filled in the space between the outer box and the inner box. The storage space formed inside the heat insulating box 1 is divided into a plurality of storage chambers for storing and storing food by one or a plurality of partition members.

As shown in FIGS. 2 and 3, here, the refrigerator 100 includes, for example, a refrigerating room 10, a switching room 20, an ice making room 30, a freezing room 40, and a vegetable room 50 as a plurality of storage rooms. These storage chambers are arranged in a heat insulating box 1 in a four-tiered structure in the vertical direction.

The refrigerating chamber 10 is arranged at the uppermost stage of the heat insulating box 1. The switching chamber 20 is arranged on one side below the refrigerating chamber 10 on the left and right sides. The cold insulation temperature zone of the switching chamber 20 can be switched by selecting any one of a plurality of temperature zones. The plurality of temperature zones that can be selected as the cold insulation temperature zone of the switching chamber 20 include, for example, a freezing temperature zone (for example, about -18 ° C.), a refrigerating temperature zone (for example, about 3 ° C.), a chilled temperature zone (for example, about 0 ° C.), and the like. It is in a soft refrigeration temperature range (for example, about -7 ° C). The ice making chamber 30 is arranged adjacent to the side of the switching chamber 20 in parallel with the switching chamber 20, that is, on the left and right other sides below the refrigerating chamber 10.

The freezing chamber 40 is arranged below the switching chamber 20 and the ice making chamber 30. The freezing chamber 40 is mainly for use when the storage object is stored frozen for a relatively long period of time. The vegetable compartment 50 is arranged at the lowermost stage below the freezing chamber 40. The vegetable compartment 50 is mainly for storing vegetables and large PET bottles having a large capacity (for example, 2 L or the like).

A rotary refrigerating room door 7 for opening and closing the opening is provided in the opening formed on the front surface of the refrigerating room 10. Here, the refrigerating room door 7 is a double door type (double door type), and is composed of a right door 7a and a left door 7b. An operation panel 6 is provided on the outer surface of the refrigerator compartment door 7 (for example, the right door 7a) on the front surface of the refrigerator 100.

Each storage room (switching room 20, ice making room 30, freezing room 40, and vegetable room 50) other than the refrigerating room 10 is opened and closed by a drawer-type door. These pull-out doors have a frame fixed to the door and slid against rails horizontally formed on the left and right inner wall surfaces of each storage room to slide the frame in the depth direction (front-back direction) of the refrigerator 100. It can be opened and closed.

As shown in FIG. 3, the refrigerating room 10 is closed on the front side (left side when facing the figure) by the refrigerating room door 7. The cross-sectional view shown in FIG. 3 is based on the cross-sectional view AA'in FIG. 2. Inside the refrigerating chamber 10, one or more refrigerating chamber shelf boards 11 are provided. Here, a case where a plurality of refrigerating room shelf boards 11 are provided is taken as an example. The inside of the refrigerating room 10 is divided into a plurality of spaces (shelves) in the vertical direction by these refrigerating room shelf boards 11. Food is placed on the refrigerating room shelf board 11.

Each refrigerating room shelf board 11 is supported by a shelf board support portion 12 at a predetermined position in the refrigerating room 10. The shelf board support portion 12 is formed so as to project from the inner surface of the side wall of the refrigerating chamber 10 to the inside of the refrigerating chamber 10. The shelf board support portion 12 may be configured so that the vertical position of one refrigerating room shelf board 11 can be changed.

The space below the refrigerating room shelf board 11 at the bottom is a chilled room 14. A chilled case 15 is installed inside the chilled chamber 14. The chilled case 15 can be pulled forward along a guide member (not shown) such as a rail. Further, a door pocket 13 is provided on the inner surface of the refrigerator compartment door 7. Food can also be placed and stored in the door pocket 13.

A camera 9 is installed on the inner surface of the refrigerator compartment door 7. The camera 9 takes an image of the inside of the storage room (here, the refrigerating room 10) from the side of the refrigerating room door 7 and outputs it as a storage room image. The refrigerator 100 according to this embodiment includes a camera moving device 90. The camera moving device 90 is a device that moves the camera 9 in the vertical direction. The camera moving device 90 is provided on the inner surface of the refrigerator compartment door 7.

The camera moving device 90 moves the camera 9 in the vertical direction within the movable range. The movable range is a preset range. Here, the upper end of the movable range is the upper end portion in the refrigerating chamber 10. Further, the lower end of the movable range is a position directly above the lowermost door pocket 13.

Next, the configuration of the camera moving device 90 will be described with reference to FIGS. 4 and 5. The camera moving device 90 includes a stepping motor 91, a worm gear 92, a pinion 93, a rack 94, a guide portion 95, and a camera support portion 98. The camera 9 is fixed to the camera support portion 98. The guide portion 95 is fixed to the refrigerator compartment door 7. The camera support portion 98 is movable with respect to the guide portion 95. The guide portion 95 is arranged along the vertical direction at least over the above-mentioned movable range. The guide unit 95 guides the movement of the camera 9 and the camera support unit 98 over the above-mentioned movable range.

As shown in FIG. 4, a stepping motor 91, a worm gear 92, and a pinion 93 are attached to the camera support portion 98. A rack 94 is attached to the guide portion 95. The rack 94 is arranged along the vertical direction over the above-mentioned movable range.

The stepping motor 91 drives the movement of the camera 9. A worm gear 92 is fixed to the drive shaft of the stepping motor 91. The pinion 93 includes a large gear and a small gear integrally. The large gear and the small gear of the pinion 93 are fixed so that the rotation axes are the same. The large gear of the pinion 93 meshes with the worm gear 92. The small gear of the pinion 93 meshes with the rack 94.

When the worm gear 92 is rotated by the stepping motor 91, the pinion 93 is rotated. When the pinion 93 in the meshed state with the rack 94 rotates, the rotation axis of the pinion 93 moves linearly along the rack 94. The rotation axis of the pinion 93 is rotatably supported by the camera support portion 98. Therefore, by such a rack and pinion mechanism, the camera moving device 90 converts the rotational movement of the stepping motor 91 into a linear operation of the camera support portion 98 with respect to the guide portion 95, and the camera 9 with respect to the refrigerating chamber door 7. Move up and down.

As shown in FIG. 3, the refrigerator 100 includes a control device 8. The control device 8 is housed in the upper part of the refrigerator 100 on the back side, for example. The control device 8 is provided with a control circuit and the like for performing various controls necessary for the operation of the refrigerator 100.

Next, the configuration of the control system of the refrigerator 100 will be described with reference to FIG. The control device 8 includes, for example, a microcomputer, and includes a processor 8a and a memory 8b. The control device 8 controls the refrigerator 100 by executing a preset process by the processor 8a executing the program stored in the memory 8b.

The refrigerator 100 includes a refrigeration cycle circuit that cools the air supplied to each storage chamber. The refrigeration cycle circuit is composed of a compressor 2, a condenser, a throttle device, a cooler, and the like (not shown). The compressor 2 compresses and discharges the refrigerant in the refrigeration cycle circuit. The condenser condenses the refrigerant discharged from the compressor 2. The squeezing device expands the refrigerant flowing out of the condenser. The cooler cools the air supplied to each storage chamber by the refrigerant expanded by the throttle device. The compressor 2 is arranged, for example, in the lower part on the back side of the refrigerator 100.

The refrigerator 100 is formed with an air passage (not shown) for supplying air cooled by the refrigeration cycle circuit to each storage chamber. This air passage is mainly arranged on the back side in the refrigerator 100. The cooler of the refrigeration cycle circuit is installed in this air passage. Further, a blower fan 4 for sending the air cooled by the cooler to each storage chamber is also installed in the air passage.

When the blower fan 4 operates, the air (cold air) cooled by the cooler is sent to the freezing chamber 40, the switching chamber 20, the ice making chamber 30, and the refrigerating chamber 10 through the air passage, and cools these storage chambers. .. The vegetable compartment 50 is cooled by introducing the return cold air from the refrigerator compartment 10 into the vegetable compartment 50 via the return air passage for the refrigerator compartment. The cold air that has cooled the vegetable compartment 50 is returned to the air passage containing the cooler through the return air passage for the vegetable chamber (these return air passages are not shown). Then, it is cooled again by the cooler, and cold air is circulated in the refrigerator 100.

A damper (not shown) is provided in the middle of the route from the air passage to each storage room. Each damper opens and closes a point leading to each storage chamber of the air passage. By changing the open / closed state of the damper, the amount of cold air supplied to each storage chamber can be adjusted. Further, the temperature of the cold air can be adjusted by controlling the operation of the compressor 2.

The refrigerating cycle circuit including the compressor 2 and the cooler provided as described above, the blower fan 4, the air passage, and the damper constitute a cooling means for cooling the inside of the storage chamber.

The operation panel 6 includes an operation unit 6a, a display unit 6b, and a speaker 6c. The operation unit 6a is an operation switch for setting the cold storage temperature of each storage room and the operation mode (thawing mode, etc.) of the refrigerator 100. The display unit 6b is a liquid crystal display that displays various information such as the temperature of each storage chamber. Further, the operation panel 6 may include a touch panel that also serves as an operation unit 6a and a display unit 6b. The speaker 6c is a voice output device for sounding a voice to a user standing around the refrigerator 100, particularly in front of the refrigerator door 7.

The refrigerator 100 includes a thermistor 81 and a door open / close detection switch 82. The thermistor 81 detects the temperature in each storage chamber. The thermistor 81 is installed in each storage room. The door open / close detection switch 82 is an open / close detecting means for detecting the open / closed of the door of the storage room. The door open / close detection switch 82 detects, for example, the opening / closing of the refrigerating room door 7 of the refrigerating room 10. The door open / close detection switch 82 is, for example, a general magnet type switch. That is, the door open / close detection switch 82 detects, for example, the proximity of the magnet embedded in the refrigerator compartment door 7 by a pair of reed switches installed on the refrigerator 100 main body side.

A detection signal of the temperature inside each storage chamber is input from the thermistor 81 to the control device 8. Further, an operation signal from the operation unit 6a of the operation panel 6 is also input to the control device 8. Further, a detection signal from the door open / close detection switch 82 is also input to the control device 8.

Based on the input signal, the control device 8 executes a process of controlling the operation of the compressor 2, the blower fan 4, etc. so that the inside of each storage chamber is maintained at a set temperature. That is, the control device 8 controls the cooling means and the like described above to control the operation of the refrigerator 100. The control device 8 also controls the shooting operation of the camera 9 and the movement of the camera 9 by the camera moving device 90. The image data taken by the camera 9 is input to the control device 8.

Next, the configuration of the refrigerator system including the refrigerator 100 configured as described above will be described with reference to FIGS. 7 and 8 in addition to FIG. As described above, the refrigerator system includes a refrigerator 100 and a server device 200. As shown in FIG. 7, the control device 8 of the refrigerator 100 includes a camera control unit 101, an internal change determination unit 102, an image upload unit 103, and a refrigerator communication unit 110. Further, as shown in FIG. 8, the server device 200 includes a storage unit 201, a feature amount extraction unit 203, a foodstuff identification unit 204, a re-learning unit 207, and a server communication unit 210.

The refrigerator communication unit 110 is for bidirectionally communicating data between the refrigerator 100 and the server device 200. Further, the server communication unit 210 is for bidirectionally communicating data between the server device 200 and the refrigerator 100.

The camera control unit 101 included in the control device 8 of the refrigerator 100 controls the operations of the camera 9 and the camera moving device 90. For example, the camera control unit 101 causes the camera 9 to take a picture of the inside of the refrigerating room 10 when the refrigerating room door 7 is opened and closed. This is because when the refrigerating room door 7 is opened and closed, food in the refrigerating room 10 may be taken in and out, and the storage state in the refrigerating room 10 may change. The camera control unit 101 detects, for example, that the refrigerating room door 7 opened by the door open / close detection switch 82 is closed, and then causes the camera 9 to take a picture while the inside of the refrigerating room 10 is illuminated. ..

The camera control unit 101 causes the camera 9 to take a picture in the refrigerating room 10 in cooperation with the movement of the camera 9 by the camera moving device 90. Next, a shooting operation in which the camera 9 and the camera moving device 90 are linked will be described. When taking a picture in the refrigerator compartment 10, the camera moving device 90 moves the camera 9 from one end to the other end of the above-mentioned movable range while stopping the movement of the camera 9 on the way. Then, the camera 9 takes a picture every time the camera moving device 90 stops the movement of the camera 9.

For example, the camera moving device 90 moves the camera 9 downward from the upper end to the lower end of the above-mentioned movable range. At this time, the camera moving device 90 stops the camera 9 at at least one place between the inner surface of the upper end of the refrigerating chamber 10 and the shelf plate 11 of the refrigerating chamber adjacent to the inner surface of the upper end. Then, the camera 9 takes an image while it is stopped at this position. The image taken by the camera 9 at this time shows the storage status of the refrigerating room shelf board 11 next to the inner surface of the upper end of the refrigerating room 10, that is, the uppermost refrigerating room shelf board 11. After the camera 9 captures the image, the camera moving device 90 resumes the descent of the camera 9.

Further, the camera moving device 90 stops the camera 9 at at least one place between the adjacent refrigerating room shelf boards 11. Then, the camera 9 takes an image while it is stopped at this position. The image taken by the camera 9 at this time shows the storage status of the refrigerating room shelf board 11 immediately below the position of the camera 9. After the camera 9 captures the image, the camera moving device 90 resumes the descent of the camera 9.

This is repeated, and when the camera 9 takes an image of the storage state at the bottom of the refrigerator compartment 10, the camera control unit 101 ends a series of shooting. The moving direction of the camera 9 is not limited to the downward direction described above. That is, the camera control unit 101 may take a picture with the camera 9 while moving the camera 9 upward from the lower end to the upper end of the movable range described above by the camera moving device 90.

The amount of movement of the camera 9 is proportional to the amount of rotation of the stepping motor 91. The rotation amount of the stepping motor 91 can be controlled by using the number of steps. In this embodiment, the camera control unit 101 controls the movement amount of the camera 9 by using the number of steps of the stepping motor 91 of the camera movement device 90. That is, the camera control unit 101 identifies the position where the camera 9 is stopped by counting the number of steps of the stepping motor 91.

When the door open / close detection switch 82 detects the closing operation of the refrigerating room door 7, the camera control unit 101 moves the camera 9 to one end of the movable range described above to correct the zero point of the stepping motor 91. good. That is, when the door open / close detection switch 82 detects the closing operation of the refrigerating room door 7, the camera control unit 101 first causes the camera moving device 90 to move the camera 9 to, for example, the upper end of the above-mentioned movable range. Then, the camera control unit 101 sets the count value of the number of steps of the stepping motor 91 to 0. By doing so, it is possible to correct the deviation between the number of steps of the stepping motor 91 and the actual position of the camera 9.

If the position of the refrigerating room shelf plate 11 is fixed and cannot be changed, the position where the camera 9 is stopped can also be fixed. Therefore, for example, the camera control unit 101 stores information about the position where the camera 9 is stopped, for example, the number of steps of the stepping motor 91 in advance. Then, when the number of steps of the stepping motor 91 reaches a value stored in advance, the camera control unit 101 may stop the camera 9 by the camera moving device 90.

The camera control unit 101 acquires the image in the refrigerating room 10 taken by the camera 9 as described above. The internal change determination unit 102 determines whether or not the image in the refrigerator compartment 10 taken by the camera 9 has changed since the previous shooting. Therefore, the memory 8b of the control device 8 holds at least the image in the refrigerating chamber 10 taken by the camera 9 last time. Then, the internal change determination unit 102 compares the previous image with the image taken by the camera 9 this time, and determines whether or not there has been a change from the time of the previous shooting.

When the internal change determination unit 102 determines that the image taken by the camera 9 has changed since the previous shooting, the image upload unit 103 uploads the image taken by the camera 9 to the server device 200 this time. The image uploaded by the image upload unit 103 to the server device 200 is transmitted to the server device 200 via the refrigerator communication unit 110. In this way, the image upload unit 103 and the refrigerator communication unit 110 constitute a transmission unit that transmits the image taken by the camera 9 to the server device 200.

The server communication unit 210 of the server device 200 receives the image transmitted from the refrigerator 100. The feature amount extraction unit 203 extracts the feature amount of the image received by the server communication unit 210. That is, the feature amount extraction unit 203 extracts the feature amount of the image taken by the camera 9. The foodstuff identification unit 204 identifies the foodstuff from the feature amount extracted by the feature amount extraction unit 203. For the identification of the food material, the learned data obtained by learning the correspondence between the feature amount of the image of the food material and the food material is used. The learned data is stored in advance in the storage unit 201.

FIG. 9 shows an example of this trained data. The trained data is a set of data in which at least one of the shape, size, color, and exterior label of the food material is associated with each food material. In this sense, the storage unit 201 stores in advance each of the foodstuffs in association with at least one of the shape, size, color, and exterior label of the foodstuff.

The trained data stores, for each of the ingredients, the characteristic attributes that can distinguish the ingredient from the other ingredients. Here, the characteristic amount of the food material itself or the container containing the food material is stored in association with the food material. Specifically, for each type of food material, information such as shape (contour), size of the linear axis, representative color (RGB value), and label (character) is used as a feature amount.

The shape of each food material to be stored in the learned data may be, for example, a geometric shape (for example, a cylinder, a trapezoid, etc.) closest to the outer shape of the food material, in addition to the contours given here. In this case, in the identification by the foodstuff identification unit 204, for example, collation with the shape stored in the learned data is performed by a method such as pattern matching. As for the color, in addition to the RGB values of the representative colors, other index values such as lightness and saturation may be used.

The foodstuff identification unit 204 collates the feature amount extracted by the feature amount extraction unit 203 with which foodstuff feature amount matches in the above-mentioned learned data. Then, the food material having the maximum degree of matching with the feature amount in the trained data is used as the identification result. At this time, if the maximum value of the matching degree is not equal to or higher than a certain reference value, in other words, if there is no food material in the trained data in which the matching degree of the feature amount is equal to or higher than the standard value, the image of the feature amount is taken. Ingredients may not be identifiable.

In the learned data illustrated in FIG. 9, the type of foodstuff is identified, but other than that, for example, the product name of the foodstuff may be identified. The feature quantity associated with one food ingredient type or product name does not have to be one value for each attribute. For example, a plurality of values may be associated with each attribute, or a range of values may be associated with each other.

Further, the identification method used by the foodstuff identification unit 204 is not limited to the pattern matching and the like described above. Alternatively, for example, a method such as a neural network may be used. At this time, the trained data is prepared appropriately according to the identification method used by the foodstuff identification unit 204.

When the food material identification unit 204 identifies the food material, the re-learning unit 207 performs re-learning in which the feature amount related to the current identification is associated with the food material. Then, the re-learning unit 207 updates the above-mentioned learned data stored in the storage unit 201 by this re-learning.

In this way, every time the food is identified, a data set for learning the feature amount of the image of the food and the correspondence with the food can be automatically acquired. Then, it is possible to automatically relearn the image used for identifying the ingredients this time as learning data to increase the amount of learning, improve the accuracy of identifying ingredients, and expand the range of identifiable ingredients. be.

In the configuration example described in this embodiment, as shown in FIG. 8, the server device 200 further includes a food ingredient list output unit 205 and a correct / incorrect input unit 206. The foodstuff list output unit 205 outputs the foodstuff identification result by the foodstuff identification unit 204 as a foodstuff list. The foodstuff list output by the foodstuff list output unit 205 is transmitted from the server communication unit 210 to the refrigerator 100.

The food list transmitted from the server device 200 is received by the refrigerator communication unit 110 of the refrigerator 100. The food list received by the refrigerator communication unit 110 is notified to the user by the operation panel 6 of the refrigerator 100. For example, the control device 8 causes the speaker 6c to reproduce the contents of the foodstuff list received by the refrigerator communication unit 110, and informs the user of the contents of the foodstuff list by voice. In addition, for example, the control device 8 may display the contents of the foodstuff list received by the refrigerator communication unit 110 on the display unit 6b as a character message or the like. In addition, for example, using a mobile terminal such as a smartphone used by the user, a television or a personal computer installed in the user's house, the contents of the food list are notified by voice, one or both of the characters. May be good. When notifying the contents of the foodstuff list, an image of a region specified by the region specifying unit 202 described later for each item of the foodstuff list may be displayed together with, for example, a display unit 6b or the like.

The correctness input unit 206 allows the user to input the correctness of each item of the foodstuff list output by the foodstuff list output unit 205. In the configuration example of this embodiment, the user does not directly input to the correct / incorrect input unit 206 of the server device 200, but instead uses the operation panel 6 of the refrigerator 100, a mobile terminal such as a smartphone, a television, a personal computer, or the like. Enter the correctness for each item in the food list via the input interface. That is, the user operates, for example, the operation unit 6a of the operation panel 6 to input correct / incorrect information as to whether the identification result is correct or incorrect for each item of the food ingredient list. At this time, if the identification result included in the foodstuff list is incorrect, not only may the user simply input that the identification result is incorrect, but the user may be able to correct the identification result.

The correct / incorrect information input to the operation panel 6 of the refrigerator 100 is transmitted from the refrigerator communication unit 110 to the server device 200. The correct / incorrect information transmitted from the refrigerator 100 is received by the server communication unit 210 of the server device 200. The correct / incorrect information received by the server communication unit 210 is input to the correct / incorrect input unit 206.

The re-learning unit 207 performs re-learning using the input contents to the correct / incorrect input unit 206. Then, the re-learning unit 207 updates the above-mentioned learned data stored in the storage unit 201 by this re-learning. In the re-learning using the input contents to the correct / incorrect input unit 206, the re-learning unit 207 corresponds the food material input to the correct / incorrect input unit 206 with the feature amount related to the present identification and the food material (corresponding to the food material). Tie) stronger.

On the other hand, the re-learning unit 207 deletes the feature amount related to the present identification from the above-mentioned learned data for the food material input as an error in the correct / incorrect input unit 206. When the corrected food material information is input to the correct / incorrect input unit 206, the re-learning unit 207 first deletes the current feature amount that led to the erroneous determination from the above-mentioned learned data, and then corrects the food material. Relearning is performed by associating this with the feature amount of this time.

The correct / incorrect input unit 206 may assume that "correct answer" is input for an item for which correct / incorrect is not input when a predetermined fixed time has elapsed after the food list output unit 205 outputs the food list. good. For example, on the correct / incorrect information input screen, "correct answer" is selected by default for each item in the food list. Then, when a certain period of time has elapsed, the correct / incorrect information selected at that time is input to the correct / incorrect input unit 206. By doing this, it is possible to automatically input the "correct answer" for the item that has been changed from the default selection state and the "error" is not selected. By doing so, it is possible to reduce the labor of the user to input correct / incorrect information.

In the configuration example described in this embodiment, as shown in FIG. 8, the server device 200 further includes an area specifying unit 202. The area specifying unit 202 identifies an area in which the food material exists in the image taken by the camera 9. This region can be specifically specified, for example, by detecting an edge in an image. Then, the feature amount extraction unit 203 extracts the feature amount for each area specified by the area specifying unit 202. Further, the food ingredient identification unit 204 identifies the foodstuff for each region specified by the region identification unit 202.

Then, the camera 9 can take an image from a plurality of different viewpoints about the same region specified by the region specifying unit 202. Specifically, in the configuration example of this embodiment, the position of the camera 9 can be moved by the camera moving device 90 as described above. With this camera moving device 90, the camera 9 can shoot the same region as described above from a plurality of different viewpoints in the vertical direction.

The camera 9 may be moved in the left-right direction. By doing so, the camera 9 can shoot the same region as described above from a plurality of different viewpoints in the left-right direction. Further, the cameras 9 may be arranged at a plurality of different positions.

The feature amount extraction unit 203 extracts each feature amount of the plurality of viewpoint images. The multi-viewpoint image is an image taken by the camera 9 from the above-mentioned plurality of viewpoints with respect to the above-mentioned same area specified by the area specifying unit 202. When the foodstuff identification unit 204 identifies the foodstuff, the re-learning unit 207 performs re-learning in which each feature amount of the above-mentioned multi-viewpoint image extracted by the feature amount extraction unit 203 is associated with the foodstuff. Then, by this re-learning, the above-mentioned learned data stored in the storage unit 201 is updated.

In this way, by using each feature amount of the above-mentioned multi-viewpoint image as a data set at the time of re-learning, more learning data can be automatically acquired. Then, it is possible to further increase the amount of learning by re-learning, further improve the accuracy of food identification, and expand the range of foods that can be identified.

Next, with reference to FIGS. 10 and 11, the operation obtained by re-learning such the same region using images taken from a plurality of viewpoints will be described. FIG. 10 is an example of explaining a range that can be identified by the trained data before re-learning. FIG. 11 is an example for explaining a range that can be identified by the trained data after re-learning from the example of FIG. 10 and 11 show an example in which the food material to be identified is milk in a so-called gable top type paper carton.

In the trained data before re-learning, it is assumed that only the feature amount when the surface A shown by the thick line in FIG. 10 is viewed from the front is associated with milk. In this case, if the image taken by the camera 9 shows the state in which the A side is viewed from the front, the food ingredient identification unit 204 can identify the foodstuff as milk.

Further, as shown by being surrounded by a broken line in FIG. 10, in an image taken from a viewpoint in which the A surface is slightly oblique from the front, the A surface is distorted and deformed and appears as the A'plane. Even in such an image showing the A'plane, if the degree of matching between the feature amount of the A'plane and the feature amount of the A plane is equal to or higher than a certain level, the foodstuff identification unit 204 can identify the foodstuff as milk.

On the other hand, as described above, in the trained data in the state of FIG. 10, only the feature amount in the state where the A side is viewed from the front is associated with milk. Therefore, the foodstuff identification unit 204 cannot identify the foodstuff as milk from the image of the viewpoint when the B side is viewed from the front and the image of the viewpoint when the C side is viewed from the front, which are the sides adjacent to the A side, respectively. .. Therefore, in the state before re-learning in FIG. 10, the range shown by the broken line in FIG. 10 is the range that can be identified as milk.

In the image showing the identifiable A'plane, the B'plane in which the B plane is deformed or the C'plane in which the C plane is deformed is also shown. In the refrigerator system of this embodiment, when milk is identified from the image showing the A'plane and the B'plane, the feature amount extracted from the image showing the A'plane and the B'plane corresponds to the milk. The attached re-learning is performed by the re-learning unit 207. Further, when milk is identified from the image showing the A'plane and the C'plane, the re-learning in which the feature amount extracted from the image showing the A'plane and the C'plane is associated with the milk is re-learned. This is done by the learning unit 207.

In this way, in the trained data after re-learning, the feature amount in the range surrounded by the thick line shown in FIG. 11 is associated with milk. That is, in addition to the features when the A surface is viewed from the front, the A'plane when the A surface is viewed from an angle, the B'plane when the B surface is viewed from an angle, and the C surface are viewed from an angle. Each feature of the C'plane is associated with milk in the trained data after retraining.

In the identification using the learned data after the re-learning, the degree of matching with the feature amount of the B'plane becomes larger than that before the re-learning, so that the image of the viewpoint when the B-plane is viewed from the front can also be seen. The foodstuff identification unit 204 can identify the foodstuff as milk. Further, from the image of the viewpoint when the C surface is viewed from the front, the degree of matching with the feature amount of the C'plane becomes larger than that before the re-learning, so that the food material identification unit 204 can identify the food material as milk. Therefore, in the state after re-learning in FIG. 11, milk can be identified up to the range shown by the broken line in the figure.

In this way, according to the refrigerator system according to this embodiment, the discriminable range can be expanded by updating the learned data by re-learning. By repeating such re-learning, the food can be finally identified regardless of the orientation in which the food is placed in the refrigerator compartment 10.

Even if the camera 9 can only take images from the same viewpoint, if the orientation of the food placed in the refrigerator 10 changes when the user puts in or takes out the food in the refrigerator 10, the camera The surface of the food shown in the image taken by 9 also changes. Therefore, even when the camera 9 can capture only images from the same viewpoint, it is possible to expand the discriminable range to a certain extent by re-learning.

Next, an example of the main operation of the refrigerator system configured as described above will be described with reference to FIGS. 12 to 14. First, FIG. 12 shows an example of an operation of uploading an image in the refrigerating room 10 from the refrigerator 100 to the server device 200.

In step S11, the camera control unit 101 confirms whether the refrigerating room door 7 has been opened or closed, or whether a predetermined fixed time has elapsed since the previous in-compartment shooting. Whether or not the refrigerating room door 7 has been opened or closed is determined based on the detection result of the door open / close detection switch 82. If the refrigerating room door 7 is not opened and closed and a certain period of time has not passed since the previous shooting inside the refrigerator, the process of step S11 is repeated. Then, when the refrigerating room door 7 is opened or closed, or when a certain period of time has elapsed since the previous shooting inside the refrigerator, the process proceeds to step S12.

In step S12, the camera control unit 101 controls the camera 9 and the camera moving device 90 to photograph the inside of the refrigerating room 10. After step S12, the process proceeds to step S13.

In step S13, the internal change determination unit 102 compares the image taken by the camera 9 in step S12 with the image taken last time. If there is no change in the image from the previous time, the image taken this time is not uploaded and the series of operations ends. On the other hand, if the image has changed from the previous time, the process proceeds to step S14.

In step S14, the image upload unit 103 transmits the image taken by the camera 9 in step S12 to the server device 200 via the refrigerator communication unit 110. When the process of step S14 is completed, the series of operations is completed.

Next, FIG. 13 shows an example of an operation of identifying foodstuffs from the image in the refrigerator compartment 10 uploaded to the server device 200. When the server communication unit 210 of the server device 200 receives the image transmitted from the refrigerator 100, in step S21, the area specifying unit 202 identifies the area where the foodstuff in the image exists. After step S21, the process proceeds to step S22.

In step S22, the feature amount extraction unit 203 extracts the feature amount from the image for each area specified by the area specifying unit 202 in step S21. After step S22, the process proceeds to step S23.

In step S23, the foodstuff identification unit 204 uses the above-mentioned learned data stored in the storage unit 201 to extract the feature amount extracted by the feature amount extraction unit 203 in step S22 for each area specified by the area identification unit 202. Identify the ingredients from. After step S23, the process proceeds to step S24.

In step S24, the re-learning unit 207 performs re-learning in which the food material identified by the food material identification unit 204 in step S23 and the feature amount extracted by the feature amount extraction unit 203 in step S22 are associated with each other. Then, the re-learning unit 207 updates the above-mentioned learned data stored in the storage unit 201 by this re-learning. After step S24, the process proceeds to step S25.

In step S25, the foodstuff list output unit 205 lists the foodstuff identification results by the foodstuff identification unit 204 in step S23 and outputs it as a foodstuff list. The foodstuff list output by the foodstuff list output unit 205 is transmitted from the server communication unit 210 to the refrigerator 100 and notified to the user. When the process of step S25 is completed, the series of operations is completed.

Next, FIG. 14 shows an example of the operation after the foodstuff list output unit 205 outputs the foodstuff list in step S25 of FIG. The output food list is transmitted to the refrigerator 100 and notified to the user as described above. The user can confirm the contents of the foodstuff list on the operation panel 6 or the like, and input correct / incorrect information as to whether or not the identification result is correct for each foodstuff item in the foodstuff list. When the user inputs the correct / incorrect information, the correct / incorrect information is sent to the server device 200 and input to the correct / incorrect input unit 206. In step S31, the correct / incorrect input unit 206 confirms whether or not correct / incorrect information has been input to the correct / incorrect input unit 206. If there is an input in the correct / incorrect input unit 206, the process proceeds to step S32.

In step S32, the re-learning unit 207 confirms whether or not the input content to the correct / incorrect input unit 206 is that the food ingredient list is incorrect. If the input content to the correct / incorrect input unit 206 does not indicate that there is an error in the food ingredient list, the process proceeds to step S33.

If the input content to the correct / incorrect input unit 206 does not indicate that the food ingredient list is incorrect, it means that the user has determined that the identification result of the foodstuff identification unit 204 output as the foodstuff list is correct. Therefore, in step S33, the re-learning unit 207 strengthens the correspondence (linkage) between the feature amount related to the present identification and the food material for the food material input to the correct / incorrect input unit 206 as the correct answer. Then, the re-learning unit 207 updates the above-mentioned learned data of the storage unit 201 to the contents after strengthening. When the process of step S33 is completed, the series of operations is completed.

On the other hand, if the content input to the correct / incorrect input unit 206 in step S32 is that there is an error in the food ingredient list, the process proceeds to step S34. In step S34, the re-learning unit 207 deletes the feature amount related to the present identification from the above-mentioned learned data for the food material input as an error in the correct / incorrect input unit 206. Then, the re-learning unit 207 updates the above-mentioned learned data of the storage unit 201 to the contents after deletion. When the process of step S34 is completed, the series of operations is completed.

On the other hand, if there is no input in the correct / incorrect input unit 206 in step S31, the process proceeds to step S35. In step S35, the correct / incorrect input unit 206 confirms whether or not a predetermined fixed time has elapsed since the food list output unit 205 outputs the food list. If a certain time has not passed since the output of the food ingredient list, the process returns to step S31.

On the other hand, when a certain time has passed after the output of the foodstuff list, it is assumed that the correct / incorrect input unit 206 has input that the identification result of the foodstuff list is correct. Therefore, in this case, the process proceeds to step S33, and the re-learning and the update of the learned data for strengthening the above-mentioned connection are performed.

Embodiment 2.
15 and 16 relate to the second embodiment of the present invention. FIG. 15 is a block diagram showing a configuration of a server device included in the refrigerator system. FIG. 16 is a flow chart showing an example of the operation related to the shelf position detection of the server device.

In the second embodiment described here, when the position of the refrigerating room shelf board can be changed in the configuration of the first embodiment described above, the position of the refrigerating room shelf board is detected from the image taken by the camera, and the camera The amount of movement of the camera is adjusted. Hereinafter, the refrigerator according to the second embodiment will be described focusing on the differences from the first embodiment. The other configurations for which the description is omitted are basically the same as those in the first embodiment.

In the refrigerator system according to this embodiment, as shown in FIG. 15, the server device 200 further includes a shelf position detecting unit 208. The shelf position detecting unit 208 detects the position of the shelf in the storage room, that is, the refrigerating room 10 in this case, from the image taken by the camera 9. In this embodiment, first, the camera moving device 90 moves the camera 9 over the entire movable range described above, and the camera 9 takes an image. The shooting position at this time is not particularly limited as long as it can finally cover the entire range of the refrigerating room 10 that can be shot by the camera 9. Then, the refrigerator communication unit 110 transmits the image thus taken to the server device 200.

The server communication unit 210 of the server device 200 receives the image transmitted from the refrigerator 100. Then, the shelf position detection unit 208 analyzes the image received by the server communication unit 210 and detects the current position of the refrigerating room shelf board 11. Information regarding the position of the refrigerating room shelf board 11 detected by the shelf position detecting unit 208 is transmitted to the refrigerator 100 by the server communication unit 210.

The refrigerator communication unit 110 receives the information regarding the position of the refrigerating room shelf board 11 transmitted from the server device 200. Then, the camera control unit 101 controls the camera moving device 90 to move the camera 9 in the vertical direction according to the position of the shelf detected by the shelf position detecting unit 208.

Next, an example of the shelf position detection operation will be described with reference to FIG. When the server communication unit 210 of the server device 200 receives the image transmitted from the refrigerator 100, the shelf position detection unit 208 detects the position of the refrigerating room shelf board 11 from the image in step S41. After step S41, the process proceeds to step S42. In step S42, the server communication unit 210 transmits information regarding the position of the refrigerating room shelf board 11 detected by the shelf position detection unit 208 in step S41 to the refrigerator 100. When the process of step S42 is completed, the series of operations is completed.

The position of the refrigerating room shelf board 11 may be detected by measuring the distance from the camera 9 to the tip of the refrigerating room shelf board 11 on the refrigerating room door 7 side with an infrared sensor or the like. In addition, a firefly phosphorescent body is attached to the tip of the refrigerating room shelf board 11 on the cold room door 7 side, the lighting of the refrigerating room 10 is turned off, a picture is taken with the camera 9, and the position of the refrigerating room shelf board 11 is located from the shining place. It may be detected.

Even in the refrigerator simtem configured as described above, the same effect as that of the first embodiment can be obtained. Further, even if the position of the refrigerating room shelf board 11 is changed or the refrigerating room shelf board 11 is removed, an appropriate image of the inside of the refrigerator can be taken according to the position of the refrigerating room shelf board 11.

Embodiment 3.
17 and 18 relate to the third embodiment of the present invention. FIG. 17 is a block diagram showing a configuration of a server device included in the refrigerator system. FIG. 18 is a flow chart showing an example of the foodstuff identification operation of the server device.

In the third embodiment described here, in the configuration of the first embodiment or the second embodiment described above, it is possible to identify whether to identify the type of foodstuff or the product name, or to identify according to the amount of learned data. The method used is changed. Hereinafter, the refrigerator according to the third embodiment will be described by taking as an example a case based on the configuration of the first embodiment and focusing on the differences from the first embodiment. The configuration omitted from the description is basically the same as that of the first embodiment or the second embodiment.

In the refrigerator system according to this embodiment, as shown in FIG. 17, the server device 200 includes an identification method determination unit 209. The identification method determination unit 209 determines the identification method used in the foodstuff identification unit 204. Then, the food ingredient identification unit 204 identifies the foodstuff from the image taken by the camera 9 by using the identification method determined by the identification method determination unit 209. The learned data to be used and the feature amount extraction method in the feature amount extraction unit 203 are also changed according to the identification method used in the foodstuff identification unit 204.

First, the foodstuff identification unit 204 uses different identification methods depending on whether the type of foodstuff is identified or the product name of the foodstuff is identified. Specifically, when identifying the type of foodstuff, the identification method determination unit 209 determines the identification method to be deep learning. Then, the foodstuff identification unit 204 acquires the learned data for deep learning from the storage unit 201 and identifies the foodstuff.

On the other hand, when identifying the product name of the food material, the identification method determination unit 209 determines the identification method to be pattern matching using SIFT (Scale-Invariant Features Trendform), SURF (Speeded Up Robust Foodures), or the like. In this case, the feature amount extraction unit 203 detects the feature points of the image taken by the camera 9 by using a technique such as SIFT, and extracts the feature amount from the feature points. This feature quantity is a vector quantity. Then, the foodstuff identification unit 204 acquires, for example, learned data for SIFT from the storage unit 201 and identifies the foodstuff. It is conceivable that the user can select whether to identify the type of food or the product name, for example, by using the operation panel 6.

Next, the foodstuff identification unit 204 uses different identification methods depending on whether or not the amount of the above-mentioned learned data stored in the storage unit 201 is equal to or more than a preset reference amount. Specifically, when the amount of the above-mentioned learned data stored in the storage unit 201 is equal to or greater than the above-mentioned reference amount, the identification method determination unit 209 determines the identification method to be deep learning. Then, the foodstuff identification unit 204 acquires the learned data for deep learning from the storage unit 201 and identifies the foodstuff.

On the other hand, when the amount of the above-mentioned learned data stored in the storage unit 201 is less than the above-mentioned reference amount, the identification method determination unit 209 determines the identification method to be pattern matching using SIFT, SURF, or the like. In this case, the feature amount extraction unit 203 extracts the feature amount from the image taken by the camera 9 by using a technique such as SIFT. Then, the foodstuff identification unit 204 acquires, for example, learned data for SIFT from the storage unit 201 and identifies the foodstuff.

Next, an example of the operation related to the determination of the identification method will be described with reference to FIG. When starting the food ingredient identification process, first, in step S51, the identification method determination unit 209 confirms whether or not to identify the product name of the foodstuff from the image. When identifying the product name of the food material from the image, the process proceeds to step S52. Then, in step S52, the identification method determination unit 209 determines the identification method as the first identification method. The first identification method is specifically pattern matching using SIFT, SURF, or the like.

On the other hand, when the product name of the food material is not identified from the image in step S51 and the type of the food material is identified, the process proceeds to step S53. In step S53, the identification method determination unit 209 confirms whether or not the amount of the above-mentioned learned data stored in the storage unit 201 is less than the above-mentioned reference amount. If the amount of trained data is less than the reference amount, the process proceeds to step S54. Then, in step S54, the identification method determination unit 209 determines the identification method as the second identification method. The second identification method is specifically pattern matching using SIFT, SURF, or the like.

On the other hand, if the amount of trained data in step S53 is equal to or greater than the reference amount, the process proceeds to step S55. Then, in step S55, the identification method determination unit 209 determines the identification method as the third identification method. The third identification method is specifically deep learning. Then, the feature amount extraction unit 203 and the food material identification unit 204 perform processing according to the method determined by the identification method determination unit 209 as described above.

Even in the refrigerator configured as described above, the same effect as that of the first embodiment or the second embodiment can be obtained. Furthermore, by changing the method used for identification according to whether the type of food or the product name is identified or the amount of learned data, it is possible to use an identification method that is more suitable for the situation. , The identification accuracy can be improved.

Embodiment 4.
19 and 20 relate to the fourth embodiment of the present invention. FIG. 19 is a block diagram showing a configuration of a second server device included in the refrigerator system. FIG. 20 is a flow chart showing an example of the operation of the server device included in the refrigerator system at the time of receiving the order history.

In the fourth embodiment described here, in any of the configurations of the first to third embodiments described above, the order history of the user on the Internet shopping site or the like is also used to identify the foodstuff. Is. Hereinafter, the refrigerator according to the fourth embodiment will be described by taking as an example a case based on the configuration of the first embodiment and focusing on the differences from the first embodiment. The configuration omitted from the description is basically the same as that of any one of the first to third embodiments.

The refrigerator system according to this embodiment includes a second server device 300. The second server device 300 is, for example, an order history management server for an Internet shopping site. The user can, for example, operate the operation unit 6a of the refrigerator 100, a mobile terminal such as a smartphone owned by the user, a personal computer, or the like, and order a desired product (foodstuff) through the second server device 300, for example.

As shown in FIG. 19, the second server device 300 includes an order history storage unit 301 and an order history transmission unit 302. The order history storage unit 301 stores the order history of the product ordered by the user. The order history transmission unit 302 transmits the user's order history stored in the order history storage unit 301 to the server device 200. The order history transmitted from the second server device 300 is received by the server communication unit 210 of the server device 200. The foodstuff identification unit 204 adjusts the range of learned data used for identification based on the order history received by the server communication unit 210. Specifically, for example, the foodstuff identification unit 204 identifies the foodstuff within the range of the products in the order history stored in the order history storage unit 301.

Next, an example of the adjustment operation of food ingredient identification using the order history will be described with reference to FIG. 20. When the server communication unit 210 of the server device 200 receives the order history transmitted from the second server device 300, in step S61, the foodstuff identification unit 204 uses the order history received by the server communication unit 210 for identification. Adjust the range of trained data to use. When the process of step S61 is completed, the series of operations is completed.

Even in the refrigerator configured as described above, the same effect as that of any one of the first to third embodiments can be obtained. Further, it is considered that the foodstuffs purchased by the user are basically stored in the storage room of the refrigerator 100. Therefore, it is expected that the identification accuracy will be improved by identifying the ingredients within the range of the products in the order history stored in the order history storage unit 301.

1 Insulation box 2 Compressor 4 Blower fan 6 Operation panel 6a Operation unit 6b Display unit 6c Speaker 7 Refrigerator room door 7a Right door 7b Left door 8 Control device 8a Processor 8b Memory 9 Camera 10 Refrigerator room 11 Refrigerator room shelf 12 Shelf Plate support 13 Door pocket 14 Chilled room 15 Chilled case 20 Switching room 30 Ice making room 40 Refrigerator room 50 Vegetable room 81 Thermista 82 Door open / close detection switch 90 Camera moving device 91 Stepping motor 92 Worm gear 93 Pinion 94 Rack 95 Guide part 98 Camera support Unit 100 Refrigerator 101 Camera control unit 102 Refrigerator change determination unit 103 Image upload unit 110 Refrigerator communication unit 200 Server device 201 Storage unit 202 Area identification unit 203 Feature quantity extraction unit 204 Foodstuff identification unit 205 Foodstuff list output unit 206 Correct / incorrect input unit 207 Re-learning unit 208 Shelf position detection unit 209 Identification method determination unit 210 Server communication unit 300 Second server device 301 Order history storage unit 302 Order history transmission unit

Claims (8)

A refrigerator with a storage room for storing food,
A camera that captures images in the storage room,
A feature amount extraction unit that extracts the feature amount of the image taken by the camera, and a feature amount extraction unit.
A storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food and the food.
Using the learned data stored in the storage unit, a foodstuff identification unit that identifies foodstuffs from the feature amount extracted by the feature amount extraction unit, and a foodstuff identification unit.
When the foodstuff identification unit identifies the foodstuff, the re-learning unit performs re-learning in which the feature amount related to the current identification is associated with the foodstuff, and updates the learned data stored in the storage unit. ,
It is provided with an area specifying portion for specifying an area where foodstuffs are present in an image taken by the camera .
The feature amount extraction unit extracts the feature amount for each region specified by the region specifying unit.
The foodstuff identification unit identifies foodstuffs for each region specified by the region identification unit, and the foodstuff identification unit identifies the foodstuff.
The camera can take an image from a plurality of different viewpoints about the same area specified by the area specifying unit.
The feature amount extraction unit extracts each feature amount of the multi-viewpoint image taken by the camera from the plurality of viewpoints in the same area.
When the foodstuff identification unit identifies the foodstuff, the re-learning unit performs re-learning in which each feature amount of the multi-viewpoint image extracted by the feature amount extraction unit is associated with the foodstuff, and the storage is performed. A refrigerator system that updates the learned data stored in the unit . A refrigerator with a storage room for storing food,
A camera that captures images in the storage room,
A feature amount extraction unit that extracts the feature amount of the image taken by the camera, and a feature amount extraction unit.
A storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food and the food.
Using the learned data stored in the storage unit, a foodstuff identification unit that identifies foodstuffs from the feature amount extracted by the feature amount extraction unit, and a foodstuff identification unit.
When the foodstuff identification unit identifies the foodstuff, the re-learning unit performs re-learning in which the feature amount related to the current identification is associated with the foodstuff, and updates the learned data stored in the storage unit. , Equipped with
The foodstuff identification unit is a refrigerator system that uses different identification methods depending on whether the type of foodstuff is identified or the product name of the foodstuff is identified. A refrigerator with a storage room for storing food,
A camera that captures images in the storage room,
A feature amount extraction unit that extracts the feature amount of the image taken by the camera, and a feature amount extraction unit.
A storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food and the food.
Using the learned data stored in the storage unit, a foodstuff identification unit that identifies foodstuffs from the feature amount extracted by the feature amount extraction unit, and a foodstuff identification unit.
When the foodstuff identification unit identifies the foodstuff, the re-learning unit performs re-learning in which the feature amount related to the current identification is associated with the foodstuff, and updates the learned data stored in the storage unit. , Equipped with
The foodstuff identification unit is a refrigerator system that uses different identification methods depending on whether or not the amount of the learned data stored in the storage unit is equal to or greater than a preset reference amount. A refrigerator with a storage room for storing food,
A camera that captures images in the storage room,
A feature amount extraction unit that extracts the feature amount of the image taken by the camera, and a feature amount extraction unit.
A storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food and the food.
Using the learned data stored in the storage unit, a foodstuff identification unit that identifies foodstuffs from the feature amount extracted by the feature amount extraction unit, and a foodstuff identification unit.
When the foodstuff identification unit identifies the foodstuff, the re-learning unit performs re-learning in which the feature amount related to the current identification is associated with the foodstuff, and updates the learned data stored in the storage unit. ,
It is equipped with an order history storage unit that stores the order history of the products ordered by the user.
The foodstuff identification unit is a refrigerator system that identifies foodstuffs within the range of products in the order history stored in the order history storage unit . A refrigerator with a storage room for storing food,
A camera that captures images in the storage room,
A feature amount extraction unit that extracts the feature amount of the image taken by the camera, and a feature amount extraction unit.
A storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food and the food.
Using the learned data stored in the storage unit, a foodstuff identification unit that identifies foodstuffs from the feature amount extracted by the feature amount extraction unit, and a foodstuff identification unit.
When the foodstuff identification unit identifies the foodstuff, the re-learning unit performs re-learning in which the feature amount related to the current identification is associated with the foodstuff, and updates the learned data stored in the storage unit. ,
A shelf position detecting unit for detecting the position of a shelf in the storage room from an image taken by the camera is provided.
The refrigerator is a refrigerator system including a camera moving device that moves the camera in the vertical direction according to the position of the shelf detected by the shelf position detecting unit . A foodstuff list output unit that outputs the foodstuff identification result by the foodstuff identification unit as a foodstuff list, and
Further provided with a correct / incorrect input unit that allows the user to input the correctness of each item of the food list output by the food list output unit.
The re-learning unit performs re-learning using the input contents to the correct / incorrect input unit, and updates the learned data stored in the storage unit according to any one of claims 1 to 5. The refrigerator system described. A refrigerator with a storage room for storing food,
A camera that captures images in the storage room,
A feature amount extraction unit that extracts the feature amount of the image taken by the camera, and a feature amount extraction unit.
A storage unit that stores in advance the learned data that learned the correspondence between the feature amount of the image of the food and the food.
Using the learned data stored in the storage unit, a foodstuff identification unit that identifies foodstuffs from the feature amount extracted by the feature amount extraction unit, and a foodstuff identification unit.
When the foodstuff identification unit identifies the foodstuff, the re-learning unit performs re-learning in which the feature amount related to the current identification is associated with the foodstuff, and updates the learned data stored in the storage unit. ,
A foodstuff list output unit that outputs the foodstuff identification result by the foodstuff identification unit as a foodstuff list, and
It is equipped with a correct / incorrect input unit that allows the user to input the correctness of each item of the food list output by the food list output unit.
The re-learning unit performs re-learning using the input contents to the correct / incorrect input unit, updates the learned data stored in the storage unit, and updates the learned data.
The correct / incorrect input unit is a refrigerator system in which correct answers are input for items for which correct / incorrect is not input when a predetermined fixed time has elapsed after the food list output unit outputs the food list. Further equipped with a server device provided so as to be able to communicate with the refrigerator,
The feature amount extraction unit, the storage unit, the foodstuff identification unit, and the relearning unit are provided in the server device.
The refrigerator system according to any one of claims 1 to 7 , wherein the refrigerator includes a transmission unit that transmits an image taken by the camera to the server device.

Images