JP7482690B2 - Cooking system, cooking method, learning device and learning method - Google Patents

Description

The technology disclosed in this specification relates to a cooking system, a cooking method, a learning device, and a learning method.

Patent document 1 discloses a heating control device that captures an image of the vicinity of a heating part (e.g., a burner) that is affected by the amount of heat from the heating part (referred to as the area above the cooking device in patent document 1), and reduces the amount of heat from the heating part if it is determined from the captured image that the amount of heat is excessive.

JP 2015-230148 A

The heating control device of Patent Document 1 determines that the amount of heat is excessive when the flame heating the cooking container extends beyond the outer periphery of the cooking container by a predetermined amount or more. In that case, the heating control device of Patent Document 1 reduces the amount of heat from the heating unit, thereby ensuring the safety of the user when the heating unit is heating. However, even after heating has stopped, the area near the heating unit remains at a high temperature for a certain period of time due to residual heat from heating. For this reason, especially after heating has stopped, the user may come into contact with the area near the heating unit without realizing that the area near the heating unit is at a high temperature. This specification provides a heating and cooking system that can notify the user of the temperature near the heating unit even after heating has stopped.

The cooking system disclosed in this specification includes a heating unit, an image acquisition unit, a nearby temperature estimation unit, and a nearby temperature information notification unit. The heating unit heats an object to be heated. The image acquisition unit acquires the image near the heating unit from an imaging device configured to capture an image near the heating unit, which is an image including a state near the heating unit that is heating the object to be heated. The nearby temperature estimation unit inputs the image near the heating unit acquired by the image acquisition unit to a trained learning device that has performed machine learning to estimate the nearby temperature, which is the temperature near the heating unit, based on the image near the heating unit, and acquires the nearby temperature from the learning device by executing arithmetic processing of the trained learning device. The nearby temperature information notification unit notifies a user of nearby temperature information, which is information related to the nearby temperature, based on the nearby temperature estimated by the nearby temperature estimation unit.

According to the above-mentioned cooking system, the nearby temperature is estimated based on the image of the vicinity of the heating unit, which is an image including the condition of the vicinity of the heating unit. Therefore, the above-mentioned heating amount value system can notify the user of the temperature of the vicinity of the heating unit even after heating has stopped.

The cooking system according to the above aspect may further include a heating information acquisition unit that acquires heating information, which is information regarding the heating performed by the heating unit on the heating object. In this case, the learning device may perform machine learning to estimate the nearby temperature based on the heating information as well. The nearby temperature estimation device may also input the heating information acquired by the heating information acquisition unit to the learning device and acquire the nearby temperature from the learning device by executing the arithmetic processing of the learned learning device.

For example, if the heating unit heats the heating object with a large amount of heat, the nearby temperature will be high. On the other hand, if the heating unit heats the heating object with a small amount of heat, the nearby temperature will be low. Also, if the heating time that the heating unit heats the heating object is long, the nearby temperature will be high. On the other hand, if the heating time that the heating unit heats the heating object is short, the nearby temperature will be low. In this way, the heating information, which is information related to the heating performed by the heating unit on the heating object, affects the nearby temperature. According to the above-mentioned heating and cooking system, the estimation accuracy of the nearby temperature can be improved by estimating the nearby temperature based on the heating information that affects the nearby temperature.

In the cooking system according to one aspect described above, the heating information may include a post-heating time that is the time that has elapsed since the heating unit finished heating the object to be heated.

For example, if the time since heating is short, residual heat from the heating unit heating the heated object still remains near the heating unit, and the nearby temperature is high. On the other hand, if the time since heating is long, the residual heat is dissipated, and the nearby temperature is low. In this way, the time since heating affects the nearby temperature. According to the above-mentioned cooking system, the estimation accuracy of the nearby temperature can be improved by estimating the nearby temperature based on the time since heating, which affects the nearby temperature.

In the cooking system according to the above aspect, the imaging device may be configured to be capable of capturing a user status image, which is an image including the status of the user. In that case, the image acquisition unit may also acquire the user status image from the imaging device. Furthermore, the cooking system according to the above aspect may further include a user intrusion detection unit that identifies a high-temperature area near the heating object based on the nearby temperature estimated by the nearby temperature estimation unit, and detects whether the user has entered the high-temperature area based on the user status image acquired by the image acquisition unit. In that case, the nearby temperature information notification unit may warn the user when the user intrusion detection unit detects that the user has entered the high-temperature area.

According to the above cooking system, if the user enters a high temperature area, a warning is given to the user, thereby notifying the user that they have entered a high temperature area.

The cooking system according to the above aspect may further include a heating object information acquisition unit that acquires heating object information, which is information about the heating object. In this case, the learning device may perform machine learning to estimate the nearby temperature based on the heating object information as well. The nearby temperature estimation unit may also input the heating object information acquired by the heating object information acquisition unit to the learning device, and acquire the nearby temperature from the learning device by executing the arithmetic processing of the learned learning device.

According to the above cooking system, the accuracy of estimating the nearby temperature can be improved by estimating the nearby temperature based on the information about the object to be heated.

In the heating and cooking system according to one aspect described above, the heating object information may include at least one of cooking container information, which is information about a cooking container placed above the heating unit, cooking menu information, which is information about a cooking menu to be cooked by the heating unit, and heating object temperature, which is the temperature of the heating object.

For example, if the size of the cooking container heated by the heating unit is large, the range affected by the radiant heat from the cooking container will be wider, and the range in which the nearby temperature becomes high will be wider. On the other hand, if the size of the cooking container is small, the range affected by the radiant heat will be narrower, and the range in which the nearby temperature becomes high will be narrower. In this way, the cooking container information affects the size of the range in which the nearby temperature becomes high. According to the above-mentioned heating and cooking system, the nearby temperature can be estimated based on the cooking container information that affects the nearby temperature, thereby improving the accuracy of estimating the nearby temperature.

For example, if the cooking menu cooked by the heating unit is different, the amount of heat of the heating unit will also be different. If the amount of heat of the heating unit is larger, the nearby temperature will be higher, and if the amount of heat of the heating unit is smaller, the nearby temperature will be lower. In other words, cooking menu information affects the nearby temperature. According to the above-mentioned heating cooking system, the estimation accuracy of the nearby temperature can be improved by estimating the nearby temperature based on the cooking menu information that affects the nearby temperature.

Furthermore, if the temperature of the object to be heated is high, the nearby temperature will also be high. On the other hand, if the temperature of the object to be heated is low, the nearby temperature will also be low. In other words, there is a correlation between the temperature of the object to be heated and the nearby temperature. According to the above cooking system, the accuracy of estimating the nearby temperature can be improved by estimating the nearby temperature based on the temperature of the object to be heated, which is correlated with the nearby temperature.

In the cooking method disclosed in this specification, a computer may execute an image acquisition step, a nearby temperature estimation step, and a nearby temperature information notification step. In the image acquisition step, the computer acquires the heating part nearby image from an imaging device configured to capture the heating part nearby image, which is an image including the state near the heating part that is heating the heated object. In the nearby temperature estimation step, the computer inputs the heating part nearby image acquired in the image acquisition step to a trained learning device that has performed machine learning to estimate the nearby temperature, which is the temperature near the heating part, based on the heating part nearby image, and acquires the nearby temperature from the learning device by executing arithmetic processing of the trained learning device. In the nearby temperature information notification step, the computer notifies the user of nearby temperature information, which is information related to the nearby temperature, based on the nearby temperature acquired in the nearby temperature estimation step.

In the cooking method according to one aspect described above, the computer may further execute a heating information acquisition step of acquiring heating information, which is information related to the heating performed by the heating unit on the heating object. In this case, the learning device may perform machine learning to estimate the nearby temperature based on the heating information as well. In the nearby temperature estimation step, the computer may also input the heating information acquired in the heating information acquisition step to the learning device, and acquire the nearby temperature from the learning device by executing the arithmetic processing of the trained learning device.

In the cooking method according to one aspect described above, the heating information may include a post-heating time that is the time that has elapsed since the heating unit finished heating the object to be heated.

In the cooking method according to one aspect of the above, the imaging device may be configured to be capable of capturing a user state image, which is an image including the state of the user. In this case, the computer may further execute a user intrusion detection step in which, in the image acquisition step, the computer also acquires the user state image from the imaging device, identifies a high-temperature area near the heating object based on the nearby temperature estimated in the nearby temperature estimation step, and detects whether the user has entered the high-temperature area based on the user state image acquired in the image acquisition step. Furthermore, in the nearby temperature information notification step, the computer may warn the user when it detects that the user has entered the high-temperature area in the user intrusion detection step.

In the cooking method according to one aspect described above, the computer may further execute a heating object information acquisition step of acquiring heating object information, which is information related to the heating object. In this case, the learning device may perform machine learning to estimate the nearby temperature based on the heating object information as well. In the nearby temperature estimation step, the computer may also input the heating object information acquired in the heating object information acquisition step to the learning device, and acquire the nearby temperature from the learning device by executing the arithmetic processing of the trained learning device.

In the heating and cooking method according to one aspect described above, the heating object information may include at least one of cooking container information, which is information about a cooking container placed above the heating unit, cooking menu information, which is information about a cooking menu to be cooked by the heating unit, and heating object temperature, which is the temperature of the heating object.

The learning device disclosed in this specification may include a learning data acquisition unit and a learning processing unit. The learning data acquisition unit acquires learning data including a nearby temperature, which is a temperature near a heating unit that heats a heated object, and a heating unit nearby image, which is associated with the nearby temperature and is acquired from an imaging device configured to be able to capture a heating unit nearby image, which is an image including a state near the heating unit that is heating the heated object. The learning processing unit performs machine learning of a learner so as to obtain an output corresponding to the nearby temperature in response to an input including the heating unit nearby image.

The above learning device can be used to construct a trained learning machine for estimating nearby temperatures based on images of the vicinity of the heated area.

In the learning device according to the above aspect, the learning data may be associated with the nearby temperature and may further include heating information that is information regarding the heating performed by the heating unit on the heating object. In that case, the input in the machine learning of the learning device may also include the heating information.

The above learning device can be used to construct a trained learning machine for estimating nearby temperatures based on heating information as well.

In the learning device according to the above aspect, the heating information may include a post-heating time that is the time that has elapsed since the heating unit stopped heating the object to be heated.

The above learning device can be used to construct a trained learning machine for estimating nearby temperatures based on heating information including the time elapsed since heating.

In the learning device according to the above aspect, the learning data may be associated with the nearby temperature and may further include heating object information, which is information about the heating object. In that case, the input in the machine learning of the learning device may also include the heating object information.

The above learning device makes it possible to construct a trained learning device for estimating nearby temperatures based on information about the heated object.

In the learning device according to one aspect described above, the heated object information may include at least one of cooking container information, which is information about a cooking container placed above the heating unit, cooking menu information, which is information about a cooking menu to be cooked by the heating unit, and heated object temperature, which is the temperature of the heated object.

The above learning device can construct a trained learning device for estimating nearby temperatures based on heating object information including at least one of cooking container information, cooking menu information, and heating object temperature.

In the learning method disclosed in this specification, a computer may execute a learning data acquisition step and a learning processing step. In the learning data acquisition step, the computer acquires learning data including a nearby temperature, which is a temperature near a heating unit that heats a heated object, and a heating unit nearby image, which is associated with the nearby temperature and is acquired from an imaging device configured to be able to capture a heating unit nearby image, which is an image including a state near the heating unit that is heating the heated object. In the learning processing step, the computer performs machine learning of a learner so as to obtain an output corresponding to the nearby temperature in response to an input including the heating unit nearby image.

In the learning method according to the above aspect, the learning data may further include heating information that is associated with the nearby temperature and is information regarding the heating performed by the heating unit on the heating object. In that case, the input in the machine learning of the learning device may also include the heating information.

In the learning method according to one aspect described above, the heating information may include a post-heating time that is the time that has elapsed since the heating unit stopped heating the object to be heated.

In the learning method according to the above aspect, the learning data may be associated with the nearby temperature and may further include heating object information, which is information about the heating object. In that case, the input in the machine learning of the learning device may also include the heating object information.

In the learning method according to one aspect described above, the heated object information may include at least one of cooking container information, which is information about a cooking container placed above the heating unit, cooking menu information, which is information about a cooking menu to be cooked by the heating unit, and heated object temperature, which is the temperature of the heated object.

Details and further improvements of the technology disclosed in this specification are explained in the "Description of Embodiments" below.

1 shows a perspective view of a cooking system according to an embodiment of the present invention. 2 shows the hardware configuration of the heating and cooking system. 2 shows the hardware configuration of a learning device. 2 shows the software configuration of the control device. 2 shows the software configuration of the learning device. FIG. 2 shows a plan view of the cooking device during heating. FIG. 4 shows a plan view of the cooking device after heating. 4 shows a flow diagram of a process executed by a control device. 1 shows a flow diagram of a process executed by a learning device.

(Embodiment)
Fig. 1 is a perspective view showing a cooking system 2 according to an embodiment of one aspect. As shown in Fig. 1, the cooking system 2 includes a range hood 4 and a cooker 10. The cooker 10 is a built-in gas-burning stove that is incorporated into a system kitchen. The range hood 4 is a ventilation device disposed above the cooker 10.

On the top plate 10u of the cooking device 10, the first burner 11a, the second burner 11b, the third burner 11c, and the exhaust port 17 communicating with the grill 28 are arranged. In addition, the burner operation unit 20 and the grill 28 are provided on the front surface 10f, which is the front surface of the cooking device 10 (i.e., the right side in FIG. 1). In this specification, to facilitate understanding, the side on which the front surface 10f of the cooking device 10 is provided may be simply referred to as the "front side", and the opposite side may be simply referred to as the "rear side". In addition, the direction connecting the front side and the rear side is referred to as the front-rear direction. The left and right sides of the horizontal direction perpendicular to the front-rear direction are the same as the right and left sides of the user located in front of the cooking device 10 and facing the front surface 10f. In addition, a grill operation unit for operating the grill burner 29a (see FIG. 2) in the grill 28 is provided on the left side of the front surface 10f of the cooking device 10, but the description thereof is omitted in this specification.

The first burner 11a, the second burner 11b, and the third burner 11c are arranged in two rows in the front-rear direction on the upper surface of the top plate 10u. The first burner 11a is equipped with a first burner burner 12a, a first sensor 14a, and a trivet 16a. A gas supply passage (not shown) is connected to the first burner 12a. The gas supply passage is provided with a flow rate adjustment valve (not shown) for adjusting the amount of gas supplied to the first burner 12a. The first burner 12a is ignited by operating an igniter (not shown) while gas is being supplied to the first burner 12a. The first burner 11a heats cooking vessels such as pots and frying pans placed above it with the ignited first burner 12a. The amount of heat from the first burner 12a can be adjusted by adjusting the amount of gas supplied to the first burner 12a. In addition, the first burner 12a is extinguished by stopping the supply of gas to the first burner 12a. The second burner 12b of the second burner 11b and the third burner 12c of the third burner 11c also have the same structure as the first burner 12a.

The first sensor 14a detects the presence of a cooking vessel placed above the first burner 11a and detects the temperature of the cooking vessel heated by the first burner burner 12a of the first burner 11a. That is, the first sensor 14a measures the temperature of the object to be heated by the first burner burner 12a (e.g., a cooking vessel). When a cooking vessel is placed on the first burner 11a, the first sensor 14a is pressed by the cooking vessel. When the first sensor 14a is pressed by the cooking vessel, it detects that a cooking vessel has been placed above the first burner 11a. When a cooking vessel is not placed on the first burner 11a, the first sensor 14a is not pressed. A thermocouple is disposed within the first sensor 14a, and the temperature of the cooking vessel in contact with the first sensor 14a can be measured. The second sensor 14b on the second burner 11b and the third sensor 14c on the third burner 11c have the same structure as the first sensor 14a. The trivets 16a to 16c support the cooking vessels placed above each burner at a fixed distance from the burner of each burner.

The burner operation unit 20 includes a power switch 24 for the cooking device 10, a first heat amount operation unit 20a, a second heat amount operation unit 20b, a third heat amount operation unit 20c, and a panel operation unit 22. The power switch 24 is a switch that starts the cooking device 10, and turning on the power switch 24 enables ignition of each burner. The first heat amount operation unit 20a corresponds to the first burner 11a. Similarly, the second heat amount operation unit 20b corresponds to the second burner 11b, and the third heat amount operation unit 20c corresponds to the third burner 11c. The first heat amount operation unit 20a is an alternate type switch that ignites and extinguishes the first burner 12a and adjusts the heat amount of the first burner 12a. The second heat amount operation unit 20b and the third heat amount operation unit 20c also have the same structure as the first heat amount operation unit 20a.

The panel operation unit 22 displays the operating status of each burner (i.e., the first burner 12a, the second burner 12b, and the third burner 12c) and the grill burner 29a (see FIG. 2). The user can use the panel operation unit 22 to set a fire extinguishing timer that will turn off each burner and the grill burner 29a after a specified time has elapsed.

The cooking appliance 10 houses a control device 40 therein. The control device 40 is a computer for automatically controlling the heat (i.e., the amount of heat) of the first burner 11a, the second burner 11b, the third burner 11c, and the grill burner 29a (see FIG. 2) in the grill 28. The control device 40 is connected to a learning device 30 via the Internet. The control device 40 and the learning device 30 will be described in detail later.

The range hood 4 houses a fan (not shown) inside. The range hood 4 includes a range cover 6 that covers the fan from below, a camera 8, and a speaker 9c. The camera 8 is located in the center of the rear of the range cover 6. A lens is located in the center of the camera 8. As will be described in detail later, the camera 8 is configured to be able to capture images from above that include the top plate 10u of the cooking appliance 10 and the state of its vicinity. The technology by which the camera 8 captures images is known, so a description of this technology is omitted here. A speaker 9c is located at the left end of the front of the range hood 4. As will be described in detail later, the speaker 9c transmits sound to the user of the cooking appliance 10.

The control configuration of the cooking system 2 will be described with reference to FIG. 2. Note that FIG. 2 only shows the first burner 12a of the first burner 11a, and omits the second burner 12b and the third burner 12c. Similarly, FIG. 2 only shows the first sensor 14a of the first burner 11a, and omits the second sensor 14b and the third sensor 14c. When the power switch 24 is turned on, the burner operation unit 20, the panel operation unit 22, the control device 40, etc. are activated.

The control device 40 is equipped with a hardware processor, a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). When the stove operation unit 20 is operated, the control device 40 adjusts the flame power of the first stove burner 12a based on the operation. The control device 40 is, for example, composed of an ECU (Electronic Control Unit).

The control device 40 includes a learning memory unit 46, which is composed of, for example, RAM, ROM, etc., a notification pattern memory unit 48, and an external interface (written as external I/F in FIG. 2) 42. The learning memory unit 46 stores a program 46p and learning result data 32r. The program 46p includes instructions for causing the control device 40 to execute a process of estimating the nearby temperature for the image captured by the camera 8. Here, the nearby temperature means information about the temperature distribution in the vicinity of each burner and grill burner 29a. The nearby temperature includes information about the area where the temperature is equal to or higher than a predetermined temperature and the maximum temperature thereof. The learning result data 32r is data for setting the learned learning device, and is generated by the learning device 30 shown in FIG. 1 and transmitted to the control device 40 via the Internet. The nearby temperature may include information about the maximum temperature within a predetermined range in the vicinity of each burner and grill burner 29a.

The notification pattern storage unit 48 of the control device 40 shown in FIG. 2 stores a notification pattern table 48t consisting of nearby temperatures and notification patterns corresponding to the nearby temperatures. The notification pattern storage unit 48 classifies the nearby temperatures into three stages, "room temperature", "medium temperature", and "high temperature", and stores them in the notification pattern table 48t. The notification pattern storage unit 48 stores notification patterns "Pattern A", "Pattern B", and "Pattern C" corresponding to each of the three stages of nearby temperatures in the notification pattern table 48t. When the estimated nearby temperature is "room temperature", the control device 40 notifies the user through the speaker 9c with a voice, for example, "The temperature in the vicinity of the first burner 12a is room temperature" (Pattern A). This allows the control device 40 to notify the user that the heated first burner 12a has cooled to room temperature. In addition, when the estimated nearby temperature is "medium temperature", for example, a voice message "The vicinity of the first burner 12a is a little hot" (pattern B) is notified to the user via the speaker 9c. This allows the control device 40 to notify the user that the first burner 12a that was heated has not yet completely cooled down. Furthermore, when the estimated nearby temperature is "high temperature (no intrusion)", for example, a voice message "The vicinity of the first burner 12a is very hot" (pattern C) is notified to the user via the speaker 9c. This allows the control device 40 to notify the user that the first burner 12a that was heated is still hot. Note that the case where the nearby temperature corresponds to "high temperature (intrusion)" will be described later with reference to FIG. 7. The classification of the nearby temperature is not limited to the above three stages, and it is sufficient that it is classified into at least two stages (i.e., whether or not it is high temperature). The nearby temperature may be further classified into multiple stages depending on the width of the nearby range and the way of classifying the temperature. The notification pattern table 48t may be stored in advance in the notification pattern storage unit 48 when the cooking appliance 10 is manufactured, or may be downloaded from an external server (not shown) via an external interface 42 capable of communicating with the external server and stored in the notification pattern storage unit 48.

The external interface 42 is an interface that connects the control device 40 to an external device. As shown in FIG. 2, the external interface 42 connects the control device 40 to the camera 8 and the speaker 9c. The external interface 42 is connected to the camera 8 and the speaker 9c, for example, by a Wi-Fi (registered trademark) system.

The learning device 30 will be described with reference to FIG. 3. The learning device 30 includes a memory unit 32, an input device 34i, an output device 34о, a control unit 36, a communication interface (written as communication I/F in FIG. 3) 38, and a drive 39. As shown in FIG. 3, the learning device 30 is a computer to which each device is electrically connected.

The control unit 36 includes a hardware processor such as a CPU, RAM, and ROM, and is configured to execute various information processing operations based on programs and data. The storage unit 32 is configured, for example, with a hard disk drive, a solid state drive, and the like. The storage unit 32 stores a learning program 32p executed by the control unit 36, learning data 32d used for machine learning by the learning device, and learning result data 32r created by executing the learning program 32p.

The learning program 32p is a program for causing the learning device 30 to execute the machine learning process (FIG. 9) described below, and for generating learning result data 32r as a result of the machine learning. The learning data 32d is data for performing machine learning on the learning device so as to estimate the nearby temperature from input information. Details will be described later.

The communication interface 38 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc., and is an interface for performing wired or wireless communication via a network. The learning device 30 may distribute the created learning result data 32r to an external device via the communication interface 38.

The input device 34i is, for example, a device for inputting data such as a mouse or keyboard. The output device 34о is, for example, a device for outputting data such as a display or speaker. An operator of the learning device 30 can operate the learning device 30 via the input device 34i and the output device 34о.

The drive 39 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 30m. The type of the drive 39 may be appropriately selected according to the type of the storage medium 30m. The learning program 32p and the learning result data 32r may be stored in this storage medium 30m.

The storage medium 30m is a medium that stores information such as a program through electrical, magnetic, optical, mechanical, or chemical action so that a computer or other device, machine, etc. can read the recorded information such as the program. The learning device 30 may obtain the learning program 32p and learning data 32d from this storage medium 30m.

In FIG. 3, a disk-type storage medium such as a CD or DVD is illustrated as an example of the storage medium 30m. However, the storage medium 30m is not limited to the disk type. The storage medium 30m may be, for example, a semiconductor memory such as a flash memory.

Regarding the specific hardware configuration of the learning device 30, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 36 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, an FPGA (field-programmable gate array), or the like. The learning device 30 may be configured with multiple information processing devices. Furthermore, the learning device 30 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a PC (Personal Computer), or the like.

The control configuration of the control device 40 will be described with reference to FIG. 4. FIG. 4 shows a schematic diagram of the control configuration of the control device 40.

The control device 40 loads the program 46p stored in the learning memory unit 46 into the RAM. The control device 40 uses the CPU to interpret and execute the program 46p loaded into the RAM to control each component. As a result, as shown in FIG. 4, the control device 40 is configured as a computer that includes, as software modules, an image acquisition unit 51, an image analysis unit 52, a resolution conversion unit 53, a heating information acquisition unit 54, a nearby temperature estimation unit 55, an alarm unit 56, a user intrusion detection unit 57, and a heated object temperature acquisition unit 59.

The image acquisition unit 51 acquires the captured image 9i from the camera 8. Details will be described with reference to FIG. 6, but the captured image 9i includes the state of the flame of each burner, the state of the cooking container placed above each burner, the state of the food being cooked in the cooking container, etc. In other words, the captured image 9i includes the state of the vicinity of each burner and grill burner 29a of the cooking device 10. Furthermore, the captured image 9i also includes the state of the user's hands approaching the vicinity of each burner and grill burner 29a.

The image analysis unit 52 analyzes the captured image 9i acquired by the image acquisition unit 51, and extracts analyzed heated object information 52b, which is information about the heated object, from the captured image 9i. The heating information acquisition unit 54 acquires the total amount of heat from each burner of the heating cooker 10, and the heating time, which is the time elapsed since heating started, as heating information 54i. The heated object temperature acquisition unit 59 acquires the heated object temperature 59t, which is the temperature of the heated object heated by each burner, from the sensor of each burner of the heating cooker 10 (see FIG. 1). In addition, when the heating of each burner has ended, the heating information acquisition unit 54 also acquires the post-heating elapsed time, which is the time elapsed since heating ended, as heating information 54i. The acquired analyzed heated object information 52b, heating information 54i, and heated object temperature 59t are input to the neural network 70 as supplementary information 60.

Details will be described with reference to FIG. 6, but the analyzed heated object information 52b includes the cooking vessel (pot 86a, frying pan 87a in FIG. 6) to be heated by the heating unit, and the foodstuffs (foods 87m, 28m in FIG. 6) to be placed on the heating unit. By inputting the analyzed heated object information 52b including such information as supplementary information 60 to the neural network 70, the analyzed heated object information 52b can be added to the conditions for estimating the nearby temperature. If the control device 40 is capable of automatic heating control that automatically adjusts the heat power according to the cooking menu, the cooking menu set by the user may also be included in the supplementary information 60 as heated object information. This allows the nearby temperature to be estimated from the amount of heat corresponding to the cooking menu. Furthermore, if the control device 40 is configured to allow the user to set information such as the type and size of the cooking vessel, the information about the cooking vessel set by the user may also be included in the supplementary information 60 as heated object information. This allows the nearby temperature to be estimated from the information about the cooking vessel set by the user.

When the amount of heating included in the heating information 54i is large, the nearby temperature is high. On the other hand, when the amount of heating is small, the nearby temperature is low. Similarly, when the heating time included in the heating information 54i is long, the nearby temperature is high, and when it is short, the nearby temperature is low. In other words, when the total amount of heating is large, the nearby temperature is high, and when the total amount of heating is small, the nearby temperature is low. When heating has ended, when the time since heating included in the heating information 54i is long, the nearby temperature is low, and when it is short, the nearby temperature is high. By inputting the heating information 54i containing such information into the neural network 70 as supplementary information 60, the heating information 54i can be added to the conditions for estimating the nearby temperature. As a result, the accuracy of estimating the nearby temperature is improved.

Furthermore, if the heated object temperature 59t acquired by the heated object temperature acquisition unit 59 is high, the nearby temperature will be high, and if it is low, the nearby temperature will be low. By inputting the heated object temperature 59t as supplementary information 60 to the neural network 70 in addition to the above heating information 54i, the heated object temperature 59t can be added to the conditions for estimating the nearby temperature. As a result, the estimation accuracy of the nearby temperature is further improved.

The resolution conversion unit 53 reduces the resolution of the captured image 9i acquired by the image acquisition unit 51. As a result, the resolution conversion unit 53 generates a low-resolution captured image 62.

The nearby temperature estimation unit 55 inputs the low-resolution captured image 62 obtained by reducing the resolution of the captured image 9i to a trained learning device (neural network 70) that has performed machine learning to estimate the nearby temperature. The supplementary information 60 described above is also input to the neural network 70. In this way, the nearby temperature estimation unit 55 acquires nearby temperature information 64 from the learning device. By inputting the low-resolution captured image 62 obtained by reducing the resolution to the neural network 70, the amount of calculation in the arithmetic processing of the neural network 70 can be reduced, and the load on the processor can be reduced. The resolution reduction process may be omitted. In this case, the nearby temperature estimation unit 55 may input the captured image 9i to the learning device (neural network 70).

The nearby temperature information 64 includes the temperature distribution near each stove burner and grill burner 29a (see FIG. 2) and the maximum temperature in the vicinity. The maximum temperatures included in the nearby temperature information 64 are classified into three stages (i.e., "room temperature," "medium temperature," and "high temperature") in the notification pattern table 48t stored in the notification pattern storage unit 48 in FIG. 2. The notification unit 56 generates a sound through the speaker 9c in a notification pattern corresponding to the three stages of the nearby temperature information 64 acquired by the nearby temperature estimation unit 55 from the learning device. This allows the control device 40 to notify the user of information related to the nearby temperature (i.e., the above-mentioned patterns A to C) based on the nearby temperature estimated from the captured image 9i and the supplementary information 60.

The user intrusion detection unit 57 acquires user information 52u obtained by analyzing the captured image 9i. The user information 52u includes the position of the user's hand 90 approaching the pot 86a placed above the first burner 11a, as shown in FIG. 7, for example. The user intrusion detection unit 57 also acquires the temperature distribution and maximum temperature near each burner from the nearby temperature information 64. When the maximum temperature included in the nearby temperature information 64 is classified as "high temperature" in the above-mentioned notification pattern table 48t, the user intrusion detection unit 57 compares the temperature distribution included in the nearby temperature information 64 with the user information 52u. When the position of the user's hand 90 is included in the range of the temperature distribution in which the maximum temperature is classified as "high temperature" (for example, the first high temperature area 15a in FIG. 7) as shown in FIG. 7, the user intrusion detection unit 57 determines that the user's hand 90 has entered the high temperature area. In this way, the user intrusion detection unit 57 of the control device 40 detects whether the user has entered the high temperature area based on the user information 52u and the estimated nearby temperature. Regardless of the three stages of the notification pattern table 48t, when the maximum temperature included in the nearby temperature information 64 exceeds a predetermined threshold, the user intrusion detection unit 57 may identify the area that exceeds the threshold temperature and compare it with the user information 52u.

In addition, when the user intrusion detection unit 57 detects the user's intrusion into the high temperature area, the notification unit 56 notifies the user through the speaker 9c with a voice saying "It is dangerous" ("Pattern D" in the notification pattern table 48t in FIG. 2). In this way, the notification unit 56 notifies the user that he or she has entered the high temperature area. When the user intrusion detection unit 57 detects the user's intrusion into the high temperature area, the user's approach is dangerous, so the notification unit 56 may notify the user through the speaker 9c with a particularly loud sound, such as a warning sound. The speaker 9c may be disposed in the cooking appliance 10 instead of the range hood 4. In addition, the notification unit 56 may display a message on a mobile terminal (not shown) of another user in a remote location to notify the user.

As shown in FIG. 4, the control device 40 uses a neural network 70 as a trained learning device that has performed machine learning to estimate the nearby temperature information 64. The neural network 70 is configured by combining multiple types of neural networks.

The neural network 70 is divided into four parts: a fully connected neural network 72, a convolutional neural network 74, a coupling layer 76, and an LSTM network 78. The fully connected neural network 72 and the convolutional neural network 74 are arranged in parallel on the input side, with the supplementary information 60 being input to the fully connected neural network 72 and the low-resolution captured image 62 being input to the convolutional neural network 74. The coupling layer 76 combines the outputs of the fully connected neural network 72 and the convolutional neural network 74. The LSTM network 78 receives the output from the coupling layer 76 and outputs the nearby temperature information 64.

The fully connected neural network 72 is a so-called multi-layered neural network, and includes, in order from the input side, an input layer 72i, an intermediate layer (hidden layer) 72m, and an output layer 72о. However, the number of layers of the fully connected neural network 72 is not limited to this example, and may be selected appropriately depending on the embodiment.

Each layer 72i, 72m, 72o of the fully-connected neural network 72 has one or more neurons (nodes). The number of neurons included in each layer may be set appropriately depending on the embodiment. The fully-connected neural network 72 is configured by each neuron included in each layer being connected to all neurons included in the adjacent layer. A weight (connection load) is set appropriately for each connection.

The convolutional neural network 74 is a forward propagation type neural network having a structure in which convolutional layers 74a and pooling layers 74b are alternately connected. In the convolutional neural network 74 according to this embodiment, multiple convolutional layers 74a and pooling layers 74b are alternately arranged on the input side. The output of the pooling layer 74b arranged on the most output side is input to the fully connected layer 74c, and the output of the fully connected layer 74c is input to the output layer 74о.

The convolution layer 74a is a layer that performs the convolution calculation of an image. Image convolution corresponds to the process of calculating the correlation between an image and a specified filter. Therefore, by performing image convolution, it is possible to detect, for example, a shading pattern similar to the shading pattern of the filter from the input image.

The pooling layer 74b is a layer that performs pooling processing. Pooling processing discards some of the information from positions where the image had a strong response to the filter, achieving invariance of the response to minute changes in the position of features that appear in the image.

The fully connected layer 74c is a layer that connects all neurons between adjacent layers. That is, each neuron included in the fully connected layer 74c is connected to all neurons included in the adjacent layer. The fully connected layer 74c may be composed of two or more layers. In addition, the number of neurons included in the fully connected layer 74c may be set appropriately depending on the embodiment.

The output layer 74о is the layer located at the most output side of the convolutional neural network 74. The number of neurons included in the output layer 74о may be set appropriately depending on the embodiment. Note that the configuration of the convolutional neural network 74 is not limited to this example, and may be set appropriately depending on the embodiment.

The connection layer 76 is disposed between the fully connected neural network 72 and the convolutional neural network 74 and the LSTM network 78. The connection layer 76 connects the output from the output layer 72о of the fully connected neural network 72 and the output from the output layer 74о of the convolutional neural network 74. The output of the connection layer 76 is input to the LSTM network 78. The number of neurons included in the connection layer 76 may be set appropriately depending on the number of outputs of the fully connected neural network 72 and the convolutional neural network 74.

The LSTM network 78 is a recurrent neural network that includes an LSTM block 78b. A recurrent neural network is a neural network that has an internal loop, such as a path from an intermediate layer to an input layer. The LSTM network 78 has a structure in which the intermediate layer of a general recurrent neural network is replaced with an LSTM block 78b.

The LSTM network 78 includes, in order from the input side, an input layer 78i, an LSTM block 78b, and an output layer 78о, and in addition to a forward propagation path, has a path from the LSTM block 78b back to the input layer 78i. The number of neurons included in the input layer 78i and the output layer 78о may be set appropriately depending on the embodiment.

LSTM block 78b is a block that has an input gate and an output gate and is configured to be able to learn the timing of storing and outputting information. LSTM block 78b may also have a forget gate that adjusts the timing of forgetting information. The configuration of LSTM network 78 can be set appropriately depending on the embodiment.

A threshold is set for each neuron, and the output of each neuron is basically determined by whether or not the sum of the products of each input and each weight exceeds the threshold. The control device 40 inputs supplementary information 60 to the fully connected neural network 72, and inputs low-resolution captured images 62 to the convolutional neural network 74. The control device 40 then performs firing determination for each neuron contained in each layer, starting from the input side. In this way, the neural network 70 performs arithmetic processing on the input. The control device 40 obtains an output value corresponding to the nearby temperature information 64 from the output layer 78о of the neural network 70.

In addition, information indicating the configuration of such neural network 70 (e.g., the number of layers in each network, the number of neurons in each layer, the connection relationships between neurons, and the transfer functions of each neuron), the weights of the connections between each neuron, and the thresholds of each neuron are included in the learning result data 32r. The control device 40 refers to the learning result data 32r to set up the trained neural network 70 used in the process of estimating the nearby temperature.

The learning device 30 will be described with reference to FIG. 5. The control unit 36 (see FIG. 3) of the learning device 30 expands the learning program 32p stored in the memory unit 32 (see FIG. 3) into the RAM. The control unit 36 then interprets and executes the learning program 32p expanded into the RAM by the CPU to control each component. As a result, as shown in FIG. 5, the learning device 30 is configured as a computer that includes a learning data acquisition unit 31 and a learning processing unit 33 as software modules.

The learning data acquisition unit 31 acquires a set of a low-resolution captured image 62t obtained by reducing the resolution of the captured image 9i, supplementary information 60t consisting of analytical heated object information 52b, heating information 54i, and heated object temperature 59t, and nearby temperature information 64t corresponding to the low-resolution captured image 62t and supplementary information 60t as learning data 66. The low-resolution captured image 62t and supplementary information 60t correspond to the low-resolution captured image 62 and supplementary information 60 described with reference to FIG. 4, respectively, and are used as input data. The nearby temperature information 64t corresponds to the nearby temperature information 64 in FIG. 4, and is used as teacher data (correct answer data). When the learning processing unit 33 receives the low-resolution captured image 62t and supplementary information 60t, it performs machine learning of the learner so as to output an output value corresponding to the nearby temperature information 64t.

As shown in FIG. 5, the learning device 30 includes a neural network 70t, similar to the control device 40 (see FIG. 4). The neural network 70t of the learning device 30 includes a fully connected neural network 72t, a convolutional neural network 74t, a coupling layer 76t, and an LSTM network 78t. The neural network 70t of the learning device 30 is configured similarly to the neural network 70 of the control device 40. Therefore, the fully connected neural network 72t, the convolutional neural network 74t, the coupling layer 76t, and the LSTM network 78t of the learning device 30 are similar to the fully connected neural network 72, the convolutional neural network 74, the coupling layer 76, and the LSTM network 78 of the control device 40, respectively. That is, the neural network 70t of the learning device 30 executes arithmetic processing in the same manner as the neural network 70 of the control device 40.

The learning processing unit 33 constructs a neural network 70t that inputs supplementary information 60t to a fully connected neural network 72t and inputs a low-resolution captured image 62t to a convolutional neural network 74t, and outputs an output value corresponding to the nearby temperature information 64t from an LSTM network 78t. The learning processing unit 33 then stores information indicating the configuration of the constructed neural network 70, the weights of the connections between each neuron, and the threshold value of each neuron as learning result data 32r in the memory unit 32. The learning processing unit 33 periodically transmits the learning result data 32r stored in the memory unit 32 to the control device 40 via the Internet. This causes the learning result data 32r in the control device 40 to be periodically updated.

With reference to FIG. 6, the captured image 9i input by the cooking system 2 to the control device 40 will be described. FIG. 6 is an example of an image captured by the camera 8. A pot 86a is placed above the first burner 11a (i.e., the front side of the paper in FIG. 6). The pot 86a is heated by the flame 13a of the first burner burner 12a (see FIG. 1) of the first burner 11a. A frying pan 87a is placed above the third burner 11c. An object to be cooked 87m is placed on the frying pan 87a. The frying pan 87a is heated by the flame 13c of the third burner burner 12c (see FIG. 1) of the third burner 11c. As shown in FIG. 6, the grill 28 is pulled out forward (i.e., the bottom side of the paper in FIG. 6) and contains an object to be cooked 28m. In this way, the camera 8 is configured to capture an image of the vicinity of the heating unit, which is an image that includes the conditions in the vicinity of each stove and the grill burner 29a (see Figure 2).

As described with reference to FIG. 4, the control device 40 estimates the temperature in the vicinity of the heating part by inputting the image 9i captured by the camera 8 (i.e., the image of the vicinity of the heating part) to the neural network 70. The image of the vicinity of the heating part includes an image of the flame 13a extending beyond the outer edge of the pot 86a. The temperature in the vicinity of the first burner 11a varies depending on the size and color of the flame 13a. The control device 40 estimates the temperature in the vicinity of the heating part by inputting the image of the vicinity of the heating part, including information about the flame 13a, to the neural network 70.

In addition, the image near the heating part includes an image of the food 87m to be cooked in addition to the flame 13c that protrudes from the outer edge of the frying pan 87a. That is, by analyzing the image near the heating part, information about the food 87m to be cooked by the third burner 12c can be extracted. This makes it possible to extract cooking menu information, which is information about the cooking menu that the user will cook using the food 87m. For example, when the food 87m is heated by the third burner 12c, the color of its surface changes. The progress of the user's cooking can be estimated based on the change in the color of the surface of the food 87m. The amount of heat of the third burner 12c can be estimated from the amount of heat required to complete the menu of the cooking menu information. The amount of heat of the third burner 12c and the temperature near the third burner 11c are correlated. That is, the cooking menu information extracted from the image of the food 87m and the temperature near the third burner 11c are correlated. The control device 40 can improve the accuracy of estimating the nearby temperature by inputting cooking menu information that has a correlation with the nearby temperature to the neural network 70 in addition to the image of the vicinity of the heating unit.

Furthermore, if the food 28m is placed on the grill 28, it is estimated that the food 28m will be heated by the grill 28. On the other hand, if the food 28m is not placed on the grill 28, it is unlikely that the food 28m will be heated by the grill 28. In other words, the image inside the grill 28 affects the estimation of the temperature near the grill 28 and the temperature near the exhaust port 17 that exhausts the grill 28. The control device 40 can improve the estimation accuracy of the nearby temperature by inputting information about the food 28m, which affects the estimation of the nearby temperature, to the neural network 70 in addition to the image near the heating part.

Figure 7 shows an image of the vicinity of the heated part after passing through the state of the image of the vicinity of the heated part shown in Figure 6. In Figure 7, the heating of the first burner 11a, the third burner 11c, and the grill 28 has stopped. Therefore, the image of the vicinity of the heated part in Figure 7 does not include an image of the flame 13a protruding from the outer edge of the pan 86a. Similarly, the image of the vicinity of the heated part in Figure 7 does not include an image of the flame 13c protruding from the outer edge of the frying pan 87a. However, the pan 86a, the frying pan 87a, and the grill 28 shown in Figure 7 are heated in the image of the vicinity of the heated part in Figure 6. The control device 40 estimates the temperatures in the vicinity of the pan 86a, the frying pan 87a, the grill 28, and the exhaust port 17 shown in Figure 7 based on the image of the vicinity of the heated part in Figure 6 before the state shown in Figure 7 is reached. The control device 40 can estimate the first high temperature area 15a of the first burner 12a, the third burner 12c, and the grill burner 29a, the third high temperature area 15c of the third burner 12c, the grill high temperature area 28b of the grill 28, and the exhaust port high temperature area 17a of the exhaust port 17, even after heating of the first burner 12a, the third burner 12c, and the grill burner 29a has been stopped.

The image of the vicinity of the heating unit shown in FIG. 7 also includes information such as the size, color, and shape of the pot 86a and frying pan 87a that are heated by the heating unit. The pot 86a has a circular shape in a plan view. Residual heat remaining in the pot 86a is dissipated around the pot 86a. Therefore, the first high temperature area 15a of the first burner 11a (i.e., the first burner burner 12a) that heats the pot 86a has a circular shape. In contrast, the frying pan 87a has a rectangular shape in a plan view. Therefore, the third high temperature area 15c of the third burner 11c (i.e., the third burner burner 12c) that heats the frying pan 87a has a rectangular shape.

Furthermore, the frying pan 87a has a handle 87b that extends in front of the cooking device 10 (i.e., below the paper surface of FIG. 7). The handle 87b is made of the same metal as the body of the frying pan 87a. Therefore, when the frying pan 87a is heated, heat is also transferred to the handle 87b, and as shown in FIG. 7, the periphery of the handle 87b is also included in the third high temperature area 15c. On the other hand, if the handle 87b is not made of metal, the heat of the body of the frying pan 87a is not easily transferred to the handle 87b. Therefore, if the handle 87b is not made of metal, the periphery of the handle 87b is not included in the third high temperature area 15c. Thus, in addition to the shape and size of the pot 86a and the frying pan 87a, the material also causes differences in the nearby temperature. Whether the handle 87b is made of metal can be estimated from the color extracted from the image of the handle 87b. The control device 40 can improve the accuracy of estimating the nearby temperature by inputting cooking vessel information (i.e., information about the pot 86a and frying pan 87a) extracted from the image of the vicinity of the heated part into the neural network 70.

As described with reference to FIG. 4, the control device 40 also inputs the heating information 54i to the neural network 70. The heating information 54i also includes the time elapsed since heating. As the time elapsed since heating increases, each high-temperature region shown in FIG. 7 shrinks.

Furthermore, even after the user has moved the pot 86a and frying pan 87a in FIG. 7 to another location, the control device 40 can estimate the temperatures near the first burner 11a, the third burner 11c, and the grill 28 using the image near the heating unit and the heating information 54i described above.

The process executed by the control device 40 (see FIG. 2) will be described with reference to FIG. 8. Note that the process executed by the control device 40 described in FIG. 8 is an example of a "heat cooking method."

When the user turns on the power switch 24 to start the cooking device 10, the control device 40 turns on the camera 8 (step S2). The turned-on camera 8 captures an image of the vicinity of the heating part at predetermined intervals. After that, the user turns on any burner (for example, the first burner 12a) to start heating (step S4). When the first burner 12a starts heating, the control device 40 starts acquiring the heating information 54i and the heating object temperature 59t (step S6). As mentioned above, the heating information 54i includes the amount of heat of the first burner 12a, the heating time, and the elapsed time after heating. Furthermore, the heating object temperature 59t is the temperature of the cooking container placed on the first burner 11a. Therefore, the control device 40 continues to acquire the heating information 54i and the heating object temperature 59t even if the first burner 12a is turned off. The control device 40 continues acquiring the heating information 54i and the heating object temperature 59t until the power switch 24 is turned off (step S26: YES).

Next, the control device 40 acquires the image near the heated part captured by the camera 8, and as described with reference to FIG. 4, the image analysis unit 52 analyzes the image near the heated part to acquire analyzed heated object information 52b (step S8). Next, as described with reference to FIG. 4, the control device 40 reduces the resolution of the image near the heated part by the resolution conversion unit 53 (step S10). Note that, as described above, step S10 of reducing the resolution of the image near the heated part can be omitted.

After that, as described with reference to FIG. 4, the control device 40 inputs the low-resolution captured image 62 and the supplementary information 60 to the neural network 70 by the nearby temperature estimation unit 55, and executes the calculation process of the neural network 70 (step S12). As described above, the neural network 70 has received the learning result data 32r from the learning device 30 via the Internet. That is, the neural network 70 is a trained learning device that has performed machine learning to estimate the nearby temperature based on the heating part nearby image. The neural network 70 to which the low-resolution captured image 62 is input outputs an output value corresponding to the nearby temperature information 64 (see FIG. 4), as described above. As a result, the control device 40 acquires the nearby temperature information 64 by the nearby temperature estimation unit 55 (step S14). After that, the control device 40 classifies the acquired nearby temperature into three stages of the notification pattern table 48t (see FIG. 2) stored in the notification pattern storage unit 48. The control device 40 causes the notification unit 56 to generate from the speaker 9c a sound of a pattern corresponding to the stage at which the estimated nearby temperature is classified (step S16). This allows the control device 40 to notify the user of the information on the estimated nearby temperature. The control device 40 may notify the user of the information on the nearby temperature at every predetermined period, or may notify the user only when the classification of the nearby temperature changes.

If the maximum temperature included in the acquired nearby temperature information 64 is a temperature classified as "high temperature" (step S18: YES), as described with reference to FIG. 4 and FIG. 7, the control device 40 uses the image analysis unit 52 to analyze the image near the heating unit and extracts the user information 52u. That is, the control device 40 acquires the user information 52u (step S20). The control device 40 uses the user intrusion detection unit 57 to compare the acquired user information 52u with the temperature distribution included in the nearby temperature information 64. That is, the control device 40 uses the user intrusion detection unit 57 to determine whether the user has intruded into the high temperature area (step S22). If the user intrusion position included in the user information 52u overlaps with the temperature distribution of the nearby temperature information 64, the control device 40 determines that the user has intruded into the high temperature area. In that case (step S22: YES), the control device 40 causes the notification unit 56 to generate a warning sound through the speaker 9c (step S24). The control device 40 then returns to step S6 and again acquires new heating information 54i and heating object temperature 59t.

Also. If the maximum temperature included in the acquired nearby temperature information 64 is not classified as "high temperature" (step S18: NO), the control device 40 returns to step S6, acquires new heating information 54i and heating object temperature 59t again, and repeats the above process. Furthermore, if the user has not entered the high temperature area (step S22: NO), and if the power switch 24 is not turned off (step S26: NO), the control device 40 returns to step S6, acquires new heating information 54i and heating object temperature 59t again, and repeats the above process. That is, the control device 40 continues to estimate the nearby temperature until the power switch 24 is turned off. Note that in the embodiment, the estimation of the nearby temperature is continued until the power switch is turned off, but the estimation of the nearby temperature may be continued not until the power switch is turned off, but until the nearby temperature becomes equal to or lower than a predetermined temperature (even if the power is turned off).

The process executed by the learning device 30 (see FIG. 5) will be described with reference to FIG. 9. The process executed by the learning device 30, which will be described with reference to FIG. 12, is an example of a "learning method."

As shown in FIG. 5, the learning device 30 acquires a set of a low-resolution image 62t, supplementary information 60t, and nearby temperature information 64t as learning data 66 via the learning data acquisition unit 31 (step S30).

The learning data 66 is data used for machine learning to enable the neural network 70t to estimate the nearby temperature. Such learning data 66 can be created by linking an image including the state of the vicinity of each burner and grill burner 29a (i.e., the image of the vicinity of the heating part described above) as shown in Figures 6 and 7 with the estimated nearby temperature. In this case, the image of the vicinity of the heating part is saved with a resolution equivalent to that of the low-resolution captured image 62 in Figure 4. The image of the vicinity of the heating part only needs to have a resolution that allows the state of the flame of each burner, the state of the cooking vessel, the state of the food being cooked, the user's hands, etc. to be distinguished.

The analyzed heated object information 52b included in the supplemental information 60 is obtained by analyzing the image near the heating part and extracting information about the color, size, and shape of the cooking vessel and the food being heated. The heating information 54i included in the supplemental information 60 is obtained by the heating information acquisition unit 54. The heated object temperature 59t included in the supplemental information 60 is obtained by the heated object temperature acquisition unit 59. The learning data 66 can also be created by linking this obtained information with the nearby temperature.

The learning data 66 may be created manually by an operator using the input device 34i (see FIG. 3), or automatically by program processing. The learning data 66 may be collected from the control device 40 at any time. The learning data 66 may also be created by an information processing device other than the learning device 30. When the learning device 30 creates the learning data 66, the control unit 36 can acquire the learning data 66 by executing the process of creating the learning data 66 in step S30. On the other hand, when the learning data 66 is created by an information processing device other than the learning device 30, the learning device 30 can acquire the learning data 66 created by the other information processing device via a network, a storage medium 30m (see FIG. 3), or the like. The number of pieces of learning data 66 acquired in step S30 may be appropriately determined according to the embodiment so that machine learning of the neural network 70t can be performed.

Next, the learning device 30 uses the learning data 66 acquired in step S30 to execute machine learning processing of the neural network 70t so that when the low-resolution captured image 62t and the supplementary information 60t are input, an output value corresponding to the nearby temperature information 64t is output (step S32).

The control unit 36 of the learning device 30 (see FIG. 3) prepares a neural network 70t for which learning processing is to be performed. The configuration of the prepared neural network 70t, the initial values of the connection weights between each neuron, and the initial values of the thresholds of each neuron may be provided by a template or by operator input. In addition, when re-learning is to be performed, the control unit 36 may prepare the neural network 70t based on the learning result data 32r for which re-learning is to be performed.

Next, the control unit 36 performs a learning process for the neural network 70 using the low-resolution captured image 62t and the supplementary information 60t included in the learning data 66 acquired in step S30 as input data and the nearby temperature information 64t as teacher data (correct answer data). A stochastic gradient descent method or the like may be used for the learning process for the neural network 70.

For example, the control unit 36 inputs the supplementary information 60t to the input layer of the fully connected neural network 72t, and inputs the low-resolution captured image 62t to the convolutional layer arranged on the most input side of the convolutional neural network 74t. Then, the control unit 36 performs a firing determination for each neuron included in each layer, starting from the input side. As a result, the control unit 36 obtains an output value from the output layer of the LSTM network 78t. Next, the control unit 36 calculates the error between the output value acquired from the output layer of the LSTM network 78t and the value corresponding to the nearby temperature information 64t. Next, the control unit 36 calculates the error of each of the connection weights between each neuron and the threshold value of each neuron using the error of the calculated output value by the back propagation through time method. Then, the control unit 36 updates the value of each of the connection weights between each neuron and the threshold value of each neuron based on each calculated error.

The control unit 36 repeats this series of processes for each piece of learning data 66 until the output value output from the neural network 70t matches the value corresponding to the nearby temperature information 64t. This allows the control unit 36 to construct a neural network 70t that outputs an output value corresponding to the nearby temperature information 64t when it receives the low-resolution captured image 62t and supplementary information 60t.

The control unit 36 stores information indicating the configuration of the constructed neural network 70, the weights of the connections between each neuron, and the threshold value of each neuron as learning result data 32r in the storage unit 32 by the learning processing unit 33 (step S34). This causes the control unit 36 to end the learning process of the neural network 70 according to this operation example.

The control unit 36 may transfer the created learning result data 32r to the control device 40 after completing the process of step S36. The control unit 36 may also periodically update the learning result data 32r by periodically executing the learning process of steps S30 to S36. The control unit 36 may also periodically update the learning result data 32r held by the control device 40 by transferring the created learning result data 32r to the control device 40 each time the learning process is executed. For example, the control unit 36 may also store the created learning result data 32r in a separate data server (not shown). In this case, the control device 40 may obtain the learning result data 32r from this data server.

(Correspondence) The pot 86a and the frying pan 87a are each an example of a "heating object". Each stove burner and the grill burner 29a of the heating cooker 10 are each an example of a "heating section". The camera 8 is an example of an "imaging device". The total amount of heat, the heating time which is the time elapsed since heating began, and the time elapsed after heating are each an example of "heating information". The image acquisition section 51 and the heating object temperature acquisition section 59 are each an example of a "heating object information acquisition section". The size, color, and shape of the pot 86a and the frying pan 87a are each an example of "cooking vessel information". The size, color, and shape of the objects to be cooked 28m and 87m are each an example of "cooking menu information".

Although the embodiments have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and variations of the specific examples given above. Modifications of the above embodiments are listed below.

(Variation 1) In the above-described heating cooking system 2, in addition to the image near the heating part, supplementary information 60 (i.e., heating information 54i, analyzed heating object information 52b, and heating object temperature 59t) is input to the trained learning device (neural network 70). Alternatively, in a variation, the configuration may be such that supplementary information 60 is not input. In that case, the control device 40 does not need to include the image analysis unit 52, the heating information acquisition unit 54, and the heating object temperature acquisition unit 59.

(Variation 2) In the above-described heating cooking system 2, the heating information acquisition unit 54 acquires the time elapsed since heating as heating information 54i, and inputs it to the trained learning device (neural network 70). Alternatively, in a variation, the configuration may be such that the time elapsed since heating is not input.

(Variation 3) In the above-described cooking system 2, the user intrusion detection unit 57 detects the user's intrusion into the high temperature area, and when the user's intrusion into the high temperature area is detected, the notification unit 56 warns the user. Alternatively, in a variation, the user may not be warned even if the user intrudes into the high temperature area. In that case, the control device 40 may not be provided with the user intrusion detection unit 57.

(Variation 4) In the above-described heating cooking system 2, the image analysis unit 52 analyzes the image near the heating unit, and cooking vessel information, which is information about the pot 86a and the frying pan 87a, is obtained as analyzed heating object information 52b, and input to the trained learning device (neural network 70). Alternatively, in a variation, cooking vessel information may not be input.

(Variation 5) In the above-described heating cooking system 2, the image analysis unit 52 analyzes the image near the heating unit, and cooking menu information, which is information about the food 87m to be cooked, is obtained as analyzed heating object information 52b, and input to the trained learning device (neural network 70). Alternatively, in a variation, the cooking menu information may not be input.

(Variation 6) The cooking system 2 described above is a built-in gas-burning stove that is incorporated into a system kitchen. Alternatively, in a variation, it may be an electromagnetic induction cooking device (IH cooking device).

Although specific examples of the present invention have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and variations of the specific examples exemplified above. The technical elements described in this specification or drawings exert technical utility alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. Furthermore, the technology exemplified in this specification or drawings can achieve multiple objectives simultaneously, and achieving one of those objectives is itself technically useful.

2: Cooking system 4: Range hood 8: Camera 9c: Speaker 9i: Captured image 10: Cooking device 11a: First burner 12a: First burner burner 13a, 13c: Flame 14a: First sensor 15a: First high temperature area 15c: Third high temperature area 17: Exhaust port 20: Burner operation unit 20a: First heat amount operation unit 22: Panel operation unit 24: Power switch 28: Grill 28m, 87m: Food to be cooked 29a: Grill burner 30: Learning device 31: Learning data acquisition unit 33: Learning processing unit 34i: Input device 34o: Output device 36: Control unit 38: Communication interface 39: Drive 40: Control device 42: External interface 46: Learning memory unit 48: Notification pattern memory unit 51: Image acquisition unit 52: Image analysis unit 53 : Resolution conversion unit 54 : Heating information acquisition unit 55 : Nearby temperature estimation unit 56 : Notification unit 57 : User intrusion detection unit 59 : Heating object temperature acquisition unit 70, 70t : Neural network 86a : Pot 87a : Frying pan 87b : Handle 90 : Hand

Claims (22)

A heating unit that heats an object to be heated;
an image acquisition unit that acquires an image of the vicinity of the heating unit from an imaging device that is configured to be able to capture an image of the vicinity of the heating unit that is heating the object to be heated;
a nearby temperature estimation unit that inputs the image near the heating unit acquired by the image acquisition unit into a trained learning device that has performed machine learning to estimate a nearby temperature, which is a temperature near the heating unit, based on the image near the heating unit, and acquires the nearby temperature from the learning device by executing a calculation process of the trained learning device;
a nearby temperature information notification unit that notifies a user of nearby temperature information, which is information relating to the nearby temperature, based on the nearby temperature estimated by the nearby temperature estimation unit. The heating device further includes a heating information acquisition unit that acquires heating information, which is information regarding the heating performed by the heating unit on the heating object,
The learning device performs machine learning to estimate the nearby temperature based on the heating information,
The heating cooking system of claim 1, wherein the nearby temperature estimation unit also inputs the heating information acquired by the heating information acquisition unit into the learning device and acquires the nearby temperature from the learning device by executing calculation processing of the learned learning device. The heating and cooking system according to claim 2, wherein the heating information includes a post-heating time that is the time that has elapsed since the heating unit finished heating the object to be heated. The imaging device is configured to be capable of imaging a user state image, which is an image including a state of the user,
The image acquisition unit also acquires the user state image from the imaging device,
a user intrusion detection unit that identifies a high-temperature area near the heating object based on the nearby temperature estimated by the nearby temperature estimation unit, and detects whether the user has entered the high-temperature area based on the user state image acquired by the image acquisition unit,
The heating and cooking system according to claim 1 , wherein the nearby temperature information notification unit warns the user when the user intrusion detection unit detects that the user has intruded into the high temperature area. The heating apparatus further includes a heating object information acquisition unit that acquires heating object information, which is information about the heating object,
The learning device performs machine learning to estimate the nearby temperature based on the heating object information as well,
The heating cooking system according to any one of claims 1 to 4, wherein the nearby temperature estimation unit also inputs the heating object information acquired by the heating object information acquisition unit into the learning device, and acquires the nearby temperature from the learning device by executing calculation processing of the learned learning device. The heating and cooking system according to claim 5, wherein the heating object information includes at least one of cooking container information, which is information about a cooking container placed above the heating unit, cooking menu information, which is information about a cooking menu to be cooked by the heating unit, and heating object temperature, which is the temperature of the heating object. The computer
An image acquisition step of acquiring an image of the vicinity of a heating unit, which is an image including a state in the vicinity of a heating unit that is heating an object to be heated, from an imaging device configured to be able to capture the image of the vicinity of the heating unit;
a nearby temperature estimation step of inputting the image near the heating unit acquired in the image acquisition step into a trained learning device that has performed machine learning to estimate a nearby temperature, which is a temperature near the heating unit, based on the image near the heating unit, and acquiring the nearby temperature from the learning device by executing a calculation process of the trained learning device;
and a nearby temperature information notifying step of notifying a user of nearby temperature information, which is information relating to the nearby temperature, based on the nearby temperature acquired in the nearby temperature estimating step. The computer further executes a heating information acquisition step of acquiring heating information which is information regarding the heating performed by the heating unit on the heating object;
The learning device performs machine learning to estimate the nearby temperature based on the heating information,
The heating and cooking method of claim 7, wherein in the nearby temperature estimation step, the computer also inputs the heating information acquired in the heating information acquisition step to the learning device, and acquires the nearby temperature from the learning device by executing calculation processing of the trained learning device. The cooking method according to claim 8, wherein the heating information includes a post-heating time that is the time that has elapsed since the heating unit finished heating the object to be heated. The imaging device is configured to be capable of imaging a user state image, which is an image including a state of the user,
The computer includes:
In the image acquisition step, the user state image is also acquired from the imaging device;
a user intrusion detection step of identifying a high-temperature area in the vicinity of the heating object based on the vicinity temperature estimated in the vicinity temperature estimation step, and detecting whether or not the user has entered the high-temperature area based on the user state image acquired in the image acquisition step;
The cooking method according to claim 7 , wherein the nearby temperature information notifying step warns the user when the user intrusion detection step detects that the user has entered the high temperature area. The computer includes:
A heating object information acquisition step is further performed to acquire heating object information, which is information about the heating object;
The learning device performs machine learning to estimate the nearby temperature based on the heating object information as well,
The cooking method according to any one of claims 7 to 10, wherein in the nearby temperature estimation step, the computer also inputs the heating object information acquired in the heating object information acquisition step into the learning device, and acquires the nearby temperature from the learning device by executing calculation processing of the trained learning device. The heating cooking method according to claim 11, wherein the heating object information includes at least one of cooking container information, which is information about a cooking container placed above the heating unit, cooking menu information, which is information about a cooking menu to be cooked by the heating unit, and heating object temperature, which is the temperature of the heating object. a learning data acquisition unit that acquires learning data including a proximity temperature, which is a temperature in the vicinity of a heating unit that heats a heating object, and a heating unit proximity image, which is associated with the proximity temperature and is acquired from an imaging device that is configured to be able to capture a heating unit proximity image, which is an image including a state in the vicinity of the heating unit that heats the heating object;
a learning processing unit that performs machine learning of a learning device for an input including an image of the vicinity of the heated portion so as to obtain an output corresponding to the vicinity temperature. The learning data is associated with the neighboring temperature and further includes heating information which is information regarding the heating performed by the heating unit on the heating object,
The learning device according to claim 13 , wherein the input in the machine learning of the learner also includes the heating information. The learning device according to claim 14, wherein the heating information includes a post-heating time that is the time that has elapsed since the heating unit stopped heating the heating object. The learning data is associated with the neighboring temperature and further includes heating object information which is information related to the heating object,
The learning device according to claim 13 , wherein the input in the machine learning of the learning unit also includes the heating object information. The learning device according to claim 16, wherein the heating object information includes at least one of cooking container information, which is information about a cooking container placed above the heating unit, cooking menu information, which is information about a cooking menu to be cooked by the heating unit, and heating object temperature, which is the temperature of the heating object. The computer
a learning data acquisition step of acquiring learning data including a proximity temperature, which is a temperature in the vicinity of a heating unit that heats a heating object, and a heating unit proximity image, which is associated with the proximity temperature and is acquired from an imaging device configured to be able to capture a heating unit proximity image, which is an image including a state in the vicinity of the heating unit that heats the heating object;
A learning method comprising: a learning process step of performing machine learning of a learning device for an input including an image of the vicinity of the heated portion so as to obtain an output corresponding to the vicinity temperature. The learning data is associated with the neighboring temperature and further includes heating information which is information regarding the heating performed by the heating unit on the heating object,
The learning method according to claim 18 , wherein the input in the machine learning of the learner also includes the heating information. The learning method according to claim 19, wherein the heating information includes a post-heating time that is the time that has elapsed since the heating unit stopped heating the object to be heated. The learning data is associated with the neighboring temperature and further includes heating object information which is information related to the heating object,
The learning method according to claim 18 , wherein the input in the machine learning of the learning device also includes the heating object information. The learning method according to claim 21, wherein the heated object information includes at least one of cooking vessel information, which is information about a cooking vessel placed above the heating unit, cooking menu information, which is information about a cooking menu to be cooked by the heating unit, and heated object temperature, which is the temperature of the heated object.

Images