JP2021181871A - Heating cooking system and heating cooking method, and learning device and learning method - Google Patents

Abstract

To provide a technology capable of controlling a heating amount of a heating portion, corresponding to a cooking menu cooked by a user.SOLUTION: A heating cooking system is equipped with a heating portion that heats a heating object, an image obtaining portion, a cooking menu estimating portion, and a heating amount control portion. The image obtaining portion obtains an object image before heating, from an imaging device configured to pick up the object image before heating that is an image including a state of the heating object subjected to preparation before heating the heating object with the heating portion. The cooking menu estimating portion inputs the object image before heating obtained by the image obtaining portion in a learned learning apparatus that has performed mechanical learning for estimating a cooking menu that a user cooks by heating the heating object with the heating portion, on the basis of the object image before heating, and executes computing processing of the learned learning apparatus to obtain the cooking menu from the learning apparatus. The heating amount control portion controls a heating amount of the heating portion on the basis of the cooking menu estimated by the cooking menu estimating portion.SELECTED DRAWING: Figure 7

Description

The techniques disclosed herein relate to cooking systems and cooking methods, learning devices and learning methods.

In Patent Document 1, an image of an object to be heated (referred to as a subject image in Patent Document 1) is imaged by an image pickup device (for example, a camera), and a heating cooker that controls the heating amount of a heating unit based on the image. Is disclosed. The cooking device of Patent Document 1 identifies the type of cooking container containing the heating object from the image of the heating object captured by the image pickup apparatus. The cooking device of Patent Document 1 controls the heating amount of the heating unit based on the type of the identified cooking container.

Japanese Unexamined Patent Publication No. 2015-68542

In the cooking device of Patent Document 1, even if the cooking menu is different, when cooking in the same type of cooking container, the heating unit is controlled by the same amount of heating. Therefore, depending on the cooking menu, the heating amount of the heating unit may be excessive or excessive. The present specification provides a technique capable of controlling the heating amount of the heating unit according to the cooking menu to be cooked by the user.

The cooking system according to the above aspect includes a heating unit, an image acquisition unit, a cooking menu estimation unit, and a heating amount control unit. The heating unit heats the object to be heated. The image acquisition unit is an image pickup device capable of capturing an image of the object to be heated, which is an image including the state of the object to be heated, which is prepared before the object to be heated is heated by the heating unit. The image of the object before heating is acquired from. The cooking menu estimation unit is a learned learning device that has been machine-learned for the user to estimate the cooking menu in which the heating object is heated by the heating unit and cooked based on the image of the object before heating. The cooking menu is acquired from the learning device by inputting the image of the object before heating acquired by the image acquisition unit and executing the arithmetic processing of the learned learning device. The heating amount control unit controls the heating amount of the heating unit based on the cooking menu estimated by the cooking menu estimation unit.

According to the above cooking system, the cooking menu is estimated based on the image of the preheating preparation performed by the user during cooking, and the heating amount of the heating unit is controlled according to the estimated cooking menu. Can be done.

Further, the cooking system according to the above aspect may further include a food material information analysis unit that analyzes food material information which is information about food materials contained in the heating object based on the image of the object before heating. .. In that case, the learning device may perform machine learning for estimating the cooking menu based on the food material information. The cooking menu estimation unit also inputs the food material information analyzed by the food material information analysis unit into the learning device, and executes arithmetic processing of the learned learning device to acquire the cooking menu from the learning device. You may.

According to the above-mentioned cooking system, the estimation accuracy of the cooking menu can be improved by estimating the cooking menu based on the food information analyzed by the food information analysis unit.

Further, in the cooking system according to the one aspect, the image pickup device may be configured to be able to capture an image of the object being heated, which is an image including the state of the object to be heated, which is heated by the heating unit. .. In that case, the image acquisition unit may also acquire the image of the object being heated from the image pickup device. The learning device may perform machine learning for estimating the cooking menu based on the image of the object being heated. The cooking menu estimation unit also inputs the image of the object being heated acquired by the image acquisition unit into the learning device, and executes arithmetic processing of the learned learning device to obtain the cooking menu from the learning device. You may get it.

According to the above-mentioned cooking system, the estimation accuracy of the cooking menu can be improved by estimating the cooking menu based on the image of the object being heated.

Further, in the cooking system according to the one aspect, the image pickup device is configured to be able to capture a heating user image which is an image including the state of the user who is heating the heating object by the heating unit. May be good. In that case, the image acquisition unit may also acquire the heating user image from the image pickup device. The learning device may perform machine learning for estimating the cooking menu based on the user image during heating. The cooking menu estimation unit also inputs the heating user image acquired by the image acquisition unit into the learning device, and executes arithmetic processing of the learned learning device to obtain the cooking menu from the learning device. You may get it.

According to the above-mentioned cooking system, the estimation accuracy of the cooking menu can be improved by estimating the cooking menu based on the user image during heating.

Further, the cooking system according to the above aspect may further include a user information analysis unit that analyzes user information that is information about the user based on the heating user image. In that case, the learning device may perform machine learning for estimating the cooking menu based on the user information. The cooking menu estimation unit also inputs the user information analyzed by the user information analysis unit into the learning device, and executes arithmetic processing of the learned learning device to acquire the cooking menu from the learning device. You may.

According to the above-mentioned cooking system, the estimation accuracy of the cooking menu can be improved by estimating the cooking menu based on the user information.

In the cooking method according to the above aspect, the computer may execute the pre-heating image acquisition step, the cooking menu estimation step, and the heating amount control step. In the pre-heating image acquisition step, the computer is configured to be able to capture an image of the pre-heating object, which is an image including the state of the pre-heating object that has been prepared before the heating object is heated by the heating unit. The image of the object before heating is acquired from the image pickup device. In the cooking menu estimation step, the computer has performed machine learning to estimate the cooking menu in which the user heats the heated object in the heating unit and cooks it based on the image of the object before heating. The cooking menu is acquired from the learning device by inputting the acquired image of the object before heating into the learning device and executing the arithmetic processing of the learned learning device. In the heating amount control step, the computer controls the heating amount of the heating unit based on the cooking menu acquired in the cooking menu estimation step.

In the cooking method according to the above aspect, the computer further executes a food information analysis step of analyzing food information, which is information about the food contained in the heating target, based on the image of the object to be heated. May be good. In that case, the learning device may perform machine learning for estimating the cooking menu based on the food material information. In the cooking menu estimation step, the computer also inputs the foodstuff information analyzed in the foodstuff information analysis step into the learning device, and executes the arithmetic processing of the learned learning device to learn the cooking menu. It may be obtained from a vessel.

In the cooking method according to the above aspect, the image pickup apparatus may be configured to be able to capture an image of the object being heated, which is an image including the state of the object being heated by the heating unit. In that case, the learner may perform machine learning for estimating the cooking menu based on the image of the object being heated. The computer may further perform the heating object image acquisition step of acquiring the heating object image from the image pickup device. In the cooking menu estimation step, the computer also inputs the heating object image acquired in the heating object image acquisition step into the learning device, and executes the arithmetic processing of the learned learning device. The cooking menu may be obtained from the learning device.

In the cooking method according to the one aspect, the image pickup apparatus may be configured to be able to capture a heating user image which is an image including a state of the user who is heating the heating object by the heating unit. .. In that case, the learning device may perform machine learning for estimating the cooking menu based on the user image during heating. The computer may further perform a heating user image acquisition step of acquiring the heating user image from the image pickup device. In the cooking menu estimation step, the computer also inputs the heating user image acquired in the heating user image acquisition step into the learning device, and executes the arithmetic processing of the learned learning device to perform the cooking. The menu may be acquired from the learner.

In the cooking method according to the above aspect, the computer may further perform a user information analysis step of analyzing user information which is information about the user based on the heating user image. In that case, the learning device may perform machine learning for estimating the cooking menu based on the user information. In the cooking menu estimation step, the computer also inputs the user information analyzed in the user information analysis step into the learning device, and executes the arithmetic processing of the learned learning device to learn the cooking menu. You may get it from the vessel.

The learning device according to the above aspect may include a learning data acquisition unit and a learning processing unit. The learning data acquisition unit is associated with a cooking menu in which the user heats and cooks the object to be heated in the heating unit and the cooking menu, and preparations are made before heating the object to be heated in the heating unit. The learning data including the pre-heated object image acquired from the image pickup apparatus configured to be able to capture the pre-heated object image which is the image including the state of the pre-heated object is acquired. The learning processing unit performs machine learning of the learning device so as to obtain an output including an output value corresponding to the cooking menu for an input including the image of the object before heating.

According to the above-mentioned learning device, it is possible to construct a learned learning device for estimating the cooking menu to be cooked by the user based on the image of the preheating preparation performed by the user during cooking.

In the learning device according to the above aspect, the learning data is associated with the cooking menu, is information analyzed based on the image of the object before heating, and is related to the foodstuff contained in the object to be heated. Ingredient information, which is information, may be further included. In that case, the input in the machine learning of the learning device may also include the food material information.

According to the above-mentioned learning device, it is possible to construct a learned learning device for estimating a cooking menu to be cooked by a user based on food material information.

In the learning device according to the one aspect, the image pickup device may be configured to be able to capture an image of the object being heated, which is an image including the state of the object to be heated being heated by the heating unit. In that case, the learning data may further include the image of the heated object associated with the cooking menu, and the input in the machine learning of the learner may also include the image of the heated object. ..

According to the above-mentioned learning device, it is possible to construct a learned learning device for estimating the cooking menu to be cooked by the user based on the image of the object being heated.

In the learning device according to the one aspect, the image pickup device may be configured to be able to capture a heating user image which is an image including the state of the user who is heating the heating object by the heating unit. In that case, the learning data may further include the heating user image associated with the cooking menu, and the input in the machine learning of the learning device may also include the heating user image.

According to the above-mentioned learning device, it is possible to construct a learned learning device for estimating a cooking menu to be cooked by a user based on a user image during heating.

In the learning device according to the one aspect, the learning data is associated with the cooking menu, and further includes user information which is information about the user as well as information analyzed based on the heating user image. But it may be. In that case, the input in the machine learning of the learner may also include the user information.

According to the above-mentioned learning device, it is possible to construct a learned learning device for estimating the cooking menu to be cooked by the user based on the user information.

In the learning method according to the above aspect, the computer may include a learning data acquisition step and a learning processing step. In the learning data acquisition step, the computer is associated with a cooking menu in which the user heats and cooks the object to be heated in the heating unit, and the cooking menu before the computer heats the object to be heated in the heating unit. Acquisition of training data including the preheated object image acquired from an image pickup apparatus configured to be capable of capturing an unheated object image which is an image including the state of the preheated object that has been prepared. do. In the learning process step, the computer performs machine learning of the learning device so as to obtain an output including an output value corresponding to the cooking menu for the input including the image of the object before heating.

In the learning method according to the above aspect, the learning data is associated with the cooking menu, is information analyzed based on the image of the object to be heated, and is related to the foodstuff contained in the object to be heated. Ingredient information, which is information, may be further included. In that case, the input in the machine learning of the learning device may also include the food material information.

In the learning method according to the above aspect, the image pickup apparatus may be configured to be able to capture an image of the object being heated, which is an image including the state of the object to be heated being heated by the heating unit. In that case, the learning data may further include the image of the heated object associated with the cooking menu, and the input in the machine learning of the learner may also include the image of the heated object. ..

In the learning method according to the above aspect, the image pickup apparatus may be configured to be able to capture a heating user image which is an image including the state of the user who is heating the heating object by the heating unit. In that case, the learning data may further include the heating user image associated with the cooking menu, and the input in the machine learning of the learning device may also include the heating user image.

In the learning method according to the above aspect, the learning data is associated with the cooking menu, and further includes user information which is information about the user as well as information analyzed based on the heating user image. But it may be. In that case, the input in the machine learning of the learner may also include the user information.

Details and further improvements to the techniques disclosed herein will be described in the "Modes for Carrying Out the Invention" section below.

The perspective view of the cooking system of embodiment which concerns on one aspect is shown. A perspective view of the range hood as viewed from below is shown. The hardware configuration of the cooking system is shown. The hardware configuration of the learning device is shown. The software configuration of the control device is shown. The software configuration of the learning device is shown. The front view of the cooking system is shown. An example of the usage status of the cooking system is shown. Another example of the usage of the cooking system is shown. The flow chart (1) of the process executed by a control device is shown. The flow chart (2) of the process executed by a control device is shown. The flow diagram of the process executed by the learning device is shown.

(Embodiment)
FIG. 1 is a perspective view schematically showing the cooking system 2 of the embodiment according to one aspect. As shown in FIG. 1, the cooking system 2 includes a range hood 4 and a cooking device 10. The cooking cooker 10 is a gas combustion type built-in stove used by incorporating it into a system kitchen. The range hood 4 is a ventilation device arranged above the cooking device 10.

On the top plate 10u of the cooking device 10, a first stove 11a, a second stove 11b, a third stove 11c, and an exhaust port 17 communicating with the grill 28 are arranged. Further, a stove operation unit 20 and a grill 28 are provided on the front surface 10f, which is the front side (that is, the right side of FIG. 1) of the cooking device 10. In the present specification, in order to help understanding, the side where the front surface 10f of the cooking device 10 is provided may be simply referred to as "front side", and the opposite side thereof may be simply referred to as "rear side". The direction connecting the front side and the rear side is the front-back direction. The left and right sides in the horizontal direction orthogonal to the front-back direction are the same as the right side and the left side of the user located on the front side of the cooking cooker 10 and facing the front surface 10f. Further, on the left side of the front surface 10f of the cooking cooker 10, a grill operation unit for operating the grill burner 29a (see FIG. 3) in the grill 28 is provided, but the description thereof is omitted in the present specification.

The first stove 11a, the second stove 11b, and the third stove 11c are arranged in two rows in the front-rear direction on the upper surface of the top plate 10u. The first stove 11a includes a first stove burner 12a, a first sensor 14a, and a trivet 16a. A gas supply path (not shown) is connected to the first stove burner 12a. The gas supply path is provided with a flow rate adjusting valve (not shown) for adjusting the amount of gas supplied to the first stove burner 12a. The first stove burner 12a is ignited by operating an igniter (not shown) with gas supplied to the first stove burner 12a. The first stove 11a heats a cooking container such as a pan or a frying pan placed above by the ignited first stove burner 12a. By adjusting the gas supply amount to the first stove burner 12a, the heating amount of the first stove burner 12a can be adjusted. Further, the fire of the first stove burner 12a is extinguished by stopping the supply of gas to the first stove burner 12a. The second stove burner 12b included in the second stove 11b and the third stove burner 12c included in the third stove 11c have the same structure as the first stove burner 12a.

The first sensor 14a detects the presence of the cooking container of the first stove 11a and also detects the temperature of the cooking container. When the cooking container is arranged on the first stove 11a, the first sensor 14a is pressed by the cooking container. When pressed by the cooking container, the first sensor 14a detects that the cooking container is placed on the first stove 11a. When the cooking container is not placed on the first stove 11a, the first sensor 14a is not pressed. A thermocouple is arranged in the first sensor 14a, and the temperature of the object in contact with the first sensor 14a can be detected. The second sensor 14b included in the second stove 11b and the third sensor 14c included in the third stove 11c have the same structure as the first sensor 14a. Further, the trivets 16a to 16c support the cooking container placed above each stove while maintaining a certain distance from each stove burner.

The stove operation unit 20 includes a power switch 24 of the cooking device 10, a first heating amount operating unit 20a, a second heating amount operating unit 20b, a third heating amount operating unit 20c, and a panel operating unit 22. .. The power switch 24 is a switch for activating the cooking cooker 10, and by turning on the power switch 24, each stove burner can be ignited. The first heating amount operation unit 20a corresponds to the first stove 11a. Similarly, the second heating amount operating unit 20b corresponds to the second stove 11b, and the third heating amount operating unit 20c corresponds to the third stove 11c. The first heating amount operating unit 20a is an alternate type switch for igniting and extinguishing the first stove burner 12a and adjusting the heating amount of the first stove burner 12a. The second heating amount operating unit 20b and the third heating amount operating unit 20c also have the same structure as the first heating amount operating unit 20a.

The panel operation unit 22 displays the operating states of the first stove burner 12a, the second stove burner 12b, the third stove burner 12c, the grill burner 29a (see FIG. 3), and the like. The user can use the panel operation unit 22 to set a fire extinguishing timer for extinguishing each stove burner and grill burner 29a after a predetermined time has elapsed. Further, the panel operation unit 22 can set an automatic heating mode, which will be described later.

The cooking device 10 houses the control device 40 inside. The control device 40 is a computer for automatically controlling the thermal power (that is, the amount of heat) of the first stove 11a, the second stove 11b, the third stove 11c, and the grill burner 29a (see FIG. 3) in the grill 28. be. The control device 40 is connected to the learning device 30 via the Internet. Details of the control device 40 and the learning device 30 will be described later.

Next, the range hood 4 will be described. The range hood 4 houses a fan (not shown) inside. The range hood 4 includes a range cover 6 that covers the fan from below, a camera cover 8, and a speaker 9c.

As shown in FIG. 2, the camera cover 8 is arranged in the rear central portion of the range cover 6. Flange extending downward is provided at the left and right end portions of the camera cover 8. A heating camera 9b and a preheating camera 9a are arranged at the center of the camera cover 8. Both the heating camera 9b and the preheating camera 9a have a hemispherical shape. The pre-heating camera 9a and the heating camera 9b have the same structure, and the imaging range is different from each other. Since the techniques for capturing images by the preheating camera 9a and the heating camera 9b are known, the description thereof will be omitted here.

Further, as shown in FIG. 1, a speaker 9c is arranged at the left end portion of the front surface of the range hood 4. The speaker 9c transmits voice to the user of the cooking device 10.

A control configuration included in the cooking system 2 will be described with reference to FIG. In FIG. 3, only the first stove burner 12a of the first stove 11a is described, and the description of the second stove burner 12b and the third stove burner 12c is omitted. Similarly, in FIG. 3, only the first sensor 14a of the first stove 11a is described, and the description of the second sensor 14b and the third sensor 14c is omitted. When the power switch 24 is turned on, the stove operation unit 20, the panel operation unit 22, the control device 40, and the like are activated.

The control device 40 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory), which are hardware processors. When the stove operation unit 20 is operated, the control device 40 adjusts the thermal power of the first stove burner 12a based on the operation. The control device 40 is configured by, for example, an ECU (Electronic Control Unit).

The control device 40 includes, for example, a learning storage unit 46 composed of RAM, ROM, or the like, a menu storage unit 48, and an external interface (described as an external I / F in FIG. 3) 42. .. The learning storage unit 46 stores the program 46p and the learning result data 32r. The program 46p includes an instruction for causing the control device 40 to execute a process of estimating a cooking menu to be cooked by the user in the cooking device 10. The learning result data 32r is data for setting the learned learning device, is generated by the learning device 30 shown in FIG. 1, and is transmitted to the control device 40 via the Internet.

The menu storage unit 48 stores a heating amount table 48t composed of a plurality of menu names and a heating amount pattern corresponding to each of the menu names. The menu name is the name of a menu such as curry or simmered food that a user cooks using the cooking device 10. The heating amount pattern is, for example, a control pattern over time in which the heating power of the first stove burner 12a is set to high heat for 15 minutes, and then the heating power is reduced and set to low heat for 20 minutes. The heating amount table 48t may be stored in advance in the menu storage unit 48 at the time of manufacturing the cooking cooker 10, or may be downloaded from an external server via an external interface 42 capable of communicating with an external server. It may be stored in the menu storage unit 48.

The external interface 42 is an interface for connecting the control device 40 and the external device. As shown in FIG. 3, the external interface 42 connects the control device 40 to the preheating camera 9a, the heating camera 9b, and the speaker 9c. The external interface 42 is connected to the preheating camera 9a, the heating camera 9b, and the speaker 9c by, for example, a Wi-Fi (registered trademark) method.

The learning device 30 will be described with reference to FIG. The learning device 30 includes a storage unit 32, an input device 34i, an output device 34о, a control unit 36, a communication interface (described as a communication I / F in FIG. 4) 38, and a drive 39. There is. As shown in FIG. 3, the learning device 30 is a computer to which each device is electrically connected.

The control unit 36 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to execute various information processing based on programs and data. The storage unit 32 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 32 stores the learning program 32p executed by the control unit 36, the learning data 32d used for machine learning of the learning device, and the learning result data 32r created by executing the learning program 32p.

The learning program 32p is a program for causing the learning device 30 to execute a machine learning process (FIG. 12) described later and generate learning result data 32r as a result of the machine learning. The learning data 32d is data for performing machine learning of the learning device so as to estimate the cooking menu from the input information. Details will be described later.

The communication interface 38 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. The learning device 30 may distribute the created learning result data 32r to an external device via the communication interface 38.

The input device 34i is, for example, a device for inputting a mouse, a keyboard, or the like. The output device 34о is, for example, a device for outputting a display, a speaker, or the like. The operator of the learning device 30 can operate the learning device 30 via the input device 34i and the output device 34о.

The drive 39 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 30 m. The type of the drive 39 may be appropriately selected according to the type of the storage medium 30 m. The learning program 32p and the learning result data 32r may be stored in the storage medium 30m.

The storage medium 30 m transfers the information of the program or the like by electrical, magnetic, optical, mechanical or chemical action so that the information of the program or the like recorded by the computer or other device, the machine or the like can be read. It is a medium to accumulate. The learning device 30 may acquire the learning program 32p and the learning data 32d from the storage medium 30m.

FIG. 4 illustrates a disc-type storage medium such as a CD or DVD as an example of the storage medium 30 m. However, the storage medium 30 m is not limited to the disk type. The storage medium 30m may be, for example, a semiconductor memory such as a flash memory.

Regarding the specific hardware configuration of the learning device 30, components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 36 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, an FPGA (field-programmable gate array), or the like. The learning device 30 may be composed of a plurality of information processing devices. Further, the learning device 30 may be a general-purpose server device, a PC (Personal Computer), or the like, in addition to an information processing device designed exclusively for the provided service.

The control configuration of the control device 40 will be described with reference to FIG. FIG. 5 schematically shows the control configuration of the control device 40.

The control device 40 expands the program 46p stored in the learning storage unit 46 into the RAM. The control device 40 controls each component by interpreting and executing the program 46p expanded in the RAM by the CPU. As a result, as shown in FIG. 5, the control device 40 has an image acquisition unit 51, an image analysis unit 52, a resolution conversion unit 53, a heating amount information acquisition unit 54, a cooking menu estimation unit 55, and an alarm unit 56 as software modules. , And a computer including a heating amount control unit 57.

The image acquisition unit 51 acquires the captured image 9i from the preheating camera 9a and the heating camera 9b. The details will be described with reference to FIGS. 7 to 9, but in the captured image 9i, the state of the foodstuff prepared before heating in each stove burner of the cooking cooker 10 and the user during heating are shown. The condition of ingredients, cooking containers, etc. is included. The image acquisition unit 51 may receive, for example, an image captured by the user with a mobile terminal.

The image analysis unit 52 analyzes the captured image 9i acquired by the image acquisition unit 51, and extracts the food material information 52f, which is information about the foodstuff, and the user information 52u, which is information about the user, from the captured image 9i. Further, the heating amount information acquisition unit 54 acquires heating amount information 57q, which is information on the heating amount of each stove burner, from each stove burner of the cooking device 10.

The resolution conversion unit 53 reduces the resolution of the captured image 9i acquired by the image acquisition unit 51. As a result, the resolution conversion unit 53 generates the low-resolution captured image 62.

The cooking menu estimation unit 55 is attached to a trained learner (neural network 70) that has been machine-learned to estimate a cooking menu that the user heats and cooks on each stove of the cooker 10. The low-resolution captured image 62 obtained by reducing the resolution of the image 62 is input. Further, the food material information 52f, the user information 52u, and the heating amount information 57q are also input as supplementary information 60 in the same manner. As a result, the cooking menu estimation unit 55 acquires the cooking menu information 64 from the learning device. By inputting the low-resolution captured image 62 obtained by reducing the resolution to the neural network 70, the amount of calculation of the arithmetic processing of the neural network 70 can be reduced, and the load on the processor can be reduced. The process of reducing the resolution may be omitted. In this case, the cooking menu estimation unit 55 may input the captured image 9i into the learning device (neural network 70).

The cooking menu information 64 includes the menu name of the heating amount table 48t stored in the menu storage unit 48 of FIG. The heating amount control unit 57 has a heating amount pattern corresponding to the menu name included in the cooking menu information 64 acquired from the learning device by the cooking menu estimation unit 55, and has a heating amount pattern corresponding to each stove burner of the cooking device 10 and the grill burner 29a in the grill 28. Controls the amount of heat (ie, thermal power) of the stove.

Further, when the pattern of the heating amount corresponding to the menu name included in the cooking menu information 64 is completed, the alarm unit 56 notifies the user that the heating is completed via the speaker 9c (see FIG. 3). Further, the alarm unit 56 may notify the user, for example, when the captured image 9i is not normally acquired. Further, the alarm unit 56 may notify the user that the heating has been completed by, for example, sending a message to the user's mobile terminal without notifying the user via the speaker 9c. Further, the speaker 9c may be arranged in the cooking device 10.

As shown in FIG. 5, the control device 40 uses the neural network 70 as a trained learner that has undergone machine learning for the user to estimate the cooking menu to be heated by each stove burner of the cooker 10. .. The neural network 70 is configured by combining a plurality of types of neural networks.

The neural network 70 is divided into four parts: a fully coupled neural network 72, a convolutional neural network 74, a coupled layer 76, and an LSTM network 78. The fully connected neural network 72 and the convolutional neural network 74 are arranged in parallel on the input side, supplementary information 60 is input to the fully connected neural network 72, and a low-resolution captured image 62 is input to the convolutional neural network 74. .. The coupling layer 76 couples the outputs of the fully coupled neural network 72 and the convolutional neural network 74. The LSTM network 78 receives the output from the coupling layer 76 and outputs the cooking menu information 64.

The fully connected neural network 72 is a so-called multi-layered neural network, and includes an input layer 72i, an intermediate layer (hidden layer) 72m, and an output layer 72о in this order from the input side. However, the number of layers of the fully connected neural network 72 does not have to be limited to such an example, and may be appropriately selected depending on the embodiment.

Each layer 72i, 72m, 72о of the fully connected neural network 72 comprises one or more neurons (nodes). The number of neurons included in each layer may be appropriately set according to the embodiment. The fully connected neural network 72 is formed by connecting each neuron included in each layer to all the neurons included in the adjacent layer. Weights (bonding loads) are appropriately set for each bond.

The convolutional neural network 74 is a feedforward neural network having a structure in which convolutional layers 74a and pooling layers 74b are alternately connected. In the convolutional neural network 74 according to the present embodiment, a plurality of convolutional layers 74a and pooling layers 74b are alternately arranged on the input side. Then, the output of the pooling layer 74b arranged on the most output side is input to the fully connected layer 74c, and the output of the fully connected layer 74c is input to the output layer 74о.

The convolution layer 74a is a layer that performs an operation for convolution of an image. Image convolution corresponds to a process of calculating the correlation between an image and a predetermined filter. Therefore, by convolving the image, for example, a shading pattern similar to the shading pattern of the filter can be detected from the input image.

The pooling layer 74b is a layer for performing a pooling treatment. The pooling process discards part of the information on the position where the response to the image filter is strong, and realizes the invariance of the response to the minute position change of the feature appearing in the image.

The fully connected layer 74c is a layer that connects all neurons between adjacent layers. That is, each neuron contained in the fully connected layer 74c is connected to all neurons contained in the adjacent layer. The fully bonded layer 74c may be composed of two or more layers. Further, the number of neurons included in the fully connected layer 74c may be appropriately set according to the embodiment.

The output layer 74о is a layer arranged on the most output side of the convolutional neural network 74. The number of neurons included in the output layer 74о may be appropriately set according to the embodiment. The configuration of the convolutional neural network 74 may not be limited to such an example, and may be appropriately set according to the embodiment.

The coupling layer 76 is arranged between the fully coupled neural network 72 and the convolutional neural network 74 and the LSTM network 78. The coupling layer 76 combines the output from the output layer 72о of the fully coupled neural network 72 and the output from the output layer 74о of the convolutional neural network 74. The output of the coupling layer 76 is input to the LSTM network 78. The number of neurons included in the connecting layer 76 may be appropriately set according to the number of outputs of the fully connected neural network 72 and the convolutional neural network 74.

The LSTM network 78 is a recurrent neural network including an LSTM block 78b. A recurrent neural network is a neural network having an internal loop, for example, a path from an intermediate layer to an input layer. The LSTM network 78 has a structure in which the intermediate layer of a general recurrent neural network is replaced with an LSTM block 78b.

The LSTM network 78 includes an input layer 78i, an LSTM block 78b, and an output layer 78о in this order from the input side, and has a path for forward propagation and a path for returning from the LSTM block 78b to the input layer 78i. The number of neurons included in the input layer 78i and the output layer 78о may be appropriately set according to the embodiment.

The LSTM block 78b is a block including an input gate and an output gate so that the timing of information storage and output can be learned. Further, the LSTM block 78b may include a forgetting gate for adjusting the timing of forgetting information. The configuration of the LSTM network 78 can be appropriately set according to the embodiment.

A threshold is set for each neuron, and basically, the output of each neuron is determined by whether or not the sum of the products of each input and each weight exceeds the threshold. The control device 40 inputs the supplementary information 60 to the fully connected neural network 72, and inputs the low-resolution captured image 62 to the convolutional neural network 74. Then, the control device 40 determines the firing of each neuron included in each layer in order from the input side. In this way, the neural network 70 executes arithmetic processing on the input. The control device 40 acquires the output value corresponding to the cooking menu information 64 from the output layer 78о of the neural network 70.

The configuration of such a neural network 70 (for example, the number of layers in each network, the number of neurons in each layer, the connection relationship between neurons, the transmission function of each neuron), the weight of the connection between each neuron, and each neuron The information indicating the threshold value is included in the learning result data 32r. The control device 40 sets the trained neural network 70 used for the process of estimating the cooking menu to be heated by the user in each stove burner of the cooking cooker 10 with reference to the learning result data 32r.

The learning device 30 will be described with reference to FIG. The control unit 36 of the learning device 30 expands the learning program 32p stored in the storage unit 32 into the RAM. Then, the control unit 36 interprets and executes the learning program 32p expanded in the RAM by the CPU, and controls each component. As a result, as shown in FIG. 6, the learning device 30 is configured as a software module as a computer including a learning data acquisition unit 31 and a learning processing unit 33.

The learning data acquisition unit 31 includes low-resolution captured images 62t obtained by lowering the resolution of the captured images 9i captured by the pre-heating camera 9a and the heating camera 9b, heating amount information 57q, food material information 52f, and user information. A set of supplementary information 60t composed of 52u, low-resolution captured image 62t, and cooking menu information 64t corresponding to supplementary information 60t is acquired as learning data 66. The low-resolution captured image 62t and the supplementary information 60t correspond to the low-resolution captured image 62 and the supplementary information 60 described with reference to FIG. 5, respectively, and are used as input data. The cooking menu information 64t corresponds to the cooking menu information 64 in FIG. 5 and is used as teacher data (correct answer data). When the low-resolution captured image 62t and the supplementary information 60t are input, the learning processing unit 33 performs machine learning of the learning device so as to output an output value corresponding to the cooking menu information 64t.

As shown in FIG. 6, like the control device 40 (see FIG. 5), the learning device 30 includes a neural network 70t. The neural network 70t of the learning device 30 includes a fully coupled neural network 72t, a convolutional neural network 74t, a coupling layer 76t, and an LSTM network 78t. The neural network 70t of the learning device 30 is configured in the same manner as the neural network 70 of the control device 40. Therefore, the fully connected neural network 72t, the convolutional neural network 74t, the connecting layer 76t, and the LSTM network 78t of the learning device 30 are the fully connected neural network 72, the convolutional neural network 74, the connecting layer 76, and the LSTM network of the control device 40, respectively. It is the same as 78. That is, the neural network 70t of the learning device 30 executes arithmetic processing in the same manner as the neural network 70 of the control device 40.

When the learning processing unit 33 inputs supplementary information 60t to the fully connected neural network 72t and inputs a low-resolution captured image 62t to the convolutional neural network 74t, the learning processing unit 33 outputs an output value corresponding to the cooking menu information 64t from the LSTM network 78t. Build a network 70t. Then, the learning processing unit 33 stores information indicating the configuration of the constructed neural network 70, the weight of the connection between each neuron, and the threshold value of each neuron in the storage unit 32 as learning result data 32r. The learning processing unit 33 periodically transmits the learning result data 32r stored in the storage unit 32 to the control device 40 via the Internet. As a result, the learning result data 32r of the control device 40 is periodically updated.

With reference to FIGS. 7 to 9, the structure in which the cooking system 2 captures the captured image 9i input to the control device 40 will be described. In the following, the case where the foodstuff is heated mainly by the first stove 11a and the grill 28 of the cooking device 10 will be described, but the same applies to the case where the second stove 11b and the third stove 11c are used for heating.

The pre-heating camera 9a and the heating camera 9b are arranged side by side in the left-right direction. The preheating camera 9a is arranged on the right side. The preheating camera 9a has an imaging area a20 extending to the right. A pre-heating preparation space 80 is provided on the right side of the heating cooker 10. That is, the preheating camera 9a is provided on the preheating preparation space 80 side of the camera cover 8.

A cutting board 82 is arranged in the pre-heating preparation space 80, and the pre-heating foodstuff 84a is placed on the upper surface of the cutting board 82. As shown in FIG. 7, a part of the preheated foodstuff 84a is cut. The pre-heated food material 84a is prepared before being heated by the first stove 11a of the cooking device 10. As shown in FIG. 7, the preheated food material 84a is placed in the imaging area a20 of the preheating camera 9a. That is, the pre-heating camera 9a is configured to be able to capture an image of the pre-heating food material including the state of the pre-heating food material 84a in which the pre-heating food material 84a is prepared before being heated by the first stove 11a.

The pre-heated food image contains information such as the type of food of the pre-heated food 84a, how to cut it, and the type and amount of seasoning added to the pre-heated food 84a in the pre-heating preparation. Therefore, as described with reference to FIG. 5, the image analysis unit 52 analyzes the preheating food material image, and the cooking menu estimation unit 55 inputs these information to the neural network 70 as supplementary information 60. can. As a result, the estimation accuracy of the cooking menu estimation unit 55 can be improved.

The heating camera 9b has an imaging area a10 extending downward. As shown in FIG. 7, the imaging area a10 of the heating camera 9b includes the first stove 11a, the second stove 11b, and the third stove 11c. The heating camera 9b can take an image of a state in which the heating foodstuff 84b housed inside the pot 86 placed on the first stove 11a is heated by the first stove 11a. That is, the heating camera 9b is configured to be able to capture an image of the heated food material including the state of the heated food material 84b heated by the first stove 11a. The pre-heating camera 9a and the heating camera 9b may be integrally formed, and in that case, the imaging area may be configured to be changeable in the left-right direction.

The image of the food being heated includes the size of the cooking container (pot 86 in FIG. 7), the type of cooking container (for example, a frying pan, etc.), the heating status (color, boiling status, etc.) of the food being heated 84b, and the inside of the cooking container. Contains information such as the amount of water, the amount of oil, and the type and amount of seasoning added to the cooking pan. Therefore, as described with reference to FIG. 5, the image analysis unit 52 analyzes the image of the food being heated, and the cooking menu estimation unit 55 inputs these information to the neural network 70 as supplementary information 60. can. As a result, the estimation accuracy of the cooking menu estimation unit 55 can be improved.

As shown in FIG. 8, the heating camera 9b is configured so that the imaging area a11 can also be imaged. As a result, the heating camera 9b can also capture a heating user image of the user 100 who executes cooking by the heating cooker 10.

In the heating user image, the posture of the user 100 (head and body orientation, hand position, etc.), the size of the cooking utensil (ladle 90 in FIG. 8) possessed by the user 100, and the type of cooking utensil (in FIG. 8). For example, information such as chopsticks) is also included. Further, by continuously capturing the state of the user 100 by the heating camera 9b, information on the movement of the user 100 (for example, the movement of adding seasonings, the movement of stirring the inside of the cooking container, etc.) is also included in the heating user image. be able to. Therefore, as described with reference to FIG. 5, the image analysis unit 52 analyzes the user image during heating, and the cooking menu estimation unit 55 inputs these information to the neural network 70 as supplementary information 60. can. As a result, the estimation accuracy of the cooking menu estimation unit 55 can be improved.

As shown in FIG. 9, the heating camera 9b is configured so that the imaging area a13 can also be imaged. As a result, the heating camera 9b can also capture an image of the food in the grill including the state of the food 88 being heated in the grill 28.

Similar to the image of the food being heated, the image of the food in the grill also contains information such as the heating status of the food 88 being heated, the amount of water in the grill 28, the amount of oil, and the type and amount of the seasoning added in the grill 28. included. Therefore, as described with reference to FIG. 5, the image analysis unit 52 analyzes the image of the food being heated, and the cooking menu estimation unit 55 inputs these information to the neural network 70 as supplementary information 60. can. As a result, the estimation accuracy of the cooking menu estimation unit 55 can be improved.

Further, as described with reference to FIG. 5, the control device 40 can also acquire the heating amount information 57q, which is information on the heating amount of each stove and the grill 28, by the heating amount information acquisition unit 54. Therefore, as described with reference to FIG. 5, the cooking menu estimation unit 55 can input the heating amount information 57q into the neural network 70 as supplementary information. As a result, the estimation accuracy of the cooking menu estimation unit 55 can be improved.

A process executed by the control device 40 (see FIG. 3) will be described with reference to FIGS. 10 and 11. The process executed by the control device 40 described with reference to FIGS. 10 and 11 is an example of the “cooking method”.

The user turns on the power switch 24 to activate the cooking cooker 10. After that, the user starts the automatic heating mode by setting in the panel operation unit 22 (step S2). When setting the automatic heating mode, the user selects a stove burner or a grill burner to heat in the automatic heating mode. Hereinafter, the case where the first stove burner 12a of the first stove 11a is selected will be described.

When the automatic heating mode is started, as described with reference to FIG. 7, the control device 40 causes the preheating camera 9a to take an image of the preheated food material, and acquires the preheating food material image from the preheating camera 9a. (Step S4). Further, the control device 40 acquires the food material information 52f as supplementary information 60 by the image analysis unit 52 shown in FIG. 5 (step S6). Next, the control device 40 lowers the resolution of the preheated food material image by the resolution conversion unit 53 shown in FIG. 5 (step S8). As described above, the step S8 for lowering the resolution of the food material image before heating can be omitted.

After that, the control device 40 inputs the low-resolution captured image 62 and the supplementary information 60 (that is, the food material information 52f) to the neural network 70 by the cooking menu estimation unit 55, as described with reference to FIG. As described above, the neural network 70 has received the learning result data 32r from the learning device 30 via the Internet. That is, the neural network 70 is a trained learner that has performed a machine learner for estimating a cooking menu based on a pre-heated food image. As described above, the neural network 70 to which the low-resolution captured image 62 is input outputs the output value corresponding to the cooking menu information 64 (see FIG. 5). As a result, the control device 40 acquires the cooking menu information 64 by the cooking menu estimation unit 55 (step S12).

After that, the first stove burner 12a selected by the user in step S2 is ignited (step S14). The control device 40 detects that the first stove burner 12a selected in step S2 is ignited by the heating amount control unit 57. The control device 40 has a heating amount corresponding to the cooking menu information 64 acquired in step S12 in the heating amount table 48t stored in the menu storage unit 48 (see FIG. 3) by the heating amount control unit 57. Control of the heating amount of the first stove burner 12a is started using the pattern of (heat) (step S16). As a result, the heating amount control unit 57 controls the heating amount of the first stove burner 12a based on the cooking menu estimated by the cooking menu estimation unit 55.

When the pattern of the heating amount corresponding to the cooking menu information 64 is completed, the control device 40 ends the pattern control of the heating amount (heat power) by the heating amount control unit 57 (step S18). After that, the control device 40 notifies the user that the heating is completed via the speaker 9c (see FIG. 3) by the alarm unit 56 (see FIG. 5) (step S20). As a result, the control device 40 and the user can be notified that the heating of the cooking menu estimated by the heating pretreatment performed by the user is completed.

As described above, the cooking system 2 of the embodiment (see FIG. 1) can also acquire an image during heating. As shown in FIG. 11, the control device 40 acquires an image of the heated food (or user) after step 16 in FIG. 10 (step S4b). As described with reference to FIGS. 7 and 8, the heating camera 9b acquires images of the food (or user) being heated. When the heating user image is acquired in step 4b, the control device 40 acquires the user information 52u as supplementary information 60 by the image analysis unit 52 shown in FIG. 5 (step S6b). Further, the control device 40 may acquire the heating amount information 57q as supplementary information 60 by the heating amount information acquisition unit 54 shown in FIG.

Next, the control device 40 lowers the resolution of the heated food (or user) image by the resolution conversion unit 53 shown in FIG. 5 (step S8b). After that, the control device 40 inputs the low-resolution captured image 62 and the supplementary information 60 (that is, the user information 52u and the heating amount information 57q) to the neural network 70 by the cooking menu estimation unit 55. As described above, the cooking menu estimation unit 55 estimates the cooking menu by the neural network 70. Hereinafter, the processing from step S10b to step S16b in FIG. 11 is the same as the step in FIG. 10 above, and thus the description thereof will be omitted. In this way, the cooking menu is estimated by the pre-heating food image and the food information 52f, and then re-estimated by the heating food image and the heating user image, whereby the estimation accuracy of the cooking menu estimation unit 55 (see FIG. 5) can be obtained. Can be improved. Further, by acquiring the user information 52u obtained by analyzing the image and the heating amount information 57q which is information on the heating amount during heating as supplementary information 60, in addition to the image during heating, the cooking menu estimation unit 55 (FIG. 5). The estimation accuracy of (see) can be improved. The image acquisition unit 51 may acquire both the image of the food being heated and the image of the user during heating at the same time, or may acquire either one first and then the other.

A process executed by the learning device 30 (see FIG. 6) will be described with reference to FIG. The process executed by the learning device 30 described with reference to FIG. 12 is an example of the “learning method”.

As shown in FIG. 6, the learning device 30 acquires a set of a low-resolution captured image 62t, supplementary information 60t, and cooking menu information 64t as learning data 66 by the learning data acquisition unit 31 (step S30).

The learning data 66 is data used for machine learning for making the cooking menu inestifiable for the neural network 70t. Such learning data 66 can be created by associating images of various ingredients prepared before heating with an estimated cooking menu. At that time, images of various foodstuffs prepared before heating are stored at the same resolution as the low-resolution captured image 62 of FIG. Images of various foodstuffs prepared before heating may be at least as long as the cutting method, size, color, etc. of the foodstuff can be discriminated.

Further, the learning data 66 can also be created by associating the images of various foodstuffs and cooking containers being heated with the cooking menu. At that time, images of various foodstuffs and cooking containers being heated are stored at the same resolution as the low-resolution captured image 62 of FIG. The images of various foodstuffs and cooking containers being heated may be at least as long as the color, size, color of foodstuffs, the color and size of the cooking container, the color inside the cooking container, and the like can be discriminated.

Further, the learning data 66 can also be created by associating the image of the user being heated with the cooking menu. At that time, the image of the user being heated is stored at the same resolution as the low-resolution captured image 62 of FIG. The image of the user during heating may be at least as long as the posture of the user, the orientation of the face, the type, size, and color of the cooking utensil possessed can be discriminated.

The food ingredient information 52f included in the supplementary information 60 can be obtained by analyzing images of various foodstuffs prepared before heating and extracting information on the type of foodstuff, how to cut the foodstuff, and the like. Similarly, the user information 52u included in the supplementary information 60 can be obtained by analyzing the image of the user during heating and extracting information regarding the posture of the user, the cooking utensils possessed by the user, and the like. The learning data 66 can also be created by associating the extracted information with the cooking menu.

The learning data 66 may be created manually by an operator or the like using the input device 34i (see FIG. 4), or may be automatically created by processing a program. The learning data 66 may be collected from the control device 40 at any time. Further, the learning data 66 may be created by an information processing device other than the learning device 30. When the learning device 30 creates the learning data 66, the control unit 36 can acquire the learning data 66 by executing the learning data 66 creation process in this step S30. On the other hand, when an information processing device other than the learning device 30 creates the learning data 66, the learning device 30 is created by the other information processing device via a network, a storage medium 30 m (see FIG. 4), or the like. The learned data 66 can be acquired. The number of learning data 66 acquired in this step S30 may be appropriately determined according to the embodiment so that machine learning of the neural network 70t can be performed.

Next, the learning device 30 outputs the output value corresponding to the cooking menu information 64t when the low-resolution captured image 62t and the supplementary information 60t are input by the learning processing unit 33 using the learning data 66 acquired in step S30. The machine learning process of the neural network 70t is executed (step S32).

The control unit 36 (see FIG. 4) of the learning device 30 prepares a neural network 70t to be subjected to the learning process. The configuration of the prepared neural network 70t, the initial value of the weight of the connection between each neuron, and the initial value of the threshold value of each neuron may be given by the template or by the input of the operator. Further, when performing re-learning, the control unit 36 may prepare a neural network 70t based on the learning result data 32r to be re-learned.

Next, the control unit 36 uses the low-resolution captured image 62t and supplementary information 60t included in the learning data 66 acquired in step S30 as input data, and uses the cooking menu information 64t as teacher data (correct answer data), and is neural. The learning process of the network 70 is performed. A stochastic gradient descent method or the like may be used for the learning process of the neural network 70.

For example, the control unit 36 inputs supplementary information 60t to the input layer of the fully connected neural network 72t, and inputs the low-resolution captured image 62t to the convolutional layer arranged on the most input side of the convolutional neural network 74t. Then, the control unit 36 determines the firing of each neuron included in each layer in order from the input side. As a result, the control unit 36 obtains an output value from the output layer of the LSTM network 78t. Next, the control unit 36 calculates an error between the output value acquired from the output layer of the LSTM network 78t and the value corresponding to the cooking menu information 64t. Subsequently, the control unit 36 calculates the connection weight between each neuron and the error of each threshold value of each neuron using the error of the output value calculated by the backpropagation through time method. do. Then, the control unit 36 updates the weight of the connection between each neuron and the value of each threshold value of each neuron based on each calculated error.

The control unit 36 repeats this series of processes for each learning data 66 until the output value output from the neural network 70t matches the value corresponding to the cooking menu information 64t. As a result, the control unit 36 can construct a neural network 70t that outputs an output value corresponding to the cooking menu information 64t when the low-resolution captured image 62t and the supplementary information 60t are input.

The control unit 36 stores information indicating the configuration of the neural network 70 constructed by the learning processing unit 33, the weight of the connection between each neuron, and the threshold value of each neuron in the storage unit 32 as learning result data 32r (step S34). ). As a result, the control unit 36 ends the learning process of the neural network 70 according to this operation example.

The control unit 36 may transfer the created learning result data 32r to the control device 40 after the process of step S36 is completed. Further, the control unit 36 may periodically update the learning result data 32r by periodically executing the learning processes of steps S30 to S36. Then, the control unit 36 may periodically update the learning result data 32r held by the control device 40 by transferring the created learning result data 32r to the control device 40 each time the learning process is executed. Further, for example, the control unit 36 may store the created learning result data 32r in another data server. In this case, the control device 40 may acquire the learning result data 32r from this data server.

(Correspondence) Ingredients 84a, 84b, 88 are examples of "heated objects", respectively. Each stove burner and grill burner 29a of the cooking device 10 is an example of a “heating unit”. The pre-heating camera 9a and the heating camera 9b are examples of the “imaging device”. The image analysis unit 52 is an example of the “food material information analysis unit” and the “user information analysis unit”.

Although the examples have been described in detail above, these are merely examples and do not limit the scope of claims. The techniques described in the claims include various modifications and modifications of the specific examples exemplified above. Modifications of the above embodiment are listed below.

(Modification 1) In the above-mentioned cooking system 2, the food material information 52f is extracted from the pre-heating food material image by the image analysis unit 52 and input to the learned learner (neural network 70) as supplementary information 60. .. Instead of this, in the modified example, the configuration may be such that the food material information 52f is not input. In that case, the control device 40 does not have to include the image analysis unit 52.

(Modification 2) In the above-mentioned cooking system 2, the image of the food being heated captured by the heating camera 9b is also acquired by the image acquisition unit 51 and input to the trained learner (neural network 70). Instead of this, in the modified example, it is not necessary to acquire the image of the food material during heating.

(Modification 3) In the above-mentioned cooking system 2, the heating user image captured by the heating camera 9b is also acquired by the image acquisition unit 51 and input to the trained learner (neural network 70). Instead of this, in the modified example, it is not necessary to acquire the user image during heating.

(Modification 4) In the above-mentioned cooking system 2, user information 52u is extracted from the heating user image by the image analysis unit 52 and input to the learned learner (neural network 70) as supplementary information 60. .. Instead of this, in the modified example, the user information 52u may not be input. In that case, the control device 40 does not have to include the image analysis unit 52.

(Variation Example 5) In the above-mentioned cooking system 2, the control device 40 detects that the stove or grill selected by the user has been ignited by the user, and controls the pattern of the heating amount of the selected stove or grill. Start. Instead, in the variant, when the sensor on the selected stove or grill detects that a cooking container or ingredient has been placed on the selected stove or grill, the controller selects the stove or grill. May be ignited.

(Modification 6) The above-mentioned cooking system 2 is a gas combustion type built-in stove used by incorporating it into a system kitchen. Instead of this, in the modified example, an electromagnetic induction heating cooker (IH cooker) may be used. In that case, the heating amount control unit 57 of the control device 40 may control the electric power of each cooker instead of the thermal power.

Although specific examples of the present invention have been described in detail above, these are merely examples and do not limit the scope of claims. The techniques described in the claims include various modifications and modifications of the specific examples exemplified above. The technical elements described herein or in the drawings exhibit their technical usefulness alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. In addition, the techniques exemplified in the present specification or the drawings can achieve a plurality of purposes at the same time, and achieving one of the purposes itself has technical usefulness.

2: Cooking system 4: Range hood 6: Range cover 8: Camera cover 9a: Pre-heating camera 9b: Heating camera 9c: Speaker 10: Heating cooker 11a: 1st stove 12a: 1st stove burner 14a: 1st Sensor 16a: Gotoku 17: Exhaust port 20: Stove operation unit 20a: First heating amount operation unit 22: Panel operation unit 24: Power switch 28: Grill 29a: Grill burner 30: Learning device 30m: Storage medium 31: Learning data acquisition unit 32: Storage unit 33: Learning processing unit 34i: Input device 34о: Output device 36: Control unit 38: Communication interface 39: Drive 40: Control device 42: External interface 46: Learning storage unit 48: Menu storage unit 48t: Heat amount Table 51: Image acquisition unit 52: Image analysis unit 53: Resolution conversion unit 54: Heating amount information acquisition unit 55: Cooking menu estimation unit 56: Alarm unit 57: Heating amount control unit 70, 70t: Neural network 80: Preparation before heating Space 84a: Unheated food 84b, 88: Heating food 86: Pot

Claims (20)

A heating unit that heats the object to be heated,
Before heating, the image pickup device configured to be able to capture an image of the object to be heated, which is an image including the state of the object to be heated, which is prepared before heating the object to be heated by the heating unit. The image acquisition unit that acquires the object image and
The image acquisition unit is attached to a learned learning device that has been machine-learned to estimate a cooking menu in which a user heats and cooks the object to be heated by the heating unit based on the image of the object before heating. A cooking menu estimation unit that acquires the cooking menu from the learning device by inputting the acquired image of the object before heating and executing arithmetic processing of the learned learning device.
A cooking system including a heating amount control unit that controls the heating amount of the heating unit based on the cooking menu estimated by the cooking menu estimation unit. It also has a food information analysis unit that analyzes food information, which is information about foods contained in the heating object, based on the image of the object before heating.
The learning device performs machine learning for estimating the cooking menu based on the food material information.
The cooking menu estimation unit also inputs the food material information analyzed by the food material information analysis unit into the learning device, and executes arithmetic processing of the learned learning device to acquire the cooking menu from the learning device. The cooking system according to claim 1. The image pickup apparatus is configured to be capable of capturing an image of an object being heated, which is an image including a state of the object to be heated that is being heated by the heating unit.
The image acquisition unit also acquires an image of the object being heated from the image pickup device.
The learning device performs machine learning for estimating the cooking menu based on the image of the object being heated.
The cooking menu estimation unit also inputs the image of the object being heated acquired by the image acquisition unit into the learning device, and executes arithmetic processing of the learned learning device to obtain the cooking menu from the learning device. The cooking system according to claim 1 or 2 to be acquired. The image pickup device is configured to be able to capture a heating user image which is an image including the state of the user who is heating the heating object by the heating unit.
The image acquisition unit also acquires the heating user image from the image pickup device.
The learning device performs machine learning for estimating the cooking menu based on the user image during heating.
The cooking menu estimation unit also inputs the heating user image acquired by the image acquisition unit to the learning device, and executes arithmetic processing of the learned learning device to obtain the cooking menu from the learning device. The cooking system according to any one of claims 1 to 3 to be acquired. It further includes a user information analysis unit that analyzes user information that is information about the user based on the heating user image.
The learning device performs machine learning for estimating the cooking menu based on the user information.
The cooking menu estimation unit also inputs the user information analyzed by the user information analysis unit into the learning device, and executes arithmetic processing of the learned learning device to acquire the cooking menu from the learning device. The cooking system according to claim 4. The computer
From the image pickup device configured to be able to capture an image of the object to be heated, which is an image including the state of the object to be heated, which is prepared before the object to be heated is heated by the heating unit, the object to be heated is said to be the object to be heated. Pre-heating image acquisition step to acquire image and
The pre-heating acquired to a learned learner that has undergone machine learning for estimating a cooking menu in which a user heats and cooks the object to be heated in the heating unit based on the image of the object before heating. A cooking menu estimation step for acquiring the cooking menu from the learning device by inputting an object image and executing arithmetic processing of the learned learning device.
A cooking method for executing a heating amount control step for controlling the heating amount of the heating unit based on the cooking menu acquired in the cooking menu estimation step. The computer further executes a food information analysis step of analyzing food information, which is information about the food contained in the heating object, based on the image of the object to be heated.
The learning device performs machine learning for estimating the cooking menu based on the food material information.
In the cooking menu estimation step, the computer also inputs the foodstuff information analyzed in the foodstuff information analysis step into the learning device, and executes the arithmetic processing of the learned learning device to learn the cooking menu. The cooking method according to claim 6, which is obtained from a vessel. The image pickup apparatus is configured to be capable of capturing an image of an object being heated, which is an image including a state of the object to be heated that is being heated by the heating unit.
The learning device performs machine learning for estimating the cooking menu based on the image of the object being heated.
The computer
Further executing the step of acquiring the image of the object being heated to acquire the image of the object being heated from the image pickup apparatus,
In the cooking menu estimation step, the heating object image acquired in the heating object image acquisition step is also input to the learning device, and the calculation process of the learned learning device is executed to display the cooking menu. The cooking method according to claim 6 or 7, which is obtained from the learning device. The image pickup device is configured to be able to capture a heating user image which is an image including the state of the user who is heating the heating object by the heating unit.
The learning device performs machine learning for estimating the cooking menu based on the user image during heating.
The computer
Further executing the heating user image acquisition step of acquiring the heating user image from the image pickup apparatus,
In the cooking menu estimation step, the heating user image acquired in the heating user image acquisition step is also input to the learning device, and the arithmetic processing of the learned learning device is executed to obtain the cooking menu. The cooking method according to any one of claims 6 to 8, which is obtained from a learning device. The computer further performs a user information analysis step of analyzing user information, which is information about the user, based on the heating user image.
The learning device performs machine learning for estimating the cooking menu based on the user information.
In the cooking menu estimation step, the computer also inputs the user information analyzed in the user information analysis step into the learning device, and executes the arithmetic processing of the learned learning device to learn the cooking menu. The cooking method according to claim 9, which is obtained from a container. The heating target is associated with a cooking menu in which a user heats an object to be heated in a heating unit and cooks the object, and preparations are made before heating the object to be heated in the heating unit. A learning data acquisition unit that acquires training data including the preheating object image acquired from an image pickup device configured to be capable of capturing an image of the unheated object, which is an image including the state of an object, and a learning data acquisition unit.
A learning device including a learning processing unit that performs machine learning of a learning device so as to obtain an output including an output value corresponding to the cooking menu for an input including the image of the object before heating. The learning data is associated with the cooking menu, and further includes food material information which is information analyzed based on the image of the object to be heated and information about the food material contained in the object to be heated. Ori,
The learning device according to claim 11, wherein the input in the machine learning of the learning device also includes the food material information. The image pickup apparatus is configured to be capable of capturing an image of an object being heated, which is an image including a state of the object to be heated that is being heated by the heating unit.
The learning data further includes the image of the object being heated associated with the cooking menu.
The learning device according to claim 11 or 12, wherein the input in the machine learning of the learning device also includes the image of the object being heated. The image pickup device is configured to be able to capture a heating user image which is an image including the state of the user who is heating the heating object by the heating unit.
The learning data further includes the heating user image associated with the cooking menu.
The learning device according to any one of claims 11 to 13, wherein the input in the machine learning of the learning device also includes the heating user image. The learning data is associated with the cooking menu, and further includes user information that is information about the user as well as information that is analyzed based on the heating user image.
The learning device according to claim 14, wherein the input in the machine learning of the learning device also includes the user information. The computer
The heating target is associated with a cooking menu in which a user heats an object to be heated in a heating unit and cooks the object, and preparations are made before heating the object to be heated in the heating unit. A training data acquisition step for acquiring training data including the preheating object image acquired from an image pickup device configured to be capable of capturing an unheated object image which is an image including the state of an object, and a training data acquisition step.
A learning process step of performing machine learning of a learning device so as to obtain an output including an output value corresponding to the cooking menu for an input including the image of the object before heating, and a learning method to be executed. The learning data is associated with the cooking menu, and further includes food material information which is information analyzed based on the image of the object to be heated and information about the food material contained in the object to be heated. Ori,
The learning method according to claim 16, wherein the input in the machine learning of the learning device also includes the food material information. The image pickup apparatus is configured to be capable of capturing an image of an object being heated, which is an image including a state of the object to be heated that is being heated by the heating unit.
The learning data further includes the image of the object being heated associated with the cooking menu.
The learning method according to claim 16 or 17, wherein the input in the machine learning of the learning device also includes the image of the object being heated. The image pickup device is configured to be able to capture a heating user image which is an image including the state of the user who is heating the heating object by the heating unit.
The learning data further includes the heating user image associated with the cooking menu.
The learning method according to any one of claims 16 to 18, wherein the input in the machine learning of the learning device also includes the heating user image. The learning data is associated with the cooking menu, and further includes user information that is information about the user as well as information that is analyzed based on the heating user image.
19. The learning method according to claim 19, wherein the input in the machine learning of the learning device also includes the user information.

Images