JP2022160322A - Heating cooking system - Google Patents

Abstract

To provide a heating cooking system capable of achieving improvement in convenience.SOLUTION: A heating cooking system includes a housing, a heating part, an imaging part, a determination part, and a control part. The housing includes a placement part on which an object to be cooked can be placed. The heating part is provided in the housing, and heats the object to be cooked. The imaging part images the object to be cooked, and the color of the object to be cooked contained in the image to be imaged is different depending on the temperature of the heating part. The determination part converts the image imaged by the imaging part into a monochromatic image, and determines the cooking conditions of the object to be cooked based on the converted monochromatic image and a learnt model in which an internal variable is adjusted by learning. The control part controls the heating part based on the determination result of the determination part.SELECTED DRAWING: Figure 4

Description

Embodiments of the present invention relate to cooker systems.

2. Description of the Related Art A microwave oven has been proposed that includes a monitor camera that monitors an object to be cooked that is placed inside the oven, and that specifies cooking ingredients based on image information of the object to be cooked that is captured by the monitor camera. By the way, further improvement of convenience is expected for the heating cooker.

JP-A-6-257756

A problem to be solved by the present invention is to provide a heating cooker system capable of improving convenience.

A heating cooker system of an embodiment has a housing, a heating section, an imaging section, a determination section, and a control section. The housing includes a placing portion on which an object to be cooked is placed. The heating unit is provided in the housing and heats the object to be cooked. The imaging unit photographs the object to be cooked, and the color of the object to be cooked included in the photographed image varies depending on the temperature of the heating unit. The determination unit converts the image captured by the imaging unit into a monochrome image, and determines the cooking condition of the cooking target based on the converted monochrome image and a learned model in which internal variables are adjusted by learning. . The control section controls the heating section based on the determination result of the determination section.

BRIEF DESCRIPTION OF THE DRAWINGS The figure which shows the whole structure of the cooking-by-heating system of 1st Embodiment. BRIEF DESCRIPTION OF THE DRAWINGS The front view which shows the heating cooker of 1st Embodiment. Sectional drawing which follows the F3-F3 line of the heating cooker shown in FIG. The block diagram which shows the system configuration|structure of the heating cooker of 1st Embodiment. 4A and 4B are diagrams showing processing of an image processing unit according to the first embodiment; FIG. The figure which shows an example of the trained model of 1st Embodiment. FIG. 5 is a diagram showing an example of contents displayed by an evaluation reception unit according to the first embodiment; 4 is a flow chart showing the flow of processing by the control unit of the first embodiment; The block diagram which shows the system configuration|structure of the heating cooker of 2nd Embodiment. The block diagram which shows the system configuration|structure of the heating cooker of 3rd Embodiment. FIG. 11 is a diagram showing an example of dirt information on the lens of the camera according to the third embodiment; The block diagram which shows the system configuration|structure of the server of 4th Embodiment. The figure which shows an example of the screen which receives correction by the correction reception part of 4th Embodiment. 10 is a flow chart showing the flow of processing by the server of the fourth embodiment;

Hereinafter, the heating cooker system of the embodiment will be described with reference to the drawings. In the following description, the same reference numerals are given to components having the same or similar functions. Duplicate descriptions of these configurations may be omitted. "Based on XX" means "based at least on XX" and may include based on other elements in addition to XX. "Based on XX" is not limited to the case where XX is used directly, but can also include the case where XX is calculated or processed. "XX or YY" is not limited to either one of XX and YY, but may include both XX and YY. This is also the case when there are three or more selective elements. "XX" and "YY" are arbitrary elements (eg, arbitrary information).

(First embodiment)
<1. Overall configuration>
FIG. 1 is a diagram showing the overall configuration of a heating cooker system 1 of the first embodiment. The heating cooker system 1 includes, for example, a heating cooker 100 and a server 200 . The Kaetsu cooking appliance system 1 can communicate with the terminal device 300 of the user U via the network NW or directly. The network NW is, for example, the Internet, a cellular network, a Wi-Fi (registered trademark) network, a LPWA (Low Power Wide Area), a WAN (Wide Area Network), a LAN (Local Area Network), or other public lines or dedicated lines. etc., can be used depending on the situation.

The heating cooker 100 is a cooking appliance that heats and cooks an object to be cooked. Heating cooker 100 is arranged in user U's residence. The heating cooker 100 is, for example, a toaster, an oven toaster, or an oven range, but may be a microwave oven or other cooking appliances that do not have an oven function. An example in which the cooker 100 is a toaster oven will be described below.

Cooking device 100 can be connected to network NW via wireless router R installed in user U's residence, for example. Heating cooker 100 can communicate with server 200 via network NW. Heating cooker 100 may be able to communicate with user U's terminal device 300 directly or via network NW. However, in this embodiment, the cooking device 100 does not have to have a communication function and may not be able to communicate with the server 200 . That is, the server 200 does not have to exist. In this case, an example of a "heating cooker system" is constituted by the heat cooker 100 alone.

Server 200 is a server that manages the operation of cooker 100 . The server 200 is composed of one or more server devices (for example, cloud servers) connected to the network NW. The server 200 may include an information processing unit that performs edge computing and fog computing, such as an information processing unit included in a router in the network NW. The server 200 can communicate with the terminal device 300 of the user U via the network NW.

User U's terminal device 300 is, for example, a mobile terminal device such as a smart phone or a tablet terminal device. The terminal device 300 includes a display device 301 including a display screen capable of displaying various information, and an operation unit 302 capable of accepting user U's operation. The display device 301 is an example of a “display unit”. The operation unit 302 is, for example, a touch sensor provided over the display screen of the display device 301 . An application program for using the cooker system 1 is installed in the terminal device 300, and the functions described below are supported. Note that the terminal device 300 is not limited to a mobile terminal device, and may be a personal computer.

<2. Heating cooker>
<2.1 Overall configuration of heating cooker>
Next, the heating cooker 100 will be described.
FIG. 2 is a front view showing the heating cooker 100. FIG. FIG. 3 is a cross-sectional view of the cooking device 100 shown in FIG. 2 taken along line F3-F3. As shown in FIGS. 2 and 3 , the heating cooker 100 has, for example, a device body (heating cooker body) 101 , a communication section 102 , and a control unit 103 .

The device main body 101 has, for example, a housing 111, a mounting section 112, a heating section 113, a display device 114, an operation section 115, a camera 116, a temperature sensor 117, and a humidity sensor 118.

The housing 111 has, for example, a housing body 121 and a door 122 . The housing main body 121 is formed in a box shape with a part of the front surface opened. That is, the front surface of the housing body 121 has an opening 121a that exposes the inside of the housing body 121 to the outside. The door 122 is provided in front of the housing body 121 . The door 122 is rotatably connected to the housing body 121 and opens and closes the opening 121a.

The door 122 has a door body 131 , a window portion 132 and a handle 133 . The door main body 131 is formed in a plate shape that covers the opening 121 a of the housing main body 121 . The window portion 132 is provided through the door body 131 . The window portion 132 is made of a transparent glass material or plastic material. The user U can visually recognize the inside of the housing body 121 from the outside through the window portion 132 . The handle 133 is provided on the front surface of the door 122 and protrudes forward compared to the front surface of the window portion 132 . The handle 133 is a portion that the user U grips when opening and closing the door 122 with respect to the housing body 121 . The handle 133 has a handle main body 133 a provided apart from the door main body 131 and connecting portions 133 b provided at both ends of the handle main body 133 a and connected to the door main body 131 .

The mounting portion 112 is provided inside the housing 111 . The placement part 112 is formed, for example, in a plate shape or a mesh shape, and the object O to be cooked is placed thereon.

The heating unit 113 is provided inside the housing 111 and heats the cooking target O placed on the placement unit 112 . The heating unit 113 includes, for example, a first heater 113a and a second heater 113b. The first heater 113a is positioned above the object O to be cooked and heats the upper surface of the object O to be cooked. The second heater 113b is positioned below the cooking target O and heats the lower surface of the cooking target. Here, the heating part 113 emits different colors according to the heat generation state of the heating part 113 . For example, the heating unit 113 emits yellow or orange light when in the first temperature state, and emits more intense orange (deeper orange) light when in the second temperature state higher than the first temperature state. luminous. Therefore, the color of the surface of the cooking object O illuminated by the heating unit 113 varies depending on the state of the temperature of the heating unit 113 .

The display device 114 is provided on the front surface of the housing body 121, for example. The display device 114 has a display screen capable of displaying various information. The display device 114 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display, but is not limited to these. The display device 114 is an example of a “display unit”.

The operation unit 115 is provided on the surface of the housing body 121, for example. The operation unit 115 can accept user U's operation. The operation unit 115 is, for example, a touch sensor provided over the display screen of the display device 114 . However, the operation unit 115 may be physical buttons or the like provided on the surface of the housing body 121 . The user U can input the type of the cooking target O by operating the operation unit 115 . The “type of cooking object” is the type of food (for example, the type of bread such as bread, muffin, wheat bread, etc.), the number of objects to be cooked (1 slice of bread, 2 slices, 3 slices of bread, etc.), and the size of the cooking object. It includes one or more of width or thickness (6 slices of bread/8 slices of bread, etc.).

The camera 116 is provided in the housing 111 and photographs the object to be cooked O placed on the placing portion 112 . Camera 116 captures an image of the surface (for example, the upper surface) of object O to be cooked. In this application, the term "image" is not limited to still images, and includes moving images (video). The image captured by the camera 116 includes content indicating the degree of baking of the object O to be cooked. For example, the camera 116 captures a color image of the surface of the object O to be cooked. In this embodiment, the color of the cooking target O included in the image captured by the camera 116 varies depending on the temperature state of the heating unit 113 . That is, when the heating unit 113 is in the first temperature state, the surface of the cooking target O is illuminated yellow or orange by the heating unit 113 . On the other hand, when the heating unit 113 is in the second temperature state, the heating unit 113 illuminates the surface of the object O to be cooked with a stronger orange color. The camera 116 includes a camera body 116a including an imaging element, and a lens 116b located between the camera body 116a and the subject (cooking target O). The camera 116 is an example of the "imaging unit".

The camera 116 is provided, for example, on the ceiling surface 121b of the housing body 121 and positioned above the object O to be cooked. For example, the camera 116 is arranged at a place where the object O to be cooked can be photographed at an angle where the thickness of the object O to be cooked can be seen in addition to the size of the object to be cooked O (for example, the size when viewed from above). Note that the camera 116 may be provided on the inner surface of the door main body 131 instead of the ceiling surface 121 b of the housing main body 121 . Further, the camera 116 may be provided on the rear surface of the handle body 133 a of the door 122 outside the housing body 121 . In this case, the camera 116 takes an image of the surface of the cooking target O housed inside the housing body 121 through the window 132 of the door 122 .

A temperature sensor 117 is provided inside the housing 111 and detects the temperature inside the housing 111 .
A humidity sensor (water vapor sensor) 117 is provided inside the housing 111 and detects the humidity inside the housing 111 . The heating cooker 100 has, for example, a steam generator (not shown) that heats the water supplied to the water supply unit to generate steam and supply the moisture to the cooking target O that is being cooked. The humidity sensor 118 is provided, for example, in the heating cooker 100 having a steam generator, and detects the moisture supplied to the object O to be cooked.

In this embodiment, the camera 116, the temperature sensor 117, and the humidity sensor 118 constitute an example of the sensor unit SU. In the following description, the image captured by the camera 116, the detection result of the temperature sensor 117, and the detection result of the humidity sensor 118 may be collectively referred to as "detection result of the sensor unit SU". Note that the sensor unit SU may include sensors other than those described above. Also, one or both of temperature sensor 117 and humidity sensor 118 may be omitted.

<2.2 System configuration of heating cooker>
FIG. 4 is a block diagram showing the system configuration of the heating cooker 100. As shown in FIG.
The communication unit 102 is, for example, a wireless communication module including an antenna and a high frequency circuit. The communication unit 102 can communicate with the server 200 via the network NW. The communication unit 102 can communicate with the terminal device 300 via the network NW or directly using short-range wireless communication (for example, Bluetooth (registered trademark)).

Control unit 103 includes information processing unit PU<b>1 and storage unit 157 . Information processing unit PU<b>1 includes type accepting unit 151 , image processing unit 152 , cooking condition determining unit 153 , control unit 154 , evaluation accepting unit 155 , and learning unit 156 . Each functional unit included in information processing unit PU1 is implemented by a hardware processor such as a CPU (Central Processing Unit) mounted in control unit 103 executing a program. However, some or all of these functional units are implemented by hardware (including circuitry) such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array). Alternatively, it may be realized by cooperation of software and hardware. Storage unit 157 is realized by a RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), flash memory, or a combination of these. Storage unit 157 stores a plurality of trained models MA and threshold information TI.

The type accepting unit 151 accepts an input from the user U indicating the type of the cooking target O based on the user U's operation on the operating unit 115 of the heating cooker 100 or the operating unit 302 of the terminal device 300 . For example, the type receiving unit 151 causes the display device 114 of the heating cooker 100 or the display device 301 of the terminal device 300 to display a plurality of candidates for the type of the cooking target O, and the user who selects one from the plurality of candidates. Accepts U input. As described above, the 'type of object to be cooked' includes one or more of the type of food, the number of objects to be cooked, the size or thickness of the object to be cooked, and the like. In this embodiment, the type reception unit 151 constitutes an example of the input reception unit IU.

The image processing unit 152 performs image processing on the image captured by the camera 116 . For example, the image processing unit 152 performs image processing for converting a color image captured by the camera 116 into a monochrome image. A "monochrome image" as used in the present application may be a black-and-white image (an image in which one pixel is expressed in two gradations (one bit) of black and white) or a grayscale image (an image including intermediate gradation). In this embodiment, the image processing unit 152 converts a color image captured by the camera 116 into a black and white image. However, the image processing unit 152 may convert the color image captured by the camera 116 into a grayscale image.

FIG. 5 is a diagram showing the processing of the image processing unit 152. As shown in FIG. (a) and (b) in FIG. 5 show the case where the temperature state of the heating unit 113 is different. The “pre-conversion” image IM shown in (a) and (b) of FIG. As described above, the color of the cooking target O included in the two images IM differs according to the temperature state of the heating unit 113 .

On the other hand, the “converted” image IMA shown in (a) and (b) of FIG. 5 is an image after processing by the image processing unit 152, and is a monochrome image. The colors of the cooking target O included in the two images IM are substantially the same. That is, by converting the image IM into a monochrome image, the influence of the color of the cooking target O included in the image IM can be reduced.

The cooking condition determination unit 153 determines the cooking condition of the cooking target O based on the monochrome image converted by the image processing unit 152 and the learned model MA whose internal variables are adjusted by learning. The trained model MA is a model including a neural network, and is trained by deep learning, for example.

FIG. 6 is a diagram showing an example of a trained model MA. The trained model MA is composed of a neural network including an input layer, a hidden layer (hidden layer), and an output layer. For example, the trained model MA is a weighting coefficient w (internal variable) between nodes n included in the input layer, hidden layer, and output layer by deep learning using teacher data obtained from experiments conducted in advance. Learning is performed by adjusting the

The learned model MA can determine the cooking condition of the cooking target O when the detection result of the sensor unit SU (for example, the image of the camera 116, the detection result of the temperature sensor 117, the detection result of the humidity sensor 118) is input as input information. Learning is performed so that the determination result indicating is output as output information. The “judgment result indicating the degree of cooking” is a numerical value between 0 and 1, for example, where the state at the start of cooking is “0” and the state of overcooking (burnt state) is “1”. be done.

Here, the threshold information TI stored in the storage unit 157 includes a threshold (for example, "0.7") corresponding to the just-right cooking condition (for example, just-just-right grilling condition). In this embodiment, the cooking condition determination unit 153 inputs the latest detection result of the sensor unit SU as input information to the learned model MA at a predetermined cycle (for example, every 5 seconds). Then, when the numerical value as the determination result output from the learned model MA exceeds the threshold for a predetermined number of times (for example, when the threshold is exceeded five times in a row), the cooking condition determination unit 153 determines whether the cooking target O It is determined that the cooking condition is just right. In this embodiment, the image processing unit 152 and the cooking condition determination unit 153 constitute an example of the determination unit DU. Cooking condition determination unit 153 is an example of a “first determination unit”.

In this embodiment, the storage unit 157 stores a plurality of learned models MA corresponding to a plurality of types of the cooking target O. FIG. Cooking condition determination unit 153 selects one learned model MA from a plurality of learned models MA based on the type of cooking target O input by user U and received by type receiving unit 151 . Then, the cooking condition determination unit 153 determines the cooking condition of the cooking target O based on the selected learned model MA.

The control unit 154 controls the heating unit 113 based on the determination result of the cooking condition determination unit 153 . For example, the control unit 154 starts the operation of the heating cooker 100 and heats the heating unit 113 based on the user U's operation on the operating unit 115 of the heating cooker 100 or the operating unit 302 of the terminal device 300 . The control unit 154 controls the heating state of the heating unit 113 by, for example, ON/OFF-controlling a switching element that supplies electric power to the heating unit 113 . Then, when the cooking condition determination unit 153 determines that the cooking condition of the object to be cooked O is just right, the control unit 154 terminates the heating of the heating unit 113 and terminates the operation of the heating cooker 100 . Instead of/in addition to ending the operation of the heating cooker 100 , the control unit 154 may adjust the temperature of the heating unit 113 based on the determination result of the cooking condition determination unit 153 .

Evaluation reception unit 155 receives user U's evaluation of the cooking condition of cooking target O after the heating cooker 100 has finished operating. For example, the evaluation reception unit 155 displays a question regarding the cooking condition on the display device 114 of the heating cooker 100 or the display device 301 of the terminal device 300 in a questionnaire format, and receives the answer to the question as the user U's evaluation of the cooking condition.

FIG. 7 is a diagram showing an example of the content displayed by the evaluation reception unit 155. As shown in FIG. The evaluation reception unit 155 asks the user U, for example, a 5-level evaluation regarding the degree of baking or a 5-level evaluation regarding the moistness, and receives the selection of the 5-level evaluation for the question as the user U's evaluation.

The learning unit 156 performs additional learning of the trained model MA based on the user U's evaluation received by the evaluation receiving unit 155 . For example, the learning unit 156 uses, as new teacher data, the operation details (a combination of the detection result of the sensor unit SU and the determination content of the cooking condition determination unit 153) when the user U evaluates that the cooking condition is good. Perform reinforcement learning to maximize evaluation. As a result, the internal variables of the trained model MA are adjusted to match the preferences of the user U.

<3. Process Flow>
FIG. 8 is a flowchart showing the flow of processing by the control unit 103. As shown in FIG. First, the type accepting unit 151 accepts the type of the cooking target O based on the user U's operation on the operating unit 115 of the heating cooker 100 (or the operating unit 302 of the terminal device 300) (S101). Then, the cooking condition determination unit 153 selects a learned model MA to be used from among a plurality of learned models MA, based on the type of the cooking target O received by the type receiving unit 151 (S102).

Next, the controller 154 starts operating the heating cooker 100 (S103). That is, the control unit 154 starts cooking the object O to be cooked by causing the heating unit 113 to heat. Then, the determination unit DU acquires the detection result of the sensor unit SU at a predetermined cycle (S104).

When the detection result of the sensor unit SU is input, the image processing unit 152 converts the color image captured by the camera 116 included in the detection result of the sensor unit SU into a monochrome image (S105). Then, the cooking condition determination unit 153 determines the cooking condition of the cooking target O using the monochrome image converted by the image processing unit 152 and the selected learned model MA, and outputs the determination result ( S106).

Next, the control unit 154 determines whether or not the determination result output by the cooking condition determination unit 153 satisfies a predetermined termination condition (S107). For example, the control unit 154 determines that the predetermined termination condition is satisfied when the numerical value included in the determination results output by the cooking condition determining unit 153 exceeds a predetermined threshold for a predetermined number of times.

If the predetermined end condition is not satisfied (S107: NO), the control unit 154 returns to S104 and repeats the processes from S104 to S106. On the other hand, if the predetermined termination condition is satisfied (S107: YES), control unit 154 terminates the operation of cooker 100 (S108). That is, the control unit 154 terminates the heating of the heating unit 113 . Thereby, the cooking of the cooking object O is completed.

<4. Advantage>
As a comparative example, consider a heating cooker in which the degree of cooking is adjusted by a timer. With such a heating cooker, the user may need to check the cooking condition each time, or the cooking condition may vary each time. In addition, when judging the degree of cooking by directly using an image taken by a camera, if the color of the surface of the object to be cooked differs according to the state of the temperature of the heating unit, it is difficult to judge the degree of cooking with high accuracy. Sometimes.

On the other hand, the cooker system 1 of the present embodiment converts the image captured by the camera 116 into a monochrome image, and determines the cooking condition of the cooking target O based on the converted monochrome image and the learned model MA. and a control unit 154 that controls the heating unit 113 based on the determination result of the determination unit DU. According to such a configuration, first, the cooking condition is determined by the determination unit DU, and the heating unit 113 is controlled based on the determination result. , and the difference in cooking condition each time is reduced. Furthermore, when the color of the surface of the object O to be cooked differs according to the temperature state of the heating unit 113, the accuracy of determination of the cooking condition can be improved by determining the cooking condition based on the monochrome image. Thereby, the improvement of the user's U convenience can be aimed at.

In the present embodiment, the cooking-by-heating system 1 further includes a type accepting unit 151 that accepts a user's input indicating the type of the object O to be cooked. The determination unit DU selects one learned model MA from a plurality of learned models MA based on the type of the cooking target O received by the type receiving unit 151, and selects the cooking target O based on the selected learned model MA. Determine the cooking condition of According to such a configuration, it is possible to further improve the accuracy of judging the degree of cooking by using the learned model MA prepared for each object to be cooked. Thereby, the convenience of the user U can be improved.

In the present embodiment, the cooking-by-heating system 1 includes an evaluation accepting unit 155 that accepts the user U's evaluation of the cooking condition after the operation is completed, and a learned model MA added based on the user U's evaluation accepted by the evaluation accepting unit 155. A learning unit 156 that performs learning is further provided. According to such a configuration, it is possible to update the learned model MA so that the evaluation by the user U is higher. Thereby, the convenience of the user U can be improved.

(Modification)
In the first embodiment described above, the type reception unit 151, the image processing unit 152, the cooking condition determination unit 153, the control unit 154, the evaluation reception unit 155, the learning unit 156, and the storage unit 157 are provided in the heating cooker 100. ing. Alternatively, one or more of type reception unit 151, image processing unit 152, cooking condition determination unit 153, control unit 154, evaluation reception unit 155, learning unit 156, and storage unit 157 are provided in server 200. good too. Also, the evaluation reception unit 155 and the learning unit 156 may be omitted.

(Second embodiment)
Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that the cooking-by-heating system 1 has a preference reception unit 161 that receives user U's preference regarding the degree of cooking. Configurations other than those described below are the same as those of the first embodiment.

FIG. 9 is a block diagram showing the system configuration of the heating cooker 100A of the second embodiment. In this embodiment, the input reception unit IU includes a preference reception unit 161 . The preference receiving unit 161 receives the user U's preference regarding the degree of cooking of the object O to be cooked, based on the user U's operation on the operation unit 115 of the heating cooker 100 or the operation unit 302 of the terminal device 300 . For example, the preference receiving unit 161 causes the display device 114 of the heating cooker 100 or the display device 301 of the terminal device 300 to display a plurality of candidates for the cooking condition, and receives an input from the user U to select one from the plurality of candidates. accept. The preferences of the user U regarding the degree of cooking are, for example, “crunchy texture”, “moist texture”, “likes well-cooked food”, and “dislikes overcooked food”.

In this embodiment, the storage unit 157 stores a plurality of pieces of threshold information TI corresponding to a plurality of cooking preferences. A plurality of pieces of threshold information TI have different thresholds (thresholds for terminating the operation of the heating cooker 100A) for determining that the cooking condition is just right. The cooking condition determination unit 153 selects one threshold information TI from a plurality of threshold information TI according to the preference of the user U received by the preference reception unit 161, and uses the selected threshold information TI to operate the heating cooker. The end timing of the operation of 100A is determined. In other words, the cooking condition determination unit 153 varies the determination result regarding the cooking condition of the object to be cooked (for example, the determination result indicating the operation end timing) based on the preference of the user U received by the preference receiving unit 161 .

According to such a configuration, the object O to be cooked can be cooked in a cooking condition that is more suitable for the preference of the user U. Thereby, the convenience of the user U can be improved. For example, according to the configuration of the present embodiment, even if the evaluation reception unit 155 and the learning unit 156 are omitted, the cooking target O can be cooked in a cooking condition that is more suitable for the user U's preference.

(Third embodiment)
Next, a third embodiment will be described. The third embodiment differs from the first embodiment in that the amount of dirt on the lens 116b of the camera 116 is also reflected in determining the degree of cooking. Configurations other than those described below are the same as those of the first embodiment.

FIG. 10 is a block diagram showing the system configuration of the heating cooker 100B of the third embodiment. Here, the heating cooker 100B is a home appliance that is cleaned less frequently than other home appliances such as a refrigerator, and the lens 116b of the camera 116 may accumulate dirt (for example, oil stains). When the dirt adhering to the lens 116b accumulates, the image captured by the camera 116 may change color (for example, turn brown). If the image captured by the camera 116 is discolored, there is a possibility that the cooking condition determination accuracy will be reduced.

In this embodiment, the information processing unit PU<b>1 includes a state management unit 171 . The state management unit 171 manages information (hereinafter referred to as "dirt information") that directly or indirectly indicates the amount of dirt adhering to the lens 116b of the camera 116. FIG. For example, the state management unit 171 collects dirt information on the lens 116 b of the camera 116 at a predetermined period (for example, once a week) and stores it as part of history information HI stored in the storage unit 157 .

FIG. 11 is a diagram showing an example of dirt information on the lens 116b of the camera 116. As shown in FIG. Dirt information on the lens 116b of the camera 116 includes, for example, dirt information HIA that directly indicates the amount of dirt adhered to the lens 116b of the camera 116, and dirt information HIA that indirectly indicates the amount of dirt adhered to the lens 116b of the camera 116. and/or information HIB. Dirt information HIA includes, for example, a history of images captured by camera 116 . For example, the contamination information HIA includes an image 1 taken at the beginning of use of the cooking device 100B and an image N taken most recently. Dirt information HIB includes the usage history of cooker 100B. The usage history includes, for example, information on one or more of the type of food cooked, the heating temperature of the heating unit 113, the heating time of the heating unit 113, and the like.

In this embodiment, the learned model MA outputs a determination result indicating the cooking condition of the cooking target O when the detection result of the sensor unit SU and the dirt information of the lens 116b of the camera 116 are input as input information. is learned by Dirt information on the lens 116b of the camera 116, which is input as input information, may be, for example, the most recently captured image N (for example, color image N), a set of image 1 and image N, or image 1 and image N. may be information indicating the difference between . Instead of/in addition to this, the contamination information of the lens 116b of the camera 116 may be the history of cooked ingredients, the integrated value of the heating temperature of the heating unit 113, or the cumulative value of the heating time of the heating unit 113, or the like. In this embodiment, the cooking condition determination unit 153 determines the cooking condition based on the detection result of the sensor unit SU and dirt information on the lens 116b of the camera 116. FIG.

According to such a configuration, it is possible to determine the degree of cooking of the object O to be cooked, taking into consideration the dirt adhering to the lens 116b of the camera 116 as well. As a result, it is possible to improve the accuracy of determination regarding the degree of cooking.

(Modification)
If the camera 116 is provided on the handle 133 of the door 122 instead of inside the housing 111, the dirt becomes a problem not on the lens 116b of the camera 116 but on the window 132 of the door 122. Therefore, when the camera 116 is provided on the handle 133 of the door 122 , the state management unit 171 manages dirt information that directly or indirectly indicates the amount of dirt adhering to the window 132 . As for the details of the dirt information in this modified example, the "camera lens" in the dirt information described in the third embodiment should be replaced with the "window". For example, the state management unit 171 of this modification manages information (dirt information HIA, HIB) that directly or indirectly indicates the amount of dirt adhering to the window 132 .

Dirt information HIA includes, for example, a history of images of window 132 taken by camera 116 or another camera. For example, the contamination information HIA includes an image 1 of the window portion 132 taken at the initial stage of use of the cooker 100B and an image N of the window portion 132 taken most recently. Dirt information HIB includes the usage history of cooker 100B. The usage history includes, for example, information on one or more of the type of food cooked, the heating temperature of the heating unit 113, the heating time of the heating unit 113, and the like. With such a configuration, it is also possible to improve the accuracy of determination regarding the cooking condition.

(Fourth embodiment)
Next, a fourth embodiment will be described. The fourth embodiment differs from the first embodiment in that the type of cooking target O is determined by the heating cooker system 1 . Configurations other than those described below are the same as those of the first embodiment. In this embodiment, the above-described trained model MA is referred to as "first trained model MA" for distinction.

FIG. 12 is a block diagram showing the system configuration of the server 200C of the fourth embodiment. The server 200C has an information processing unit PU2 and a storage unit 206, for example. Information processing unit PU2 includes, for example, an information acquisition unit 201, a type determination unit 202, an information output unit 203, a correction reception unit 204, and a learning unit 205. Each functional unit included in the information processing unit PU2 is implemented by executing a program by a hardware processor such as a CPU installed in the server 200C. However, some or all of these functional units may be realized by hardware (including circuitry) such as ASIC, PLD, or FPGA, or by cooperation between software and hardware. good too. Storage unit 206 is implemented by RAM, ROM, HDD, flash memory, or a combination of these. The storage unit 206 stores the second trained model MB.

The information acquisition unit 201 acquires from the cooking device 100 a color image captured by the camera 116 of the cooking device 100 (or a monochrome image converted by the image processing unit 152 ). Hereinafter, the color image captured by the camera 116 and the monochrome image converted by the image processing unit 152 are collectively referred to as "image acquired by the information acquisition unit".

The type determination unit 202 determines the type of the cooking target O based on the image (for example, color image) acquired by the information acquisition unit 201 and the second learned model MB. The second trained model MB is a model including a neural network, and is trained by deep learning, for example. The type determination unit 202 is an example of a “second determination unit”. In the present embodiment, an example of determination unit DU is configured by image processing unit 152 and cooking condition determination unit 153 provided in heating cooker 100, and type determination unit 202 provided in server 200C.

For example, the second trained model MB, like the first trained model MA shown in FIG. 6, is composed of a neural network including an input layer, a hidden layer (middle layer), and an output layer. For example, the trained model MB is a weighting factor w (internal variable) between nodes n included in the input layer, hidden layer, and output layer by deep learning using teacher data obtained from experiments conducted in advance. Learning is performed by adjusting the

The second trained model MB is trained so that when the image acquired by the information acquisition unit 201 is input as input information, the determination result indicating the type of the cooking target O is output as output information. . As described above, the type of cooking target O includes one or more of the type of ingredients, the number of cooking targets, the size or thickness of the cooking targets, and the like. The output information output from the second trained model MB includes information indicating the likelihood of the determination result of the type of cooking target O (for example, including information indicating the probability that the determination result is correct).

The information output unit 203 causes the cooking device 100 or the terminal device 300 to output information indicating the type of the cooking target O determined by the type determining unit 202, and displays the information indicating the type of the cooking target O on the display of the heating cooker 100. Displayed on the device 114 or the display device 301 of the terminal device 300 .

The correction accepting unit 204 accepts a correction to the determination result of the type of the cooking target O determined by the type determining unit 202 and output by the information output unit 203 . For example, the correction accepting unit 204 accepts a correction to the determination result of the type of the cooking target O based on the user U's operation on the operating unit 115 of the heating cooker 100 or the operating unit 302 of the terminal device 300 .

FIG. 13 is a diagram showing an example of a screen for accepting correction by the correction accepting unit 204. As shown in FIG. In this embodiment, the determination result of the type of the cooking target O by the type determination unit 202 is displayed on the display device 114 of the heating cooker 100 or the display device 301 of the terminal device 300 . Then, the correction accepting unit 204 first accepts whether or not to correct the determination result of the type of the object O to be cooked. Then, if there is a correction, the correction receiving unit 204 receives specific content of the correction.

The learning unit 205 performs additional learning of the second trained model MB based on the correction content of the user U received by the correction receiving unit 204 . For example, the learning unit 205 performs additional learning of the second trained model MB using a combination of the image acquired by the information acquisition unit 201 and the correct type of cooking target O corrected by the user U as new teacher data. . Thereby, the improvement of the determination precision of the kind determination part 202 can be aimed at.

In the present embodiment, the control unit 154 of the heating cooker 100 may adjust the temperature of the heating unit 113 based on the determination result of the type determining unit 202 instead of/in addition to ending the operation of the heating cooker 100. good.

FIG. 14 is a flow chart showing the flow of processing by the server 200C. First, the type determination unit 202 determines the type of the cooking target O based on the image acquired by the information acquisition unit 201 and the second learned model MB (S201). Next, the type determination unit 202 determines whether the determination result is correct (whether the probability that the determination result is correct is equal to or greater than a threshold) (S202).

The information output unit 203 outputs the determination result of the type of the cooking target O to the heating cooker 100 when the determination result is correct (when the determination result is correct probability is equal to or greater than the threshold, S202: YES). In this case, the heating cooker 100 automatically selects (automatically sets) the operating conditions of the heating cooker 100 (such as the heating temperature and heating time of the heating unit 113) based on the determination result of the type of the cooking target O, Operation of the heating cooker 100 is started (S206). Note that the information output unit 203 displays the determination result indicating the type of the cooking target O even when the determination result is not wrong (when the probability that the determination result is correct is equal to or greater than the threshold value). 114 or the display device 301 of the terminal device 300 .

On the other hand, when there is a possibility that the determination result is wrong (if the probability that the determination result is correct is less than the threshold, S202: NO), the information output unit 203 outputs the determination result of the type of cooking target O to the heating cooker. 100 or the display device 301 of the terminal device 300 (S203). Next, the correction receiving unit 204 receives a correction to the determination result of the type of cooking target O (S204).

The information output unit 203 outputs the determination result of the type of the cooking target O to the heating cooker 100 when receiving the user U's input indicating that there is no correction (S204: NO). Then, the process proceeds to S206. On the other hand, when the correction receiving unit 204 receives an input from the user U to correct the type of the cooking target O (S204: YES), the correction receiving unit 204 corrects the type of the cooking target O (S205). Then, the information output unit 203 outputs the corrected type of the cooking target O to the heating cooker 100 . Then, the process proceeds to S206. Subsequent processing is the same as the processing of the first embodiment.

According to such a configuration, convenience for the user U can be improved, as in the first embodiment. Furthermore, in the present embodiment, the type of cooking target O is automatically determined by the type determination unit 202 . Thereby, the further improvement of the user's U convenience can be aimed at.

In this embodiment, the information output unit 203 that outputs the type of the cooking target O determined by the type determination unit 202 to the display devices 114 and 301 and the input of the user U correcting the type of the cooking target O can be received. A correction receiving unit 204 is further provided. According to such a configuration, it is possible to cook based on the correct content corrected by the user U when the determination of the type of the cooking target O is erroneous. Thereby, the further improvement of the user's U convenience can be aimed at.

In this embodiment, the cooking condition determination unit 153 is provided in the cooking device 100 . The type determination unit 202 is provided in the server 200C. As a result, the cooker 100 can determine the cooking condition that requires quick determination, and the server 200C can determine the type of the cooking target O that has more time than the cooking condition. As a result, the required capacity of the storage unit 157 of the heating cooker 100 can be reduced, and the stability of processing and the cost reduction of the heating cooker 100 can be achieved.

In this embodiment, the cooking condition determination unit 153 determines the cooking condition based on the monochrome image. The type determination unit 202 determines the type of the cooking target O based on the color image. According to such a configuration, when the color changes depending on the temperature of the heating unit 113, the influence of the color difference of the cooking object O illuminated by the heating unit 113 is reduced, and the type of the cooking object O is determined. can make highly accurate judgments based on color images.

(Modification)
In the fourth embodiment described above, the information acquisition unit 201, the type determination unit 202, the information output unit 203, the correction reception unit 204, the learning unit 205, and the storage unit 206 are provided in the server 200C. Alternatively, one or more of the information acquisition unit 201, the type determination unit 202, the information output unit 203, the correction reception unit 204, the learning unit 205, and the storage unit 206 may be provided in the heating cooker 100. .

Several embodiments and modifications have been described above. However, the embodiments and modifications are not limited to the above examples. For example, the embodiments and modifications described above may be implemented in combination with each other. The “server” referred to in this application is not limited to a cloud server, and may be a computer provided at the user U's home.

According to at least one embodiment described above, the heating cooker system converts an image captured by the imaging unit into a monochrome image, converts the monochrome image, and the learned model whose internal variables are adjusted by learning. Based on, it has the determination part which determines the cooking condition of cooking object. According to such a configuration, it is possible to improve convenience.

While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, as well as the scope of the invention described in the claims and equivalents thereof.

DESCRIPTION OF SYMBOLS 1... Cooking device system 100, 100A, 100B... Cooking device, 111... Housing, 112... Placement part, 113... Heating part, 114... Display device (display part), 116... Camera (imaging part), 151...Type accepting unit 152...Image processing unit 153...Cooking condition determining unit (first determining unit) 154...Control unit 155...Evaluation accepting unit 156...Learning unit 161...Preference accepting unit 200, 200C Server 202 Type determination unit (second determination unit) 204 Correction reception unit 205 Learning unit 300 Terminal device 301 Display device (display unit) DU Determination unit MA First learning Already model, MB: Second learned model, O: Cooking target.

Claims (14)

a housing including a placing portion on which the object to be cooked is placed;
a heating unit provided in the housing for heating the object to be cooked;
an image capturing unit that captures an image of the object to be cooked, and that includes different colors of the object to be cooked included in the image captured by the temperature of the heating unit;
A judgment unit that converts an image captured by the imaging unit into a monochrome image, and judges the cooking condition of the cooking target based on the converted monochrome image and a learned model whose internal variables have been adjusted by learning;
a control unit that controls the heating unit based on the determination result of the determination unit;
Heating cooker system with Further comprising a type reception unit that receives user input indicating the type of the cooking target,
The learned model includes a plurality of learned models corresponding to the type of cooking target,
The determination unit selects one learned model from the plurality of learned models based on the type of the cooking target received by the type receiving unit, and cooks the cooking target based on the selected learned model. determine the condition
The heating cooker system according to claim 1. The trained model includes a first trained model used to determine the cooking condition and a second trained model used to determine the type of the cooking target,
The determination unit includes a first determination unit that determines the cooking condition based on the monochrome image and the first trained model, an image captured by the imaging unit or the monochrome image, and the second trained model. and a second determination unit that determines the type of the cooking target based on
The heating cooker system according to claim 1. an information output unit that outputs the type of the cooking target determined by the second determination unit to a display unit;
a correction receiving unit capable of receiving user input to correct the type of the cooking target when the type of the cooking target displayed on the display unit is incorrect;
further comprising
The heating cooker system according to claim 3. a heating cooker including the housing, the heating unit, the imaging unit, and the control unit;
a server connectable to the cooking device via a network;
with
The first determination unit is provided in the heating cooker,
The second determination unit is provided in the server,
The heating cooker system according to claim 3 or 4. The first determination unit determines the cooking condition based on the monochrome image and the first learned model,
The second determination unit determines the type of the cooking target based on the color image captured by the imaging unit and the second learned model.
The heating cooker system according to any one of claims 3 to 5. The image captured by the imaging unit includes content indicating the degree of grilling of the cooking target,
The heating cooker system according to any one of claims 1 to 6. an evaluation reception unit that receives a user's evaluation of the cooking condition after the operation is finished;
a learning unit that performs additional learning of the trained model based on the user's evaluation received by the evaluation receiving unit;
further comprising
The heating cooker system according to any one of claims 1 to 7. Further comprising a preference reception unit that receives user's preference regarding the cooking condition,
The determination unit varies the determination result regarding the cooking condition of the cooking target based on the user's preference received by the preference reception unit.
The heating cooker system according to any one of claims 1 to 8. A heating cooker including the housing, the heating unit, the imaging unit, and the control unit,
The cooking device is a toaster or a toaster oven,
The heating cooker system according to any one of claims 1 to 9. The housing has a housing body having an opening and a door that closes the opening,
The imaging unit is provided on the ceiling of the housing body or on the inner surface of the door,
The heating cooker system according to claim 10. The determination unit determines the cooking condition based on information directly or indirectly indicating the amount of dirt attached to the lens of the imaging unit, the monochrome image, and the learned model.
The heating cooker system according to claim 11. The housing has a housing body having an opening, and a door that closes the opening and includes a window and a handle that allows the inside of the housing body to be viewed from the outside,
The image capturing unit is provided in the handle and captures an image of the object to be cooked that is housed inside the housing body through the window.
The heating cooker system according to claim 10. The determination unit determines the cooking condition based on information that directly or indirectly indicates the amount of dirt attached to the window, the monochrome image, and the learned model.
The heating cooker system according to claim 13.

Images