CN115398153A

CN115398153A - Heating state recognition device, heating control method, heating state recognition system, and heating control system

Info

Publication number: CN115398153A
Application number: CN202180028426.6A
Authority: CN
Inventors: 三浦伸之
Original assignee: Zensho Holdings Co Ltd
Current assignee: Zensho Holdings Co Ltd
Priority date: 2020-04-14
Filing date: 2021-01-15
Publication date: 2022-11-25
Also published as: JP6894032B1; JP2021169875A; JP2021169920A; WO2021210230A1

Abstract

The invention provides a heating state recognition device, a heating control method, a heating state recognition system and a heating control system. A heating state recognition device according to an embodiment includes: an acquisition unit that acquires a plurality of time-series images obtained by imaging the inside of a container containing a heated object and a liquid in time series; and a recognition unit that recognizes a plurality of heating states of the liquid using time-series image feature quantities acquired based on the plurality of time-series images.

Description

Heating state recognition device, heating control method, heating state recognition system, and heating control system

Technical Field

Embodiments of the present invention relate to a heating state recognition device, a heating control method, a heating state recognition system, and a heating control system.

Background

In a restaurant, a store staff has conventionally made an effort to determine the amount of heat and provide a customer with a cooked product that is cooked with a more delicious taste. For example, there is a cooking method in which the temperature of the pan is adjusted so as not to boil. However, the heating adjustment of a heating device that heats a pot or the like may be influenced by experience of a worker (see japanese patent laid-open publication No. 2017-133722).

Disclosure of Invention

Accordingly, an object of the present invention is to provide a heating state recognition device, a heating control method, a heating state recognition system, and a heating control system that enable cooking with more appropriate heating control without being affected by experience of a cook.

A heating state recognition device according to an embodiment includes:

an acquisition unit that acquires a plurality of time-series images obtained by imaging the inside of a container containing an object to be heated and a liquid in time series; and

and a recognition unit that recognizes a plurality of in-container states of the liquid using the time-series feature quantities acquired based on the plurality of time-series images.

The image processing apparatus may further include a feature amount generation unit having a convolutional neural network that generates an image feature amount for each of a plurality of time-series images and arranges the image feature amounts in time series to generate time-series image feature amounts,

the identification unit has a recurrent neural network that identifies a plurality of heating states based on time-series image feature quantities generated by the recurrent neural network.

The heating apparatus may further comprise a learning unit for performing machine learning by using a learning set including an accurate label corresponding to the states of the object and the liquid and time-series image feature values generated by the convolutional neural network,

the recognition unit recognizes the state based on the result of the machine learning.

The heating control device may be a heating power control unit that controls the heating power of the heating device for heating the container based on the recognition result of the recognition unit of the heating state recognition device.

The heating power control unit may perform control for changing the time for maintaining the predetermined state in accordance with the predetermined state before the liquid boils.

When the recognition unit recognizes the state in which the object to be heated is input, independently of the liquid state of the liquid, the heating power control unit may cause the display device to display a display image urging to discard the object to be heated in the container after a predetermined time has elapsed from the time when the state in which the object to be heated is recognized.

A heating state recognition method according to an embodiment includes:

an acquisition step of acquiring a plurality of time-series images obtained by imaging and shooting the inside of a container containing an object to be heated and a liquid in time series; and

and a recognition step of recognizing a plurality of heating states of the liquid using time-series image features obtained by acquiring the time-series image features based on the plurality of time-series images.

The heating apparatus may further include a heating power control step of controlling the heating power of the heating device for heating the container based on the recognition result of the recognition step of the heating state recognition method.

A heating state recognition system according to an embodiment includes:

an imaging device that images the inside of a container from the upper part of the container having an object to be heated and a liquid;

an acquisition unit that acquires a plurality of images captured in time series by an imaging device; and

and a recognition unit that recognizes a plurality of heating states of the liquid using time-series image feature quantities acquired based on the plurality of time-series images.

The heating control system may further include a heating power control unit that controls heating power of the heating device for heating the container based on the recognition result of the recognition unit of the heating state recognition system.

According to the present invention, it is possible to provide a heating control device, a heating control method, and a heating control system that can perform cooking with more appropriate heating control without being affected by experience of a cook.

Drawings

Fig. 1 is a block diagram showing the configuration of a heating control system according to the present embodiment.

Fig. 2 is a diagram showing an example of the position of an imaging device for imaging a cooking container.

Fig. 3 is a diagram showing a configuration example of the feature amount generation unit.

Fig. 4 is a table for explaining the states of the object to be heated and the liquid.

Fig. 5 is a flowchart showing an example of the learning process of the learning unit.

Fig. 6 is a flowchart showing a control example of the heating power control unit.

Fig. 7 is a table showing an example of a correct tag corresponding to a human operation.

Fig. 8 is a diagram showing an example of a state of changing to the third state after changing to the fourth state occurs multiple times.

Fig. 9 is a block diagram showing a configuration of a cooking instruction device according to a second embodiment.

Fig. 10 is a flowchart showing a control example of the heating power control unit according to the second embodiment.

Detailed Description

Hereinafter, a heating state recognition device, a heating control method, a heating state recognition system, and a heating control system according to embodiments of the present invention will be described in detail with reference to the drawings. The embodiments described below are examples of the embodiments of the present invention, and the present invention is not to be construed as being limited to these embodiments. In the drawings referred to in the present embodiment, the same or similar reference numerals are given to the same or similar parts and their overlapping description will be omitted. For convenience of explanation, the dimensional ratio of the drawings may be different from the actual ratio, or a part of the structure may be omitted from the drawings.

(first embodiment)

Fig. 1 is a block diagram showing the configuration of a heating control system 1 according to a first embodiment. The heating control system 1 of fig. 1 is a system capable of controlling the heating power of the heating device. As shown in fig. 1, the heating control system 1 of the present embodiment includes: heating device 10, cooking container 20, imaging device 30, display device 40, operation unit 50, and heating control device 60.

Fig. 2 is a diagram showing an example of the position of the imaging device 30 that images the inside of the cooking container 20 placed on the heating device 10 from above. As shown in fig. 2, the heating device 10 is, for example, a gas stove, and includes a burner as a heat generating portion on an upper surface portion of a casing thereof. Further, the heating power of the burner (adjustment of the amount of heat generation) can be adjusted by the rotational operation of the motor of the burner operation button. The heating device 10 according to the present embodiment is a gas furnace, but is not limited thereto. For example, an electric furnace is also possible.

The cooking container 20 is, for example, a pot, and stores cooking water in which food as an object to be heated, water, and seasonings are mixed.

The imaging device 30 is, for example, a video camera, and can capture images representing the state of the inside of the cooking container 20 in time series from the upper part of the cooking container 20. The imaging device 30 outputs 3 kinds of digital image data, for example, a red image (R image), a green image (G image), and a blue image (B image), to the heating control device 60. The 3 digital image data are constituted by pixels of 1080 × 1920 rows, for example.

As shown again in fig. 1, the display device 40 is, for example, a monitor. The display device 40 displays the image generated by the heating control device 60.

The operation unit 50 is constituted by, for example, a keyboard, a pointing device, and the like. The touch panel can be incorporated with the display device 40. The operation unit 50 outputs an instruction signal corresponding to an operation by a user using the heating control device 60.

The heating control device 60 is a device capable of controlling the heating power of the heating device 10, and the heating control device 60 includes: the image processing apparatus includes an acquisition unit 600, a storage unit 602, a feature amount generation unit 603, a learning unit 604, a recognition unit 607, a heating power control unit 608, and an image generation unit 610. Such a heating control device 60 is implemented by a personal computer, for example. That is, the heating control device 60 is configured to include, for example, a CPU (Central Processing Unit). The heating state recognition device 61 according to the present embodiment includes an acquisition unit 600, a feature amount generation unit 603, a learning unit 604, and a recognition unit 607, and the heating state recognition system 62 according to the present embodiment includes the imaging device 30 and the heating state recognition device 61. The heating state recognition device 61 according to the present embodiment includes the learning unit 604, but is not limited thereto. For example, the heating state recognition device 61 may not include the learning unit 604.

The acquisition unit 600 may acquire time-series images captured by the imaging device 30 and store the images in the storage unit 602. The acquisition unit 600 can also be connected to a network, for example, and acquire an image from a server or the like via the network.

The storage unit 602 is configured by, for example, an HDD (hard disk drive), an SSD (solid state drive), or the like. The storage unit 602 can also store the images acquired by the acquisition unit 600 for learning. The storage unit 602 stores various programs for executing heating control. Thus, the heating control device 60 configures each processing unit by executing a program stored in the storage unit 602, for example. The storage unit 602 stores a machine learning model for recognition. In the present embodiment, the machine learning model that has been learned is sometimes referred to as a learned model.

The feature amount generator 603 generates image feature amounts in time series from the time-series images acquired by the acquisition unit 600, for example. The learning data is, for example, data obtained by associating an accurate label corresponding to the state of the object to be heated and the liquid in the cooking container 20 with the time-series image feature amount generated by the feature amount generator 603. Details of the feature amount generating section 603 will be described later.

The learning unit 604 performs machine learning, for example, supervised learning, using the learning data stored in the storage unit 602. The learning unit 604 performs machine learning, for example, by a neural network. The learning unit 604 may perform the arithmetic processing in a computer having higher calculation capability for learning, for example, other than the heating control device 60. The learned model (learned model) is stored in the storage unit 602 via a network and used by the recognition unit 607, for example. In the present embodiment, the learning unit 604 performs arithmetic processing in a computer other than the heating control device 60, but the present invention is not limited to this. For example, the learning unit 604 may be provided in the heating control device 60. In this case, data obtained on site can be reflected in learning in real time.

The machine learning model for the learning unit 604 to perform machine learning is not limited to the neural network. For example, the recognition function or the like may be learned as a machine learning model by a general clustering method such as a K-means method. Further, details of the learning section 604 are described later.

The recognition unit 607 recognizes the predetermined state of the liquid in the cooking container 20 using the machine learning result in the learning unit 604. The recognition unit 607 according to the present embodiment uses the neural network learned by the learning unit 604. The neural network can recognize a plurality of heating states including a predetermined state of the liquid in the cooking container 20 before boiling by using the time-series feature amount generated by the feature amount generating unit 603 based on the plurality of time-series images captured by the imaging device 30. Details of the identification portion 607 are described later.

The heating power control unit 608 controls the heating power of the heating apparatus 10 based on the recognition result of the recognition unit 607. For example, the heating power control unit 608 controls the heating power of the heating device 10 by controlling the rotation of the motor of the heating device 10. The details of the fire power control section 608 are also described later.

The image generating unit 610 generates an image indicating the state of the inside of the cooking container 20 based on the recognition result of the recognition unit 607, and outputs the image to the display device 40. Further, the image generation unit 610 may generate an image indicating an operation instruction based on the recognition result of the recognition unit 607.

Here, details of the feature amount generation unit 603 and the learning unit 604 will be described with reference to fig. 3.

Fig. 3 is a diagram showing a configuration example of the feature amount generation unit 603. As shown in fig. 3, the feature amount generator 603 includes a plurality of image feature amount extractors 603a. The image feature extraction unit 603a is, for example, a Convolutional Neural Network (CNN). In the present embodiment, the plurality of image feature extracting units 603a are used, but the present invention is not limited to this. For example, the images t1 to tn may be sequentially input to one image feature extraction unit 603a, and the image features t1 to tn may be sequentially generated.

In this CNN, neurons of each layer are classified into filters, and each filter detects a different repetitive feature in input data. The multi-layer neural network architecture in a convolutional neural network, for example, has a plurality of convolutional layers, a plurality of sampling layers, and a plurality of fully connected layers. The CNN is a learned CNN (for example, imageNet pre-trained network) that has been learned using a conventional labeled image. That is, the feature extraction unit 603a generates a value of the intermediate layer output as a feature by fixing the convolutional neural network and feeding forward the input image.

The images t1 to tn captured in time series are input to each of the plurality of feature value extracting units 603a. For example, 3 digital images, i.e., a red image (R image), a green image (G image), and a blue image (B image), are reduced to 224 × 224 pixels for 1 image. In this embodiment, the three primary colors R, G, B are held by respective 8-bit (256-gray) values. Thus, the pixel data of 224 × 224 × 3 is input to the first convolution layer of the CNN. Then, time-series image feature quantities t1 to tn are generated. For example, n is 60, and 60 time-series images captured in one second are input to the learning unit 604 or the recognition unit 607. The image feature generated by the feature extraction unit 603a is, for example, 1024 dimensions. In this way, the CNN generates the image feature quantities t1 to tn for each of the images t1 to tn. The feature value generation unit 603 outputs the image feature values t1 to tn arranged in time series.

The data for learning will be described in detail with reference to fig. 4.

Fig. 4 is a table for explaining the states of the object to be heated and the liquid. The states of the object to be heated and the liquid in the cooking container 20 change in time series with the elapse of the heating time. That is, as the heating of the cooking container 20 progresses, for example, the first state changes to the fourth state in time series. For example, the first state is "a state of not boiling, not performing any operation". The second state is a "state in which the liquid surface is stable and the bubbles rise all over". The third state is a "state in which the liquid surface is surged, but no bubble appears or is inconspicuous". The fourth state is the "boiling state". For example, the first to third states represent states before boiling. In the data for learning, a label indicating the first state is assigned as a correct label to the time-series image obtained by the acquisition unit 600 or the time-series image feature obtained by the feature amount generation unit 603 in the first state, a label indicating the second state is assigned as a correct label to the time-series image obtained by the acquisition unit 600 or the time-series image feature obtained by the feature amount generation unit 603 in the second state, a label indicating the third state is assigned as a correct label to the time-series image obtained by the acquisition unit 600 or the time-series image feature obtained by the feature amount generation unit 603 in the third state, and a label indicating the fourth state is assigned as a correct label to the time-series image obtained by the acquisition unit 600 or the time-series image feature obtained by the feature amount generation unit 603 in the fourth state. These labels can be imparted manually, for example. These labeled time-series images or time-series image feature values are stored as learning data in the storage unit 602.

The learning unit 604 will be described with reference to fig. 3 again. The learning unit 604 learns a neural network, which is an example of a machine learning model, using the learning data. The neural network learned by the learning unit 604 learns the time-series image feature amount with a label, for example, and outputs the label (the first state to the fourth state) for the input time-series image feature amount. For example, the neural network is a multi-level connection of layers having a plurality of nodes (LSTM: long Short-term Memory). LSTM is one way of learning a recurrent neural network (recurrent neural network) of time series data.

For example, in this neural network, a connection coefficient between nodes is initialized, and image feature quantities of a labeled time series are sequentially input. The neural network is generated by processing such as back propagation (error back propagation) for correcting the connection coefficient so that an error between an output of the neural network and a label given to the input time-series image feature amount is reduced. For example, a learned neural network is generated by performing processing such as back propagation so as to minimize a predetermined loss function. By repeating the above-described processing, the neural network can reproduce and output the label for the input feature quantity more favorably. The neural network learning method is not limited to the above-described method, and any known technique can be applied.

The recognition unit 607 has a neural network on which the learning described above is performed. The neural network can recognize a state before boiling, which has been difficult to distinguish in the past, as a plurality of states. For example, if the state is the first state, that is, the "non-boiling, non-operating state", the time-series image feature amount does not change, and the neural network can be identified as being in the first state.

For example, if the liquid level is in the second state, that is, the "state where the liquid level is stable and the bubble rises up, the time-series fluctuation of the value of the pixel data corresponding to the bubble is captured as the time-series image feature amount. In this way, for example, when capturing a time-series variation in the image feature amount corresponding to the bubble, the neural network can recognize that the state is the second state.

For example, if the liquid surface is in a third state, that is, a "state in which the liquid surface is surged but no bubble is present or is inconspicuous", the time-series variation of the value of the pixel data corresponding to the surging of the liquid surface is captured as the time-series image feature amount. In this way, for example, when capturing a time-series variation in the image feature amount corresponding to a surge in the liquid surface, the neural network can recognize that the state is the third state.

For example, if the image is in the "boiling state" which is the fourth state, the time-series fluctuation of the values of the pixel data corresponding to the surge of the liquid surface and the bubble is captured from the entire image area. In this case, the time-series variation of the value of the pixel data corresponding to the bubble is larger than the variation in the third state. In this way, the time-series fluctuation of the value of the pixel data corresponding to the surge of the liquid surface is captured as the time-series image feature amount. Thus, for example, when a time-series variation in the pixel data having a larger value corresponding to the liquid level and the bubble is captured as the time-series image feature amount, the neural network can recognize that the state is the fourth state.

In this way, it is possible to capture a slight change before boiling from the time-series image feature amount acquired based on the plurality of time-series images obtained by imaging the inside of the cooking container 20 containing the object to be heated and the liquid in time series. Thereby, a plurality of heating states of the liquid can be recognized using these time-series image feature quantities. In the present embodiment, the heating state is classified into 4 types, but the present invention is not limited to this. For example, the classification may be 5 or 10.

Here, an example of the learning process of the learning unit 604 will be described with reference to fig. 5. Fig. 5 is a flowchart showing an example of the learning process of the learning unit 604. As shown in fig. 5, first, the learning unit 604 acquires a set of learning data from the storage unit 602 (step S100). When the learning data is a time-series image, the feature amount generator 603 converts the learning data into a time-series image feature amount. Next, the learning unit 604 starts learning the neural network (step S102). In this case, learning is performed in which the time-series image feature amount is provided as an input, and any one of the tags 1 to 4 corresponding to the time-series image feature amount is provided as a teacher signal (step S104). That is, learning is performed based on back propagation that corrects the connection coefficient so that an error between the tag corresponding to the input feature amount and the output value of the neural network is reduced.

Then, the learning unit 604 determines whether or not the learning termination condition is satisfied (step S106), and if the learning termination condition is not satisfied (step S106: NO), the processing from step S104 is repeated. The end condition is, for example, the number of learning times.

On the other hand, when the termination condition is satisfied (step S106: YES), the entire process is terminated. In this manner, a neural network for outputting the state of the time-series image feature amount obtained from the time-series plurality of images is learned.

Here, an example of the processing program of the heating power control unit 608 will be described. The processing program of the heating power control unit 608 has programmed therein the duration of the first to fourth states, for example, for each meal. Here, in the case of cooking in a state where the cooking is not boiling, the heating power control unit 608 controls the heating state and the cooling state of the object and the liquid by setting the time in the third state immediately before boiling to be the longest, for example.

For example, the heating power control section 608 uses the recognition result of the recognition section 607 to control the intensity of the heating power of the heating apparatus 10 and the holding time of the heating power to shift the programmed state.

A control example of the heating power control unit 608 will be described with reference to fig. 6. Fig. 6 is a flowchart showing a control example of the heating power control section 608. As shown in fig. 6, first, the acquisition unit 600 acquires a plurality of time-series images from the imaging device 30 (step S200).

Next, the feature amount generation unit 605 generates a time-series image feature amount from 60 × 60 × 3 time-series pixel data of 60 sheets, inputs the image feature amount to the identifier 607, and identifies the state of the plurality of time-series images (step S202).

Next, the heating power control unit 608 controls the heating power of the heating apparatus 10 based on the state recognized by the recognition unit (step S204). For example, when the current state is maintained, the heating power of the heating apparatus 10 is controlled so as not to change the heating power, and when the state is heated to the next state, the heating power of the heating apparatus 10 is increased. On the other hand, when the state is cooled to the next state, the heating power of the heating device 10 is reduced.

Subsequently, the heating power control unit 608 determines whether or not the programmed cooking is finished (step S206), and if not (step S206: NO), repeats the processing from step S200. On the other hand, when the process is completed (step S206: YES), the heating apparatus 10 is stopped, and the whole process is completed. In the case of heating control using a thermometer, the temperature may vary depending on the position of the object to be heated and the state of occurrence of convection. On the other hand, in recognition of images based on time series, information on a state change of the entire image is used, and therefore, recognition of the state is more accurate. Therefore, the heating control by the heating power control unit 608 can be performed with higher accuracy.

As described above, according to the present embodiment, the learning unit 604 performs machine learning of a predetermined state of the liquid before boiling on the basis of a plurality of time-series images obtained by imaging the container including the object to be heated and the liquid in time series. Thus, the recognition unit 607 can recognize the predetermined state before boiling by capturing a slight change in the surface image of the liquid as a time-series change in the image feature amount based on the learning result of the learning unit 604. Since the recognition unit 607 recognizes a predetermined state before boiling, the heating power control unit 608 can perform heating control for cooking so as not to boil a container containing an object to be heated and a liquid, for example.

(first modification of the first embodiment)

The heating control system 1 according to the first embodiment recognizes the heating state of the cooking container 20 by the heating device 10, but the heating control system 1 according to the first modification of the first embodiment is different from the heating control system 1 according to the first embodiment in that it is also possible to recognize the operation of a person on the cooking container 20. Next, differences from the first embodiment will be described.

Fig. 7 is a table showing an example of a correct tag corresponding to the operation of a person of a cook. For example, the fifth state is "a state in which stirring is performed by a tool without depending on the liquid state". The sixth state is "a state where liquid is not supplied and a tool is used for skimming or the like". The seventh state is "a state in which the food is not in a liquid state but is charged".

That is, in the heating control system 1 according to the modification of the first embodiment, the machine learning model used by the recognition unit 607 is learned using time-series images of the fifth to seventh states in addition to the first to fourth states. In this case, for example, the machine learning model used by the recognition unit 607 is learned by assigning a correct label of 5 to the fifth state, a correct label of 6 to the sixth state, and a correct label of 7 to the seventh state.

This makes it possible to recognize the state of the pot and preferentially detect the movement of the operating cooker. For example, when the recognition unit 607 recognizes the seventh state, the heating power control unit 608 causes the display device 40 to display a display image prompting the waste of the object to be heated in the cooking container 20 after a predetermined time elapses from the time when the seventh state is recognized. Accordingly, the quality of the object to be heated in the cooking container 20 can be maintained, and the recognition unit 607 can recognize any one of the fifth to seventh states in addition to the first to fourth states, using the plurality of time-series images captured in time series.

(second modification of the first embodiment)

The heating control system 1 according to the second modification of the first embodiment also has a function of stopping heating by issuing a warning when the heating state of the cooking container 20 is changed to a predetermined state.

Fig. 8 is a diagram showing an example of a state in which the state changes to the third state in which the liquid surface surges but bubbles are not present or are inconspicuous after the fourth state changes to the "boiling state" a plurality of times.

As shown in fig. 8, in the heating control system 1, when the heating control that is stopped in the third state is performed, there is a possibility that the state will be changed to the fourth state. In this way, when the number of times of the fourth state becomes, for example, 3 times or more, which is a predetermined number of times, a warning is issued to the display device 40, and heating is stopped.

An object to be heated may contain an article whose quality may be deteriorated by boiling for a long time. In this way, by issuing a warning according to the number of times of boiling, confirmation of the quality can be promoted, and the object to be heated can be discarded according to the quality state.

(third modification of the first embodiment)

The heating control device 60 according to a third modification of the first embodiment further includes: a liquid temperature measuring unit (not shown) for measuring the temperature of the liquid in the cooking container 20; and an air pressure measuring unit (not shown) for measuring the atmospheric pressure. The identification unit 607 identifies a plurality of heating states of the liquid using at least one of the liquid temperature measured by the liquid temperature measurement unit and the air pressure measured by the air pressure measurement unit, and the time-series image feature amount acquired based on the plurality of time-series images. For example, the recognition unit 607 can correct the recognition results of the plurality of heating states of the liquid recognized using the time-series image feature values acquired based on the plurality of time-series images, based on the result of measurement by at least one of the liquid temperature measurement unit and the air pressure measurement unit.

The heating power control unit 608 controls the heating power of the heating device 10 for heating the cooking container 20 using the recognition result of the recognition unit 607 based on the result measured by at least one of the liquid temperature measuring unit and the air pressure measuring unit. For example, the pressure varies depending on the height of the place where the heating control device 60 is installed, and the boiling point varies. The recognition unit 607 can correct such variation due to the environment based on the result of measurement by at least one of the liquid temperature measurement unit and the air pressure measurement unit, and the heating control device 60 can perform more accurate control.

(fourth modification of the first embodiment)

The heating control device 60 according to the fourth modification of the first embodiment further includes a time measuring unit (not shown) that measures a plurality of heating states and other times. The time measuring unit measures the elapsed time in each state (for example, in the first to fourth states (fig. 4) and in the fifth to seventh states (fig. 7) described later).

The heating power control unit 608 controls the heating power of the heating device 10 for heating the cooking container 20 based on the recognition result of the recognition unit 607 and the elapsed time from the input of the object to be heated measured by the time measuring unit. For example, the heating may be stopped after the cooking is continued for a certain period of time, and the temperature may be maintained. Thus, the heating power control unit 608 can perform control such as stopping heating at a specific time after the food material (e.g., meat) is charged.

(second embodiment)

The heating control system 1 according to the second embodiment differs from the heating control system 1 according to the first embodiment in that the neural network used in the identification unit is changed in accordance with the shape of the cooking container 20. Next, differences from the first embodiment will be described.

Fig. 9 is a block diagram showing the configuration of the heating control system 1 according to the second embodiment. As shown in fig. 9, the heating control system 1 of the present embodiment is further provided with a shape recognition unit 612 of the heating control device 60.

The cooking container 20 has a plurality of different shapes such as a circular shape and a square shape. The learning section 604 learns the machine learning model used at the shape recognition section 612 for shape recognition. In this case, the input data is image data captured from above the cooking container 20, and the learning data in which the teacher category is a shape is used.

The machine learning model for the recognition unit 607 is also learned using the learning data 1 obtained by imaging the circular cooking container 20 and the learning data 2 obtained by imaging the square cooking container 20. That is, the recognition unit 607 includes a machine learning model 1 of a learning result 1 obtained by learning with the learning data 1 and a machine learning model 2 of a learning result 2 obtained by learning with the learning data 2.

Convection of the circular cooking container 20 occurs with the center of the circular pan as a convection center. On the other hand, the convection of the square-shaped cooking container 20 shows a tendency that the center of convection is shifted to a position influenced by the heating position. Therefore, the recognition accuracy is further improved by using the machine learning model 1 and the machine learning model 2 separately according to the shape of the cooking container 20.

A control example of the heating power control unit 608 according to the second embodiment will be described with reference to fig. 10. Fig. 10 is a flowchart showing a control example of the heating power control unit 608 according to the second embodiment. As shown in fig. 12, first, the acquisition unit 600 acquires an image from the imaging device 30 (step S300).

Next, the shape recognition unit 612 converts the image into 60 × 60 × 3 pixel data, inputs the pixel data to the machine learning model used by the shape recognition unit 612, recognizes the state of the image, and determines whether or not the image is a circular shape (step S302).

Next, if it is a circular shape (step S302: YES), the recognition unit 607 uses the machine learning model 1 of the learning result 1, and if it is a square shape (step S302: NO), the recognition unit 607 uses the machine learning model 2 of the learning result 2.

As described above, according to the present embodiment, the neural network used in the recognition unit 607 is changed according to the shape of the cooking container 20. Thus, even when the state in the container differs depending on the shape of the cooking container 20, the state in the container can be recognized with higher accuracy.

At least a part of the heating control device 60 and the heating control system 1 described in the above embodiments may be configured by hardware or software. In the case of software, a program for realizing at least a part of the functions of the heating control device 60 and the heating control system 1 may be stored in a recording medium such as a flexible disk or a CD-ROM, and the program may be read by a computer and executed. The recording medium is not limited to a removable recording medium such as a magnetic disk or an optical disk, and may be a fixed type recording medium such as a hard disk unit or a memory.

Further, a program for realizing at least a part of the functions of the heating control device 60 and the heating control system 1 may be distributed via a communication line (including wireless communication) such as the internet. The program may be distributed via a wired line such as the internet or a wireless line, or may be distributed after being stored in a recording medium in a state where the program is encrypted, modulated, or compressed.

Although several embodiments have been described above, these embodiments are presented as examples only, and are not intended to limit the scope of the invention. The novel apparatus, method, and program described in this specification can be implemented in other various ways. The embodiments of the apparatus, method, and program described in the present specification may be variously omitted, replaced, and modified without departing from the spirit of the present invention.

Description of the reference numerals

1: a cooking system; 10: a heating device; 20: a cooking container; 30: a camera device; 40: a display device; 60: a heating control device; 61: a heating state recognition device; 62: a heating state identification system; 600: an acquisition unit; 603: a feature value generation unit; 604: a learning unit; 606: a recognizer learning unit; 607: an identification unit; 608: a fire control unit; 612: a shape recognition unit.

Claims

1. A heating state recognition device is provided with:

an identification unit that identifies a plurality of heating states of the liquid using the time-series image feature amounts acquired based on the plurality of time-series images.

2. The heating state identifying device according to claim 1,

the image processing apparatus further includes a feature amount generation unit having a convolutional neural network that generates an image feature amount for each of the plurality of time-series images and arranges the image feature amounts in time series to generate the time-series image feature amount,

the identification unit has a recurrent neural network that identifies the plurality of heating states based on time-series image feature quantities generated by the recurrent neural network.

3. The heating state identifying device according to claim 1 or 2,

further comprising a learning unit for performing machine learning by using a learning set including an accurate label corresponding to the states of the object to be heated and the liquid and a time-series image feature amount generated by the convolutional neural network,

the recognition portion recognizes a state based on a result of the machine learning.

4. The heating state identifying device according to claim 3,

the learning unit performs the machine learning using a learning set that differs depending on a shape of the container,

the recognition portion recognizes a state based on a result of the machine learning corresponding to a shape of the container.

5. A kind of heating control device is disclosed,

a heating power control unit that controls the heating power of the heating device for heating the container based on the recognition result of the recognition unit of the heating state recognition device according to any one of claims 1 to 5 is provided.

6. The heating control device according to claim 5,

the heating power control unit performs control for changing a time for maintaining a predetermined state of the liquid before boiling, in accordance with the predetermined state.

7. The heating control device according to claim 5 or 6,

when the recognition unit recognizes the state in which the object is input, independently of the liquid state of the liquid, the heating power control unit causes the display device to display a display image urging discarding of the object in the container after a predetermined time has elapsed from the time when the state in which the object is input is recognized.

8. A heating state recognition method includes the following steps:

and a recognition step of recognizing a plurality of heating states of the liquid using the time-series image feature values obtained by acquiring time-series image feature values based on the plurality of time-series images.

9. A method for controlling the heating of a substrate,

the heating apparatus includes a heating power control step of controlling the heating power of a heating device for heating the container based on the recognition result of the recognition step of the heating state recognition method according to claim 8.

10. A heating state recognition system is provided with:

an imaging device that images the inside of a container from the upper part of the container having the object to be heated and the liquid;

an acquisition unit that acquires a plurality of images captured by the imaging device in time series; and

11. A heating control system is provided, which comprises a heating control system,

the heating apparatus according to claim 10, further comprising a heating power control unit for controlling the heating power of the heating device for heating the container based on the recognition result of the recognition unit of the heating state recognition system.