JP2021169875A - Heating state identification device, heating control device, heating control method, heating state identification system, and heating control system - Google Patents

Abstract

To provide a heating state identification device, a heating control device, a heating control method, a heating state identification system, and a heating control system.SOLUTION: A heating state identification device pertaining to an embodiment is provided with: an acquisition unit for acquiring multiple time-series images by capturing the inside of a container containing an object to be heated and a liquid in a time-series manner; and an identification unit for identifying multiple heating states in the liquid by using time-series image feature amounts acquired on the basis of the multiple time-series images.SELECTED DRAWING: Figure 1

Description

Embodiments of the present invention relate to a heating state identification device, a heating control device, a heating control method, a heating state identification system, and a heating control system.

Traditionally, in restaurants, store employees have endeavored to determine the amount of heat and provide customers with more delicious cooked food. For example, there is a cooking method in which the temperature of the pot is adjusted so that it does not boil. However, the heating adjustment of the heating device that heats the pot or the like may depend on the experience of the employee.

JP-A-2017-133722

Therefore, the problem to be solved by the present invention is a heating state identification device, a heating control device, a heating control method, and a heating state capable of performing cooking with more appropriate heating control without being influenced by the experience of the cook. It is to provide an identification system and a heating control system.

The heating state identification device according to one embodiment includes an acquisition unit that acquires a plurality of time-series images of the inside of a container having a heated object and a liquid taken in time series.
An identification unit that identifies multiple container states in a liquid using time-series features acquired based on multiple time-series images.
To be equipped.

A feature amount generator having a convolutional neural network that generates an image feature amount for each image of a plurality of time-series images and arranges them in a time series to generate a time-series image feature amount is further provided.
The identification unit uses a recurrent neural network that identifies multiple heating states based on the time-series image features generated by the convolutional neural network.
You may have.

Further equipped with a learning unit that performs machine learning using a learning set including a correct label corresponding to the state of the object to be heated and the state of the liquid and a time-series image feature amount generated by the convolutional neural network.
The identification unit may identify the state based on the result of machine learning.

The heating control device may be a thermal power control unit that controls the thermal power of the heating device that heats the container based on the identification result of the identification unit of the heating state identification device described above.

The thermal power control unit may perform control to change the time for maintaining the predetermined state according to the predetermined state before boiling of the liquid.

When the identification unit identifies the state in which the object to be heated is charged regardless of the liquid state of the liquid, the thermal power control unit elapses a predetermined time from the time when the state in which the object to be heated is input is identified. After that, a display image prompting the disposal of the object to be heated in the container may be displayed on the display device.

The heating control method according to one embodiment includes an acquisition step of acquiring a plurality of time-series images obtained by taking time-series images of the inside of a container having an object to be heated and a liquid.
An identification step of identifying a plurality of heating states in a liquid by using a time-series image feature amount obtained by acquiring a time-series image feature amount based on a plurality of time-series images.
To be equipped.

A thermal power control step that controls the thermal power of the heating device that heats the container based on the identification result of the identification process of the heating state identification method, and
May be provided.

The heating state identification system according to the embodiment includes a photographing device that photographs the inside of the container from the upper part of the container having the object to be heated and the liquid.
An acquisition unit that acquires multiple images taken by the imaging device in chronological order,
An identification unit that identifies multiple heating states in a liquid using time-series image features acquired based on multiple time-series images.
To be equipped.

The heating control system includes a thermal power control unit that controls the thermal power of the heating device that heats the container based on the identification result of the identification unit of the heating state identification system described above.
May be provided.

According to the present invention, it is possible to provide a heating control device, a heating control method, and a heating control system capable of performing cooking with more appropriate heating control without being influenced by the experience of the cook.

The block diagram which shows the structure of the heating control system by this embodiment. The figure which shows an example of the position of the photographing apparatus which photographes a cooking container. The figure which shows the structural example of the feature amount generation part. A table that describes the state of the object to be heated and the liquid in text. The flowchart which shows the learning processing example of the learning part. The flowchart which shows the control example of the thermal power control part. A table showing examples of correct labels corresponding to human operations. The figure which shows the example which the state which becomes the 3rd state occurred a plurality of times after becoming the 4th state. The block diagram which shows the structure of the cooking instruction apparatus by 2nd Embodiment. The flowchart which shows the control example of the thermal power control part which concerns on 2nd Embodiment.

Hereinafter, the heating state identification device, the heating control device, the heating control method, the heating state identification system, and the heating control system according to the embodiment of the present invention will be described in detail with reference to the drawings. The embodiments shown below are examples of the embodiments of the present invention, and the present invention is not construed as being limited to these embodiments. Further, in the drawings referred to in the present embodiment, the same parts or parts having similar functions are designated by the same reference numerals or similar reference numerals, and the repeated description thereof may be omitted. In addition, the dimensional ratio of the drawing may differ from the actual ratio for convenience of explanation, or a part of the configuration may be omitted from the drawing.

(First Embodiment)
FIG. 1 is a block diagram showing a configuration of a heating control system 1 according to the first embodiment. The heating control system 1 of FIG. 1 is a system capable of controlling the thermal power of the heating device. As shown in FIG. 1, the heating control system 1 according to the present embodiment includes a heating device 10, a cooking container 20, a photographing device 30, a display device 40, an operation unit 50, and a heating control device 60. Will be done.

FIG. 2 is a diagram showing an example of the position of the photographing device 30 in which the inside of the cooking container 20 placed on the heating device 10 is photographed from above. As shown in FIG. 2, the heating device 10 is, for example, a gas stove, and has a burner as a heat generating portion on the upper surface of the housing thereof. In addition, it is possible to adjust the thermal power of the burner (adjust the amount of heat generated) by rotating the burner operation button with the motor. The heating device 10 according to the present embodiment is a gas stove, but the heating device 10 is not limited to this. For example, it may be an electric stove.

The cooking container 20 is, for example, a pot, and stores foodstuffs to be heated and cooking water in which water and seasonings are mixed.
The photographing device 30 is, for example, a camera, and can photograph an image showing the state inside the cooking container 20 in chronological order from the upper part of the cooking container 20. The photographing device 30 outputs three types of digital image data, for example, a red image (R image), a green image (G image), and a blue image (B image) to the heating control device 60. Each of the three types of digital image data is composed of, for example, 1080 vertical × 1920 horizontal pixels.

As shown again in FIG. 1, the display device 40 is, for example, a monitor. The display device 40 displays the image generated by the heating control device 60.

The operation unit 50 is composed of, for example, a keyboard, a pointing device, or the like. It can also be integrated into the display device 40 as a touch panel. The operation unit 50 outputs an instruction signal according to the operation of the user who uses the heating control device 60.

The heating control device 60 is a device capable of controlling the thermal power of the heating device 10, and includes an acquisition unit 600, a storage unit 602, a feature amount generation unit 603, a learning unit 604, an identification unit 607, and a thermal power. It includes a control unit 608 and an image generation unit 610. Such a heating control device 60 is realized by, for example, a personal computer. That is, the heating control device 60 includes, for example, a CPU (Central Processing Unit). Further, the heating state identification device 61 according to the present embodiment includes an acquisition unit 600, a feature amount generation unit 603, a learning unit 604, and an identification unit 607, and the heating state identification system 62 according to the present embodiment includes an acquisition unit 600, a feature amount generation unit 603, a learning unit 604, and an identification unit 607. A photographing device 30 and a heating state identification device 61 are provided. The heating state identification device 61 according to the present embodiment includes, but is not limited to, the learning unit 604. For example, the heating state identification device 61 may be configured without the learning unit 604.

The acquisition unit 600 can also acquire time-series images taken by the photographing apparatus 30 and store them in the storage unit 602. The acquisition unit 600 is also connected to a network, for example, and can acquire an image from a server or the like via the network.

The storage unit 602 is composed of, for example, an HDD (hard disk drive), an SSD (solid state drive), or the like. The storage unit 602 can also store the image acquired by the acquisition unit 600 for learning. Further, the storage unit 602 stores various programs for executing the heating control. As a result, the heating control device 60 constitutes each processing unit by, for example, executing a program stored in the storage unit 602. Further, the storage unit 602 stores a machine learning model for identification. In this embodiment, the trained machine learning model may be referred to as a trained model.

The feature amount generation unit 603 generates, for example, an image feature amount in time series from each of the time-series images acquired by the acquisition unit 600. The learning data is, for example, data in which the time-series image feature amount generated by the feature amount generation unit 603 is associated with the correct label corresponding to the state of the object to be heated and the liquid in the cooking container 20. Details of the feature amount generation unit 603 will be described later.

The learning unit 604 performs machine learning, for example, supervised learning, using the learning data stored in the storage unit 602. The learning unit 604 performs machine learning with, for example, a neural network. Further, the learning unit 604 may perform arithmetic processing outside the heating control device 60, for example, in a computer having a higher computing power for learning. The learned model (learned model) is stored in the storage unit 602 via a network, for example, and is used by the identification unit 607. In the present embodiment, the learning unit 604 is calculated and processed in a computer outside the heating control device 60, but the learning unit 604 is not limited to this. For example, the learning unit 604 may be provided in the heating control device 60. In this case, it is possible to immediately reflect the data obtained in the field in the learning.

Further, the machine learning model that the learning unit 604 performs machine learning is not limited to the neural network. For example, a discriminant function or the like may be learned as a machine learning model by a general clustering method such as the K-means method. The details of the learning unit 604 will be described later.

The identification unit 607 identifies a predetermined state of the liquid in the cooking container 20 by using the machine learning result in the learning unit 604. The identification unit 607 according to the present embodiment uses the neural network learned by the learning unit 604. This neural network uses the time-series feature amount generated by the feature amount generation unit 603 based on the plurality of time-series images taken by the photographing device 30, and determines a predetermined amount of the liquid in the cooking container 20 before boiling. It is possible to identify multiple heating states, including states. Details of the identification unit 607 will be described later.

The thermal power control unit 608 controls the thermal power of the heating device 10 based on the identification result of the identification unit 607. For example, the thermal power control unit 608 controls the rotation of the motor of the heating device 10 to control the thermal power of the heating device 10. Details of the thermal power control unit 608 will also be described later.

The image generation unit 610 generates an image showing the state inside the cooking container 20 based on the identification result of the identification unit 607, and outputs the image to the display device 40. Further, the image generation unit 610 may generate an image indicating an operation instruction based on the identification result of the identification unit 607.

Here, the details of the feature amount generation unit 603 and the learning unit 604 will be described with reference to FIG.
FIG. 3 is a diagram showing a configuration example of the feature amount generation unit 603. As shown in FIG. 3, the feature amount generation unit 603 has a plurality of image feature amount extraction units 603a. The image feature amount extraction unit 603a is, for example, a convolutional neural network (CNN, convolutional neural network, hereinafter may be simply referred to as CNN). In this embodiment, a plurality of image feature amount extraction units 603a are used, but the present invention is not limited to this. For example, the images t1 to tn may be input to one image feature amount extraction unit 603a in order, and the image feature amounts t1 to tn may be generated in order.

In this CNN, neurons in each layer are grouped into filters, and each filter detects different overlapping features in the input data. A multi-layer neural network architecture in a convolutional neural network has, for example, a plurality of convolutional layers, a plurality of sampling layers, and a plurality of fully connected layers. In addition, this CNN is a trained CNN (for example, ImageNet pre-trained net) trained by an existing labeled image.
work). That is, the feature amount extraction unit 603a fixes the convolutional neural network, feedforwards the input image, and generates the value output from the intermediate layer as the feature amount.

Images t1 to tn taken in time series are input to each of the plurality of feature amount extraction units 603a. For example, as one image, an image obtained by reducing each of three types of digital images, a red image (R image), a green image (G image), and a blue image (B image) to 224 × 224 pixels is used. In the present embodiment, the three primary colors R, G, and B are held as numerical values of 8 bits (256 gradations) each. As a result, 224 × 224 × 3 pixel data is input to the first-stage convolution layer of the CNN. Then, time-series image feature quantities t1 to tn are generated. For example, n is 60, and 60 time-series images taken per second are input to the learning unit 604 or the identification unit 607. The image feature amount generated by the feature amount extraction unit 603a is, for example, 1024 dimensions. In this way, the CNN generates image feature amounts t1 to tn for each image t1 to tn. Then, the feature amount generation unit 603 outputs the image feature amounts t1 to tn arranged in time series.

The data for learning will be described in detail with reference to FIG.
FIG. 4 is a table for explaining the states of the object to be heated and the liquid in text. The state of the object to be heated and the liquid in the cooking container 20 changes in time series with the passage of the heating time. That is, as the heating of the cooking container 20 progresses, for example, the state changes from the first state to the fourth state in chronological order. For example, the first state is "a state in which nothing is boiled and no operation is performed". The second state is "a state in which the liquid level is stable and bubbles rise in places". The third state is "a state in which the liquid level is raised, but bubbles are not generated or are inconspicuous". The fourth state is the "boiling state". For example, the first to third states indicate the states before boiling. In the data for learning, the first state is shown as a correct label on the time-series image obtained by the acquisition unit 600 or the time-series image feature amount obtained by the feature amount generation unit 603 in the first state. A label is given to the time-series image obtained by the acquisition unit 600 or the time-series image feature amount obtained by the feature amount generation unit 603 in the second state, and a label indicating the second state is added as a correct answer label. A label indicating the third state is given as a correct answer label to the time-series image obtained by the acquisition unit 600 or the time-series image feature amount obtained by the feature amount generation unit 603 in the third state. , A time-series image obtained by the acquisition unit 600 in the fourth state or a time-series image feature amount obtained by the feature amount generation unit 603 is given a label indicating the fourth state as a correct answer label. These labels can be attached manually, for example. These labeled time-series images or time-series image features are stored in the storage unit 602 as learning data.

Returning to FIG. 3, the learning unit 604 will be described. The learning unit 604 learns a neural network, which is an example of a machine learning model, using the above-mentioned learning data. The neural network learned by the learning unit 604 learns, for example, a labeled time-series image feature amount and outputs a label (first to fourth states) for the input time-series image feature amount. do. For example, this neural network is a neural network (LSTM: Long Short-term Memory) in which layers having a plurality of nodes are connected in multiple stages. LSTM is a form of recurrent neural network (recurrent neural network) that learns time series data.

For example, in this neural network, the connection coefficient between each node is initialized, and labeled time-series image features are input in order. Then, the neural network performs processing such as backpropagation (error backpropagation method) that corrects the connection coefficient so that the error between the output in the neural network and the label given to the input time-series image feature amount is reduced. Is generated by. For example, a trained neural network is generated by performing processing such as backpropagation so as to minimize a predetermined loss function. By repeating the above-mentioned processing, the neural network can better reproduce and output the label for the input feature amount. The learning method of the neural network is not limited to the above-mentioned method, and any known technique can be applied.

The identification unit 607 has a neural network that has been trained as described above. This neural network can discriminate the state before boiling into a plurality of states, which was difficult to discriminate in the past. For example, in the first state, "a state in which nothing is boiling and no operation is performed", there is no change in the time-series image features, and the neural network can be identified as the first state. ..

Further, for example, in the second state, "the liquid level is stable and bubbles rise in places", the time-series fluctuation of the pixel data value corresponding to the bubbles is a time-series image feature amount. Is caught as. In this way, for example, when the time-series fluctuation of the image feature amount corresponding to the bubble is captured, the neural network can be identified as the second state.

Further, for example, in the third state, "the liquid level is raised, but bubbles are not generated or are inconspicuous", the value of the pixel data corresponding to the liquid level rise is changed over time. It is captured as a time-series image feature amount. In this way, for example, when the time-series fluctuation of the image feature amount corresponding to the swelling of the liquid level is captured, the neural network can be identified as the third state.

Further, for example, in the fourth state of "boiling state", the swelling of the liquid level and the time-series fluctuation of the pixel data value corresponding to the bubble can be captured from the entire area of the image. In this case, the time-series fluctuation of the value of the pixel data corresponding to the bubble becomes larger than that in the third state. In this way, the time-series fluctuation of the pixel data value corresponding to the swelling of the liquid level is captured as the time-series image feature amount. This makes it possible for the neural network to be identified as the fourth state, for example, when a larger time-series variation in the pixel data values corresponding to the liquid level and bubbles is captured as a time-series image feature amount.

In this way, it is possible to capture minute changes before boiling by the time-series image features acquired based on a plurality of time-series images of the inside of the cooking container 20 having the object to be heated and the liquid. Become. Thereby, a plurality of heating states in the liquid can be identified by using these time-series image features. In this embodiment, the heating state is classified into four categories, but the heating state is not limited to this. For example, 5 classifications or 10 classifications may be used.

Here, an example of learning processing of the learning unit 604 will be described with reference to FIG. FIG. 5 is a flowchart showing an example of learning processing of the learning unit 604. As shown in FIG. 5, first, the learning unit 604 acquires a set of learning data from the storage unit 602 (step S100). When the training data is a time-series image, the feature amount generation unit 603 is used here to convert the training data into a time-series image feature amount. Next, the learning unit 604 starts learning the neural network (step S102). In this case, learning is performed by giving a time-series image feature amount as an input and giving a label of any one of 1 to 4 corresponding to the time-series image feature amount as a teacher signal (step S104). That is, learning is performed by backpropagation that corrects the connection coefficient so that the error between the label corresponding to the input feature amount and the output value of the neural network is reduced.

Next, the learning unit 604 determines whether or not the learning end condition is satisfied (step S106).
If the end condition is not satisfied (NO in step S106), the process from step S104 is repeated. The end condition is, for example, the number of learnings.

On the other hand, when the end condition is satisfied (YES in step S106), the entire process is terminated. In this way, a neural network that outputs a state for a time-series image feature amount obtained from a plurality of time-series images is learned.

Here, an example of the processing program of the thermal power control unit 608 will be described. In the processing program of the thermal power control unit 608, the duration of the first to fourth states is programmed for each dish, for example. Here, when cooking in a state where boiling is not performed, the thermal power control unit 608 controls, for example, the heating state and the cooling state of the object to be heated and the liquid by setting the time of the third state immediately before boiling as the longest.

For example, the thermal power control unit 608 controls the strength and weakness of the thermal power of the heating device 10 and the maintenance time of the thermal power so as to transition the programmed state by using the identification result of the identification unit 607.

A control example of the thermal power control unit 608 will be described with reference to FIG. FIG. 6 is a flowchart showing a control example of the thermal power control unit 608. As shown in FIG. 6, first, the acquisition unit 600 acquires a plurality of time-series images from the photographing device 30 (step S200).

Next, the feature amount generation unit 605 generates a time-series image feature amount from 60 time-series pixel data of 60 × 60 × 3, inputs it to the classifier 607, and states a plurality of time-series images. Is identified (step S202).
Next, the thermal power control unit 608 controls the thermal power of the heating device 10 according to the identified state of the identification unit (step S204). For example, when maintaining the current state, the heating power of the heating device 10 is controlled so as not to change the heating power, and when heating the state to the next state, the heating power of the heating device 10 is increased. On the other hand, when the state is cooled to the next state, the thermal power of the heating device 10 is reduced.

Next, the thermal power control unit 608 determines whether or not the programmed cooking is completed (step S206), and if not (No in step S206), repeats the process from step S200. On the other hand, when it is finished (YES in step S206), the heating device 10 is stopped and the whole process is finished. In the case of heating control using a thermometer, the temperature becomes uneven depending on the position of the object to be heated and the state of convection. On the other hand, in the time-series image identification, the information on the state change of the entire image is used, so that the state identification becomes more accurate. Therefore, the heating control by the thermal power control unit 608 can be performed with higher accuracy.

As described above, according to the present embodiment, the learning unit 604 machine-learns a predetermined state before boiling of the liquid by using a plurality of time-series images of the container having the object to be heated and the liquid taken in time series. I decided. As a result, the identification unit 607 can grasp a slight change in the surface image of the liquid as a time-series variation of the image feature amount based on the learning result of the learning unit 604, and can identify a predetermined state before boiling. .. Further, since the identification unit 607 identifies a predetermined state before boiling, the heating power control unit 608 can control the heating to cook the container having, for example, the object to be heated and the liquid so as not to boil.

(First Modified Example of First Embodiment)
The heating control system 1 according to the first embodiment has identified the heating state of the cooking container 20 by the heating device 10, but the heating control system 1 according to the first modification of the first embodiment has the same with respect to the cooking container 20. The first embodiment differs from the heating control system 1 in that human operations can also be identified. Hereinafter, the differences from the first embodiment will be described.

FIG. 7 is a table showing an example of a correct label corresponding to a human operation of a cook. For example, the fifth state is "a state of stirring with a tool regardless of the liquid state". The sixth state is "a state in which lye is removed using a tool regardless of the liquid state". The seventh state is "a state in which ingredients are added regardless of the liquid state".

That is, in the heating control system 1 according to the modified example of the first embodiment, the machine used by the identification unit 607 using time-series images of the fifth to seventh states in addition to the first to fourth states. Learn the learning model. In this case, for example, a correct label of 5 is given to the 5th state, 6 is given to the 6th state, and 7 is given to the 7th state, and the machine learning model used by the identification unit 607 is learned.

This makes it possible to preferentially detect the movement of the cook who recognizes and operates the state of the pot. For example, when the identification unit 607 identifies the seventh state, the thermal power control unit 608 prompts the disposal of the heated object in the cooking container 20 after a predetermined time has elapsed from the time when the seventh state is identified. The image is displayed on the display device 40. Thereby, the quality of the object to be heated in the cooking container 20 can be maintained. Thus, the identification unit 607 uses a plurality of time-series images taken in time-series in addition to the first to fourth states. , Any one of the 5th to 7th states can be identified.

(Second modification of the first embodiment)
The heating control system 1 according to the second modification of the first embodiment further has a function of issuing a warning and stopping the heating when the heating state of the cooking container 20 reaches a predetermined state.

FIG. 8 shows an example in which the third state of “the liquid level is rising but no bubbles or inconspicuous” occurs multiple times after the fourth state of “being in a boiling state”. It is a figure which shows.

As shown in FIG. 8, in the heating control system 1, there is a possibility that the fourth state may be reached when the heating control for stopping in the third state is performed. In this way, when the number of times of the fourth state becomes, for example, three times or more, which is a predetermined number of times, a warning is issued to the display device 40 and heating is stopped.

The object to be heated may contain an item whose quality may deteriorate due to boiling for a long time. In this way, by issuing a warning based on the number of boiling times, it is possible to prompt confirmation of quality, and depending on the quality state, it is also possible to discard the object to be heated.

(Third variant of the first embodiment)
The heating control device 60 according to the third modification of the first embodiment includes a liquid temperature measuring unit (not shown) for measuring the liquid temperature in the cooking container 20 and a pressure measuring unit (not shown) for measuring the atmospheric pressure. Further prepare. The identification unit 607 uses at least one of a time-series image feature amount acquired based on a plurality of time-series images, a liquid temperature measured by the liquid temperature measuring unit, and an atmospheric pressure measured by the atmospheric pressure measuring unit in a liquid. Identify multiple heating conditions. For example, the identification unit 607 obtains at least the identification results of a plurality of heating states in the liquid identified using the time-series image feature quantities acquired based on the plurality of time-series images of the liquid temperature measurement unit and the pressure measurement unit. It is possible to make corrections based on the result measured by one of them.

The thermal power control unit 608 controls the thermal power of the heating device 10 for heating the cooking container 20 by using the identification result of the identification unit 607 based on the result measured by at least one of the liquid temperature measurement unit and the atmospheric pressure measurement unit. .. For example, the atmospheric pressure changes and the boiling point also differs depending on the altitude of the place where the heating control device 60 is installed. The identification unit 607 can correct such fluctuations due to the environment based on the results measured by at least one of the liquid temperature measuring unit and the atmospheric pressure measuring unit, and the heating control device 60 can perform more accurate control. It becomes.

(Fourth Modified Example of First Embodiment)
The heating control device 60 according to the fourth modification of the first embodiment further includes a time measuring unit (not shown) for measuring the time of a plurality of heating states and the like. The time measuring unit measures the elapsed time of each state (for example, the first to fourth states (FIG. 4) and the fifth to seventh states (FIG. 7) described later).

The thermal power control unit 608 controls the thermal power of the heating device 10 that heats the cooking container 20 based on the identification result of the identification unit 607 and the elapsed time from the time when the object to be heated by the time measuring unit is charged. For example, after boiling for a certain period of time, heating can be stopped and heat retention can be performed. As a result, the thermal power control unit 608 can control the heating, such as stopping the heating at a specific time after the food material (for example, meat) is added.

(Second Embodiment)
The heating control system 1 according to the second embodiment is different from the heating control system 1 in the first embodiment in that the neural network used by the identification unit is changed depending on the shape of the cooking container 20. Hereinafter, the differences from the first embodiment will be described.

FIG. 9 is a block diagram showing a configuration of the heating control system 1 according to the second embodiment. As shown in FIG. 9, the heating control system 1 according to the present embodiment is further provided with the shape identification unit 612 of the heating control device 60.

The cooking container 20 has a plurality of different shapes such as a round shape and a square shape. The learning unit 604 learns the machine learning model used in the shape identification unit 612 for shape identification. In this case, the input data is image data taken from above the cooking container 20, and the learning data with the teacher category as the shape is used.

Further, the machine learning model for the identification unit 607 also learns by using the learning data 1 in which the round cooking container 20 is photographed and the learning data 2 in which the square cooking container 20 is photographed. That is, the identification unit 607 has a machine learning model 1 of the learning result 1 learned by the learning data 1 and a machine learning model 2 of the learning result 2 learned by the learning data 2.

In the convection of the round cooking container 20, convection occurs with the central portion of the circular pan as the convection center. On the other hand, the convection of the square-shaped cooking container 20 tends to shift the convection center to a position affected by the heating position. Therefore, depending on the shape of the cooking container 20, it is better to use the machine learning model 1 and the machine learning model 2 properly to improve the identification accuracy.

A control example of the thermal power control unit 608 according to the second embodiment will be described with reference to FIG. FIG. 10 is a flowchart showing a control example of the thermal power control unit 608 according to the second embodiment. As shown in FIG. 12, first, the acquisition unit 600 acquires an image from the photographing device 30 (step S300).

Next, the shape identification unit 612 converts the pixel data into 60 × 60 × 3 pixel data, inputs the data to the machine learning model used in the shape identification unit 612, identifies the state of the image, and determines whether or not the image has a round shape. Step S302).
Next, if the identification unit 607 has a round shape (YES in step S302), the machine learning model 1 of the learning result 1 is used, and if it has a square shape (NO in step S302), the machine learning of the learning result 2 Model 2 is used.

As described above, according to the present embodiment, the neural network used in the identification unit 607 is changed according to the shape of the cooking container 20. As a result, even if the state inside the container differs depending on the shape of the cooking container 20, the state inside the container can be identified with higher accuracy.

At least a part of the heating control device 60 and the heating control system 1 described in the above-described embodiment may be configured by hardware or software. In the case of software configuration, a program that realizes at least a part of the functions of the heating control device 60 and the heating control system 1 is stored in a recording medium such as a flexible disk or a CD-ROM, read by a computer, and executed. You may. The recording medium is not limited to a removable one such as a magnetic disk or an optical disk, and may be a fixed recording medium such as a hard disk or a memory.

Further, a program that realizes at least a part of the functions of the heating control device 60 and the heating control system 1 may be distributed via a communication line (including wireless communication) such as the Internet. Further, the program may be encrypted, modulated, compressed, and distributed via a wired line or wireless line such as the Internet, or stored in a recording medium.

Although some embodiments have been described above, these embodiments are presented only as examples and are not intended to limit the scope of the invention. The novel devices, methods and programs described herein can be implemented in a variety of other forms. In addition, various omissions, substitutions, and changes can be made to the forms of the apparatus, method, and program described in the present specification without departing from the gist of the invention.

1: Cooking system, 10: Heating device, 20: Cooking container, 30: Imaging device, 40: Display device, 60: Heating control device, 61: Heating state identification device, 62: Heating state identification system, 600: Acquisition unit, 603: Feature amount generation unit, 604: Learning unit, 606: Discriminator learning unit, 607: Identification unit, 608: Thermal power control unit, 612: Shape identification unit.

Claims (11)

An acquisition unit that acquires a plurality of time-series images of the inside of a container having a heated object and a liquid taken in chronological order.
Using the time-series image feature amount acquired based on the plurality of time-series images, an identification unit for identifying a plurality of heating states in the liquid, and an identification unit.
A heating state identification device comprising. A feature amount generation unit having a convolutional neural network that generates an image feature amount for each of the images of the plurality of time-series images and arranges them in a time series to generate the time-series image feature amount is further provided.
The identification unit uses a recurrent neural network that identifies the plurality of heating states based on the time-series image features generated by the convolutional neural network.
The heating state identification device according to claim 1. A learning unit that performs machine learning using a learning set including a correct label corresponding to the state of the object to be heated and the state of the liquid and a time-series image feature amount generated by the convolutional neural network is further provided.
The heated state identification device according to claim 1 or 2, wherein the identification unit identifies a state based on the result of the machine learning. The learning unit performs the machine learning using different learning sets according to the shape of the container.
The heating state identification device according to claim 3, wherein the identification unit identifies a state based on the result of the machine learning according to the shape of the container. A thermal power control unit that controls the thermal power of the heating device that heats the container based on the identification result of the identification unit of the heating state identification device according to claims 1 to 5.
A heating control device. The heating control device according to claim 5, wherein the thermal power control unit controls to change the time for maintaining the predetermined state according to a predetermined state before boiling of the liquid. When the identification unit identifies the state in which the object to be heated is charged regardless of the liquid state of the liquid, the thermal power control unit identifies the state in which the object to be heated is charged. The heating control device according to claim 5 or 6, wherein a display image prompting the disposal of the heated object in the container is displayed on the display device after a predetermined time has elapsed. An acquisition process for acquiring a plurality of time-series images obtained by taking time-series images of the inside of a container having a heated object and a liquid, and
An identification step of identifying a plurality of heated states in the liquid by using the time-series image feature amount obtained by acquiring the time-series image feature amount based on the plurality of time-series images.
A heating state identification method. A thermal power control step of controlling the thermal power of the heating device for heating the container based on the identification result of the identification step of the heating state identification method according to claim 8.
A heating control method. An imaging device that photographs the inside of the container from the top of the container that has the object to be heated and the liquid,
An acquisition unit that acquires a plurality of images captured by the imaging device in time series, and
Using the time-series image feature amount acquired based on the plurality of time-series images, an identification unit for identifying a plurality of heating states in the liquid, and an identification unit.
A heating state identification system. A thermal power control unit that controls the thermal power of the heating device that heats the container based on the identification result of the identification unit of the heating state identification system according to claim 10.
A heating control system.

Images