KR20230029444A - Platform apparatus for directly generating a cooking diary from cooking video obtained from a camer of a range hood or cooking device using deep learning technology - Google Patents

Abstract

The present invention relates to a platform device for generating a cooking diary from a cooking video using deep learning technology. According to the present invention, the platform device for generating a cooking diary comprises: at least one camera for photographing a cooking video in which a chef cooks while explaining a cooking process; a server for receiving the cooking video transmitted from the at least one camera and generating a cooking diary; and a cooking information DB for storing cooking-related raw materials, cooking methods, cooking utensils, cooking energy information, and text data or voice data related to cooking.

Description

Platform apparatus for directly generating a cooking diary from cooking video obtained from a camer of a range hood or cooking using deep learning technology device using deep learning technology}

The present invention relates to a platform device for generating a cooking diary from a cooking video using deep learning technology, and more specifically, to simply generating a cooking diary using a cooking video through voice recognition and deep learning technology. It relates to a platform device for generating a cooking diary from a cooking video using possible deep learning technology.

Generally, cooks or people who like to cook make and use a cooking diary. Since it is difficult to remember all the dishes, I have created and used a cooking diary that summarizes the cooking process and recipes.

In the past, cooking diaries have been produced in such a way that chefs or other cooking enthusiasts handwritten cooking processes and recipes disclosed on TV, SNS, or practice sites in notebooks, and take pictures and attach them to notebooks.

However, this conventional method has problems in that it takes a long time to make a cooking diary, it is difficult to write down instantaneously passing voices in a note, and it is difficult to capture an image, so the difficulty of production is high. In some cases, there was a problem that it was difficult to modify or add.

Accordingly, there has been a demand for the creation of a cooking diary capable of easily and simply creating a cooking diary and enabling easy modification or addition of characters or images in the cooking diary.

Republic of Korea Patent Publication No. 10-2021-0096956 (2021.08.06.)

Accordingly, an object of the present invention is to provide a platform device for generating a cooking diary from a cooking video using deep learning technology that can overcome the above conventional problems.

Another object of the present invention is to create a cooking diary from a cooking video using a deep learning technology that can easily create a cooking diary only by shooting a cooking video and easily modify or add text or images in the cooking diary. It is to provide a platform device for

According to the embodiment of the present invention for achieving some of the above technical problems, a platform device for creating a cooking diary according to the present invention includes at least one device for taking a cooking video in which a chef cooks while explaining a cooking process. with a camera; a server for receiving the cooking video transmitted from the at least one camera and generating a cooking diary; A cooking information DB for storing cooking-related raw materials, cooking methods, cooking utensils, cooking energy information, and text data or audio data related to cooking is provided, and the server includes a video receiving unit to receive a cooking video transmitted from the at least one camera. and; a video separator for dividing the received cooking video into audio data and image data by assigning a time label; Using STT (Speech to Text) module, the voice data is recognized and converted into text data, and at the same time, cooking steps divided into a plurality of steps are recognized, and text data for each step recognized within each cooking step is generated. , a voice data processing unit that provides to be combined with the bottom of the corresponding step-by-step representative image; For each cooking step recognized by the voice data processing unit, the image at the time of recognition of the corresponding cooking step is extracted and specified as a representative image for each stage, the extracted representative image for each stage is input to a predetermined location, and provided by the voice data processing unit An image data processing unit is provided to combine the character data of each step with the bottom of the representative image for each step.

The cooking video is filmed while having a chef pronounce a specific term for classifying each cooking step, and the audio data processing unit may recognize the time when the specific term is recognized as the start time of each cooking step.

When recognizing the voice data, the voice data processing unit compares terms with a low voice recognition rate with terms stored in the cooking information DB, selects and displays a term having the highest similarity, and displays the terms stored in the cooking information DB. If there is no similarity with a term or if it has a similarity lower than a certain level, it can be recognized as a new term and stored in the cooking information DB.

The voice data processing unit includes an STT conversion unit for recognizing and converting the voice data into text data using a speech to text (STT) module; a cooking step recognizing unit recognizing a cooking stage through a specific term recognized by the STT conversion unit; A step-by-step text data generation unit providing step-by-step text data recognized in each cooking step together with time label information at the time of recognition of each cooking step to the image data processing unit so as to be combined with the bottom of the step-by-step representative image. can

The image data processor includes: a representative image extractor for extracting and specifying an image at a time of recognition of a corresponding cooking step as a representative image for each step according to time label information provided by the audio data processor; an image data processing unit inputting the extracted representative image for each step to a predetermined location; a cooking information combining unit for generating a cooking step image by combining the step-by-step text data provided by the voice data processing unit with the bottom of the step-by-step representative image input at a predetermined location; A diary generating unit may be provided to generate a cooking diary for one dish by combining the step-by-step cooking step images according to the order of time labels.

According to the present invention, there is an advantage in that a cooking diary can be easily created only by an operation of photographing a cooking video. In addition, it has the advantage of being able to easily modify or add characters or images in the cooking diary.

1 is a schematic block diagram of a platform device for generating a cooking diary according to an embodiment of the present invention;
2 is a schematic block diagram of the server of FIG. 1;
3 is a schematic block diagram of the audio data processing unit of FIG. 2;
Figure 4 is a schematic block diagram of the image data processing unit of Figure 2,
5 is an operation flowchart of the platform device of FIG. 1;
6 to 9 show images of each cooking step.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, without any intention other than to provide a thorough understanding of the present invention to those skilled in the art to which the present invention pertains.

1 is a schematic block diagram of a platform device for generating a cooking diary according to an embodiment of the present invention, FIG. 2 is a schematic block diagram of a server in FIG. 1, and FIG. 3 is a schematic block diagram of a voice data processing unit in FIG. , and FIG. 4 is a schematic block diagram of the image data processing unit of FIG. 2 .

1 to 4, the platform device 500 for generating a cooking diary according to an embodiment of the present invention includes a server 100, at least one camera 200 and a cooking information DB 300 to provide

The at least one camera 200 is used to record a cooking video while the chef explains the cooking process step by step, and may be provided one or more times. When the at least one camera 200 is provided in plurality, it is respectively installed around the range hood and the cooking table in the kitchen, and the main camera and the sub camera take pictures while exchanging roles depending on the location along the cooking line. It is possible to shoot a single cooking video. The at least one camera 200 may be a general camera, but may include a camera mounted on a smartphone.

The at least one camera 200 may be installed at the lower part of the range hood, the head of the chef, or the cooking table, and one of them becomes the main camera by the chef's operation or the control of the server 100, and the others become the sub-cameras. It can be controlled by the camera.

The cooking information DB 300 stores cooking-related raw materials, cooking methods, cooking utensils, cooking energy information, and cooking-related text data or voice data.

The cooking information DB 300 generally includes all voice data that can be pronounced while cooking, such as cooking ingredients, cooking methods, cooking utensils, and cooking energy (crater) information that can be pronounced when a chef explains cooking. Information and text data information corresponding thereto are stored. In addition, the cooking information DB 300 may add data of e-books related to cooking, collect various cooking recipes published on the Internet and SNS in a scrolling manner, and classify the data into categories using a machine learning algorithm It is possible to save as

The server 100 receives the cooking video transmitted from the at least one camera 200 and processes it to create a cooking diary.

The server 100 includes a video receiver 110, a video separator 120, an audio data processor 130, and an image data processor 140.

The video receiver 110 receives the cooking video transmitted from the at least one camera 200 . The cooking video may refer to a video in which a cooking process is photographed while a chef explains the cooking step by step.

The video separator 120 separates the cooking video received through the video receiver 110 into audio data and image data by giving a time label.

The voice data processing unit 130 recognizes the voice data using a speech to text (STT) module and converts the voice data into text data. At the same time, cooking steps divided into a plurality of steps are recognized, and text data for each step recognized within each cooking step is provided to the image data processing unit 140 to be combined with the bottom of the corresponding step-by-step representative image.

The voice data processing unit 130 includes an STT conversion unit 132, a cooking step recognition unit 134, and a step-by-step text data generation unit 136.

The STT conversion unit 132 recognizes the voice data and converts it into text data using a speech to text (STT) module.

The STT conversion unit 132 can be pronounced while cooking, such as cooking ingredients, cooking methods, cooking utensils, and cooking energy (cooker) information that can be pronounced when a chef explains cooking through learning through deep learning. All possible voice data information and corresponding text data information are learned to recognize the voice of the voice data. In addition, when recognizing the voice data, a term having a low voice recognition rate is compared with terms stored in the cooking information DB 300 for similarity, and a term having the highest similarity is selected and displayed, and the cooking information DB 300 If there is no similarity with the term stored in , or if it has a similarity lower than a certain level, it is recognized as a new term and stored in the cooking information DB 300 according to a machine learning algorithm.

The cooking step recognition unit 134 recognizes the cooking step through a specific term recognized by the STT conversion unit. That is, the time point at which the specific term is recognized is recognized as the time point at which each cooking step starts. In order for the cooking step recognizing unit 134 to recognize the cooking step, the cooking video is filmed while having the chef pronounce specific terms for classifying each cooking step in the cooking preparation step and cooking process. do. For example, if a cook uses the words "prepare" or "I'm going to start cooking", it is possible to recognize this as a preparation step, and the cook may say "step 1, step 2...etc" or When words such as “first, second…, etc.” or “primary, second…, etc.” are converted into text in the STT conversion unit 132, each of the words is converted through the converted text. It is possible to recognize the cooking stages of Therefore, in the case of a sentence including a keyword such as 'first step' among text data converted to text (text), it can be judged as a first step cooking method.

The step-by-step text data generation unit 136 provides the image data processor 140 with step-by-step text data recognized in each cooking step to be combined with the bottom of the step-by-step representative image.

The step-by-step character data generation unit 136, for example, in the cooking preparation step, after the chef pronounces "I will start cooking", "The name of the dish is 'Shrimp Chanmuk', and the ingredients used in this dish (raw material) ) is described (pronounced) as 'shrimp, egg, pea, potato, carrot, lemon'", you can know the cooking preparation stage through the recognition of "I will start cooking", and the name and It is possible to generate character data in the cooking preparation step through the recognition of ingredients.

In general, in the first video of cooking ingredients, the chef arranges the ingredients necessary for cooking on the table, starts with the sentence "Ingredients used for this dish~", and explains each ingredient. It is extracted as the raw material of the dish and generated as character data for each stage of the cooking preparation step. The generated text data for each step is transmitted to the image data processing unit 140 together with time label information at the point at which the description of "the ingredients used for this dish" starts. In this case, if information corresponding to a recipe of a dish cooked by the chef is stored in the cooking information DB 300, voice recognition may be facilitated.

In addition, if the sentence included after "Step 1~" includes "Soak gelatin in water for 20 minutes, boil over low heat for 5 minutes to dissolve", this is recognized as an explanation of the cooking method in Step 1 and removed. It is created as character data for each step of step 1. At this time, the time of the time label where the character (text) of 'Step 1' is located is specified, and this time label information is transmitted to the image data processing unit together with the character data for each step or at different times. That is, the time label information may be transmitted first, and the text data for each step may be transmitted subsequently or simultaneously.

For each cooking step recognized by the voice data processor 130, the image data processing unit 140 extracts and specifies the image at the time of recognition of the corresponding cooking step as a representative image for each stage, and places the extracted representative image for each stage at a predetermined location. , and a cooking diary is created by combining the step-by-step text data provided by the voice data processing unit 130 with the bottom of the step-by-step representative image.

The image data processing unit 140 includes a representative image extraction unit 142, an image data processing unit 144, a cooking information combination unit 146, and a diary creation unit 148.

The representative image extractor 142 extracts the image at the time of recognition of the corresponding cooking step as the representative image for each step according to the time label information provided from the step-by-step text data generator 136 of the voice data processor 130. to be specific

For example, in the case of step 1 of cooking, the representative image extractor 142 corresponds to the time label received through the step-by-step character data generator 136 among the image data separated through the video separator 120. The video image at the point of time, that is, the point of pronunciation of "first step ~" is captured and extracted as a representative image of the first step dish.

In this way, it is possible to extract representative images for each stage.

The image data processing unit 144 inputs the step-by-step representative image extracted through the representative image extraction unit 142 to a predetermined location to create a cooking diary.

The cooking information combination unit 146 is at the bottom of the step-by-step representative image input to a predetermined position through the image data processing unit 144, in the step-by-step text data generation unit 136 of the voice data processing unit 130. The provided step-by-step character data is combined with the bottom of the step-by-step representative image to create a step-by-step cooking step image.

The diary creation unit 148 creates a cooking diary for one dish by combining the step-by-step cooking step images in the order of time labels.

Hereinafter, the operation of the platform device 500 for generating the cooking diary will be described with reference to FIG. 5 .

5 shows a schematic operation flow chart of the platform device, and FIGS. 6 to 9 show images of the step-by-step cooking steps.

As shown in FIG. 5, the platform device 500 for generating the cooking diary captures a cooking video using at least one camera 200 and transmits it to the server 100 to create the cooking diary. (S110).

The server 100 assigns a time label to the received cooking video and separates it into audio data and image data (S112).

The separated voice data is converted into text (text) through the STT conversion module of the voice data processing unit 130, and the cooking steps are recognized through specific terms pronounced by the chef, and the text for each step within each cooking step. Data is generated and provided to the image processing unit (S114).

On the other hand, the image data processor 140 extracts the image at the time of recognition of the corresponding cooking step for each cooking step recognized by the voice data processor 130 as a representative image for each step to generate a representative image for each step (S116).

The step-by-step representative image may be created similarly or identically to the step-by-step text data creation point, or may be created after the step-by-step text data is created. This depends on when the time label at the time of recognizing the corresponding cooking step is transmitted from the voice data processing unit 130. It may be created similarly or identically to the time of creation, and if the time label is transmitted together with the text data for each step, the representative image for each step will be created after the text data for each step is generated.

Thereafter, the image data processing unit 140 inputs the representative image for each stage to a predetermined position, and combines the text data for each stage provided by the voice data processing unit 130 with the bottom of the representative image for each stage to show FIGS. 6 to 9 As shown, a step-by-step cooking step image is generated (S118).

6 shows the cooking step image of the cooking preparation step, the name of the dish is written at the top, the representative image for each step of the cooking preparation step is input below it, and the character data for each step of the cooking preparation step is displayed at the bottom of the representative image for each step. It can be seen that they are connected.

FIG. 7 shows a cooking step image of the first step of cooking. In the upper left corner, the cooking step is displayed in numbers, a representative image for each step of the first step is located, and text data for each step of the first step is combined at the bottom. Figure 8 shows the cooking step image of the second step of cooking, the cooking step is displayed in numbers on the upper left, the representative image of the second step is located, and the text data of the second step is combined at the bottom.

9 shows an image of the cooking stage at the cooking end stage where cooking is finally completed, and the cook judges the cooking end point by voice recognition of a specific term (cooking, completion, etc.) saying “this dish is completed”, and the cooking end stage is recognized, and representative images for each stage of the cooking end stage are extracted by extracting the image data of the time label at the time when these characters are recognized, and by combining the representative images for each stage and the character data for each stage, Likewise, the cooking step image of the cooking end step is created.

Thereafter, the cooking diary for one dish is created by combining the cooking step images according to the order of time labels (S120).

The created cooking diary may change or add text or image data in each stage of cooking step image according to selection.

As described above, according to the present invention, there is an advantage in that a cooking diary can be easily created by simply taking a video while a chef cooks.

The description of the above embodiment is merely an example with reference to the drawings for a more thorough understanding of the present invention, and should not be construed as limiting the present invention. In addition, it will be apparent to those skilled in the art that various changes and modifications are possible within a range that does not deviate from the basic principles of the present invention.

100: server 110: video receiver
120: video separation unit 130: audio data processing unit
140: image data processing unit 200: camera
300: Cooking information DB

Claims (5)

at least one camera for recording a cooking video in which the chef explains the cooking process;
a server for receiving the cooking video transmitted from the at least one camera and generating a cooking diary;
A cooking information DB for storing cooking-related raw materials, cooking methods, cooking utensils, cooking energy information, and text data or voice data related to cooking,
The server,
a video receiver for receiving the cooking video transmitted from the at least one camera;
a video separator for dividing the received cooking video into audio data and image data by assigning a time label;
Using STT (Speech to Text) module, the voice data is recognized and converted into text data, and at the same time, cooking steps divided into a plurality of steps are recognized, and text data for each step recognized within each cooking step is generated. , a voice data processing unit that provides to be combined with the bottom of the corresponding step-by-step representative image;
For each cooking step recognized by the voice data processing unit, the image at the time of recognition of the corresponding cooking step is extracted and specified as a representative image for each stage, the extracted representative image for each stage is input to a predetermined location, and provided by the voice data processing unit A platform device for generating a cooking diary, characterized by comprising an image data processing unit for combining step-by-step text data with the bottom of the step-by-step representative image.
The method of claim 1,
The cooking video is filmed while the chef pronounces a specific term for classifying each cooking step, and the audio data processing unit recognizes the time when the specific term is recognized as the starting point of each cooking step. Characterized in that A platform device for creating a cooking diary.
The method of claim 1,
When recognizing the voice data, the voice data processing unit compares terms with a low voice recognition rate with terms stored in the cooking information DB, selects and displays a term having the highest similarity, and displays the terms stored in the cooking information DB. A platform device for creating a cooking diary, characterized in that when there is no similarity with a term or it has a low similarity above a certain level, it is recognized as a new term and stored in the cooking information DB.
The method of claim 2,
The voice data processing unit,
an STT conversion unit for recognizing the voice data and converting it into text data using a speech to text (STT) module;
a cooking step recognizing unit recognizing a cooking stage through a specific term recognized by the STT conversion unit;
Equipped with a step-by-step character data generation unit that provides step-by-step character data recognized in each cooking step together with time label information at the time of recognition of each cooking step to the image data processing unit so as to be combined with the bottom of the step-by-step representative image Platform device for generating a cooking diary characterized by.
The method of claim 4,
The image data processing unit,
a representative image extractor for extracting and specifying an image at the time of recognition of a corresponding cooking step as a representative image for each step according to the time label information provided by the voice data processor;
an image data processing unit inputting the extracted representative image for each step to a predetermined location;
a cooking information combining unit for generating a cooking step image by combining the step-by-step text data provided by the voice data processing unit with the bottom of the step-by-step representative image input at a predetermined location;
and a diary generator for generating a cooking diary for one dish by combining the step-by-step cooking step images according to the order of time labels.

Images