JP2021029677A - Placement state management device, placement state management method, and placement state management program - Google Patents

Abstract

To provide a placement state management device, a placement state management method, and a placement state management program capable of determining that a placement state of Sushi is a non-standard one.SOLUTION: A placement state management device includes: a storage unit for storing a first learning model that has learned a food placement state in advance on the basis of an image of food placed on a placement member; an imaging unit for imaging a placement member on which food is placed; an extraction unit for extracting a part of images of the placement member as a first image on the basis of the images captured by the imaging unit; a determination unit for determining the placement state of the food placed on the placement member on the basis of the first image extracted by the extraction unit and the first learning model; and an action unit for executing a predetermined action when the placement state of the food is determined to be a non-standard one by the determination unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to a mounting state management device, a mounting state management method, and a mounting state management program.

Traditionally, the food service industry has endeavored to provide users with cheaper, faster, and higher quality food products. For example, at a conveyor belt sushi restaurant, a device for gripping rice may be introduced, and efforts are being made to efficiently provide sushi such as sushi. By the way, for example, when putting the finished grip on the lane, the material may shift from the rice. In this case, in order to provide the user with higher quality food, it is necessary to inform the store employees that the grip of the sushi rice is out of the standard.
The technique described in Patent Document 1 images a dish flowing through the lane from above, removes the pattern described on the dish, and then recognizes the grip. Further, the technique described in Patent Document 1 measures the peripheral length of the recognized grip, and warns when the peripheral length is longer than the normal peripheral length (when the material shifts and falls). It is designed to output.

Japanese Unexamined Patent Publication No. 2008-17902

Generally, the circumference of the grip varies depending on the material. Therefore, since the technique described in Patent Document 1 does not recognize the type of grip, it may not be possible to measure the peripheral length for each type of grip (for each material). Further, although the technique described in Patent Document 1 removes the pattern described on the dish, there is a possibility that the pattern on the dish cannot be removed when the pattern is blurred or the like. In that case, it may not be possible to accurately measure the circumference of the grip. Further, in the technique described in Patent Document 1, it is necessary to register the pattern of the plate each time, and there is a possibility that the registration may take time and effort.

An object of the present invention is to provide a placement state management device, a placement state management method, and a placement state management program capable of determining that the placement state of food is out of the standard by a method different from the conventional method. ..

One aspect of the placement state management device is a storage unit that stores a first learning model in which the placement state of the food is learned in advance based on an image of the food placed on the placement member, and a storage unit on which the food is placed. An imaging unit that captures an image of the mounting member to be mounted, an extraction unit that extracts a part of the image of the mounting member as a first image based on the image captured by the imaging unit, and a first extraction unit extracted by the extraction unit. Based on the image and the first learning model, a determination unit that determines the placement state of the food to be placed on the placement member, and a determination unit that determines that the placement state of the food is out of the standard. It is provided with an operation unit that performs a predetermined operation.

In one aspect of the placement state management device, the operation unit has an output unit that outputs a warning when the determination unit determines that the food placement state is out of the standard, and the determination unit determines the food placement state as a reference. It may be at least one of the transport lanes that stops the transport of the mounting member when it is determined to be outside.

In one aspect of the mounting state management device, when the determination unit determines that the food loading state is out of the standard, the moving unit switches the transport destination of the mounting member transported by the transport lane from the transport lane. , Another transport lane connected to the transport lane, and a robot device that removes the mounting member transported by the transport lane from the transport lane when the determination unit determines that the food placement state is out of the standard. , Either of them may be used.

In one aspect of the mounting state management device, the storage unit stores a second learning model in which the shape of the mounting member is learned in advance based on the image of the mounting member, and the extraction unit is imaged by the imaging unit. The first image may be extracted based on the image and the second learning model.

In one aspect of the mounting state management device, the extraction unit recognizes the mounting member based on the image captured by the imaging unit and the second learning model, and has a predetermined range inscribed in the recognized mounting member. The image may be extracted as the first image.

In one aspect of the mounting state management device, the mounting member is a plate on which sushi as food is placed and whose planar shape is substantially circular, and the imaging unit transports the plate on which sushi is placed. The plate that moves with the movement of the lane is imaged from above with respect to the surface on which the sushi is placed, and the extraction unit is a square area inscribed in the plate as an image of a predetermined range inscribed in the mounting member. The image of may be extracted as the first image.

In one aspect of the placement state management device, the determination unit determines whether the sushi material placed on the plate is out of the sushi rice, the sushi is lying down, or the seaweed is peeling off from the sushi. , At least one may be determined as out of the reference of the mounted state.

In one aspect of the placement state management method, a computer including a storage unit that stores a first learning model in which the placement state of the food is learned in advance based on an image of the food placed on the placement member is used as the food. An imaging step in which the mounting member on which the food is placed is imaged by the imaging unit, an extraction step in which a part of the image of the mounting member is extracted as a first image based on the image captured in the imaging step, and an extraction step. Based on the first image extracted in step 1 and the first learning model, the determination step for determining the placement state of the food to be placed on the placement member and the determination step for determining the placement state of the food are out of the standard. If it is determined, the operation step of performing a predetermined operation and the operation step are executed.

One aspect of the placement state management program is a computer provided with a storage unit that stores a first learning model in which the placement state of the food is learned in advance based on an image of the food placed on the placement member. Is extracted by an imaging function that captures an image of the mounting member on which the device is placed, an extraction function that extracts a part of the image of the mounting member as a first image based on the image captured by the imaging function, and an extraction function. Based on the first image and the first learning model, the determination function for determining the placement state of the food to be placed on the placement member and the judgment function determine that the placement state of the food is out of the standard. If this is the case, an operation function that performs a predetermined operation and an operation function are realized.

In one aspect of the mounting state management device, the food to be mounted on the mounting member is placed on the mounting member based on the first image as a partial image of the mounting member on which the food is placed and the learning model. When the placement state is determined and the placement state of the food is determined to be non-standard, a predetermined operation is performed, so that it is possible to prevent the food having the non-standard placement state from being provided to the customer.
Further, the mounting state management method and the mounting state management program of one aspect can exert the same effect as the mounting state management device of the above-described aspect.

It is a figure for demonstrating the mounting state management system which concerns on one Embodiment. It is the first figure for demonstrating an example of the case where the food is placed on the placing member. It is a 2nd figure for demonstrating an example of the case where the food is placed on the placing member. (A) shows an example in which the food placement state is out of the standard, and (B) shows an example in which the food placement state is within the standard. It is a 3rd figure for demonstrating an example of the case where the food is placed on the placing member. (A) shows an example in which the food placement state is out of the standard, and (B) shows an example in which the food placement state is within the standard. It is a figure which shows an example of the aggregation result when it is judged that the sushi placement state is out of the standard. It is a flowchart for demonstrating the mounting state management method which concerns on one Embodiment.

Hereinafter, an embodiment of the present invention will be described.
FIG. 1 is a diagram for explaining the mounting state management system 1 according to the embodiment.

The mounting state management system 1 includes, for example, a mounting state management device 10 and a headquarters terminal 31 that receives information transmitted from the mounting state management device 10.

As an example, the mounting state management device 10 is arranged in a restaurant such as a conveyor belt sushi restaurant, a food factory, or the like. The placement state management device 10 determines the placement state of the food to be placed on the placement member. As a specific example, in the conveyor belt sushi restaurant, when the plate 21 as a mounting member is transported on the transport lane 20, the mounting state management device 10 places the sushi 22 to be mounted on the plate 21. Determine the state. The placement state management device 10 outputs a warning when the placement state of the sushi 22 is out of the standard. Further, when the placement state of the sushi 22 is out of the standard, the placement state management device 10 may transmit a report (information) of the placement state to the headquarters terminal 31.
In addition, one or a plurality of mounting state management devices 10 may be arranged in one restaurant or one food factory. In addition, there may be a plurality of restaurants and food factories.

The headquarters terminal 31 is arranged at the management headquarters 30 that manages restaurants, food factories, and the like. The headquarters terminal 31 is, for example, a terminal such as a personal computer. Based on the report transmitted from the restaurant, the food factory, etc., the headquarters terminal 31 has, for example, the food placement state when the food is provided at the restaurant, and the food placement member (for example,) at the food factory. , Tray, etc.) may be used to tabulate the status of whether or not the food is placed outside the standard. The headquarters terminal 31 may display the result of the aggregation on the terminal display unit (not shown) and store it in the terminal storage unit (not shown). Employees of the management headquarters 30 can educate employees of restaurants and food factories by checking the result of aggregation displayed on the terminal display unit of the terminal 31 of the headquarters.

Hereinafter, the mounting state management device 10 will be specifically described.
FIG. 2 is a first diagram for explaining an example in which food is placed on the mounting member.
FIG. 3 is a second diagram for explaining an example in which food is placed on the mounting member. FIG. 3A shows an example in which the food placement state is out of the standard, and FIG. 3B shows an example in which the food placement state is within the standard.
FIG. 4 is a third diagram for explaining an example in which food is placed on the placing member. FIG. 4A shows an example in which the food placement state is out of the standard, and FIG. 4B shows an example in which the food placement state is within the standard.

As shown in FIG. 1, the mounting state management device 10 includes an extraction unit 12, a determination unit 13, an imaging unit 14, a storage unit 15, an output unit 16, and a communication unit 17. The extraction unit 12 and the determination unit 13 are realized as one function of the control unit 11 (for example, an arithmetic processing unit or the like) of the mounting state management device 10.

The communication unit 17 can send and receive information to and from the headquarters terminal 31.

The storage unit 15 stores a first learning model in which the placement state of the food is learned in advance based on the image of the food placed on the placement member.
The mounting member is, for example, a plate on which food is placed. Further, the mounting member may be, for example, a tray, a container, a pack, a bowl, a bowl, a box, a lunch box, or the like on which food is placed. The planar shape of the mounting member may be, for example, a circle, a quadrangle, a polygon, or the like. When the planar shape of the mounting member is circular, the circular shape does not have to be a perfect circle, and may be a substantially circular shape such as an ellipse. Hereinafter, the case where the mounting member is a plate 21 having a circular (substantially circular) shape when viewed in a plan view on which the sushi 22 as food is placed, as illustrated in FIG. 2, will be described. The food may be various foods, but sushi 22 will be described below as an example. Here, the sushi 22 may be at least one of a grip, a warship, a sushi roll and a thin roll.

The food placement state is the state of the food placed on the placement member, and the food is placed on the placement member in the standard state, or the food is placed in the standard state. Indicates whether it is placed on a member. The food placement state is, for example, a state in which the sushi 22 is placed on the plate 21. Specifically, the food placement state is that the sushi 22 is placed on the plate 21 according to the standard (see FIGS. 3 (B) and 4 (B)), and the sushi 22 is based on the plate 21. It is not placed on the street (see FIGS. 3 (A) and 4 (A)). When the sushi 22 is not placed according to the standard, for example, the sushi material placed on the plate 21 is out of the sushi rice, the sushi is lying down, and the seaweed is peeling off from the sushi. And so on.
In addition, the food placement state may indicate whether or not the food is placed at a predetermined position (position within the reference range) of the placement member.

As an example of the first learning model, as shown in FIG. 3 (A), a state in which the sushi rice is deviated from the rice sushi rice (a state in which the sushi is placed outside the standard) and an example in FIG. 3 (B). As shown in the above, it is generated based on the result of learning the state in which the sushi rice is not deviated from the rice (the state in which the sushi is placed is within the standard). In addition, as shown in FIG. 4A, the first learning model includes a state in which the seaweed of the warship is peeled off, a state in which the warship is collapsed (a state in which the sushi is placed outside the standard), and a state in which the sushi is placed outside the standard. As shown in FIG. 4B as an example, it is generated based on the result of learning the state where the seaweed of the warship is not peeled off and the state where the warship is not collapsed (the state where the sushi is placed is within the standard). .. The first learning model is generated based on the result of learning various placement states of sushi in addition to the examples of FIGS. 3 and 4. The first learning model may be a model in which a plurality of learning results are integrated, or may be a model for each learning result.

The storage unit 15 stores a second learning model in which the shape of the mounting member is learned in advance based on the image of the mounting member. The shape of the mounting member is, for example, the flat shape of a plate used in a restaurant. As a specific example, the shape of the mounting member is the planar shape (circular (substantially circular)) of the plate 21 transported on the transport lane 20 at the conveyor belt sushi restaurant. Alternatively, the shape of the mounting member may be, for example, a flat shape such as a food tray, a container, a pack, a bowl, a bowl, a box, and a lunch box.

The image pickup unit 14 takes an image of a mounting member on which food is placed. For example, the imaging unit 14 images the transport lane 20 that transports the plate 21 on which the sushi 22 is placed from above. That is, the imaging unit 14 images the plate 21 that moves with the movement of the transport lane 20 from above with respect to the surface on which the sushi 22 is placed. The image pickup unit 14 is a camera or the like that captures a still image or a moving image. When the imaging unit 14 captures a still image, the imaging unit 14 captures the still image of the plate 21 when the plate 21 passes below the imaging unit 14. Alternatively, when the imaging unit 14 captures a moving image, the imaging unit 14 continuously images the transport lane 20 and images the moving image of the dish 21 passing below the imaging unit 14. The transport lane 20 transports the plate 21 on which the sushi 22 is placed, for example, in a conveyor belt sushi restaurant. The transport lane 20 may be a circular transport lane or a straight transport lane.

The imaging unit 14 may continuously image the lower part and send the captured image to the extraction unit 12 when the dish 21 is located at a specific coordinate in the image.
Alternatively, in order to image the dish 21 when the dish 21 passes below the imaging unit 14, a detection sensor (not shown) for detecting the dish 21 is arranged in the mounting state management device 10. You may leave it. In this case, the imaging unit 14 may image the dish 21 with a still image or a moving image based on the detection result of the detection sensor. That is, the image pickup unit 14 may take an image of the dish 21 when the detection sensor detects that the dish 21 passes below the image pickup section 14. Further, the image pickup unit 14 may take an image of the dish 21 passing under the image pickup unit 14 after a predetermined time (for example, several seconds to several tens of seconds later) after detecting the dish 21 by the detection sensor. Good. When the imaging unit 14 images the plate 21 after a predetermined time from the detection of the plate 21 by the detection sensor, for example, the detection sensor is arranged on the upstream side of the transport lane 20 with respect to the lower part of the imaging unit 14. It may or may not be arranged. The detection sensor may include, for example, an infrared light emitting unit and an infrared light receiving unit. For example, if the infrared rays emitted from the light emitting unit are shielded by the dish 21 and the infrared rays are not received by the light receiving unit, the detection sensor may detect the dish 21. The mounting state management device 10 can reduce the amount of acquired image data when a still image is captured by the imaging unit 14 as compared with the case where a moving image is captured.

The imaging unit 14 may not only image the dish 21 transported on the transport lane 20, but may also image the dish 21 placed on a table or the like. Further, the imaging unit 14 may not only image one dish 21 but also a plurality of dishes 21 at a time.
Further, in a conveyor belt sushi restaurant or the like, a custom-made product ordered by a customer (sushi 22 placed on a plate 21) may be placed on a table such as a tub and transported in a transport lane 20. Even in such a case, the imaging unit 14 can image a custom-made product on the table.

The extraction unit 12 extracts a part of the image of the mounting member (dish 21) as the first image based on the image captured by the image pickup unit 14. In this case, the extraction unit 12 extracts the first image based on the image captured by the imaging unit 14 and the second learning model. Specifically, the extraction unit 12 recognizes the mounting member (dish 21) based on the image captured by the imaging unit 14 and the second learning model, and the recognized mounting member (dish 21) is used. An inscribed image in a predetermined range is extracted as the first image. That is, the extraction unit 12 recognizes the planar shape of the dish 21 based on, for example, deep learning and machine learning. In this case, the extraction unit 12 can recognize the outer edge of the dish 21.

When a still image is captured by the imaging unit 14, the extraction unit 12 extracts the first image from the still image. When a moving image is captured by the imaging unit 14, the extraction unit 12 extracts a frame in which the entire dish 21 is captured from the moving image, and extracts a first image based on the frame and the second learning model.
The extraction unit 12 inscribes the plate 21 from the image even when the entire plate 21 is not captured in the image captured by the imaging unit 14 (when a part of the plate 21 is captured in the still image or the frame). If an image in a predetermined range (first image) can be extracted, the first image may be extracted from the image.
Further, when a plurality of dishes 21 are captured in an image (still image or frame) captured by the imaging unit 14, the extraction unit 12 recognizes each of the plurality of dishes 21 and corresponds to each of the plurality of dishes 21. The first image may be extracted.

The extraction unit 12 extracts an image of the quadrangular region S (see FIG. 2) inscribed in the dish 21 as a first image as an image of a predetermined range inscribed in the mounting member. That is, when the planar shape of the dish 21 is circular, the extraction unit 12 extracts the quadrangular region S inscribed in the outer edge of the dish 21 as the first image as an image of a predetermined range inscribed in the dish 21.
Alternatively, even when the planar shape of the dish 21 is circular, the extraction unit 12 may not only extract the quadrangular region S described above, but may also extract, for example, a polygonal region inscribed in the outer edge of the dish 21. Then, a circular region which is inside by a predetermined distance from the outer edge of the dish 21 may be extracted.
The mounting state management device 10 can record the sushi 22 in the first image by extracting the quadrangular region S inscribed in the outer edge of the plate 21, or compared with the case where the quadrangular region S is not extracted. The amount of data can be reduced.

Even when the planar shape of the plate 21 is quadrangular, the polygonal region or the circular region inscribed in the outer edge of the plate 21 may be extracted as the first image.
Further, when the mounting member is a food tray, container, pack, vessel, bowl, box, lunch box, etc., a partition or the like may be arranged inside. The extraction unit 12 may recognize the inside of the partition (the portion on which the food is placed) and extract the polygonal region or the circular region inscribed in the portion as the first image.

Further, when the custom-made product is imaged by the imaging unit 14, the extraction unit 12 can extract, for example, an image (first image) of the quadrangular region S inscribed in the dish 21. Here, in the case of a custom-made product, since the plate 21 is placed on the table, the size of the first image is larger than that of a general product (a non-custom-made product). Therefore, the mounting state management device 10 can determine whether it is a custom-made product or a general product based on the size of the first image.

Further, the extraction unit 12 inputs images continuously captured by the imaging unit 14, and when the entire dish 21 is captured in the image (when the dish 21 is located at a specific coordinate), the first image is displayed. It may be extracted.

The determination unit 13 determines the placement state of the food to be placed on the placement member based on the first image extracted by the extraction unit 12 and the first learning model. That is, the determination unit 13 determines whether the sushi 22 placed on the plate 21 is placed according to the standard (whether the sushi 22 is not placed according to the standard) based on, for example, the result of deep learning or machine learning. To judge.

Specifically, the determination unit 13 determines whether the material of the sushi 22 placed on the plate 21 is out of the sushi rice, the sushi 22 is lying down, or the seaweed is peeling off from the sushi 22. , At least one is judged as non-standard in the mounted state.
For example, the determination unit 13 determines what the sushi 22 placed on the plate 21 is based on the first image. As a specific example, the determination unit 13 determines the type of sushi 22 (grip, warship, thin roll, etc.) placed on the plate 21 based on deep learning, machine learning, and the like. Further, the determination unit 13 determines the placement state for each type of sushi 22 based on deep learning, machine learning, and the like. More specifically, when the sushi 22 placed on the plate 21 is a grip, the determination unit 13 determines whether or not the placement state of the grip is out of the standard. Similarly, when the sushi 22 placed on the plate 21 is a warship, the determination unit 13 determines whether or not the placement state of the warship is out of the standard. Similarly, when the sushi 22 placed on the plate 21 is a thin roll, the determination unit 13 determines whether or not the placement state of the thin roll is out of the standard.
The determination unit 13 may determine the position where the sushi 22 is placed on the plate 21 as the placement state of the food (sushi 22).
Further, the determination unit 13 determines the placement state of the sushi 22 placed on the plate 21 based on the first image and the first learning model in both the custom-made product and the general product.
The determination unit 13 is not limited to the above example, and may determine the placement state of various foods that are out of the standard in restaurants and food factories based on the learning model.

The operation unit may perform a predetermined operation when the determination unit 13 determines that the placement state of the sushi 22 is out of the standard. The operating unit may be at least one of the output unit 16 and the transport lane 20. When the moving unit is the transport lane 20, the moving unit may include a transport control unit (control unit 11).

The output unit 16 outputs a warning when the determination unit 13 determines that the placement state of the sushi 22 is out of the standard. The output unit 16 may be, for example, at least one of a speaker and a display unit (display device). When the output unit 16 is a speaker, the output unit 16 (speaker) outputs a voice or a warning sound indicating that the placement state of the sushi 22 placed on the plate 21 is out of the standard. When the output unit 16 is a display unit, the output unit 16 (display unit) displays a warning image indicating that the placement state of the sushi 22 placed on the plate 21 is out of the standard. When the warning is output by the output unit 16, the employee of the restaurant can take measures such as removing the plate 21 on which the nonstandard sushi 22 is placed from the transport lane 20.

Further, the transport lane 20 may stop the transport of the plate 21 when the determination unit 13 determines that the placement state of the sushi 22 is out of the standard. For example, the transport lane 20 includes a circuit transport lane and a straight transport lane. In particular, when the transport lane 20 is a straight transport lane, the transport lane 20 may stop the transport of the dish 21 on which the dish 21 is placed (do not transport the dish 21). In this case, the transport control unit (control unit 11) that controls the transport lane 20 may control the transport lane 20 to stop when the determination unit 13 determines that the placement state of the sushi 22 is out of the standard. Good.

Further, the above-mentioned operation unit may perform the following predetermined operation when the determination unit 13 determines that the placement state of the sushi 22 is out of the standard. The moving unit may be either another transport lane (separate lane) or a robot device (for example, a robot arm). In this case, the operation unit may include the control unit 11.
In the loading state management device 10, when another lane to the kitchen or the like is connected to the transport lane 20 where the customer can pick up the sushi 22, the sushi is transported on the transport lane 20 by the determination unit 13. When it is determined that the placement state of 22 is out of the standard, the lane switching device (not shown) is operated based on the control of the control unit 11, and the plate 21 on which the sushi 22 is placed is moved to another lane. You may let me.
Alternatively, when the mounting state management device 10 is equipped with a robot arm capable of removing the dishes 21 transported on the transport lane 20, the mounted state of the sushi 22 transported on the transport lane 20 is changed. If it is determined that the sushi is out of the standard, the robot arm may be operated under the control of the control unit 11 to remove the plate 21 on which the sushi 22 is placed from the transport lane 20.

FIG. 5 is a diagram showing an example of the aggregation result when the placement state of sushi is determined to be out of the standard.
When the determination unit 13 determines that the food placement state is out of the standard, the control unit 11 may aggregate the determination results. For example, as shown in FIG. 5 as an example, the control unit 11 may aggregate the number of foods for which the placement state is determined to be out of the standard for each day (day of the week). In addition, the control unit 11 may, for example, total the number of foods in which the food placement state is determined to be out of the standard for each time zone. Further, the control unit 11 may, for example, distinguish between a custom-made product and a general product for totaling.

For example, the control unit 11 transmits the aggregation result (report of the mounting state) to the headquarters terminal 31 by the communication unit 17 at predetermined intervals.
The headquarters terminal 31 displays, for example, the aggregation result transmitted from the restaurant or the like on the terminal display unit or the like. For example, when the number of times the sushi 22 is determined to be out of the standard for each restaurant is equal to or greater than the threshold value, the headquarters terminal 31 may display a warning or the like on the terminal display unit. Here, the headquarters terminal 31 may set different threshold values for the custom-made product and the general product. That is, for example, the headquarters terminal 31 displays a warning or the like on the terminal display unit when the number of times the custom-made product is determined to be out of the standard for each restaurant is equal to or greater than the first threshold value, and is a general product. A warning or the like may be displayed on the terminal display unit when the number of times the placement state is determined to be out of the standard is equal to or greater than the second threshold value (second threshold value> first threshold value).
By checking the display of the warning etc. on the terminal display unit, the employees of the management headquarters 30 can refer to the employees of the restaurant where the number of times the food placement state is determined to be out of the standard is equal to or greater than the threshold value. You can educate.

The mounting state management device 10 transmits the determination result by the determination unit 13 (for example, the determination result that the placement state of the sushi 22 is out of the standard) to the headquarters terminal 31 via the communication unit 17. May be good.
The headquarters terminal 31 may aggregate the determination results transmitted from the mounting state management device 10 for each store. The headquarters terminal 31 displays the result of the aggregation on the terminal display unit or the like, and displays a warning or the like on the terminal display unit when the number of times the sushi 22 placement state of each restaurant is determined to be out of the standard is equal to or greater than the threshold value. You may do it.

Next, the mounting state management method according to the embodiment will be described.
FIG. 6 is a flowchart for explaining the mounting state management method according to the embodiment.

In step ST11, the imaging unit 14 images the mounting member on which the food is placed. Specifically, the imaging unit 14 images the plate 21 on which the sushi 22 is placed, which moves on the transport lane 20, from above.

In step ST12, the extraction unit 12 extracts the first image based on the image captured in step ST11 and the second learning model. Specifically, the extraction unit 12 recognizes the dish 21 based on the image captured in step ST11 and the second learning model, and the first image is an image of the quadrangular region S inscribed in the recognized dish 21. Extract as. For example, when a moving image is captured in step ST11, the extraction unit 12 extracts a frame in which the entire plate 21 is captured from the moving image, and extracts a first image based on the frame and the second learning model. ..

In step ST13, the determination unit 13 determines the placement state of the sushi 22 placed on the plate 21 based on the first image extracted in step ST12 and the first learning model. Specifically, the determination unit 13 determines whether the material of the sushi 22 placed on the plate 21 is out of the sushi rice, the sushi 22 is lying down, or the seaweed is peeling off from the sushi 22. , At least one is judged as non-standard in the mounted state. If the mounting state is out of the standard (Yes), the process proceeds to step ST14. If the mounting state is not out of the standard (No), the process ends.

In step ST14, the output unit 16 outputs a warning. The output unit 16 may be, for example, at least one of a speaker and a display unit.
Further, in step ST14, the transport lane 20 may stop the transport operation of the dish 21 on which the dish 21 is placed (do not transport the dish 21).

In step ST15, the control unit 11 transmits information to the management headquarters 30 that the placement state of the sushi 22 is out of the standard. The control unit 11 may transmit to the management headquarters 30 the result of totaling the number of times that the placement state of the sushi 22 is determined to be out of the standard for each day (day of the week) and for each time zone.

The headquarters terminal 31 can display the aggregation result transmitted in step ST15 on the terminal display unit or the like. For example, when the number of times that the food placement state of each restaurant is determined to be out of the standard is equal to or greater than the threshold value, the headquarters terminal 31 may display a warning or the like on the terminal display unit. Employees and the like of the management headquarters 30 can educate the employees and the like of restaurants by confirming the display of warnings and the like on the terminal display unit.

Next, the effect of this embodiment will be described.
The mounting state management device 10 includes an extraction unit 12 that extracts a part of the image of the dish 21 as a first image based on the image captured by the imaging unit 14, and a first image extracted by the extraction unit 12. , When the determination unit 13 for determining the placement state of the sushi 22 placed on the plate 21 and the determination unit 13 determine that the placement state of the sushi 22 is out of the standard based on the first learning model. , An output unit 16 for outputting a warning.
As a result, the placement state management device 10 can determine that the placement state of the sushi 22 is out of the standard. Further, the placement state management device 10 outputs a warning when the placement state of the sushi 22 is out of the standard, so that the employee of the restaurant can be notified. Further, the placement state management device 10 prevents the non-standard sushi 22 from being provided to the customer by removing the non-standard sushi 22 (dish 21) from the transport lane 20 by the employee. It is possible to improve the quality of the products offered at the conveyor belt sushi restaurant.

In the mounting state management device 10, the extraction unit 12 may extract the first image based on the image captured by the image pickup unit 14 and the second learning model.
As a result, the placement state management device 10 can recognize the dish 21 based on, for example, deep learning and machine learning. Further, the mounting state management device 10 can extract the first image based on the recognized dish 21.

In the mounting state management device 10, the extraction unit 12 recognizes the dish 21 based on the image captured by the image pickup unit 14 and the second learning model, and obtains an image in a predetermined range inscribed in the recognized dish 21. It may be extracted as a first image.
Since the placement state management device 10 extracts an image (first image) in a predetermined range inscribed in the plate 21, only the portion on which the sushi 22 is placed can be extracted. Therefore, the mounting state management device 10 suppresses the occurrence of erroneous determination in the mounting state of the sushi 22 because the first image does not include an image of a portion unnecessary for determining the mounting state of the sushi 22. be able to. Further, since the mounting state management device 10 extracts the first image, the amount of data can be reduced as compared with the image generated by the imaging unit 14 (the image before extracting the first image), and the determination unit can be used. The amount of processing such as 13 can be reduced.

In the mounting state management device 10, the mounting member may be a plate 21 having a circular planar shape on which sushi 22 as food is placed. Sushi 22 may be at least one of a grip, a warship, a sushi and a thin roll. The imaging unit 14 images the plate 21 moving on the transport lane 20 from above with respect to the surface on which the sushi 22 is placed, and the extraction unit 12 has a predetermined range inscribed in the mounting member. The image of the quadrangular region S inscribed in the dish 21 as an image may be extracted as the first image.
The loading state management device 10 can determine the loading state of the sushi 22 placed on the plate 21 transported through the transport lane 20 of the restaurant. In addition, the placement state management device 10 can determine the placement state according to the type of sushi 22.

In the placement state management device 10, the determination unit 13 determines that the material of the sushi 22 placed on the plate 21 is out of the sushi rice, the sushi 22 is lying down, and the seaweed is peeled off from the sushi 22. Of the cases, at least one may be determined as out of the reference of the mounted state.
The placement state management device 10 can determine that the placement state of the sushi 22 placed on the plate 21 is out of the standard. That is, the placement state management device 10 can suppress the provision of sushi 22 whose placement state is out of the standard to the customer, and can improve the quality of the sushi 22 provided at the restaurant.

In the mounting state management method, the computer extracts a part of the image of the dish 21 as the first image based on the image captured by the imaging unit 14, and the first image extracted in the extraction step. , A determination step for determining the placement state of the sushi 22 placed on the plate 21 based on the first learning model, and a warning when the determination step determines that the placement state of the sushi 22 is out of the standard. Is output from the output unit 16 and the output step is executed.
As a result, the placement state management method can determine that the placement state of the sushi 22 is out of the standard. Further, the placement state management method outputs a warning when the placement state of the sushi 22 is out of the standard, so that the employee of the restaurant can be notified. In addition, the placement state management method can prevent the non-standard sushi 22 from being provided to the customer by removing the non-standard sushi 22 (dish 21) from the transport lane 20 by the employee. It is possible to improve the quality of the products offered at the conveyor belt sushi restaurant.

The placement state management program extracts a part of the image of the plate 21 as the first image based on the imaging function of imaging the plate 21 on which the sushi 22 is placed and the image captured by the imaging function on the computer. Based on the extraction function, the first image extracted by the extraction function, and the first learning model, a determination function for determining the placement state of the sushi 22 placed on the plate 21, and a determination function for sushi. When it is determined that the mounting state of 22 is out of the standard, an output function for outputting a warning is executed.
As a result, the placement state management program can determine that the placement state of the sushi 22 is out of the standard. In addition, the placement state management program outputs a warning when the placement state of the sushi 22 is out of the standard, so that the employee of the restaurant can be notified. In addition, the placement state management program can prevent the non-standard sushi 22 from being provided to the customer by removing the non-standard sushi 22 (dish 21) from the transport lane 20 by the employee. It is possible to improve the quality of products offered at conveyor belt sushi restaurants.

Each part of the mounting state management device 10 described above may be realized as a function of a computer arithmetic processing unit or the like. That is, the extraction unit 12 and the determination unit 13 of the mounting state management device 10 may be realized as extraction functions and determination functions by a computer arithmetic processing unit or the like, respectively.
The on-board state management program can realize each of the above-mentioned functions on a computer. The mounting state management program may be recorded on a non-temporary recording medium that can be read by a computer, such as an external memory or an optical disk.
Further, as described above, each part of the mounting state management device 10 may be realized by a computer arithmetic processing device or the like. The arithmetic processing unit or the like is composed of, for example, an integrated circuit or the like. Therefore, each part of the mounting state management device 10 may be realized as a circuit constituting an arithmetic processing unit or the like. That is, the extraction unit 12 and the determination unit 13 of the mounting state management device 10 may be realized as an extraction circuit and a determination circuit that constitute an arithmetic processing unit or the like of a computer.
The imaging unit 14, the storage unit 15, the output unit 16, and the communication unit 17 of the mounting state management device 10 may be realized as an imaging function unit, a storage function, an output function, and a communication function. Further, the imaging unit 14, the storage unit 15, the output unit 16, and the communication unit 17 of the mounting state management device 10 are configured as, for example, an integrated circuit or the like, and thus serve as an imaging circuit, a storage circuit, an output circuit, and a communication circuit. It may be realized. Further, the image pickup unit 14, the storage unit 15, the output unit 16, and the communication unit 17 of the mounting state management device 10 are configured as, for example, a plurality of devices as an image pickup device, a storage device, an output device, and a communication device. It may be configured. Further, the operating unit may be realized as an operating function or an operating device. Since the operating unit is at least one of the output unit 16 and the transport lane 20 (or another lane), the operating function (operating device) may be at least one of the output function (output device) and the transport function (transport device). Good. Further, the operation function may be a robot function. Further, the operation function (operation device) may include a control function (operation device) as a function of the control unit 11.

1 Mounted state management system 10 Mounted state management device 12 Extraction unit 13 Judgment unit 14 Imaging unit 15 Storage unit 16 Output unit 20 Transport lane 30 Management headquarters 31 Headquarters terminal

Claims (9)

A storage unit that stores a first learning model in which the food placement state is learned in advance based on an image of the food placed on the placement member, and a storage unit.
An imaging unit that captures images of the mounting members on which food is placed,
An extraction unit that extracts a part of the image of the above-mentioned placing member as a first image based on the image captured by the imaging unit, and an extraction unit.
Based on the first image extracted by the extraction unit and the first learning model, a determination unit for determining a placement state of food to be placed on the above-mentioned placement member, and a determination unit.
When the determination unit determines that the food placement state is out of the standard, the operation unit that performs a predetermined operation and the operation unit that performs a predetermined operation.
A mounting state management device including. The moving part
An output unit that outputs a warning when the determination unit determines that the food placement state is out of the standard, and
The mounting state management device according to claim 1, which is at least one of the transport lanes for stopping the transport of the mounting member when the determination unit determines that the loading state of the food is out of the standard. The moving part
When the determination unit determines that the food placement state is out of the standard, another transport lane connected to the transport lane, in which the transport destination of the mounting member transported by the transport lane is switched from the transport lane, as well as,
The first aspect of claim 1, wherein when the determination unit determines that the food placement state is out of the standard, the robot device removes the mounting member transported by the transport lane from the transport lane. Mounted state management device. The storage unit stores a second learning model in which the shape of the pre-described member is pre-learned based on the image of the pre-described member.
The mounting state management device according to any one of claims 1 to 3, wherein the extraction unit extracts the first image based on the image captured by the imaging unit and the second learning model. .. The extraction unit recognizes the preambled member based on the image captured by the imaging unit and the second learning model, and obtains an image of a predetermined range inscribed in the recognized predicated member. The mounting state management device according to claim 4, which is extracted as an image. The above-mentioned placing member is a plate having a substantially circular planar shape on which sushi as food is placed.
The imaging unit takes an image of the plate that moves with the movement of the transport lane that transports the plate on which the sushi is placed from above with respect to the surface on which the sushi is placed.
The mounting state management device according to claim 5, wherein the extraction unit extracts an image of a quadrangular region inscribed in the dish as an image of the predetermined range inscribed in the above-mentioned placing member as the first image. The determination unit is in a state where at least one of the cases where the sushi material placed on the plate is misaligned from the sushi rice, the sushi is collapsed, and the seaweed is peeled off from the sushi is placed. The mounting state management device according to claim 6, which is determined to be out of the standard. A computer provided with a storage unit that stores a first learning model in which the food placement state is learned in advance based on an image of the food placed on the placement member.
An imaging step in which an imaging unit captures an image of a mounting member on which food is placed,
An extraction step of extracting a part of the image of the above-mentioned placing member as a first image based on the image captured in the imaging step,
Based on the first image extracted in the extraction step and the first learning model, a determination step for determining a placement state of food to be placed on the above-mentioned placement member, and a determination step.
When the food placement state is determined to be out of the standard in the determination step, an operation step for performing a predetermined operation and an operation step for performing a predetermined operation,
Placement state management method to execute. A computer provided with a storage unit that stores a first learning model in which the food placement state is learned in advance based on an image of the food placed on the placement member.
An imaging function that captures images of the mounting members on which food is placed,
An extraction function that extracts a part of the image of the above-mentioned placing member as a first image based on the image captured by the imaging function, and
Based on the first image extracted by the extraction function and the first learning model, a determination function for determining a placement state of food to be placed on the above-mentioned placement member, and a determination function.
When the food placement state is determined to be out of the standard by the determination function, an operation function that performs a predetermined operation and an operation function that performs a predetermined operation,
A mounting state management program that realizes.

Images