JP7401995B2 - Placement status management device, placement status management method, and placement status management program - Google Patents

Description

The present invention relates to a placement state management device, a placement state management method, and a placement state management program.

Conventionally, the restaurant industry has been making efforts to provide cheaper, faster, and higher quality food to users. For example, in some conveyor belt sushi restaurants, devices for making sushi rice are sometimes introduced, and efforts are being made to efficiently serve sushi such as sushi rice. By the way, for example, when putting the finished nigiri on the lane, the topping may deviate from the shari. In this case, in order to provide higher quality food to users, it is necessary to notify store employees that a nigiri with a topping that deviates from the rice is out of the standard.
The technology described in Patent Document 1 images a plate flowing in a lane from above, removes a pattern written on the plate, and then recognizes a grip. Furthermore, the technology described in Patent Document 1 measures the circumference of the recognized grip, and issues a warning if the circumference is longer than the normal circumference (if the material is slipping or falling over). It is designed to be output.

Japanese Patent Application Publication No. 2008-17902

Generally, the circumference of a nigiri varies depending on the topping. For this reason, the technique described in Patent Document 1 does not recognize the type of grip, so it may not be possible to measure the circumference for each type of grip (for each topping). Further, the technique described in Patent Document 1 removes the pattern written on the plate, but if the pattern has scratches or the like, there is a possibility that the pattern on the plate cannot be removed. In that case, there is a possibility that the circumference of the grip cannot be accurately measured. Further, the technique described in Patent Document 1 requires registering the pattern of the plate each time, and the registration may take time and effort.

An object of the present invention is to provide a placement condition management device, a placement condition management method, and a placement condition management program that can determine that the placement condition of food is outside the standard using a method different from conventional methods. .

One aspect of the placement state management device includes a storage unit that stores a first learning model that has learned the placement state of the food in advance based on an image of the food placed on the placement member; an imaging section that captures an image of the mounting member to be placed; an extraction section that extracts a part of the image of the mounting member as a first image based on the image captured by the imaging section; and a first image extracted by the extraction section. a determination unit that determines the placement state of the food placed on the placement member based on the image and the first learning model; and when the determination unit determines that the placement state of the food is outside the standard; An operating section that performs a predetermined operation.

In one aspect of the placement state management device, the operation unit includes an output unit that outputs a warning when the determination unit determines that the food placement status is out of the standard; It may be at least one of the transport lanes that stop transporting the mounting member when it is determined that the mounting member is outside the transport lane.

In one embodiment of the placement state management device, when the determination unit determines that the placement state of the food is outside the standard, the operation unit switches the destination of the placement member transported by the transport lane from that transport lane. , another conveyance lane connected to the conveyance lane, and a robot device that removes the placement member conveyed by the conveyance lane from the conveyance lane when the determination unit determines that the food placement state is out of standard. It may be either one of the following.

In one embodiment of the placement state management device, the storage unit stores a second learning model that has learned the shape of the placement member in advance based on the image of the placement member, and the extraction unit stores the second learning model that has learned the shape of the placement member in advance based on the image of the placement member, and the extraction unit stores the second learning model that has learned the shape of the placement member in advance based on the image of the placement member. The first image may be extracted based on the image and the second learning model.

In one embodiment of the placement state management device, the extraction unit recognizes the placement member based on the image captured by the imaging unit and the second learning model, and extracts a predetermined range inscribed in the recognized placement member. The image may be extracted as the first image.

In one embodiment of the placement state management device, the placement member is a plate having a substantially circular planar shape on which sushi as food is placed, and the imaging unit is configured to transport the plate on which the sushi is placed. A plate that moves as the lane moves is imaged from above with respect to the surface on which sushi is placed on the plate, and the extraction unit extracts a rectangular area inscribed in the plate as an image of a predetermined range inscribed in the placement member. The image may be extracted as the first image.

In one aspect of the placement state management device, the determination unit determines whether the sushi toppings placed on the plate are misaligned with the sushi rice, the sushi has fallen, or the seaweed has peeled off from the sushi. , at least one may be determined to be outside the criteria for the placement state.

In one aspect of the placement state management method, a computer includes a storage unit that stores a first learning model that has learned the placement state of the food in advance based on an image of the food placed on the placement member. an imaging step of causing an imaging unit to image the mounting member on which the mounting member is placed, an extraction step of extracting a part of the image of the mounting member as a first image based on the image captured in the imaging step, and an extraction step a determination step of determining the placement condition of the food placed on the placement member based on the first image extracted in and the first learning model; and a determination step in which the placement condition of the food is outside the standard. If it is determined that this is the case, an operation step of performing a predetermined operation is executed.

In one aspect, the placement state management program includes a storage unit that stores a first learning model that has learned the placement state of the food in advance based on an image of the food placed on the placement member. an image capture function that captures an image of the mounting member on which is placed; an extraction function that extracts a part of the image of the mounting member as a first image based on the image captured by the imaging function; a determination function that determines the placement state of the food placed on the placement member based on the first image and the first learning model; In this case, an operation function for performing a predetermined operation is realized.

A placement state management device according to one aspect of the present invention includes a placement state management device for placing food on a placement member based on a first image as an image of a part of the placement member on which the food is placed and a learning model. If the placement condition of the food is determined and it is determined that the placement condition of the food is outside the standard, a predetermined operation is performed, so that it is possible to prevent food whose placement condition is out of the standard from being provided to the customer.
Further, the placement state management method and the placement state management program of one embodiment can achieve the same effects as the above-described placement state management device of one embodiment.

FIG. 1 is a diagram for explaining a placement state management system according to an embodiment. FIG. 3 is a first diagram for explaining an example of a case where food is placed on a placing member. FIG. 7 is a second diagram for explaining an example of a case where food is placed on the placing member. (A) shows an example in which the food placement condition is outside the standard, and (B) shows an example in which the food placement condition is within the standard. FIG. 7 is a third diagram for explaining an example of a case where food is placed on a placing member. (A) shows an example in which the food placement condition is outside the standard, and (B) shows an example in which the food placement condition is within the standard. FIG. 7 is a diagram illustrating an example of a tally result when it is determined that the placement state of sushi is outside the standard. 3 is a flowchart for explaining a placement state management method according to an embodiment.

An embodiment of the present invention will be described below.
FIG. 1 is a diagram for explaining a placement state management system 1 according to an embodiment.

The placement state management system 1 includes, for example, a placement state management device 10 and a headquarters terminal 31 that receives information transmitted from the placement state management device 10.

The placement state management device 10 is placed, for example, in a restaurant such as a conveyor belt sushi restaurant, a food factory, or the like. The placement state management device 10 determines the placement state of food placed on the placement member. As a specific example, when a plate 21 as a mounting member is transported on a conveyance lane 20 in a conveyor belt sushi restaurant, the placement state management device 10 controls the placement of sushi 22 placed on the plate 21. Determine the condition. The placement state management device 10 outputs a warning when the placement state of the sushi 22 is outside the standard. Further, the placement state management device 10 may transmit a report (information) of the placement state to the headquarters terminal 31 when the placement state of the sushi 22 is outside the standard.
Note that one or more placement state management devices 10 may be placed in one restaurant or one food factory. Furthermore, there may be a plurality of restaurants and food factories.

The headquarters terminal 31 is located at a management headquarters 30 that manages restaurants, food factories, and the like. The headquarters terminal 31 is, for example, a terminal such as a personal computer. Based on reports sent from restaurants, food factories, etc., the headquarters terminal 31 determines, for example, the placement state of food when it is provided at a restaurant, and the placement of the food on a placement member (e.g. , trays, etc.), the status of whether the food is placed outside the standard may be counted. The headquarters terminal 31 may display the results of the tally on a terminal display section (not shown) and store them in a terminal storage section (not shown). Employees of the management headquarters 30 can provide education to employees of restaurants and food factories by checking the tally results displayed on the terminal display section of the headquarters terminal 31.

Hereinafter, the placement state management device 10 will be specifically explained.
FIG. 2 is a first diagram for explaining an example of a case where food is placed on a placing member.
FIG. 3 is a second diagram for explaining an example of a case where food is placed on the placing member. FIG. 3(A) shows an example in which the food placement condition is outside the standard, and FIG. 3(B) shows an example in which the food placement condition is within the standard.
FIG. 4 is a third diagram for explaining an example in which food is placed on the placing member. FIG. 4(A) shows an example in which the food placement condition is outside the standard, and FIG. 4(B) shows an example in which the food placement condition is within the standard.

As shown in FIG. 1, the placement state management device 10 includes an extraction section 12, a determination section 13, an imaging section 14, a storage section 15, an output section 16, and a communication section 17. The extraction unit 12 and the determination unit 13 are realized as a function of the control unit 11 (for example, an arithmetic processing unit, etc.) of the placement state management device 10.

The communication unit 17 is capable of transmitting and receiving information to and from the headquarters terminal 31 .

The storage unit 15 stores a first learning model that has previously learned the placement state of the food based on the image of the food placed on the placement member.
The placement member is, for example, a plate on which food is placed. Further, the placing member may be, for example, a tray, a container, a pack, a bowl, a bowl, a box, a lunch box, etc. on which the food is placed. The planar shape of the mounting member may be, for example, circular, square, polygonal, or the like. Note that when the planar shape of the mounting member is circular, the circular shape does not have to be a perfect circle, and may be a substantially circular shape such as an ellipse. In the following, a case will be described in which the mounting member is a plate 21 having a circular (substantially circular) shape when viewed from above, on which sushi 22 as food is placed, as illustrated in FIG. Furthermore, although the food may be various kinds of food, sushi 22 will be explained below as an example. Here, the sushi 22 may be at least one of sushi, gunkan, tsutsumi, and hosomaki.

The food placement condition is the condition of the food placed on the placement member, and whether the food is placed on the placement member in the standard condition or the food is placed in the standard condition. Indicates whether it is placed on a member. The state in which the food is placed is, for example, a state in which sushi 22 is placed on the plate 21. Specifically, the state in which the food is placed is that the sushi 22 is placed on the plate 21 according to the standard (see FIGS. 3(B) and 4(B)), and the sushi 22 is placed on the plate 21 according to the standard. It is in a state where it is not placed on the street (see FIGS. 3(A) and 4(A)). Conditions in which the sushi 22 is not placed according to the standard include, for example, a condition in which the toppings of the sushi placed on the plate 21 are misaligned with the sushi rice, a condition in which the sushi is fallen down, and a condition in which the seaweed is peeled off from the sushi. etc.
Furthermore, the placement state of the food may indicate whether the food is placed at a predetermined position (position within a reference range) on the placement member.

The first learning model is used for the state in which the topping of the nigiri is off the rice (the state in which the sushi is placed is out of the standard), as shown in FIG. 3(A), and the state shown in FIG. 3(B) as an example. As shown, it is generated based on the result of learning the state in which the toppings of the nigiri do not deviate from the rice (the state in which the sushi is placed is within the standard). In addition, as an example shown in FIG. 4(A), the first learning model is based on a state in which the seaweed on a warship is peeling off, a state in which the warship is fallen down (a state in which the state in which the sushi is placed is outside the standard), and As shown in an example in Figure 4 (B), it is generated based on the results of learning the state in which the seaweed on the warship has not peeled off and the state in which the warship has not fallen down (the state in which the sushi is placed is within the standard). . The first learning model is generated based on the results of learning various placement states of sushi in addition to the examples shown in FIGS. 3 and 4. The first learning model may be a model that integrates a plurality of learning results, or may be a model for each learning result.

The storage unit 15 stores a second learning model in which the shape of the placement member is learned in advance based on the image of the placement member. The shape of the mounting member is, for example, the planar shape of a plate used in a restaurant. As a specific example, the shape of the mounting member is the planar shape (circular (substantially circular)) of the plate 21 that is conveyed on the conveyance lane 20 in a conveyor belt sushi restaurant. Alternatively, the shape of the mounting member may be, for example, a planar shape such as a food tray, container, pack, bowl, bowl, box, or lunch box.

The imaging unit 14 images the placement member on which food is placed. For example, the imaging unit 14 images the conveyance lane 20 that conveys the plate 21 on which the sushi 22 is placed from above. That is, the imaging unit 14 images the plate 21 that moves along with the movement of the transport lane 20 from above with respect to the surface on which the sushi 22 is placed on the plate 21 . The imaging unit 14 is a camera or the like that captures still images or moving images. When the imaging unit 14 captures a still image, the imaging unit 14 captures a still image of the plate 21 when the plate 21 passes below the imaging unit 14 . Alternatively, when the imaging unit 14 images a moving image, the imaging unit 14 continuously images the conveyance lane 20 and images the moving image of the plate 21 passing below the imaging unit 14 . The transport lane 20 transports a plate 21 on which sushi 22 is placed, for example, in a conveyor belt sushi restaurant. The conveyance lane 20 may be a circular conveyance lane or a straight conveyance lane.

Note that the imaging unit 14 may continuously capture images of the lower part and send the captured image to the extraction unit 12 when the plate 21 is located at a specific coordinate within the image.
Alternatively, in order to capture an image of the plate 21 when the plate 21 passes below the imaging unit 14, the placement state management device 10 is provided with a detection sensor (not shown) for detecting the plate 21. You can leave it there. In this case, the imaging unit 14 may take a still image or a moving image of the plate 21 based on the detection result of the detection sensor. That is, the imaging unit 14 may image the plate 21 when its detection sensor detects that the plate 21 passes below the imaging unit 14 . The imaging unit 14 may also capture an image of the plate 21 passing below the imaging unit 14 after a predetermined period of time (for example, several seconds to several tens of seconds) after the detection sensor detects the plate 21. good. Note that when the imaging unit 14 images the plate 21 after a predetermined period of time after the detection sensor detects the plate 21, the detection sensor is arranged on the upstream side of the conveyance lane 20 with respect to the lower part of the imaging unit 14, for example. It may be placed or may not be placed. The detection sensor may include, for example, an infrared light emitting section and an infrared light receiving section. For example, the detection sensor may detect the dish 21 when the infrared rays emitted from the light emitting part are blocked by the dish 21 and the infrared rays are not received by the light receiving part. Note that when the imaging unit 14 captures a still image, the placement state management device 10 can reduce the amount of acquired image data compared to when capturing a moving image.

The imaging unit 14 may not only take an image of the plate 21 being conveyed on the conveyance lane 20, but also take an image of the plate 21 placed on a mounting base such as a table. Moreover, the imaging unit 14 may not only image one plate 21 but also image a plurality of plates 21 at once.
Further, in conveyor belt sushi restaurants and the like, there are cases where custom-ordered items (sushi 22 placed on a plate 21) ordered by customers are placed on a stand such as a bucket and transported along the transport lane 20. Even in such a case, the imaging unit 14 can take an image of the custom-made product on the table.

The extraction unit 12 extracts a partial image of the mounting member (dish 21) as a first image based on the image captured by the imaging unit 14. In this case, the extraction unit 12 extracts the first image based on the image captured by the imaging unit 14 and the second learning model. Specifically, the extraction unit 12 recognizes the mounting member (dish 21) based on the image captured by the imaging unit 14 and the second learning model, and applies the information to the recognized mounting member (dish 21). An image in a predetermined inscribed range is extracted as a first image. That is, the extraction unit 12 recognizes the planar shape of the plate 21 based on, for example, deep learning and machine learning. In this case, the extraction unit 12 can recognize the outer edge of the dish 21.

When a still image is captured by the imaging unit 14, the extraction unit 12 extracts a first image from the still image. When a video is captured by the imaging unit 14, the extraction unit 12 extracts a frame in which the entire plate 21 is captured from the video, and extracts a first image based on the frame and the second learning model.
Note that even if the entire plate 21 is not captured in the image captured by the imaging unit 14 (a part of the plate 21 is captured in a still image or frame), the extraction unit 12 inscribes the plate 21 from the image. If an image (first image) in a predetermined range can be extracted, the first image may be extracted from that image.
Further, when a plurality of dishes 21 are captured in the image (still image or frame) captured by the imaging unit 14, the extraction unit 12 recognizes each of the plurality of dishes 21 and corresponds to each of the plurality of dishes 21. The first image may also be extracted.

The extraction unit 12 extracts an image of a rectangular area S (see FIG. 2) inscribed in the plate 21 as a first image as an image of a predetermined range inscribed in the mounting member. That is, when the planar shape of the plate 21 is circular, the extraction unit 12 extracts the rectangular area S inscribed in the outer edge of the plate 21 as the first image as an image of a predetermined range inscribed in the plate 21.
Alternatively, even when the planar shape of the plate 21 is circular, the extraction unit 12 may extract not only the quadrangular area S described above but also a polygonal area inscribed in the outer edge of the plate 21, for example. However, a circular region that is a predetermined distance inside from the outer edge of the plate 21 may be extracted.
The placement state management device 10 can record the sushi 22 in the first image by extracting the rectangular area S inscribed in the outer edge of the plate 21, or can record the sushi 22 in the first image, compared to the case where the rectangular area S is not extracted. The amount of data can be reduced.

Note that even when the planar shape of the plate 21 is a quadrilateral, a polygonal region or a circular region inscribed in the outer edge of the plate 21 may be extracted as the first image.
Further, when the mounting member is a food tray, container, pack, bowl, bowl, box, lunch box, etc., a partition or the like may be arranged inside. The extraction unit 12 may recognize the inside of the partition (the part where the food is placed) and extract a polygonal area or a circular area inscribed in that part as the first image.

Furthermore, when the custom-made product is imaged by the imaging unit 14, the extraction unit 12 can extract, for example, an image (first image) of a rectangular area S inscribed in the plate 21. Here, in the case of a custom-made item, since the plate 21 is placed on a stand, the size of the first image is larger than that of a general item (an item that is not a custom-made item). Therefore, the placement state management device 10 can determine whether the item is a custom-made item or a general item based on the size of the first image.

Further, the extraction unit 12 inputs images continuously captured by the imaging unit 14, and extracts the first image when the entire plate 21 is captured in the image (when the plate 21 is located at a specific coordinate). It may also be extracted.

The determination unit 13 determines the placement state of the food placed on the placement member based on the first image extracted by the extraction unit 12 and the first learning model. That is, the determination unit 13 determines whether the sushi 22 placed on the plate 21 is placed according to the standard (or whether the sushi 22 is not placed according to the standard), for example, based on the result of deep learning or machine learning. Determine.

Specifically, the determining unit 13 determines whether the toppings of the sushi 22 placed on the plate 21 are off the rice, the sushi 22 has fallen down, or the seaweed has peeled off from the sushi 22. , at least one is determined to be out of the criteria for the placement state.
For example, the determining unit 13 determines what kind of sushi 22 is placed on the plate 21 based on the first image. As a specific example, the determination unit 13 determines the type of sushi 22 placed on the plate 21 (sushi, gunkan, hosomaki, etc.) based on deep learning, machine learning, and the like. Further, the determination unit 13 determines the placement state of each type of sushi 22 based on deep learning, machine learning, and the like. More specifically, when the sushi 22 placed on the plate 21 is a nigiri, the determination unit 13 determines whether the placement state of the nigiri is outside the standard. Similarly, when the sushi 22 placed on the plate 21 is a warship, the determination unit 13 determines whether the placement state of the warship is outside the standard. Similarly, when the sushi 22 placed on the plate 21 is a thin roll, the determination unit 13 determines whether the placement state of the thin roll is outside the standard.
Note that the determination unit 13 may determine the position where the sushi 22 is placed on the plate 21 as the placement state of the food (sushi 22).
In addition, the determining unit 13 determines the placement state of the sushi 22 placed on the plate 21 based on the first image and the first learning model, whether the sushi is a custom-made item or a general item.
Note that the determination unit 13 is not limited to the above example, and may determine the placement state of various foods that are considered to be out of standard at restaurants and food factories based on the learning model.

The operating section may perform a predetermined operation when the determining section 13 determines that the placement state of the sushi 22 is outside the standard. The operating section may be at least one of the output section 16 and the transport lane 20. Note that when the operating section is the transport lane 20, the operating section may include a transport control section (control section 11).

The output unit 16 outputs a warning when the determination unit 13 determines that the placement state of the sushi 22 is outside the standard. The output unit 16 may be, for example, at least one of a speaker and a display unit (display device). If the output unit 16 is a speaker, the output unit 16 (speaker) outputs a sound or warning sound indicating that the state of the sushi 22 placed on the plate 21 is out of standard. When the output unit 16 is a display unit, the output unit 16 (display unit) displays a warning image indicating that the state of the sushi 22 placed on the plate 21 is outside the standard. When a warning is output by the output unit 16, an employee of the restaurant can take measures such as removing the plate 21 on which the non-standard sushi 22 is placed from the conveyance lane 20.

Further, the conveyance lane 20 may stop conveying the plate 21 when the determination unit 13 determines that the state in which the sushi 22 is placed is outside the standard. For example, the transport lane 20 includes a circular transport lane and a straight transport lane. In particular, when the transport lane 20 is a straight transport lane, the transport lane 20 may stop transporting the plate 21 placed thereon (not transporting the plate 21). In this case, the conveyance control unit (control unit 11) that controls the conveyance lane 20 may control the conveyance lane 20 to stop when the determination unit 13 determines that the placement state of the sushi 22 is outside the standard. good.

Further, the above-mentioned operation unit may perform the following predetermined operation when the determination unit 13 determines that the placement state of the sushi 22 is outside the standard. The operating unit may be either another transport lane (another lane) or a robot device (for example, a robot arm). In this case, the operating section may include the control section 11.
When the conveyance lane 20 from which customers can pick up sushi 22 is connected to another lane heading toward a cooking area, the placement state management device 10 determines whether the sushi being conveyed on the conveyance lane 20 by the determination unit 13 is When the placement state of the sushi 22 is determined to be outside the standard, a lane switching device (not shown) is operated under the control of the control unit 11 to move the plate 21 on which the sushi 22 is placed to another lane. You may let them.
Alternatively, if a robot arm that can remove the plate 21 being transported on the transport lane 20 is installed, the placement state management device 10 can control the mounting state of the sushi 22 being transported on the transport lane 20. If it is determined that the sushi 22 is outside the standard, the robot arm may be operated under the control of the control unit 11 to remove the plate 21 on which the sushi 22 is placed from the conveyance lane 20.

FIG. 5 is a diagram illustrating an example of a tally result when the state in which the sushi is placed is determined to be outside the standard.
When the determination unit 13 determines that the food placement state is outside the standard, the control unit 11 may compile the determination results. For example, as shown in FIG. 5, the control unit 11 may total the number of food items determined to be outside the standard for each date (day of the week). Further, the control unit 11 may, for example, total the number of times when the food placement state is determined to be out of the standard. Further, the control unit 11 may, for example, distinguish between custom-made items and general items and tally them.

For example, the control unit 11 transmits the tally result (report of the placement state) to the headquarters terminal 31 via the communication unit 17 every predetermined period.
The headquarters terminal 31 displays, for example, the tally results sent from restaurants and the like on a terminal display section and the like. For example, the headquarters terminal 31 may display a warning or the like on the terminal display unit when the number of times the placement state of the sushi 22 in each restaurant is determined to be outside the standard is equal to or greater than a threshold value. Here, the headquarters terminal 31 may set different threshold values for custom-made items and general items. That is, the headquarters terminal 31 displays a warning etc. on the terminal display section when the number of times the placement condition of custom-ordered items for each restaurant is determined to be out of standard, for example, is greater than or equal to the first threshold value, and also displays a warning etc. on the terminal display section. A warning or the like may be displayed on the terminal display unit when the number of times the placement state of the device is determined to be outside the standard is equal to or greater than a second threshold (second threshold>first threshold).
By checking warnings and other displays on the terminal display, employees of the management headquarters 30 will notify employees of restaurants whose number of times the food placement condition has been determined to be out of standard to a threshold value or more. Can provide education, etc.

Note that the placement state management device 10 transmits the judgment result by the judgment unit 13 (for example, the judgment result that the placement state of the sushi 22 is outside the standard) to the headquarters terminal 31 via the communication unit 17. Good too.
The headquarters terminal 31 may aggregate the determination results sent from the placement state management device 10 for each store. The headquarters terminal 31 displays the results of the aggregation on the terminal display section, etc., and displays a warning etc. on the terminal display section if the number of times the placement state of the sushi 22 for each restaurant is determined to be outside the standard is a threshold value or more. You can also do it.

Next, a placement state management method according to an embodiment will be described.
FIG. 6 is a flowchart for explaining a placement state management method according to an embodiment.

In step ST11, the imaging unit 14 images the placement member on which the food is placed. Specifically, the imaging unit 14 images the plate 21 on which the sushi 22 is placed, which is moving on the conveyance lane 20, from above.

In step ST12, the extraction unit 12 extracts the first image based on the image captured in step ST11 and the second learning model. Specifically, the extraction unit 12 recognizes the plate 21 based on the image captured in step ST11 and the second learning model, and converts the image of the rectangular area S inscribed in the recognized plate 21 into the first image. Extract as. Note that, for example, when a video is captured in step ST11, the extraction unit 12 extracts a frame in which the entire plate 21 is captured from the video, and extracts the first image based on the frame and the second learning model. .

In step ST13, the determining unit 13 determines the placement state of the sushi 22 placed on the plate 21 based on the first image extracted in step ST12 and the first learning model. Specifically, the determining unit 13 determines whether the toppings of the sushi 22 placed on the plate 21 are off the rice, the sushi 22 has fallen down, or the seaweed has peeled off from the sushi 22. , at least one is determined to be out of the criteria for the placement state. If the placement state is outside the standard (Yes), the process proceeds to step ST14. If the placement state is not outside the standard (No), the process ends.

In step ST14, the output unit 16 outputs a warning. The output unit 16 may be, for example, at least one of a speaker and a display unit.
Further, in step ST14, the transport lane 20 may stop the transport operation of the plate 21 placed thereon (not transport the plate 21).

In step ST15, the control unit 11 transmits information to the management headquarters 30 that the placement state of the sushi 22 is outside the standard. The control unit 11 may transmit to the management headquarters 30 the results of aggregating the number of times the placement state of the sushi 22 has been determined to be outside the standard for each date (day of the week) and for each time period.

The headquarters terminal 31 can display the total results transmitted in step ST15 on a terminal display section or the like. For example, the headquarters terminal 31 may display a warning or the like on the terminal display section when the number of times the food placement condition of each restaurant is determined to be outside the standard is equal to or greater than a threshold value. The employees of the management headquarters 30 can educate the employees of the restaurant by checking the display of warnings and the like on the terminal display section.

Next, the effects of this embodiment will be explained.
The placement state management device 10 includes an extraction unit 12 that extracts a part of the image of the plate 21 as a first image based on the image captured by the imaging unit 14, and a first image extracted by the extraction unit 12. , a first learning model, and a determination unit 13 that determines the placement state of the sushi 22 placed on the plate 21, and when the determination unit 13 determines that the placement state of the sushi 22 is outside the standard. , and an output unit 16 that outputs a warning.
Thereby, the placement state management device 10 can determine that the placement state of the sushi 22 is outside the standard. Moreover, since the placement state management device 10 outputs a warning when the placement state of the sushi 22 is out of the standard, it is possible to notify an employee of the restaurant. Furthermore, the placement state management device 10 prevents non-standard sushi 22 from being provided to customers by removing sushi 22 (plates 21) whose placement state is non-standard by an employee from the conveyance lane 20. It is possible to improve the quality of products provided at conveyor belt sushi restaurants.

In the placement state management device 10, the extraction unit 12 may extract the first image based on the image captured by the imaging unit 14 and the second learning model.
Thereby, the placement state management device 10 can recognize the plate 21 based on, for example, deep learning and machine learning. Further, the placement state management device 10 can extract the first image based on the recognized plate 21.

In the placement state management device 10, the extraction unit 12 recognizes the plate 21 based on the image captured by the imaging unit 14 and the second learning model, and extracts an image of a predetermined range inscribed in the recognized plate 21. It may be extracted as the first image.
Since the placement state management device 10 extracts an image (first image) of a predetermined range inscribed in the plate 21, only the portion where the sushi 22 is placed can be extracted. Therefore, since the first image does not include images of parts unnecessary for determining the placement state of the sushi 22, the placement state management device 10 suppresses misjudgment of the placement state of the sushi 22. be able to. Furthermore, since the placement state management device 10 extracts the first image, the amount of data can be reduced compared to the image generated by the imaging unit 14 (an image before extracting the first image), and the determination unit 13 etc. processing amount can be reduced.

In the placement state management device 10, the placement member may be a plate 21 having a circular planar shape on which sushi 22 as food is placed. The sushi 22 may be at least one of nigiri, gunkan, tsutsumi, and hosomaki. The imaging unit 14 images the plate 21 moving on the conveyance lane 20 from above with respect to the surface on which the sushi 22 is placed on the plate 21, and the extraction unit 12 images a predetermined range inscribed in the placement member. An image of a rectangular region S inscribed in the plate 21 may be extracted as the first image.
The placement state management device 10 can determine the placement state of the sushi 22 placed on the plate 21 being transported through the transportation lane 20 of the restaurant. Further, the placement state management device 10 can determine the placement state according to the type of sushi 22.

In the placement state management device 10, the determination unit 13 determines whether the toppings of the sushi 22 placed on the plate 21 are off the rice, the sushi 22 has fallen down, and the seaweed has peeled off from the sushi 22. At least one of these cases may be determined to be outside the criteria for the placement state.
The placement state management device 10 can determine whether the placement state of the sushi 22 placed on the plate 21 is outside the standard. That is, the placement state management device 10 can prevent sushi 22 whose placement condition does not meet the standards from being provided to customers, and can improve the quality of sushi 22 provided at restaurants.

In the placement state management method, the computer extracts a part of the image of the plate 21 as a first image based on the image captured by the imaging unit 14; and the first image extracted in the extraction step. , a determination step of determining the placement state of the sushi 22 placed on the plate 21 based on the first learning model, and a warning if the placement state of the sushi 22 is determined to be outside the standard in the determination step. an output step of outputting from the output unit 16.
Thereby, the placement state management method can determine that the placement state of the sushi 22 is outside the standard. Moreover, since the placement state management method outputs a warning when the placement state of the sushi 22 is outside the standard, employees of the restaurant can be notified. In addition, the placement state management method prevents non-standard sushi 22 from being provided to customers by removing sushi 22 (plates 21) whose placement state is non-standard by an employee from the conveyance lane 20. It is possible to improve the quality of products provided at conveyor belt sushi restaurants.

The placement state management program provides a computer with an imaging function that images the plate 21 on which the sushi 22 is placed, and extracts a partial image of the plate 21 as a first image based on the image captured by the imaging function. a determination function that determines the state of the sushi 22 placed on the plate 21 based on the first image extracted by the extraction function and the first learning model; 22 is determined to be outside the standard, an output function for outputting a warning is executed.
Thereby, the placement state management program can determine that the placement state of the sushi 22 is outside the standard. Moreover, since the placement state management program outputs a warning when the placement state of the sushi 22 is out of the standard, it is possible to notify the employees of the restaurant. In addition, the placement state management program prevents non-standard sushi 22 (plates 21) from being provided to customers by having employees remove the sushi 22 (plates 21) whose placement state is non-standard from the conveyance lane 20. It is possible to improve the quality of products provided at conveyor belt sushi restaurants.

Each part of the above-mentioned placement state management device 10 may be realized as a function of an arithmetic processing unit of a computer or the like. That is, the extraction unit 12 and the determination unit 13 of the placement state management device 10 may be implemented as an extraction function and a determination function, respectively, by a computer processing unit or the like.
The placement state management program can cause a computer to implement each of the above-mentioned functions. The placement state management program may be recorded on a computer-readable non-temporary recording medium such as an external memory or an optical disk.
Further, as described above, each part of the placement state management device 10 may be realized by a processing unit of a computer or the like. The arithmetic processing device and the like are constituted by, for example, an integrated circuit or the like. Therefore, each part of the placement state management device 10 may be realized as a circuit that constitutes an arithmetic processing device or the like. That is, the extraction unit 12 and the determination unit 13 of the placement state management device 10 may be realized as an extraction circuit and a determination circuit that constitute an arithmetic processing unit of a computer or the like.
The imaging unit 14, storage unit 15, output unit 16, and communication unit 17 of the placement state management device 10 may be realized as an imaging function unit, a storage function, an output function, and a communication function. In addition, the imaging section 14, storage section 15, output section 16, and communication section 17 of the placement state management device 10 are configured with integrated circuits, etc., so that they can function as an imaging circuit, a storage circuit, an output circuit, and a communication circuit. May be realized. In addition, the imaging unit 14, storage unit 15, output unit 16, and communication unit 17 of the placement state management device 10 are configured with a plurality of devices, so that they function as an imaging device, a storage device, an output device, and a communication device. may be configured. Furthermore, the operating section may be realized as an operating function or an operating device. Since the operating section is at least one of the output section 16 and the transport lane 20 (or another lane), the operating function (operating device) may be at least one of the output function (output device) and the transport function (transporting device). good. Further, the motion function may be a robot function. Further, the operating function (operating device) may include a control function (operating device) as a function of the control unit 11.

1 Placement state management system 10 Placement state management device 12 Extraction unit 13 Determination unit 14 Imaging unit 15 Storage unit 16 Output unit 20 Transport lane 30 Management headquarters 31 Headquarters terminal

Claims (9)

a storage unit that stores a first learning model that has previously learned a placement state of the food based on an image of the food placed on the placement member;
an imaging unit that captures an image of a placement member on which food is placed;
an extraction unit that extracts a partial image of the mounting member as a first image based on the image captured by the imaging unit;
a determination unit that determines a placement state of food placed on the placement member based on the first image extracted by the extraction unit and the first learning model;
an operation unit that performs a predetermined operation when the placement state of the food is determined to be out of the standard by the determination unit;
The extraction unit recognizes the outer edge of the placement member from the image taken by the imaging unit of the placement member, based on a second learning model that has learned the shape of the placement member, and determines whether the outer edge is circular. If the outer edge is a quadrangle, the inscribed quadrangular region or polygonal region is extracted as a predetermined range , and the image of the extracted predetermined range is placed as the first image. State management device. The operating section is
an output unit that outputs a warning when the determination unit determines that the food placement state is outside the standard;
The placement state management device according to claim 1, wherein the placement state management device is at least one of a transportation lane that stops transportation of the placement member when the placement state of the food is determined to be outside the standard by the determination unit. The operating section is
another conveyance lane connected to the conveyance lane, where the conveyance destination of the placement member conveyed by the conveyance lane is switched from the conveyance lane when the placement state of the food is determined to be outside the standard by the determination section; as well as,
2. The robot device according to claim 1, wherein the robot device removes a placement member transported by the transport lane from the transport lane when the judgment unit determines that the food mounting state is outside the standard. Placement status management device. The storage unit stores a second learning model that has learned the shape of the mounting member in advance based on an image of the mounting member,
The placement state management device according to any one of claims 1 to 3, wherein the extraction unit extracts the first image based on the image captured by the imaging unit and the second learning model. . The extraction unit recognizes the mounting member based on the image captured by the imaging unit and the second learning model, and extracts an image of a predetermined range inscribed in the recognized mounting member from the first learning model. The placement state management device according to claim 4, which extracts the information as an image. The mounting member is a plate having a substantially circular planar shape on which sushi as food is placed;
The imaging unit images a plate that moves along with the movement of a conveyance lane that conveys the plate on which the sushi is placed, from above with respect to a surface on which the sushi is placed on the plate;
The placement state management device according to claim 5, wherein the extraction unit extracts, as the first image, an image of a rectangular area inscribed in the dish as the image of the predetermined range inscribed in the placement member. The determination unit determines at least one of the following when the sushi toppings placed on the plate are misaligned with the sushi rice, when the sushi has fallen down, and when the seaweed has peeled off from the sushi. The placement state management device according to claim 6, which determines that the placement state management device is out of the standard. A computer includes a storage unit that stores a first learning model that has learned the placement state of the food in advance based on an image of the food placed on the placement member,
an imaging step of causing an imaging unit to take an image of the placement member on which the food is placed;
an extraction step of extracting a part of the image of the mounting member as a first image based on the image taken in the imaging step;
a determination step of determining a placement state of the food placed on the placement member based on the first image extracted in the extraction step and the first learning model;
If the placement state of the food is determined to be outside the standard in the determination step, an operation step of performing a predetermined operation;
In the extraction step, the outer edge of the placement member is recognized from an image of the placement member taken by the imaging unit based on a second learning model that has learned the shape of the placement member, and the outer edge is circular. If the outer edge is a quadrangle, the inscribed quadrangular region or polygonal region is extracted as a predetermined range , and the image of the extracted predetermined range is placed as the first image. State management method. A computer including a storage unit that stores a first learning model that has previously learned the placement state of the food based on an image of the food placed on the placement member;
an imaging function that images a placement member on which food is placed;
an extraction function that extracts a part of the image of the mounting member as a first image based on the image captured by the imaging function;
a determination function that determines a placement state of food placed on the placement member based on the first image extracted by the extraction function and the first learning model;
an operation function that performs a predetermined operation when the placement state of the food is determined to be outside the standard by the determination function;
The extraction function recognizes the outer edge of the placement member from an image of the placement member taken by the imaging function based on a second learning model that has learned the shape of the placement member, and determines whether the outer edge is circular. If the outer edge is a quadrangle, the inscribed quadrangular region or polygonal region is extracted as a predetermined range , and the image of the extracted predetermined range is placed as the first image. State management program.

Images