JP6239492B2 - Product management apparatus and program - Google Patents

Description

  Embodiments described herein relate generally to a product management apparatus and a program.

  Traditionally, there is a sushi restaurant called Kaiten Sushi. In such a store, sushi as a product is placed on a plate and placed on a rotating lane provided in contact with the kitchen or the audience seat, thereby circulating the sushi to the audience seat.

  By the way, in the above stores, sushi on a rotating lane may be conveyed in an inappropriate state. For example, while being circulated and conveyed, the shape of sushi may collapse or the freshness of sushi may deteriorate, resulting in an inappropriate state that deviates from an appropriate state. Further, when there is an error in topping during sushi production, the sushi is conveyed in an inappropriate state. In such a case, the store side needs to cope by removing sushi in an inappropriate state from the rotating lane. However, in the conventional configuration, the store clerk monitors each sushi on the rotating lane, which is labor intensive and may cause oversight.

  The problem to be solved by the present invention is to provide a merchandise management apparatus and program capable of efficiently managing merchandise at stores such as carousel sushi.

The product management apparatus according to the embodiment includes a capturing unit that captures a captured image from an imaging device that captures an image on a transport path on which a transport body on which products are placed is transported, a type of the transport body, and an appearance of the transport body Storage means for storing a dictionary file in which data representing the characteristics, the type of product placed on the carrier, and the appearance characteristics of the product are associated and registered; and the capture means A carrier region detection means for detecting a carrier region representing the carrier from the captured image, a feature on the appearance of the carrier registered in the dictionary file, and a feature on the appearance of the carrier region. The first identification means for comparing and identifying the type of the carrier represented in the carrier area, the commodity detection means for detecting the commodity area representing the commodity from the carrier area, and registered in the dictionary file Features on the appearance of the product The second identification means for comparing the appearance characteristics of the product area and identifying the type of the product represented in the product area, and whether the state of the product represented in the product area is in a predetermined appropriate state Determining means for determining whether or not the determining means determines that the state is inappropriate. The determining means includes: a type of the carrier specified by the first specifying means; (2) When the combination with the type of product specified by the specifying means is not registered in association with the dictionary file, it is determined as an inappropriate state.

FIG. 1 is a diagram schematically illustrating a store system according to the embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the commodity management apparatus. FIG. 3 is a diagram schematically illustrating an example of the data configuration of the first dictionary file. FIG. 4 is a diagram schematically illustrating an example of the data configuration of the second dictionary file. FIG. 5 is a block diagram illustrating an example of a functional configuration of the product management apparatus. FIG. 6 is a flowchart illustrating a procedure of product management processing performed by the product management apparatus. FIG. 7 is a diagram schematically illustrating another configuration example of the store system.

  Below, the merchandise management apparatus and program used in a sushi restaurant (store) called “conveyor sushi” will be described with reference to the drawings.

  FIG. 1 is a diagram schematically illustrating a store system 1 according to the present embodiment. As shown in FIG. 1, the store system 1 includes a transport device 10, a camera 20, a notification device 30, and a product management device 40. Here, the camera 20, the notification device 30, and the product management device 40 are communicably connected via a communication line (not shown).

  The transport apparatus 10 is a transport apparatus for transporting sushi that is a product in the main store. The transport apparatus 10 includes a circulation-type transport path (rotation lane) 10a formed in a substantially U shape. A part of the rotating lane 10a is disposed in the backyard R1, and the other part is disposed in the passenger cabin R2.

  In the backyard R1, mainly sushi and the like that are commodities are manufactured. In the backyard R1, the store clerk places the manufactured sushi on the dish D serving as a transporter, and places the dish D on the rotating lane 10a. Guest room R2 is provided with a passenger seat (not shown) such as a counter table along rotation lane 10a. The transport device 10 circulates and delivers the products to the passenger seats by transporting the dish D placed on the rotation lane 10a in the direction of the arrow in the figure. In addition, about a conveyance of the conveying apparatus 10, a well-known technique shall be used. The shape of the rotation lane 10a is not limited to that shown in FIG.

  The camera 20 is an imaging device including an imaging element such as a color CCD (Charge Coupled Device) or a color CMOS (Complementary Metal Oxide Semiconductor). The camera 20 is provided on the backyard R1 side, for example, at the boundary with the cabin R2, and images the rotating lane 10a from above. That is, the camera 20 images the dish D placed on the rotation lane 10a from above. The camera 20 captures a moving image at a frame rate of, for example, 30 fps and transmits the captured image to the product management apparatus 40 as a captured image. In the present embodiment, two cameras 20 are provided. However, the number of cameras 20 is not particularly limited. In the present embodiment, the camera 20 is provided on the backyard R1 side. However, the configuration is not limited to this, and the camera 20 may be provided on the cabin R2 side.

  The notification device 30 is a notification device provided in the backyard R1 or the like. Specifically, the notification device 30 operates according to the control of the product management device 40 to notify the store clerk that there is an inappropriate product. Note that the notification method of the notification device 30 is not particularly limited. For example, it is good also as a structure which alert | reports by the output of a message or an image by making the alerting | reporting apparatus 30 into an electric bulletin board and a display apparatus. Moreover, it is good also as a structure which alert | reports by the output of an audio | voice by making the alerting | reporting apparatus 30 into audio | voice output apparatuses, such as a buzzer and a speaker.

  The merchandise management apparatus 40 is an information processing apparatus such as a PC (Personal Computer) provided in the backyard R1 or the like. The product management device 40 manages the state of the product conveyed by the rotation lane 10a based on the image of the product (such as sushi) captured by the camera 20. Hereinafter, the merchandise management apparatus 40 will be described.

  FIG. 2 is a block diagram illustrating an example of a hardware configuration of the merchandise management apparatus 40. The merchandise management apparatus 40 includes a microcomputer 41 as an information processing unit that executes information processing. The microcomputer 41 mainly includes a CPU (Central Processing Unit) 42, a ROM (Read Only Memory) 43, and a RAM (Random Access Memory) 44. The ROM 43 stores various programs that can be executed by the CPU 42. The RAM 44 is a main storage device of the product management device 40.

  A communication unit 45 is connected to the CPU 42 of the product management apparatus 40. The communication unit 45 performs data communication with the camera 20 and the notification device 30 under the control of the CPU 42.

  In addition, a storage unit 46 such as an HDD or an SSD is connected to the CPU 42 of the merchandise management apparatus 40. The storage unit 46 stores various programs and various files that can be executed by the CPU 42. In addition, the storage unit 46 stores a first dictionary file F1 that stores the feature amount of each dish D on which the product is placed, and a second dictionary file F2 that stores the feature amount of each product.

  FIG. 3 is a diagram schematically illustrating an example of the data configuration of the first dictionary file F1. The first dictionary file F1 stores the dish feature amount in association with the dish identifier. The dish identifier is an identifier uniquely assigned to each dish type. In general, in a conveyor belt sushi restaurant, prices are classified for each product, and products are conveyed using dishes D having different patterns for each price. Therefore, an identifier that can identify the type is stored in the dish identifier of the first dictionary file F1 for each type of dish D used for transporting the product. The dish feature amount is data obtained by parameterizing appearance features of each dish D such as the shape of the dish D and the pattern of the dish D.

  FIG. 4 is a diagram schematically illustrating an example of the data configuration of the second dictionary file F2. The second dictionary file F2 stores information on each product. Specifically, the second dictionary file F2 stores a product name, a dish identifier, a first product feature value, a second product feature value, and the like representing an item of the product in association with the product identifier of each product.

  Here, the product identifier is a unique identifier assigned to each product type. The dish identifier is a dish identifier of the dish D on which the product is placed. The first product feature amount is data obtained by parameterizing features on the appearance (surface) of foods constituting the product, such as the color, gloss, pattern, and unevenness of the product. This first product feature amount is mainly used to specify the type of product. The second product feature amount is data obtained by parameterizing features on the outer shape such as the shape, layout, and topping of the product when the product is placed on the dish D. The second product feature amount is mainly used to determine whether or not the product is placed on the dish D with a predetermined outer shape (appropriate outer shape). It is assumed that the first product feature amount and the second product feature amount are created based on products in an appropriate state.

  Next, the functional configuration of the merchandise management apparatus 40 will be described with reference to FIG. FIG. 5 is a block diagram illustrating an example of a functional configuration of the product management device 40. The product management device 40 cooperates with the CPU 42 and a program stored in the ROM 43 or the storage unit 46 to obtain an image capture unit 411, a dish detection unit 412, a product identification unit 413, an outer shape determination unit 414, and a product management unit 415. The notification processing unit 416 is provided as a functional unit.

  The image capturing unit 411 is a functional unit that captures a captured image captured by the camera 20 via the communication unit 45. Specifically, the image capturing unit 411 sequentially captures captured images captured by the camera 20 via a communication line (not shown) and stores the captured images on the RAM 44.

  The dish detection unit 412 detects a region representing the dish D or the like from the captured image captured by the image capturing unit 411 by using a known technique such as a pattern matching technique. Here, the detection method of the dish detection unit 412 is not particularly limited.

  For example, the dish detection unit 412 compares the contour line extracted from the captured image with the shape defined in the dish feature value of the first dictionary file F1. Then, the dish detection unit 412 detects an area in the captured image that matches the shape of the dish feature amount as a dish area. Further, the dish detection unit 412 compares the pattern represented in the dish area with the pattern defined in the dish feature amount of the first dictionary file F1, and identifies a matching pattern. And the dish detection part 412 detects the remaining area | region remove | excluding the matched pattern from the dish area | region as a goods area | region. In addition, the dish detection unit 412 identifies the dish identifier of the dish feature amount pattern that matches the pattern of the dish area as the type of the dish D.

  The product specifying unit 413 specifies the product included in the captured image from the captured image captured by the image capturing unit 411. Specifically, the product specifying unit 413 extracts data representing the appearance (surface) feature from the product region detected by the dish detection unit 412 as the first product feature amount. The product specifying unit 413 compares the first product feature amount extracted from the product region with the first product feature amount of each registered product registered in the second dictionary file F2, and calculates the similarity. Here, the degree of similarity quantitatively indicates how similar the product in the captured image is to each registered product. The product specifying unit 413 calculates the similarity based on the first product feature amount of each registered product associated with the plate identifier based on the plate identifier specified by the plate detection unit 412. Then, the product specifying unit 413 specifies (recognizes) a registered product having a highest similarity that exceeds a predetermined threshold among the calculated similarities as a product item represented in the product area.

Recognizing an object included in an image in this way is called generic object recognition. Regarding general object recognition, for example, various recognition techniques are described in the following documents.
Keiji Yanai, “Current Status and Future of General Object Recognition”, IPSJ Journal, Vol. 48, no. SIG16, [Search September 10, 2012], Internet <URL: http://mm.cs.uec.ac.jp/IPSJ-TCVIM-Yanai.pdf>

Further, techniques for performing general object recognition by dividing an image into regions for each object are described in the following documents.
Jamie Shotton et al., “Semantic Texton Forests for Image Categorization and Segmentation”, [searched on September 10, 2012], Internet <URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1. 1.145.3046 & rep = rep1 & type = pdf>

  Note that the method of calculating the similarity is not particularly limited. For example, the similarity may be calculated as an absolute evaluation or a relative evaluation. When calculating the similarity as an absolute evaluation, the first product feature amount of the product in the captured image is compared with each first product feature amount registered in the second dictionary file F2 on a one-to-one basis. The similarity derived as a result of the above may be adopted as it is. Further, when calculating the similarity as a relative evaluation, the total of the similarities may be calculated to be 100%.

  The outer shape determination unit 414 determines whether or not the outer shape of the product included in the captured image is appropriate by using a known technique such as a pattern matching technique. Specifically, the outer shape determination unit 414 uses, as the second product feature amount, features on the outer shape such as the shape, layout, and topping of the products existing in the region from the dish region and the product region detected by the dish detection unit 412. Extract. In addition, the outer shape determination unit 414 reads the second product feature amount of the registered product specified by the product specifying unit 413 from the second dictionary file F2. Then, the outer shape determination unit 414 compares the second product feature amount extracted from the dish region with the second product feature amount read from the second dictionary file F2, thereby determining whether or not the outer shape of the product matches. judge. Here, when the determination result of the outer shape determination unit 414 indicates a match, it means that the outer shape of the product is in an appropriate state.

  The product management unit 415 is a functional unit that manages each product transported on the rotating lane 10 a based on the processing results of the dish detection unit 412 to the outer shape determination unit 414. Specifically, the product management unit 415 determines whether each product is in an appropriate state based on the processing results of the dish detection unit 412 to the outer shape determination unit 414. For example, the notification processing unit 416 determines that the product is in an inappropriate state in the following processing results (first processing result, second processing result, and third processing result).

  As the first processing result, there is a case where the dish detection unit 412 cannot detect the product area even though the dish detection unit 412 detects the dish area. Moreover, as a 2nd process result, the case where the goods specific | specification part 413 cannot identify goods from the goods area which the dish detection part 412 detected is mentioned. Possible causes of these processing results are that the product is not placed on the dish D and that the characteristics (color, gloss, etc.) inherent to the product are lost due to deterioration of freshness and the like. Moreover, it is possible that the combination of the dish D and goods is wrong. Therefore, the product management unit 415 determines that the product is in an inappropriate state when the first or second processing result is obtained.

  Further, as a third processing result, there is a case where the outer shape determination unit 414 determines that the product is different even though the product specifying unit 413 has specified the product. The cause of the third processing result may be that the shape or layout of the product placed on the dish D is broken, or there is an error in topping. Therefore, the product management unit 415 determines that the product is in an inappropriate state when the third processing result is obtained.

  The notification processing unit 416 is a functional unit that controls the notification device 30 via a communication line (not shown) to perform notification. Specifically, the notification processing unit 416 notifies the store clerk that the product is in an inappropriate state by operating the notification device 30 when the product management unit 415 determines that it is an illegal act. Note that the notification processing unit 416 may be configured to switch a notification message or the like and a notification method in accordance with a processing result (first to third processing results). As a result, the store clerk can grasp the state of the product by confirming the notification content, so that the convenience related to the product management can be improved.

  Hereinafter, the product management process performed by the product management apparatus 40 will be described with reference to FIG. Here, FIG. 6 is a flowchart showing the procedure of the merchandise management process performed by the merchandise management apparatus 40.

  First, the image capturing unit 411 sequentially captures captured images captured by the camera 20 (step S11). The dish detection unit 412 waits until a dish area is detected from the captured images sequentially taken in step S11 (step S12; No). When detecting the dish area (step S12; Yes), the dish detection unit 412 determines whether or not the commodity area has been detected from the dish area (step S13). Here, when the dish detection unit 412 cannot detect the product area (step S13; No), the product management unit 415 determines that the product is in an inappropriate state, and executes step S19.

  On the other hand, when the dish detection unit 412 detects the product area (step S13; Yes), the process proceeds to step S14. Note that the image capturing unit 411 also specifies the dish type (dish identifier) when detecting the dish area.

  Subsequently, the product specifying unit 413 extracts the first product feature amount from the product region detected in Step S12 (Step S14). Next, the product specifying unit 413 compares the extracted first product feature quantity with the first product feature quantity of each registered product registered in the first dictionary file F1, and calculates a similarity (step S15). Then, the product identification unit 413 identifies a product (product type) based on the similarity calculated in step S14 (step S16). Note that the product specifying unit 413 performs comparison with the first feature amount of the registered product associated with the plate identifier based on the plate identifier specified in step S12.

  If the product identification unit 413 cannot identify the product (step S16; No), the product management unit 415 determines that the product is in an inappropriate state and causes step S19 to be executed.

  In addition, when the product specifying unit 413 specifies a product (step S16; Yes), the outer shape determination unit 414 extracts a feature related to the outer shape of the product represented in the product region as a second product feature amount (step S17). . Subsequently, the outer shape determination unit 414 determines whether or not the second product feature value extracted in step S17 matches the second product feature value of the product specified in step S15 (step S18).

  When the determination result of the outer shape determination unit 414 indicates a difference (step S18; No), the product management unit 415 determines that the product is in an inappropriate state, and executes step S19.

  In step S19, the notification processing unit 416 performs notification using the notification device 30 (step S19), and returns to step S12. If the determination result of the outer shape determination unit 414 indicates a match (step S18; Yes), the product management unit 415 determines that the product is in an appropriate state and returns to step S12.

  As described above, the product management apparatus 40 according to the present embodiment detects a product in an inappropriate state from the products transported on the rotating lane 10a, and notifies the store clerk to that effect. As a result, it is possible to save the labor of the store clerk involved in monitoring the product status, and to prevent a situation in which the product in an inappropriate state continues to be conveyed, thereby enabling efficient management of the product. .

  As mentioned above, although embodiment of this invention was described, the said embodiment was shown as an example and is not intending limiting the range of invention. The above embodiments can be implemented in various other forms, and various omissions, replacements, changes, additions, combinations, and the like can be made without departing from the scope of the invention. Moreover, the said embodiment and its deformation | transformation are included in the range of the invention, the summary, and the invention described in the claim, and its equal range.

  For example, in the merchandise management apparatus 40 of the above-described embodiment, the notification is performed when it is determined that the merchandise is in an inappropriate state, but the present invention is not limited thereto. For example, the product management device 40 may be configured to exclude the product from the rotation lane 10a when it is determined that the product is in an inappropriate state. When this configuration is adopted, the store system 1 further includes an exclusion device 50 as shown in FIG.

  Here, FIG. 7 is a diagram illustrating another configuration example (modification) of the store system 1. The exclusion device 50 is provided on one side of the rotation lane 10a on the backyard R1 side. The exclusion device 50 is connected to the product management device 40 via a communication line (not shown), and operates according to the control of the product management device 40. Specifically, the exclusion device 50 excludes the dish D from the rotation lane 10a by pushing out the dish D (product) placed on the rotation lane 10a to the other side (the arrow direction in the drawing). In addition, it is preferable to install the exclusion apparatus 50 in the back | latter stage of the area | region on the rotation lane 10a which the camera 20 images in the conveyance direction of the rotation lane 10a.

  On the other hand, in the product management device 40, when the product management unit 415 detects the processing result to be notified, the product management unit 415 notifies the notification using the notification device 30, and uses the exclusion device 50 to identify the corresponding product from the rotation lane 10a. Exclude. Specifically, when the product management unit 415 detects the processing result to be notified, the product management unit 415 operates the exclusion device 50 at the timing when the product targeted for the processing is conveyed to the installation position of the removal device 50. In addition, the timing which operates the exclusion apparatus 50 shall be predetermined according to the installation position of the exclusion apparatus 50, the conveyance speed of the rotation lane 10a, etc. Thereby, since the product in an improper state can be excluded from the rotation lane 10a, the labor of the store clerk involved in product management can be reduced.

  In this modification, the merchandise management unit 415 is configured to control the exclusion device 50. However, the configuration is not limited thereto, and a dedicated functional unit that controls the exclusion device 50 may be provided.

  Moreover, in the said embodiment, although rotating sushi was mentioned as an example as an example of a store, you may apply to other stores which handle goods other than this, not only this. In addition, it does not ask whether a conveyance path is a circulation type.

  In addition, the program executed by each device of the above embodiment is provided by being incorporated in advance in a storage medium (ROM or storage unit) included in each device, but is not limited thereto. For example, the program may be provided in a form that can be installed or executed and recorded on a computer-readable recording medium. Examples of the storage medium include a CD-ROM, a flexible disk (FD), a CD-R, and a DVD (Digital Versatile Disk). Further, the storage medium is not limited to a medium independent of a computer or an embedded system. For example, the storage medium includes a storage medium that is stored or temporarily stored by downloading a program transmitted via a LAN, the Internet, or the like.

  The program executed by each device of each of the above embodiments may be provided or distributed via a network such as the Internet.

DESCRIPTION OF SYMBOLS 1 Store system 10 Conveyance apparatus 10a Rotation lane 20 Camera 30 Notification apparatus 40 Merchandise management apparatus 50 Exclusion apparatus 411 Image capture part 412 Dish detection part 413 Commodity specification part 414 Shape determination part 415 Commodity management part 416 Notification processing part

JP 2006-20881 A

Claims (6)

Capture means for capturing a captured image from an image capturing apparatus that captures an image on a transport path on which a transport body on which products are placed is transported;
Data indicating the type of the carrier, the appearance characteristics of the carrier, the type of product placed on the carrier, and the appearance characteristics of the product are registered in association with each other. Storage means for storing dictionary files;
A carrier region detection unit for detecting a carrier region representing the carrier from the captured image captured by the capture unit;
A first specifying unit that compares the appearance feature of the carrier registered in the dictionary file with the appearance feature of the carrier region, and identifies the type of the carrier represented in the carrier region; ,
Product detection means for detecting a product region representing the product from the carrier region;
A second specifying means for comparing the appearance characteristics of the product registered in the dictionary file with the appearance characteristics of the product area, and specifying the type of the product represented in the product area;
Determining means for determining whether the state of the product represented in the product area is in a predetermined appropriate state;
An informing means for informing when the judging means judges an inappropriate state;
Equipped with a,
The determination unit is inappropriate when the pair of the carrier type specified by the first specifying unit and the product type specified by the second specifying unit is not registered in association with the dictionary file. A merchandise management apparatus that determines a state . The second specifying means compares the appearance characteristics of the product associated with the type of the carrier specified by the first specifying means and the appearance characteristics of the product area,
The product management apparatus according to claim 1 , wherein the determination unit determines that the state is inappropriate when the second specifying unit cannot specify the type of the product. An outer shape determination means for determining whether the outer shape of the product represented in the product area matches a predetermined appropriate outer shape;
The merchandise management apparatus according to claim 1, wherein the determination unit determines that the state is inappropriate when it is determined that the outer shape determination unit is different. The product management apparatus according to any one of claims 1 to 3, wherein the determination unit determines that the product detection unit is in an inappropriate state when the product region cannot be detected from the transport body region.   The product management apparatus according to any one of claims 1 to 4, further comprising an excluding unit that excludes the product determined by the determining unit as being in an inappropriate state from the conveyance path. Computer
Capture means for capturing a captured image from an image capturing apparatus that captures an image on a transport path on which a transport body on which products are placed is transported;
Data indicating the type of the carrier, the appearance characteristics of the carrier, the type of product placed on the carrier, and the appearance characteristics of the product are registered in association with each other. Storage means for storing dictionary files;
A carrier region detection unit for detecting a carrier region representing the carrier from the captured image captured by the capture unit;
A first specifying unit that compares the appearance feature of the carrier registered in the dictionary file with the appearance feature of the carrier region, and identifies the type of the carrier represented in the carrier region; ,
Product detection means for detecting a product region representing the product from the carrier region;
A second specifying means for comparing the appearance characteristics of the product registered in the dictionary file with the appearance characteristics of the product area, and specifying the type of the product represented in the product area;
Determining means for determining whether the state of the product represented in the product area is in a predetermined appropriate state;
An informing means for informing when the judging means judges an inappropriate state;
To function ,
The determination unit is inappropriate when the pair of the carrier type specified by the first specifying unit and the product type specified by the second specifying unit is not registered in association with the dictionary file. A program that determines a state .

Images