WO2020045464A1

WO2020045464A1 - Merchandise positioning device, merchandise positioning method, merchandise positioning system, and merchandise positioning program

Info

Publication number: WO2020045464A1
Application number: PCT/JP2019/033613
Authority: WO
Inventors: 亀山博史; 大西弘之; 石田良介
Original assignee: グローリー株式会社
Priority date: 2018-08-30
Filing date: 2019-08-28
Publication date: 2020-03-05
Also published as: JP2020035196A

Abstract

Provided are a merchandise positioning device, merchandise positioning method, merchandise positioning system, and merchandise positioning program, which are easy for an operator to use in self-checkout, etc. This merchandise positioning device comprises: a setting part which, for each of a plurality of areas into which a positioning region for positioning merchandise has been divided, sets identification information for identifying the area while associating the information with the area; and a reporting part which reports the identification information having been set by the setting part.

Description

Product placement device, product placement method, product placement system, and product placement program

The present invention relates to a product placement device, a product placement method, a product placement system, and a product placement program for placing a product at a cash register or the like provided in a store.

Conventionally, a store system is known that can photograph a plurality of products in a paper box and products taken out of a shopping basket with a camera (for example, see Patent Documents 1 and 2).

The store system described in Patent Document 1 stores a donut ordered by a purchaser in a paper box provided with a plurality of partitions by a clerk, and based on an image obtained by photographing the plurality of donut stored in the paper box with a camera. Recognize the type of Then, the recognized product information such as the product name and unit price of the donut and the image of the paper box containing the donut are simultaneously displayed to the purchaser.

The store system described in Patent Document 2 is a so-called self-checkout system, in which a purchaser operates a touch panel to input the presence or absence of a barcode, and in the case of a product with a barcode, the purchaser reads the barcode with a barcode scanner. Let On the other hand, in the case of a product without a barcode, the purchaser inputs the product name on the touch panel, and displays the product image taken by the camera on the touch panel possessed by the clerk to display the product name and the product image input by the purchaser. It is configured so that correctness can be confirmed.

JP 2012-043371 A JP 2010-055460 A

The store system described in Patent Literature 1 stores goods in a paper box by a clerk, and a purchaser cannot grasp which product is stored in which position in the paper box. Therefore, since the purchaser must check a plurality of products on the image on the display screen, it takes time to perform the checking operation.

(4) In the store system described in Patent Document 2, the purchaser checks out the merchandise by himself / herself and pays for the merchandise. However, since the barcode reading process is basically required, the operation of holding the barcode over the barcode scanner when the position of the barcode is difficult to understand is complicated for the elderly and the like. In the case of products without barcodes, it is necessary for the purchaser to enter the product name on the touch panel, which is troublesome, and the product image taken by the camera is displayed on the touch panel owned by the clerk, and the product entered by the purchaser The operation of checking the correctness of the name and the product image is also very complicated.

Therefore, there is a need for a product placement device, a product placement method, a product placement system, and a product placement program that are easy for the operator to use in self-checkout and the like.

The characteristic configuration of the product placement device according to the present invention includes a setting unit that sets a plurality of areas into which a placement area for placing a product is divided, in association with specific information for specifying each area. And a notifying unit for notifying the specific information set by the setting unit.

Further, a feature of the product placement method according to the present invention is that a setting step of setting, in association with a plurality of areas into which a placement area for placing a product is divided, identification information for identifying each area, is set. And a notifying step of notifying the specific information set in the setting step.

In addition, the characteristic configuration of the product placement system according to the present invention is such that setting is performed in such a manner that a plurality of areas into which a placement area for placing a product is divided are associated with specific information for specifying each area. And a notifying unit for notifying the specific information set by the setting unit.

Further, the characteristic configuration of the product placement program to be executed by the computer according to the present invention is such that, for a plurality of areas into which the placement area for placing the product is divided, specific information for specifying each area is specified. The present invention is characterized in that it comprises a setting process for setting the information in association with each other, and a notifying process for notifying the specific information set in the setting process.

According to the above configuration or method, the specification information for specifying each area is set in association with the plurality of areas in which the arrangement areas for arranging the products are divided. For this reason, when goods are arranged in each area, each area where goods are arranged can be clearly distinguished by specific information.

Further, in the above configuration or method, since the specific information for specifying each area is notified, it is easy to check each area in which the arrangement area is divided, and the area in which the operator has arranged the product is not limited to any area. Can be easily grasped. In particular, in the case where the product placement device is provided at the self-checkout, if the placement area is imaged by the camera, it is not necessary to hold the products one by one over the barcode, which is convenient for the purchaser. As described above, it is possible to provide a product placement device, a product placement method, a product placement system, and a product placement program that are easy for the operator to use at a self-checkout or the like.

特徴 Another characteristic configuration is that the arrangement area is a placement surface on which goods are arranged.

れば If the placement surface is the arrangement region as in this configuration, the operator can easily visually recognize the planar arrangement region. For this reason, goods can be easily arranged on a mounting surface based on the specific information of each area divided on the mounting surface, and it is easy for an operator to use.

Another characteristic configuration further includes a receiving unit that receives a specification of a method regarding the division of the plurality of areas, and the setting unit is configured to specify each of the plurality of areas according to the specification received by the receiving unit. The point is to set specific information.

As in this configuration, if the setting unit sets the specific information for specifying each of the plurality of areas according to the specification of the method regarding the plurality of area divisions received by the receiving unit, the area setting is appropriately performed according to the usage mode of the device. Highly convenient because it can be.

Another characteristic configuration further includes a recognition unit that recognizes products arranged in the plurality of areas for each area, and the notification unit notifies a result recognized by the recognition unit in association with the specific information. It is in.

When the recognition unit recognizes the product for each area and notifies the user of the recognition result in association with the specific information as in the present configuration, the operator can easily recognize the recognition result of the product arranged in each area of the arrangement area. be able to.

Another feature is that the notifying unit notifies in a different manner according to the result recognized by the recognizing unit.

In this configuration, since the notification is performed in a different manner according to the result recognized by the recognition unit, the case where the recognition unit can recognize the product and the case where the recognition unit cannot recognize the product are clearly distinguished from each other by the operator. Can be notified.

Another characteristic configuration further includes a detection unit that detects an arrangement location of the commodity arranged in the arrangement area, and the setting unit, according to the arrangement location of the commodity detected by the detection unit, each of the plurality of areas Is to set the specific information for specifying the information.

The setting unit of this configuration sets the specific information for specifying each of the plurality of areas according to the location of the product detected by the detection unit, so that the product can be placed in any location of the placement area. Is notified of the appropriate location. For this reason, the operator can arrange the goods in the arrangement area at random regardless of the size and shape of the goods, which is highly convenient.

Another feature is that the notifying unit notifies the arrangement area of the specific information in association with the arrangement information.

If the specific information for specifying each area is notified in association with the arrangement area itself as in the present configuration, it is possible to arrange the product while confirming the specific information for each area in the arrangement area. Cheap.

Another characteristic configuration further includes: an acquisition unit that acquires an image of the arrangement region, and a display unit that displays the image acquired by the acquisition unit, and the notification unit is configured to execute the arrangement region through the display unit. The specific information is displayed in association with the image.

If the specific information is displayed in association with the image of the arrangement area via the display unit as in the present configuration, the specific information of each area in the arrangement area can be easily confirmed.

The characteristic configuration of the product placement device according to the present invention is an acquisition unit that acquires an image including a placement region for placing a product, and a division unit that divides the placement region included in the image into a plurality of divided regions. And a detector for detecting a product included in the divided area based on the similarity information of the divided area.

According to this configuration, since the similarity (for example, normalized correlation coefficient) information is used when detecting the product by the detection unit, the product can be detected stably with respect to illumination fluctuation. As a result, a product placement device with high product detection accuracy and easy for the operator to use is provided.

特徴 Another feature is that the arrangement area is provided with a predetermined pattern.

(4) If a pattern (pattern, color, etc.) that is hardly used for a product as a predetermined pattern is used in the arrangement area as in the present configuration, the detection accuracy of the product is improved.

1 is an overall schematic diagram of a product management system. It is a block diagram of a goods management system. It is a figure showing an example of an arrangement field. It is a figure showing an example of an arrangement field. It is a figure showing an example of an arrangement field. It is a figure showing the example of a display of a touch panel. It is a figure showing the example of a display of a touch panel. It is a flowchart of an area designation process. It is a flowchart of an area change process. It is a flowchart of a purchase process. It is a block diagram of a goods management system concerning another embodiment. It is a block diagram of a goods management device concerning another embodiment. It is a key map showing the detecting method of goods concerning other embodiments.

Hereinafter, embodiments of a product placement device, a product placement method, a product placement system, and a product placement program according to the present invention will be described with reference to the drawings. In the present embodiment, as a product placement device, a product placement method, a product placement system, and a product placement program, a purchaser (operator) operates without intervention of a clerk at a small convenience store installed in a hospital, a nursing home, an office building, or the like. An example used in a self-checkout system X (an example of a product placement system) in which a person performs accounting by himself / herself will be described. However, without being limited to the following embodiments, various modifications can be made without departing from the scope of the invention.

As shown in FIG. 1, the self-checkout system X includes a plurality (two in the present embodiment) of self-purchase devices 1 (an example of a product placement device) having a terminal 14 for a purchaser, and a host server 2. I have. The purchaser's terminal 14 and the upper server 2 are connected to each other by a wired or wireless network, and are configured so that sales information from the self-purchase device 1 is transmitted to the upper server 2.

The self-purchase device 1 includes a base 10, an imaging unit 11 (an example of an acquisition unit) including a visible light camera, a distance sensor 12, an embedding tray 13, and a terminal 14 for a purchaser. I have. The purchaser's terminal 14, the imaging unit 11, the distance sensor 12, and the embedding tray 13 are network-connected by wire or wirelessly.

The base 10 is a housing that supports the imaging unit 11, the distance sensor 12, the embedding tray 13, and the terminal 14 for the purchaser. A shopping basket (not shown) can be placed beside the embedding tray 13. A space 10A is provided. The base 10 also has a built-in money storage unit (not shown) for storing money to be changed.

The imaging unit 11 is a visible light camera including an imaging device such as a CCD or a CMOS. The imaging range of the imaging unit 11 is set to a range where the entire surface of the embedding tray 13 can be photographed. The imaging unit 11 photographs a product placed on the surface of the embedding tray 13 and transmits the image to a recognition unit 14C described later. The recognition unit 14C recognizes a product based on the received image using a known object authentication technology. The distance sensor 12 is a sensor that emits light from a light source such as an LED, receives reflected light reflected by a product, and converts the reflected light into a distance. The distance sensor 12 can measure the height of a product placed on the surface of the embedding tray 13 and transmits the measured value to a recognition unit 14C described later. The recognizing unit 14C compares the received measurement value with the height of the registered product stored in the storage unit 14G, which will be described later, and recognizes a state in which a plurality of products are arranged in an overlapping manner.

配置 An arrangement area for arranging (transferring) the goods stored in the shopping basket by the purchaser's hand is formed on the surface (an example of the mounting surface) of the embedding tray 13. The embedding tray 13 in the present embodiment is embedded in the base 10 so that the plane-shaped arrangement area is visible to the purchaser. Although the mode of the embedding tray 13 will be described in detail later, the surface of the embedding tray 13 is divided into a plurality of areas, and specific information (numbers and frames) for specifying each area is visually recognized by the purchaser. It has a notifying unit 13A for making the notification (see FIG. 2). In the present embodiment, the notification unit 13A displays each area divided on the embedding tray 13 by a number as shown in FIG. 3 or lights up a frame by an LED as shown in FIGS. Or Based on the specific information notified to the purchaser by the notification unit 13A, the purchaser arranges one product in each area.

As shown in FIGS. 1 and 2, the buyer terminal 14 includes a setting unit 14A, a notification control unit 14B, a recognition unit 14C, and a touch panel 14D (an example of a reception unit, an example of a display unit, and an example of a notification unit). It has a unit 14E, a detection unit 14F, a storage unit 14G, a speaker 14H, and a barcode reader 14I. The setting unit 14A, the notification control unit 14B, the recognition unit 14C, and the detection unit 14F are configured by software that is various programs executed by the CPU. The touch panel 14D, the settlement unit 14E, the speaker 14H, and the barcode reader 14I The storage unit 14G is configured by hardware such as a RAM and an HDD.

(4) The setting unit 14A sets a plurality of areas in which the arrangement areas for arranging commodities are divided in association with specific information for specifying each area. The arrangement area is the surface of the above-described embedding tray 13, and a plurality of areas divided with respect to the embedding tray 13 are set in association with a number, a frame, and the like as specific information. 3 to 5 show specific examples of the display method relating to the specific information set by the setting unit 14A.

FIG. 3 shows the “area display method”. On both sides of the embedding tray 13, a plurality of line segments for dividing each area are drawn at equal intervals, and a number serving as an index of each area is provided between the pair of line segments (this embodiment). 1 to 8) are displayed in order at the center. The central portion between both sides of the embedding tray 13 is blank so that commodities may protrude from each area into the central portion. It is preferable that the “area display method” is stored in advance in the storage unit 14G as a template, and is selected by an administrator (operator) on the touch panel 14D (see FIG. 7). As the “manual input”, the administrator may digitally input the division line using a touch pen, or the administrator may input the division line on the surface of the embedding tray 13 (an example of a receiving unit) using magic or tape. The setting unit 14A may set the specific information for specifying each area based on the image captured by the imaging unit 11.

FIG. 4 shows the “fixed frame LED lighting method”. On both sides of the embedding tray 13, a plurality of fixed frames for dividing each area are set at equal intervals, and these fixed frames are lit by LEDs. It is preferable that the “fixed frame LED lighting method” is stored in advance in the storage unit 14G as a template and is selected by the administrator on the touch panel 14D (see FIG. 7). Further, as “manual input”, the administrator may digitally input a fixed frame with a touch pen.

FIG. 5 shows the “variable area lighting method”. A grid-like virtual line (dashed line in the figure) is drawn on the embedding tray 13 so that the setting unit 14A specifies each area according to the location of the product detected by the detection unit 14F. Set the specific information of When setting the specific information, the setting unit 14A specifies a frame surrounded by the closest virtual line among the virtual lines outside the contour of the product detected by the detection unit 14F. Then, the inner area of the frame is each area divided with respect to the arrangement area where the product is arranged, and the setting unit 14A turns on the frame serving as the specific information for specifying each area with the LED. . In the "variable area lighting method" shown in FIG. 5, since the specific information for specifying each of the plurality of areas is set in accordance with the location of the product detected by the detection unit 14F, the product can be placed in any location of the location area. Even when the products are placed, the appropriate placement location of the product is notified. This allows the purchaser to randomly place the product on the surface of the embedding tray 13 regardless of the size and shape of the product, which is highly convenient.

As described above, the recognition unit 14 </ b> C recognizes a product arranged in each area of the embedding tray 13 based on an image captured by the imaging unit 11 using a known object recognition technology. Here, “recognizing a product” means to recognize a product ID, a product classification and a product name associated with the product ID, etc., for the products arranged in each area of the embedded tray 13. A detailed description of the known object recognition technology is omitted, but the target product feature amounts such as the contour, color and surface shape of the object, and characters attached to the object, and the product information (registered product feature) stored in the storage unit 14G (Agreement, 100%, perfect match if 100%). If the score is equal to or more than a predetermined threshold (for example, 90%), the recognition unit 14C determines that the product is recognized by the recognition unit 14C (normally recognized product), and if the score is less than the predetermined threshold, the recognition unit 14C recognizes the product. It is determined as a product not recognized by the unit 14C (unknown product). For recognition as an unknown product by the recognition unit 14C, it is also possible to recognize a state in which a plurality of products are arranged in an overlapping manner by the measured value of the height of the product arranged on the surface of the embedded tray 13 by the distance sensor 12. included. Recognition as an unknown product by the recognition unit 14C also includes recognizing that a plurality of products are arranged in one area.

The notification control unit 14B controls the notification mode of the notification unit 13A of the embedded tray 13 and the notification mode of the touch panel 14D. The notification control unit 14B controls the notification to the purchaser in different modes according to the result recognized by the recognition unit 14C. For example, in the example of FIG. 3, it is assumed that the second and third products are the products recognized by the recognition unit 14C (normally recognized products), and OK is displayed on the touch panel 14D as shown in FIG. As an unrecognized product (unknown product), NG is displayed on the touch panel 14D as shown in FIG. In the example of FIGS. 4 and 5, the product (unknown product) that is not recognized by the recognition unit 14C is compared with the lighting color (for example, blue) of the frame surrounding the product (normally recognized product) recognized by the recognition unit 14C. The surrounding frame is highlighted with different lighting colors (for example, red). 4 and 5 are displayed on at least one of the notification unit 13A of the embedding tray 13 and the touch panel 14D.

In the case of an unknown product, the notification control unit 14B sends the sound of the speaker 14H so that the product is placed again in a predetermined area or the barcode attached to the product is read by the barcode reader 14I. The control to notify the purchaser on the screen display of the touch panel 14D is executed. Further, the notification control unit 14B may perform notification control so that a candidate product is displayed on the touch panel 14D and the purchaser is selected.

The touch panel 14D is a display device that can be operated on a screen by a purchaser or an administrator with a finger or a touch pen. The touch panel 14D functions as a receiving unit that receives a designation of a plurality of areas provided on the surface (arranged area) of the embedding tray 13 and a notification unit that notifies a result recognized by the recognition unit 14C in association with specific information. And a function as a display unit that displays an image captured by the imaging unit 11.

FIG. 7 shows a display screen of the reception unit (area designation), and one of “area display method”, “fixed frame LED lighting method”, “variable area LED lighting method”, and “manual input”. This is a screen for accepting selection of a method. The function of the reception unit is input by the administrator when the self-checkout system X is introduced into the store, and the setting unit 14A sets the specific information for specifying each of the plurality of areas according to the input specification. In addition, when the self-checkout system X is operated, if the user wants to change the area division, the administrator operates the function of the receiving unit of the touch panel 14D again. The touch panel 14D may be provided with an "area change" function, and the "area change" includes a process of integrating or dividing a plurality of areas.

FIG. 6 shows a display screen of a touch panel 14D as a notification unit corresponding to FIG. 3 (area display method), and a screen for notifying (displaying) a result recognized by the recognition unit 14C in association with specific information. is there. As the recognition result, No. 2 and No. 3 as the specific information are OK displayed as the products recognized by the recognition unit 14C (normally recognized products), and No. 7 as the specific information is not recognized by the recognition unit 14C. (Unknown product) is displayed as NG. At this time, as the function of the display unit, the specific information (number) is displayed in association with the image of the surface (arrangement area) of the embedding tray 13 captured by the imaging unit 11, and it is determined that the seventh product is an unknown product. It is highlighted with a frame. In addition, regarding the 7th product that is an unknown product, the product is placed on a predetermined area again, or the barcode attached to the product is read by the barcode reader 14I on the screen display of the touch panel 14D. Notify buyers. Although not shown, FIGS. 4 (fixed frame LED lighting method) and FIG. 5 (variable area LED lighting method) also correspond to images of the surface (arranged area) of the embedded tray 13 captured by the imaging unit 11. Alternatively, the specific information may be displayed on the touch panel 14D.

The settlement unit 14E includes a money handling device, a card reader, and the like. When all of the products are recognized by the recognition unit 14C or the barcode reader 14I, a product name and a price corresponding to the recognized product and associated with the product ID stored in the storage unit 14G are acquired. Then, as shown in No. 2 and No. 3 in FIG. 6, the product name, the quantity and the amount (including the total amount) are displayed on the touch panel 14D, and the purchaser confirms the touch panel 14D and presses a checkout button (not shown). By doing so, the settlement unit 14E accepts payment by money, a credit card, or the like. After the payment is completed, a receipt is printed and issued by a printer (not shown), and change and the like are refunded. Note that the settlement method in the settlement unit 14E may be a method of displaying a barcode on the screen of a smartphone to make a payment, a method of making a payment by biometric authentication (such as face authentication using a camera), or the like.

The detection unit 14 </ b> F detects a placement location of a product placed on the surface (placement area) of the embedding tray 13 in the “variable area lighting method” illustrated in FIG. 5. Specifically, the detection unit 14F detects the outline of the product from the image captured by the imaging unit 11. Then, the detection unit 14F detects the position coordinates of the contour of the product on the surface (arrangement area) of the embedding tray 13. Also, in the “area display method” shown in FIG. 3 and the “fixed frame LED lighting method” shown in FIG. 4, the detection unit 14F detects the location of the product placed on the surface of the embedded tray 13 and sets the setting unit. 14A is configured to execute an area change described later.

The storage unit 14G stores various programs, and stores a product classification, a product name, a unit price, and image data (including a registered product feature amount) in association with a product ID. Further, the storage unit 14G stores a display format template related to the specific information set by the setting unit 14A. Further, the storage unit 14G stores the height of each product direction, and the recognition unit 14C stores the height of the product measured by the distance sensor 12 and the height of the registered product stored in the storage unit 14G. By comparing with the above, it can be recognized whether or not it is one product.

As shown in FIG. 2, the host server 2 includes a sales information storage unit 21, a sales management control unit 22, and a display device 23.

The sales information storage unit 21 stores the sales information of the product transmitted from the purchaser's terminal 14. As the sales information of the merchandise, a merchandise classification, a merchandise name, a sales amount, a sales quantity, and the like associated with the merchandise ID of the purchased merchandise are stored. The sales management control unit 22 includes a CPU that executes sales registration control on the sales information storage unit 21 and executes display control on the screen of the display device 23. The display device 23 is a display device that displays a screen for sales management by an administrator.

Next, referring to FIG. 8, a description will be given of a process when the administrator specifies a method of a plurality of areas on the surface (arranged area) of the embedding tray 13 before use by the purchaser.

(4) When the administrator selects the area designation from the initial screen (not shown) of the purchaser terminal 14 (# 81 Yes), the screen shifts to the screen of the touch panel 14D shown in FIG. 7 (# 82). Next, when the administrator selects one of the prescribed templates (“area display method” or “fixed frame LED lighting method”) from the area designation methods shown in FIG. 7 (# 83 Yes), the setting unit 14A sets A plurality of areas are automatically set on the surface (arrangement area) of the loading tray 13 and specific information (number or frame) for specifying each area is set in association with the plurality of areas (setting step, Setting processing, # 84 to # 85). When the administrator selects the “variable area LED lighting method” (# 83 Yes), the setting unit 14A determines the detection result of the detection unit 14F when the product is arranged on the surface (arrangement area) of the embedding tray 13. And automatically sets a plurality of areas based on the information, and sets the plurality of areas in association with specific information (number or frame) for specifying each area (setting step, setting processing, # 84 to # 85) ). Next, the notification unit 13A controlled by the notification control unit 14B notifies the specific information (number or frame) set by the setting unit 14A as shown in FIGS. 3 to 5 (notification step, notification processing, # 86).

On the other hand, if the administrator selects “manual input” from the area designation method shown in FIG. 7 (No in # 83), the administrator must divide the surface (arranged area) of the embedding tray 13 into a plurality of areas. Performs digital input of division lines on the touch panel 14D with a touch pen, or analog input of division lines on the surface of the embedding tray 13 with magic or tape (# 87). When a dividing line is digitally input to the touch panel 14D, the setting unit 14A sets a plurality of areas in association with specific information (number or frame) for specifying each area, and sets the embedding line. When the dividing line is input to the surface of the tray 13 by magic, tape, or the like, identification information for identifying each area for a plurality of areas based on an image of the dividing line captured by the imaging unit 11 (Setting number, setting process, # 88). Next, the notification unit 13A (embedded tray 13) and the touch panel 14D controlled by the notification control unit 14B transmit the specific information (number or frame) set by the setting unit 14A, as shown in FIGS. Notification (display on the embedded tray 13) (notification step, notification processing, # 89).

As described above, if the setting unit 14A sets the specific information for specifying each of the plurality of areas according to the specification regarding the plurality of area divisions received by the touch panel 14D as the receiving unit, the setting unit 14A can set the specific information according to the usage form of the self-checkout system X. Since the area can be set appropriately, the convenience is high. By the way, a purchaser arranges products one by one in each area designated on the surface (arrangement area) of the embedding tray 13 and performs a purchase procedure, but an inexperienced purchaser purchases products over a plurality of areas. May be placed. Therefore, a process of automatically changing each area after the area designation process is completed will be described with reference to FIG.

As a result of imaging the product by the imaging unit 11 and detecting the product by the detection unit 14F (# 70), if the purchaser arranges the product over a plurality of areas (# 71 Yes), the setting unit 14A changes the area. (Area change step, area change processing, # 72). Next, specific information (number or frame) for specifying each changed area is associated and reset (resetting step, resetting process, # 73). In this area change and resetting, for example, when a product is arranged across No. 6 and No. 7 shown in FIG. 3, No. 6 and No. 7 are integrated and the specific information is reset as No. 6, and No. 7 is set as specific information. Then, the notification unit 13A (embedded tray 13) and the touch panel 14D controlled by the notification control unit 14B re-notify the specific information (number or frame) reset by the setting unit 14A (re-notification step, re-notification). Notification process, # 74).

Next, a purchase process in which a purchaser purchases a product will be described with reference to FIG.

The notification unit 13A of the embedding tray 13 notifies (displays on the embedding tray 13) the specific information set by the setting unit 14A as described above (# 91). The purchaser who put the product in the shopping basket places the shopping basket in the space 10A of the base 10, and places the product in each area notified by the notification unit 13A of the embedded tray 13 (# 92 Yes). Next, based on the detection result of the detection unit 14F based on the image of the product captured by the imaging unit 11, when the product is arranged over a plurality of areas (# 93 Yes), the notification of the embedding tray 13 is performed as described above. Unit 13A re-notifies the specific information (number or frame) reset by setting unit 14A (# 94). Next, when the product is properly arranged in each area, and when the product is arranged across a plurality of areas and the specific information (number or frame) is notified again, the image is taken by the imaging unit 11. The recognition unit 14C recognizes the product based on the product image (# 95). Then, the recognition result by the recognition unit 14C is notified in association with the specific information (# 96). In the example of FIG. 3, as shown in FIG. 6, a number, a product name, an amount, and a recognition result (OK, NG) corresponding to the display on the notification unit 13A of the embedding tray 13 are displayed (notified) on the touch panel 14D. ing. In the examples of FIGS. 4 and 5, the notifying unit 13A of the embedding tray 13 highlights and notifies a product (unknown product) not recognized by the recognizing unit 14C.

As described above, if the recognition unit 14C notifies the recognition result of the recognition of the product for each area in association with the specific information, the purchaser can easily grasp the type of the product placed in each area of the placement area. Can be. In addition, since the notification is performed in a different manner according to the result recognized by the recognition unit 14C, it is clear whether the recognition unit 14C can recognize the product type or the recognition unit 14C cannot recognize the product type. The buyer can be notified separately.

Next, when there is an unknown product (# 97 Yes), the product is repositioned one by one in the frame by the display on the touch panel 14D or the sound from the speaker 14H shown in FIG. An instruction is given as to whether the barcode is read by the barcode reader 14I. Next, when the purchaser reads the barcode attached to the product by holding it over the barcode reader 14I (# 98 Yes), the process returns to $ 96 again, notifies the recognition result in association with the specific information, and notifies the unknown product. It is determined whether or not there is (# 97). On the other hand, when the purchaser changes the direction of the product in the frame (# 98 No) and returns to $ 92 again, it is determined whether or not the product has been arranged in each area (# 92). The determination up to whether or not there is (# 97) is repeated.

If there is no unknown product and all products are normally recognized (No in # 97), the settlement unit 14E executes the settlement process (# 99). As described above, the notifying unit 13A of the embedding tray 13 notifies the specific information for specifying each area, so that it is easy to confirm each area where the arrangement area is divided, and the purchaser arranges the product. The user can easily recognize which area the selected area is. In particular, in the self-checkout, it is highly convenient because the purchaser can confirm the arrangement location of the commodities only by arranging a plurality of commodities in each area.

[Other Embodiments]
(1) As shown in FIG. 11, the upper server 2a is responsible for the functions of the setting unit 14A and the like in the above-described embodiment, and the buyer terminal 14a and the upper server 2a can be bidirectionally communicated by wire or wirelessly. May be configured. In the present embodiment, the purchaser terminal 14a includes a touch panel 14D, a settlement unit 14E, a speaker 14H, and a barcode reader 141 similar to those described above. The purchaser terminal 14a also has a program for operating the touch panel 14D, the settlement unit 14E, the speaker 14H, and the barcode reader 14I, and a CPU for executing the program, but is not shown.

The host server 2a includes a setting unit 24A, a recognition unit 24C, and a detection unit 24D having the same configuration as the setting unit 14A, the recognition unit 14C, and the detection unit 14F of the purchaser terminal 14. The host server 2a has a control unit 24B including the notification control unit 14B of the purchaser terminal 14 and the sales management control unit 22 of the host server 2 described above. Furthermore, the upper server 2a has a storage unit 24E including the storage unit 14G of the purchaser terminal 14 and the sales information storage unit 21 of the upper server 2 described above.

(2) FIG. 12 is a block diagram according to another embodiment of the purchaser terminal 14. The purchaser's terminal 14 according to the present embodiment may include the respective functional units according to the above-described embodiment, but only the functions necessary for the description are illustrated in the drawing. The purchaser terminal 14 according to the present embodiment includes a setting unit 14A, a dividing unit 14J, a feature amount extracting unit 14K, a similarity calculating unit 14L, a detecting unit 14F, and a storage unit 14G.

The setting unit 14A sets a plurality of areas into which the arrangement areas for arranging the products are divided, in association with specific information (number or frame) for specifying each area, and generates a reference frame. (Setting step, setting processing). In the reference frame shown in FIG. 13, an “area display method” similar to that of FIG. 3 described above is shown as an example. The division unit 14J divides the arrangement region included in the image into a plurality of meshes M (an example of a division region) based on the image including the arrangement region for arranging the products captured by the imaging unit 11 (partition). Step, division process). As an example, the dividing unit 14J divides the image into meshes M at regular intervals (for example, when an image of 500 × 400 pixels is divided into 50 × 40, each mesh M has 10 × 10 pixels) (see FIG. 13). The storage unit 14G stores a reference frame DB1 (an image including a placement area where no product is placed, a background image) to which the specific information set by the setting unit 14A is added before the product is placed in the placement area. The dividing unit 14J divides the reference frame DB1 stored in the storage unit 14G into a plurality of meshes M. In creating the reference frame DB1, a predetermined pattern is given to the arrangement area. Here, the predetermined pattern is a pattern (a pattern, a color, and the like) that is hardly used for a product.

The feature amount extraction unit 14K extracts feature amounts (brightness, color, etc.) of each mesh M divided by the division unit 14J (feature amount extraction step, feature amount extraction processing). The similarity calculation unit 14L compares the feature amount of each mesh M of the reference frame DB1 with the feature amount of each mesh M of the captured image corresponding to each mesh M of the reference frame DB1, and determines the similarity between the meshes M (for example, A normalized correlation coefficient) is calculated as similarity information (similarity calculation step, similarity calculation process). Here, for example, the higher the normalized correlation coefficient is, the higher the similarity between the meshes M is. The similarity calculating unit 14L calculates the similarity between the meshes M each time a new feature amount of the captured image is input, and stores the calculation result of the similarity in the storage unit 14G as the similarity information DB2.

The detection unit 14F detects a product included in the divided area based on the similarity information DB2 of the divided area (detection step, detection processing). As illustrated in FIG. 13, the detecting unit 14F captures the feature amount of each mesh M of the reference frame DB1 (the divided area of the image including the arrangement area where no product is arranged) and the imaging corresponding to each mesh M of the reference frame DB1. If the similarity information DB2 calculated by comparing the feature amount of each mesh M of the image (the divided area of the image including the arrangement area where the article is arranged) is equal to or smaller than a predetermined threshold, the article exists. It is determined that the mesh is M. Then, an aggregate area adjacent to each mesh M determined to have an object by the detection unit 14F is a product placement location. As a conventional technique for detecting a product area in an image, there is a method using a chroma key. However, in a place where the illumination changes due to sunlight or the like, the brightness or color of the background used for the chroma key fluctuates, so that only the product region cannot be extracted. In the present embodiment, by using a similarity (for example, a normalized correlation coefficient) that is not easily affected by a change in brightness, it is possible to stably detect a product area even in a situation where lighting varies. Further, since a predetermined pattern given to the arrangement area is used as a chroma key, the detection accuracy of a product can be improved. Here, the predetermined pattern is a pattern (a pattern, a color, and the like) that is hardly used for a product. Note that each functional unit of the present embodiment may be provided in the host server 2, and is not particularly limited.

(3) The arrangement area in which the articles are arranged is not limited to the surface of the embedded tray 13 but may be the bottom of the shopping basket. In this case, the setting line 14A is provided based on the image captured by the imaging unit 11 when the shopping basket is placed on the embedding tray 13 by providing a dividing line and specific information (number and the like) on the bottom surface of the shopping basket. Set the specific information for specifying the area. Further, the placement area where the products are placed may be a portable tray used in a company cafeteria or the like. In this case, the setting section 14A specifies each area based on the image captured by the image capturing section 11 when the partition line and the specific information (number or the like) are provided on the surface of the tray and the tray is placed on the self-checkout register. Is set (it may be set in advance as a common tray), and the employee performs settlement in the settlement unit 14E.
(4) The distance sensor 12 may be built in the imaging unit 11 including a visible light camera. Alternatively, the embedding tray 13 may be configured to be transparent, the product may be raised by irradiating light from below, and the product may be three-dimensionally photographed with a visible light camera.

(5) The recognizing unit 14C may determine the score using a multi-level threshold, and the notification control unit 14B may perform the notification control in a different manner according to the recognition result of the recognizing unit 14C. For example, if the value is equal to or more than the first threshold (for example, 90%), it is determined that the product is recognized by the recognition unit 14C (normally recognized product). The product may be selected by the purchaser, and if less than the second threshold, the barcode attached to the product may be read by the barcode reader 14I. Further, the threshold value may be different for each product.
(6) In the above-described embodiment, when the purchaser places the product over a plurality of areas, the setting unit 14A changes the area. Alternatively, when the purchaser arranges the product over a plurality of areas, the purchaser may issue an instruction to replace the product, or if most of the product (for example, 80% or more) is in one area. , The recognizing unit 14C may recognize it as being arranged in the one area.

(7) The notification mode of the notification unit may be any mode. For example, a blinking display may be used to highlight an unknown product, or an alphabet or the like may be used to display specific information. When notifying the specific information, the specific information may be projected onto the arrangement area using a projection device. Further, when notifying the specific information, the notification may be performed using sound.
(8) The barcode reader 14I may have a function of reading a QR code (registered trademark).
(9) The self-checkout system X may be composed of one self-purchase device 1 and the host server 2, or the function of the host server 2 may be built in the self-purchase device 1. In addition, the self-purchase device 1 is illustrated as an example of the product placement device. However, the product placement device may be configured by the embedded tray 13 and the terminal 14 for the purchaser, or the product placement device may be configured only by the terminal 14 for the purchaser. May be.
(10) The self-checkout system X is not limited to being applied to a small-scale convenience store, but can also be applied to a large-scale convenience store, a supermarket, and the like. Further, the product placement device, the product placement method, the product placement system, and the product placement program in the above-described embodiment can be applied to a manned cash register where a clerk intervenes.

The present invention is applicable to a product placement device, a product placement method, a product placement system, and a product placement program for placing a product.

1: Self-purchasing device (product placement device)
11: imaging unit (acquisition unit)
13: Embedded tray (placement area)
13A: notification unit 14A: setting unit 14C: recognition unit 14D: touch panel (notification unit, reception unit, display unit)
14F: detection unit 14J: division unit DB2: similarity information M: mesh (division region)
X: Self-checkout system (product management system)

Claims

A setting unit that sets, in association with a plurality of areas in which the arrangement areas for arranging the products are divided, identification information for identifying each area,
A notification unit for notifying the specific information set by the setting unit.
The product placement apparatus according to claim 1, wherein the placement area is a placement surface on which a product is placed.
The apparatus further includes a receiving unit that receives specification of a method regarding division of the plurality of areas,
3. The product placement device according to claim 1, wherein the setting unit sets the specifying information for specifying each of the plurality of areas according to the specification received by the receiving unit. 4.
A recognition unit that recognizes products arranged in the plurality of areas for each area,
4. The product placement device according to claim 1, wherein the notification unit notifies a result recognized by the recognition unit in association with the specific information.
5. The product placement device according to claim 4, wherein the notification unit performs notification in a different manner according to a result recognized by the recognition unit. 6.
The apparatus further includes a detection unit that detects an arrangement location of the product arranged in the arrangement area,
6. The apparatus according to claim 1, wherein the setting unit sets the identification information for identifying each of the plurality of areas according to a location of a product detected by the detection unit. Item placement device according to the item.
The product placement device according to any one of claims 1 to 6, wherein the notification unit notifies the placement area by associating the specific information with the placement area.
An acquisition unit that acquires an image of the arrangement region,
A display unit that displays the image obtained by the obtaining unit,
The product placement device according to any one of claims 1 to 7, wherein the notification unit displays the specific information in association with an image of the placement area via the display unit.
A setting step of setting, in association with a plurality of areas in which the arrangement area for arranging the products is divided, identification information for identifying each area,
A notification step of notifying the specific information set in the setting step.
A setting unit that sets, in association with a plurality of areas in which the arrangement areas for arranging the products are divided, identification information for identifying each area,
A notification unit for notifying the specific information set by the setting unit.
A setting process of setting a plurality of areas in which the arrangement areas for arranging the products are divided, in association with specific information for specifying each area;
And a notifying process of notifying the specific information set in the setting process.
An acquisition unit that acquires an image including an arrangement area for arranging products,
A dividing unit that divides the placement area included in the image into a plurality of divided areas;
A merchandise placement device comprising: a detection unit that detects a merchandise included in the divided area based on the similarity information of the divided area.
13. The apparatus according to claim 12, wherein a predetermined pattern is provided in the arrangement area.