JP2020009466A - Display state determination system - Google Patents

Abstract

To provide a display state determination system which can accurately specify the state of display of a product displayed on a display shelf by comparing product identification information of the displayed product and product identification information of the product on the product tag.SOLUTION: A display state determination system 1 uses a management terminal 2 and a taken image information input terminal 3. The taken image information input terminal 3 is a terminal for inputting image information of an image of a display shelf in a shop. The management terminal 2 includes: a taken image information input reception processing unit 20; a taken image information storage unit 21; a taken image information normal-positioning processing unit 22; a position specification processing unit 23; a displayed product recognition processing unit 24; a product tag recognition processing unit 25; a sample information storage unit 26; a commodity identification information storage unit 27; and a display situation determination processing unit 28.SELECTED DRAWING: Figure 1

Description

  The present invention compares the product identification information of the displayed product with the product identification information of the product in the product tag, and thereby accurately determines the display status of the product displayed on the display shelf. It relates to a judgment system.

  In various stores such as convenience stores and supermarkets, it is common to sell products placed on display shelves. Therefore, by displaying a plurality of products on the display shelf, even if one of the products is purchased, the same product can be purchased by another person. If the display of the goods displayed on the display shelf disappears, the sales opportunity of the goods is lost, which is not a preferable state.

  Therefore, it is required to manage the display status of the products, especially whether or not the products displayed on the display shelf are out of stock.

  In order to achieve this object, a method using a POS system introduced in a store can be considered. In the POS system, the purchase and sale of goods are managed in an integrated manner, so that the stock quantity of the goods can be properly grasped.

  However, it is not possible to manage the shortage of goods only from the POS system. The POS system can specify the number of stocks based on the number of purchases and sales of goods, but not all the stocks are lined up on display shelves and may exist in store warehouses. is there. In addition, there is a decrease in the number of commodities not via the POS system, for example, a decrease in commodities due to shoplifting. In this case as well, the situation is such that the merchandise is not actually displayed, even though it is present as the stock quantity of the POS system.

  For this reason, it is preferable to grasp from the actual display on the display shelf in order to appropriately grasp the display state of the product (during display, out of stock, etc.), in particular, the presence or absence of the out of stock. In order to realize this, it is sufficient that the display shelf is photographed by the photographing device and the missing item can be identified from the photographed image information. In order to do this accurately, it is conceivable that humans manually specify the displayed product and the missing product from image information obtained by photographing the display shelf. However, it is burdensome and inefficient for a human to identify a missing product every time from image information obtained by photographing a display shelf.

  In addition to product shortages, identifying what products are on display is also an important issue for product manufacturers and sales companies in terms of product development strategies and sales strategies. . Also, the burden of manually specifying them is large and inefficient.

  Therefore, it is necessary to automatically identify the products displayed on the display shelves of the store and to grasp the display status of the products. For example, there is a method of using an image recognition technique for an image of a display shelf in a store based on a sample image of each product. For example, Patent Literatures 1 and 2 disclose the following prior arts. Patent Document 3 discloses a technique for identifying a missing item on a display shelf of a product.

JP-A-5-342230 JP-A-5-334409 International Publication 2012/029548

  The invention of Patent Literature 1 is a system for assisting a person who does not have knowledge of which display shelf to display a product on. For this reason, it is not possible to grasp the missing items of the products actually displayed. Patent Document 2 discloses a shelf layout support system that supports display of products, which is a system that supports input of product images. However, the system of Patent Document 2 only supports input of a product image when using the shelf layout support system. Even if this system is used, whether the product actually displayed is out of stock? , It is not possible to grasp the display status indicating whether the vehicle is being displayed.

  Patent Literature 3 is an invention in which, when there is an empty space in a display shelf, a product to be displayed in the empty space is specified, or when a product to be displayed on the display shelf is misplaced, the notification is made. In the case of the present invention, an image when all the products are arranged on the display shelf is taken in advance, and a difference from the current image is obtained to specify the position of the empty space, or to specify the position of the shelf board. When a color or pattern is captured, it is identified as an empty space. However, since the products displayed on the display shelves include replacement of the products, the difference between the current image and the time when all the products are arranged can be reduced to the amount of free space. There is no mention or suggestion of whether it can be specified or its specific processing. For example, if there is a change in the product displayed between the first image taken and the second image taken, a difference in the image will occur, of course. If the user does not know, it is unclear for a person skilled in the art how to specifically process the difference to make it recognized as an empty space, and there is no feasibility at all. Even when the color or pattern of the shelf is recognized, the area near the back of the shelf is often dark, and the color information of the shelf of the shelf often changes. For this reason, it is extremely difficult to recognize the empty space as an actual computer process simply by capturing the color and pattern of the display shelf. In this case, there is no feasibility for those skilled in the art.

  Therefore, even if Patent Literature 3 is used, it is not possible to actually determine whether a product displayed on a display shelf is out of stock or is being displayed.

  In view of the above problems, the present inventor compares the product identification information of the product displayed on the display shelf with the product identification information of the product on the product tag to determine the display status of the product displayed on the display shelf. We have invented a display situation determination system that can specify with high accuracy.

  The invention according to claim 1 is a display status determination system that determines a display status of a product, wherein the display status determination system includes a display product recognition processing unit that recognizes product identification information of a product displayed on a display shelf. A product tag recognition processing unit for recognizing product identification information of a product tag attached to the display shelf, and comparing product identification information of the displayed product with product identification information of the product tag, A display status determination system that determines a display status of the product.

  According to the present invention, the display status of the product is determined based on the product displayed on the display shelf and the product tag, so that the display status of the product can be accurately determined.

  In the above invention, the displayed product recognition processing unit recognizes the product identification information of the product displayed on the display shelf by determining the similarity between the image information in which the display shelf is captured and the sample information. The system can be configured like a display situation determination system.

  There are various methods for recognizing the product identification information of the products displayed on the display shelf. As in the present invention, the recognition can be performed based on image information obtained by photographing the display shelf. In this case, by performing a matching process between the image information showing the display shelf and the sample information and determining the similarity, it is possible to determine the product identification information of the product displayed on the display shelf.

  In the above invention, the displayed product recognition processing unit may determine that the similarity between the image information showing the display shelf and the sample information having the highest similarity with the sample information corresponds to a predetermined threshold value. Otherwise, the system can be configured as a display situation determination system that does not recognize the product identification information.

  Since the matching process is performed between the image information showing the display shelf and the sample information, the similarity can always be derived. However, if the similarity is low, it cannot be said that the reliability is high. Therefore, when the similarity is not equal to or greater than the predetermined threshold, by not using the result of the matching process, it is possible to improve the accuracy of determining the display status of the product.

  In the above invention, the display product recognition processing unit is configured to read the display shelf based on product identification information read by a predetermined reading device and stored in an electronic tag attached to a product displayed on the display shelf. Can be configured as a display status determination system that recognizes the product identification information of the product displayed in the product.

  In order to recognize the displayed product, in addition to the process using the image information as described above, an electronic tag such as an IC tag may be attached to the product and read by a reading device. In the case of this method, it is possible to reduce erroneous recognition of the displayed product.

  In the above invention, the product tag recognition processing unit includes a product tag specification processing unit that specifies a product tag area in the image information of the display shelf, and an OCR in a part or all of the specified product tag area. It can be configured as a display situation determination system having a product tag in-tag information specifying processing unit that recognizes product identification information by performing recognition processing.

  There are various methods for recognizing the product identification information of the product tag. As in the present invention, the recognition can be performed based on image information obtained by photographing the display shelf. In this case, a process such as the present invention may be performed.

  In the above invention, the product tag specification processing unit can be configured as a display situation determination system that specifies a product tag area using brightness information of the image information.

  In order to recognize the product identification information of the product tag, it is necessary to specify the area of the product tag as accurately as possible. In general, since the color of the ground of a product tag is mostly white, when the display shelf is photographed, the portion of the product tag becomes bright. Focusing on this point, the product tag area is identified with high accuracy by specifying the product tag area using brightness information (information on the amount of brightness when image information is decomposed into hue, brightness, and saturation information). Can be identified.

  In the above invention, the product tag specification processing unit specifies the upper side position and the lower side position of the product tag by integrating the brightness information of the image information in the horizontal direction, and converts the brightness information of the image information in the vertical direction. By integrating, it is possible to configure a display situation determination system that specifies the left side position and the right side position of the product tag.

  In the above invention, the product tag identification processing unit identifies the rising edge and the falling edge of a histogram obtained by integrating the brightness information of the image information in the horizontal direction, thereby identifying the upper side position and the lower side position of the product tag. By determining the rise and fall of the histogram obtained by integrating the brightness information of the image information in the vertical direction, the left and right sides of the product tag are identified as a pair of the fall within a predetermined distance from the rise, and the display situation determination It can be configured like a system.

  When the product tag area is specified by focusing on the brightness information of the image information, the product tag area can be specified with high accuracy by integrating the brightness information of the image information in the vertical direction and the horizontal direction.

  In the above invention, the information-in-product-tag identification processing unit identifies the box by binarizing the identified product tag area and performing a labeling process, and among the identified boxes, boxes that satisfy a predetermined condition Is specified as a block, and the OCR recognition process is performed on the specified block.

  In order to recognize the information described on the product tag, OCR recognition processing may be performed on the entire product tag area. However, in that case, a large amount of noise or the like is detected, and accuracy may deteriorate. Connect. Also, the processing time increases. Therefore, it is preferable to execute the OCR recognition processing only on a part of the product tag area. Therefore, by executing the OCR recognition process on the block specified by the process of the present invention, the recognition accuracy can be improved.

  In the above invention, the product tag recognition processing unit performs edge detection in a product tag placement area of the image information, which is an area where the product tag may be attached, and satisfies a predetermined condition of a vertical angle. A product tag placement area normalization processing unit that detects a straight line group, specifies at least two straight lines from the detected straight line group, and executes a trapezoidal correction process using the specified two straight lines; It can be configured like a display situation determination system.

  The surface of the display shelf is vertical, whereas the surface of the product tag is often mounted more upwardly (toward the customer) than the vertical surface. This is to make it easier for the customer to visually recognize the product tag. Therefore, by performing the normalization process on the product tag placement area, the product tag area can be specified with higher accuracy. This is because, for example, the rise and fall of the hiss and gram in the brightness information become steep. Further, according to the present invention, the accuracy of the OCR recognition processing in the product tag area can be improved.

  In the above invention, the product tag recognition processing unit reads product identification information stored in a product tag attached to the display shelf or an electronic tag attached to the vicinity thereof, which has been read by a predetermined reading device, or has a coding function. It can be configured as a display situation determination system that recognizes product identification information in the product tag based on the provided product identification information.

  In order to recognize the product identification information of the product tag, in addition to the processing using the image information as described above, an electronic tag such as an IC tag is attached to the product tag or its vicinity and read by a reader. Thus, the product tag recognition processing unit may recognize the product identification information of the product tag. Further, the product tag recognition processing unit may recognize the product identification information of the product tag by reading the product tag and the coded product identification information attached to the vicinity thereof with the reading device. In the case of these methods, erroneous recognition of the product identification information of the product tag can be reduced.

  The invention according to claim 1 can be realized by causing a computer to read and execute the program of the present invention. In other words, the computer is configured to display a product identification processing unit for recognizing product identification information of a product displayed on a display shelf, a product tag recognition processing unit for recognizing product identification information of a product tag, and a product identification of the displayed product. A display status determination program that functions as a display status determination processing unit that determines display status of the product by comparing information with product identification information of the product tag.

  By using the display status determination system of the present invention, the display status of the products displayed on the display shelf can be accurately determined.

It is a block diagram showing typically an example of composition of a display situation judging system of the present invention. It is a block diagram showing typically an example of composition of a display goods recognition processing part in a display situation judging system of the present invention. It is a block diagram showing typically an example of composition of a goods tag recognition processing part in a display situation judging system of the present invention. FIG. 3 is a block diagram schematically illustrating an example of a hardware configuration of a computer used in the display status determination system of the present invention. It is a flow chart which shows an example of the whole processing process in the display situation judging system of the present invention. FIG. 4 is a diagram illustrating an example of captured image information. FIG. 4 is a diagram illustrating an example of captured image information. FIG. 7 is a diagram illustrating an example of normal image information obtained by performing a trapezoidal correction process on the captured image information of FIG. 6. FIG. 8 is a diagram illustrating an example of normal image information obtained by performing a trapezoidal correction process on the captured image information of FIG. 7. FIG. 4 schematically shows a state in which an input of designation of a shelf area and a product tag placement area is received with respect to normal image information obtained by normalizing image information of a display shelf in which products such as beverage cans are displayed. FIG. The state where the input of the designation of the shelf area and the product tag placement area has been received for the normal image information obtained by normalizing the image information obtained by photographing the display shelf on which the product such as the toothbrush is suspended and displayed. It is a figure which shows typically. FIG. 4 is a diagram illustrating an example of sample information stored in a sample information storage unit. It is a figure which shows typically the relationship between a place and a face in a display shelf. It is a figure which shows an example of the image information of the product tag arrangement | positioning area | region which was set to the image information of the product tag arrangement | positioning area of FIG. It is a figure which shows an example of the image information of the goods tag arrangement | positioning area | region of FIG. It is a figure which shows typically the process which specifies the upper side position and the lower side position of a goods tag area | region in a goods tag specification processing part. It is a figure which shows typically the process which specifies the position of the goods tag area | region in the horizontal direction in a goods tag specification processing part. It is a figure which shows typically the state which specified the goods tag area | region from the image information of the goods tag arrangement | positioning area of the normal position of FIG. It is a figure which shows typically the image information which binarized the goods tag area | region in the process in the information specification processing part in goods tags. It is a figure which shows typically the state which generated the box in the process in the information identification processing part in goods tag. It is a figure which shows typically the state which generated the block in the process in the information identification processing part in goods tag. It is a figure which shows an example of the product name described in the product tag, the product name with the shortest edit distance among the product names stored in a product dictionary as a result of OCR recognition, and the edit distance. It is a figure showing an example of correspondence of edit distance which may be decided for every character number of the last candidate. It is a figure which shows typically an example of the process which links the goods identification information recognized by the display goods recognition processing part with the goods identification information of the goods tag recognized by the goods tag recognition processing part. FIG. 4 is a diagram illustrating an example of captured image information. FIG. 26 is a diagram illustrating an example of normal image information obtained by performing a trapezoidal correction process on the captured image information of FIG. 25. FIG. 4 is a diagram schematically illustrating a correspondence relationship between captured image information and depth information. It is a figure showing an example of depth information. FIG. 9 is a diagram schematically illustrating a state in which a conversion process similar to a normalization process such as a trapezoidal correction process in captured image information is also performed on depth information. FIG. 29 is a diagram schematically illustrating a state in which the same conversion processing as the normalization processing such as the trapezoid correction processing in the captured image information has been performed on the depth information in FIG. 28. It is a figure which shows typically an example of the depth from the regression plane of a place about each place. It is a figure which shows typically the process which calculates | requires the regression plane of a place position. It is a figure which shows an example of the process which makes a determination based on the change of the state of the place between the last time (N-1 times) and this time (N times), and the change of the product tag. FIG. 19 is a diagram schematically illustrating an N-th process in a product identification information specifying processing unit according to a fifth embodiment.

  An example of the processing functions of the display status determination system 1 of the present invention is shown in block diagrams in FIGS. The display situation determination system 1 uses a management terminal 2 and a captured image information input terminal 3. FIG. 1 is a block diagram showing the overall function of the display status determination system 1, FIG. 2 is a block diagram showing the function of a displayed product recognition processing unit 24 described later, and FIG. 3 is a product tag recognition processing unit 25 described later. FIG. 4 is a block diagram showing the function of (1).

  The management terminal 2 is a computer used by an organization such as a company that operates the display status determination system 1. Further, the photographed image information input terminal 3 is a terminal for inputting image information of a photographed display shelf of a store.

  The management terminal 2 and the captured image information input terminal 3 in the display status determination system 1 are realized using a computer. FIG. 4 schematically shows an example of a hardware configuration of a computer. The computer includes an arithmetic device 70 such as a CPU for executing arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk for storing information, a display device 72 such as a display for displaying information, and a computer capable of inputting information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 for transmitting and receiving processing results of the arithmetic device 70 and information stored in the storage device 71 via a network such as the Internet or a LAN.

  When the computer has a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, for example, but are not limited thereto.

  The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be directly performed on the display by a predetermined input device (such as a touch panel pen) or a finger.

  The photographed image information input terminal 3 may include a photographing device such as a camera in addition to the above devices. As the captured image information input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, and a tablet computer can be used.

  Each means in the present invention is only logically distinguished in its function, and may physically or virtually have the same area. The processing order in each means of the present invention can be appropriately changed. Further, a part of the processing may be omitted. For example, a normalization process described later can be omitted. In that case, it is possible to execute processing on image information that has not been subjected to normalization processing.

  When determining the display status of a product based on captured image information, the display status determination system 1 determines the captured image information input reception processing unit 20, the captured image information storage unit 21, the captured image information normalization processing unit 22, and the position identification. It has a processing unit 23, a displayed product recognition processing unit 24, a product tag recognition processing unit 25, a sample information storage unit 26, a product identification information storage unit 27, and a display status determination processing unit 28. Therefore, as described later, when the image information is not used, the photographed image information input reception processing unit 20, the photographed image information storage unit 21, and the photographed image information normalization processing unit 22 need not be provided. In the recognition processing section 24 and the product tag recognition processing section 25, the processing relating to the image information may not be provided.

  The photographed image information input reception processing unit 20 receives an input of image information (photographed image information) of a display shelf of a store photographed by the photographed image information input terminal 3 and stores the image information in a photographed image information storage unit 21 described later. From the photographed image information input terminal 3, it is preferable to receive input together with photographed date and time, store identification information such as a store name, image information identification information for identifying image information, and the like, in addition to the photographed image information. 6 and 7 show examples of the photographed image information. FIGS. 6 and 7 show captured image information in which there are three shelves on the display shelf, and products are displayed there.

  The photographed image information storage unit 21 stores the photographed image information received from the photographed image information input terminal 3, the photographing date and time, the store identification information, the image information identification information, and the like in association with each other. The photographed image information may be any image information to be processed by the present invention. In general, it is difficult to take a photographed object with the photographing object facing directly when simply photographing. Therefore, it is necessary to execute a correction process for correcting the photographing object to the facing state, for example, a trapezoidal correction process. Is good. When one display shelf is photographed with a plurality of images, image information obtained by synthesizing the image as one image information is also included. The image information after the distortion correction processing has been executed is also included in the captured image information.

  The photographed image information emplacement processing unit 22 corrects the photographed image information stored in the photographed image information storage unit 21 so that the photographing object is in a state of being directly opposed (foreground processing), for example, The normal image information that has undergone the trapezoidal correction processing is generated. The trapezoidal correction process is a correction process performed so that the shelf of the display shelf shown in the captured image information is horizontal and the product tag for the product displayed there is vertical. Normalization is to transform the image information so that the optical axis of the lens of the photographing device is taken along the direction perpendicular to the plane to be photographed so as to be the same as when photographed from a far distance. There is a correction process.

  The trapezoidal correction process executed by the captured image information normalization processing unit 22 receives an input of designation of four vertices in the captured image information, and executes the trapezoidal correction process using each vertex. The four vertices for receiving the designation may be the four vertices of the shelf of the display shelf or the four vertices of the shelf position of the display shelf. Alternatively, four vertices of a set of two or three shelves may be used. Any four points can be designated as the four vertices. FIG. 8 shows an example of the photographed image information of FIG. 6 and FIG. 9 shows an example of the photographed image information (normal image information) obtained by normalizing the photographed image information of FIG.

  The position specifying processing unit 23 performs a trapezoidal area where a product is likely to be placed, out of the in-place image information obtained by performing the trapezoidal correction processing on the captured image information in the captured image information in-place processing unit 22. (A shelf area) and an area where a product tag may be attached (a product tag placement area). In other words, the display shelves are shown in the photographed image information and the normal image information, and the display shelves have a shelf area in which the products are displayed and a possibility that a product tag for the products displayed there may be attached. There is a certain product tag arrangement area. Therefore, the shelf area and the product tag arrangement area are specified from the normal image information. As for the specification of the shelf area and the product tag arrangement area, the operator of the management terminal 2 may manually designate the shelf area and the product tag arrangement area, and the position specification processing unit 23 may accept the designation, The shelf area and the product tag placement area may be automatically specified for the second and subsequent times based on the information on the shelf area and the product tag placement area that have been manually input.

  FIG. 10 shows a state in which an input of designation of a shelf area and a product tag placement area is received with respect to normal image information obtained by normalizing image information obtained by photographing a display shelf on which a product such as a beverage can is displayed. Is schematically shown. Further, FIG. 11 shows the specification of the shelf area and the product tag placement area for the normal image information obtained by normalizing image information obtained by photographing a display shelf on which a product such as a toothbrush is suspended and displayed. 5 schematically shows a state in which an input has been received.

  The displayed product recognition processing unit 24 executes a process of recognizing a product displayed on a shelf in a display shelf shown in the normal image information. Details of the processing of the displayed product recognition processing unit 24 will be described later. In the process described later, the case where the displayed product recognition processing unit 24 recognizes the product identification information of the displayed product based on the photographed image information of the display shelf will be described. By attaching the stored IC tag (electronic tag) or the like and reading the product identification information of the displayed product from the IC tag, the product identification information of the displayed product may be recognized.

  The product tag recognition processing unit 25 performs a process of recognizing the product identification information described in the product tag in the product tag area of the display shelf shown in the normal image information. Details of the processing of the product tag recognition processing unit 25 will be described later. In the following description, the case where the product identification information is recognized by reading the character information written on the product tag will be described. The product identification information may be recognized by reading information (code information) in which product identification information such as a code, a two-dimensional code, and a color code is encoded. Alternatively, the product identification information may be recognized by reading the product identification information from a product tag or an IC tag (electronic tag) installed near the product tag. When reading the product identification information from the code information or the IC tag, the product identification information is read by a reader such as a code reader or a non-contact communication device 74, and the information read by the reader is recognized by the product tag recognition processing unit 25. I do.

  The sample information storage unit 26 stores sample information for identifying which product is displayed on the shelf of the display shelf shown in the normal image information. The sample information is image information obtained by photographing a product that may be displayed on a display shelf from a plurality of angles such as up, down, left, right, and oblique. FIG. 12 shows an example of the sample information stored in the sample information storage unit 26. FIG. 12 shows a case where canned beer is photographed from various angles as specimen information, but is not limited to canned beer. The sample information storage unit 26 stores the sample information and the product identification information in association with each other.

  The sample information storage unit 26 stores information extracted from the sample information together with or instead of the sample information and necessary for calculating similarity, for example, information of a pair of an image feature amount and its position. You may memorize it. It is assumed that the sample information also includes information necessary for calculating the similarity. In this case, the display product recognition processing unit 24 does not have to calculate the image feature amount for the sample information every time when performing the matching process between the image information of the area of the place described later and the sample information. Can be shortened.

  The product identification information storage unit 27 stores the product identification information of the product displayed in each place of the shelf of the display shelf or information indicating that the product is “empty”. For example, the product identification information is associated with the product identification information, and is associated with the photographing date / time information, the store information, the image information identification information of the photographed image information, the image identification information of the normal image information, and the place identification information for identifying the place. The information is stored in the information storage unit 27. FIG. 13 schematically shows the relationship between places and faces on a display shelf.

  The display status determination processing unit 28 determines the display status as to whether the product is being displayed or is out of stock. For example, when determining that a product is out of stock as a display situation, a product that can be recognized from the product tag but is not recognized as being displayed on the display shelf is determined to be out of stock. Specifically, among the product identification information recognized by the product tag recognition processing unit 25, the product identification information that is not in the product identification information recognized by the displayed product recognition processing unit 24 is replaced with the product identification information of the missing product. Is determined.

  For example, a missing item is determined by the following method. When the same product identification information is included in an adjacent place among the product identification information (product identification information of a face) for each place recognized by the display product recognition processing unit 24, they are regarded as one product group. Collect product identification information. In addition, among the product identification information recognized by the product tag recognition processing unit 25, the same product identification information is combined into one piece of product identification information. Then, the collected product identification information is compared with each other to determine whether there is matching product identification information. In this case, the order of the product identification information does not matter.

  Then, among the product identification information recognized by the product tag recognition processing unit 25, it is determined that the product identification information recognized by the displayed product recognition processing unit 24 is being displayed. Among the product identification information recognized by the displayed product recognition processing unit 24, the product identification information that is not included in the product identification information recognized by the product tag recognition processing unit 25 is determined to have no product tag. Further, among the product identification information recognized by the product tag recognition processing unit 25, the product identification information that is not included in the product identification information recognized by the displayed product recognition processing unit 24 is determined as the product identification information of the missing product. . At this time, if the place corresponding to the product tag of the product identification information of the product determined to be missing is “empty”, the product can be accurately determined as missing. Even if the place is not “empty”, it can be determined as a missing item. FIG. 24 schematically shows this processing.

  When recognizing the product identification information of the product displayed on the display shelf based on the captured image information, the displayed product recognition processing unit 24 includes a shelf area processing unit 241, a place specifying processing unit 242, and a product identification information specifying processing unit. 243 and a shelf image matching unit 244.

  The shelf area processing unit 241 identifies the image information of the shelf area identified by the position identification processing unit 23 as the shelf area image information. The shelf area processing unit 241 may actually cut out the image as image information, or may actually cut out the area by specifying the image information of the area without actually cutting out the image information. If the display shelf has a plurality of shelves, each is cut out as shelf area image information. The coordinates indicating the area of the shelf are the coordinates of the vertices for specifying the area, and may be, for example, four points, the upper right and lower left, and the upper left and lower right two points in the normal image information. Further, the coordinates are relative coordinates with respect to a predetermined location (for example, the upper left vertex of the display shelf) in the image information such as the display shelf in the normal image information.

  The place specifying unit 242 specifies a place area for each shelf in the shelf area in the normal image information. The place is an area where a product is placed, and it does not matter whether the product is placed or not. The size of the area of the place is the same or substantially the same as the size of the product to be placed there. The place where the product is actually displayed is referred to as “face”. A case where no product is displayed in the place is defined as “empty”. It is preferable that the place specifying processing unit 242 be divided into a first place specifying process and a second and subsequent place specifying processes.

  In the first place specification processing in the place specification processing unit 242, a place is specified in a range (preferably a rectangular area) configured by the coordinates of the shelf identified by the position specification processing unit 23. Specifically, constraints such as identifying narrow and narrow shadows between products in the shelf area, identifying repetitive patterns of images, identifying steps on the top side of the package, and having the same product width The area of the place is specified by specifying the break position based on the. As the place specifying process, any method can be adopted depending on the category of the product and the form of the product, and is not limited to the above. In addition, a correction input by a person in charge may be received for the automatically specified place. Further, an input of the position of the place may be received from the person in charge. The coordinates of the area forming the specified place are the coordinates of the place area in the normal image information, the shooting date and time information, the store information, the image information identification information of the shooting image information, the image identification information of the normal image information, and the place. It is managed in association with place identification information for identification. The coordinates indicating the area of the place are coordinates of vertices for specifying the rectangular area, and may be, for example, four points in the normal image information, coordinates of two points of upper right and lower left, upper left and lower right, and the like. Further, the coordinates are relative coordinates with respect to a predetermined location (for example, the upper left vertex of the display shelf) in the image information such as the display shelf in the normal image information.

  In general, one shelf often contains similar types of products. In that case, the size of the product is also substantially the same. Therefore, when the area of the place of one product is specified, the place may be specified by repeatedly setting the area on the shelf.

  The second and subsequent places specification processing by the place specification processing unit 242 is based on the coordinates of the area of the place specified by the previous (N-1) -th normal image information for the same shelf on the same display shelf. The coordinates of the area of the place specified by the (Nth) normal image information may be used.

  The product identification information specification processing unit 243 specifies, for each shelf of the display shelf, the product identification information of the product displayed in the place using the sample information stored in the sample information. The product identification information includes, but is not limited to, a product name, a JAN code assigned to the product, and the like. Any information may be used as long as it can identify the product.

  The product identification information specification processing unit 243 executes the following processing. That is, for each place, the image information of the place and the sample information of the product stored in the sample information storage unit 26 are matched to specify the product identification information of the product displayed in the place. Specifically, first, the similarity between the image information of the area constituted by the coordinates of the place to be processed and the sample information stored in the sample information storage unit 26 is determined, and the sample information having the highest similarity is determined. Is specified, and if the specified similarity is equal to or greater than a predetermined threshold, the product is specified as the product identification information of the product displayed in the place constituted by the coordinates.

  Here, the following processing is performed to determine the similarity between the image information of the place and the specimen information. First, in the processing up to the processing for specifying the product identification information in the product identification information specifying processing unit 243, the direction of the image information of the place area on the shelf of the normal image information and the direction of the sample information are the same (overturn or inversion). The size of each piece of image information is substantially the same (if the size of the image information is different over a predetermined range, the respective pieces of image information are determined before similarity determination). The size is adjusted so that the size of the image information is within a predetermined range.)

  The product identification information identification processing unit 243 determines the similarity between the image information of the place and the sample information, and determines the similarity between the feature point based on the image feature amount (for example, the local feature amount) of the image information of the place and the sample information. Feature points based on image feature amounts (for example, local feature amounts) are extracted. Then, a pair having the highest similarity between the feature point of the image information of the place and the feature point of the sample information is detected, and the difference between the coordinates of the corresponding point is calculated. Then, an average value of the differences is obtained. The average value of the difference indicates the average moving amount of the entire image information of the place and the sample information. Then, the coordinate difference between all feature point pairs is compared with the average coordinate difference, and pairs having a large degree of deviation are excluded. Then, similarity is ranked according to the number of remaining corresponding points.

  The similarity between the image information of the place and the sample information can be calculated by the above method. In addition, in order to improve the accuracy, an EMD (Earth Movements Distance) between the color histograms may be obtained and used as a similarity measure. This makes it possible to compare the similarity relatively strong to environmental changes such as the brightness information of the captured image information, and to specify with high accuracy.

  As a similarity determination, an EMD between signatures (a set of image feature amounts and weights) of the image information of the area of each place may be obtained and used as a similarity measure. As the signature image feature amount, for example, the frequency distribution of the image information of the place area in the HSV color space is obtained, the hue and the saturation are grouped, and the number of features and the image feature amount by the area in the HSV color space are determined. can do. The reason why the hue and the saturation are grouped is to reduce the dependence on the lightness so that the hue and the saturation are not largely influenced by the photographing conditions.

  Further, in order to speed up the processing, similarity such as L2 distance between image features such as color correlogram and color histogram of image information in an appropriate color space can be used instead of signature and EMD.

  The determination of similarity is not limited to the above. The identified product identification information is associated with shooting date and time information, store information, image information identification information of shot image information, image identification information of normal image information, and place identification information for identifying a place, and a product identification information storage unit. 27.

  Since the place where the product identification information can be specified is the face, information indicating that the place is “face” is stored in the product identification information storage unit 27, and the place where the product identification information cannot be specified is “place”. Since it is “empty”, information indicating that the place is “empty” is stored in the product identification information storage unit 27.

  When the displayed product recognition processing unit 24 recognizes the product identification information stored in the IC tag attached to the product, the information read by the portable communication terminal equipped with a predetermined IC tag reading device is read. The information may be received from the portable communication terminal together with the position information on the shelf. In this case, each process in the display product recognition processing unit 24 described above may or may not be provided. The place in this case can also be specified by determining in advance the order in which the reading device reads the IC tags. That is, by placing the reading device over the shelf in order from right to left, the places are sequentially specified in the order in which the product identification information or the inability to read is recognized. In this case, the reader may be held up even at a position where there is no product (gap) to recognize that the product identification information cannot be read, that is, it is "empty". Is also good. Further, in the above-described place specifying processing unit 242, the place may be specified from the captured image information or the normal image information, and the product identification information may be specified by associating the place with the product identification information read by the reading device.

  The product identification information specified as described above is shooting date and time information, store information, image information identification information of shot image information, image identification information of normal image information, place identification information for identifying a place, and a face. The information is stored in the product identification information storage unit 27 in association with information indicating whether the item is empty or empty.

  The shelf image matching processing unit 244 determines the image information of the shelf area in the previous (N-1) -th normal image information and the image information of the shelf area in the current (N-th) normal image information. If the similarity is high, it is determined that the product identification information or “empty” of each place in the shelf is the same. As described above, this similarity determination process is performed by comparing the image feature amount of the image information of the shelf area in the previous (N−1) th normal image information with the current (Nth) normal image information. The similarity may be determined based on the image information of the shelf area, or the EMD of the color histograms may be used. It is not limited to them. Then, instead of the specific processing for each place unit in the product identification information specifying processing unit 243, the product identification information specifying processing unit 243 transmits the product identification information of each place in the shelf in the N-th regular image information to the N- The product identification information is stored in the product identification information storage unit 27 as the same as the product identification information of each place in the same shelf at the first time. As a result, it is possible to omit a process for a shelf that hardly changes, such as a shelf where the product does not move much or a shelf managed in a very short cycle. The processing by the shelf image matching processing unit 244 may not be provided.

  When recognizing the product identification information of the product tag based on the captured image information, the product tag recognition processing unit 25 includes a product tag placement area processing unit 251, a product tag placement area normalization processing unit 252, and a product tag identification processing unit 253. And a product tag information specifying unit 254.

  The product tag placement area processing unit 251 cuts out image information of the product tag placement area specified by the position specification processing unit 23 as product tag placement area image information. The product tag placement area processing unit 251 may actually cut out as image information, or may cut out virtually as image information without actually cutting it out. To virtually cut out image information means to execute processing with a specified area, for example, a range of a product tag arrangement area as a processing target. If there are a plurality of product tag placement areas on the display shelf, each is cut out as product tag placement area image information. The coordinates indicating the product tag placement area are coordinates of vertices for specifying the area, and may be, for example, four points, two points of upper right and lower left, upper left and lower right in the normal image information, and the like. Further, the coordinates are relative coordinates with respect to a predetermined location (for example, the upper left vertex of the display shelf) in the image information such as the display shelf in the normal image information.

  The product tag placement area normalization processing unit 252 executes a trapezoidal correction process for normalizing the product tag placement area image information cut out by the product tag placement area processing unit 251. While the display shelf surface is vertical, the product tag surface is often more upward than the vertical surface so that customers can easily see it. Therefore, the recognition accuracy is improved by normalizing the image information of the product tag arrangement area. FIG. 14 shows an example of the image information of the product tag placement area which is set to the normal position with respect to the image information of the product tag placement area of FIG. FIG. 14A shows image information of the product tag placement area obtained by normalizing the image information of the product tag placement area of the upper shelf in FIG. 10, and FIG. 14B shows the image information of the lower shelf in FIG. It is the image information of the product tag placement area in which the image information of the product tag placement area is set as the normal.

  The product tag placement area normalization processing unit 252 performs the following processing to normalize the image information of the product tag placement area. That is, edge detection is performed in the image information of the product tag placement area, and a near vertical contour line having a certain length or more (for example, a vertical (90 °, ) Is specified within a predetermined range of angles. Note that it is preferable to extract a contour line at a position near both left and right ends, but the present invention is not limited to this. In the case of the image information of the product tag arrangement area in FIG. 10, L1 to L4 are specified as shown in FIG. FIG. 15A shows the product tag placement area of the upper shelf in FIG. 10, and FIG. 15B shows the product tag placement area of the lower shelf in FIG. Note that the specified contour lines L1 to L4 are not actually drawn in the image information of the product tag arrangement area. Then, trapezoidal correction processing is performed on the image information of the product tag arrangement area such that L1 and L2 in FIG. 15A and L3 and L4 in FIG. 15B respectively become vertical lines. By executing such processing, the image information of the product tag arrangement area is set to the normal position, and the image information of the set product tag arrangement area shown in FIG. 14 can be obtained. The accuracy of the product tag identification processing unit 253 and the product tag information identification processing unit 254 can be improved by executing the process of the product tag placement area normalization processing unit 252. It is preferred, but can be omitted. In that case, the product tag specification processing unit 253 and the product tag internal information specification processing unit 254 are executed for the product tag placement area cut out by the product tag placement area processing unit 251.

  The product tag specification processing unit 253 specifies the area of each product tag (product tag area) from the placed product tag placement area image information. In the process of specifying the product tag area, mainly two types of methods can be used. The first method is a method of specifying a product tag area based on a contour line, and the second method is to match overall characteristics such as overall light and dark distribution with image information of a product tag template. This is a method of specifying the product tag area. Note that a method other than the first method and the second method can be used.

  The first method is to use the fact that the ground color (background color) of the product tag is mostly white (brighter than the background). That is, first, a histogram is generated in which the brightness information of the image information is integrated in the horizontal direction from among the image information of the product tag arrangement area that has been placed in the normal position. Then, the rising and falling positions of the histogram are specified, and the upper side position A and the lower side position B of the product tag are specified. FIG. 16 schematically shows this processing. The rising is a portion of the histogram that increases steeply from black to white (a predetermined ratio or more), and the falling is a portion of the histogram that sharply decreases from white to black. .

  Then, a cutout is made between the upper side position A and the lower side position B of the image information of the placed product tag placement area, and a histogram is generated by vertically integrating the brightness information of the placed product tag placement area image information. I do. Then, the rising and falling positions of the histogram are specified, the rising and the falling within a predetermined distance range to the right are paired, and the product tag area is specified as the left side position and the right side position, respectively. If a product tag area does not exist within a predetermined distance, it is specified as a product tag area on the right side of a rising edge where a pair could not be formed, and on the left side of a falling edge where a pair could not be formed. FIG. 17 schematically shows this processing.

  FIG. 18 shows a state where the product tag area is specified from the image information of the product tag placement area shown in FIG. 14 by the first method. A rectangular area composed of each of the upper side position A, the lower side position B, the left side position (rising) U, and the right side position (falling) D is the area of the specified product tag.

  There is also a product tag in which the ground color (background color) of the product tag is other than white, for example, red, and the characters are white. Therefore, when the price information and the product identification information in the product tag cannot be recognized by the above-described processing, the relationship between white and black is inverted (the brightness is inverted) in the image information of the product tag placement area, The same processing as described above may be executed. This makes it possible to correspond to a product tag with a dark ground and a bright character.

  The process of inverting the brightness of the image information in the product tag arrangement area is performed when the information in the product tag cannot be recognized or when the product identification information cannot be normally recognized in any of the processes of identifying the product identification information. It may be executed irrespective of whether it can be recognized or not.

  The second method is so-called template matching. That is, the image information of the product tag is registered in advance as a template, and the template is matched with the image information of the product tag placement area that has been set in place, thereby specifying the product tag area.

  The product tag includes a tax-excluded price, a tax-included price, product identification information (such as a product name), a maker name, and a rating. Therefore, if the image information of the product tag used as a template includes product identification information such as the product name and specific numbers and characters of the price, that portion is also subject to image matching processing determination, and that portion is converted into a mosaic. , It is preferable to neutralize or exclude from the determination target by deleting. Neutralization means that high and low points are not assigned to any input, and exclusion means that the part is excluded from matching in image matching processing.

  The product tag information specification processing unit 254 executes a process of specifying information described in the product tag area specified by the product tag specification processing unit 253 by OCR recognition or the like. The OCR recognition can be performed for all or a part of the area specified as the product tag area. If it is performed on all the regions specified as the product tag regions, there is a high possibility that erroneous recognition will occur due to noise or the like. In this case, processes corresponding to the first method and the second method used in the product tag identification processing unit 253 may be executed.

  When the first method is used in the product tag specification processing unit 253, first, binarization processing is performed in the specified product tag area. As the binarization processing, one of a method using the lightness of the HSV color space, a method of Otsu, and a method of extracting red (red and red are frequently used for numbers and character strings of product tags) is one or more. There is a method of selecting a combination extracted as a candidate.

  Then, a labeling process is performed on the binarized image information. The labeling process is a process of assigning the same number (identification information) to pixels having continuous white or black in binary image information, thereby forming continuous islands into one island (group). is there. Then, a rectangular area (box) including the island detected by the labeling process is generated, and the height, width, and baseline of the box are obtained. When generating a box, it is preferable, but not limited to, to generate a rectangle enclosed by the smallest, vertical and horizontal line segments that enclose the area labeled with the same number. When a box is generated, islands that do not satisfy a predetermined threshold height and width do not generate a box as noise, but are removed as is from the processing target. For example, islands that are too small may be horizontal ruled lines or dust on the image, and islands that are too wide may be logos, which are removed as noise.

  In general, characters used in a product tag are often used in a font such as a bold Gothic font in which the outline of the entire character is clear. Therefore, even when the image information has a slight blur, a character group forming one character string can be detected as an island having the same height as the base line.

  Then, the information-in-product-tag identification processing unit 254 combines adjacent boxes having predetermined similarity with each other to form a block. That is, a block is formed by merging continuous boxes whose height matches the baseline within a predetermined range and whose height and width are within a certain threshold range. At this time, small boxes and the like between the blocks to be merged are collectively configured as one block. As a result, a cloud point, a semi-voice point, a hyphen, and the like are also taken into one block. A block is an area to be subjected to OCR recognition. Then, the block having the highest height among the blocks is estimated as a price area (price area without tax), and OCR recognition is performed. The OCR recognition is similarly performed for the other blocks. By performing the above-described processing, the character recognition processing can be performed with higher accuracy than when performing OCR recognition on a document portion freely laid out over a plurality of lines in the product tag area. FIGS. 19 to 21 schematically show this processing. FIG. 19 shows binarized image information, and FIG. 20 shows a state in which a box (region indicated by a broken line) is generated. FIG. 21 is a diagram showing a state in which blocks (regions indicated by broken lines) have been generated.

  As described above, the information specification processing unit 254 in the product tag can recognize the information described in the product tag by characters.

  When the second method is used in the product tag specification processing unit 253, product identification information such as a tax-excluded price, a tax-included price, a manufacturer name, a product name, and the like are described in advance in the image information of the product tag as a template. The position and size (height, width) of each character frame are set. Therefore, the image information of the corresponding portion is cut out from the product tag area specified by the product tag specification processing unit 253, and the OCR recognition processing is executed. At this time, product identification information such as price, manufacturer name, product name, etc., and the character type used by the rating (for example, whether it contains numbers, Roman characters, symbols, or Kana-Kanji characters) are defined as constraints. The accuracy of the OCR recognition process can be improved.

  Further, the information-in-product-tag identification processing unit 254 may execute a process of confirming the consistency of the read information. As the consistency check processing, it is preferable to perform two kinds of processing: a check processing of consistency by dictionary matching and a logical check.

  The process of checking the consistency by dictionary matching is executed, for example, as follows. The display situation determination system 1 stores a product dictionary (FIG. 1) that stores product identification information such as product names of products that may be displayed on a display shelf in association with corresponding code information (for example, JAN code). (Not shown). Then, character string similarity evaluation between a character string other than the character string read from the area indicating the price recognized by the information identification processing unit 254 in the product tag and product identification information such as all the product names registered in the product dictionary. I do. For example, an edit distance (Levenshtein distance), a Jaro-Winkler distance, or the like can be used.

  If the maximum value of the character string similarity determined above is one, the product identification information such as the product name is determined as the final candidate. The allowable similarity is determined in advance for the length of the character string of the product identification information such as the product name as the final candidate. If the similarity is equal to or greater than the allowable similarity, the product identification information such as the product name is identified. I do. If the allowable similarity has not been reached, the read character string is undetermined. If there are a plurality of products having the maximum similarity, the read character string is not determined.

  The edit distance, which is one of the character string similarity evaluations, is a type of distance that indicates how different two character strings are. Specifically, the edit distance is determined by inserting, deleting, or replacing one character. The minimum number of steps required to transform one string into another. FIG. 22 shows an example of the product name described in the product tag, the product name having the smallest editing distance among the product names stored in the product dictionary as a result of OCR recognition, and the editing distance. FIG. 23 shows a table of the correspondence of the edit distances that may be determined for each number of characters of the final candidate. In this specification, a case in which processing is performed using the edit distance will be described. However, a character string similarity evaluation defined so that the substitution has a greater similarity than the insertion or deletion may be used.

  Then, after obtaining the product identification information such as the product name of the minimum editing distance, a rating dictionary (not shown) is provided separately for the character string obtained by removing the relevant part of the product identification information such as the product name from the OCR recognition result. The edit distance from each rating stored in the manufacturer name and the edit distance from each manufacturer name stored in the manufacturer name dictionary (not shown) are obtained. For example, in the example of "350 ml of throat raw" in FIG. 22, "350 ml" matches "350 ml" in the rated dictionary at an editing distance of 0, and the character string of the rated part is identified. Similarly, the editing distance is obtained for the maker name, and the character string at the minimum editing distance is identified as the maker name. Then, the character strings of the rated part and the maker name part are removed from the OCR recognition result, and the character string at the shortest edit distance in the product name dictionary is obtained for the removed character strings. Determine whether or not. If the editing distance is within the allowable range, the product identification information such as the product name, the manufacturer name, and the rating are determined. By performing such processing, it is possible to appropriately determine even when the product identification information such as the product name includes the rating and the manufacturer name. The rating dictionary is a dictionary that stores ratings (capacity, etc.) of products that may be displayed on the display shelf. The maker name dictionary is a dictionary that stores maker names of products that are likely to be displayed on display shelves.

  In order to allow the user to finally determine and approve the recognition result by the information identification processing unit 254 in the product tag, when performing display for accepting an input such as selection of the recognition result, a confirmed character string, an unconfirmed character string Is preferably displayed in such a manner that it can be specified. For example, the colors of the determined character string and the undetermined character string are divided, and the determined character string and / or the undetermined character string are provided with information indicating the determined or undetermined character string. For an undetermined character string, if there is a plurality of product identification information such as a product name having a minimum editing distance, each product identification information may be displayed.

  The logical check (the process of determining the logical consistency of the recognized information) in the product identification information processing unit 254 can be performed as follows. For example, when two prices are read as price areas, one is the tax-excluded price and one is the tax-included price, but if the tax-excluded price is higher than the tax-included price, they are reversed and recognized. Let it. If the value obtained by multiplying the tax-excluded price by the consumption tax rate does not match the tax-included price, it is determined that one or both of them has an erroneous recognition. Further, when the product identification information such as the product name is recognized, it is determined whether or not the product or the product category falls within the normal price range. Further, it may be determined whether the correspondence between the product identification information such as the product name, the manufacturer name, and the rating is the same.

  As described above, by confirming the consistency of the information in the product tag, the information included in the product tag in the captured image information can be determined. The information determined in this way can be output in, for example, a table format.

  Next, an example of a processing process of the display status determination system 1 of the present invention will be described with reference to the flowchart of FIG. In the following description, a case is described in which the product identification information of the displayed product and the product identification information of the product tag are identified from the captured image information, and the lack of the product is determined. By reading the product identification information stored in the IC tag installed in the vicinity with the reading device, the product identification information can be specified, and the product shortage can be determined.

  The photographed image information obtained by photographing the display shelf of the store is input from the photographed image information input terminal 3, and the input is received by the photographed image information input reception processing unit 20 of the management terminal 2 (S100). FIG. 25 shows an example of captured image information. In addition, input of image capturing date and time, store identification information, and image information identification information of captured image information is received. Then, the photographed image information input reception processing unit 20 stores the photographed image information, the photographing date and time, the shop identification information, and the image information identification information of the photographed image information, which have been received, in the photographed image information storage unit 21 in association with each other.

  When a predetermined operation input is received in the management terminal 2, the normal image information normalization processing unit extracts the photographed image information stored in the photographed image information storage unit 21 and performs a normalization process such as a trapezoidal correction process. The input of four points of the shelf position (display shelf position), which are vertices, is performed, and a positive value conversion process is executed (S110). FIG. 26 shows an example of photographed image information (normal image information) on which the normalization process has been performed in this manner.

  Then, by receiving a predetermined operation input in the management terminal 2 with respect to the normal image information, the position specifying processing unit 23 specifies the shelf position area and the product tag arrangement area (S120). That is, the input of the shelf area and the product tag arrangement area in the normal image information is accepted. FIG. 10 is a diagram showing a state where the shelf area and the product tag arrangement area are specified.

  When the shelf area and the product tag placement area are specified as described above, the display product recognition processing unit 24 performs the process of recognizing the displayed product in the shelf area and the product tag recognition process in the product tag placement area. The steps are executed by the units 25 (S130, S140). The display product recognition processing by the display product recognition processing unit 24 and the product tag recognition processing by the product tag recognition processing unit 25 may be performed in parallel or at different timings.

  First, a process of recognizing product identification information of a product in the display product recognition processing unit 24 will be described.

  The shelf area processing unit 241 in the display product recognition processing unit 24 cuts out the image information of the shelf area from the normal image information based on the area of the shelf received in S120. Then, a process of specifying a place is executed for each shelf in the shelf area image information. More specifically, for a shelf in the shelf area, a repetitive pattern of an image is specified, which specifies a narrow narrow shadow between products within a rectangular area formed by coordinates of four points. The place is specified by specifying a step on the upper side of the package, or specifying a break position based on restrictions such as the same product width. Place identification information for identifying the place is added to the identified place. The identified coordinates of each place are stored in association with the photographing date and time, store identification information, image information identification information of photographed image information, image information identification information of stationary image information, and place identification information for identifying the place. Let it. Note that the coordinates of the place may be two points that can specify a rectangular area without storing four points.

  As described above, when each place of each shelf in the shelf position area image information of the normal image information is specified, the product identification information specifying processing unit 243 determines the sample information stored in the sample information storage unit 26 for each place. A matching process is executed to specify the product identification information of the product displayed in the place. That is, each image feature amount is calculated from the image information in the rectangular area of the place of a certain shelf (X is the place identification information of the area of this place) and the sample information stored in the sample information storage unit 26. Then, similarity is determined by obtaining a pair of feature points. Then, the sample information having the highest similarity is specified. If the similarity at that time is equal to or greater than a predetermined threshold, the product identification information corresponding to the sample information is specified based on the sample information storage unit 26. Then, the specified product identification information is set as the product identification information of the product displayed in the place of the place identification information X. Note that, for a place determined to be not similar to any of the sample information, information indicating that the place identification information is “empty” is added. The product identification information specification processing unit 243 converts the specified product identification information or the information indicating “empty” into the shooting date and time, the store identification information, the image information identification information of the shot image information, and the image information identification of the normal image information. The information is stored in the product identification information storage unit 27 in association with the information and the place identification information.

  Note that it is not always possible to specify the product identification information of all places. Therefore, for a place that cannot be identified, instead of adding “empty” information from the beginning as described above, input of product identification information is accepted, and the received product identification information is used as the shooting date and time and store identification. The information is stored in the product identification information storage unit 27 in association with the information, the image information identification information of the captured image information, the image information identification information of the normal image information, and the place identification information. Similarly, an input may be accepted for the process of correcting the specified product identification information. Further, as a result of the correction processing, when an input indicating that the content is “empty” is received, information of “empty” may be added.

  By performing the above processing, it is possible to specify the product identification information of the product displayed on the shelf of the display shelf shown in the captured image information.

  Next, a process of recognizing information of a product tag in the product tag recognition processing unit 25 will be described.

  The product tag placement area processing unit 251 in the product tag recognition processing unit 25 cuts out the image information of the product tag placement area specified in S120, and the product tag placement area normalization processing unit 252 executes the trapezoidal correction processing. A normalization process is performed on the image information in the tag arrangement area.

  As the normalization processing for the image information of the product tag placement area, first, edge detection is performed on the image information of the product tag placement area. Then, of the detected edges, a contour line having a predetermined length or more and within an angle within a predetermined range from the vertical is specified. Then, the leftmost and rightmost contour lines are specified from the specified contour lines. FIG. 15 shows an example of the contour line thus specified. Then, the trapezoidal correction process is performed on the image information of the product tag placement area so that the specified contour lines (L1, L2 in FIG. 15A, L3, L4 in FIG. 15B) become vertical lines. I do. By executing such processing, the image information of the product tag arrangement area is set to the normal position, and the image information of the set product tag arrangement area shown in FIG. 14 can be obtained.

  When the normalization processing of the image information of the product tag placement area is completed, the product tag identification processing unit 253 converts the individual product tag areas from the image information of the product tag placement area using the first method or the second method. Identify.

  In the case of the first method, a histogram in which the brightness information of the image information is integrated in the horizontal direction is generated from the image information of the product tag placement area that has been set, and the rising and falling positions of the histogram are specified. Then, the rise of the specified histogram is specified as a top position A of the product tag, and the fall position of the histogram is specified as a bottom position B of the product tag.

  Next, of the image information of the product tag placement area that has been set, a histogram obtained by cutting out the upper side position A and the lower side position B and integrating the brightness information of the image information of the product tag placement area that has been set in the vertical direction. Generate

  In the generated histogram, the rising position U and the falling position D are specified, and the rising position U (the left side position) and the falling position D (the right side position) within a predetermined distance range to the right are specified as a pair. Specify as an area.

  The rising position U where a pair could not be formed is on the right side, and the falling position D where a pair could not be formed is on the left side. If there is no product tag area within a predetermined distance, it is specified as a product tag area. I do.

  FIG. 18 shows a state in which each product tag area is specified by the processing as described above.

  In the case of using the second method, the product tag identification processing unit 253 performs an image matching process between the image information of the product tag of the template registered in advance and the image information of the product tag placement area that has been set in place. By executing, the product tag area is specified.

  When the product tag specification processing unit 253 specifies the product tag area as described above, the information specification processing unit 254 within the product tag specifies information in the product tag.

  The first method in the product tag information specification processing unit 254 performs binarization processing on the specified product tag area to obtain binarized image information. Then, a box is specified by a labeling process on the binarized image information. It should be noted that islands that do not satisfy the predetermined threshold height and width do not generate a box as noise and are removed from the processing target as they are.

  Then, the height, width, and baseline of the generated box are obtained, and the adjacent box whose height matches the height within a predetermined range and whose height and width are within a certain threshold range is specified. The specified boxes are merged to form a block. Then, of the blocks, the block having the highest height is estimated as a price area (price area without tax), and OCR recognition is performed. As a result of the OCR recognition, if price information cannot be obtained (if a numeric character string cannot be recognized), the block having the next highest height is estimated as a price area (excluding tax area), and OCR recognition is similarly performed. .

  In the above-described OCR recognition, the accuracy of reading the price can be improved by adding, as a constraint, character recognition that constitutes a price display such as “number” or “comma” used for the price.

  As described above, when the price information is acquired from the block estimated as the price area (excluding tax area) by OCR recognition, the blocks other than the price information acquired and estimated as the price area (excluding tax area) are identified and identified. OCR recognition is performed for each block. For this OCR recognition, it is preferable to perform two types of OCR recognition processing. That is, a normal standard OCR recognition process and an OCR recognition process in which character recognition constituting price display is added as a constraint.

  The blocks specified above include a block in a price area (tax-included price area) and a block in an area of information such as product identification information such as a manufacturer name and a product name and rating. Then, two types of OCR recognition processing are executed for each block. For blocks in the area of product identification information such as manufacturer name and product name, and information such as rating, character strings are recognized in standard OCR recognition processing, and characters containing errors are often used in OCR recognition processing with constraints. Become a column. In this case, it is determined whether or not the recognition results of the two recognition processes differ by a predetermined value or more. If they differ, the character string executed in the standard OCR recognition process is used as the recognition result, and the price area (price area including tax) is used. It is determined that the block is a block other than the block. On the other hand, for the block in the price area (tax-included price area), the character string of the price information is recognized by the standard OCR recognition processing, and the character string of the price information is also recognized by the OCR recognition processing with the constraint. In this case, it is determined whether or not the recognition results of the two recognition processes differ by a predetermined value or more. If the two do not differ, it is determined that the block is a block in a price area (tax-included price area), and the recognition processing with a constraint added is performed. The character string is used as the price information of the recognition result.

  As described above, the information specification processing unit 254 in the product tag can recognize the information described in the product tag by characters. In order to determine the display status of a product, it is sufficient to have the product identification information of the product, but by recognizing other information such as the tax-excluded price and the tax-included price, the price information etc. can be associated with the display status. It becomes possible to manage.

  In the case where the second method is used in the product tag identification processing unit 253, the product tag used as a template includes in advance product identification information such as a tax-excluded price, a tax-included price, a manufacturer name, a product name, and a rating. The position and size of the character frame to be described are set. Therefore, the image information of the corresponding portion may be cut out from the product tag area specified by the product tag specification processing unit 253, and the OCR recognition processing may be executed.

  Then, the in-product-tag information specification processing unit 254 executes a dictionary matching process with the specified product name or the like. That is, the editing distance between the read character string and the product identification information such as each product name in the product dictionary is determined, the product identification information such as the product name with the minimum editing distance is specified, and the product identification information is determined within a predetermined distance. If so, it is identified as product identification information such as a product name. Then, the product name portion with the shortest editing distance is removed from the read character string, and the editing distance between each remaining character string and each rating in the rating dictionary is obtained. It is determined whether the minimum editing distance is within a predetermined distance. Judgment is made, and if it is within a predetermined distance, that part is identified as a rated character string. Similarly, the product name part and the rated part with the minimum editing distance are removed from the read character string, and the editing distance between each remaining character string and each manufacturer name in the manufacturer name dictionary is determined. It is determined whether it is within the distance, and if it is within the predetermined distance, that part is identified as a character string of the manufacturer name. When the Jaro-Winkler distance is used instead of the edit distance, processing corresponding to the Jaro-Winkler distance may be appropriately executed.

  Further, the product tag information identification processing unit 254 executes a logical check process on the character string whose character has been recognized. That is, it is determined whether or not the recognized character string is logically inconsistent.

  If there is no inconsistency as a result of the logical check, for each recognized character string, product identification information such as tax-excluded price, tax-included price, product name, etc., manufacturer, and rating are specified, and they are identified as shooting date and time and store identification. The information is stored and output in a predetermined storage area in association with the image and image information identification information of the photographed image information. For example, output in table format.

  As described above, after the displayed product recognition processing unit 24 recognizes the product identification information of the product displayed on the shelf and the product tag recognition processing unit 25 recognizes the product identification information of the product tag, the display status determination processing unit 24, out of the product identification information recognized by the product tag recognition processing unit 25, product identification information that is not included in the product identification information recognized by the displayed product recognition processing unit 24, as shown in FIG. It is determined as the product identification information of the product (S150).

  In other words, the display situation determination processing unit 28 determines that the same product identification information is included in an adjacent place among the product identification information (face product identification information) for each place recognized by the display product recognition processing unit 24. , The product identification information is put together as one product group. In addition, the display situation determination processing unit 28 combines the same product identification information among the product identification information recognized by the product tag recognition processing unit 25 into one piece of product identification information.

  Then, the collected product identification information is compared with each other, and it is determined whether or not there is matching product identification information. Is determined.

  That is, when determining the missing item, the product identification information that is not included in the product identification information recognized by the displayed product recognition processing unit 24 among the product identification information recognized by the product tag recognition processing unit 25 is missing. The information is determined as the product identification information of the product, and information indicating that the product is out of stock is stored in the product identification information storage unit 27 in association with the identification information of the place.

  In order to judge a missing item, it is sufficient to execute only this judgment process. However, as a product manufacturer or a sales company, what kind of product of the company or other company is exposed on the display shelf of the store and to what extent. By knowing whether the product is displayed, it can be linked to the development strategy and sales strategy of the company's products. Therefore, by also performing the judgment of the product on display, the demands of such a manufacturing company and a sales company can be achieved. For this reason, the display status determination processing unit 28 may also recognize a product being displayed in addition to the determination of a missing item. That is, the display status determination processing unit 28 may determine that the product identification information recognized by the displayed product recognition processing unit 24 is in display, among the product identification information recognized by the product tag recognition processing unit 25. . When it is recognized that the product is being displayed, the product identification information and information indicating the face are stored in the product identification information in association with the identification information of the place.

  Further, the absence of the product tag prevents the customer from knowing the price of the product, which is one of the causes of the customer refraining from purchasing the product. Therefore, it is also important for the product manufacturer and the sales company to know the absence of the product tag. Therefore, the display situation determination processing unit 28 may recognize the absence of the product tag in addition to the determination of the missing item. That is, the display status determination processing unit 28 determines that the product identification information that is not included in the product identification information recognized by the product tag recognition processing unit 25 among the product identification information recognized by the displayed product recognition processing unit 24 has no product tag. Is determined. When the absence of the product tag is recognized, the product identification information and the information indicating the absence of the product tag are stored in the product identification information in association with the identification information of the place.

  By executing the above-described processing, it is possible to recognize a product that is missing, in some cases, a product that is being displayed, and a product that does not have a product tag in the display situation of the display shelf. Further, since such information is stored for each place on the display shelf, it is possible to grasp information such as a missing item of a product for each position of the place.

  In the first embodiment, the case where processing is performed using image information and sample information of a place area as recognition of “sky” has been described. However, if only image information matching processing is performed, the processing is affected by shooting conditions and the like. Cheap. Therefore, in addition to the matching between the image information of the place area and the sample information, a process for recognizing "sky" can be performed to further improve the accuracy.

  In this case, the displayed product recognition processing unit 24 executes the following process of recognizing “empty”.

  First, as described above, the product identification information specifying processing unit 243 in the display product recognition processing unit 24 performs image information of an area formed by coordinates of a place to be processed and sample information stored in the sample information storage unit 26. Is determined, the product identification information corresponding to the sample information having the highest similarity is specified, and a place whose similarity is not more than a predetermined threshold is specified.

  Therefore, the displayed product recognition processing unit 24 determines whether any of the following processes A and B is applicable as the process of the first method for recognizing “empty”.

  That is, the process A determines that the image information of the specified place area is “sky” when the average value of the brightness of the area is darker than the predetermined reference value. In the process B, when the image information of the place area is subjected to the matching process with the image information of the rear plate of the shelf, if the similarity is the highest, it is determined to be “empty”. In this case, image information of the rear plate of the shelf may be stored as sample information. Note that a determination process other than the process A and the process B can be used.

  In the process A, since the sample information of the product has a color close to black as a whole and there is usually no product name that is not specified, there is no problem in recognizing it as "empty". Regarding the process B, since the rear plate of the shelf is not usually similar to the sample information of the product, if it is determined that the similarity with the rear plate of the shelf is high, it can be recognized as “empty”. it can.

  In addition, the displayed product recognition processing unit 24 recognizes a void in the display shelf (an area where a product is originally placed, but a void caused by a missing item) as a process of the second method for recognizing “empty”. It may be. The process of recognizing the void in the display shelf can be determined as follows. In this case, when the display shelf is photographed and the photographed image information is acquired, the photographing device (for example, including a portable communication terminal such as a smartphone) includes distance information (eg, distance between the display shelf to be photographed and the photographing device). (Depth information) as two-dimensional information. In this case, when photographing the display shelf in the photographing device, the photographing is performed by irradiating a light beam of a specific wavelength, for example, infrared ray, and measuring the amount of reflected light from each direction and the time until the reflected light reaches. Other than acquiring distance information (depth information) to the target or irradiating a light beam of a specific wavelength, for example, an infrared dot pattern, and measuring distance information (depth information) from the reflection pattern to the imaging target However, there is a method using parallax of a stereo camera, but the method is not limited to these. 27 (a) and 27 (b) schematically show the correspondence between the captured image and the depth information. The positions of the captured image information and the depth information correspond to each other. If the depth information is given by the distance from the imaging device (one point in three dimensions), the depth information is converted into depth information from the imaging surface (plane) of the imaging device. . FIG. 28 schematically shows an example of the depth information.

  Then, the photographed image information is set as the normal image information in the photographed image information normalization processing section 22. Then, similarly, the same conversion is executed for the depth information. That is, the same conversion processing as the normalization processing such as the trapezoid correction processing in the captured image information is also performed on the depth information. This is schematically shown in FIGS. 29 (a) and 29 (b). FIG. 29A shows the fixed image information obtained by performing the normalization process on the captured image information of FIG. 27A. FIG. 29B is a diagram showing depth information in a state where the same processing as the normalization processing of FIG. 29A is performed on each piece of depth information with respect to the depth information of FIG. 27B. It is. FIG. 30 schematically illustrates an example of the depth information in a state where the normalization process has been performed on the depth information of FIG.

  As described above, by executing the same processing on the depth information as on the normal image information, the depth information can be specified with the same positional relationship as the normal image information.

  Next, for the place area specified by the displayed product recognition processing unit 24, one of the minimum value, the average value, and the maximum value of the depth information in the place area is obtained (this is called a place distance). Then, a plane determined by the majority of the depth information of each place is obtained, and the depth of each place is obtained from the plane. FIG. 31 schematically shows the depth of each place. If the depth is larger than a predetermined threshold, the place is determined to be “empty”.

  If the regression plane of all the place positions in the three-dimensional space is obtained, the depth of the value that deviates the most from the regression plane is removed, and the regression plane is calculated again. 31 can be obtained by setting the distance from the plane to each place position as "depth of the place". FIG. 32 schematically shows a process of obtaining a regression plane of a place position. FIG. 32A is a front view, and FIG. 32B is a view from above.

  The displayed product recognition processing unit 24 can more accurately specify that the place is “empty” by executing the processing of the first method or the processing of the second method. The processing of the first method and the processing of the second method may be performed in addition to the matching processing between the image information of the place area and the sample information in the product identification information specifying processing unit 243, or the image information of the place area. May be performed instead of the matching process with the sample information.

  Further, when the position of the place area has already been specified, for example, when processing is performed for the same display shelf for the second time or later, the processing is performed in place of the matching processing between the image information of the place area and the sample information. When executed, it is possible to quickly and accurately specify whether or not the place is “empty”.

  Note that the recognition processing of “empty” in the first embodiment, the present embodiment, and the like is performed by the reading device reading the product identification information from the IC tag attached to the product, and the displayed product recognition processing unit 24 converts the product identification information based on the information. It may be applied when specifying. That is, when the product identification information is read from the IC tag attached to the product, if the product does not exist, the product identification information cannot be read because there is no IC tag. For this reason, only places of products displayed in the horizontal direction may be recognized. On the other hand, originally, goods should be displayed in the "empty". Therefore, the present embodiment is executed for each area corresponding to the horizontal and vertical sizes of the product corresponding to the product identification information read by the reading device (for example, one product or an average value of the products read). By separating the “empty” area recognized in the example, the place can be automatically specified. In this way, even when the product identification information is read from the IC tag attached to the product, the “empty” place, which is the area where the product should exist, can be automatically specified.

  In the first embodiment and the second embodiment, when a product is missing, the matching process is performed between the image information of the place area and the sample information. However, the same display shelf is used for the second and subsequent times. When executing the processing, the display status determination processing unit 28 can execute the determination processing using the previous processing result.

  In this case, the display status determination processing unit 28 makes the determination based on the change in the state of the place between the previous time (N-1 times) and the current time (N times) and the change in the product tag. FIG. 33 shows an example of this determination processing.

  That is, the display situation determination processing unit 28 determines the product identification information in the current (N-th) place recognized by the display product recognition processing unit 24, the product identification information in the same (N-1) previous place, and the product Using the information of the current (N times) product tag recognized by the tag recognition processing unit 25 and the information of the previous (N-1 times) product tag, the state change of the current time and the previous time is shown in FIG. Based on the above, it is determined whether or not the place is out of stock. By making such a determination, the determination can be made more quickly.

  Further, a modified example of the specification processing in the place specification processing unit 242 according to the first to third embodiments will be described. In the present embodiment, the process of specifying a place in the first to third embodiments may be performed as an initial process, and the process of specifying a place may be automatically performed as a process of specifying a second or subsequent place. The processing in this case will be described.

  Note that, in addition to the first time, a case where the processing of the first to third embodiments is performed at an arbitrary timing in order to correct a shift at the time of automatic identification is also included. The second and subsequent times are other than the first time. The second and subsequent times also include cases in which processing is performed on photographed image information and normal image information of photographed image information and normal image information obtained by photographing the same display shelf at the same opportunity and at different angles. .

  The place identification processing unit 242 executes the same processing as that of the first embodiment as the first place identification processing. The place specification processing unit 242 specifies the second and subsequent places by specifying the coordinates of the area of the place specified by the previous (N-1) -th normal image information for the same shelf of the same display shelf. The coordinates are extracted and the coordinates are set as the coordinates of the area of the place specified by the current (N-th) normal image information.

  The coordinates of the area of the place are, similarly to the coordinates of the position of the shelf, relative coordinates based on a predetermined position in the display shelf in the normal image information.

  Further, as a modification of the first embodiment, the process of the first embodiment is performed for the first time as the process of specifying the product identification information of the product displayed in the place for each shelf of the display shelf in the product identification information specifying processing unit 243. The following processing is executed as the processing for specifying the product identification information for the second and subsequent times.

  The product identification information identification processing unit 243 first identifies the place identification information of the place to be processed as the identification processing of the product identification information of the place in the N-th normal image information. Let X be the identified place identification information. Then, the image information of the area of the place identification information X in the N-th normal image information is compared with the image information of the area in the (N-1) th normal image information at the position corresponding to the place identification information X. For the determination of the similarity, in addition to the processing of the above-described first to fourth embodiments, the EMD of the signatures (sets of image feature amounts and weights) of the image information of the area of each place is obtained, and the similarity scale and can do. As the signature image feature amount, for example, the frequency distribution of the image information of the place area in the HSV color space is obtained, the hue and the saturation are grouped, and the number of features and the image feature amount by the area in the HSV color space are determined. can do. The reason why the hue and the saturation are grouped is to reduce the dependence on the lightness so that the hue and the saturation are not largely influenced by the photographing conditions.

  Further, in order to speed up the processing, similarity such as L2 distance between image features such as color correlogram and color histogram of image information in an appropriate color space can be used instead of signature and EMD.

  If the similarity of the image information of the place area is equal to or more than a certain threshold value, the product identification information corresponding to the place of the area in the (N-1) th normal image information is extracted from the product identification information storage unit 27, and the Nth time Is the product identification information of the place identification information X in the normal image information. Thereby, the product identification information of the place identification information X in the N-th normal image information to be processed can be specified. If the similarity is less than a certain threshold value, the image information of the area of the place identification information X in the N-th normal image information and the sample information stored in the sample information storage unit 26 are similar to the case of the first time. In comparison, the product identification information having the similarity equal to or higher than the predetermined threshold value and having the highest similarity is specified as the product identification information of the place of the place identification information X in the N-th normal image information.

  In the comparison with the image information of the place area in the (N-1) th normal image information, not only the comparison with the corresponding place position but also a place within a predetermined range may be included as a comparison target. For example, when comparing with the image information of the area of the place identification information X in the N-th normal image information, the comparison target is not only the area of the place identification information X in the (N-1) th normal image information but also the area. Areas of a place within a predetermined range, for example, a place located at a position separated by one or more in the left-right direction, and areas of places located at upper and lower shelves may be included. Further, in addition to the area of the place identification information X in the N-1th normal image information, the area of a plurality of adjacent places such as the place identification information X-2, X-1, X, X + 1, X + 2 is included. You may.

  In this case, the image information of the area of the place identification information X in the N-th normal image information and the image information of the area of the place to be compared in the (N-1) -th normal image information are Are compared, and the place identification information of the (N−1) th normal image information having the highest similarity is identified. Note that the similarity may be a condition that the similarity is equal to or more than a certain threshold. Then, the product identification information corresponding to the specified place identification information of the (N-1) th normal image information is extracted from the product identification information storage unit 27, and the product identification information of the place identification information X in the Nth normal image information is extracted. I do. FIG. 34 schematically shows this processing. FIG. 34A shows the previous (N-1) -th set image information, and FIG. 34B shows the current (N-th) set image information. Then, the image information of the area of each place of the shelf 1 of the N-th normal image information is compared with the image information of the area of each place of the shelf 1 of the N-1th normal image information. To determine the similarity. The product identification information of the place of the shelf 1 in the (N-1) th normal image information having the highest similarity is replaced with the product identification information of the place of the shelf 1 in the Nth normal image information. Is specified as FIG. 34 shows a case where, in addition to the (N-1) th place of the normal image information located at a position corresponding to the place of the Nth place image information, a comparison is made with two places each on the left and right sides. . In addition, not only the same shelf but also the image information of the place position of the upper and lower shelf may be compared. For example, in the case of FIG. 34, when specifying the product identification information at the place position of the center of the shelf 2 of the N-th normal image information, the center of the shelf 2 of the (N-1) -th normal image information and the left and right thereof Not only is the image information compared with the image information of every two places, but also the image information of the center of the shelf 1 and the two places of the left and right places of the shelf 1 of the (N-1) th normal image information, May be compared with the image information of the center of the shelf 3 and the image information of two places on the left and right sides of the shelf 3 of the normal image information.

  If the candidate of the product identification information cannot be specified because the similarity does not satisfy the threshold as a result of the comparison of the (N-1) -th place image information with the place image information, the N-th place image information Is compared with the sample information stored in the sample information storage unit 26, the product identification information having a similarity equal to or greater than a predetermined threshold value and having the highest similarity is identified by N It is specified as the product identification information of the place of the place identification information X in the second regular image information. In this case, the similarity determination process can be performed in the same manner as the first process of specifying the product identification information.

  The product identification information candidates identified as described above correspond to shooting date and time information, store information, image information identification information of shot image information, image identification information of normal image information, and place identification information for identifying a place. In addition, it is stored in the product identification information storage unit 27.

  In the first to fifth embodiments, in each piece of image information obtained by photographing the same display shelf at the same opportunity at different angles, the product identification information in the place is specified, and the corresponding places are compared with each other. The product identification information in was determined. Therefore, instead of specifying the product identification information in the place for each piece of image information shot at different angles, the image information shot at different angles is first synthesized to eliminate the reflected parts of light, etc. The processing for specifying the product identification information in the place may be performed on the obtained one piece of image information.

  The processes of the above-described first to sixth embodiments can be appropriately combined. In addition, each processing is not limited to the order described in the specification of the present invention, and can be appropriately changed as long as the object is achieved. Further, each processing in the display product recognition processing unit 24 and the product tag recognition processing unit 25 is performed on the in-place image information in which the in-place processing is performed on the captured image information. You may do it. In that case, the normal image information may be replaced with the photographed image information.

  Further, in the processing of the above-described first to sixth embodiments, the displayed product recognition processing unit 24 specifies the identification information of the product from the captured image information or the normal image information, and the product tag recognition processing unit 25 specifies the product tag. By identifying the identification information and collating them, a missing item of the product is determined. In addition to recognizing character information described in the product tag, the product tag recognition processing unit 25 may read information obtained by coding the product tag or product identification information provided in the vicinity of the product tag. The product identification information may be read from an IC tag or the like provided in the vicinity thereof. That is, the product tag recognition processing unit 25 is not limited to the character recognition processing described on the product tag, but uses the product tag or the product identification information near the product tag to display the product tag recognition processing unit 24. It is only necessary to perform the matching process with the product identification information to determine the missing item.

  Further, the display status determination processing unit 28 may determine the missing item by comparing the identification information of the product specified by the displayed product recognition processing unit 24 with the stock information of the product managed by the POS system. . That is, the number of stocks of products managed by the POS system is extracted, and the number of stocks for each product is compared with the product identification information specified by the displayed product recognition processing unit 24. Then, although the POS system has a stock quantity (the stock quantity is 1 or more), the product identification information of the product that could not be specified by the displayed product recognition processing unit 24 is determined to be likely to be out of stock. May be.

  Further, the displayed product recognition processing unit 24 specifies the product displayed on the display shelf based on the captured image information, attaches an IC tag to the displayed product, and stores the IC tag in the IC tag. By reading the product identification information, the product identification information of the product displayed on the display shelf may be specified. When the displayed product recognition processing unit 24 reads the product identification information from the IC tag attached to the product, and the product tag recognition processing unit 25 reads the product identification information from the product tag or the IC tag installed near the product tag, Reads the product identification information from the IC tag, the IC tag recognizes that the product identification information of the IC tag attached to the product and the product identification information of the IC tag attached to the product tag can be identified by a flag or the like. It is preferred that it is.

  If it is only necessary to specify whether the product is on display or out of stock, the process for each place is not performed, and the displayed product recognition processing unit 24 checks the displayed product. The process of specifying the product identification information and the process of specifying the product identification information of the product tag in the product tag recognition processing unit 25 are performed, and by comparing them, it is possible to specify whether or not the product is missing during the display of the product. .

  By using the display status determination system 1 of the present invention, the display status of the products displayed on the display shelf can be accurately determined.

1: Display status determination system 2: Management terminal 3: Photographed image information input terminal 20: Photographed image information input reception processing unit 21: Photographed image information storage unit 22: Photographed image information normalization processing unit 23: Position identification processing unit 24 : Display product recognition processing unit 25: Product tag recognition processing unit 26: Sample information storage unit 27: Product identification information storage unit 28: Display situation determination processing unit 70: Arithmetic device 71: Storage device 72: Display device 73: Input device 74 : Communication device 241: shelf area processing section 242: place identification processing section 243: article identification information identification processing section 244: shelf image matching processing section 251: article tag arrangement area processing section 252: article tag arrangement area normalization processing Unit 253: Product tag specification processing unit 254: Information specification processing unit in product tag

Claims (12)

A display status determination system for determining a display status of a product,
The display status determination system,
A display product recognition processing unit that recognizes product identification information of a product displayed on the display shelf;
A product tag recognition processing unit that recognizes product identification information of a product tag attached to the display shelf;
A display status determination processing unit that determines the display status of the product by comparing the product identification information of the displayed product with the product identification information of the product tag;
A display status determination system comprising: The displayed product recognition processing unit includes:
By judging the similarity between the image information showing the display shelf and the specimen information, the product identification information of the product displayed on the display shelf is recognized.
The display status determination system according to claim 1, wherein: The displayed product recognition processing unit includes:
If the similarity of the sample information having the highest similarity between the image information showing the display shelf and the sample information corresponds to the sample information and the similarity is not greater than or equal to a predetermined threshold value, the product identification information is not recognized. ,
The display status determination system according to claim 1 or 2, wherein: The displayed product recognition processing unit includes:
Recognizing the product identification information of the product displayed on the display shelf based on the product identification information read by a predetermined reading device and stored in the electronic tag attached to the product displayed on the display shelf. ,
The display status determination system according to any one of claims 1 to 3, wherein: The product tag recognition processing unit,
A product tag specification processing unit that specifies a product tag area in the image information that includes the display shelf;
A product tag information specifying processor that performs OCR recognition processing on part or all of the specified product tag area to recognize product identification information;
The display status determination system according to any one of claims 1 to 4, further comprising: The product tag identification processing unit includes:
Specifying a product tag area using the brightness information of the image information;
The display status determination system according to claim 5, wherein: The product tag identification processing unit includes:
By integrating the brightness information of the image information in the horizontal direction, the upper and lower positions of the product tag are specified,
By integrating the brightness information of the image information in the vertical direction, the left side position and the right side position of the product tag are specified.
The display status determination system according to claim 5 or 6, wherein: The product tag identification processing unit includes:
By specifying the rising and falling edges of the histogram obtained by integrating the brightness information of the image information in the horizontal direction, the upper and lower positions of the product tag are specified.
By specifying the rise and fall of the histogram obtained by integrating the brightness information of the image information in the vertical direction, the fall within a predetermined distance from the rise is specified as a pair to identify the left side position and the right side position of the product tag.
The display status determination system according to any one of claims 5 to 7, wherein: The information identification processing unit in the product tag,
The bin is identified by binarizing the identified product tag area and performing a labeling process,
Of the identified boxes, boxes that satisfy a predetermined condition are identified as blocks,
Performing the OCR recognition process on the specified block;
The display status determination system according to any one of claims 5 to 8, wherein: The product tag recognition processing unit,
Edge detection is performed in a product tag placement area in the image information where the product tag is likely to be attached, and a straight line group that satisfies a predetermined condition of a vertical angle is detected. A product tag placement area normalization processing unit that specifies at least two straight lines and performs a trapezoidal correction process using the specified two straight lines.
The display status determination system according to any one of claims 5 to 9, wherein: The product tag recognition processing unit,
Based on the product identification information read by a predetermined reading device and stored in a product tag attached to the display shelf or an electronic tag attached to the vicinity thereof, or based on coded product identification information, Recognize product identification information,
The display status determination system according to any one of claims 1 to 10, wherein: Computer
A display product recognition processing unit that recognizes product identification information of a product displayed on a display shelf;
A product tag recognition processing unit that recognizes product identification information of the product tag attached to the display shelf;
A display status determination processing unit that determines the display status of the product by comparing the product identification information of the displayed product with the product identification information of the product tag;
A display status determination program characterized by functioning as a display.