JP6885563B2 - Display status judgment system - Google Patents

Description

According to the present invention, the display status for accurately determining the display status of the product displayed on the display shelf by comparing the product identification information of the displayed product with the product identification information of the product in the product tag. Regarding the judgment system.

At various stores such as convenience stores and supermarkets, it is common to place the products for sale on display shelves for sale. Therefore, by displaying a plurality of products on the display shelf, even if one of the products is purchased, the same product can be purchased by another person. If the products displayed on the display shelves are no longer displayed, the sales opportunity for the products will be lost, which is not a preferable state.

Therefore, it is required to manage the display status of products, especially whether the products displayed on the display shelves are out of stock.

In order to achieve this purpose, a method using the POS system introduced in the store can be considered. Since the POS system manages the purchase and sale of products in an integrated manner, it is possible to appropriately grasp the number of products in stock.

However, it is not possible to manage product shortages only from the POS system. The POS system can specify the number of inventories from the number of items purchased and the number of items sold, but not all of the inventories are lined up on the display shelves and may exist in the store's warehouse. is there. There is also a decrease in products that do not go through the POS system, such as shoplifting. In this case as well, even if the POS system is in stock, the product is not actually displayed.

For this reason, in order to properly grasp the display status of products (during display, shortage, etc.), especially the presence or absence of shortage, it is preferable to grasp from the actual display on the display shelf. In order to realize this, it is sufficient to photograph the display shelf with an imaging device and identify the missing item from the captured image information. In order to do this accurately, it is conceivable that a human manually identifies the products on display and the products that are out of stock from the image information obtained by photographing the display shelves. However, it is burdensome and inefficient for human beings to identify a product shortage every time from the image information obtained by photographing the display shelf.

In addition to product shortages, identifying what kind of products are on display is also an important issue for product manufacturers and sales companies from the perspective of product development strategies and sales strategies. .. And the burden of manually identifying them is heavy, and the same is true of inefficiency.

Therefore, it is required to automatically identify the products displayed on the display shelves of the store and grasp the display status of the products. For example, there is a method of using image recognition technology for an image of a store display shelf based on a sample image of each product. As these conventional techniques, for example, there are the following Patent Documents 1 and 2. Further, there is Patent Document 3 as a technique for identifying a missing item on a product display shelf.

Japanese Unexamined Patent Publication No. 5-342230 Japanese Unexamined Patent Publication No. 5-334409 International release 2012/029548

The invention of Patent Document 1 is a system that supports a person who does not have knowledge about which display shelf a product should be displayed on. Therefore, it is not possible to grasp the shortage of the products actually displayed. Further, Patent Document 2 is a system that supports input of a product image in a shelf allocation support system that supports the display of products. However, the system of Patent Document 2 only supports the input of product images when using the shelf allocation support system, and even if this system is used, is the actually displayed product out of stock? ， It is not possible to grasp the display status indicating whether the product is on display.

Patent Document 3 is an invention that specifies a product to be displayed in the empty space when there is an empty space on the display shelf, or notifies the product when the product to be displayed on the display shelf is mistakenly placed. In the case of the present invention, an image of all the products placed on the display shelf is taken in advance, and the difference from the current image is obtained to identify the position of the empty space, or the shelf board. It identifies that it is an empty space when the color or pattern is captured. However, since the products displayed on the display shelves include replacement of products, what kind of processing should be done specifically for the difference between the time when all the products are placed and the current image is used as an empty space. There is no description or suggestion as to whether it can be specified or its specific processing. For example, if there is a change in the product displayed between the first captured image and the next captured image, there will naturally be a difference between the images, but the specific content of how to handle the difference. If you do not know, it is unclear to those skilled in the art what kind of processing should be performed to recognize the difference as an empty space, and there is no feasibility at all. In addition, even when recognizing the color and pattern of the shelf board, it is often dark near the back of the display shelf, and the color information of the shelf board of the display shelf is often changed. Therefore, it is extremely difficult for a person skilled in the art to recognize the color or pattern of the display shelf as an empty space as an actual computer process, and even in this case, it is not feasible for a person skilled in the art.

Therefore, even if Patent Document 3 is used, it is not possible to actually determine whether the product displayed on the display shelf is out of stock or is on display.

In view of the above problems, the present inventor compares the product identification information of the product displayed on the display shelf with the product identification information of the product on the product tag to determine the display status of the product displayed on the display shelf. We invented a display status judgment system that can identify accurately.

The invention of claim 1 is a display status determination system for determining the display status of products, wherein the display status determination system includes a display product recognition processing unit that recognizes product identification information of products displayed on a display shelf. By comparing the product tag recognition processing unit that recognizes the product identification information of the product tag attached to the display shelf with the product identification information of the displayed product and the product identification information of the product tag. It is a display status determination system having a display status determination processing unit for determining the display status of the product.

In the present invention, since the display status of the product is determined based on the product displayed on the display shelf and the product tag, the display status of the product can be accurately determined.

In the above-described invention, the display product recognition processing unit includes image information showing the display shelf, image information obtained by photographing a product that may be displayed on the display shelf, and / or an image feature amount thereof. By determining the similarity with the sample information, it can be configured like a display status determination system that recognizes the product identification information of the products displayed on the display shelf.

There are various methods for recognizing the product identification information of the products displayed on the display shelf, but as in the present invention, it can be performed based on the image information obtained by photographing the display shelf. In this case, the product identification information of the products displayed on the display shelves can be determined by performing matching processing between the image information showing the display shelves and the sample information and determining the similarity.

In the above-described invention, the display product recognition processing unit includes image information showing the display shelf, image information obtained by photographing a product that may be displayed on the display shelf, and / or an image feature amount thereof. Of the product identification information corresponding to the sample information having the highest similarity to the sample information, if the similarity is not equal to or higher than a predetermined threshold, the product identification information is not recognized, and the system is configured like a display status determination system. be able to.

Since the matching process is performed between the image information showing the display shelf and the sample information, the similarity can always be derived. However, if the similarity is low, it cannot be said that the reliability is high. Therefore, if the similarity is not equal to or higher than a predetermined threshold value, the accuracy of determining the display status of the products can be improved by not using the result of the matching process.

In the above-described invention, the display product recognition processing unit is based on the product identification information stored in the electronic tag attached to the product displayed on the display shelf, which is read by a predetermined reading device. It can be configured like a display status judgment system that recognizes the product identification information of the products displayed in.

In order to recognize the displayed product, in addition to the processing using the image information as described above, the product may be attached with an electronic tag such as an IC tag and read by a reading device. In the case of this method, it is possible to reduce the misrecognition of the displayed products.

In the above-described invention, the product tag recognition processing unit performs OCR in the product tag identification processing unit that specifies the product tag area in the image information showing the display shelf and a part or all of the specified product tag area. It can be configured like a display status determination system having an information identification processing unit in a product tag that recognizes product identification information by performing recognition processing.

There are various methods for recognizing the product identification information of the product tag, but as in the present invention, it can be performed based on the image information obtained by photographing the display shelf. In this case, it is advisable to execute the process as in the present invention.

In the above-described invention, the product tag identification processing unit can be configured like a display status determination system that specifies a product tag area using the brightness information of the image information.

In order to recognize the product identification information of the product tag, it is necessary to specify the area of the product tag as accurately as possible. Generally, the background color of the product tag is mostly white, so when the display shelf is photographed, the part of the product tag becomes bright. Therefore, paying attention to this point, by specifying the product tag area using the lightness information (information on the amount of lightness when the image information is decomposed into hue, lightness, and saturation information), the product tag area can be accurately defined. Can be identified.

In the above-described invention, the product tag specifying processing unit identifies the upper side position and the lower side position of the product tag by integrating the brightness information of the image information in the horizontal direction, and displays the brightness information of the image information in the vertical direction. By integrating, it can be configured like a display status judgment system that specifies the left side position and right side position of the product tag.

In the above-described invention, the product tag identification processing unit specifies the upper side position and the lower side position of the product tag by specifying the rising and falling edges of the histogram obtained by integrating the brightness information of the image information in the horizontal direction. By specifying the rise and fall of the histogram that integrates the brightness information of the image information in the vertical direction, the left side position and right side position of the product tag are specified as a pair of the fall within a predetermined distance from the rise, and the display status is judged. It can be configured like a system.

When the product tag area is specified by focusing on the brightness information of the image information, the product tag area can be specified accurately by integrating the brightness information of the image information in the vertical direction and the horizontal direction, respectively.

In the above-described invention, the product tag information specifying processing unit specifies a box by binarizing the specified product tag area and performing labeling processing, and among the specified boxes, boxes satisfying a predetermined condition are used. Can be configured as a display status determination system that specifies the block as a block and executes the OCR recognition process on the specified block.

In order to recognize the information written on the product tag, OCR recognition processing may be executed for the entire product tag area, but in that case, a lot of noise etc. will be detected and the accuracy will deteriorate. Connect. It also increases the processing time. Therefore, it is preferable to execute the OCR recognition process only on a part of the product tag area. Therefore, by executing the OCR recognition process on the block specified by the process of the present invention, the recognition accuracy can be improved.

In the above-described invention, the product tag recognition processing unit performs edge detection in the product tag arrangement area, which is an area in the image information where the product tag may be attached, and satisfies a predetermined condition of a vertical angle. A product tag placement area normalization processing unit that detects a straight line group, identifies at least two straight lines from the detected straight line group, and executes a trapezoidal correction process using the specified two straight lines. It can be configured like a display status judgment system.

While the surface of the display shelf is vertical, the surface of the product tag is often attached facing upward (facing the customer) from the vertical surface. This is to make it easier for customers to see the product tag. Therefore, the product tag area can be specified more accurately by performing the product tag placement area in the normal position. This is because, for example, the rise and fall of hiss and grams in the histogram information become steep. Further, according to the present invention, the accuracy of the OCR recognition process in the product tag area can be improved.

In the above-described invention, the product tag recognition processing unit reads the product identification information or coded in the product tag attached to the display shelf or an electronic tag attached in the vicinity thereof, which is read by a predetermined reading device. It can be configured like a display status determination system that recognizes the product identification information in the product tag based on the product identification information.

In order to recognize the product identification information of the product tag, in addition to the processing using the image information as described above, an electronic tag such as an IC tag is attached to the product tag or its vicinity and read by a reading device. Therefore, the product tag recognition processing unit may recognize the product identification information of the product tag. Further, the product tag recognition processing unit may recognize the product identification information of the product tag by reading the coded product identification information attached to the product tag or its vicinity with a reading device. In the case of these methods, it is possible to reduce the misrecognition of the product identification information of the product tag.

The invention of claim 1 can be realized by loading and executing the program of the present invention in a computer. That is, the computer is a display product recognition processing unit that recognizes product identification information of products displayed on the display shelf, a product tag recognition processing unit that recognizes product identification information of product tags, and product identification of the displayed products. This is a display status determination program that functions as a display status determination processing unit that determines the display status of the product by comparing the information with the product identification information of the product tag.

By using the display status determination system of the present invention, it is possible to accurately determine the display status of the products displayed on the display shelf.

It is a block diagram which shows typically an example of the structure of the display condition determination system of this invention. It is a block diagram which shows typically an example of the structure of the display product recognition processing unit in the display situation determination system of this invention. It is a block diagram which shows typically an example of the structure of the product tag recognition processing part in the display situation determination system of this invention. It is a block diagram which shows typically an example of the hardware composition of the computer used in the display situation determination system of this invention. It is a flowchart which shows an example of the whole processing process in the display situation determination system of this invention. It is a figure which shows an example of photographed image information. It is a figure which shows an example of photographed image information. It is a figure which shows an example of the orthodox image information which performed the keystone correction processing with respect to the photographed image information of FIG. It is a figure which shows an example of the orthodox image information which performed the keystone correction processing with respect to the photographed image information of FIG. Schematically, the state in which the input of the designation of the shelf stage area and the product tag placement area is accepted for the vertical image information obtained by placing the image information of the display shelf on which products such as beverage cans are displayed in a normal position. It is a figure which shows. A state in which input for specifying the shelf area and product tag placement area is accepted for the orthodox image information in which the image information of the display shelf on which products such as toothbrushes are hung and displayed is placed upright. It is a figure which shows typically. It is a figure which shows an example of the sample information which is stored in the sample information storage part. It is a figure which shows typically the relationship between a place and a face in a display shelf. It is a figure which shows an example of the image information of the product tag arrangement area which was put upright with respect to the image information of the product tag arrangement area of FIG. It is a figure which shows an example of the image information of the product tag arrangement area of FIG. It is a figure which shows typically the process of specifying the upper side position and the lower side position of the product tag area in the product tag identification processing unit. It is a figure which shows typically the process which specifies the position in the lateral direction of the product tag area in the product tag identification processing unit. It is a figure which shows typically the state which specified the product tag area from the image information of the product tag arrangement area which was put upright in FIG. It is a figure which shows typically the image information which binarized the product tag area in the processing in the product tag information identification processing unit. It is a figure which shows typically the state which generated the box in the processing in the information identification processing part in a product tag. It is a figure which shows typically the state which generated the block in the processing in the information identification processing part in a product tag. It is a figure which shows an example of the product name with the smallest edit distance, and the edit distance among the product names stored in the product dictionary as a result of the product name and OCR recognition written on the product tag. It is a figure which shows an example of the correspondence relation of the edit distance which may be decided for each number of characters of a final candidate. It is a figure which shows typically an example of the process of associating the product identification information recognized by the display product recognition processing unit with the product identification information of a product tag recognized by a product tag recognition processing unit. It is a figure which shows an example of photographed image information. It is a figure which shows an example of the orthodox image information which performed the keystone correction processing on the photographed image information of FIG. It is a figure which shows typically the correspondence relationship between photographed image information and depth information. It is a figure which shows an example of the depth information. It is a figure which shows typically the state which the conversion processing which is the same as the keystone correction processing such as the keystone correction processing in the photographed image information is executed also for the depth information. It is a figure which shows typically the state which performed the same conversion process as the keystone correction process such as the keystone correction process in the photographed image information with respect to the depth information of FIG. 28. It is a figure which shows typically an example of the depth from the regression plane of a place for each place. It is a figure which shows typically the process of finding the regression plane of a place position. It is a figure which shows an example of the process which makes a judgment based on the change of the place state and the change of a product tag between the last time (N-1 time) and this time (N times). It is a figure which shows typically the Nth processing in the product identification information identification processing part in Example 5.

An example of the processing function of the display status determination system 1 of the present invention is shown in FIGS. 1 to 3 in a block diagram. The display status determination system 1 uses the management terminal 2 and the captured image information input terminal 3. FIG. 1 is a block diagram showing the overall functions of the display status determination system 1, FIG. 2 is a block diagram showing the functions of the display product recognition processing unit 24 described later, and FIG. 3 is a block diagram showing the functions of the product tag recognition processing unit 25 described later. It is a block diagram which shows the function of.

The management terminal 2 is a computer used by an organization such as a company that operates the display status determination system 1. Further, the photographed image information input terminal 3 is a terminal for inputting image information obtained by photographing the display shelf of the store.

The management terminal 2 and the photographed image information input terminal 3 in the display status determination system 1 are realized by using a computer. FIG. 4 schematically shows an example of the hardware configuration of the computer. The computer can input information from an arithmetic unit 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, and a display device 72 such as a display that displays information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits and receives processing results of the arithmetic unit 70 and information stored in the storage device 71 via a network such as the Internet or LAN.

When the computer is provided with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used, for example, in portable communication terminals such as tablet computers and smartphones, but are not limited thereto.

The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be performed directly on the display with a predetermined input device (such as a pen for a touch panel) or a finger.

The captured image information input terminal 3 may include a photographing device such as a camera in addition to the above-mentioned devices. As the captured image information input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used.

Each means in the present invention has only a logical distinction in its function, and may form the same area physically or substantially. The processing order of the processing in each means of the present invention may be changed as appropriate. In addition, a part of the processing may be omitted. For example, the emplacement process described later can be omitted. In that case, it is possible to execute processing on the image information that has not been subjected to the orthostatic processing.

When determining the display status of a product based on the photographed image information, the display status determination system 1 identifies the position of the photographed image information input reception processing unit 20, the photographed image information storage unit 21, the photographed image information normalization processing unit 22, and the position. It has a processing unit 23, a display product recognition processing unit 24, a product tag recognition processing unit 25, a sample information storage unit 26, a product identification information storage unit 27, and a display status determination processing unit 28. Therefore, as will be described later, when the image information is not used, it is not necessary to provide the photographed image information input reception processing unit 20, the photographed image information storage unit 21, and the photographed image information emplacement processing unit 22, and the display product. The recognition processing unit 24 and the product tag recognition processing unit 25 do not have to provide processing related to image information.

The photographed image information input reception processing unit 20 receives the input of the image information (photographed image information) of the display shelf of the store photographed by the photographed image information input terminal 3 and stores it in the photographed image information storage unit 21 described later. In addition to the photographed image information, the photographed image information input terminal 3 may accept input including the store identification information such as the shooting date and time and the store name, and the image information identification information for identifying the image information. 6 and 7 show an example of captured image information. In FIGS. 6 and 7, the display shelves have three shelves, and the photographed image information on which the products are displayed is shown.

The photographed image information storage unit 21 stores the photographed image information, the photographed date and time, the store identification information, the image information identification information, and the like received from the photographed image information input terminal 3 in association with each other. The captured image information may be any image information to be processed according to the present invention. In general, when simply taking a picture, it is difficult to take a picture with the object to be photographed facing the subject. Therefore, a correction process for correcting the object to be photographed, for example, a keystone correction process, should be executed. Is good. When a single display shelf is photographed with a plurality of images, the image information combined as one image information is also included. In addition, the captured image information also includes the image information after the distortion correction processing is executed.

The photographed image information emplacement processing unit 22 corrects the photographed image information stored in the photographed image information storage unit 21 so that the object to be photographed faces the photographed image information (imposition processing), for example. Generates orthodox image information that has been subjected to trapezoidal correction processing. The keystone correction process is a correction process performed so that the shelves of the display shelves shown in the photographed image information are horizontal and the product tags for the products displayed there are vertical. Orthogonalization is to transform the image information so that the optical axis of the lens of the photographing device is the same as when the image is taken from a sufficient distance along the perpendicular direction of the plane to be photographed. For example, a trapezoid. There is a correction process.

The keystone correction process executed by the captured image information emplacement processing unit 22 receives input for designating four vertices in the photographed image information, and executes the keystone correction process using each of the vertices. The four vertices that accept the designation may be the four vertices of the shelf stage of the display shelf or the four vertices of the shelf position of the display shelf. Further, it may be four vertices of a group of two or three shelves. Any four points can be specified as the four vertices. FIG. 8 shows an example of the photographed image information (orthogonal image information) in which the photographed image information of FIG. 6 and the photographed image information of FIG. 7 are embossed.

The position identification processing unit 23 is the area of the shelf where products may be arranged among the normal image information obtained by performing the trapezoidal correction processing on the photographed image information in the photographed image information normalization processing unit 22. Identify the area (shelf area) and the area where the product tag may be attached (product tag placement area). That is, although the display shelves are shown in the photographed image information and the vertical image information, there is a possibility that the shelf area where the products are displayed and the product tags for the products displayed there are attached to the display shelves. There is a certain product tag placement area. Therefore, the shelf area and the product tag placement area are specified from the normal image information. To specify the shelf area and the product tag placement area, the operator of the management terminal 2 may manually specify the shelf area and the product tag placement area, and the position identification processing unit 23 may accept them for the first time. The shelf area and the product tag placement area may be automatically specified from the second time onward based on the information of the shelf area and the product tag placement area for which input is manually received.

FIG. 10 shows a state in which input for designating the shelf stage area and the product tag placement area is accepted for the orthodox image information in which the image information obtained by photographing the display shelf on which products such as beverage cans are displayed is placed upright. Is schematically shown. Further, in FIG. 11, the shelf stage area and the product tag placement area are designated with respect to the vertical image information obtained by rectifying the image information obtained by photographing the display shelf in which products such as toothbrushes are hung and displayed. The state in which the input is accepted is schematically shown.

The display product recognition processing unit 24 executes a process of recognizing the products displayed on the shelves in the display shelf shown in the normal image information. The details of the processing of the display product recognition processing unit 24 will be described later. In the processing described later, the case where the display product recognition processing unit 24 recognizes the product identification information of the displayed product based on the photographed image information obtained by photographing the display shelf will be described, but the product identification information is provided to the product. The product identification information of the displayed product may be recognized by attaching a stored IC tag (electronic tag) or the like and reading the product identification information of the displayed product from the IC tag.

The product tag recognition processing unit 25 executes a process of recognizing the product identification information described in the product tag in the product tag area of the display shelf shown in the normal image information. Details of the processing of the product tag recognition processing unit 25 will be described later. In the following description, as the recognition of the product identification information, the case where the product identification information is recognized by reading the character information described on the product tag will be described, but the bar described on the product tag or its vicinity will be described. The product identification information may be recognized by reading the information (code information) in which the product identification information such as the code, the two-dimensional code, and the color code is coded. Further, the product identification information may be recognized by reading the product identification information from the product tag or an IC tag (electronic tag) installed in the vicinity thereof. When reading the product identification information from the code information or the IC tag, for example, the product identification information is read by a reading device such as a code reader or a non-contact communication device 74, and the information read by the reading device is recognized by the product tag recognition processing unit 25. To do.

The sample information storage unit 26 stores sample information for identifying which product is displayed on the shelf of the display shelf shown in the normal image information. Specimen information is image information obtained by photographing products that may be displayed on a display shelf from a plurality of angles such as up and down, left and right, and diagonally. FIG. 12 shows an example of sample information stored in the sample information storage unit 26. FIG. 12 shows the case where canned beer is photographed from various angles as sample information, but the sample information is not limited to canned beer. The sample information storage unit 26 stores the sample information and the product identification information in association with each other.

In addition, in the sample information storage unit 26, information necessary for calculating similarity, for example, information of a pair of an image feature amount and its position, which is extracted from the sample information and is extracted together with the sample information or instead of the sample information. You may remember it. The sample information shall also include the information required to calculate the similarity. In this case, the display product recognition processing unit 24 does not have to calculate the image feature amount for the sample information each time when performing the matching process between the image information in the place area described later and the sample information, and the calculation time. Can be shortened.

The product identification information storage unit 27 stores the product identification information of the product displayed in each place on the shelf of the display shelf or the information indicating that the product is “empty”. For example, product identification is associated with product identification information, such as shooting date / time information, store information, image information identification information of shot image information, image identification information of normal image information, and place identification information for identifying a place. It is stored in the information storage unit 27. FIG. 13 schematically shows the relationship between the place and the face on the display shelf.

The display status determination processing unit 28 determines the display status of whether the product is on display or out of stock. For example, when determining a product shortage as a display status, a product that can be recognized from the product tag but is not recognized as being displayed on the display shelf is determined as a product shortage. Specifically, among the product identification information recognized by the product tag recognition processing unit 25, the product identification information that is not included in the product identification information recognized by the display product recognition processing unit 24 is the product identification information of the missing product. Judge as.

For example, the shortage is determined by the following method. Of the product identification information (product identification information of the face) for each place recognized by the display product recognition processing unit 24, if the same product identification information is included in the adjacent place, they are regarded as one product group. Summarize product identification information. Further, among the product identification information recognized by the product tag recognition processing unit 25, the same product identification information is combined into one product identification information. Then, the product identification information collected from each is compared with each other, and it is determined whether or not there is matching product identification information. In this case, the order of the product identification information does not matter.

Then, among the product identification information recognized by the product tag recognition processing unit 25, the product identification information recognized by the display product recognition processing unit 24 is determined to be on display. Of the product identification information recognized by the display product recognition processing unit 24, the product identification information that is not included in the product identification information recognized by the product tag recognition processing unit 25 is determined to be nonexistent of the product tag. Further, among the product identification information recognized by the product tag recognition processing unit 25, the product identification information that is not included in the product identification information recognized by the display product recognition processing unit 24 is determined as the product identification information of the missing product. .. At this time, if the corresponding place of the product tag of the product identification information of the product determined to be out of stock is "empty", it can be accurately determined as out of stock. Even if the place is not "empty", it can be judged as a missing item. FIG. 24 schematically shows this process.

When recognizing the product identification information of the products displayed on the display shelf based on the photographed image information, the display product recognition processing unit 24 includes the shelf area processing unit 241 and the place identification processing unit 242 and the product identification information identification processing unit. It has 243 and a shelf image matching processing unit 244.

The shelf area processing unit 241 specifies the image information of the shelf area specified by the position specifying processing unit 23 as the shelf area image information. The shelf area processing unit 241 may actually cut out as image information, or may virtually cut out by specifying the image information of the area without actually cutting out as image information. If there are multiple shelves on the display shelf, each is cut out as shelf area image information. The coordinates indicating the area of the shelf are the coordinates of the vertices for specifying the area, and may be, for example, the coordinates of four points in the orthodox image information, the coordinates of the upper right and the lower left, and the coordinates of the upper left and the lower right. In addition, the coordinates are relative to a predetermined location (for example, the upper left apex of the display shelf) in the image information such as the display shelf in the normal image information.

The place identification processing unit 242 specifies the place area for each shelf in the shelf area in the normal image information. A place is an area where a product is placed, and it does not matter whether or not the product is placed. The size of the area of the place is the same as or approximately the same size as the goods to be placed there. The place where the product is actually displayed is called the "face". If no products are displayed in the place, it is regarded as "empty". The place specifying processing unit 242 is preferably divided into a place specifying process for the first time and a place specifying process for the second and subsequent times.

In the first place identification process in the place identification processing unit 242, the place is specified within the range of the area (preferably a rectangular area) composed of the coordinates of the shelf step specified by the position identification processing unit 23. Specifically, there are restrictions such as identifying narrow and narrow shadows that occur between products in the shelf area, identifying repeating patterns of images, identifying steps on the upper side of the package, and having the same product width. The area of the place is specified by specifying the delimiter position based on. As the place identification process, any method can be adopted depending on the product category and the product form, and the process is not limited to the above. In addition, the person in charge may accept correction input for the automatically specified place. Furthermore, the person in charge may accept input of the position of the place. The coordinates of the area that constitutes the specified place are the coordinates of the place area in the normal image information, the shooting date and time information, the store information, the image information identification information of the shot image information, the image identification information of the normal image information, and the place. The place identification information for identification is associated and managed. The coordinates indicating the place area are the coordinates of the vertices for specifying the rectangular area, and may be, for example, the coordinates of four points in the orthodox image information, the coordinates of the upper right and the lower left, and the coordinates of the upper left and the lower right. In addition, the coordinates are relative to a predetermined location (for example, the upper left apex of the display shelf) in the image information such as the display shelf in the normal image information.

In general, similar types of products are often placed on one shelf. In that case, the size of the product is almost the same. Therefore, when the area of the place of one product is specified, the place on the shelf may be specified by repeatedly setting the area on the shelf.

In the second and subsequent place identification processes in the place identification processing unit 242, the coordinates of the place area specified in the previous (N-1th) normal image information for the same shelf on the same display shelf are used this time (N-1). It may be set to the coordinates of the area of the place specified by the normal image information of the Nth time).

The product identification information identification processing unit 243 specifies the product identification information of the product displayed in the place for each shelf of the display shelf by using the sample information stored in the sample information. The product identification information includes, but is not limited to, the product name and the JAN code assigned to the product. Any information that can identify the product may be used.

The product identification information identification processing unit 243 executes the following processing. That is, by matching the image information of the place with the sample information of the product stored in the sample information storage unit 26 for each place, the product identification information of the product displayed in the place is specified. Specifically, first, the similarity between the image information of the area composed of the coordinates of the place to be processed and the sample information stored in the sample information storage unit 26 is determined, and the sample information having the highest similarity is determined. If the specified similarity is equal to or higher than a predetermined threshold, the product identification information corresponding to is specified as the product identification information of the product displayed in the place composed of the above coordinates.

Here, in order to determine the similarity between the image information of the place and the sample information, the following processing is performed. First, in the processing before the product identification information identification processing in the product identification information identification processing unit 243, the directions of the image information of the place area on the shelf of the orthodox image information and the sample information are the same (overturn or inverted). (Not), and the size of each image information is roughly the same (if the size of the image information is different within a predetermined range, each image information is before the judgment of similarity. Adjust the size so that the size of the image information is within the specified range).

The product identification information identification processing unit 243 determines the similarity between the image information of the place and the sample information, so that the feature points based on the image feature amount (for example, the local feature amount) of the image information of the place and the sample information Each feature point based on an image feature amount (for example, a local feature amount) is extracted. Then, the pair with the highest similarity between the feature points of the image information of the place and the feature points of the sample information is detected, and the difference in the coordinates of the corresponding points is obtained. Then, find the average value of the difference. The average value of the difference indicates the total average movement amount between the image information of the place and the sample information. Then, the coordinate difference of all the feature point pairs is compared with the average coordinate difference, and the pair with a large degree of deviation is excluded. Then, the similarity is ranked by the number of remaining corresponding points.

The similarity between the image information of the place and the sample information can be calculated by the above method. Further, in order to improve the accuracy, the EMD (Earth Movers Distance) between the color histograms may be obtained and used as a measure of similarity. As a result, it is possible to compare similarities that are relatively strong against environmental changes such as brightness information of captured image information, and it is possible to identify with high accuracy.

In addition to the determination of similarity, the EMD of the signatures (set of image features and weights) of the image information in each place area may be obtained and used as a measure of similarity. As the image feature amount of the signature, for example, the frequency distribution of the image information in the place area in the HSV color space is obtained, the hue and saturation are grouped, and the number of features and the image feature amount by the area in the HSV color space are obtained. can do. The reason for grouping the hue and saturation is to reduce the dependence on brightness so that it is not greatly affected by the shooting conditions.

Further, in order to speed up the processing, it is possible to use the similarity such as the L2 distance between the image features such as the color colorogram and the color histogram of the image information in an appropriate color space instead of the signature and the EMD.

The determination of similarity is not limited to the above. The identified product identification information corresponds to the shooting date / time information, store information, image information identification information of shot image information, image identification information of normal image information, and place identification information for identifying a place, and is associated with the product identification information storage unit. Store in 27.

Since the place where the product identification information can be specified is a face, the product identification information storage unit 27 stores information indicating that the place is a "face", and the place where the product identification information cannot be specified is ". Since it is "empty", information indicating that the place is "empty" is stored in the product identification information storage unit 27.

When the display product recognition processing unit 24 recognizes the product identification information stored in the IC tag attached to the product, the information read by the portable communication terminal equipped with a predetermined IC tag reading device is used. It may be received from the portable communication terminal together with the position information on the shelf. In this case, each process in the display product recognition processing unit 24 described above may or may not be provided. The place in this case can also be specified by predetermining the order in which the IC tags are read by the reading device. That is, by holding the reading device in order from the right to the left of the shelf, the places are specified in order in the order in which the product identification information or the unreadable items are recognized. In this case, the reader may be held over the position (void) where there is no product to recognize that the product identification information cannot be read, that is, it is "empty", and the position where there is no product is not read. May be good. Further, the place identification processing unit 242 described above may specify the place from the photographed image information or the orthotopic image information, and specify the product identification information by associating it with the product identification information read by the reading device.

The product identification information identified as described above is shooting date / time information, store information, image information identification information of shot image information, image identification information of orthodox image information, place identification information for identifying a place, and a face. It is stored in the product identification information storage unit 27 in association with the information indicating whether it is empty or empty.

The shelf image matching processing unit 244 includes the image information of the shelf area in the previous (N-1st) normal image information and the image information of the shelf area in the current (Nth) normal image information. If the similarity is high, it is determined that the product identification information or "empty" of each place on the shelf is the same. As described above, this similarity determination process is performed on the image feature amount of the image information of the shelf area in the previous (N-1st) vertical image information and the current (Nth) vertical image information. The similarity may be determined based on the image information of the shelf area, or the EMD of the color histograms may be used. Moreover, it is not limited to them. Then, instead of the specific processing for each place in the product identification information identification processing unit 243, the product identification information identification processing unit 243 is provided with the product identification information of each place on the shelf in the Nth normal image information. The product identification information storage unit 27 stores the product identification information as the same as the product identification information of each place on the same shelf for the first time. As a result, it is possible to omit the processing for the shelves that hardly change, such as the shelves where the products do not move much and the shelves that are managed in an extremely short cycle. It is not necessary to provide the processing by the shelf image matching processing unit 244.

When recognizing the product identification information of the product tag based on the photographed image information, the product tag recognition processing unit 25 includes the product tag arrangement area processing unit 251 and the product tag arrangement area normalization processing unit 252 and the product tag identification processing unit 253. And 254 of the information identification processing unit in the product tag.

The product tag arrangement area processing unit 251 cuts out the image information of the product tag arrangement area specified by the position identification processing unit 23 as the product tag arrangement area image information. The product tag arrangement area processing unit 251 may actually cut out as image information, or may cut out virtually without actually cutting out as image information. Virtually cutting out image information means executing processing with the specified area, for example, the range of the product tag placement area as the processing target. If there are a plurality of product tag placement areas on the display shelf, each of them is cut out as product tag placement area image information. The coordinates indicating the product tag placement area are the coordinates of the vertices for specifying the area, and may be, for example, the coordinates of four points in the orthodox image information, the coordinates of the upper right and the lower left, and the coordinates of the upper left and the lower right. In addition, the coordinates are relative to a predetermined location (for example, the upper left apex of the display shelf) in the image information such as the display shelf in the normal image information.

The product tag placement area normalization processing unit 252 executes trapezoidal correction processing for normalizing the product tag placement area image information cut out by the product tag placement area processing unit 251. While the surface of the display shelf is vertical, the surface of the product tag is often facing upward from the vertical surface so that the customer can easily see it. Therefore, the recognition accuracy is improved by making the image information of the product tag placement area normal. FIG. 14 shows an example of the image information of the product tag arrangement area that has been rectified with respect to the image information of the product tag arrangement area of FIG. FIG. 14 (a) shows the image information of the product tag placement area in which the image information of the product tag placement area of the upper shelf in FIG. 10 is rectified, and FIG. 14 (b) shows the image information of the lower shelf in FIG. This is the image information of the product tag placement area in which the image information of the product tag placement area is rectified.

The product tag placement area normalization processing unit 252 normalizes the image information of the product tag placement area by executing the following processing. That is, in the image information of the product tag placement area, edge detection is performed, and a nearly vertical contour line of a certain length or more (for example, 70 degrees to 110 degrees) is vertical (90 degrees) at a location near the left and right ends. ) To specify the contour line) within a predetermined range of angles. It is preferable, but not limited to, to extract the contour lines near the left and right ends. In the case of the image information of the product tag arrangement area of FIG. 10, L1 to L4 are specified, respectively, as shown in FIG. FIG. 15A is a product tag placement area on the upper shelf in FIG. 10, and FIG. 15B is a product tag placement area on the lower shelf in FIG. The contour lines L1 to L4 to be specified are not actually drawn in the image information of the product tag arrangement area. Then, trapezoidal correction processing is executed for the image information of the product tag arrangement area so that L1 and L2 in FIG. 15A and L3 and L4 in FIG. 15B are vertical lines, respectively. By executing such a process, the image information of the product tag arrangement area is made normal, and the image information of the product tag arrangement area shown in FIG. 14 can be obtained. By executing the processing of the product tag placement area normalization processing unit 252, the accuracy of the product tag specifying processing unit 253 and the information specifying processing unit 254 in the product tag can be improved, so that processing is executed. It is preferable, but it can be omitted. In that case, the product tag identification processing unit 253 and the product tag in-information identification processing unit 254 execute the product tag arrangement area cut out by the product tag arrangement area processing unit 251.

The product tag identification processing unit 253 specifies an area (product tag area) of each product tag from the product tag arrangement area image information that has been placed upright. Two main methods can be used for the process of specifying the product tag area. The first method is a method of specifying the product tag area based on the contour line, and the second method is to match the overall features such as the distribution of the entire light and darkness with the image information of the product tag template. This is a method of specifying the product tag area. A method other than the first method and the second method can also be used.

The first method is a method that utilizes the fact that the background color (background color) of the product tag is often white (brighter than the background). That is, first, among the image information of the product tag placement area that has been placed upright, a histogram that integrates the brightness information of the image information in the horizontal direction is generated. Then, the rising and falling positions of the histogram are specified, and the upper side position A and the lower side position B of the product tag are specified. FIG. 16 schematically shows this process. The rising edge is the point where the histogram increases sharply from black to white (more than a predetermined ratio), and the falling edge is the point where the histogram sharply decreases from white to black. ..

Then, out of the image information of the product tag placement area that has been placed upright, a histogram is generated in which the brightness information of the image information of the product tag placement area that has been placed upright is integrated in the vertical direction by cutting out between the upper side position A and the lower side position B. To do. Then, the rising and falling positions of the histogram are specified, and the rising edge and the falling edge within a predetermined distance range to the right of the rising edge are paired, and the product tag area is specified by setting each as the left side position and the right side position. Then, if there is no product tag area within a predetermined distance, it is specified as a product tag area on the right side of the rising edge where the pair could not be formed and on the left side of the falling edge where the pair could not be formed. FIG. 17 schematically shows this process.

Further, FIG. 18 shows a state in which the product tag area is specified from the image information of the product tag placement area that is placed upright in FIG. 14 by the first method. The rectangular area composed of each of the upper side position A, the lower side position B, the left side position (rising) U, and the right side position (falling) D is the specified product tag area.

There are also product tags in which the background color (background color) of the product tag is other than white, for example, red, and the characters are white. Therefore, if the price information or product identification information in the product tag cannot be recognized by the above processing, the relationship between white and black is reversed (brightness is reversed) in the image information of the product tag placement area. The same process as described above may be executed. This makes it possible to correspond to a product tag with a dark background and bright characters.

Note that the process of reversing the brightness of the image information in the product tag placement area is executed not only when the information in the product tag cannot be recognized, but also when it cannot be recognized normally in any of the processes in the process of identifying the product identification information. It may be executed regardless of whether it can be recognized or not.

The second method is so-called template matching. That is, the product tag area is specified by registering the image information of the product tag as a template in advance and matching the template with the image information of the product tag placement area that has been placed upright.

The product tag includes the tax-excluded price, the tax-included price, the product identification information (product name, etc.), the manufacturer name, the rating, and the like. Therefore, if the image information of the product tag used as a template includes product identification information such as the product name, specific price numbers, and characters, that part is also included in the judgment target of the image matching process, so that part is mosaicked. , It is preferable to neutralize or exclude it from the judgment target by deleting it. Neutralization means not allocating high points and low points for any input, and exclusion means excluding that part from the matching target during image matching processing.

The information identification processing unit 254 in the product tag executes a process of specifying the information described in the product tag area specified by the product tag identification processing unit 253 by OCR recognition or the like. When OCR recognition is performed, it can be performed for all or a part of the area specified as the product tag area. If it is applied to all the areas specified as the product tag area, there is a high possibility that erroneous recognition will occur due to noise or the like, so it is preferable to limit the target area for OCR recognition. In this case, the processes corresponding to the first method and the second method used in the product tag specifying processing unit 253 may be executed.

When the first method is used by the product tag specifying processing unit 253, the binarization processing is first performed in the specified product tag area. As the binarization process, one of the method using the brightness of the HSV color space, the method of Otsu, the method of extracting red (the number and character string of the product tag are frequently used in addition to black, and red is also frequently used). There is a method of selecting a candidate that is extracted by combining a plurality of items.

Then, the labeling process is executed on the binarized image information. The labeling process is a process of assigning the same number (identification information) to pixels in which white or black is continuous in binary image information, thereby forming consecutive pixels into one island (group). is there. Then, a rectangular area (box) including the islands detected by the labeling process is generated, and the height, width, and baseline of the box are obtained. When generating a box, it is preferable, but not limited to, to generate a rectangle surrounded by the smallest vertical and horizontal line segments surrounding the areas labeled with the same number. When generating a box, islands that do not satisfy the height and width that are the predetermined thresholds are not generated as noise and are removed from the processing target as they are. For example, an island that is too small may have horizontal ruled lines or dust on the image, and an island that is too wide may have a logo, etc., which are removed as noise.

In general, the characters used in product tags are often in bold Gothic fonts, which have a clear outline of the entire characters. Therefore, even if the image information is slightly out of focus, the character group forming one character string can be detected as an island having the same height as the baseline.

Then, the information identification processing unit 254 in the product tag merges adjacent boxes having predetermined similarities to form a block. That is, a block is formed by merging consecutive boxes whose baseline and height match within a predetermined range and whose height and width are within a certain threshold range. At this time, small boxes and the like between the blocks to be merged are also collectively configured as one block. As a result, voiced sound marks, semi-voiced sound marks, hyphens, etc. are also incorporated into one block. The block is an area to be recognized by OCR. Then, among the blocks, the block with the highest height is estimated as the price range (price range excluding tax), and OCR recognition is performed. Similarly, OCR recognition is performed for other blocks. By executing the above processing, it is possible to execute the character recognition processing more accurately than performing OCR recognition on the manuscript portion freely laid out over a plurality of lines in the product tag area. 19 to 21 show this process schematically. FIG. 19 is binarized image information, and FIG. 20 is a diagram showing a state in which a box (region shown by a broken line) is generated. Further, FIG. 21 is a diagram showing a state in which a block (region shown by a broken line) is generated.

As described above, the information identification processing unit 254 in the product tag can recognize the information described in the product tag as characters.

When the second method is used by the product tag identification processing unit 253, the product identification information such as the tax-excluded price, the tax-included price, the manufacturer name, and the product name, and the rating are written in advance in the image information of the product tag used as the template. The position and size (height, width) of each character frame are set. Therefore, the image information of the corresponding portion is cut out from the product tag area specified by the product tag specifying processing unit 253, and the OCR recognition process is executed. At this time, by defining the product identification information such as price, manufacturer name, product name, and the character type used by the rating (for example, which of numbers, romaji, symbols, and kana-kanji is included) as constraint conditions. The accuracy of OCR recognition processing can be improved.

Further, the information identification processing unit 254 in the product tag may execute a process of confirming the consistency of the read information. It is preferable to perform two types of consistency confirmation processing: consistency confirmation processing by dictionary collation and logical check.

The consistency confirmation process by dictionary collation is executed as follows, for example. The display status determination system 1 stores product identification information such as product names of products that may be displayed on display shelves and corresponding code information (for example, JAN code) in association with each other (Fig.). Not shown). Then, the character string similarity evaluation between the character string other than the character string read from the area indicating the price recognized by the information identification processing unit 254 in the product tag and the product identification information such as all the product names registered in the product dictionary. I do. For example, the editing distance (Levenshtein distance) and the Jaro-Winkler distance can be used.

If the maximum value of the character string similarity obtained above is one, the product identification information such as the product name is used as the final candidate. Then, the allowable similarity is determined in advance for the length of the character string of the product identification information such as the final candidate product name, and if the similarity is equal to or higher than the acceptable similarity, the product identification information such as the product name is identified. To do. If the acceptable similarity is not reached, the read character string is undetermined. If there are multiple products with the maximum similarity, the read character string is undetermined.

The edit distance, which is one of the character string similarity evaluations, is a kind of distance that indicates how different the two character strings are. Specifically, one of them is inserted, deleted, or replaced by inserting, deleting, or replacing one character. The minimum number of steps required to transform a string into the other string. FIG. 22 shows an example of the product name and the editing distance with the shortest editing distance among the product names stored in the product dictionary as a result of the product name and OCR recognition written on the product tag. Further, FIG. 23 shows a table of correspondences of edit distances that may be determined for each number of characters of the final candidate. In this specification, the case of processing using the edit distance will be described, but the character string similarity evaluation may be defined so that the substitution has a greater similarity than the insertion and deletion.

Then, after obtaining the product identification information such as the product name with the minimum editing distance described above, a rating dictionary (shown) is separately prepared for the character string excluding the relevant part of the product identification information such as the product name from the OCR recognition result. The editing distance with each rating stored in (not shown) and the editing distance with each manufacturer name stored in the manufacturer name dictionary (not shown) are obtained. For example, in the example of "350 ml of throat raw" in FIG. 22, the part of "350 ml" matches "350 ml" in the rated dictionary with an editing distance of 0, and it is identified that it is a character string of the rated part. Similarly, the editing distance is obtained for the manufacturer name, and the character string with the minimum editing distance is identified as the manufacturer name. Then, each character string of the rated part and the manufacturer name part is removed from the OCR recognition result, and the character string having the shortest editing distance in the product name dictionary is obtained from the removed remaining character string, and is it an acceptable editing distance? Judge whether or not. If it is within the allowable editing distance, determine the product identification information such as the product name, the manufacturer name, and the rating. By performing such processing, even if the product identification information such as the product name includes the rating and the manufacturer name, it can be appropriately determined. The rating dictionary is a dictionary that stores the ratings (capacity, etc.) of products that may be displayed on the display shelves. The manufacturer name dictionary is a dictionary that stores the manufacturer names of products that may be displayed on the display shelf.

In order for the user to finally judge and approve the recognition result by the information identification processing unit 254 in the product tag, when displaying an input such as selection of the recognition result, a confirmed character string or an unconfirmed character string is displayed. It is good that the display is performed in a identifiable way. For example, the color of a confirmed character string and an unconfirmed character string may be separated, and information indicating confirmed or unconfirmed may be added to a confirmed character string and / or an unconfirmed character string. For undetermined character strings, if there are multiple product identification information such as the product name with the minimum editing distance, each product identification information may be displayed.

The logical check (determination process of logical consistency of the recognized information) in the information identification processing unit 254 in the product tag can be performed as follows. For example, if you read two prices as the price range, one will be the tax-excluded price and one will be the tax-included price, but if the tax-excluded price is higher than the tax-included price, they will be recognized in reverse. Let me. If the value obtained by multiplying the tax-excluded price by the consumption tax rate does not match the tax-included price, it is assumed that either or both are misrecognized. Furthermore, when the product identification information such as the product name is recognized, it is determined whether the product or the product category is within the normal price range. In addition, it may be determined whether the correspondence between the product identification information such as the product name, the manufacturer name, and the rating matches.

By confirming the consistency of the information in the product tag as described above, the information included in the product tag reflected in the photographed image information can be determined. The information determined in this way can be output in tabular format, for example.

Next, an example of the processing process of the display status determination system 1 of the present invention will be described with reference to the flowchart of FIG. In the following description, the case where the product identification information of the displayed product and the product identification information of the product tag are specified from the photographed image information and the product is judged to be out of stock will be described. By reading the product identification information stored in the IC tag installed in the vicinity with a reading device, the product identification information can be specified and the product shortage can be determined.

The photographed image information photographed on the display shelf of the store is input from the photographed image information input terminal 3, and the input is accepted by the photographed image information input reception processing unit 20 of the management terminal 2 (S100). FIG. 25 shows an example of captured image information. It also accepts input of image information identification information such as shooting date and time, store identification information, and shooting image information. Then, the captured image information input reception processing unit 20 stores the captured image information, the shooting date and time, the store identification information, and the image information identification information of the captured image information in association with each other in the captured image information storage unit 21.

When the management terminal 2 receives a predetermined operation input, the normal image information normalization processing unit extracts the photographed image information stored in the photographed image information storage unit 21 and performs positive value processing such as trapezoidal correction processing. The input of four points of the shelf position (the position of the display shelf), which is the apex of the image, is received, and the positive value processing is executed (S110). FIG. 26 shows an example of the captured image information (normal image information) in which the normalization process is executed in this way.

Then, the position specifying processing unit 23 specifies the shelf stage position area and the product tag placement area by receiving a predetermined operation input in the management terminal 2 for the normal image information (S120). That is, the input of the shelf area and the product tag arrangement area in the normal image information is accepted. FIG. 10 is a diagram showing a state in which the shelf stage area and the product tag arrangement area are specified.

When the shelf area and the product tag placement area are specified as described above, the display product recognition processing unit 24 performs the display product recognition process in the shelf area, and the product tag recognition process in the product tag placement area performs the product tag recognition process. Each unit 25 executes (S130, S140). The display product recognition processing unit 24 may perform the display product recognition processing, and the product tag recognition processing unit 25 may perform the product tag recognition processing in parallel or at different timings.

First, the product identification information recognition process of the product in the display product recognition processing unit 24 will be described.

The shelf area processing unit 241 of the display product recognition processing unit 24 cuts out the image information of the shelf area from the normal image information based on the area of the shelf that has received the input in S120. Then, the process of specifying the place is executed for each shelf in the shelf area image information. Specifically, for the shelf in the shelf area, within the range of the rectangular area composed of the coordinates of four points, the narrow and narrow shadows generated between the products are specified, and the repeating pattern of the image is specified. ， Specify the place by specifying the step on the upper side of the package, specifying the dividing position based on the restrictions such as the same product width. Place identification information for identifying the place is attached to the specified place. Then, the coordinates of each specified place are stored in association with the shooting date / time, store identification information, image information identification information of the shot image information, image information identification information of the orthodox image information, and place identification information for identifying the place. Let me. The coordinates of the place may be two points that can specify the rectangular area without storing four points.

When each place of each shelf in the shelf position area image information of the normal image information is specified as described above, the product identification information identification processing unit 243 and the sample information stored in the sample information storage unit 26 for each place Execute the matching process and identify the product identification information of the product displayed in that place. That is, each image feature amount is calculated from the image information in the rectangular area of a place on a shelf (where the place identification information in this place area is X) and the sample information stored in the sample information storage unit 26. Then, the similarity is judged by finding the pair of feature points. Then, the sample information having the highest similarity is specified, and if the similarity at that time is equal to or higher than a predetermined threshold value, the product identification information corresponding to the sample information is specified based on the sample information storage unit 26. Then, the specified product identification information is used as the product identification information of the product displayed in the place of the place identification information X. For places that are judged not to be similar to any sample information, information indicating that the place identification information is "empty" is added. The product identification information identification processing unit 243 uses the identified product identification information or information indicating that it is "empty" to identify the shooting date and time, the store identification information, the image information identification information of the shot image information, and the image information identification of the normal image information. The information and the place identification information are associated with each other and stored in the product identification information storage unit 27.

Not all place product identification information can be specified. Therefore, for places that have not been identified, instead of adding "empty" information from the beginning as described above, the input of product identification information is accepted, and the product identification information for which the input is accepted is used for shooting date and time and store identification. The information, the image information identification information of the captured image information, the image information identification information of the normal image information, and the place identification information are stored in the product identification information storage unit 27 in association with each other. Similarly, input may be accepted for the correction process of the specified product identification information. In addition, if the input of "empty" is accepted as a result of the correction process, the information of "empty" may be added.

By performing the above processing, it is possible to identify the product identification information of the product displayed on the shelf of the display shelf shown in the photographed image information.

Next, the process of recognizing the product tag information in the product tag recognition processing unit 25 will be described.

The product tag placement area processing unit 251 in the product tag recognition processing unit 25 cuts out the image information of the product tag placement area specified in S120, and the product tag placement area normalization processing unit 252 executes the trapezoidal correction processing to perform the product. Executes the normalization process for the image information in the tag placement area.

As the normalization process for the image information of the product tag placement area, first, edge detection is performed in the image information of the product tag placement area. Then, among the detected edges, a contour line having a certain length or more and within an angle within a predetermined range from the vertical is specified. Then, among the identified contour lines, the contour lines on the left and right are specified. An example of the contour line thus specified is shown in FIG. Then, the trapezoidal correction process is executed for the image information of the product tag arrangement area so that the specified contour lines (L1, L2 in FIG. 15A and L3, L4 in FIG. 15B) become vertical lines, respectively. To do. By executing such a process, the image information of the product tag arrangement area is made normal, and the image information of the product tag arrangement area shown in FIG. 14 can be obtained.

When the process of normalizing the image information of the product tag placement area is completed, the product tag identification processing unit 253 uses the first method or the second method to select individual product tag areas from the image information of the product tag placement area. Identify.

In the case of the first method, among the image information of the product tag placement area that has been placed upright, a histogram that integrates the brightness information of the image information in the horizontal direction is generated, and the rising and falling positions of the histogram are specified. Then, the rising position of the specified histogram is specified as the upper side position A of the product tag, and the falling position of the histogram is specified as the lower side position B of the product tag.

Next, of the image information of the product tag placement area that has been placed upright, a histogram that cuts out between the upper side position A and the lower side position B and integrates the brightness information of the image information of the product tag placement area that has been placed upright in the vertical direction. To generate.

In the generated histogram, the rising position U and the falling position D are specified, the rising position U (left side position) and the falling position D (right side position) within a predetermined distance range to the right of the rising position U are specified as a pair, and the product tag is specified. Specify as an area.

If there is no product tag area within a predetermined distance, specify it as a product tag area on the right side of the rising position U where the pair could not be formed, and on the left side of the falling position D where the pair could not be formed. To do.

FIG. 18 shows a state in which each product tag area is specified by the above processing.

When the second method is used, the product tag identification processing unit 253 performs image matching processing between the image information of the product tag of the template registered in advance and the image information of the product tag arrangement area that has been placed upright. By executing, the product tag area is specified.

When the product tag specifying processing unit 253 specifies the product tag area as described above, the information specifying processing unit 254 in the product tag specifies the information in the product tag.

The first method in the product tag information specifying processing unit 254 is to binarize the specified product tag area to obtain binarized image information. Then, the box is specified by the labeling process on the binarized image information. Islandes that do not satisfy the height and width that are the predetermined thresholds are removed from the processing target as they are without generating a box as noise.

Then, by finding the height, width, and baseline of the generated box, and identifying adjacent boxes whose baseline and height match within a predetermined range and whose height and width are within a certain threshold range, Merge the specified boxes to form a block. Then, among the blocks, the block with the highest height is estimated as the price range (price range excluding tax), and OCR recognition is performed. If the price information cannot be obtained as a result of OCR recognition (if the numerical character string cannot be recognized), the block with the next highest height is estimated as the price range (tax-excluded price range), and OCR recognition is performed in the same manner. ..

In the above-mentioned OCR recognition, the reading accuracy of the price can be improved by adding the character recognition constituting the price display such as "number" and "comma" used for the price as a constraint condition.

When the price information is acquired by OCR recognition from the blocks estimated as the price area (tax-excluded price area) in this way, the blocks other than the acquired price information estimated as the price area (tax-excluded price area) are identified and specified. OCR recognition is performed for each block. For this OCR recognition, it is preferable to perform two types of OCR recognition processing. That is, it is an ordinary standard OCR recognition process and an OCR recognition process in which character recognition constituting a price display is added as a constraint condition.

The blocks specified above include blocks in the price area (price area including tax) and blocks in the area of information such as product identification information such as manufacturer name and product name, and rating. Then, two types of OCR recognition processing are executed for each block. For blocks in the area of product identification information such as manufacturer name and product name, and information area such as rating, character strings are recognized by standard OCR recognition processing, and most of the characters include errors in OCR recognition processing with constraints. Become a column. In this case, it is determined whether the recognition results of the two recognition processes differ by a predetermined value or more, and if they differ, the character string executed by the standard OCR recognition process is used as the recognition result, and the price area (tax-included price area). It is determined that the block is other than the block of. On the other hand, for blocks in the price area (price area including tax), the standard OCR recognition process recognizes the price information character string, and the constrained OCR recognition process also recognizes the price information character string. In this case, it is determined whether the recognition results of the two recognition processes differ by a predetermined value or more, and if they do not differ, it is determined that the block is in the price area (tax-included price area), and the recognition process with constraints added. The character string is used as the price information of the recognition result.

As described above, the information identification processing unit 254 in the product tag can recognize the information described in the product tag as characters. In order to judge the display status of a product, it is sufficient to have the product identification information of the product, but by recognizing other information such as the tax-excluded price and the tax-included price, the price information can be associated with the display status. It becomes possible to manage.

When the second method is used by the product tag specifying processing unit 253, the product tag used as a template has the product identification information such as the tax-excluded price, the tax-included price, the manufacturer name, and the product name, and the rating, respectively. The position and size of the displayed character frame are set. Therefore, the image information of the corresponding portion may be cut out from the product tag area specified by the product tag specifying processing unit 253, and the OCR recognition process may be executed.

Then, the information identification processing unit 254 in the product tag executes a dictionary collation process with the specified product name or the like. That is, the edit distance between the read character string and the product identification information such as each product name in the product dictionary is obtained, the product identification information such as the product name with the minimum edit distance is specified, and the edit distance is within a predetermined distance. If so, it is identified as product identification information such as a product name. Then, the product name part with the shortest edit distance is removed from the read character string, the edit distance with each rating in the rating dictionary is obtained for the remaining substring, and whether the minimum edit distance is within the predetermined distance is determined. Judgment is made, and if it is within a predetermined distance, that part is identified as a rated character string. Similarly, the product name part and the rated part with the minimum edit distance are removed from the read character string, the edit distance with each manufacturer name in the manufacturer name dictionary is obtained for the remaining substring, and the minimum edit distance is determined. It is determined whether it is within the distance, and if it is within the predetermined distance, that part is identified as a character string of the manufacturer name. If the Jaro-Winkler distance is used instead of the edit distance, the corresponding processing may be executed as appropriate.

Further, the product tag information identification processing unit 254 executes a logical check process for the character-recognized character string. That is, it is judged whether or not the character recognition character strings are logically inconsistent.

If there is no contradiction as a result of the logical check, identify the product identification information such as tax-excluded price, tax-included price, product name, manufacturer, and rating for each recognized character string, and identify them as the shooting date and time and store identification. Information and captured image information is stored and output in a predetermined storage area in association with the image information identification information. For example, output in tabular format.

As described above, after the product tag recognition processing unit 25 recognizes the product identification information of the products displayed on the shelf by the display product recognition processing unit 24, the display status determination processing unit 25 recognizes the product identification information of the product tag. As shown in FIG. 24, 28 is out of product identification information that is not included in the product identification information recognized by the display product recognition processing unit 24 among the product identification information recognized by the product tag recognition processing unit 25. It is determined as the product identification information of the product (S150).

That is, when the display status determination processing unit 28 includes the same product identification information in the adjacent places among the product identification information (product identification information of the face) for each place recognized by the display product recognition processing unit 24. The product identification information is put together as one product group. Further, the display status determination processing unit 28 collects the same product identification information into one product identification information among the product identification information recognized by the product tag recognition processing unit 25.

Then, by comparing the product identification information collected from each other and determining whether or not there is matching product identification information, whether the product with the recognized product identification information is displayed, the product tag is absent, or the product is out of stock. To judge.

That is, when determining a shortage, among the product identification information recognized by the product tag recognition processing unit 25, the product identification information that is not included in the product identification information recognized by the display product recognition processing unit 24 is out of stock. It is determined as the product identification information of the product, and the information indicating that the product is out of stock is stored in the product identification information storage unit 27 in association with the identification information of the place.

In order to determine the shortage, it is sufficient to execute only this determination process, but as a product manufacturer and sales company, what kind of products of the company or other companies are exposed on the display shelves of the store and to what extent. By grasping whether it is displayed, it can be linked to the development strategy and sales strategy of the company's products. Therefore, by also determining the products on display, it is possible to meet the demands of such manufacturing companies and sales companies. Therefore, the display status determination processing unit 28 may determine the shortage of products and also recognize the products being displayed. That is, the display status determination processing unit 28 may determine that among the product identification information recognized by the product tag recognition processing unit 25, the product identification information recognized by the display product recognition processing unit 24 is being displayed. .. When it is recognized that the product is on display, the product identification information and the information indicating that the face is displayed are stored in the product identification information in association with the identification information of the place.

In addition, the absence of the product tag makes it impossible for the customer to know the price of the product, which also contributes to the customer refraining from purchasing the product. Therefore, it is important for product manufacturers and sales companies to know the absence of product tags. Therefore, the display status determination processing unit 28 may recognize the absence of the product tag in addition to the determination of the missing item. That is, among the product identification information recognized by the display product recognition processing unit 24, the product identification information that is not included in the product identification information recognized by the product tag recognition processing unit 25 is not present in the product tag. Is determined to be. When it is recognized that the product tag does not exist, the product identification information and the information indicating that the product tag does not exist are stored in the product identification information in association with the identification information of the place.

By executing the above processing, it is possible to recognize products that are out of stock, in some cases, products that are on display, and products that do not have product tags in the display status of the display shelves. In addition, since the information is stored for each place on the display shelf, it is possible to grasp information such as a product shortage for each place position.

In the first embodiment, the case where the processing is performed using the image information and the sample information of the place area as the recognition of the “sky” has been described, but if only the image information matching processing is performed, it depends on the shooting conditions and the like. Cheap. Therefore, in addition to matching the image information in the place area with the sample information, it is possible to further improve the accuracy by performing processing for recognizing the "sky".

In this case, the display product recognition processing unit 24 executes the following processing for recognizing the “empty”.

First, as described above, the product identification information identification processing unit 243 in the display product recognition processing unit 24 has the image information of the area composed of the coordinates of the place to be processed and the sample information stored in the sample information storage unit 26. The similarity with the product is determined, the product identification information corresponding to the sample information having the highest similarity is specified, and the place whose similarity is not equal to or higher than the predetermined threshold is specified.

Therefore, the display product recognition processing unit 24 determines whether any of the following processes A and B is applicable as the process of the first method for recognizing "empty".

That is, the process A determines that the image information of the specified place area is “empty” when the average value of the brightness of the area is darker than the predetermined reference value. Process B determines that the image information in the place area is "empty" when the image information of the rear plate of the shelf is matched with the image information of the rear plate and the similarity is the highest. In this case, the image information of the rear plate of the shelf may be stored as sample information. It should be noted that a determination process other than process A and process B can also be used.

Regarding process A, the color of the sample information of the product is close to black as a whole, and there is usually no product whose product name is not specified, so there is no problem in recognizing it as "empty". Regarding process B, the rear plate of the shelf does not usually resemble the sample information of the product, so if it is judged to have high similarity to the rear plate of the shelf, it can be recognized as "empty". it can.

In addition, the display product recognition processing unit 24 recognizes a gap in the display shelf (a space where the product is originally placed, but a gap created by a shortage) as a process of the second method for recognizing "empty". You may do it. The process of recognizing the voids in the display shelves can be determined by the following. In this case, when the display shelf is photographed and the photographed image information is acquired, the distance information (distance information) between the display shelf to be photographed and the photographing device by the photographing device (including a portable communication terminal such as a smartphone). Depth information) is acquired as two-dimensional information. In this case, when photographing the display shelf in the photographing device, the light beam of a specific wavelength, for example, infrared rays is irradiated, and the amount of reflected light from each direction and the time until the reflected light arrives are measured. In addition to acquiring distance information (depth information) to the target, or irradiating a ray of a specific wavelength, for example, an infrared dot pattern, and measuring the distance information (depth information) from the reflection pattern to the shooting target. , There is a method of using the parallax of a stereo camera, but it is not limited to these. The correspondence between the captured image and the depth information is schematically shown in FIGS. 27 (a) and 27 (b). The positions of the captured image information and the depth information correspond to each other. If the depth information is given by the distance from the photographing device (one point on three dimensions), the depth information is converted into the depth information from the photographing surface (plane) of the photographing device. .. Further, FIG. 28 schematically shows an example of depth information.

Then, the captured image information is set as the captured image information in the captured image information normalization processing unit 22. Then, similarly, the same conversion is executed for the depth information. That is, the same conversion process as the keystone correction process for the captured image information is executed for the depth information. This is schematically shown in FIGS. 29 (a) and 29 (b). FIG. 29 (a) is the emplacement image information obtained by performing the emplacement processing on the captured image information of FIG. 27 (a). FIG. 29 (b) is a diagram showing the depth information in a state in which the depth information of FIG. 27 (b) is subjected to the same processing as the emplacement processing of FIG. 29 (a) for each depth information. Is. FIG. 30 schematically shows an example of the depth information in the state where the orthostatic processing is executed with respect to the depth information of FIG. 28.

By executing the same processing as the normal image information for the depth information in this way, the depth information can be specified in the same positional relationship as the normal image information.

Next, with respect to the place area specified by the display product recognition processing unit 24, either the minimum value, the average value, or the maximum value of the depth information in the place area is obtained (this is called the place distance). Then, a plane determined by a large amount of depth information of each place is obtained, and the depth of each place is obtained from the plane. FIG. 31 schematically shows the depth of each place. Then, if the depth is larger than the predetermined threshold value, the place is determined as "empty".

If the regression plane of all the place positions in the three-dimensional space is obtained, the depth that is the most deviated from the regression plane is excluded, and the regression plane is obtained again, and the plane fits a certain value or more. , If it is a surface composed of the places (faces) of the products on display and the distance from the same plane to each place position is set as the "place depth" again, FIG. 31 can be obtained. FIG. 32 schematically shows the process of obtaining the regression plane of the place position. FIG. 32 (a) is a front view, and FIG. 32 (b) is a view from above.

The display product recognition processing unit 24 can more accurately identify that the place is "empty" by executing the above-mentioned processing of the first method or the processing of the second method. The processing of the first method and the processing of the second method may be performed in addition to the matching processing of the image information of the place area and the sample information in the product identification information identification processing unit 243, or the image information of the place area. And, it may be performed instead of the matching process with the sample information.

Furthermore, if the position of the place area has already been specified, for example, when processing is executed for the same display shelf from the second time onward, the processing is performed instead of the matching processing between the image information of the place area and the sample information. When executed, it is possible to quickly and accurately identify whether or not the place is "empty".

The reading device reads the product identification information from the IC tag attached to the product, and the display product recognition processing unit 24 reads the product identification information based on the "empty" recognition process in the first embodiment and the present embodiment. It may be applied when specifying. That is, when reading the product identification information from the IC tag attached to the product, if the product does not exist, the product identification information cannot be read because there is no IC tag. Therefore, it may be possible to recognize only the place of the product displayed in the horizontal direction. On the other hand, originally, the products should be displayed in the "empty". Therefore, this implementation is carried out for each area in which the product identification information read by the reader corresponds to the horizontal and vertical sizes of the corresponding products (for example, one product or the average value of the read products). Places can be automatically identified by dividing the "empty" area recognized in the example. By doing so, even when the product identification information is read from the IC tag attached to the product, the "empty" place, which is the area where the product should exist, can be automatically specified.

In Example 1 and Example 2, in determining the product shortage, the matching process of the image information of the place area and the sample information was used, but the same display shelf was used for the second and subsequent times. When executing the processing, the display status determination processing unit 28 can execute the determination processing using the previous processing result.

In this case, the display status determination processing unit 28 makes a determination based on the change in the place state between the previous time (N-1 times) and the current time (N times) and the change in the product tag. An example of this determination process is shown in FIG. 33.

That is, the display status determination processing unit 28 includes the product identification information in the current (N times) place recognized by the display product recognition processing unit 24, the product identification information in the same place last time (N-1 times), and the product. Using the product tag information of this time (N times) recognized by the tag recognition processing unit 25 and the product tag information of the previous time (N-1 times), the determination table showing the previous state change this time is shown in FIG. 33. Based on the above, it is determined whether or not the place is out of stock. By making such a judgment, the judgment can be made more quickly.

Further, a modification of the specific processing in the place specifying processing unit 242 of Examples 1 to 3 will be described. In this embodiment, the place identification process of Examples 1 to 3 may be performed as the first process, and the place may be automatically specified as the second and subsequent place identification processes. The processing in this case will be described.

In addition to the first time, the first time includes the case where the processing of Examples 1 to 3 is performed at an arbitrary timing in order to correct the deviation at the time of automatic identification. The second and subsequent times are other than the first time. In addition, from the second time onward, it is also included that the photographed image information and the normal image information taken on the same display shelf at the same opportunity are processed for the photographed image information and the normal image information taken at different angles. ..

The place specifying processing unit 242 executes the same processing as the processing of the first embodiment as the first place specifying processing. Then, in the second and subsequent place identification processes in the place identification processing unit 242, the coordinates of the place area specified by the previous (N-1th) normal image information are used for the same shelf on the same display shelf. Extract and use the coordinates as the coordinates of the area of the place specified by the normal image information of this time (Nth time).

The coordinates of the place area are relative coordinates based on a predetermined position in the display shelf in the normal image information, similar to the coordinates of the position of the shelf.

Further, as a modification of the first embodiment, as a process of specifying the product identification information of the product displayed in the place for each shelf of the display shelf in the product identification information identification processing unit 243, the process of the first embodiment is performed for the first time. The following processing is executed as the identification processing of the product identification information of the above, and as the identification processing of the product identification information from the second time onward.

The product identification information identification processing unit 243 first identifies the place identification information of the place to be processed as the identification processing of the product identification information of the place in the Nth orthodox image information. Let X be the specified place identification information. Then, the image information of the area of the place identification information X in the Nth normal image information is compared with the image information of the area in the N-1th normal image information at the position corresponding to the place identification information X. Regarding the determination of similarity, in addition to the processing of Examples 1 to 4 described above, the EMD of the image information signatures (set of image features and weights) of the area of each place is obtained, and the similarity is determined. can do. As the image feature amount of the signature, for example, the frequency distribution of the image information in the place area in the HSV color space is obtained, the hue and saturation are grouped, and the number of features and the image feature amount by the area in the HSV color space are obtained. can do. The reason for grouping the hue and saturation is to reduce the dependence on brightness so that it is not greatly affected by the shooting conditions.

Further, in order to speed up the processing, it is possible to use the similarity such as the L2 distance between the image features such as the color colorogram and the color histogram of the image information in an appropriate color space instead of the signature and the EMD.

If the similarity of the image information in the place area is equal to or higher than a certain threshold value, the product identification information corresponding to the place in the area in the N-1th orthodox image information is extracted from the product identification information storage unit 27, and the Nth time. Let it be the product identification information of the place identification information X in the normal image information of. As a result, the product identification information of the place identification information X in the Nth normal image information to be processed can be specified. If the similarity is less than a certain threshold, the image information in the area of the place identification information X in the Nth orthodox image information and the sample information stored in the sample information storage unit 26 are stored in the sample information storage unit 26 as in the case of the first time. By comparison, the product identification information having the similarity equal to or higher than a predetermined threshold value and having the highest similarity is specified as the product identification information of the place of the place identification information X in the Nth orthodox image information.

In the comparison with the image information of the place area in the N-1th orthodox image information, not only the comparison with the position of the corresponding place but also the place in a predetermined range may be included as the comparison target. For example, when comparing with the image information in the area of the place identification information X in the Nth normal image information, the comparison target is from the area of the place identification information X in the N-1th normal image information and from that area. The area of the place within a predetermined range, for example, the area of the place located one or more apart in the left-right direction thereof, and the area of the place located on the upper and lower shelves may be included. Further, in addition to the area of the place identification information X in the N-1th orthodox image information, the area of a plurality of adjacent places such as the place identification information X-2, X-1, X, X + 1, X + 2 is included. You may.

In this case, the image information of the area of the place identification information X in the Nth normal image information and the image information of each of the areas of the place range to be compared in the N-1th normal image information. To identify the place identification information of the N-1th orthodox image information having the highest similarity. It should be noted that the similarity may be conditional on being above a certain threshold. Then, the product identification information corresponding to the place identification information of the specified N-1th normal image information is extracted from the product identification information storage unit 27, and the product identification information of the place identification information X in the Nth normal image information is obtained. To do. FIG. 34 schematically shows this process. FIG. 34 (a) is the previous (N-1st) normal image information, and FIG. 34 (b) is the current (Nth) normal image information. Then, the image information of the area of each place on the shelf 1 of the Nth time normal image information is compared with the image information of the area of each place of the shelf 1 of the N-1th normal image information. The similarity is determined by, and the product identification information of the place on the shelf 1 of the N-1th normal image information, which has the highest similarity, is used as the product identification information of the place on the shelf 1 of the Nth normal image information. Indicates that it is specified as. FIG. 34 shows a case where, in addition to the N-1th place of the normal image information at the position corresponding to the Nth place of the normal image information, a comparison is performed with two places on the left and right of the place. .. Not only the same shelf, but also the image information of the place positions of the upper and lower shelves may be compared. For example, in the case of FIG. 34, when specifying the product identification information of the place position at the center of the shelf 2 of the Nth normal image information, the center of the shelf 2 of the N-1th normal image information and its left and right sides thereof. Not only comparing with the image information of the area of two places, but also the image information of the area of two places on the left and right of the center of the shelf 1 of the N-1th orthodox image information, N-1st. You may compare the image information and the similarity with the image information of the center of the shelf 3 and the area of each of the two places on the left and right of the vertical image information of the above.

As a result of comparing the N-1th orthodox image information with the image information of the place, if the candidate for the product identification information cannot be identified because the similarity does not satisfy the threshold, the Nth orthitudinal image information The image information in the area of the place identification information X in the above is compared with the sample information stored in the sample information storage unit 26, and the product identification information having a similarity equal to or higher than a predetermined threshold and having the highest similarity is obtained as N. It is specified as the product identification information of the place of the place identification information X in the second orthodox image information. The similarity determination process in this case can be performed in the same manner as the initial product identification information identification process.

The product identification information candidates identified as described above correspond to the shooting date / time information, store information, image information identification information of the shot image information, image identification information of the orthodox image information, and place identification information for identifying the place. It is also stored in the product identification information storage unit 27.

In the first to fifth embodiments, the product identification information in the place is specified in each of the image information obtained by shooting the same display shelf at different angles at the same opportunity, and the corresponding places are compared with each other. The product identification information in is confirmed. Therefore, instead of specifying the product identification information in the place for each of the image information taken at different angles, the image information taken at different angles is first combined to eliminate the reflected part such as light. It may be configured to execute the identification process of the product identification information in the place for one piece of image information.

The processes of Examples 1 to 6 described above can be combined as appropriate. Further, each process is not limited to the order described in the specification of the present invention, and can be appropriately changed as long as the object is achieved. Further, each process in the display product recognition processing unit 24 and the product tag recognition processing unit 25 was executed for the normal image information in which the normalization process was executed for the photographed image information, but for the photographed image information. You may do it. In that case, the normal image information may be read as the captured image information.

Further, in the processing of Examples 1 to 6 described above, the display product recognition processing unit 24 identifies the product identification information from the photographed image information or the emplacement image information, and the product tag recognition processing unit 25 identifies the product. By identifying the identification information of the product and collating it, it was determined that the product was out of stock. In addition to recognizing the character information described in the product tag, the product tag recognition processing unit 25 may read the information encoding the product identification information provided in or near the product tag, or the product tag or The product identification information may be read from an IC tag or the like provided in the vicinity thereof. That is, the product tag recognition processing unit 25 is not limited to the character recognition processing described in the product tag, and uses the product tag or the product identification information in the vicinity of the product tag to display the product tag recognition processing unit 24. It suffices to determine whether the product is out of stock by collating it with the identification information of the product.

Further, the display status determination processing unit 28 may determine the shortage by collating the product identification information specified by the display product recognition processing unit 24 with the inventory information of the product managed by the POS system. .. That is, the inventory quantity of the product managed by the POS system is extracted, and the inventory quantity for each product is collated with the product identification information specified by the display product recognition processing unit 24. Then, it is determined that there is a possibility that the product identification information of the product that cannot be specified by the display product recognition processing unit 24 is out of stock even though the stock quantity exists in the POS system (the stock quantity is 1 or more). You may.

Further, the display product recognition processing unit 24 identifies the products displayed on the display shelf based on the photographed image information, attaches an IC tag to the displayed products, and stores the products in the IC tag. By reading the product identification information, the product identification information of the product displayed on the display shelf may be specified. When the display product recognition processing unit 24 reads the product identification information from the IC tag attached to the product, and the product tag recognition processing unit 25 reads the product identification information from the product tag or the IC tag installed in the vicinity thereof, respectively. Reads the product identification information from the IC tag, so that the IC tag can be identified by the product identification information of the IC tag attached to the product and the product identification information of the IC tag attached to the product tag by a flag or the like. It is preferable that

Further, if it is only necessary to identify whether the product is on display or out of stock, the product on display in the display product recognition processing unit 24 does not perform the processing for each place. By performing the product identification information specifying process and the product tag recognition processing unit 25 specifying the product identification information of the product tag and collating them, it is possible to identify whether or not the product is out of stock during display. ..

By using the display status determination system 1 of the present invention, it is possible to accurately determine the display status of the products displayed on the display shelf.

1: Display status determination system 2: Management terminal 3: Photographed image information input terminal 20: Photographed image information input reception processing unit 21: Photographed image information storage unit 22: Photographed image information emplacement processing unit 23: Position identification processing unit 24 : Display product recognition processing unit 25: Product tag recognition processing unit 26: Specimen information storage unit 27: Product identification information storage unit 28: Display status determination processing unit 70: Computing device 71: Storage device 72: Display device 73: Input device 74 : Communication device 241: Shelf stage area processing unit 242: Place identification processing unit 243: Product identification information identification processing unit 244: Shelf stage image matching processing unit 251: Product tag placement area processing unit 252: Product tag placement area normalization processing Department 253: Product tag identification processing unit 254: Information identification processing unit in the product tag

Claims (12)

It is a display status judgment system that judges the display status of products.
The display status determination system is
The display product recognition processing unit that recognizes the product identification information of the products displayed on the display shelf,
A product tag recognition processing unit that recognizes the product identification information of the product tag attached to the display shelf, and
A display status determination processing unit that determines the display status of the product by comparing the product identification information of the displayed product with the product identification information of the product tag.
A display status determination system characterized by having. The display product recognition processing unit
By determining the similarity between the image information showing the display shelf and the image information obtained by photographing a product that may be displayed on the display shelf and / or the sample information including the image feature amount thereof, the above-mentioned Recognize the product identification information of the products displayed on the display shelf,
The display status determination system according to claim 1. The display product recognition processing unit
Correspondence is the sample information having the highest similarity between the image information showing the display shelf and the image information obtained by photographing a product that may be displayed on the display shelf and / or the sample information including the image feature amount thereof. If the similarity of the product identification information is not equal to or higher than a predetermined threshold, the product identification information is not recognized.
The display status determination system according to claim 1 or 2, wherein the display status determination system is characterized in that. The display product recognition processing unit
Recognizes the product identification information of the product displayed on the display shelf based on the product identification information stored in the electronic tag attached to the product displayed on the display shelf, which is read by a predetermined reading device. ，
The display status determination system according to any one of claims 1 to 3, wherein the display status determination system is characterized in that. The product tag recognition processing unit
A product tag identification processing unit that specifies a product tag area in the image information showing the display shelf, and a product tag identification processing unit.
An information identification processing unit in a product tag that recognizes product identification information by performing OCR recognition processing in a part or all of the specified product tag area, and
The display status determination system according to any one of claims 1 to 4, wherein the display status determination system is characterized by having. The product tag identification processing unit
The product tag area is specified using the brightness information of the image information.
The display status determination system according to claim 5. The product tag identification processing unit
By integrating the brightness information of the image information in the horizontal direction, the upper side position and the lower side position of the product tag are specified.
By integrating the brightness information of the image information in the vertical direction, the left side position and the right side position of the product tag are specified.
The display status determination system according to claim 5 or 6, wherein the display status determination system is characterized in that. The product tag identification processing unit
By specifying the rising and falling edges of the histogram that integrates the brightness information of the image information in the horizontal direction, the upper and lower side positions of the product tag are specified.
By specifying the rise and fall of the histogram, which is the sum of the brightness information of the image information in the vertical direction, the left side position and the right side position of the product tag are specified by pairing the fall within a predetermined distance from the rise.
The display status determination system according to any one of claims 5 to 7, wherein the display status determination system is characterized. The information identification processing unit in the product tag
The box is specified by binarizing the specified product tag area and performing labeling processing.
Of the specified boxes, the boxes that satisfy the specified conditions are specified as blocks, and the boxes are specified as blocks.
The OCR recognition process is executed for the specified block.
The display status determination system according to any one of claims 5 to 8, wherein the display status determination system is characterized in that. The product tag recognition processing unit
Edge detection is performed in the product tag arrangement area, which is an area where the product tag may be attached in the image information, a straight line group satisfying a predetermined condition of a vertical angle is detected, and among the detected straight line groups. It has a product tag placement area normalization processing unit that identifies at least two straight lines and executes trapezoidal correction processing using the two specified straight lines.
The display status determination system according to any one of claims 5 to 9, wherein the display status determination system is characterized in that. The product tag recognition processing unit
In the product tag, based on the product identification information stored in the product tag attached to the display shelf or the electronic tag attached in the vicinity thereof, or the coded product identification information read by a predetermined reader. Recognize product identification information,
The display status determination system according to any one of claims 1 to 10. Computer,
Display product recognition processing unit that recognizes product identification information of products displayed on display shelves,
Product tag recognition processing unit that recognizes the product identification information of the product tag attached to the display shelf,
A display status determination processing unit that determines the display status of the product by comparing the product identification information of the displayed product with the product identification information of the product tag.
A display status determination program characterized by functioning as.

Images