JP6664675B2 - Image recognition system - Google Patents

Description

  The present invention relates to an image recognition system for accurately specifying identification information of an object such as a product shown in image information.

  In the retail industry, it is known that the display status of a product affects the sale of the product. Therefore, as a product manufacturing company or a sales company, knowing what kind of product of the company or another company is displayed in the store can be linked to the development strategy and sales strategy of the product.

  On the other hand, in order to realize this, it is important to accurately identify products displayed at stores. Therefore, it is conceivable that a display shelf of a store is photographed, and a displayed product is manually specified by a human based on the image information. In this case, the product can be specified almost accurately. However, in order to keep track of the display status of a product, it is necessary to know the display status at regular intervals. However, every time a human identifies a product from image information obtained by photographing a store display shelf. Is burdensome and inefficient.

  Therefore, it is required to automatically identify the products displayed there from image information obtained by photographing the display shelves of the store, and to grasp the display status of the products. For example, there is a method of using an image recognition technique for an image of a display shelf in a store based on a sample image of each product. As these prior arts, for example, there is a system that manages the display status of commodities by using the technologies described in Patent Documents 1 to 3 below.

  In some cases other than the retail industry, there is a case where it is required to take an image of an object and specify the object in the image information of the imaged object.

JP-A-5-342230 JP-A-5-334409 International Publication 2012/029548

  The invention of Patent Literature 1 is a system for assisting a person who does not have knowledge of which display shelf to display a product on. Therefore, it is not possible to grasp the products actually displayed. Patent Document 2 discloses a shelf layout support system that supports display of products, which is a system that supports input of product images. However, the system of Patent Document 2 only supports the input of a product image when using the shelf allocation support system, and even if this system is used, it is not possible to grasp the products actually displayed. .

  Patent Literature 3 is an invention in which, when there is an empty space in a display shelf, a product to be displayed in the empty space is specified, or when a product to be displayed on the display shelf is misplaced, the notification is made. In the present invention as well, the product displayed on the display shelf is specified by performing image matching processing between the product and the product displayed on the display shelf, but the recognition accuracy is actually low.

  When image recognition technology is used for an image of a display shelf based on a sample image of a product, the image feature between the sample image and the partial image of the product display portion in the image of the display shelf is displayed. Recognize a product using similarity. On the other hand, in the image of the display shelf or the specimen image, if the surface of the product package reflects light, unnecessary objects such as lighting and outside light in the store, the photographer and the photographing device may be located at the position of the photographing device. Depending on the relationship, it may be reflected on the surface of the product. Such a factor is a factor that affects the accuracy of image recognition because the original image feature amount cannot be obtained.

  The same problem applies not only to the identification of the products displayed on the display shelf, but also to the case where the target is specified from image information obtained by photographing a general target. And even if each of the above-mentioned patent documents is used, it is not possible to solve the problem due to reflection of light or the like.

  In view of the above problems, the present inventor has invented an image recognition system capable of specifying identification information of an object such as a product with higher accuracy than before even when light or the like is reflected in an image.

According to the first aspect of the present invention, a part of a plurality of pieces of image information photographed by changing the relative position between a product and a light source without changing the relative position between the product and a photographing device is cut out into one piece of image information. A sample information storage unit that stores the sample information of the product generated by combining, a part or all of the image information showing the display shelf displaying the product, and a sample of the product that is stored in the sample information storage unit An identification information identification processing unit that identifies the product identification information of the product in the image information in which the display shelf is captured by comparing the information with the information.

  As the sample information of the product used in the comparison processing, the use of information without reflection of light or the like leads to an improvement in accuracy. Therefore, by using the sample information of the product based on a plurality of pieces of image information obtained by changing the relative position of the light source without changing the relative position of the product and the photographing device, it is possible to improve accuracy in the comparison process. be able to.

According to the second aspect of the present invention, illumination is applied from light sources located on the left and right sides of the photographing device without changing the relative position between the product and the photographing device whose horizontal cross section is convex or substantially convex. accepting an input of the captured image information Te, among the image information the input of which is accepted, the image information sources on the right across the imaging device is taken to position the left area from the center line of the image information For image information obtained by cutting out and photographing with the light source positioned on the left side of the photographing device, the right side region is cut out from the center line of the image information, and the image information of the cut out left side region and right side region is combined into one. This is an image recognition system that generates sample information of the product by combining with image information.

  As the sample information of the product used in the comparison processing, the use of information without reflection of light or the like leads to an improvement in accuracy. Therefore, it is preferable that the sample information of the product is generated as in the present invention.

The invention described in claim 1 can be realized by causing a computer to read and execute the program of the present invention. That is, the above-described method is performed by changing a relative position between a product and a light source without changing a relative position between a product and a photographing device, cutting out a part of a plurality of pieces of image information captured and combining the pieces of image information into one piece of image information. A computer that uses a sample information storage unit that stores sample information of a product is used to convert a part or all of the image information showing the display shelf displaying the product and the sample information of the product that is stored in the sample information storage unit. By performing the comparison, an image recognition program can be configured to function as an identification information specifying processing unit that specifies product identification information of a product in the image information in which the display shelf is captured.

The invention described in claim 2 can be realized by causing a computer to read and execute the program of the present invention. That is, an image obtained by irradiating a computer with illumination from light sources located on the left and right sides of the photographing device without changing the relative positions of the product and the photographing device whose horizontal cross section is convex or substantially convex. A function of accepting input of information, of the image information having received the input, with respect to image information photographed with the light source positioned on the right side of the photographing device, a left region is cut out from a center line of the image information, A function of cutting out the right side area from the center line of the image information with respect to the image information shot with the light source positioned on the left side of the photographing device, and combining the cut out left side area and right side area image information into one image information Thus, the function of generating the sample information of the product can be realized as an image recognition program.

  By using the image recognition system of the present invention, even when light or the like is reflected in image information, an object such as a product can be specified with higher accuracy than before.

1 is a block diagram schematically illustrating an example of a configuration of an image recognition system according to the present invention. FIG. 2 is a block diagram schematically illustrating an example of a configuration of an image information processing unit in the image recognition system of the present invention. FIG. 3 is a block diagram schematically illustrating an example of a hardware configuration of a computer used in the image recognition system of the present invention. 5 is a flowchart illustrating an example of an entire processing process in the image recognition system of the present invention. 5 is a flowchart illustrating an example of a processing process in an image information processing unit according to the present invention. 5 is a flowchart illustrating an example of a processing process in an identification information comparison processing unit according to the present invention. It is a figure which shows an example of a sample information storage part typically. It is a figure which shows an example of the photographic image information which image | photographed the display shelf of a store from the different angle on the same opportunity. It is a figure which shows another example of the captured image information which image | photographed about the display shelf of a store from the different angle at the same opportunity. FIG. 9 is a diagram illustrating an example of normal image information obtained by performing a trapezoidal correction process on the captured image information of FIG. 8. FIG. 10 is a diagram illustrating an example of normal image information obtained by performing a trapezoidal correction process on the captured image information of FIG. 9. FIG. 11 is a diagram illustrating a state where a shelf area is specified in the captured image information in FIG. 10. FIG. 12 is a diagram illustrating a state where a shelf area is specified in the captured image information of FIG. 11. It is an image figure of an example of the typical processing in the image recognition system of the present invention. FIG. 9 is a diagram illustrating an example of a state in which a feature amount collection area is set for the (N-1) th captured image information; FIG. 14 is a diagram illustrating an example of a state in which a feature amount collection area is set for Nth captured image information. It is a figure which shows the relationship of the pair of the feature-value collection area | region of N-1 time, and the N-th feature-quantity collection area. FIG. 11 is a diagram illustrating a state where a shelf position C in the (N-1) th captured image information is projected as a shelf position D in the Nth captured image information by a function F. FIG. 4 is a diagram illustrating an example of captured image information. FIG. 11 is a diagram illustrating a state in which a feature amount collection area is set in the (N−1) th captured image information. FIG. 14 is a diagram illustrating a state in which a feature amount collection area is set in N-th captured image information. FIG. 13 is a diagram illustrating an example of a state in which shelf positions D1 to D4 are specified in an N-th imaging in the square method; It is a figure which shows typically the N-th process in an identification information identification processing part. FIG. 14 is a block diagram schematically illustrating an example of a configuration of an image recognition system according to a fifth embodiment. FIG. 19 is a block diagram schematically illustrating an example of a configuration of an image information processing unit according to a fifth embodiment. 19 is a flowchart illustrating an example of an overall processing process of an image recognition system according to a fifth embodiment. 19 is a flowchart illustrating an example of a processing process of a process of generating processed normal image information according to the fifth embodiment. FIG. 9 is a diagram illustrating an example of a state in which normal image information is divided into meshes for each predetermined reference unit. It is a figure which shows typically the process which produces | generates processed normal image information by arranging the image information of the area | region of the mesh specified by satisfy | filling a predetermined condition. It is a figure which shows an example when the state which image | photographs goods is seen from above. FIG. 9 is a diagram schematically illustrating a process of generating appropriate sample information by cutting out portions where no reflection occurs in a plurality of pieces of captured image information and combining them with each other. It is a figure which shows an example of the external appearance of the goods used as a sample typically. It is a figure showing an example of the photography method in case the shape of the appearance of the product is the first case. It is a figure showing an example of the photography method in case the appearance of goods is the 2nd case. It is a figure showing an example of a photography method in case the shape of the appearance of goods is the third case.

  FIG. 1 shows an example of a system configuration of the image recognition system 1 of the present invention. The image recognition system 1 uses a management terminal 2 and a captured image information input terminal 3.

  The management terminal 2 is a computer used by an organization such as a company that operates the image recognition system 1. Further, the photographed image information input terminal 3 is a terminal for inputting image information of a photographed display shelf of a store.

  The management terminal 2 and the captured image information input terminal 3 in the image recognition system 1 are realized using a computer. FIG. 3 schematically shows an example of a hardware configuration of a computer. The computer includes an arithmetic device 70 such as a CPU for executing arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk for storing information, a display device 72 such as a display for displaying information, and a computer capable of inputting information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 for transmitting and receiving processing results of the arithmetic device 70 and information stored in the storage device 71 via a network such as the Internet or a LAN.

  When the computer has a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, for example, but are not limited thereto.

  The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be directly performed on the display by a predetermined input device (such as a touch panel pen) or a finger.

  The photographed image information input terminal 3 may include a photographing device such as a camera in addition to the above devices. As the captured image information input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, and a tablet computer can be used.

  Each means in the present invention is only logically distinguished in its function, and may physically or virtually have the same area. The processing order in each means of the present invention can be appropriately changed. Further, a part of the processing may be omitted. For example, a normalization process described later can be omitted. In that case, it is possible to execute processing on image information that has not been subjected to normalization processing.

  The image recognition system 1 includes a captured image information storage unit 20, a sample information storage unit 21, an identification information candidate storage unit 22, an identification information storage unit 23, an image information processing unit 24, and an identification information comparison processing unit 25.

  The photographed image information storage unit 20 stores the photographed image information received from the photographed image information input terminal 3, the photographing date and time, the shop identification information, the image information identification information, and the like in association with each other. The photographed image information may be image information for which the keystone correction processing is executed, and includes image information obtained by combining a plurality of pieces of one display shelf as one piece of image information. . The image information after the distortion correction processing has been executed is also included in the captured image information.

  The sample information storage unit 21 stores sample information for identifying the product of each face in the shelf of the display shelf shown in the normal image information. The sample information is image information obtained by photographing a product that may be displayed on a display shelf from a plurality of angles such as up, down, left, right, and oblique. FIG. 7 shows an example of the sample information stored in the sample information storage unit 21. FIG. 7 shows a case where canned beer is photographed from various angles as specimen information, but is not limited to canned beer. The sample information storage unit 21 stores sample information and product identification information in association with each other.

  The sample information storage unit 21 stores information necessary for calculating similarity, such as information of a pair of an image feature amount and its position, extracted from the sample information together with or instead of the sample information. You may memorize it. It is assumed that the sample information also includes information necessary for calculating the similarity. In this case, when performing the matching process between the image information of the face area and the sample information, the identification information specifying processing unit 246 does not have to calculate the image feature amount for the sample information every time, and shortens the calculation time. can do.

  The identification information candidate storage unit 22 stores, for each captured image information, a candidate for product identification information of a product displayed on each face of the shelf of the display shelf. For example, in association with a candidate for product identification information, image date / time information, store information, image information identification information of captured image information, image identification information of normal image information, and face identification information for identifying a face are associated. The information is stored in the identification information candidate storage unit 22. The candidates for the product identification information stored here are stored for each piece of captured image information. Therefore, since there are a plurality of pieces of photographed image information for the same display shelf, the product identification information candidates specified for each photographed image information are stored in the corresponding faces of the shelf of the same display shelf. .

  The identification information storage unit 23 stores the specified (confirmed) product identification information of the product displayed on each face of the shelf of the display shelf. For example, in correspondence with product identification information, identification information in association with shooting date / time information, store information, image information identification information of captured image information, image identification information of normal image information, and face identification information for identifying a face. The information is stored in the storage unit 23.

  The image information processing unit 24 specifies, in the photographed image information received from the photographed image information input terminal 3, a candidate of the commodity identification information of the commodity displayed on the shelf in the display shelf shown in the photographed image information Execute

  The image information processing unit 24 identifies the captured image information input reception processing unit 241, the captured image information normalization processing unit 242, the shelf position identification processing unit 243, the shelf area extraction processing unit 244, and the face identification processing unit 245. It has an information specifying processor 246 and a shelf image matching processor 247.

  The photographed image information input reception processing unit 241 receives input of image information (photographed image information) of a display shelf of a store photographed by the photographed image information input terminal 3 and stores the image information in the photographed image information storage unit 20. From the photographed image information input terminal 3, it is preferable to receive input together with photographed date and time, store identification information such as a store name, image information identification information for identifying image information, and the like, in addition to the photographed image information. 8 and 9 show examples of the photographed image information. FIGS. 8 and 9 show captured image information in which there are three shelves on the display shelf, and goods are displayed there.

  The photographed image information emplacement processing unit 242 performs trapezoidal correction processing on the photographed image information stored in the photographed image information storage unit 20 to generate the imposed image information. The trapezoidal correction process is a correction process performed so that the shelf of the display shelf shown in the captured image information is horizontal and the product tag for the product displayed there is vertical. Normalization is to transform the image information so that the optical axis of the lens of the photographing device is taken along the direction perpendicular to the plane to be photographed so as to be the same as when photographed from a far distance. There is a correction process.

  The trapezoidal correction process executed by the captured image information normalization processing unit 242 receives an input of designation of four vertices in the captured image information, and executes the trapezoidal correction process using each vertex. The four vertices for receiving the designation may be the four vertices of the shelf of the display shelf or the four vertices of the shelf position of the display shelf. Alternatively, four vertices of a set of two or three shelves may be used. Any four points can be designated as the four vertices. FIG. 10 and FIG. 11 show examples of photographed image information (normal image information) on which normalization processing has been performed.

  The shelf position specification processing unit 243 identifies a shelf area (shelf area) from the captured image information or the normal image information. That is, the display shelves shown in the photographed image information and the normal image information include a shelf area where products are displayed and a product tag placement area where product tags for the products displayed there are attached. . Therefore, the shelf area is specified from the normal image information. As for the identification of the shelf area, the operator of the management terminal 2 may manually designate the shelf area, and the shelf position specification processing unit 243 may accept the designation, or the shelf manual that has been manually received for the first time may be used. Based on the area information, the shelf area may be automatically specified for the second and subsequent times. 12 and 13 show a state where the shelf area is specified.

  The shelf area cutout processing section 244 cuts out image information of the area of the shelf identified by the shelf position specifying processing section 243 as shelf area image information. The shelf area cutout processing unit 244 may actually cut out as image information, or virtually cut out by specifying coordinates forming the shelf area without actually cutting out the image information. May be. If the display shelf has a plurality of shelves, each is cut out as shelf area image information. The coordinates indicating the area of the shelf are the coordinates of the vertices required to specify the area, and are, for example, four points in the normal image information, the coordinates of two points of upper right and lower left, upper left and lower right, and the like. Good. Further, the coordinates are relative coordinates with respect to a predetermined location (for example, the upper left vertex of the display shelf) in the image information such as the display shelf in the normal image information.

  The face specification processing unit 245 specifies a face where a product is placed (an area where a product is placed) for each shelf in the shelf area in the normal image information. The face identification processing unit 245 is divided into first face identification processing and second and subsequent face identification processing.

  In the first face identification processing by the face identification processing unit 245, commodities are placed within an area (preferably a rectangular area) constituted by the coordinates of the shelf identified by the shelf position identification processing unit 243. Specify the area (face). Specifically, it is used to identify narrow and narrow shadows that occur between products, to specify repetitive patterns of images, to specify steps on the top side of a package, and to divide based on constraints such as the same product width. The face area is specified by specifying the position. As the face identification processing, any method can be adopted depending on the category of the product and the form of the product, and is not limited to the above method. In addition, a correction input by a person in charge may be received for the face automatically specified. Further, the input of the position of the face may be received from the person in charge. The coordinates of the area constituting the identified face are the coordinates of the face area in the normal image information, the shooting date and time information, the store information, the image information identification information of the shooting image information, the image identification information of the normal image information, and the face. The information is managed in association with face identification information for identification. Further, the coordinates indicating the face area are coordinates of vertices necessary for specifying the rectangular area, and may be, for example, four points in the normal image information, coordinates of two points of upper right and lower left, upper left and lower right, and the like. . Further, the coordinates are relative coordinates with respect to a predetermined location (for example, the upper left vertex of the display shelf) in the image information such as the display shelf in the normal image information.

  The face identification processing of the second and subsequent faces by the face identification processing unit 245 is based on the coordinates of the area of the face identified by the previous (N−1) th normal image information for the same shelf on the same display shelf. The coordinates of the face area specified by the (Nth) normal image information may be used.

  The identification information specification processing unit 246 specifies, for each shelf of the display shelf, a candidate for the product identification information of the product displayed on the face. The product identification information includes, but is not limited to, a product name, a JAN code assigned to the product, and the like. Any information may be used as long as it can identify the product.

  The identification information specifying processing unit 246 performs the following processing. That is, by matching the image information of the face with the sample information of the product stored in the sample information storage unit 21 for each face, the candidate of the product identification information of the product displayed on the face is specified. Specifically, first, the similarity between the image information of the area constituted by the coordinates of the face to be processed and the sample information stored in the sample information storage unit 21 is determined, and the sample information having the highest similarity is determined. Is specified and identified as a candidate for the product identification information of the product displayed on the face constituted by the coordinates.

  Here, the following processing is performed to determine the similarity between the image information of the face and the sample information. First, in the processing up to the processing for specifying the candidate of the product identification information in the identification information specifying processing unit 246, the direction of the image information of the face area on the shelf of the normal image information and the direction of the sample information are the same (overturning and And the size of each image information is approximately the same (if the size of the image information is different over a predetermined range, the size of each Are adjusted so that the size of the image information is within a predetermined range).

  The identification information specifying processing unit 246 determines the similarity between the image information of the face and the sample information by using a feature point based on the image feature amount (for example, a local feature amount) of the image information of the face and the image of the sample information. Feature points based on feature amounts (for example, local feature amounts) are respectively extracted. Then, a pair having the highest similarity between the feature points of the image information of the face and the feature points of the sample information is detected, and the difference between the coordinates of the corresponding points is calculated. Then, an average value of the differences is obtained. The average value of the difference indicates the average moving amount of the entire image information of the face and the sample information. Then, the coordinate difference between all feature point pairs is compared with the average coordinate difference, and pairs having a large degree of deviation are excluded. Then, similarity is ranked according to the number of remaining corresponding points.

  The similarity between the image information of the face and the sample information can be calculated by the above method. In addition, in order to improve the accuracy, an EMD (Earth Movements Distance) between the color histograms may be obtained and used as a similarity measure. This makes it possible to compare the similarity relatively strong to environmental changes such as the brightness information of the captured image information, and to specify with high accuracy.

  As a similarity determination, an EMD between signatures (a set of image feature amounts and weights) of image information of each face area may be obtained and used as a similarity measure. As the signature image feature amount, for example, the frequency distribution of the image information of the face area in the HSV color space is obtained, the hue and the saturation are grouped, and the number of features and the image feature amount by the area in the HSV color space are determined. can do. The reason why the hue and the saturation are grouped is to reduce the dependence on the lightness so that the hue and the saturation are not largely influenced by the photographing conditions.

  Further, in order to speed up the processing, similarity such as L2 distance between image features such as color correlogram and color histogram of image information in an appropriate color space can be used instead of signature and EMD.

  The determination of similarity is not limited to the above. The identified product identification information candidates are identification information candidates associated with shooting date and time information, store information, image information identification information of shot image information, image identification information of normal image information, and face identification information for identifying a face. The information is stored in the storage unit 22. Note that the candidate for the product identification information specified here is not the determined product identification information because it is a candidate for the product identification information of the product appearing in the face in the normal image information based on certain photographed image information.

  The product identification information candidates identified as described above correspond to shooting date and time information, store information, image information identification information of captured image information, image identification information of normal image information, and face identification information for identifying a face. In addition, it is stored in the identification information candidate storage unit 22.

  The shelf image matching processing unit 247 compares the image information of the shelf area in the previous (N-1) -th normal image information, the image information of the shelf area in the current (N-th) normal image information, and If the similarity is high, it is determined that the candidates for the product identification information of each face in the shelf are the same. As described above, this similarity determination process is performed by comparing the image feature amount of the image information of the shelf area in the previous (N−1) th normal image information with the current (Nth) normal image information. The similarity may be determined based on the image information of the shelf area, or the EMD of the color histograms may be used. It is not limited to them. Then, instead of the identification processing for each face unit in the identification information identification processing unit 246, the identification information identification processing unit 246 sends the N-th candidate image identification information of each face in the shelf in the normal image information to the N- It is stored in the identification information candidate storage unit 22 as the same as the candidate for the product identification information of each face in the same shelf at the first time. As a result, it is possible to omit a process for a shelf that hardly changes, such as a shelf where the product does not move much or a shelf managed in a very short cycle. The processing by the shelf image matching processing section 247 may not be provided.

  The processing in the shelf image matching processing unit 247 is used for the N-th and N-1th comparisons, and for the N-th comparison of the normal image information obtained by photographing the same display shelf at different angles. May be used.

  The identification information comparison processing unit 25 uses the image information (normal image information) obtained by photographing the same display shelf at the same opportunity at different angles based on the product identification information candidates identified by the identification information identification processing unit 246. Then, a process of comparing the candidates for the product identification information of the corresponding faces and specifying (determining) the product identification information of the product shown in the face is executed. FIG. 14 shows an image diagram of the processing of the identification information comparison processing unit 25.

  The identification information comparison processing unit 25 receives the same display shelf at the same opportunity, for example, in the case of normal image information based on different photographed image information photographed on the same day, the merchandise identification information of each face identified by the image information processing unit 24. Each candidate is specified from the identification information candidate storage unit 22. For example, a candidate for product identification information corresponding to the same face identification information is specified. Then, each of the candidates for the product identification information is compared with a predetermined condition, and the product identification information corresponding to the face is specified (determined). For example, as shown in FIG. 14, it is assumed that there are two pieces of photographed image information (photographed image information 1 and photographed image information 2) that photograph the same display shelf at the same opportunity and have different angles. At this time, the product identification information 1 is associated with the face A in the fixed image information 1 obtained by normalizing the photographed image information 1 as a candidate for the product identification information. It is assumed that there is no product identification information candidate for face A in image information 2 (the product identification information for face A could not be identified based on normal image information 2). In this case, the product identification information 1 based on the normal image information 1 is specified as the product identification information of the face A. As another example, product identification information 1 is associated with face A in normal image information 1 obtained by normalizing captured image information 1 with a likelihood of 40% as a candidate for product identification information. If the product identification information 2 is associated with the face A in the fixed image information 2 obtained by normalizing the information 2 with the likelihood of 90% as the product identification information candidate, the product identification information having the highest likelihood 2 is specified as face A product identification information.

  As described above, the identification information comparison processing unit 25 determines whether or not there is a candidate for the product identification information, whether or not there is a candidate for the product identification information, in the image information at different angles obtained by photographing the same display shelf at the same opportunity. Is specified as the product identification information corresponding to the face based on the likelihood of the candidate. If one face in one piece of image information has a plurality of product identification information candidates corresponding to the likelihood, the likelihood may be calculated. For example, the face A in the fixed image information 1 obtained by normalizing the photographed image information 1 is associated with the product identification information 1 as a candidate for the product identification information with a likelihood of 65% and the product identification information 2 with a likelihood of 30%. The face A in the fixed image information 2 obtained by normalizing the photographed image information 2 is associated with the product identification information 2 as a candidate for the product identification information with a likelihood of 70% and the product identification information 1 with a likelihood of 25%. If the same product identification information is used, the values based on the likelihood are calculated by adding or multiplying the values based on the likelihood, and based on the value, one of the product identification information is specified. Is also good. For example, in the case described above, the product identification information 1 may be specified as 90, and the product identification information 2 may be determined as 100 by comparing the product identification information 2 with 100. As described above, the identification information comparison processing unit 25 compares, under a predetermined condition, the candidates for the product identification information specified from a plurality of pieces of image information at different angles when the same display target, that is, the display shelf, is photographed at the same opportunity. Then, the product identification information corresponding to the face is specified. The accuracy is improved by executing the processing in units of faces. However, the present invention is not limited to this, and it is only necessary to compare the candidates for the product identification information.

  Note that the image recognition system 1 in the present specification can be applied to a process of specifying the product identification information of the product shown in the image information of the product displayed on the display shelf, but is not limited thereto. Instead, the present invention can be applied to a case where identification information of an object is specified from image information in which an object is captured.

  A processing process using the image recognition system 1 of the present invention will be described with reference to block diagrams of FIGS. 1 and 2 and flowcharts of FIGS. In the following description, a case will be described in which two pieces of photographed image information photographed from two directions are used for the same display shelf. However, the same processing can be performed with three or more pieces.

  The person in charge of photographing the display shelves of the store photographs the same display shelves at the same opportunity from a plurality of angles so that the reflection of light or the like does not overlap. The photographed image information (FIGS. 8 and 9) of the display shelf of the store photographed from different angles at the same opportunity is input from the photographed image information input terminal 3 and photographed by the image information processing unit 24 of the management terminal 2. The image information input reception processing unit 241 receives the input (S100, S110). In addition, input of image capturing date and time, store identification information, and image information identification information of captured image information is received. Then, the captured image information input reception processing unit 241 stores the captured image information, the captured date and time, the shop identification information, and the image information identification information of the captured image information in the captured image information storage unit 20 in association with each other.

  When a predetermined operation input is received in the management terminal 2, the photographed image information normalization processing unit 242 extracts the photographed image information stored in the photographed image information storage unit 20, and is a vertex for performing a trapezoidal correction process. Input of four points (positions of display shelves) is received, and trapezoidal correction processing is executed (S120). FIGS. 10 and 11 show examples of photographed image information (normal image information) on which the trapezoidal correction processing has been performed in this manner.

  Then, the shelf position identification processing unit 243 identifies a shelf area by receiving a predetermined operation input in the management terminal 2 for the normal image information (S130). That is, the input of the shelf area in the normal image information is accepted. FIG. 12 and FIG. 13 are diagrams showing a state where the shelf area is specified.

  When the shelf area is specified as described above, the shelf area cutout processing unit 244 cuts out the image information of the shelf area from the normal image information based on the area of the shelf that has received the input in S130. (S140). Then, a process of specifying a face is executed for each shelf in the shelf area image information (S150). More specifically, for a shelf in the shelf area, a repetitive pattern of an image is specified, which specifies a narrow narrow shadow between products within a rectangular area formed by coordinates of four points. The face is specified by specifying the step on the upper side of the package, specifying the break position based on constraints such as the same product width, and the like. The identified face is provided with face identification information for identifying the face. This face identification information may be the same face identification information so that image information obtained by photographing the same display shelf can be identified as a corresponding face even if the image information is different. The correspondence of the identification information is stored.

  The identified coordinates of each face are stored in association with the photographing date and time, store identification information, image information identification information of photographed image information, image information identification information of normal image information, and face identification information for identifying a face. Let it. The coordinates of the face may be two points that can specify a rectangular area without storing four points.

  As described above, when each face of each shelf in the shelf area image information of the normal image information is specified, the identification information specifying processing unit 246 performs, for each face, the sample information stored in the sample information storage unit 21 and the matching process. Is performed, and the product identification information corresponding to the sample information having the highest similarity is specified. Then, it is specified as a candidate for product identification information of the product displayed on the face (S160). That is, each image feature amount is calculated from the image information in the rectangular area of the face of a certain shelf (X is the face identification information of the face in this area) and the sample information stored in the sample information storage unit 21. Then, similarity is determined by obtaining a pair of feature points. Then, the sample information having the highest similarity is specified. If the similarity at that time is equal to or greater than a predetermined threshold, the product identification information corresponding to the sample information is specified based on the sample information storage unit 21. Then, the specified product identification information is set as a candidate for the product identification information of the product displayed on the face of the face identification information X. Then, the identification information identification processing unit 246 associates the identified candidates for the product identification information with the shooting date and time, the store identification information, the image information identification information of the shot image information, the image information identification information of the normal image information, and the face identification information. Then, it is stored in the identification information candidate storage unit 22 (S170).

  Note that not all face product identification information can be specified. Therefore, for a face that cannot be identified (such as a face whose highest similarity is less than a predetermined threshold), input of product identification information is accepted, and the received product identification information is used as the shooting date and time, store identification information, and captured image. The image information identification information of the information, the image information identification information of the normal image information, and the face identification information may be stored in the identification information candidate storage unit 22 as candidates for the product identification information in association with each other. Similarly, an input may be received for the process of correcting the identified product identification information candidate. Since such a face is originally highly reliable, the face is determined as the product identification information without executing the processing of the identification information comparison processing unit 25 described later, and the identification information candidate storage unit 22 and the identification information storage unit 23. At this time, a flag indicating that the manual input has been received is attached to the identification information candidate storage unit 22, and if the flag is present, the processing of the identification information comparison processing unit 25 can be skipped.

  By performing the processing described above, it is possible to specify a candidate for the product identification information of the product displayed on the shelf of the display shelf shown in the captured image information.

  When a candidate for product identification information can be identified for a face in the normal image information based on each piece of captured image information, the identification information comparison processing unit 25 stores the candidate for product identification information between the corresponding faces in the identification information candidate storage unit 22. Then, by comparing them based on a predetermined condition, they are specified as the product identification information of the face (S200).

  More specifically, first, the identification information comparison processing unit 25 specifies a corresponding face in a plurality of pieces of normal image information having different angles photographed on the same display shelf at the same opportunity (S210). The method of specifying the corresponding face includes a method of specifying the face based on the same face identification information and a method of specifying the face associated as the face.

  Then, the candidate of the product identification information for the identified face is extracted from the identification information candidate storage unit 22, and the candidate of the product identification information of the corresponding face is compared with a predetermined condition or not (S220). (S230). Then, the specified product identification information is stored in the identification information storage unit 23 (S240).

  By performing the above processing, it is possible to specify the product identification information of the product displayed on the shelf of the display shelf shown in the captured image information. In addition, since the identification is performed based on the image information captured at a plurality of angles, the influence of the reflection of light or the like can be easily eliminated, and the accuracy of specifying the product can be improved.

  Also, in the second or subsequent processing, the processing in the shelf image matching processing unit 247 can be executed as appropriate.

  In the above-described first embodiment, the trapezoidal correction process is executed by designating four points. However, it is burdensome to designate and input the vertices that are the references each time. Therefore, a configuration may be adopted in which a vertex serving as a reference for the trapezoidal correction processing is automatically specified. The processing in this case will be described.

  In this case, the captured image information normalization processing unit 242 is divided into a first keystone correction process and a second and subsequent keystone correction processes. In addition to the first time, the first time includes a case where the operation is manually performed at an arbitrary timing in order to correct a deviation when the vertex is automatically specified. The second and subsequent times are other than the first time. In the present invention, the same display shelf is photographed at the same opportunity from a plurality of angles. Therefore, trapezoidal correction processing is executed by specifying the vertices for the first one (N-1st), and the vertices that are the reference of the trapezoidal correction processing are automatically specified for the second and subsequent sheets (N-th). You may make it.

  In the first trapezoidal correction process in the photographed image information normalization processing unit 242, similarly to the first embodiment, an input of designation of four vertexes of a rectangular area of the display shelf is received. The four vertices of the rectangular area of the display shelf may be the four vertices of the shelf position of the display shelf, the four vertices of the shelf, or the four vertices of the area where the product tag is attached. Alternatively, four vertices of a set of two or three shelves may be used. Here, the four vertices whose specification has been accepted are stored in association with the image date and time information, the store information, and the image information identification information of the captured image information. Then, the photographed image information normalization processing unit 242 executes trapezoidal correction processing on the photographed image information based on the coordinates of the four vertexes for which the designation has been received, and sets the trapezoidal correction image information.

  The photographed image information normalization processing unit 242 executes the second and subsequent trapezoidal correction processing as follows.

  First, the photographed image information normalization processing unit 242 calculates the vertex coordinates of the (N−1) -th photographed image information obtained by photographing the same (substantially the same) region corresponding to the N-th photographed image information in the previous processing. Identify from stored information. The vertex coordinates of the (N−1) -th photographed image information obtained by photographing the same (substantially the same) region corresponding to the N-th photographed image information are stored in store identification information, image identification information, photographing date / time information, and the like corresponding to the photographed image information. Specify based on. Then, for the (N-1) th captured image information, a rectangular area of a predetermined size including the specified vertex coordinates of the four vertices, for example, a square having a width of about 1/5 of the width of the shelf, is taken as the feature amount collection area 2420. Set as FIG. 15 shows an example of a state in which the feature amount collection area 2420 is set for the (N-1) th captured image information. The feature amount collection area 2420 may be a rectangular area including vertex coordinates. On the other hand, if the backgrounds of the display shelves are matched, the background will be greatly displaced even if the photographing position is slightly shifted. Therefore, it is preferable to set the feature amount collection area 2420 at a position that includes as much of the inside of the display shelf as possible. That is, it is preferable that the vertex coordinates are located outside the display shelf in the feature amount collection area 2420 with respect to the center point of the feature amount collection area 2420.

  For example, the four points of the vertex coordinates are located at the upper left, upper right, lower left, and lower right. Then, when the rectangular region of the feature amount collection region 2420 is divided into two regions each of which is divided into two at the vertical and horizontal centers, the feature amount collection region 2420 including the upper left vertex coordinates has the vertex coordinates of the upper left region of the rectangular region. Is set so as to be located at. Similarly, in the feature quantity collection area 2420 including the upper right vertex coordinates, the feature quantity collection area 2420 is set so that the vertex coordinates are located in the upper right area of the rectangular area, and the feature quantity collection area including the lower left vertex coordinates is set. In the region 2420, the feature amount collection region 2420 is set such that the vertex coordinates are located in the lower left region of the rectangular region. In the feature amount collection region 2420 including the lower right vertex coordinates, the vertex coordinates are set in the rectangular region. The feature amount collection area 2420 is set so as to be located in the lower right area. Thus, the vertex coordinates are located outside the display shelf in the feature amount collection area 2420 with respect to the center point of the feature amount collection area 2420.

  Next, the photographed image information normalization processing unit 242 includes the feature amount collection area 2420 set in the (N-1) th photographed image information in the Nth photographed image information, and performs the (N-1) th photographed image information A feature amount collection area 2421 having a size equal to or larger than the feature amount collection area 2420 is set. The feature amount collection area 2421 set in the N-th captured image information does not exceed half the size of the short side. Further, when the image is outside the captured image information, the range is trimmed. FIG. 16 shows an example of a state in which the feature amount collection area 2421 is set for the N-th captured image information.

  Then, the photographed image information normalization processing unit 242 collects a local feature amount in each of the feature amount collection regions 2420 set for the (N-1) th photographed image information, and calculates a feature point based on the local feature amount and its coordinates. Set and memorize. In addition, in each feature amount collection area 2421 set for the N-th captured image information, a local feature amount is collected, and a feature point based on the local feature amount and a set of its coordinates are stored.

  The captured image information normalization processing unit 242 corresponds to the local feature amount of the feature point in the feature amount collection region 2420 of the (N-1) th captured image information and the feature amount collection region 2420 of the (N-1) th captured image information. Then, the local feature amount of the feature point in the feature amount collection area 2421 of the N-th captured image information at the position to be compared is compared. Then, a feature point of each local feature amount of the N-th shot image information that is closest to each local feature amount of each feature point of the (N-1) th shot image information is specified. Then, the feature points of the closest local feature quantities are paired, and the coordinates of the feature points based on the paired local feature quantities are associated with each other. Note that, at this time, pairs in which the proximity (similarity) between the local feature values is less than a predetermined threshold are excluded. As a result, a pair of a feature point of a local feature amount in the (N−1) th feature amount collection region 2420 and a feature point of the closest local feature amount in the N-th feature amount collection region 2421 can be specified. FIG. 17 shows a pair relationship between the feature points of the local feature amounts of the (N−1) th feature amount collection region 2420 and the feature points of the local feature amounts of the N-th feature amount collection region 2421. In FIG. 17, the point group of the feature points by the local feature amount in the (N−1) th feature amount collection region 2420 is A, the point group of the feature point by the local feature amount in the Nth feature amount collection region 2421 is B, N− The vertex used in the first trapezoidal correction process is indicated by C.

  The coordinates of the point group A of the feature points based on the local feature amounts in the (N-1) th feature amount collection region 2420 and the point group B of the feature points based on the local feature amounts in the N-th feature amount collection region 2421 corresponding to the point group A A function F (perspective transformation / homography) for projecting the point group A onto the point group B is obtained based on the coordinates of the point group A and the point group A. The function F includes, but is not limited to, a method of repeating sampling estimation and using RANSAC of OpenCV which is a kind of robust estimation. Note that a pair having a large deviation from the relationship line in the projection relationship is excluded from the processing target.

  After obtaining the function F in the captured image information normalization processing unit 242, the captured image information normalization processing unit 242 calculates the coordinates of the vertex C used in the (N-1) -th keystone correction process based on the function F. The image is projected onto the Nth captured image information and specified as the coordinates of the vertex D for the Nth trapezoidal correction process. This is schematically shown in FIG.

  By performing the above processing on each of the feature amount collection areas 2420 and 2421, the four vertices of the shelf position for the trapezoidal correction processing in the Nth captured image information are specified. Then, the photographed image information normalization processing unit 242 performs trapezoidal correction processing on the Nth photographed image information based on the specified four vertices to perform normalization, generates and stores the normal image information. At this time, the photographed image information normalization processing unit 242 associates the photographed date and time information, the store information, the image information identification information of the photographed image information, and the image identification information of the fixed image information with the corresponding image information. Then let me remember. The coordinates of the specified vertex corresponding to the N-th photographed image information are stored in association with the photographing date and time information, store information, and image information identification information of the photographed image information.

  Note that the process of specifying the vertices used in the trapezoidal correction process in the photographed image information normalization processing unit 242 is not limited to the case where a product is specified from image information obtained by photographing a display shelf as in the present invention. It can also be applied to the image recognition system 1 that normalizes a plurality of pieces of image information obtained by shooting and generates normal image information. This makes it possible to normalize a plurality of pieces of image information obtained by photographing the same photographing target object, and generate normal image information of the photographing target object.

  In addition, in order to specify the four vertices of the shelf position for the trapezoidal correction processing in the N-th captured image information, in the above description, the feature points based on the local feature amount in the feature amount collection area 2420 in the (N-1) -th captured image information A function F is obtained using the point group A of the above and the point group B of the feature points based on the local feature amounts in the feature amount collection area 2421 in the N-th captured image information, and is used in the (N-1) -th keystone correction process. The process of projecting the coordinates of the vertex C onto the N-th captured image information using the function F and specifying the coordinates as the coordinates of the vertex D for the N-th trapezoidal correction process has been described. However, in such processing, the coordinates (positions) of corresponding points of similar image information may be found in the (N-1) th captured image information and the Nth captured image information. Instead, any feature amount may be used as long as it is a type of feature amount that specifies a location in the image information. For example, there are sharp points and highlight points in the image information. In this specification, a feature amount that specifies a location in image information, such as a local feature amount, is referred to as an image feature amount (position specifying image feature amount). In the description of the present specification, a case where the local feature value is used as described above as the image feature value will be described.

  Next, a vertex specifying process for performing the trapezoidal correction process according to the second embodiment will be described. In this case, when the vertex for performing the trapezoidal correction processing is already specified in the photographed image information obtained by photographing an arbitrary display shelf, and the photographed image information is photographed at the same opportunity and at a different angle on the same display shelf. Will be described.

  The photographed image information obtained by photographing the display shelf of the store is input from the photographed image information input terminal 3, and the input is received by the photographed image information input reception processing unit 241 of the management terminal 2. FIG. 19 shows an example of captured image information. In addition, input of image capturing date and time, store identification information, and image information identification information of captured image information is received. Then, the photographed image information input reception processing unit 241 causes the photographed image information storage unit 20 to store the photographed image information, the photographing date and time, the shop identification information, and the image information identification information that have received the input in association with each other.

  When a predetermined operation input is received at the management terminal 2, the photographed image information normalization processing unit 242 extracts the photographed image information stored in the photographed image information storage unit 20, and determines the shelf position for executing the trapezoidal correction process. A process for specifying the vertex D (D1 to D4) is executed.

  The vertex coordinates (for example, vertex coordinates C1 to C4 and the vertex coordinates C1 to C4) of the previous (N-1) th photographed image information (FIG. 20) that photographed the same or almost the same area corresponding to the photographed image information this time (Nth time) (FIG. 19). To do). The vertex coordinates of the previous shot image information may be specified based on the shooting date and time, store identification information, image identification information of the shot image information, and the like.

  The photographed image information normalization processing unit 242 extracts the (N-1) th photographed image information from the photographed image information storage unit 20, and for each of the vertices C1 to C4, a rectangle of a predetermined size including one vertex at a time. The area is set as the (N−1) th captured image information as a feature amount collection area 2420. FIG. 20 shows a state in which the feature amount collection area 2420 is set in the (N-1) th captured image information.

  Further, the photographed image information normalization processing unit 242 extracts the N-th photographed image information (FIG. 19) from the photographed image information storage unit 20, and extracts a feature of a wider range than the (N−1) -th feature amount collection area 2420. The quantity collection area 2421 is set as the Nth captured image information. FIG. 21 shows a state in which the feature amount collection area 2421 is set in the N-th captured image information. Each feature amount collection region 2421 in the N-th captured image information includes one N-1th feature amount collection region 2420. In FIG. 21, the inclusion relationship is shown by showing the (N−1) -th feature amount collection region 2420 in the N-th feature amount collection region 2421.

  Then, the photographed image information normalization processing unit 242 collects a local feature amount in each of the feature amount collection regions 2420 set for the (N-1) th photographed image information, and sets a set of feature points and coordinates by the local feature amount. Is stored. In addition, a local feature amount is collected in each feature amount collection region 2421 set for the Nth captured image information, and a set of feature points and coordinates based on the local feature amount is stored.

  The photographed image information normalization processing unit 242 generates a feature amount collection area 2421 of the N-th photographed image information closest to each local feature amount of each feature point in the feature amount collection area 2420 of the (N-1) th photographed image information. , The feature points of the local feature amounts are specified, and these coordinates are associated with each other as the feature points of the local feature amounts forming a pair. FIG. 17 shows the relationship between the pair of the (N-1) th N-1th feature amount collection area 2420 and the Nth feature amount collection area 2421.

  The point group of the feature points based on the local feature amounts in the (N−1) th feature amount collection area 2420 is A, the point group of the feature points based on the local feature amounts in the Nth feature amount collection area 2421 is B, and the (N−1) th time. Let C (C1 to C4) be the vertices used in the trapezoidal correction processing of, the photographed image information emplacement processing unit 242 converts the point group A into the point group B based on the coordinates of the point group A and the point group B. A function F (perspective transformation / homography) to be projected on is obtained.

  Then, the photographed image information normalization processing unit 242 projects the coordinates of the vertices C (C1 to C4) used in the (N-1) th trapezoidal correction processing based on the obtained function F, and performs the Nth trapezoidal correction processing. Are specified as the coordinates of the vertex D (D1 to D4).

  By performing the above processing on each of the feature amount collection areas 2420 and 2421, the four vertices D (D1 to D4) for the Nth trapezoidal correction processing can be automatically specified. The coordinates of the specified N-th vertex D (D1 to D4) are stored in association with the photographing date and time, the store identification information, and the image information identification information of the photographed image information. FIG. 22 shows the specified vertices D1 to D4.

  As described above, when the apex D (D1 to D4) of the shelf position for the trapezoidal correction processing on the Nth captured image information is specified, the captured image information emplacement processing unit 24222 specifies the vertex D (D1 to D4). ), A trapezoidal correction process is performed on the Nth captured image information.

  By executing the above processing, the corresponding vertices can be automatically specified for the second and subsequent keystone correction processing without specifying the four vertices used in the keystone correction processing. The burden on the person can be reduced.

  Further, a modified example of the face specification processing in the face specification processing unit 245 according to the first embodiment will be described. In this embodiment, the face identification processing of the first embodiment may be performed as the first processing, and the face identification processing may be automatically performed as the second and subsequent face identification processing. The processing in this case will be described.

  Note that, in addition to the first time, the case of performing the processing of the first embodiment at an arbitrary timing in order to correct a shift at the time of automatic identification is also included. The second and subsequent times are other than the first time. The second and subsequent times also include cases in which processing is performed on photographed image information and normal image information of photographed image information and normal image information obtained by photographing the same display shelf at the same opportunity and at different angles. .

  The face identification processing unit 245 executes the same processing as the processing of the first embodiment as the first face identification processing. Then, the face specification processing unit 245 performs the face specification processing for the second and subsequent faces by using, for the same shelf on the same display shelf, the coordinates of the area of the face specified by the previous (N−1) th normal image information. The extracted coordinates are set as the coordinates of the face area specified by the current (N-th) normal image information.

  Similar to the coordinates of the position of the shelf, the coordinates of the face area are relative coordinates with respect to a predetermined location (for example, the upper left vertex C1 of the display shelf) in the display shelf in the normal image information.

  Further, as a modified example of the first embodiment, the process of the first embodiment is performed as the process of specifying the product identification information of the product displayed on the face for each shelf of the display shelf in the identification information specifying processing unit 246. The following process is executed as the process for specifying the product identification information for the second and subsequent times as the process for specifying the product identification information.

  The identification information identification processing unit 246 first identifies the face identification information of the face to be processed in the identification processing of the product identification information of the face in the N-th normal image information. Let X be the identified face identification information. Then, the image information of the area of the face identification information X in the N-th normal image information is compared with the image information of the area in the (N-1) -th normal image information at the position corresponding to the face identification information X. For the determination of the similarity, in addition to the processing of the first to third embodiments described above, a signature (a set of image feature amounts and weights) of the image information of the area of each face is obtained, and a similarity measure and a similarity measure are obtained. can do. As the signature image feature amount, for example, the frequency distribution of the image information of the face area in the HSV color space is obtained, the hue and the saturation are grouped, and the number of features and the image feature amount by the area in the HSV color space are determined. can do. The reason why the hue and the saturation are grouped is to reduce the dependence on the lightness so that the hue and the saturation are not largely influenced by the photographing conditions.

  Further, in order to speed up the processing, similarity such as L2 distance between image features such as color correlogram and color histogram of image information in an appropriate color space can be used instead of signature and EMD.

  If the similarity of the image information of the face area is equal to or more than a certain threshold, a candidate for the product identification information corresponding to the face of the area in the (N-1) th normal image information is extracted from the identification information candidate storage unit 22, It is a candidate for the product identification information of the face identification information X in the N-th normal image information. As a result, it is possible to specify the candidate of the product identification information of the face identification information X in the N-th normal image information to be processed. If the similarity is less than a certain threshold value, the image information of the area of the face identification information X in the N-th normal image information and the sample information stored in the sample information storage unit 21 are similar to the case of the first time. In comparison, the product identification information having the similarity equal to or higher than the predetermined threshold value and having the highest similarity is specified as the candidate of the product identification information of the face of the face identification information X in the N-th normal image information.

  In comparison with the image information of the face area in the N-1th normal image information, not only the comparison with the corresponding face position but also a face in a predetermined range may be included as a comparison target. For example, when comparing with the image information of the area of the face identification information X in the N-th normal image information, the comparison target is not only the area of the face identification information X in the (N-1) th normal image information but also that area. A face area within a predetermined range, for example, a face located at a position separated by one or more in the left-right direction, and a face area located on the upper and lower shelves may be included. Furthermore, in addition to the area of the face identification information X in the N-1th normal image information, the area of a plurality of adjacent faces such as face identification information X-2, X-1, X, X + 1, and X + 2 is included. You may.

  In this case, the image information of the area of the face identification information X in the N-th normal image information and the image information of the area of the face to be compared in the (N−1) -th normal image information are And identifies the face identification information of the (N−1) th normal image information having the highest similarity. Note that the similarity may be a condition that the similarity is equal to or more than a certain threshold. Then, the candidate of the product identification information corresponding to the identified face identification information of the (N-1) th normal image information is extracted from the identification information candidate storage unit 22, and the product identification of the face identification information X in the Nth normal image information is performed. Information candidates. FIG. 23 schematically shows this processing. FIG. 23A shows the previous (N-1) -th normal image information, and FIG. 23B shows the current (N-th) normal-image information. Then, the image information of the area of each face of the shelf 1 of the N-th regular image information is compared with the image information of the area of each face of the shelf 1 of the N-1th regular image information. To determine the candidate of the product identification information of the face of the shelf 1 of the (N-1) -th normal image information having the highest similarity, and the product of the face of the shelf 1 of the N-th normal image information. Indicates that the identification information is specified as a candidate. FIG. 23 shows a case in which, in addition to the face of the (N−1) -th regular image information located at a position corresponding to the face of the N-th regular image information, comparison is performed with two faces each on the left and right sides. . Note that not only the same shelf but also the image information of the face positions of the upper and lower shelf may be compared. For example, in the case of FIG. 23, when specifying the candidate of the product identification information at the face position at the center of the shelf 2 of the N-th normal image information, the center of the shelf 2 of the N-1th normal image information and Not only is it compared with the image information of the two face areas on the left and right sides, but also the image information of the center of the shelf 1 and the face area on the left and right sides of the N-1th regular image information, A comparison may be made between the image information of the center of the shelf 3 and the image information of two faces on the left and right sides of the shelf 3 in the first normal image information.

  If the candidate of the product identification information cannot be specified due to the similarity not satisfying the threshold value as a result of the comparison of the N-1th normal image information with the face image information, the Nth normal image information Is compared with the sample information stored in the sample information storage unit 21, and the product identification information having the similarity equal to or more than a predetermined threshold value and having the highest similarity is identified by N It is specified as a candidate for the product identification information of the face of the face identification information X in the second regular image information. In this case, the similarity determination process can be performed in the same manner as the first process of specifying the product identification information.

  The product identification information candidates identified as described above correspond to shooting date and time information, store information, image information identification information of captured image information, image identification information of normal image information, and face identification information for identifying a face. In addition, it is stored in the identification information candidate storage unit 22.

  In the first to fourth embodiments, in each piece of image information obtained by photographing the same display shelf at the same opportunity and at different angles, the candidates of the product identification information in the face are specified, and the corresponding faces are compared. The product identification information on the face has been determined. In the present embodiment, for each piece of image information photographed at different angles, a candidate for product identification information on the face is not specified, but image information photographed at different angles is first synthesized to eliminate the reflected portion of light or the like. A description will be given of a case where the identification processing of the product identification information on the face is performed on one piece of image information obtained after the insertion. An example of a block diagram of the system configuration of the image recognition system 1 in this case is shown in FIGS.

  The image recognition system 1 according to the present embodiment includes a captured image information storage unit 20, a specimen information storage unit 21, an identification information storage unit 23, and an image information processing unit 24. The photographed image information storage unit 20, the specimen information storage unit 21, and the identification information storage unit 23 have the same configuration as the first to fourth embodiments.

  The image information processing unit 24 includes a photographed image information input reception processing unit 241, a photographed image information positive value processing unit, a shelf position identification processing unit 243, a shelf area cutout processing unit 244, a face identification processing unit 245, and identification information. It has a specific processing unit 246, a shelf image matching processing unit 247, and an image information processing processing unit 248.

  The photographed image information input reception processing unit 241 and the photographed image information positive value processing unit execute the same processing as in the first to fourth embodiments. That is, the photographed image information input reception processing unit 241 receives input of image information (photographed image information) of the display shelf of the store photographed by the photographed image information input terminal 3 and stores the image information in the photographed image information storage unit 20. Further, the photographed image information normalization processing unit 242 performs trapezoidal correction processing on the photographed image information stored in the photographed image information storage unit 20 to generate normal image information.

  The image information processing unit 248 executes a process of synthesizing one or more ranges of part or all of the normal image information that has been subjected to the positive value conversion process based on the captured image information. That is, a part or the entire range of each piece of normal image information obtained by photographing the same display shelf at different angles at the same opportunity is divided into meshes using a predetermined reference unit, for example, about 10 to 100 pixels as a reference unit. Then, the average value of the brightness of the image information of the area of each mesh is calculated. The average value of the brightness is compared for each of the corresponding meshes in the normal image information photographed at different angles, and the mesh of the image information with the smallest average value of the brightness is specified. This process is performed for all meshes. By arranging the image information of the specified mesh area in the vertical or horizontal direction as one piece of image information (processed normal image information) and synthesizing the image information, it is possible to generate image information in which the influence of light reflection is reduced. . It should be noted that the image information of the mesh area is not limited to the average value of the brightness, and the mesh to be combined may be specified using an index value based on another evaluation criterion for the brightness. Further, it is preferable that the image information to be processed by the image information processing unit 248 is corrected so that the photographing conditions such as the shutter speed and the aperture value are the same or equal among the image information. When the above-described processing using the lightness is executed between image information having different shooting conditions or not being corrected equivalently, image information having a low index value based on lightness (dark) is selected for a portion where no image is reflected. It will be. Even in this case, the reflection itself can be removed, but problems such as a reduction in contrast and brightness of the generated image information or a noticeable roughness are likely to occur, which may affect recognition accuracy. It is.

  Each processing in the face identification processing unit 245, the identification information identification processing unit 246, and the shelf image matching processing unit 247 is the same as that in the first to fourth embodiments, but the processing target is the shelf area of the normal image information. Instead, it is performed on the processed normal image information generated by the image information processing unit 248. In the first to fourth embodiments, the identification information identification processing unit 246 identifies the candidate for the product identification information of the product in the face area and stores the candidate in the identification information candidate storage unit 22. The product identification information of the product in the face area is specified, and the specified information is stored in the identification information storage unit 23.

  By executing the processing on the processed normal image information in this manner, the image in the range of the shelf area can be identified without specifying the candidate for the product identification information on the face as in the first to fourth embodiments. Since the information is subjected to the synthesis processing by the image information processing unit 248, the influence of the reflection of light on the shelf area can be reduced. Therefore, the product identification information on the face can be directly specified with high accuracy.

  The processing for the normal image information in the image information processing unit 248 includes the image information of the area of the shelf position specified by the shelf position identification processing unit 243 and the image of the shelf area extracted by the shelf area extraction processing unit 244. It is preferable to execute the processing on the information, but the present invention is not limited to this. The processing may be executed on a part or the entire range of the normal image information generated by the photographed image information positive value processing unit.

  Next, an example of a processing process according to the present embodiment will be described with reference to the flowcharts of FIGS.

  The person in charge of photographing the display shelves in the store photographs the same display shelves from a plurality of angles so that the reflection of light or the like does not overlap. The photographed image information (FIGS. 8 and 9) of the display shelf of the store photographed from different angles at the same opportunity is input from the photographed image information input terminal 3 and photographed by the image information processing unit 24 of the management terminal 2. The image information input reception processing unit 241 receives the input (S300). In addition, input of image capturing date and time, store identification information, and image information identification information of captured image information is received. Then, the captured image information input reception processing unit 241 stores the captured image information, the captured date and time, the shop identification information, and the image information identification information of the captured image information in the captured image information storage unit 20 in association with each other.

  When a predetermined operation input is received in the management terminal 2, the photographed image information normalization processing unit 242 extracts the photographed image information stored in the photographed image information storage unit 20, and is a vertex for performing a trapezoidal correction process. The input of four positions (display shelf positions) is received, and trapezoidal correction processing is executed (S310). FIGS. 10 and 11 show examples of photographed image information (normal image information) on which the trapezoidal correction processing has been performed in this manner.

  Then, the shelf position identification processing unit 243 identifies a shelf area by receiving a predetermined operation input in the management terminal 2 with respect to the normal image information (S320). That is, the input of the shelf area in the normal image information is accepted. FIG. 12 and FIG. 13 are diagrams showing a state where the shelf area is specified.

  When the shelf area is specified as described above, the shelf area cutout processing unit 244 cuts out the image information of the shelf area from the normal image information based on the area of the shelf that has received the input in S320. (S330).

  The image information processing unit 248 executes a processed normal image information generation process on the image information of the shelf area extracted by the shelf area extraction processing unit 244 in S330 (S340).

  First, the image information processing unit 248 changes the magnification of the cut-out shelf area information so that the vertical and horizontal sizes of the corresponding shelf area coincide or substantially coincide with each other. And keep the size. Then, the normal image information of the cut-out shelf area is sectioned by a predetermined reference unit, for example, every 10 pixels with a mesh (S400). FIG. 28 schematically shows this state. FIG. 28A shows a state in which the shelf area of the normal image information in FIG. 13 is meshed with FIG.

  Then, the image information processing unit 248 calculates, for each mesh, an average value of brightness for each piece of image information in the mesh area (S410). Note that the average value may be a simple average, a weighted average, or the like, or may be a value other than the average as long as it is a reference value based on the brightness of each image information of the mesh area.

  Then, the image information processing unit 248 compares the average value (reference value) of the brightness for each corresponding mesh (S420), and specifies the image information of the mesh with the smallest brightness among the comparisons (S430). In the case of FIG. 28, in each of the normal image information of FIGS. 28 (a) and 28 (b), for each mesh in the shelf area of the normal image information, the brightness of each image information of the mesh area is determined. Calculate the average value. Since the mesh A1 (FIG. 28A) and the mesh A2 (FIG. 28B) correspond, the average value of the brightness of the mesh A1 and the average value of the brightness of the mesh A2 are compared. For example, if the average value of the brightness of the mesh A1 is small, the mesh A1 is specified. Further, since the mesh B1 (FIG. 28A) and the mesh B2 (FIG. 28B) correspond, the average value of the brightness of the mesh B1 is compared with the average value of the brightness of the mesh B2. For example, if the average value of the brightness of the mesh B2 is small, the mesh B2 is specified. In this manner, the brightness comparison and the specific processing for the corresponding meshes are performed for all the meshes.

  Then, by arranging the specified meshes vertically and horizontally, one image information (processed normal image information) is generated (S440). In the above example, since the meshes A1 and B2 are specified, the meshes specified in the processing of S430, such as the meshes A1 and B2, are arranged vertically and horizontally to arrange one mesh in the shelf area. It is synthesized and generated as image information. FIG. 29 schematically shows this processing. In FIG. 29, for the sake of simplicity, the processing for the highest shelf area in the shelf area of FIG. 28 is shown as an example.

  By executing the above-described processing, it is possible to generate image information with reduced reflection of light in the shelf area.

  As in the first to fourth embodiments, the face identification processing unit 245 performs face identification processing on the processed normal image information, and the identification information identification processing unit 246 outputs the product identification information on a face-by-face basis. A specific process is executed (S350, S360). That is, the face specifying processing unit 245 specifies the face area in the processed normal image information, and the identification information specifying processing unit 246 determines the sample information stored in the sample information storage unit 21 and the image information in the face area. By executing the process of specifying the product identification information by the image matching process, the product identification information of the product in the face area is specified. Then, the specified product identification information is stored in the identification information storage unit 23 as in the first to fourth embodiments (S370).

  Even if the processing according to the present embodiment is used, it is not always possible to specify the product identification information of all faces. Therefore, for the face that cannot be identified, the input of the product identification information is received, and the received product identification information is used as the shooting date and time, the store identification information, the image information identification information of the shot image information, and the image information of the normal image information. The identification information and the face identification information may be stored in the identification information candidate storage unit 22 as candidates for the product identification information in association with each other. Similarly, an input may be received for the process of correcting the identified product identification information candidate.

  By performing the above processing, it is possible to specify the product identification information of the product displayed on the shelf of the display shelf shown in the captured image information.

  Also in this embodiment, the modification processing of each processing unit in the first to fourth embodiments can be used.

  If the specimen information stored in the specimen information storage unit 21 includes reflection of light or surrounding colors, the image characteristics (image characteristic amount) of the appearance (eg, package or packaging) of the product are affected. Affects the accuracy of the matching process. Therefore, it is preferable to use, as the image information used as the sample information, image information captured so that the original color of the product is not impaired by reflection of light or surrounding colors. However, depending on the external shape of the product and the material, it is sometimes difficult to avoid reflection due to light reflection. For example, in the case of a metal can, since the reflectance is high, it is inevitable that the reflection of the illumination light is reflected in the photograph.

  Factors that cause disturbance such as reflection when photographing specimen information of a product include the photographing environment of the product and the shape of the appearance of the product. The shooting environment includes the background of the product and the state of the surface on which the product is placed, in addition to the light source such as a lighting device. In addition, depending on the direction of the surface constituting the appearance of the product, the appearance of the product reflects light such as a photographing device and a lighting device.

  Therefore, in the present embodiment, a process for obtaining sample information without reflection will be described. The processing in the present embodiment is also applicable to the sample information of the first to fifth embodiments.

  The following shooting method can be applied as a method for reflection in the shooting environment.

  In order to prevent distortion of the sample information, the photographing by the photographing device is performed from the horizontal facing direction of the product. In this case, since it is sometimes unavoidable that reflection is caused on the appearance of the product due to reflection or the like, the reflection is performed on a flat object having no specific hue such as white or gray. Specifically, an image of a product with sample information is set against a white or gray background having no specific hue. As the background of the product, white or gray paper, cloth, or the like is preferably used. The white or gray paper or cloth used as the background is not folded on the surface on which the product is to be placed, and is spread out to the front of the product until the reflected image becomes a flat plate. Thereby, the influence of the reflection due to the shooting environment can be reduced.

  The following photographing method can be applied as a method for reflection by the appearance shape of the product.

  For a product to be photographed for sample information, the product and the lighting are changed without changing the relative position between the product and the photographing device (for example, the relative position between the surface of the product photographed by the photographing device and the photographing device). By changing the relative position with respect to the light source (for example, the surface of the product irradiated with light by the light source and the relative position with respect to the light source), the product is photographed a plurality of times. In other words, the relative position between the imaging device and the light source is changed without changing the relative position between the product and the imaging device, and the product is imaged a plurality of times. In other words, the product is captured at the same position, the same size (size), and the same angle in the captured image information, and the light from the light source such as illumination is radiated to the product from different angles, so that the product is photographed multiple times. Then, the image recognition system 1 receives the input. FIG. 30 shows an example of a state in which a product is imaged when viewed from above. FIG. 30A is a diagram illustrating an example of a state in which an image is shot from the left side of a product with illumination, and FIG. 30B is a diagram illustrating an example of a state in which an image is shot with illumination from the right side of the product. Although FIG. 30 shows a case where images are taken from two places on the left and right across the image taking device, images may be taken from three or more places. In FIG. 30, the light path of the reflection generated at the position closest to the center axis of the product by the light emitted from the light source closest to the photographing device is indicated by a broken arrow. In this case, the angle of the incident light (incident angle) and the angle of the reflected light (reflection angle) are the same, with the perpendicular to the surface of the product as the axis of symmetry.

  Then, the image recognition system 1 obtains appropriate sample information by cutting out portions where no reflection has occurred in the plurality of pieces of image information whose input has been received and combining them with each other. Appropriate sample information is stored in the sample information storage unit 21. FIG. 31 schematically shows this processing. FIG. 31A shows image information shot in the state of FIG. 30A, FIG. 31B shows image information shot in the state of FIG. 30B, and FIG. 31 (a) and 31 (b) are appropriate sample information obtained by synthesizing the image information. That is, in FIG. 30A, since the product is photographed by irradiating light from the right side, the reflection of light is likely to occur on the right side of the product. In addition, in FIG. 30B, since the product is photographed by irradiating light from the left side, reflection of light is likely to occur on the left side of the product. In other words, no light is reflected on the left side (A) of the product in FIG. 31A, and no light is reflected on the right side (B) of the product in FIG. 31B. Therefore, the image information of the left side (A) of the product in FIG. 31A and the image information of the right side (B) of the product in FIG. 31B are cut out and combined into one piece of image information. (FIG. 31C). Thereby, it is possible to synthesize appropriate sample information without reflection of light.

  In FIGS. 30A and 30B, it is preferable that the sums α and β of the incident angle and the reflection angle be the same or substantially the same. As a result, in the photographed image information, if the product is located near the center of the image information in the horizontal direction, the left half or the right half where no reflection occurs in the photographed image information is cut out. Since it is sufficient to combine the images, the image combining process is simplified. When the sums α and β of the incident angle and the reflection angle are the same or substantially the same, the accuracy of the image information synthesizing process can be improved, but the angles α and β need not be the same or substantially the same.

  The processing of FIGS. 30 and 31 will be described more specifically as follows. In general, the appearance of a product to be imaged can be roughly classified into three types shown in FIG. 32 since the horizontal sectional view of the product is convex or substantially convex. That is, in the case of a rectangular parallelepiped or a substantially rectangular parallelepiped (a front is flat) as in a product in a box (FIG. 32 (a)), and in the case of a beverage can having a columnar shape or the like and a vertical side (see FIG. 32 (b)), a case where a part of the side surface is not vertical, such as a wine bottle having a curved surface such as a spherical surface at the top (FIG. 32 (c)). For convenience, FIG. 32A is a first case, FIG. 32B is a second case, and FIG. 32C is a third case.

  FIG. 33 is a diagram illustrating a case where the first case is photographed by the photographing device. FIG. 33 (a) is a view from above when shooting a product, and FIG. 33 (b) is a view from the side when shooting a product. In the case of the first case, the light source such as illumination is, for example, one light so that the photographing device is directly opposed to the product and the light illuminating the product comes from one direction. The light emitted from a light source such as a lighting device is reflected on a surface (reflection surface in a range indicated by a one-dot chain line) facing the photographing device in the product, and is incident on a focal point of the photographing device.

  Then, assuming a straight line connecting the focal point of the photographing device and the right or left side of the reflecting surface of the product and a straight line with the perpendicular line from the reflecting surface of the product to the photographing device as the axis of symmetry, outside the straight line (opposite to the photographing device) If the light source is installed at any position in the region (region X, region Y) of the (side), no reflection occurs on the product. Here, the angle formed by a straight line connecting the focal point of the photographing apparatus and the right side or the left side of the reflection surface of the product and the perpendicular, and the angle between the perpendicular line and the boundary line between the regions X and Y are the incident angles. From the relationship between the angle and the reflection angle, both are angles a. Therefore, it is only necessary to take an image from any one of the area X and the area Y (composition does not need to be performed). Alternatively, as shown in FIG. 33 (b), an area above a horizontal line from the top of the product and a straight line (a line with the horizontal line as a symmetric axis) having the same angle b as the angle b with respect to the top of the product from the photographing device is defined. A region Z is set, and a light source is set in the region Z to perform photographing. In the case of the first case, if the image is taken as shown in FIG. 33 (a) or FIG. 33 (b), the light source is not reflected on the product.

  FIG. 34 is a diagram illustrating a case where the second case is photographed by the photographing device. In the second case, as shown in FIG. 34 (a), the angle b is the same as the angle b from the horizontal line from the top of the product to the top of the product from the photographing device, as in the case of FIG. 33 (b). A region above a straight line (a line having a horizontal line as a symmetry axis) is defined as a region Z, and a light source is installed in the region Z to perform photographing. When photographing cannot be performed as shown in FIG. 34A, photographing is performed as shown in FIG. FIG. 34 (b) is a view from above when a product is photographed. In FIG. 34 (b), as in FIG. 30, the relative position between the product and the photographing device (for example, the relative position between the product photographed by the photographing device and the photographing device) is not changed. The product is photographed a plurality of times by changing the relative position between the product and a light source such as lighting (for example, the surface of the product irradiated with light by the light source and the relative position to the light source). Then, the image information photographed at each position is combined with a part without reflection to obtain sample image information.

  That is, the light emitted from the illumination is reflected by the reflection surface of the product, and is incident on the focal point of the photographing device. Therefore, when lighting is installed in the left area (area X) of the product, light emitted from the lighting causes reflection on the left side of the product, and lighting is installed in the right area (area Y) of the product. The light emitted from the illumination causes reflection on the right side of the product. Therefore, the right side is specified in the image information when the light source is located on the left side of the product, and the left side is specified in the image information when the light source is positioned on the right side of the product. Sample information.

  FIG. 35 is a diagram illustrating a case where the third case is photographed by the photographing device. FIG. 35 shows a case where the relative position between the product and the photographing device (for example, the relative position between the surface of the product photographed by the photographing device and the photographing device) is changed similarly to FIGS. 30 and 34B. Instead, the merchandise is photographed a plurality of times by changing the relative position between the merchandise and a light source such as lighting (for example, the surface of the merchandise irradiated with light by the light source and the relative position to the light source). Then, the image information photographed at each position is combined with a part without reflection to obtain sample image information.

  As described above, when photographing the specimen information of a product, a photographing method corresponding to the first to third cases may be applied according to the shape of the appearance of the product. However, since the appearance of the product generally has various shapes, the product is not classified into three cases, but is classified into a third case (that is, a photographing method similar to that of FIGS. 30, 34 (b), and 35 described above). ) Can be used for general purposes.

  By photographing the specimen information of the product by the photographing method as described above, it is possible to obtain the specimen information in which disturbance due to both the photographing conditions and the appearance of the commodity is reduced.

  As a modification of the sixth embodiment, the processing of the fifth embodiment may be used. In other words, in order to generate appropriate sample information, a plurality of pieces of image information obtained by changing the position of the light source without changing the relative positional relationship between the product and the image capturing apparatus are used. Execute the process. More specifically, of the plurality of pieces of image information obtained by photographing the product as in the fifth embodiment, all or a part of the image information (region including the product) is divided into meshes, and the average value of brightness in the mesh is calculated. . Then, the average value of the brightness is calculated for each corresponding mesh, and the image information of the mesh having the smallest value is specified. By arranging the image information of the mesh thus specified vertically and horizontally, appropriate sample information is obtained.

  The sample information generated by using the processing in the present embodiment performs the same processing as in the first to fifth embodiments to specify the product identification information of the product shown in the image information and to obtain the photographed image information. Alternatively, the image recognition system 1 may be configured to execute the process of specifying the product identification information by simply executing the image matching process on the normal image information and the sample information.

  In the above-described first to sixth embodiments, the case has been described in which the product identification information of the product displayed on the display shelf of a store or the like is specified from the image information obtained by photographing the display shelf. The present invention can be used not only for processing but also for specifying identification information of a specific target object from image information. That is, an image object (for example, an object in which a plurality of specific objects are arranged, etc.) or a specific object (an object to be identified in identification information) is photographed from a plurality of angles at the same opportunity, By comparing the similarities of the above, the present invention can be applied to the image recognition system 1 that specifies the identification information of the specific target object appearing in the image information. In that case, the processing of the first to sixth embodiments can be applied as specific processing. In this case, the processing on the display shelf may be read as the processing on the photographing target, and the processing on the product may be read as the processing on the specific target.

  By using the image recognition system 1 of the present invention, even when light or the like is reflected in image information, an object such as a product can be specified with higher accuracy than before.

1: image recognition system 2: management terminal 3: photographed image information input terminal 20: photographed image information storage unit 21: sample information storage unit 22: identification information candidate storage unit 23: identification information storage unit 24: image information processing unit 25: Identification information comparison processing unit 70: Arithmetic device 71: Storage device 72: Display device 73: Input device 74: Communication device 241: Photographed image information input reception processing unit 242: Photographed image information normalization processing unit 243: Shelf stage position identification Processing unit 244: shelf area cutout processing unit 245: face identification processing unit 246: identification information identification processing unit 247: shelf image matching processing unit 248: image information processing processing unit 2420: feature in N-1st image information Quantity sampling area 2421: Feature quantity sampling area in N-th image information

Claims (4)

Without changing the relative position between the product and the photographing device, changing the relative position between the product and the light source, cutting out a part of a plurality of pieces of image information taken and combining them into one piece of image information, A sample information storage unit for storing sample information;
By comparing a part or all of the image information showing the display shelf displaying the product with the sample information of the product stored in the sample information storage unit, the product information in the image information showing the display shelf is compared. An identification information identification processing unit that identifies product identification information;
An image recognition system comprising: Accepts input of image information photographed by illuminating each of the light sources located on the left and right sides of the photographing device without changing the relative position between the product whose horizontal cross section is convex or substantially convex and the photographing device. ,
Of the image information for which input has been received, for image information in which a light source is positioned on the right side of the imaging device, a region on the left side is cut out from the center line of the image information, and the region on the left side of the imaging device is cut out. For image information captured with the light source positioned, cut out the area to the right from the center line of the image information,
Generating the sample information of the product by synthesizing the image information of the cut-out left region and the right region into one image information;
An image recognition system, characterized in that: Without changing the relative position between the product and the photographing device, changing the relative position between the product and the light source, cutting out a part of a plurality of pieces of image information taken and combining them into one piece of image information, A computer that uses a sample information storage unit that stores sample information
By comparing a part or all of the image information showing the display shelf displaying the product with the sample information of the product stored in the sample information storage unit, the product information in the image information showing the display shelf is compared. Identification information identification processing unit for identifying product identification information,
An image recognition program characterized by functioning as a computer. On the computer,
Accepts input of image information captured by illuminating each of the light sources located on the left and right sides of the photographing device without changing the relative position between the product whose horizontal cross section is convex or substantially convex and the photographing device. function,
Of the image information for which input has been received, for image information in which a light source is positioned on the right side of the imaging device, a region on the left side is cut out from the center line of the image information, and the region on the left side of the imaging device is cut out. For image information captured with the light source positioned, a function to cut out the right area from the center line of the image information,
A function of generating the sample information of the product by combining the extracted image information of the left region and the right region with one image information;
An image recognition program characterized by realizing: