JP6746123B2 - Image recognition system - Google Patents

Description

The present invention relates to an image recognition system for accurately identifying identification information of an object such as a product shown in image information.

In the retail industry, it is known that the display status of products affects the sales of products. Therefore, as a product manufacturing company or a sales company, by grasping what kind of products of its own company or other companies are displayed in the store, it is possible to connect to the development strategy and sales strategy of its own products.

On the other hand, to realize this, it is important to accurately identify the products displayed in the store. Therefore, it is conceivable that a person manually photographs the display shelves of the store and manually identifies the displayed products from the image information. In this case, the product can be specified almost accurately. However, in order to continuously grasp the display situation of the product, it is necessary to grasp the display condition at regular intervals, but humans specify the product every time from the image information of the display shelves of the store. Is burdensome and inefficient.

Therefore, it is required to automatically identify the products displayed on the display shelf from the image information of the display shelves and grasp the display status of the products. For example, there is a method of using image recognition technology for an image of a store display shelf based on a sample image of each product. As these conventional techniques, for example, there are systems that manage the display status of products by using the techniques described in Patent Documents 1 to 3 below.

In addition to the retail industry, there are cases where it is required to photograph an object and identify the object in the photographed image information.

JP-A-5-342230 JP-A-5-334409 International publication 2012/029548

The invention of Patent Document 1 is a system that assists even a person who does not have knowledge of which display shelf to display a product. Therefore, it is not possible to grasp the products actually displayed. Patent Document 2 is a system that supports input of product images in a shelf allocation support system that supports display of products. However, the system of Patent Document 2 only supports the input of product images when using the shelving allocation support system, and even if this system is used, it is not possible to grasp the products that are actually displayed. ..

Patent Document 3 is an invention that specifies a product to be displayed in the empty space when there is an empty space in the display shelf or notifies when the product to be displayed in the display shelf is misplaced. In the present invention as well, the image matching processing between the product and the product displayed on the display shelf is performed to specify the product displayed on the display shelf. However, the recognition accuracy is low.

When the image recognition technology is used for the image of the display shelf based on the sample image of the product, the image feature amounts of the sample image and the partial image of the display part of the product in the image of the display shelf are compared. Recognize products using similarities. On the other hand, in the image of the display shelf or the sample image, when the surface of the product package reflects light, unnecessary objects such as lighting and outside light in the store and the photographer or the photographing device are located with respect to the photographing device. Depending on the relationship, it may be reflected on the surface of the product. This is a factor that affects the accuracy of image recognition because the original image feature amount cannot be obtained.

Further, the same problem applies not only to the identification of the product displayed on the display shelf but also to the identification of the general object from the image information obtained by photographing the general object. And even if each of the above-mentioned patent documents is used, it is not possible to solve the problem caused by the reflection of light.

In view of the above problems, the inventor of the present invention has invented an image recognition system that can identify identification information of a target object such as a product more accurately than before even when light or the like is reflected in the image.

The invention according to claim 1 is an image information processing unit that specifies, for each piece of image information obtained by photographing a display shelf displaying a product from a plurality of angles, a candidate for product identification information of the product shown in the image information. An identification information comparison processing unit that identifies the product identification information of the product shown in the image information by using the likelihoods of the identified product identification information candidates in the corresponding image information captured at different angles. Therefore, when there are a plurality of candidates for the identified product identification information in the corresponding image information captured at the different angles, the identification information comparison processing unit performs a predetermined calculation using the likelihood. It is an image recognition system that specifies the product identification information .

With the configuration according to the present invention, it is possible to recognize the product more accurately than before even if light or the like is reflected in the image information of the display shelf.

When the product identification information is specified from the product identification information candidates in each image information, it is preferable to specify it as in the present invention. That is, if there are cases where the product identification information candidates are specified and cases where they are not specified, the specified product identification information candidates are specified as the product identification information. Further, when there is a candidate for the product identification information, the product identification information with the highest likelihood can be specified. Further, when there are a plurality of candidates according to the likelihood, the product identification information can be specified from the value based on the likelihood.

In the above-mentioned invention, the image information processing unit is included in the processing unit that generates image information by performing the positive value conversion process on the image information of the display shelf, and the image information that is normalized. An image having an identification information specifying processing unit for specifying a candidate of the product identification information of the product shown in the region by comparing the image information of a partial region of the display shelf with the sample information of the product. It can be configured like a recognition system.

Since the image information of the display shelf is distorted or the like, the recognition accuracy increases when the positive value processing is performed and then the processing is performed. In addition, since a plurality of products are often displayed on the display shelf, it is preferable to specify the product identification information candidates by, for example, performing the comparison process with the sample information on the area for each product.

In the above-mentioned invention, the identification information specifying processing unit accepts input of the product identification information and inputs at least the product identification information when the candidate of the product identification information of the product shown in the area cannot be specified. It is possible to configure the image recognition system such that the received product identification information is specified as the product identification information of the area without performing the process in the identification information comparison processing unit for the area.

If the product identification information candidate cannot be specified, the input may be accepted. In this case, since the input product identification information is considered to have high accuracy, if the product identification information is specified (fixed) as it is without performing the comparison process, the processing time can be reduced.

According to a fourth aspect of the present invention, for each piece of image information obtained by photographing the object to be photographed from a plurality of angles, a first processing unit that identifies candidates for identification information of the specific object included in the image information, in image information corresponding taken at different angles, and a second processing unit for identifying the identification information of the specific object that is reflected in the image information using the likelihood of the candidate of the specific identification information, the second When there are a plurality of candidates for the identified identification information in the corresponding image information captured at the different angles, the processing unit of (1) performs a predetermined calculation using the likelihood to identify the image captured in the image information. It is an image recognition system that identifies the identification information of an object .

In the invention according to claim 1, the image recognition system specifies the identification information of the product displayed on the display shelf, but the invention is not limited to this. The image information obtained by shooting the shooting target from a plurality of angles. In the above, it can be applied even in the case of identifying the identification information of the specific object shown in the image information. That is, similarly to the invention described in claim 1, even if the captured image information has a reflection of light or the like, it is possible to recognize the identification information of the specific object more accurately than in the past.

The invention according to claim 1 can be realized by causing a computer to read and execute the program of the present invention. That is, for each of the image information obtained by photographing the display shelves displaying the products from a plurality of angles, the computer uses an image information processing unit that identifies candidates for the product identification information of the product shown in the image information. In the corresponding image information, the image recognition program that functions as an identification information comparison processing unit that identifies the product identification information of the product shown in the image information by using the likelihood of the identified candidate of the product identification information. Then, the identification information comparison processing unit performs a predetermined operation using the likelihood when the identified product identification information candidates are plural in corresponding image information captured at different angles. It can be configured like an image recognition program that identifies identification information .

The invention according to claim 4 can be realized by causing a computer to read and execute the program of the present invention. That is, for each of the image information obtained by shooting the imaged object from a plurality of angles, the computer takes a first processing unit that identifies candidates for the identification information of the specific object shown in the image information. An image recognition program that causes the corresponding image information to function as a second processing unit that identifies the identification information of the specific object shown in the image information by using the likelihood of the identified identification information candidate. When the plurality of candidates of the identified identification information are present in the corresponding image information captured at the different angles, the second processing unit performs a predetermined calculation using the likelihood to obtain the image information. It can be configured like an image recognition program for identifying the identification information of the specific object in the image.

By using the image recognition system of the present invention, even when light or the like is reflected in the image information, it is possible to specify an object such as a product more accurately than before.

It is a block diagram which shows typically an example of a structure of the image recognition system of this invention. It is a block diagram which shows typically an example of a structure of the image information processing part in the image recognition system of this invention. It is a block diagram which shows typically an example of the hardware constitutions of the computer used with the image recognition system of this invention. It is a flow chart which shows an example of the whole processing process in the image recognition system of the present invention. It is a flow chart which shows an example of a processing process in an image information processing part of the present invention. It is a flow chart which shows an example of the processing process in the discernment information comparison processing part of the present invention. It is a figure which shows an example of a sample information storage part typically. It is a figure which shows an example of the captured image information which image|photographed the display shelf of a store from the different angle at the same opportunity. It is a figure which shows another example of the captured image information which image|photographed the display shelf of a store from the different angle at the same opportunity. It is a figure which shows an example of the normal image information by which the keystone correction process was performed about the picked-up image information of FIG. FIG. 10 is a diagram showing an example of orthotopic image information on which the trapezoidal correction processing has been performed on the captured image information of FIG. It is a figure which shows the state by which the shelf area was specified in the picked-up image information of FIG. It is a figure which shows the state by which the shelf area was specified in the picked-up image information of FIG. It is an image figure of an example of typical processing in the image recognition system of the present invention. It is a figure which shows an example of the state which set the feature-value extraction area with respect to the captured image information of the N-1st time. It is a figure which shows an example of the state which set the feature-value extraction area with respect to the captured image information of the Nth time. It is a figure which shows the relationship of the pair of the feature-value extraction area|region of the N-1th time, and the feature-value extraction area of the N-th time. It is a figure which shows the state which projects the shelf position C in the captured image information of the N-1st time as the shelf position D in the captured image information of the Nth time by the function F. It is a figure which shows an example of captured image information. It is a figure which shows the state which set the feature-value extraction area|region in the captured image information of the N-1st time. It is a figure which shows the state which set the feature-value extraction area|region in the captured image information of the Nth time. It is a figure which shows an example of the state which specified shelf position D1 thru|or D4 in the square method by the Nth photography. It is a figure which shows typically the process of the Nth time in an identification information specific processing part. 13 is a block diagram schematically showing an example of the configuration of an image recognition system in Example 5. FIG. FIG. 14 is a block diagram schematically showing an example of the configuration of an image information processing unit in Example 5. 16 is a flowchart showing an example of the overall processing process of the image recognition system in Embodiment 5. 16 is a flowchart illustrating an example of a processing process of generating processed fixed image information according to the fifth exemplary embodiment. FIG. 6 is a diagram showing an example of a state in which the normal image information is divided by a mesh for each predetermined reference unit. It is a figure which shows typically the process which produces|generates the processed normal image information by arranging the image information of the area|region of the mesh specified by satisfy|filling a predetermined condition. It is a figure which shows an example at the time of seeing the state which image|photographs a product from above. It is a figure which shows typically the process which generate|occur|produces suitable sample information by cutting out the part which does not produce a reflection in the several image information image|photographed, and synthesize|combining it. It is a figure which shows typically an example of the external appearance of the product used as a sample. It is a figure which shows an example of the imaging method in case the external shape of a product is a 1st case. It is a figure which shows an example of the imaging method in case the external shape of a product is a 2nd case. It is a figure which shows an example of the imaging method in case the external shape of goods is a 3rd case.

An example of the system configuration of the image recognition system 1 of the present invention is shown in FIG. The image recognition system 1 uses a management terminal 2 and a captured image information input terminal 3.

The management terminal 2 is a computer used by an organization such as a company that operates the image recognition system 1. The photographed image information input terminal 3 is a terminal for inputting image information of photographed display shelves of a store.

The management terminal 2 and the captured image information input terminal 3 in the image recognition system 1 are realized by using a computer. FIG. 3 schematically shows an example of the hardware configuration of the computer. The computer is capable of inputting information, such as an arithmetic device 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 for transmitting and receiving the processing result of the arithmetic device 70 and information stored in the storage device 71 via a network such as the Internet or a LAN.

When the computer includes a touch panel display, the display device 72 and the input device 73 may be integrally configured. The touch panel display is often used in, for example, a portable communication terminal such as a tablet computer or a smartphone, but is not limited thereto.

The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that the input can be directly performed on the display with a predetermined input device (such as a pen for a touch panel) or a finger.

The photographed image information input terminal 3 may be provided with a photographing device such as a camera in addition to the above devices. As the captured image information input terminal 3, a portable communication terminal such as a mobile phone, a smart phone, or a tablet computer can be used.

The functions of the respective means in the present invention are logically distinguished from each other, and may physically or virtually form the same area. The processing order of the processing in each means of the present invention can be appropriately changed. Also, part of the processing may be omitted. For example, the orthodontic process described below can be omitted. In that case, it is possible to execute processing for image information that has not been subjected to orthonormalization processing.

The image recognition system 1 includes a captured image information storage unit 20, a sample information storage unit 21, an identification information candidate storage unit 22, an identification information storage unit 23, an image information processing unit 24, and an identification information comparison processing unit 25.

The photographed image information storage unit 20 stores the photographed image information received from the photographed image information input terminal 3, the photographing date and time, the store identification information, the image information identification information, and the like in association with each other. The captured image information may be any image information to be subjected to the trapezoidal correction process, and also includes image information that is combined as one image information when one display shelf is photographed by a plurality of sheets. .. Further, the image information after the distortion correction processing is executed is also included in the captured image information.

The sample information storage unit 21 stores sample information for identifying which product is the product of each face on the shelf of the display shelf shown in the normal image information. The sample information is image information obtained by photographing a product that may be displayed on a display shelf from a plurality of angles such as vertical, horizontal, and diagonal. FIG. 7 shows an example of the sample information stored in the sample information storage unit 21. FIG. 7 shows the case where canned beer is photographed from various angles as the sample information, but it is not limited to canned beer. The sample information storage unit 21 stores the sample information and the product identification information in association with each other.

It should be noted that the sample information storage unit 21 stores, together with the sample information or in place of the sample information, information necessary for calculating the similarity extracted from the sample information, for example, information on a pair of an image feature amount and its position. You may remember. It is assumed that the sample information also includes the information necessary for calculating the similarity. In this case, the identification information identification processing unit 246 does not have to calculate the image feature amount for each sample information when performing the matching process between the image information of the face area and the sample information, and the calculation time is shortened. can do.

The identification information candidate storage unit 22 stores the candidate of the product identification information of the product displayed on each face of the shelf of the display shelf for each captured image information. For example, in association with product identification information candidates, in association with shooting date/time information, store information, image information identification information of captured image information, image identification information of upright image information, face identification information for identifying a face. It is stored in the identification information candidate storage unit 22. The product identification information candidates stored here are stored for each captured image information. Therefore, since there are a plurality of captured image information items for the same display shelf, the corresponding face of the shelf of the same display shelf stores the candidate of the product identification information specified for each captured image information item. ..

The identification information storage unit 23 stores the identified (confirmed) product identification information of the product displayed on each face of the shelf of the display shelf. For example, in association with product identification information, shooting date/time information, store information, image information identification information of captured image information, image identification information of orthotopic image information, face identification information for identifying a face, identification information. It is stored in the storage unit 23.

In the captured image information received from the captured image information input terminal 3, the image information processing unit 24 identifies a candidate for the product identification information of the product displayed on the shelf in the display shelf shown in the captured image information. To execute.

The image information processing unit 24 distinguishes the photographed image information input reception processing unit 241, the photographed image information normalization processing unit 242, the shelf position specifying processing unit 243, the shelf region cutting processing unit 244, and the face specifying processing unit 245. It has an information identification processing unit 246 and a shelf image matching processing unit 247.

The captured image information input reception processing unit 241 receives input of image information (captured image information) of the display shelf of the store captured by the captured image information input terminal 3, and stores it in the captured image information storage unit 20. From the photographed image information input terminal 3, in addition to the photographed image information, it is preferable to accept input together with photographed date and time, store identification information such as a store name, image information identification information for identifying image information, and the like. 8 and 9 show examples of photographed image information. In FIG. 8 and FIG. 9, there are three shelves on the display shelves, and the photographed image information in which products are displayed there.

The captured image information normalization processing unit 242 performs trapezoidal correction processing on the captured image information stored in the captured image information storage unit 20 to generate normal image information, which is orthonormalized. The trapezoidal correction process is a correction process performed so that the shelves of the display shelves shown in the photographed image information are horizontal and the product tags for the products displayed there are vertical. Orthogonalization is to transform the image information so that the optical axis of the lens of the image capturing device is the same as when the image is captured from a sufficient distance along the perpendicular direction of the plane to be imaged. There is a correction process.

The trapezoidal correction processing executed by the captured image information orthotopic processing unit 242 receives an input of designation of four vertices in the captured image information, and executes the trapezoidal correction processing using each of the vertices. The four vertices for receiving the designation may be the four vertices of the shelves of the display rack or the four vertices of the rack position of the display rack. Further, it may be four apexes of a set of two or three shelves. Any four points can be designated as the four vertices. 10 and 11 show an example of photographed image information (orthogonal image information) that has been orthostatically processed.

The shelf position specifying processing unit 243 specifies the shelf area (shelf area) from the captured image information or the normal image information. That is, the display shelf shown in the photographed image information and the normal image information has a shelf area where products are displayed and a product tag arrangement area where product tags for the products displayed are attached. .. Therefore, the shelf area is specified from the fixed image information. To specify the shelf area, the operator of the management terminal 2 may manually specify the shelf area and the shelf position specifying processing unit 243 may accept it, or the shelf for which the input was manually accepted for the first time. The shelf area may be automatically specified after the second time based on the area information. 12 and 13 show a state in which the shelf area is specified.

The shelf area cutout processing unit 244 cuts out the image information of the shelf area specified by the shelf position specification processing unit 243 as shelf area image information. The shelf area cutout processing unit 244 may actually cut out the image information, or may actually cut out the image information by specifying the coordinates forming the shelf area without cutting out the image information. May be When there are a plurality of shelves on the display shelf, each of them is cut out as shelf area image information. Further, the coordinates indicating the area of the shelf are the coordinates of the vertices necessary for specifying the area, and are, for example, the coordinates of four points in the normal image information, the coordinates of the upper right and lower left, and the upper left and lower right. Good. Further, the relative coordinates are based on a predetermined position in the image information (for example, the upper left apex of the display shelf) such as a display shelf in the normal image information.

The face identification processing unit 245 identifies the face on which the product is placed (the region in which the product is placed) for each shelf in the shelf area in the normal image information. The face identification processing unit 245 is divided into a first face identification process and a second and subsequent face identification process.

In the first face specifying processing in the face specifying processing unit 245, the product is placed within the area (preferably rectangular area) constituted by the coordinates of the shelf specified by the shelf position specifying processing unit 243. Specify the area (face). Specifically, it is based on constraints such as identifying thin and narrow shadows that occur between products, identifying repeating patterns of images, identifying steps on the top side of the package, and having the same product width. The face area is specified by specifying the position. As the face identifying process, any method can be adopted depending on the category of the product and the form of the product, and is not limited to the above. Further, correction input by a person in charge may be accepted for the automatically identified face. Further, the input of the face position may be accepted from the person in charge. The coordinates of the area forming the identified face are the shooting date/time information, the store information, the image information identification information of the captured image information, the image identification information of the orthostatic image information, and the face at the coordinates of the area of the face in the regular image information. The face identification information for identification is associated and managed. The coordinates indicating the face area are the coordinates of the vertices necessary for specifying the rectangular area, and may be, for example, the coordinates of four points in the orthonormal image information, the upper right and lower left, and the upper left and lower right. .. Further, the relative coordinates are based on a predetermined position in the image information (for example, the upper left apex of the display shelf) such as a display shelf in the normal image information.

In the second and subsequent face identification processes in the face identification processing unit 245, the coordinates of the face area identified by the previous (N-1)th fixed image information for the same shelf on the same display shelf are set this time (( The coordinates of the face area specified by the Nth (normal) image information may be used.

The identification information identification processing unit 246 identifies a candidate for the item identification information of the item displayed on the face for each shelf of the display shelf. The merchandise identification information includes, but is not limited to, the merchandise name and the JAN code assigned to the merchandise. Any information may be used as long as it can identify the product.

The identification information identification processing unit 246 executes the following processing. That is, by matching the image information of the face with the sample information of the product stored in the sample information storage unit 21 for each face, the candidate of the product identification information of the product displayed on the face is specified. Specifically, first, the similarity between the image information of the area formed by the coordinates of the face to be processed and the sample information stored in the sample information storage unit 21 is determined, and the sample information having the highest similarity is determined. The product identification information corresponding to is specified, and is specified as a candidate for the product identification information of the product displayed on the face composed of the coordinates.

Here, in order to determine the similarity between the face image information and the sample information, the following processing is performed. First, in the processing up to the identification processing of the product identification information candidates in the identification information identification processing unit 246, the image information of the face area in the shelf of the normal image information and the sample information have the same direction (rollover or rollover). It is not inverted, and the size of each image information is almost the same (when the size of the image information is different in a predetermined range or more, before the similarity determination, The size is adjusted so that the size of the image information is within a predetermined range).

The identification information identification processing unit 246 determines the similarity between the face image information and the sample information, and therefore, the image of the feature information based on the image feature amount (for example, local feature amount) of the face image information and the image of the sample information. Feature points based on the feature amount (for example, local feature amount) are extracted. Then, the pair having the highest similarity between the feature points of the image information of the face and the feature points of the sample information is detected, and the difference between the coordinates of the corresponding points is found. Then, the average value of the differences is obtained. The average value of the differences indicates the total average movement amount between the face image information and the sample information. Then, the coordinate difference of all pairs of feature points is compared with the average coordinate difference, and the pairs having a large deviation degree are excluded. Then, the similarities are ranked according to the number of remaining corresponding points.

The similarity between the face image information and the sample information can be calculated by the above method. Further, in order to improve the accuracy, the EMD (Earth Moves Distance) between the color histograms may be obtained and used as a measure of similarity. As a result, it is possible to perform comparative comparison of similarities relatively strong to environmental changes such as brightness information of photographed image information, and to specify with high accuracy.

In addition, as the determination of the similarity, the EMD between the signatures (the set of the image feature amount and the weight) of the image information of the area of each face may be obtained and used as the similarity measure. As the image feature amount of the signature, for example, the frequency distribution of the image information of the face region in the HSV color space is obtained, and the hue and saturation are grouped to determine the number of features and the image feature amount of the region in the HSV color space. can do. The reason for grouping the hue and the saturation is to reduce the dependence on the lightness so as not to be largely influenced by the shooting conditions.

Further, in order to speed up the processing, instead of the signature and the EMD, the similarity such as the L2 distance between the image feature amounts such as the color correlogram and the color histogram of the image information in an appropriate color space can be used.

The determination of similarity is not limited to the above. The identified product identification information candidates are identification information candidates in association with shooting date/time information, store information, image information identification information of captured image information, image identification information of upright image information, and face identification information for identifying a face. It is stored in the storage unit 22. The candidate of the product identification information specified here is not the confirmed product identification information because it is the candidate of the product identification information of the product shown on the face in the normal image information based on a certain captured image information.

The candidate product identification information identified as described above corresponds to the shooting date/time information, the shop information, the image information identification information of the shot image information, the image identification information of the upright image information, and the face identification information for identifying the face. It is also stored in the identification information candidate storage unit 22.

The shelf image matching processing unit 247 stores the image information of the shelf area in the previous (N−1)th normal image information and the image information of the shelf area in the current (Nth) normal image information. If the similarity is high, the product identification information candidates of each face in the shelf are determined to be the same. As described above, this similarity determination processing is performed on the image feature amount of the image information of the shelf area in the previous (N-1)th normal image information and the current (Nth) normal image information. Similarity determination may be performed based on the image information of the shelf area, or EMD between color histograms may be used. Moreover, it is not limited to them. Then, instead of the identification processing for each face in the identification information identification processing unit 246, the identification information identification processing unit 246 sends N-th candidate product identification information of each face in the shelf in the normal image information. It is stored in the identification information candidate storage unit 22 as the same as the candidate of the product identification information of each face in the same first shelf. As a result, it is possible to omit processing for a shelf that hardly changes, such as a shelf that does not move much or a shelf that is managed in an extremely short cycle. Note that the processing by the shelf image matching processing unit 247 may not be provided.

The processing in the shelf image matching processing unit 247 is used not only in the Nth comparison and the N-1th comparison, but also in the Nth comparison of the normal image information obtained by photographing the same display shelf at different angles. You may use.

Based on the product identification information candidates identified by the identification information identification processing unit 246, the identification information comparison processing unit 25 uses the captured image information (normal image information) obtained by capturing the same display shelf at different angles on the same occasion. A process of comparing the product identification information candidates of the corresponding faces and identifying (determining) the product identification information of the product shown in the face is executed. An image diagram of the processing of the identification information comparison processing unit 25 is shown in FIG.

The identification information comparison processing unit 25 displays the product identification information for each face identified by the image information processing unit 24 in the normal image information based on different photographed image information photographed on the same display shelf at the same opportunity, for example, on the same day. Each candidate is specified from the identification information candidate storage unit 22. For example, each candidate of product identification information corresponding to the same face identification information is specified. Then, with respect to each candidate of the product identification information, the product identification information corresponding to the face is specified (determined) by comparing with a predetermined condition. For example, as shown in FIG. 14, it is assumed that there are two pieces of photographed image information (photographed image information 1 and photographed image information 2) that are photographed at the same opportunity on the same display shelf but at different angles. At this time, the face A in the normal image information 1 obtained by normalizing the photographed image information 1 is associated with the product identification information 1 as a candidate for the product identification information, and the normal image obtained by normalizing the photographed image information 2 is arranged. It is assumed that the face A in the image information 2 has no candidate for the product identification information (the product identification information of the face A could not be specified based on the normal image information 2). In this case, the product identification information 1 based on the normal image information 1 is specified as the product identification information of the face A. Further, as another example, the face A in the orthonormal image information 1 obtained by orthonormalizing the photographed image information 1 is associated with the product identification information 1 as a candidate of the product identification information with a likelihood of 40%. If the product identification information 2 is associated with the likelihood A of 90% as a candidate of the product identification information to the face A in the orthotopic image information 2 obtained by normalizing the information 2, the product identification information with the highest likelihood. 2 is specified as the product identification information of face A.

In this way, the identification information comparison processing unit 25 identifies the identified product identification information candidates in the image information of different angles obtained by photographing the same display shelf at the same opportunity, whether or not there is a product identification information candidate, and product identification information. Based on the likelihood of the candidate, the product identification information corresponding to the face is specified. When one face in one image information has a plurality of product identification information candidates according to the likelihood, the likelihoods may be calculated respectively. For example, product identification information 1 is associated with a likelihood of 65% and product identification information 2 is associated with a likelihood of 30% as a face of the face A in the orthotopic image information 1 obtained by orientating the photographed image information 1. Therefore, the face A in the normal image information 2 obtained by normalizing the photographed image information 2 is associated with the product identification information 2 as a candidate of the product identification information with a likelihood of 70% and the product identification information 1 with a likelihood of 25%. In the case of the same product identification information, the values based on the likelihood are added to each other, a calculation such as multiplication is performed to calculate a value, and one of the product identification information is specified based on the value. Good. For example, in the above case, the product identification information 1 may be 90 and the product identification information 2 may be 100, and the product identification information 2 may be specified. In this way, the identification information comparison processing unit 25 compares the candidates of the product identification information specified from the plurality of image information at different angles of the display shelf, which is the same image capturing object, at the same opportunity under predetermined conditions. The product identification information corresponding to the face is specified. It should be noted that the accuracy is improved by executing the process for each face, but the present invention is not limited to this, and it is only necessary to compare the candidates of the product identification information.

Note that the image recognition system 1 in the present specification can be applied to the process of specifying the product identification information of the product displayed on the image information of the product displayed on the display shelf, but is not limited thereto. It can also be applied to the case where the identification information of an object is specified based on the image information of the object, not the object.

A processing process using the image recognition system 1 of the present invention will be described with reference to the block diagrams of FIGS. 1 and 2 and the flowcharts of FIGS. 4 to 6. It should be noted that in the following description, the case where two pieces of photographed image information photographed from two directions are used for the same display shelf will be described, but the same processing can be executed even when there are three or more pieces.

The person in charge of shooting the display shelves of the store shoots the same display shelf from a plurality of angles at the same opportunity so that the reflection of light or the like does not overlap. Then, the photographed image information (FIGS. 8 and 9) photographed from different angles at the same opportunity on the display shelf of the store is input from the photographed image information input terminal 3 and photographed by the image information processing unit 24 of the management terminal 2. The image information input reception processing unit 241 receives the input (S100, S110). Further, the input of image information identification information such as shooting date/time, store identification information, and shot image information is accepted. Then, the photographed image information input reception processing unit 241 stores the photographed image information, the photographing date and time, the store identification information, and the image information identification information of the photographed image information, which have been input, in the photographed image information storage unit 20 in association with each other.

When the management terminal 2 receives a predetermined operation input, the photographed image information normalization processing unit 242 extracts the photographed image information stored in the photographed image information storage unit 20 and performs a trapezoidal correction process on a shelf. The four inputs of the position (position of the display shelf) are accepted, and the keystone correction process is executed (S120). 10 and 11 show an example of the captured image information (normal image information) on which the trapezoidal correction process is executed in this manner.

Then, by receiving a predetermined operation input in the management terminal 2 with respect to the fixed image information, the shelf position specifying processing unit 243 specifies the shelf area (S130). That is, the input of the shelf area in the normal image information is accepted. 12 and 13 are diagrams showing a state in which the shelf area is specified.

When the shelf area is specified as described above, the shelf area cut-out processing unit 244 cuts out the image information of the shelf area from the normal image information based on the shelf area received in S130. (S140). Then, a process for identifying a face is executed for each shelf in the shelf area image information (S150). Specifically, for a shelf in the shelf area, a repeating pattern of images is specified that specifies a thin and narrow shadow that occurs between products within a rectangular area formed by the coordinates of four points. The face is specified by specifying the step on the upper side of the package, specifying the separation position based on the constraint that the product width is the same, and the like. Face identification information for identifying the face is attached to the identified face. The face identification information is the same face identification information for the image information obtained by photographing the same display shelf even if the image information is different, so that it can be identified as the corresponding face. The correspondence of the identification information is stored.

The identified coordinates of each face are stored in association with the shooting date and time, the store identification information, the image information identification information of the captured image information, the image information identification information of the normal image information, and the face identification information for identifying the face. Let It should be noted that the face coordinates may be two points that can specify a rectangular area without storing four points.

When each face of each shelf in the shelf area image information of the normal image information is specified as described above, the identification information specifying processing unit 246 performs a matching process with the sample information stored in the sample information storage unit 21 for each face. Then, the product identification information corresponding to the sample information having the highest similarity is specified. Then, it is specified as a candidate for the product identification information of the product displayed on the face (S160). That is, each image feature amount is calculated from the image information in the rectangular area of the face of a certain shelf (X is the face identification information of the face in this area) and each piece of sample information stored in the sample information storage unit 21. Then, the similarity is judged by finding a pair of feature points. Then, the sample information having the highest similarity is specified, and if the similarity at that time is equal to or more than a predetermined threshold value, the product identification information corresponding to the sample information is specified based on the sample information storage unit 21. Then, the identified product identification information is set as a candidate for the product identification information of the product displayed on the face of the face identification information X. Then, the identification information identification processing unit 246 associates the identified product identification information candidates with the shooting date and time, the store identification information, the image information identification information of the shot image information, the image information identification information of the upright image information, and the face identification information. And stores it in the identification information candidate storage unit 22 (S170).

Note that it is not always possible to specify the product identification information for all faces. Therefore, for faces that have not been identified (such as faces whose highest similarity is less than a predetermined threshold value), input of product identification information is accepted, and the product identification information for which the input is accepted is taken as the shooting date, store identification information, and taken image. The image information identification information of information, the image information identification information of orthostatic image information, and the face identification information may be associated with each other and stored in the identification information candidate storage unit 22 as a candidate of the product identification information. Further, the input of the correction processing of the identified product identification information candidate may be similarly received. Since such a face has a high degree of certainty from the beginning, it is decided as the product identification information without executing the process of the identification information comparison processing unit 25 described later, and the identification information candidate storage unit 22 and the identification information storage unit are executed. 23 may be stored. At this time, a flag indicating that the manual input is accepted can be added to the identification information candidate storage unit 22, and if there is such a flag, the processing of the identification information comparison processing unit 25 can be skipped.

By performing the above-described processing, it is possible to specify the product identification information candidate of the product displayed on the shelf of the display shelf shown in the photographed image information.

When the product identification information candidate can be specified for the face in the normal image information based on each captured image information, the identification information comparison processing unit 25 determines the product identification information candidates of the corresponding faces as the identification information candidate storage unit 22. The product identification information of the face is identified by comparing the above items based on a predetermined condition (S200).

Specifically, first, the identification information comparison processing unit 25 identifies a corresponding face in a plurality of pieces of normal image information having different angles taken on the same display shelf at the same opportunity (S210). For identifying the corresponding face, there are methods such as identifying based on the same face identification information and identifying a face associated as a face.

Then, the product identification information candidates for the identified face are extracted from the identification information candidate storage unit 22, and the product identification information candidates for the corresponding faces are compared with each other to determine whether a predetermined condition is satisfied (S220). (S230). Then, the identified product identification information is stored in the identification information storage unit 23 (S240).

By performing the above-described processing, it is possible to specify the product identification information of the product displayed on the shelf of the display shelf shown in the photographed image information. Further, since the identification is performed based on the captured image information captured at a plurality of angles, it becomes easy to eliminate the influence of the reflection of light or the like, and the accuracy of identifying the product can be improved.

In the second or subsequent processing, the processing in the shelf image matching processing unit 247 can be appropriately executed.

In the above-described first embodiment, the trapezoidal correction process is executed by designating four points, but it is a heavy burden to designate and input the reference vertices every time. Therefore, it may be configured to automatically identify the apex serving as the reference of the trapezoidal correction process. The processing in this case will be described.

In this case, the captured image information normalization processing unit 242 is divided into a first trapezoidal correction process and a second and subsequent trapezoidal correction processes. In addition to the first time, it also includes the case of performing the operation manually at an arbitrary timing in order to correct the deviation when the vertices are automatically specified. The second and subsequent times are other than the first time. Further, in the present invention, the same display shelf is photographed from a plurality of angles at the same opportunity. Therefore, the trapezoidal correction process is executed by designating the vertices for the first sheet (N-1), and the vertices that are the reference for the trapezoidal correction process are automatically specified for the second and subsequent sheets (Nth). You may do so.

In the first trapezoidal correction process in the captured image information normalization processing unit 242, as in the first embodiment, the input of designation of the four vertices of the rectangular area of the display shelf is accepted. The four vertices of the rectangular area of the display shelf may be the four vertices of the shelf position of the display shelf, the four vertices of the shelf, or the four vertices of the area to which the product tag is attached. Further, it may be four apexes of a set of two or three shelves. The four vertices for which the designation has been accepted here are stored in association with the image date/time information, the store information, and the image information identification information of the captured image information. Then, the captured image information normalization processing unit 242 performs trapezoidal correction processing on the captured image information based on the coordinates of the four vertices for which designation has been received, and obtains the normal image information.

The captured image information normalization processing unit 242 executes the trapezoidal correction processing after the second time as follows.

First, the captured image information normalization processing unit 242 determines the vertex coordinates of the (N-1)th captured image information obtained by capturing the same (substantially the same) region corresponding to the Nth captured image information, in the previous processing. Identify from stored information. The vertex coordinates of the (N-1)th captured image information obtained by capturing the same (almost the same) area corresponding to the Nth captured image information are stored in the store identification information, the image identification information, the shooting date/time information, etc. Identify based on Then, with respect to the (N-1)th captured image information, a rectangular area having a predetermined size including the identified vertex coordinates of the four vertices, for example, a square having a width of about 1/5 of the shelf is used as the feature amount collection area 2420. Set as. FIG. 15 shows an example of a state in which the feature amount collection area 2420 is set for the (N-1)th captured image information. The feature amount collection area 2420 may be a rectangular area including vertex coordinates. On the other hand, if the backgrounds of the display shelves are matched with each other, the backgrounds are greatly shifted even if the shooting position is slightly shifted. Therefore, it is preferable to set the feature amount collection area 2420 to a position that includes as much the inside of the display shelf as possible. That is, it is preferable that the vertex coordinates are located in the feature amount collection area 2420 in a direction outward of the display shelf with respect to the center point of the feature amount collection area 2420.

For example, four vertex coordinates are located at the upper left, upper right, lower left, and lower right. Then, when the rectangular area of the feature amount collection area 2420 is divided into a total of four areas by dividing the rectangular area in the vertical and horizontal centers into two areas, in the feature quantity collection area 2420 including the upper left vertex coordinates, the vertex coordinates of the upper left area The feature amount collection area 2420 is set so as to be located at. Similarly, in the feature amount collection area 2420 including the upper right vertex coordinates, the feature amount collection area 2420 is set so that the vertex coordinates are located in the upper right area of the rectangular region, and the feature amount collection area including the lower left vertex coordinates is set. In the area 2420, the feature amount sampling area 2420 is set so that its vertex coordinates are located in the lower left area of the rectangular area, and in the feature amount collecting area 2420 including the lower right vertex coordinates, the vertex coordinates are in the rectangular area. The feature amount collection area 2420 is set so as to be located in the lower right area. As a result, the vertex coordinates are located in the feature amount collection area 2420 in a direction outward of the display shelf from the center point of the feature amount collection area 2420.

Next, the captured image information orthotopicization processing unit 242 includes the feature amount collection area 2420 set in the N-1th captured image information in the Nth captured image information, and the N-1th captured image information. A feature amount collection area 2421 having a size equal to or larger than the feature amount collection area 2420 is set. The feature amount collection area 2421 set in the Nth captured image information does not exceed the size of 1/2 of the short side. Further, when it goes outside the photographed image information, the range is trimmed. FIG. 16 shows an example of a state in which the feature amount collection area 2421 is set for the Nth captured image information.

Then, the captured image information normalization processing unit 242 collects the local feature amount in each feature amount collection area 2420 set for the (N-1)th captured image information, and extracts the feature point and the coordinates of the feature point by the local feature amount. Remember the set and. In addition, the local feature quantity is collected in each feature quantity collection area 2421 set for the Nth captured image information, and the feature point and the set of coordinates of the local feature quantity are stored.

The captured image information normalization processing unit 242 corresponds to the local feature amount of the feature point in the feature amount collection area 2420 of the N−1th shot image information and the feature amount collection area 2420 of the N−1th shot image information. The local feature amount of the feature point in the feature amount sampling area 2421 of the Nth captured image information at the position is compared. Then, the feature point of each local feature amount of the N-th captured image information that is closest to each local feature amount of each feature point of the N-1th captured image information is specified. Then, the feature points of the closest local feature quantities are paired, and the coordinates of the feature points according to the paired local feature quantities are associated. At this time, pairs in which the closeness (similarity) between the local feature quantities is less than a predetermined threshold value are excluded. As a result, it is possible to specify a pair of local feature amount feature points in the (N-1)th feature amount collection region 2420 and closest feature points of the local feature amount in the Nth feature amount collection region 2421. FIG. 17 shows a pair relationship between the feature points of the local feature quantity of the N-1th feature quantity collection area 2420 and the feature points of the local feature quantity of the Nth feature quantity collection area 2421. In FIG. 17, the point cloud of feature points by the local feature quantity in the N-1th feature quantity collection area 2420 is A, the point cloud of feature points by the local feature quantity in the Nth feature quantity collection area 2421 is B, N−. The vertex used in the first trapezoidal correction process is indicated by C.

The coordinates of the point group A of the feature points by the local feature amount in the (N-1)th feature amount sampling area 2420 and the point group B of the feature points by the local feature amount in the Nth feature amount sampling area 2421 corresponding to the point group A. Then, a function F (perspective transformation/homography) for projecting the point group A onto the point group B is obtained based on the coordinates of. The function F includes, but is not limited to, methods such as repeating sampling estimation and using RANSAC of OpenCV which is a kind of robust estimation. It should be noted that pairs that have a large deviation from the relational line having a projective relationship are excluded from the processing target.

After obtaining the function F in the captured image information normalization processing unit 242, the captured image information normalization processing unit 242 calculates the coordinates of the vertex C used in the (N-1)th trapezoidal correction processing based on the function F. It is projected onto the Nth captured image information, and is specified as the coordinates of the vertex D for the Nth trapezoidal correction process. FIG. 18 schematically shows this.

By performing the above processing on each of the feature amount collection areas 2420 and 2421, the four vertices of the shelf position for the trapezoidal correction processing in the Nth captured image information are specified. Then, the captured image information normalization processing unit 242 performs trapezoidal correction processing on the Nth captured image information based on the identified four vertices to perform normalization, and generates and stores normal image information. At this time, the captured image information normalization processing unit 242 is associated with the normal image information, and is associated with the shooting date/time information, store information, image information identification information of the captured image information, and image identification information of the normal image information. Let them remember. The identified coordinates of the vertex corresponding to the Nth captured image information are stored in association with the image date/time information, the store information, and the image information identification information of the captured image information.

The apex identification process used in the trapezoidal correction process in the captured image information orthotopic processing unit 242 is not limited to the case in which the product is identified from the image information of the display shelf image as in the present invention, and the same image capturing target object is used. It can also be applied to the image recognition system 1 that orients a plurality of image information of captured images to generate orthotopic image information. As a result, a plurality of pieces of image information of the same imaged object can be orthonormalized to generate orthotopic image information of the imaged object.

Further, in order to identify the four vertices of the shelf position for the trapezoidal correction processing in the Nth captured image information, in the above description, the feature points based on the local feature amount in the feature amount collection area 2420 in the N−1th captured image information. Of the feature point by the local feature amount in the feature amount sampling area 2421 in the Nth captured image information and the function F is obtained and used in the N-1th trapezoidal correction process. The process of projecting the coordinates of the vertex C onto the captured image information of the Nth time by the function F and specifying the coordinates of the vertex D for the trapezoidal correction process of the Nth time has been described. However, in such processing, it is sufficient to find the coordinates (position) of the corresponding points of similar image information in the (N-1)th captured image information and the Nth captured image information. However, any feature amount may be used as long as it is a type of feature amount that specifies a location in the image information. For example, there are sharp points and highlight points in the image information. In this specification, a feature amount such as a local feature amount that specifies a location in image information is referred to as an image feature amount (position specific image feature amount). In the description of this specification, the case where the local feature amount is used as the image feature amount as described above will be described.

Next, the vertex specifying process for performing the trapezoidal correction process according to the second embodiment will be described. In this case, the apex for performing trapezoidal correction processing has already been specified in the captured image information obtained by capturing an arbitrary display shelf, and the captured image information captured at different angles on the same display shelf at the same opportunity Will be explained.

The captured image information of the display shelves of the store is input from the captured image information input terminal 3, and the captured image information input reception processing unit 241 of the management terminal 2 receives the input. FIG. 19 shows an example of photographed image information. Further, the input of image information identification information such as shooting date/time, store identification information, and shot image information is accepted. Then, the photographed image information input reception processing unit 241 stores the photographed image information, the photographing date and time, the store identification information, and the image information identification information, which have been input, in the photographed image information storage unit 20 in association with each other.

When the management terminal 2 receives a predetermined operation input, the photographed image information normalization processing unit 242 extracts the photographed image information stored in the photographed image information storage unit 20 and determines the rack position for executing the keystone correction process. The process of identifying the vertex D (D1 to D4) is executed.

The vertex coordinates (eg, vertex coordinates C1 to C4) of the previous (N-1)th captured image information (FIG. 20) of capturing the same or almost the same area corresponding to the captured image information of this time (Nth) (FIG. 19). Specify). The vertex coordinates of the last captured image information may be specified based on the capturing date and time, the store identification information, the image identification information of the captured image information, and the like.

The captured image information normalization processing unit 242 extracts the captured image information of the (N-1)th time from the captured image information storage unit 20, and for each of the vertices C1 to C4, a rectangle of a predetermined size including one vertex. The region is set as the feature amount collection region 2420 and set in the (N-1)th captured image information. FIG. 20 shows a state in which the feature amount collection area 2420 is set in the (N-1)th captured image information.

Further, the photographed image information normalization processing unit 242 extracts the Nth photographed image information (FIG. 19) from the photographed image information storage unit 20 and features in a wider range than the N-1th feature amount collection area 2420. The volume sampling area 2421 is set to the Nth captured image information. FIG. 21 shows a state in which the feature amount collection area 2421 is set in the Nth captured image information. Each feature amount collection area 2421 in the Nth captured image information includes one N-1th feature amount collection area 2420. In FIG. 21, the N-th feature amount collection area 2421 is shown as the (N-1)th feature amount collection region 2420 to show the inclusion relation.

Then, the captured image information normalization processing unit 242 collects the local feature amount in each feature amount collection area 2420 set for the (N-1)th captured image information, and sets a set of feature points and coordinates according to the local feature amount. Memorize In addition, local feature quantities are collected in each feature quantity collection area 2421 set for the Nth captured image information, and a set of feature points and coordinates based on the local feature quantities is stored.

The captured image information orthotopicization processing unit 242 uses the feature amount collection area 2421 of the Nth captured image information closest to each local feature amount of each feature point in the feature amount collection area 2420 of the N−1th captured image information. The feature points of the local feature quantity at are identified, and the coordinates are associated with each other as the feature point of the local feature quantity forming a pair. FIG. 17 shows a pair relationship between the (N-1)th N-1th feature amount collection area 2420 and the Nth feature amount collection area 2421.

Then, the point cloud of feature points by the local feature quantity in the N-1th feature quantity collection area 2420 is A, the point cloud of feature points by the local feature quantity in the Nth feature quantity collection area 2421 is B, and the N-1th time. Assuming that the vertices used for the trapezoidal correction processing of C are C (C1 to C4), the captured image information orthotopic processing unit 242 determines the point group A to the point group B based on the coordinates of the point group A and the point group B. A function F (perspective transformation/homography) that is projected onto is obtained.

Then, the captured image information normalization processing unit 242 projects the coordinates of the vertices C (C1 to C4) used in the (N-1)th keystone correction processing based on the obtained function F, and performs the Nth keystone correction processing. Is specified as the coordinates of the vertex D (D1 to D4).

By performing the above processing on each of the feature amount collection areas 2420 and 2421, the four vertices D (D1 to D4) for the Nth trapezoidal correction processing can be automatically specified. The specified coordinates of the N-th vertex D (D1 to D4) are stored in association with the shooting date/time, the store identification information, and the image information identification information of the shot image information. FIG. 22 shows the identified vertices D1 to D4.

As described above, when the vertices D (D1 to D4) of the shelf position for the trapezoidal correction processing for the Nth captured image information are specified, the captured image information normalization processing unit 24222 determines that the vertices D (D1 to D4). ), the trapezoidal correction process is performed on the Nth captured image information.

By performing the above processing, it becomes possible to automatically identify the corresponding vertices for the second and subsequent trapezoidal correction processes without specifying the four vertices used in the trapezoidal correction process. The burden on the person can be reduced.

Further, a modified example of the face identification processing in the face identification processing unit 245 of the first embodiment will be described. In the present embodiment, the face identifying process of the first embodiment may be performed as the first process, and the face identifying process may be automatically performed as the second and subsequent face identifying processes. The processing in this case will be described.

In addition to the first time, the case of performing the process of the first embodiment at an arbitrary timing is also included in order to correct the deviation in the automatic identification. The second and subsequent times are other than the first time. In addition, the second and subsequent times also include the case where the processing is performed on the captured image information and the normal image information obtained by capturing the same display shelf at the same opportunity on the same occasion. ..

The face identification processing unit 245 executes the same processing as that of the first embodiment as the initial face identification processing. Then, in the second and subsequent face identification processes in the face identification processing unit 245, the coordinates of the face area identified by the previous (N-1)th fixed image information are set for the same shelf of the same display shelf. The extracted coordinates are used as the coordinates of the area of the face specified by the current (N-th) orthographic image information.

The coordinates of the face area are, like the coordinates of the positions of the shelves, relative coordinates based on a predetermined position in the display shelf (for example, the top left apex C1 of the display shelf) in the normal image information.

Further, as a modified example of the first embodiment, the processing of the first embodiment is performed as the processing in the identification information specifying processing unit 246 for specifying the product identification information of the product displayed on the face for each shelf of the display shelf. The following process is executed as the product identification information identification process and the product identification information identification process from the second time onward.

The identification information identification processing unit 246 first identifies the face identification information of the face to be processed in the identification process of the product identification information of the face in the Nth normal image information. Let X be the identified face identification information. Then, the image information of the area of the face identification information X in the Nth normal image information is compared with the image information of the area in the N-1th normal image information at the position corresponding to the face identification information X. To determine the similarity, in addition to the processing of the above-described first to third embodiments, the EMD between the signatures (the set of image feature amounts and weights) of the image information of the area of each face is obtained, and the similarity measure is obtained. can do. As the image feature amount of the signature, for example, the frequency distribution of the image information of the face region in the HSV color space is obtained, and the hue and saturation are grouped to determine the number of features and the image feature amount of the region in the HSV color space. can do. The reason for grouping the hue and the saturation is to reduce the dependence on the lightness so as not to be largely influenced by the shooting conditions.

Further, in order to speed up the processing, instead of the signature and the EMD, the similarity such as the L2 distance between the image feature amounts such as the color correlogram and the color histogram of the image information in an appropriate color space can be used.

If the similarity of the image information of the face area is equal to or more than a certain threshold value, the product identification information candidates corresponding to the face of the area in the N-1th fixed image information are extracted from the identification information candidate storage unit 22, The face identification information X in the Nth normal image information is used as a candidate for the product identification information. As a result, it is possible to identify a candidate for the product identification information of the face identification information X in the Nth normal image information to be processed. If the similarity is less than a certain threshold value, as in the case of the first time, the image information of the area of the face identification information X in the Nth normal image information and the sample information stored in the sample information storage unit 21 are obtained. By comparison, the product identification information whose similarity is greater than or equal to a predetermined threshold value and which has the highest similarity is specified as a candidate for the product identification information of the face of the face identification information X in the Nth normal image information.

In comparison with the image information of the area of the face in the N-1th normal position image information, not only the comparison with the position of the corresponding face but also a face within a predetermined range may be included as a comparison target. For example, when comparing with the image information of the area of the face identification information X in the Nth normal position image information, the comparison target is the area of the face identification information X in the N-1th normal position image information, and from that area. Areas of faces within a predetermined range, for example, faces at positions separated by one or more in the left-right direction, and face areas at upper and lower shelves may be included. Further, in addition to the area of the face identification information X in the N-1th fixed image information, a plurality of adjacent face areas such as face identification information X-2, X-1, X, X+1, X+2 are included. May be.

In this case, the image information of the area of the face identification information X in the Nth normal image information, and the image information of each area of the face range of the comparison target in the N-1th normal image information. Are compared, and the face identification information of the N-1th orthotopic image information having the highest similarity is specified. The similarity may be a condition that it is equal to or more than a certain threshold. Then, a product identification information candidate corresponding to the identified face identification information of the N-1th normal image information is extracted from the identification information candidate storage unit 22, and the product identification of the face identification information X in the Nth normal image information is performed. Make it a candidate for information. FIG. 23 schematically shows this processing. FIG. 23A shows the last (N-1)th fixed image information, and FIG. 23B shows the current (Nth) fixed image information. Then, the image information of the area of each face of the shelf 1 of the Nth normal image information is compared with the image information of the area of each face of the shelf 1 of the N-1th normal image information. The similarity is determined with, and the product identification information candidate of the face of the shelf 1 of the N-1th normal image information having the highest similarity is identified as the product of the face of the shelf 1 of the Nth normal image information. Indicates that identification is performed as a candidate for identification information. In FIG. 23, a case is shown in which, in addition to the face of the (N-1)th regular image information at the position corresponding to the face of the Nth regular image information, two faces on the left and right sides of the face are compared. .. It should be noted that not only the same shelf but also the image information of the face positions of the upper and lower shelves may be compared. For example, in the case of FIG. 23, when specifying the candidate of the product identification information of the face position at the center of the shelf 2 of the Nth normal image information, the center of the shelf 2 of the N-1th normal image information and Not only is it compared with the image information of the two face areas on the left and right sides, but also the image information of the center of the shelf 1 of the N-1th fixed image information and the image information of the two face areas on the left and right sides, N- The similarity may be compared with the image information of the center of the shelf 3 of the first normal image information and the image information of two face regions on the left and right sides thereof.

As a result of the comparison of the N-1th normal image information with the face image information, if the product identification information candidate cannot be specified because the similarity does not satisfy the threshold value, the Nth normal image information. The image information in the area of the face identification information X in the above is compared with the sample information stored in the sample information storage unit 21, and the product identification information whose similarity is equal to or higher than a predetermined threshold value and which has the highest similarity is N. It is specified as a candidate of the product identification information of the face of the face identification information X in the normal image information for the first time. The similarity determination process in this case can be performed in the same manner as the initial product identification information identification process.

The candidate product identification information identified as described above corresponds to the shooting date/time information, the shop information, the image information identification information of the shot image information, the image identification information of the upright image information, and the face identification information for identifying the face. It is also stored in the identification information candidate storage unit 22.

In each of the first to fourth embodiments, by identifying the candidate of the product identification information on the face and comparing the corresponding faces with each other in each of the image information of the same display shelf taken at the same opportunity at different angles. The product identification information on the face is confirmed. In this embodiment, the candidates of the product identification information on the face are not specified for each of the image information captured at different angles, but the image information captured at different angles is first combined to eliminate the reflected portion such as light. A case will be described in which the identification processing of the product identification information on the face is performed on the one piece of image information obtained by incorporating. An example of a block diagram of the system configuration of the image recognition system 1 in this case is shown in FIGS. 24 and 25.

The image recognition system 1 of the present embodiment includes a photographed image information storage unit 20, a sample information storage unit 21, an identification information storage unit 23, and an image information processing unit 24. The photographed image information storage unit 20, the sample information storage unit 21, and the identification information storage unit 23 have the same configurations as those in the first to fourth embodiments.

The image information processing unit 24 includes a captured image information input reception processing unit 241, a captured image information positive value conversion processing unit, a shelf position specifying processing unit 243, a shelf area cutout processing unit 244, a face specifying processing unit 245, and identification information. It has a specific processing unit 246, a shelf image matching processing unit 247, and an image information processing unit 248.

The captured image information input acceptance processing unit 241 and the captured image information positive value conversion processing unit execute the same processing as that in the first to fourth embodiments. That is, the captured image information input reception processing unit 241 receives the input of image information (captured image information) of the display shelf of the store captured by the captured image information input terminal 3 and stores it in the captured image information storage unit 20. Further, the photographed image information emplacement processing unit 242 generates trapezoidal image information in which the photographed image information stored in the photographed image information storage unit 20 is subjected to trapezoidal correction processing to be orthonormalized.

The image information processing unit 248 performs a process of synthesizing a part or the entire range of the orthonormal image information, which is subjected to the positive value conversion process based on the captured image information, into one image information. That is, a part or the whole range of each normal image information obtained by photographing the same display shelf at the same occasion at different angles is divided by a mesh using a predetermined reference unit, for example, about 10 to 100 pixels as a reference unit. Then, the average value of the brightness of the image information in the area of each mesh is calculated. The average value of the lightness is compared for each mesh corresponding to the orthonormal image information captured at different angles, and the mesh of the image information having the smallest average value of the lightness is specified. This process is performed for all meshes. If the image information of the specified mesh region is arranged side by side in the vertical or horizontal direction as one piece of image information (processed normal image information), it is possible to generate image information in which the influence of light reflection is reduced. .. It should be noted that the image information of the mesh area is not limited to the average value of lightness, and the mesh to be combined may be specified by using the index value according to another evaluation standard for lightness. Further, it is preferable that the image information to be processed by the image information processing unit 248 is corrected such that the image information such as the shutter speed and the aperture value is the same or equivalent between the image information. When the above-mentioned processing using lightness is performed between image information that has different shooting conditions or is not corrected equivalently, image information with a low lightness-based index value (dark) is selected for the part without reflection. Will be done. Even in this case, the reflection itself can be removed, but problems such as a decrease in contrast and brightness of the entire image information to be generated or a noticeable roughness are likely to occur, which may affect recognition accuracy. Is.

The respective processes in the face identification processing unit 245, the identification information identification processing unit 246, and the shelf image matching processing unit 247 are the same as those in the first to fourth embodiments, but the processing target is the shelf region of the normal image information. Instead, the processing is performed on the processed normal image information generated by the image information processing unit 248. Further, the identification information identification processing unit 246 identifies the candidate of the product identification information of the product in the face area in the first to fourth embodiments and stores the candidate in the identification information candidate storage unit 22, but in the present embodiment, The product identification information of the product in the face area is specified, and the specified information is stored in the identification information storage unit 23.

By performing the processing on the processed fixed image information in this way, the image of the range of the shelf area can be obtained without specifying the candidate of the product identification information on the face as in the first to fourth embodiments. Since the image information processing unit 248 performs the combining process on the information, it is possible to reduce the influence of the reflection of light on the shelf area. Therefore, the product identification information on the face can be directly specified accurately.

The image information processing unit 248 processes the normal image information by the image information of the area at the shelf position specified by the shelf position specifying processing unit 243 and the image of the shelf area cut out by the shelf area cutting processing unit 244. Although it is preferable to execute it for information, the present invention is not limited to this, and may be executed for a part or the entire range of the orthonormal image information generated by the captured image information positive value conversion processing unit.

Next, an example of the processing process in this embodiment will be described with reference to the flowcharts of FIGS. 26 and 27.

The person in charge of photographing the display shelves of the store photographs the same display shelf from a plurality of angles so that the reflection of light or the like does not overlap. Then, the photographed image information (FIGS. 8 and 9) photographed from different angles at the same opportunity on the display shelf of the store is input from the photographed image information input terminal 3 and photographed by the image information processing unit 24 of the management terminal 2. The image information input reception processing unit 241 receives the input (S300). Further, the input of image information identification information such as shooting date/time, store identification information, and shot image information is accepted. Then, the photographed image information input reception processing unit 241 stores the photographed image information, the photographing date and time, the store identification information, and the image information identification information of the photographed image information, which have been input, in the photographed image information storage unit 20 in association with each other.

When the management terminal 2 receives a predetermined operation input, the photographed image information normalization processing unit 242 extracts the photographed image information stored in the photographed image information storage unit 20 and performs a trapezoidal correction process on a shelf. Input of four points of position (position of display shelf) is accepted, and keystone correction processing is executed (S310). 10 and 11 show an example of the captured image information (normal image information) on which the trapezoidal correction process is executed in this manner.

Then, by receiving a predetermined operation input in the management terminal 2 with respect to the fixed image information, the shelf position specifying processing unit 243 specifies the shelf area (S320). That is, the input of the shelf area in the normal image information is accepted. 12 and 13 are diagrams showing a state in which the shelf area is specified.

When the shelf area is specified as described above, the shelf area cutout processing unit 244 cuts out the image information of the shelf area from the normal image information based on the shelf area input in S320. (S330).

The image information processing unit 248 executes the processed normal position image information generation process for the image information of the shelf area cut out by the shelf area cutout processing unit 244 in S330 (S340).

First, the image information processing unit 248 changes the magnification so that the normal image information of the cut out shelf area is matched or substantially matched in vertical and horizontal size for each corresponding shelf area. And arrange the sizes. Then, the normal image information of the cut out shelf area is divided into meshes in predetermined reference units, for example, every 10 pixels (S400). FIG. 28 schematically shows this state. FIG. 28A shows a state in which meshing is performed on the shelf area of the normal image information in FIG. 12 and FIG. 28B in FIG.

Then, the image information processing unit 248 calculates, for each mesh, an average value of brightness for each image information in the area of the mesh (S410). The average value may be a simple average, a weighted average, or the like, and may be a value other than the average as long as it is a reference value based on the brightness of each image information in the area of the mesh.

Then, the image information processing unit 248 compares the average value (reference value) of the brightness for each corresponding mesh (S420), and specifies the image information of the mesh having the smallest brightness among the comparisons (S430). In the case of FIG. 28, in each of the normal image information of FIGS. 28(a) and 28(b), for each mesh in the shelf area of the respective normal image information, the brightness of each image information of the mesh area is calculated. Calculate the average value. Since the mesh A1 (FIG. 28(a)) and the mesh A2 (FIG. 28(b)) correspond to each other, the average value of the brightness of the mesh A1 and the average value of the brightness of the mesh A2 are compared. For example, if the average value of the brightness of the mesh A1 is small, the mesh A1 is specified. Since the mesh B1 (FIG. 28(a)) and the mesh B2 (FIG. 28(b)) correspond to each other, the average value of the brightness of the mesh B1 and the average value of the brightness of the mesh B2 are compared. For example, if the average value of the brightness of the mesh B2 is small, the mesh B2 is specified. In this way, the comparison of the brightness of the corresponding meshes and the specific processing are performed for all the meshes.

By arranging the identified meshes side by side vertically and horizontally, one piece of image information (processed normal image information) is generated (S440). In the above example, since the meshes A1, B2, etc. are specified, by arranging the meshes A1, B2, etc. specified in the process of S430 side by side vertically and horizontally, one It is synthesized and generated as image information. FIG. 29 schematically shows this processing. Note that, in FIG. 29, for ease of understanding, the processing for the highest shelf area of the shelf areas in FIG. 28 is shown as an example.

By performing the above-described processing, it is possible to generate image information with reduced light reflection in the shelf area.

As described above, the face identification processing unit 245 performs face identification processing, and the identification information identification processing unit 246 displays the product identification information in units of faces with respect to the processed normal image information as in the first to fourth embodiments. Specific processing is executed (S350, S360). That is, the face identification processing unit 245 identifies the face area in the processed normal image information, and the identification information identification processing unit 246 stores the sample information stored in the sample information storage unit 21 and the image information in the face area. The product identification information of the product in the area of the face is specified by executing the product identification information specifying process by the image matching process. Then, the identified product identification information is stored in the identification information storage unit 23 as in the first to fourth embodiments (S370).

Even if the processing according to the present embodiment is used, it is not always possible to specify the product identification information of all faces. Therefore, for a face that cannot be identified, the input of product identification information is accepted, and the product identification information that has been input is used as the shooting date/time, store identification information, image information identification information of captured image information, image information of orthostatic image information. The identification information and face identification information may be associated with each other and stored in the identification information candidate storage unit 22 as candidates for the product identification information. Further, the input of the correction processing of the identified product identification information candidate may be similarly received.

By performing the above-described processing, it is possible to specify the product identification information of the product displayed on the shelf of the display shelf shown in the photographed image information.

Also in this embodiment, the modification process of each processing unit in the first to fourth embodiments can be used.

If the sample information stored in the sample information storage unit 21 includes reflection of light or surrounding colors, the image characteristics (image characteristic amount) of the appearance of the product (for example, package or packaging) will be affected. Affects the accuracy of the matching process. Therefore, it is preferable to use, as the image information used as the sample information, image information captured so that the original color of the product is not damaged by the reflection of light or surrounding colors. However, depending on the shape and material of the appearance of the product, it may be difficult to avoid reflection due to the reflection of light. For example, in the case of a metal can, it is inevitable that the reflection of the light of the illumination is reflected in the photograph because of its high reflectance.

Factors that give rise to disturbances such as glare when photographing sample information of a product include the photographing environment of the product and the external shape of the product. In the shooting environment, in addition to the light source such as a lighting device, there are the background of the product and the condition of the surface on which the product is placed. In addition, depending on the direction of the surface that constitutes the appearance of the product, the appearance shape of the product causes reflection of light from a photographing device, a lighting device, or the like.

Therefore, in this embodiment, a process for obtaining sample information without reflection will be described. The processing in this embodiment can be applied to the sample information in the first to fifth embodiments.

The following shooting methods can be applied as a method for imprinting depending on the shooting environment.

In order to prevent distortion of sample information, shooting with the shooting device is performed from the horizontal facing direction of the product. In this case, it may be unavoidable that the appearance of the product is reflected due to reflection or the like. Therefore, a flat object that does not have a specific hue such as white or gray is reflected. Specifically, a product that is used as sample information is photographed against a background of white or gray that does not have a specific hue. White or gray paper or cloth should be used as the background of the product. The white or gray paper or cloth used as the background should be spread to the front of the product to the extent that the result of reflection is flat, without creases on the surface on which the product is placed. As a result, it is possible to reduce the influence of reflection in the shooting environment.

The following shooting methods can be applied as a method for imprinting on the appearance of the product.

Regarding the product for which the sample information is captured, the product and the illumination are changed without changing the relative position between the product and the image capturing device (for example, the relative position between the surface of the product captured by the image capturing device and the image capturing device). The relative position with respect to the light source (such as the relative position between the surface of the product illuminated by the light source and the light source) is changed, and the product is photographed multiple times. In other words, the relative position between the product and the photographing device is not changed, but the relative position between the photographing device and the light source is changed, and the product is photographed a plurality of times. In other words, in the captured image information, the product is captured at the same position, the same size (size), and the same angle, and the light from the light source such as the illumination is applied to the product from different angles, so that the image is captured multiple times. Then, the image recognition system 1 receives the input. FIG. 30 shows an example of a state in which a product is photographed as seen from above. FIG. 30(a) is a diagram showing an example of a state in which a product is illuminated and illuminated from the left side, and FIG. 30(b) is a diagram showing an example of a state in which a product is illuminated and photographed from the right side. Although FIG. 30 shows a case where images are taken from two places on the left and right sides of the image taking device, the images may be taken from three places or more. In FIG. 30, an optical path of a reflection that occurs at a position closest to the central axis of the product due to the light emitted from the light source closest to the photographing device is indicated by a dashed arrow. In this case, the angle of incident light (incident angle) and the angle of reflected light (reflection angle) are the same with the perpendicular to the surface of the product as the axis of symmetry.

Then, the image recognition system 1 obtains appropriate sample information by cutting out portions of the plurality of input image information that are not reflected and combining them with each other. Appropriate sample information is stored in the sample information storage unit 21. FIG. 31 schematically shows this processing. FIG. 31(a) is image information taken in the state of FIG. 30(a), FIG. 31(b) is image information taken in the state of FIG. 30(b), and FIG. It is appropriate sample information obtained by combining the image information of 31(a) and 31(b). That is, in FIG. 30A, since the product is photographed by shining light from the right side, light is likely to be reflected on the right side of the product. Further, in FIG. 30(b), since the product is photographed by shining light from the left side, light is likely to be reflected on the left side of the product. In other words, there is no light reflection on the left side (A) of the product in FIG. 31(a), and no light reflection on the right side (B) of the product in FIG. 31(b). Therefore, the image information of the area on the left side (A) of the product in FIG. 31(a) and the image information of the area on the right side (B) of the product in FIG. 31(b) are respectively cut out and combined into one image information. (FIG. 31(c)). This makes it possible to synthesize appropriate sample information without the reflection of light.

In addition, in FIGS. 30A and 30B, the sums α and β of the incident angle and the reflection angle are preferably the same or substantially the same. As a result, if the photographed image information shows the product near the horizontal center of the image information, the left half or the right half where no reflection in the photographed image information occurs is cut out. Since the images may be combined, the image combining process becomes simple. When the sums α and β of the incident angle and the reflection angle are the same or substantially the same, the accuracy of the image information combining process can be improved, but the angles α and β do not have to be the same or substantially the same.

The process of FIGS. 30 and 31 will be described more specifically as follows. The external shape of the product to be photographed can be roughly classified into three types shown in FIG. 32, since the horizontal cross-sectional view of the product is generally convex or substantially convex. That is, when the product is a rectangular parallelepiped or a substantially rectangular parallelepiped like a boxed product (the front surface is flat) (Fig. 32(a)), like a beverage can, it is cylindrical and the side faces are vertical (Fig. 32(b)), a case where a part of the side surface is not vertical such as a wine bottle having a curved surface such as a spherical surface (FIG. 32(c)). For convenience, FIG. 32A is a first case, FIG. 32B is a second case, and FIG. 32C is a third case.

FIG. 33 is a diagram showing a case where the first case is photographed by a photographing device. FIG. 33A is a view from above when photographing a product, and FIG. 33B is a side view when photographing a product. In the case of the first case, the light source such as a lighting is, for example, one light so that the photographing device is directly faced to the product and the light illuminating the product comes from one direction. Then, the light emitted from the light source such as the illumination device is reflected on the surface of the product facing the image capturing device (the reflecting surface in the range indicated by the alternate long and short dash line) and enters the focal point of the image capturing device.

Then, if a straight line connecting the focal point of the photographing device and the right side or the left side of the reflection surface of the product is assumed to be a straight line with the perpendicular line from the reflection surface of the product to the photographing device as the axis of symmetry, the outside of that straight line If the light source is installed at any position of the (side) area (area X, area Y), no reflection occurs in the product. Here, the angle formed by the straight line connecting the focal point of the photographing device and the right side or the left side of the reflecting surface of the product and the perpendicular, and the angle between the line serving as the boundary line between the regions X and Y and the perpendicular are the incident angles. From the relationship between the angle and the reflection angle, both are the angle a. Therefore, it suffices to photograph from either one of the area X and the area Y (it is not necessary to combine the images). Alternatively, as shown in FIG. 33(b), a region above the horizontal line from the top of the product and the straight line (line with the horizontal line as the axis of symmetry) which is the same angle b as the angle b to the top of the product from the imaging device An area Z is set, and a light source is installed in the area Z to take an image. In the case of the first case, if the image is taken as shown in FIG. 33(a) or FIG. 33(b), the light source is not reflected on the product.

FIG. 34 is a diagram showing a case where the second case is photographed by a photographing device. In the second case, as in FIG. 34(a), as in the case of FIG. 33(b), the angle b is the same as the horizontal line from the top of the product and the angle b from the photographing device to the top of the product. An area above a straight line (a line having a horizontal line as an axis of symmetry) is defined as an area Z, and a light source is installed in the area Z to perform photographing. If the image cannot be captured as shown in FIG. 34(a), the image is captured as shown in FIG. 34(b). FIG. 34B is a view from above when a product is photographed. In FIG. 34(b), similar to FIG. 30, the relative position between the product and the photographing device is not changed (for example, the relative position between the surface of the product photographed by the photographing device and the photographing device), The relative position between the product and a light source such as illumination (for example, the relative position between the surface of the product illuminated by the light source and the light source) is changed, and the product is photographed multiple times. Then, the image information captured at each position is combined with the non-reflective portion to obtain sample image information.

That is, the light emitted from the illumination is reflected by the reflecting surface of the product and enters the focal point of the photographing device. Therefore, when the illumination is installed in the area on the left side of the product (area X), the light emitted from the illumination causes reflection on the left side of the product, and the illumination is installed in the area on the right side of the product (area Y). , The light emitted from the illumination causes a reflection on the right side of the product. Therefore, in the image information when the light source is located on the left side of the product, the right side is specified, and in the image information when the light source is located on the right side of the product, the left side is specified and combined into one image information. Use as sample information.

FIG. 35 is a diagram showing a case where the third case is photographed by a photographing device. 35 is similar to FIGS. 30 and 34(b), the relative position between the product and the photographing device (for example, the surface of the product photographed by the photographing device and the relative position between the photographing device) can be changed. Instead, the relative position between the product and a light source such as illumination (for example, the relative position between the surface of the product illuminated by the light source and the light source) is changed, and the product is photographed multiple times. Then, the image information captured at each position is combined with the non-reflective portion to obtain sample image information.

As described above, when the sample information of the product is photographed, the photographing method corresponding to the first case to the third case may be applied according to the shape of the appearance of the product. However, since the shapes of the appearances of the products are generally different, the products are not classified into three cases, but the third case (that is, the imaging method similar to those in FIG. 30, FIG. 34B, and FIG. 35 described above). ) Can be used for general purposes.

By photographing the sample information of the product by the above-described photographing method, it is possible to obtain the sample information in which the disturbance caused by both the photographing condition and the appearance shape of the product is reduced.

As a modified example of the sixth embodiment, the process of the fifth embodiment may be used. That is, in order to generate appropriate sample information, a plurality of pieces of image information captured by changing the position of the light source without changing the relative positional relationship between the product and the image capturing apparatus are the same as in the fifth embodiment. Execute the process. Specifically, as in the fifth embodiment, all or a part of the image information (a region including the product) is divided into meshes among a plurality of pieces of image information obtained by photographing the product, and the average value of the brightness in the mesh is calculated. .. Then, the average value of the brightness is calculated for each corresponding mesh, and the image information of the mesh having the smallest value is specified. Appropriate sample information can be obtained by arranging the image information of the mesh thus identified in a vertical and horizontal manner.

The sample information generated using the processing in this embodiment performs the same processing as that of the first to fifth embodiments to specify the product identification information of the product shown in the image information, and also the captured image information. Alternatively, the image recognition system 1 may be configured to execute the process of specifying the product identification information by simply executing the image matching process between the normal image information and the sample information.

In the first to sixth embodiments described above, the case where the product identification information of the product displayed on the display shelf of the store or the like is specified from the image information of the image of the display shelf has been described. However, the identification information of the product is specified. It can be used not only for processing but also for identifying identification information of a specific object from image information. That is, image information (samples in which a plurality of specific objects are arranged, etc.) or specific objects (objects to be specified in the identification information) are taken from a plurality of angles at the same opportunity, and sample information. It can be applied to the image recognition system 1 that identifies the identification information of the specific object shown in the image information by comparing the similarities. As the specific processing in that case, the processing of the first to sixth embodiments can be applied. In this case, the process for the display shelf may be read as the process for the photographing target object, and the process for the product may be read as the process for the specific target object.

By using the image recognition system 1 of the present invention, even when light or the like is reflected in the image information, it is possible to identify an object such as a product more accurately than before.

1: Image recognition system 2: Management terminal 3: Photographed image information input terminal 20: Photographed image information storage unit 21: Specimen information storage unit 22: Identification information candidate storage unit 23: Identification information storage unit 24: Image information processing unit 25: Identification information comparison processing unit 70: arithmetic device 71: storage device 72: display device 73: input device 74: communication device 241: photographed image information input acceptance processing unit 242: photographed image information emplacement processing unit 243: shelf position specification Processing unit 244: Shelf region cutout processing unit 245: Face identification processing unit 246: Identification information identification processing unit 247: Shelf image matching processing unit 248: Image information processing unit 2420: Features in the (N-1)th image information Quantity collection area 2421: Feature quantity collection area in Nth image information

Claims (6)

An image information processing unit that specifies a candidate of product identification information of a product shown in the image information for each of the image information of the display shelves displaying the product from a plurality of angles,
In image information corresponding taken at different angles, have a, an identification information comparing unit for identifying the merchandise identification information item that is reflected in the image information using the likelihood of the candidate of the specified merchandise identification information The
The identification information comparison processing unit,
In the corresponding image information captured at different angles, when there are a plurality of candidates for the specified product identification information, the product identification information is specified by performing a predetermined calculation using the likelihood.
An image recognition system characterized by the above. The image information processing unit is
A processing unit for generating image information of executing the Sei置 processing the image information obtained by photographing the display shelf,
By comparing the image information of a part of the display shelf in the orthographic image information with the sample information of the product, a candidate for the product identification information of the product shown in the region is identified. An identification information specifying processing unit
The image recognition system according to claim 1, further comprising: The identification information identification processing unit,
When the candidate of the product identification information of the product shown in the area cannot be specified, the input of the product identification information is accepted,
For at least the area for which the input of the product identification information is accepted, the accepted product identification information is specified as the product identification information of the area without performing the process in the identification information comparison processing unit.
The image recognition system according to claim 2, wherein: A first processing unit that specifies candidates for identification information of the specific target imaged in the image information, for each of the image information of the imaged target imaged from a plurality of angles;
A second processing unit that, in corresponding image information captured at different angles, uses the likelihood of the identified identification information candidate to identify the identification information of the specific object shown in the image information;
The second processing unit is
In the corresponding image information captured at the different angles, when there are a plurality of identified identification information candidates, the identification information of the specific target imaged in the image information is obtained by performing a predetermined calculation using the likelihood. Identify,
An image recognition system characterized by the above. Computer,
An image information processing unit that specifies, for each piece of image information obtained by photographing a display shelf on which products are displayed from a plurality of angles, candidates for product identification information of the product shown in the image information,
An image that functions as an identification information comparison processing unit that identifies the product identification information of the product shown in the image information by using the likelihood of the identified product identification information candidates in the corresponding image information captured at different angles. A recognition program,
The identification information comparison processing unit,
In the corresponding image information captured at different angles, when there are a plurality of candidates for the specified product identification information, the product identification information is specified by performing a predetermined calculation using the likelihood.
An image recognition program characterized by the above. Computer,
A first processing unit that specifies, for each of image information obtained by shooting the shooting target from a plurality of angles, candidates for identification information of the specific target shown in the image information,
An image that functions as a second processing unit that identifies the identification information of the specific object shown in the image information by using the likelihood of the identified identification information candidates in the corresponding image information captured at different angles. A recognition program,
The second processing unit is
In the corresponding image information captured at the different angles, when there are a plurality of identified identification information candidates, the identification information of the specific target imaged in the image information is obtained by performing a predetermined calculation using the likelihood. Identify,
An image recognition program characterized by the above.