JP2021184125A - Verification system - Google Patents

Abstract

To provide a verification system for image recognition processing.SOLUTION: A verification system includes an image recognition processing section for executing first comparison processing of a part or all of an area in image information, and a verification processing section for executing second comparison processing of a part or all of the recognition result by the image recognition processing section. The verification processing section executes, as the second comparison processing, processing by a processing method that is not same as the first comparison processing, or a processing method in which all of elements for use in the first comparison processing and elements for use in the second comparison processing are not common.SELECTED DRAWING: Figure 1

Description

The present invention relates to a verification system for image recognition processing.

It may be required to determine whether or not an object to be determined is reflected in a certain image information. For example, there is a case where it is determined whether or not the product is displayed by determining whether or not the product to be determined is reflected in the image information obtained by photographing the display shelf of the product.

In this way, when determining whether an object is reflected in the image information, it is common to execute an image recognition process (matching process) between the image information to be processed and the image information of the object. It is a target. For example, in Patent Document 1 below, there is a system for grasping products handled by vending machines by executing image recognition processing on image information taken by vending machines based on registered images for each product. It has been disclosed.

Japanese Unexamined Patent Publication No. 2014-191423

The specific processing of Patent Document 1 is to shoot a vending machine from a plurality of directions, align the positional relationships of the shot images, and then superimpose the shot images to generate a composite image. Then, the product handled by the vending machine is specified by collating the generated composite image with the registered image representing the product that may be displayed on the vending machine (image recognition processing).

In the collation process (image recognition process) between the composite image and the registered image representing the product, it is common to execute the process by using each feature amount. Since the accuracy of such image recognition processing is not perfect, various studies and developments have been conducted to improve the accuracy.

However, at present, even if the latest image recognition processing technology is used, its recognition accuracy has not reached the human recognition ability. When it is determined whether or not the product to be determined is displayed by determining whether or not the product is displayed, the recognition accuracy cannot be said to be high.

In addition, since the conventional image recognition system is a system that determines which of the pre-registered images is the most similar, the superiority or inferiority of the relative ranking is compared, and the image with the highest ranking is the first. It has been identified as a candidate. However, it is unclear whether the judgment result of the image recognition system is correct or not, and it may be necessary for a human to judge the correctness of the first candidate of the image recognition system.

As described above, in the conventional image recognition system, it is necessary for humans to confirm and correct the recognition result. Confirmation and correction work by humans becomes a bottleneck for labor saving and the burden is extremely heavy.

For these reasons, it is required to save labor for human confirmation and correction work for the recognition result of image recognition processing.

In addition to improving the accuracy of image recognition processing, it is also considered to use multiple image recognition processing techniques in parallel and use the most accurate judgment result as the result of image recognition processing.

However, in this case, since a plurality of image recognition processes are treated equally and the output result is output by each image recognition processing system, processing load and processing time are required for each image recognition processing system. For example, in the case of searching for a product in an area with image information from 10,000 samples, the system uses three image recognition processing systems and adopts the most accurate output result. In this case, it is necessary to perform image recognition processing on the image information area and 10,000 samples in each image recognition processing system. Therefore, the image recognition processing is performed on a total of 30,000 specimens, which causes a problem that the processing load and the processing time are significantly increased.

Therefore, in view of the above problems, the present inventor has invented a verification system that verifies the recognition result in the image recognition processing system.

The first invention is a second aspect of an image recognition processing unit that executes a first comparison process for a part or all areas of image information, and a part or all of recognition results by the image recognition processing unit. A verification processing unit having a verification processing unit for executing the comparison processing of the above, wherein the verification processing unit has a processing method that is not the same as the first comparison processing as the second comparison processing, or the above. This is a verification system that executes processing by a processing method in which all the elements used in the first comparison processing and the elements used in the second comparison processing are not common.

By configuring as in the present invention, the independence of the first comparison process and the second comparison process can be ensured. Further, in the second comparison process, unlike the first comparison process, the comparison process is executed for the judgment result of the image recognition processing unit, so that the processing load is different from the case where a plurality of comparison processes are simply coexisted. Can be significantly reduced, and it becomes easier to make a correct / incorrect judgment without intervening human judgment.

In the above-described invention, the verification processing unit uses a processing method different from that of the first comparison processing, or an element used in the first comparison processing and the second comparison processing as the second comparison processing. It can be configured like a verification system in which all the elements used execute processing by different processing methods.

By configuring as in the present invention, the independence of the first comparison process and the second comparison process can be further ensured, and the accuracy of the verification process can be improved.

In the above-described invention, the verification system includes a sample information storage unit that stores sample information of an object, and a verification result processing unit that executes processing for a determination result of a second comparison processing in the verification processing unit. In the first comparison process in the image recognition processing unit, the area to be processed in the image information is compared with the sample information in the sample information, and the image is reflected in the area to be processed. In the second comparison process in the verification processing unit, the corresponding information of a part or all of the objects specified in the first comparison process and the processing target are performed. When the comparison result with the region is satisfied with the predetermined conditions, it is determined that the result of the first comparison process is correct, and when the verification result processing unit determines that the verification result processing unit is correct, the verification result processing unit determines that the result is correct. It can be configured like a verification system that specifies the result of the first comparison process as an object reflected in the area to be processed.

By executing the processing as in the present invention, if the verification processing unit determines that the answer is correct in the area to be processed, the object reflected in the correct answer can be identified as it is, so that the verification is performed manually. The work can be saved.

In the above-described invention, in the second comparison process in the verification processing unit, if the comparison result does not satisfy a predetermined condition, the result of the first comparison process is determined to be an error, and the verification result processing unit determines that the result is an error. , If the verification processing unit determines that an error has occurred, it can be configured like a verification system that gives a predetermined notification.

By executing the process as in the present invention, if the verification processing unit determines that an error has occurred in the area to be processed, this notification is given. Therefore, only the area to which this notification is given is manually verified. You just have to do. Therefore, the manual verification work can be saved.

In the above-described invention, the sample information storage unit includes at least one verification key including information indicating a verification area that characterizes an object, information on the type of image filter that extracts the verification area, and image information of the verification area. At least one or more verification descriptors are stored for each object, and the verification processing unit specifies the verification descriptor corresponding to some or all of the objects specified in the first comparison process. Then, for each verification key in the specified verification descriptor, it is determined whether the area corresponding to the verification area can be extracted from the area to be processed by using the information of the verification key, and the area can be extracted. When the degree of matching between the image information of the extracted area and the image information of the verification key is calculated, the calculated degree of matching is compared with a predetermined threshold, and the comparison result satisfies the predetermined condition. Can be configured like a verification system that specifies that the result of the first comparison process is correct as an object reflected in the area to be processed.

In the above-described invention, the verification processing unit specifies the verification descriptor corresponding to a part or all of the objects specified in the first comparison process, and each verification key in the specified verification descriptor. Of the areas to be processed, it is determined whether the area can be extracted from the area corresponding to the verification area in the verification key by using the image filter in the verification key, and if the area can be extracted, if the area can be extracted. The degree of matching between the extracted area and the image information of the verification key is calculated, the calculated degree of matching is compared with a predetermined threshold value, and when the comparison result satisfies the predetermined condition, the first comparison process is performed. It can be configured like a verification system that identifies the object in the area to be processed as the result of the above is correct.

Various processing methods can be used as the second comparison processing, but by using the processing method configured as in the present invention, the manual work is reduced as much as possible, and the second comparison processing is executed with high accuracy. Can be done.

In the above-described invention, the verification processing unit accepts a closed region that characterizes the object as a verification region from the image information or the sample information of the object as the sample information, and sets the verification region and the peripheral region with respect to the verification region. It can be configured like a verification system in which an image filter that distinguishes between the above is selected and the image information in the verification area is stored in the sample information storage unit as information in the verification key.

The verification key in the verification descriptor can be configured as in the present invention.

In the above-mentioned invention, the verification processing unit searches for an image filter that is compatible in the widest area inside the verification area and is incompatible in the widest area of the peripheral area among the image filters registered in advance. By doing so, it can be configured like a verification system that selects an image filter.

In the above-described invention, the verification processing unit sets the number of conforming pixels in the verification area / the number of pixels in the verification area to AT, and the number of non-conforming pixels in the peripheral area / the peripheral area among the image filters registered in advance. When the number of pixels of is BF, it can be configured like a verification system that selects an image filter that maximizes the harmonic mean of AT and BF.

The work of manually specifying the image filter used for the verification key in the verification descriptor is burdensome. Therefore, by configuring as in the present invention, the image filter used for the verification key can be automatically specified.

In the above invention, like a verification system that selects an image filter that distinguishes between the verification region and the peripheral region with respect to the verification region after the binary image is expanded or contracted with respect to the verification region and the peripheral region. Can be configured in.

The result detected by applying the image filter in the verification area and the peripheral area may be a so-called missing image in which the internal pixels are not filled. Therefore, it is preferable to calculate the conforming / non-conforming pixels described above after filling the missing pixels by expanding / contracting the binary image as in the present invention. This stabilizes the results and allows the selection of a more appropriate image filter.

In the above-described invention, the sample information storage unit stores at least one or more feature information categorizing the portion that characterizes the object and the superiority as a verification descriptor, and the feature information is as the feature information. It has an object that is a part that characterizes the object, the attribute of the object, and the attribute value for the attribute, and the verification processing unit has a part or all of the object specified in the first comparison process. The verification descriptor corresponding to the object of the object is specified, and for each feature information in the specified verification descriptor, the area to be processed sets the attributes and attribute values in the feature information in the order of the feature information having the highest degree of superiority. It is configured like a verification system that determines whether it is satisfied, and if the predetermined condition is satisfied, identifies it as an object reflected in the area to be processed, assuming that the result of the first comparison process is correct. can do.

By configuring as in the present invention, the verification descriptor can be configured more easily and its processing can be simplified.

In the above-described invention, the sample information storage unit stores at least one profile data including information of a partial region representing the shape of the object in the image information, and the verification processing unit stores at least one profile data for each object. By reducing the color of the area to be processed to a predetermined number of colors, dividing the reduced image information into areas for each color, and specifying the specified area as a component, the area to be processed can be obtained. Profile data is generated, and the similarity is compared between the profile data of the area to be processed and the profile data of one or all of the objects specified in the first comparison process. When the similarity satisfies a predetermined condition, it can be configured like a verification system that identifies the object as being reflected in the area to be processed as the result of the first comparison processing is correct.

In the above-described invention, the verification processing unit is one of a predetermined number of components having a large area among the components of the profile data of the area to be processed and one of the objects specified in the first comparison processing. A predetermined number of components having a large area among the components of the profile data of all the objects are specified as nuclei, and the adjacent components of the core components in each of the profile data are connected by the centers of gravity. When the similarity is calculated using the pattern and the calculated similarity satisfies the predetermined condition, the corresponding component is specified in each of the profile data among the components not included in the pattern. Then, it is added to the pattern, the similarity is calculated from the correspondence between the components in the pattern of each profile data, and when the similarity satisfies a predetermined condition, the first comparison process is performed. It can be configured like a verification system that identifies the object in the area to be processed as the result of the above is correct.

Various processing methods can be used as the second comparison processing, but by using the processing method configured as in the present invention, the manual work is reduced as much as possible, and the second comparison processing is executed with high accuracy. Can be done.

In the above-described invention, the verification processing unit reduces the image information or sample information of the object to a predetermined number of colors, divides the reduced image information into regions for each color, and constitutes a specified region. It can be configured like a verification system that is specified as an element and stores the information indicating the area of the shape and the image information of the area as profile data in the sample information storage unit for each of the specified components. ..

By configuring as in the present invention, profile data can be automatically generated, so that manual work can be reduced and labor can be saved.

The first invention can be applied to the case where the product displayed on the display shelf is specified from the image information obtained by photographing the display shelf as in the present invention. That is, the face area specified from the image information obtained by photographing the display shelf is reflected in the face area by performing the first comparison process using the sample information of the product stored in the sample information storage unit. A verification system including an image recognition processing unit that identifies an object and a verification processing unit that executes a second comparison process for a part or all of the recognition results by the image recognition processing unit. In the verification processing unit, as the second comparison processing, processing that is not the same as the image recognition processing, or that all the elements used in the first comparison processing and the elements used in the second comparison processing are common. It is a verification system that executes no processing.

In the above-described invention, the second comparison process in the verification processing unit compares the information of a part or all of the objects specified in the first comparison process with the face region. When the comparison result satisfies a predetermined condition, it can be configured like a verification system in which the result of the first comparison process is regarded as correct and specified as an object reflected in the face region.

As one of the product marketing information, it is important to understand how much the product is displayed on the display shelf of the store. Therefore, in the past, the person in charge went to each store and manually specified each. However, the work is extremely burdensome and limited. Therefore, it is being considered to automatically identify it using image processing technology. However, at present, even if the latest image recognition processing technology is used, the recognition accuracy has not reached the human recognition ability, so it is necessary to manually confirm and correct the recognition result. turn into.

Therefore, by configuring as in these inventions, it is possible to significantly reduce the labor for human confirmation and correction work without significantly increasing the processing load by the computer, and thus the work required for grasping the display status of the product. The burden can be greatly reduced.

The first invention can be realized by loading and executing the program of the present invention in a computer. That is, the computer is subjected to the first comparison processing for a part or all of the area in the image information, and the second comparison with respect to a part or all of the recognition result by the image recognition processing unit. A verification program that functions as a verification processing unit that executes processing, and the verification processing unit performs processing that is not the same as the image recognition processing or the first comparison processing as the second comparison processing. This is a verification program that executes a process in which all the elements used in the above-mentioned second comparison process and the elements used in the second comparison process are not common.

The fifteenth invention can be realized by loading and executing the program of the present invention in a computer. That is, the computer performs the first comparison process on the face area specified from the image information obtained by photographing the display shelf by using the sample information of the product stored in the predetermined storage area, thereby performing the first comparison process in the face area. It is a verification program that functions as an image recognition processing unit that identifies an object in the image, and a verification processing unit that executes a second comparison process for a part or all of the recognition results by the image recognition processing unit. In the verification processing unit, as the second comparison processing, processing that is not the same as the image recognition processing, or all of the elements used in the first comparison processing and the elements used in the second comparison processing are all. A verification program that executes non-common processing.

By using the verification system of the present invention, it is possible to save labor in the verification of the recognition result in the image recognition processing system. Therefore, the conventional manual work can be significantly reduced or eliminated.

It is a block diagram schematically showing an example of the whole processing function of the verification system of this invention. It is a block diagram schematically showing an example of the hardware configuration of the computer used in the verification system of this invention. It is a figure which shows an example of the photographed image information schematically. It is a figure which shows an example of a sample information. It is a figure which shows typically an example of the image information in the state which performed the orthotopic process with respect to the photographed image information of FIG. It is a figure which shows an example of the image information of the face area cut out from the photographed image information. It is a figure which shows typically an example of the case where the elements of the image feature amount are classified and systematized. It is a flowchart which shows an example of the generation process of a verification descriptor. It is a flowchart which shows an example of the processing process of the whole processing in the verification system of this invention. It is a flowchart which shows an example of the verification process. It is a figure which shows an example of the verification descriptor in Example 1 schematically. It is a figure which shows an example of the verification area set in the sample information. It is a figure which shows typically the process of calculating the degree of coincidence between the image information obtained by binarizing the image information in the area to be compared, and the extracted graphic information corresponding to the verification key. It is a figure which shows an example of the verification descriptor in Example 2 schematically. It is a figure which shows the other example of the verification descriptor in Example 2 schematically. It is a flowchart which shows an example of the verification process in Example 2. FIG. It is a figure which shows typically the verification process in Example 2. It is a figure which shows an example of profile data schematically. It is a flowchart which shows an example of the processing process which generates the profile data corresponding to the sample information in Example 3. FIG. It is a figure which shows typically an example of the image information of a product as a sample information and the corresponding profile. It is a flowchart which shows an example of the processing process which generates profile data from the image information of a face area in Example 3. FIG. It is a flowchart which shows an example of the verification process in Example 3 schematically. It is a figure which shows an example of the image information of a face area. It is a figure which shows an example of the profile generated with respect to the image information of the face area of FIG. 23A schematically. The similarity is determined by using a profile based on the image information of the face area and a profile in which the product identification information of the product similar to the product reflected in the face area to be processed is associated with the corresponding sample information. It is a figure which shows typically the process to evaluate. It is a figure which shows typically the correspondence between the sample information of FIG. 20 and the profile, the image of the face area of FIG. 23, and the respective components in the profile of FIG. 24. It is a figure which shows an example of the image filter and the list of the parameter.

FIG. 1 shows a block diagram of an example of the overall processing function of the verification system 1 of the present invention. The verification system 1 uses a management terminal 2 and an input terminal 3.

The management terminal 2 is a computer that realizes the core processing function of the verification system 1. The input terminal 3 is a terminal for acquiring image information of a photographed object such as a display shelf of a store. The input terminal 3 may photograph and acquire an object such as a product as a sample used in the comparison process (image recognition process) described later.

The management terminal 2 and the input terminal 3 in the verification system 1 are realized by using a computer. FIG. 2 schematically shows an example of a computer hardware configuration. The computer can input information into an arithmetic unit 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk for storing information, a display device 72 such as a display for displaying information, and information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits / receives the processing result of the arithmetic unit 70 and the information stored in the storage device 71 via a network such as the Internet or LAN.

When the computer is provided with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited thereto.

The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be performed directly on the display with a predetermined input device (such as a pen for a touch panel) or a finger.

The input terminal 3 may include a photographing device such as a camera in addition to the above-mentioned devices. As the input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used. The input terminal 3 captures image information (photographed image information or object image information described later) by visible light or the like with a photographing device.

Each means in the present invention has only a logical distinction in its function, and may form the same area physically or substantially. The processing in each means of the present invention may be appropriately changed in the processing order. In addition, a part of the processing may be omitted. Further, a part or all of the functions in the management terminal 2 may be executed in the input terminal 3.

The verification system 1 includes an image information input reception processing unit 20, an image information storage unit 21, a sample information storage unit 22, an image information processing unit 23, an image recognition processing unit 24, a verification processing unit 25, and a verification result processing unit 26. ..

The image information input reception processing unit 20 receives input of image information (photographed image information) taken by the input terminal 3 and stores it in the image information storage unit 21 described later. For example, the input of the photographed image information of the display shelf of the store is accepted and stored in the image information storage unit 21. In addition to the captured image information, the input terminal 3 may accept input together with information indicating the shooting date and time and the shooting target, for example, store identification information such as a store name, image information identification information for identifying image information, and the like. FIG. 3 schematically shows an example of captured image information. FIG. 3 schematically shows photographed image information obtained by photographing a state in which products are displayed on three shelves on the display shelf.

The image information storage unit 21 stores the captured image information received from the input terminal 3, the shooting date and time, the image information identification information, and the like in association with each other.

The sample information storage unit 22 stores the sample information of the object in order to identify the object reflected in the photographed image information. The sample information is image information (object image information) obtained by photographing a part or all of an object and / or information on a feature amount thereof. The sample information may be the identification information of the object, for example, the product code such as the product name, model number, JAN code, and the image information of the object may be the image information of the entire object, or a part of the object, for example. When the object is a PET bottle beverage, it may be the image information of the label portion. When the feature amount information of the object image information is used as the sample information, it is not necessary to extract the feature amount each time the processing is performed.

The sample information storage unit 22 may generate one sample information for one object, or may generate one sample information for one object at a plurality of angles, for example, at a position that does not appear when the product is photographed in a face-to-face manner. In order to capture a certain surface, sample information from front, back, top, bottom, and both sides may be stored. Further, when one object has a plurality of exteriors (packages, etc.), the sample information for each exterior may be associated with one object. FIG. 4 shows an example of sample information. FIG. 4 is an example in which the image information of the exterior of the product is used as the sample information.

Since the object to be used as sample information is usually photographed from the position facing it, it is not necessary to perform the orthostatic processing and the distortion correction processing described later, but it may be performed.

The image information processing unit 23 sets the photographed image information received by the image information input reception processing unit 20 or the photographed image information stored in the image information storage unit 21 into a state of facing the photographed image information, and photographs. Face processing is executed to specify the area (face area) to execute the comparison processing (image recognition processing) with the sample information from the image information.

In general, when the object to be photographed is simply photographed, it is difficult to shoot the object in the orthotopic position in the face-to-face state. This is a process of transforming the image information so that the optical axis of the lens of the photographing device is the same as when the image is taken from a sufficient distance along the perpendicular direction of the plane to be photographed. An example of such correction processing is trapezoidal correction processing. If the image information is distorted, distortion correction processing may be added. FIG. 5 shows the image information in a state in which the orthostatic processing is executed on the captured image information of FIG.

If the captured image information is the image information captured in a facing state, or if there is no distortion, it is not necessary to execute the orthostatic processing and the distortion correction processing.

The captured image information may be any image information to be processed by the present invention. The captured image information also includes the image information after the correction processing for the captured image information such as the orthostatic processing is executed.

Further, when the image information processing unit 23 shoots a plurality of images, the image information processing unit 23 executes a process of synthesizing the images into one image information, and the image information obtained by the synthesizing process is positive. Placement processing and face processing may be executed. A known method can also be used as a process of synthesizing a plurality of image information into one image information.

The face processing specifies an area (face area) for executing comparison processing with sample information described later in the captured image information. For example, in the case of photographed image information obtained by photographing a product display shelf and the sample information is a product, the area of the product displayed on the display shelf or the area of the product label is specified as the face area. If the product is a plastic bottle beverage, the area of the product label is the face area, and if the product is a boxed product (for example, sweets), the entire product package is the face area. Depending on the situation, the face area can be set as appropriate. Even when the product is a PET bottle beverage, the product area may be the face area.

There are various methods for specifying the face area, and it can be set arbitrarily according to the characteristics of the sample information. When the sample information is a product (for example, a PET bottle beverage) and the area of the product label is specified as the face area from the photographed image information of the display shelf, for example, between the shelves of the display shelf. Identify the vertical thin and narrow shadows that occur between products in the area (shelf area), identify the repeating pattern of the image, identify the step on the upper side of the package, the product width is the same, etc. Specify the product area by specifying the delimiter position based on the constraint. Then, from the area of the product, a predetermined rectangular area is specified as a label area, and that area is specified as a face area.

As the method for specifying the face area, any method can be adopted depending on the product category and the product form, and the method is not limited to the above. Further, the correction input by the operator may be accepted for the automatically specified face area. Further, the input of the position of the face area may be accepted from the operator.

Further, the image information processing unit 23 may specify the face region by using deep learning. In this case, the above captured image information is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the face area is set based on the output value. It may be specified. Further, as the learning model, it is possible to use various captured image information in which the face region is given as correct answer data.

The image information processing unit 23 cuts out the face region specified as described above. Cutting out the face area may actually cut out the face area specified from the captured image information, and also includes setting the area as a processing target for subsequent processing.

When cutting out the face area, the face area specified in each of the above processes may be cut out as it is, or the face area may be specified by a plurality of methods and the target to be cut out using the result of the face area specified by each method. You may specify the face area to be. For example, the area of a product is specified by identifying the vertical thin and narrow shadows that occur between the products in the area between the shelves of the display shelf (shelf area), and a predetermined area is specified from that area. A method of specifying a rectangular area and a method of specifying a face area by deep learning are performed, and the results of the face areas specified by each of these methods are finally cut out by a predetermined calculation. May be specified. The method for specifying the face area to be cut out is not limited to the above, and can be set arbitrarily. FIG. 6 shows an example of the image information of the face region cut out from the captured image information.

The image recognition processing unit 24 performs image recognition processing (first comparison processing) by comparing the face area specified by the image information processing unit 23 with each sample information of each object stored in the sample information storage unit 22. To execute. For example, by extracting the feature amount of the image information in the face area and executing the comparison processing between the extracted feature amount and the feature amount of the sample information, the similarity between the image information in the face area and the sample information is determined. The identification information of the object included in the face area, for example, the product name, is determined.

As the comparison process, not only the entire image information in the face area is matched with the sample information, but also the feature amount of a part of the image information in the face area is extracted, and the feature amount of the sample information and the comparison process are performed. You may do it.

Further, the comparison process may be performed by a method other than the feature amount.

Further, the image recognition processing unit 24 may execute a comparison process with the sample information by using deep learning. In this case, all or part of the image information in the face region is input to the learning model in which the weighting coefficient between neurons in each layer of the neural network consisting of many layers is optimized, and the output value thereof. Similar sample information may be identified based on. As the learning model, it is possible to use image information of various face regions with sample information given as correct answer data.

In the above description, the case where the image recognition processing unit 24 performs image recognition processing on the image information in the face region specified by the image information processing unit 23 has been described, but the captured image is taken without using the image information processing unit 23. The image recognition processing unit 24 may perform image recognition processing on the information.

The verification processing unit 25 determines in the image recognition processing unit 24 as the verification processing for verifying whether or not the identification information of the object included in the face area to be processed is correct, which is determined by the image recognition processing unit 24. The sample information corresponding to the identification information of the object is specified from the sample information storage unit 22, and the comparison process (second comparison process) between the specified sample information and the image information of the face area to be processed is executed. Do by. Further, in the second comparison processing, the comparison processing is executed between the sample information corresponding to the identification information of the object determined by the image recognition processing unit 24 and the image information of the face region to be processed, so that the image recognition processing is performed. Unlike the image recognition process (first comparison process) in the image recognition process unit 24, it is not necessary to execute the comparison process with all the sample information stored in the sample information storage unit 22. The identification information of the objects in the range and the comparison process may be executed. For example, a comparison process may be performed between the sample information of the top three objects determined by the image recognition processing unit 24 and the image information of the face region to be processed. It may be preferable to perform a comparison process between the top-level sample information and the image information of the face region to be processed.

The image recognition process (first comparison process) in the image recognition processing unit 24 and the comparison process (second comparison process) in the verification process are not the same processing method, or the image recognition process (image recognition process) in the image recognition processing unit 24. A processing method is used in which all the elements used for the verification process (second comparison process) in the verification process unit 25 and the elements used for the first comparison process) are not common. An element is a standard used in image recognition processing to determine the similarity of image information.

Specifically, for example, when the two elements of the local feature amount and the EMD are used in the first comparison processing (image recognition processing) in the image recognition processing unit 24, the second comparison processing (verification processing) in the verification processing unit 25 In), a comparison processing method that does not use all of the two elements of local features and EMD is used. More preferably, the same element may not be used in the first comparison process in the image recognition processing unit 24 and the second comparison process in the verification processing unit 25.

For example, as an example of a method for systematizing image features, it is assumed that the elements are classified as shown in FIG. In this case, when the local feature amount and the EMD are used as the elements in the first comparison processing in the image recognition processing unit 24, the elements in the second comparison processing in the verification processing unit 25 are, for example, regions. Use comparison processing that uses the layout information, dominant color, shape and color of the object area as elements.

In this way, the first comparison process in the image recognition processing unit 24 and the second comparison process in the verification processing unit 25 use a processing method that is not the same or a process in which all the elements are not common. The independence of the comparison process can be ensured. Further, the first comparison processing in the image recognition processing unit 24 and the second comparison processing in the verification processing unit 25 should use a processing method having no commonality or a processing method in which most of the elements are not common. By doing so, the independence between the first comparison process and the second comparison process is ensured, and the error included in the first comparison process is minimized to occur in the second comparison process as well. , It is possible to reduce the possibility that an incorrect answer generated in the first comparison process is accidentally turned into a correct answer in the second comparison process.

Further, in the second comparison process, unlike the first comparison process, the comparison process is executed only for the determination result determined by the image recognition processing unit 24, so that the case where a plurality of comparison processes are simply coexisted is not possible. Unlike this, the processing load can be significantly reduced.

The second comparison process is not limited to one, and a plurality of comparison processes may be used.

As for the second comparison process in the verification processing unit 25, various processes can be used as long as the above-mentioned restrictions are satisfied with the first comparison process in the image recognition processing unit 24, and each of them will be described later. An example will be given.

When the verification result processing unit 26 determines that the determination result in the image recognition processing unit 24 is "correct" in the verification processing unit 25, the verification result processing unit 26 directly uses the identification information of the object determined by the image recognition processing unit 24 as the face. It is specified as the identification information of the object reflected in the area, and is stored in a predetermined storage area in association with the face area. Further, when the verification processing unit 25 determines that the determination result in the image recognition processing unit 24 is "error", the notification that the identification information of the object determined by the image recognition processing unit 24 needs to be corrected is notified. To make corrections. For example, there are various notification methods such as marking the face area in the captured image information and highlighting that correction is necessary. The person in charge who confirms this notification visually checks the image information of the face area and makes corrections. When a correction input is received, the identification information of the object for which the input is received is stored in a predetermined storage area in association with the face area.

Next, an example of the processing process using the verification system 1 of the present invention will be described with reference to the flowcharts of FIGS. 8 to 10. FIG. 8 is a flowchart showing an example of a processing process of a verification descriptor generation process which is a pre-process for executing the verification process (second comparison process) in the present embodiment, and FIG. 9 shows the entire process of the present invention. It is a flowchart which shows an example of a processing process, and FIG. 10 is a flowchart which shows an example of the processing process of the verification process (second comparison process) in this Example.

In each of the following examples, a case where a display shelf of a store is photographed and a product on the display shelf is specified will be described. Therefore, the photographed image information is the display shelf of the store where the product is displayed, and the product may be displayed as the object in the sample information.

First, the process of generating the verification descriptor, which is the pre-process, will be described with reference to the flowchart of FIG. The verification descriptor is used to execute verification processing (second comparison processing) between the sample information corresponding to the identification information of the object determined by the image recognition processing unit 24 and the image information of the face region in the captured image information. This is information indicating a standard for determining whether or not two image informations are similar. An example of the verification descriptor is shown in FIG.

The verification descriptor is information about an area (verification area) that characterizes the object, which is set in association with the sample information to be the object, and one or more, preferably a plurality, are set for one sample information. Information about each validation area is represented as a validation key. The verification keys include the position coordinates of the verification area (x-coordinates and y-coordinates of the reference point), the size from the position coordinates (size in the x-axis direction and y-axis direction), and the image for extracting each verification area. The type of filter is associated with some or all of the information such as the information of the parameters to be extracted and the image information of the verification area in some cases. The verification descriptor may be stored in association with the sample information of the object, or may be stored in a different storage device 71.

FIG. 12 shows an example of the verification area set in the sample information. In the sample information of FIG. 12, the area near the nozzle (verification area 1), the area of the logo mark near the center (verification area 2), and the area of the design on the package (verification area 3) are set as verification areas, respectively. .. The verification region in FIG. 12 is a region shown by a solid line, and the region of the broken line portion will be described later.

The operator photographs the product as sample information from the opposite position using the input terminal 3, receives the input of the captured object image information, identification information, product name, and product code on the management terminal 2, and associates them. It is stored in the sample information storage unit 22 (S100). Then, the sample information to be used in the image recognition processing in the image recognition processing unit 24 is extracted and stored. For example, a local feature amount and an EMD are extracted as an image feature amount and stored in the sample information storage unit 22.

Processing for such sample information is performed in advance for a plurality of products, preferably all products that may be displayed on a display shelf.

Further, unnecessary information, for example, a background portion is removed from the image information or the sample information of the object (S110). The background part may be removed automatically or manually. Then, input is accepted by using one or more closed regions that characterize the object as a verification region (closed region of the solid line portion in FIG. 12) (S120). Then, the verification area and the peripheral area near the verification area (the area excluding the verification area from the closed area of the broken line in FIG. 12) are distinguished, and the image filter for extracting each verification area is automatically selected (). S130). The peripheral area can be set as a square-shaped area in which the verification area is expanded outward by a predetermined range, for example, 20%. As an image filter selection process, a plurality of image filters are prepared in advance, and these image filters are sequentially applied to the verification area to search for an image filter that separates the verification area from the peripheral area in the vicinity thereof. By doing so, the image filter to be separated most is selected.

The most separation between the verification region and the region in the vicinity means that the reaction occurs in the widest area inside the verification region and does not react in the outer region near the verification region. Some image filters such as color gamut extraction require parameters, but in that case, the parameters that can be separated are also stored. In the case of FIG. 12, since the shape of the nozzle portion is characteristic in the verification area 1, the applicable binarization image filter is selected, the brightness image filter is selected in the verification area 2, and the verification area 3 is the color gamut extraction. It shows the case where the color corresponding to orange is selected as the parameter in the image filter.

As a method of searching for an image filter that separates the verification area and the peripheral area in the vicinity thereof, as large an area (number of pixels) as possible in the verification area is suitable, and as large as possible area (number of pixels) in the peripheral area is not suitable. It suffices to search for such an image filter and set its parameters. For example, as an image filter and its parameters, as shown in FIG. 27, a list is prepared in advance, the image filters are selected in order, and all combinations are made while changing the values within the range of possible values as parameters. To carry out. If the number of combinations is enormous, the implementation time can be improved by first trying in a large range and then retrying in a smaller range for the range where good results are obtained.

Then, in this trial, the image filter and parameters that have the maximum number of matching pixels in the verification area and the minimum number of matching pixels in the peripheral area are selected. Since there is a trade-off between these two, (the number of conforming pixels in the verification area / the number of pixels in the verification area) is AT, and (the number of non-conforming pixels in the peripheral area / the number of pixels in the peripheral area) is BF. Then, the one that maximizes the harmonic mean of AT and BF may be selected. That is,
I = 2 × (AT × BF) / (AT + BF)
You can select the image filter and parameters that maximize. If the area ratio between the verification area and the peripheral area is 1: 1, the harmonic mean may be calculated as it is without weighting AT and BF, but if the area ratios are different, AT and BF are weighted. To calculate the harmonic mean.

The result detected by applying the image filter in the verification area and the peripheral area may be a so-called missing image in which the internal pixels are not filled. Therefore, it is preferable to perform the expansion / contraction processing of the binary image to fill in the missing pixels, and then calculate the conforming / non-conforming pixels described above. This stabilizes the results and allows the selection of more appropriate image filters and parameters.

As the image filter, known image filters such as applicable binarization, color gamut extraction, and warm color extraction can be used.

When the image filter is selected as described above, the image filter and parameters for extracting the verification area and the area corresponding to the verification area are registered as the verification key in association with the extracted graphic information for each verification area. Then, a verification descriptor that summarizes the verification keys of each verification area is registered in a predetermined storage area including the sample information storage unit 22 by storing the object or the sample information in the object (S140). FIG. 11 is an example of the verification descriptor generated in this way. The extracted graphic information in the verification descriptor of FIG. 11 is a binarized image as a result of extracting the image information in the verification area with an image filter.

By executing the above processing, the verification descriptor generation processing can be executed.

Next, an example of the entire processing process of the present invention will be described with reference to the flowchart of FIG.

When the image information input reception processing unit 20 of the management terminal 2 receives the photographed image information obtained by photographing the display shelf of the store from the input terminal 3 (S200), the image information storage unit 21 receives the photographed date and time and the identification information of the photographed image information. And memorize it in association with.

The image information processing unit 23 executes an orthostatic processing for correcting the received captured image information so that the captured image information faces each other. In addition, face processing is performed to specify the face area from the image information that has been placed upright.

Then, the image recognition processing unit 24 executes an image recognition process (first comparison process) between the face region specified by the image information processing unit 23 and each sample information stored in the sample information storage unit 22 (S210). .. As the first comparison process, for example, by executing a comparison process between the feature amount extracted from the image information of the face area and the feature amount of the sample information, the similarity between the image information of the face area and the sample information is determined. Then, the identification information of the object included in the face area, for example, the product name, is determined. Further, as another method of the first comparison processing, a method using deep learning may be used. In this case, all or part of the image information in the face region is input to the learning model in which the weighting coefficient between neurons in each layer of the neural network consisting of many layers is optimized, and the output value is used. Based on this, similar sample information may be identified.

As a result output by the image recognition processing unit 24, the product identification information of the most similar product as the product reflected in the image information in the face area to be processed is specified, and the product with high similarity is selected first. Product identification information of a plurality of products may be specified in order.

As described above, the image recognition processing unit 24 executes the image recognition processing for each face area of the captured image information.

When the product identification information of a product similar to the product shown in the face area is specified for each face area of the photographed image information, the verification processing unit 25 executes verification processing as to whether or not the product identification information is specified correctly. (S220).

An example of the processing process of the verification processing in the verification processing unit 25 will be described with reference to the flowchart of FIG.

First, unnecessary information, for example, a background portion is removed from the image information of the face area to be processed, and the image information to be processed is cut out (S300). Further, the parameter ACC is set to the initial value 0 (S310).

Then, based on the recognition result of the image recognition processing in the image recognition processing unit 24, the verification descriptor in which the product identification information of the product similar to the product shown in the face area to be processed is associated with the corresponding sample information is generated. Identify. Then, using the information of the verification area of the verification key in the specified verification descriptor and the extraction method, the image information of the area corresponding to the verification area is extracted from the image information cut out in S300 (S320). At this time, the extraction range may be adjusted and retries may be performed so that the position shift due to the rotation of the product or the fluctuation of the shooting angle can be followed. That is, if the extraction fails, the area corresponding to the verification area may be extracted by moving the area to be extracted up / down / left / right or retrying to expand the area.

When the area to be compared is successfully extracted as described above (S330), the degree of matching between the image information obtained by binarizing the image information in the area and the extracted graphic information associated with the verification key is determined. Calculate (S340).

Various methods can be used for the process of calculating the degree of matching between the image information in the extracted area and the extracted graphic information associated with the verification key. For example, the following method can be used. An example of this process is shown in FIG. In FIG. 13, the image information A imitates the image information of the region extracted by S320, and the image information B imitates the extracted graphic information.

First, the corresponding elements are specified in the two image information. Elements that do not have a corresponding element are excluded from processing. Then, for each verification key, the degree of matching between the image information in the extracted area and the extracted graphic information associated with the verification key is calculated by calculating the following mathematical formula.
Match = w1 x Σ (match between corresponding elements) -w2 x Σ (area of non-corresponding elements)

Here, w1 and w2 are predetermined weights and can be set as arbitrary values. W2 can also be set as 0. That is, when it is not appropriate to consider the area of the uncorresponding element, such as when an unrelated object is reflected, the influence can be eliminated by setting w2 as 0.

Further, in the above-mentioned formula for calculating the degree of matching, the degree of matching between the corresponding elements is calculated by matching the scales of the sizes of the two image information and the centers of gravity of the elements in the two profiles. It can be calculated by doing.
Match between corresponding elements = (Area of common part of elements of face profile and sample information profile- (Area of element of face profile but no sample information profile + Area of sample information profile and face Area of the element that is not the profile of) / 2) / Area of the profile of the face

At this time, in order to correspond to the inclination of the product (the inclination of the element of the image information), the above formula is calculated while rotating one of them at a minute angle, and the one with the largest value is adopted. good. Further, it may be possible to emphasize the one in which the number of corresponding element sets is large in the two profiles.

The degree of matching calculated as described above is added to the parameter ACC (S350). If the verification descriptor has another verification key, the processes from S320 to S350 are repeated for the remaining verification keys (S360).

Then, when the above processing is executed for all the verification keys in the verification descriptor associated with the product identification information of the product similar to the product shown in the face area to be processed and the corresponding sample information, the parameter ACC and the parameter ACC are executed in advance. Compared with the determined threshold (S370), if the parameter ACC is equal to or greater than the threshold, the verification processing unit 25 performs image recognition processing in the image recognition processing unit 24 for the product shown in the face region to be processed. The product identification information of the specified product is determined to be "correct answer" (S380). On the other hand, if the parameter ACC is less than the threshold value, the verification processing unit 25 describes the product identification information of the product specified by the image recognition processing in the image recognition processing unit 24 for the product shown in the face area to be processed. It is determined as "error" (S390).

If there is a plurality of product identification information of the product specified by the image recognition processing in the image recognition processing unit 24, the verification processing is executed for all the product identification information, and the product identification information of the product determined to be "correct" is executed. Of these, the product identification information of the product specified with the highest accuracy or the highest order in the image recognition processing is specified as the product identification information of the product shown in the face area to be processed.

By executing the above verification processing for each face area in the captured image information, the result of the image recognition processing in the image recognition processing unit 24 is verified.

The verification process can be executed as described above.

Then, for the face area determined to be the correct answer, the verification result processing unit 26 identifies the identification information of the object determined by the image recognition processing unit 24 as the identification information of the object reflected in the face area as it is. It is stored in a predetermined storage area in association with the face area (S250).

On the other hand, when the verification processing unit 25 determines that the determination result in the image recognition processing unit 24 is "error", the verification result processing unit 26 corrects the identification information of the object determined by the image recognition processing unit 24. Is required, and the operator is made to make corrections (S260). When the operator confirms this notification, he / she visually checks the image information of the face area and makes a correction input. When a correction input is received, the identification information of the object for which the input is received is stored in a predetermined storage area in association with the face area.

In this way, the image recognition processing unit 24 executes the image recognition processing (first comparison processing) for the face region in the captured image information, and the verification processing unit 25 verifies the recognition result (second comparison processing). By executing the comparison process), the processing load can be significantly reduced as compared with the case where a plurality of image recognition processing systems are used in parallel as in the conventional case.

Next, a case where a verification process different from the verification process of the first embodiment is used in the verification system 1 of the present invention will be described. An example of the processing process of the verification process in the second embodiment will be described with reference to the flowchart of FIG. In the verification process in this embodiment, a case where a simpler setting is provided as a verification descriptor will be described.

In this embodiment, local features in the image information of the face region are extracted, and whether or not the preset conditions (conditions set in the verification descriptor) are satisfied is verified by the verification process in the verification processing unit 25. ..

The verification descriptor is characteristic information that categorizes the part of the sample information that characterizes the object. The verification descriptor stores one or more feature information in association with the superiority (weighting). The feature information has an object that is a part that characterizes the object in the sample information, an "attribute" that indicates the color and outer shape of the object, and an "attribute value" that indicates the specific color and shape of the attribute. There is. For example, if the object in the sample information is a PET bottle of beverage, "cap" is set as the "object" in "characteristic information", "color" is set as the attribute, and "white" is set as the attribute value. .. One or more feature information is set as a verification descriptor for one object. The superiority is information that ranks the verification in the verification descriptor, but the superiority does not have to be set.

14 and 15 show an example of the verification descriptor in this embodiment. FIG. 14 shows an example of a verification descriptor for a PET bottled beverage. In the verification descriptor of FIG. 14, one characteristic information is set for the outer shape of the PET bottle of the beverage, two characteristic information for the mark, and two characteristic information for the cap, for a total of five characteristic information. Attributes, attribute values and superiority are set. In FIG. 14, the attribute "bottle shoulder", its attribute value "Ikari shoulder", and the priority (weight) "0.4" are set for the feature information "bottle outer shape", and the "attribute" of the feature information "mark" is set. "Layout" and its attribute "center on the label" and priority "0.3" are set for, and the attribute "base color" and its attribute value "white" and priority "0." for the feature information "mark". 1 ”is set, the attribute“ color ”and its attribute value“ white ”, and the priority“ 0.1 ”are set for the feature information“ cap ”, and the attribute“ shape ”is set for the feature information“ cap ”. The attribute value "standard" and priority "0.1" are set.

FIG. 15 shows an example of a verification descriptor for a PET bottled beverage of a product different from that of FIG. Also in the verification descriptor of FIG. 15, a total of five characteristic information is set, one characteristic information for the outer shape of the PET bottle of the beverage, two characteristic information for the mark, and two characteristic information for the cap. Attributes, attribute values and superiority are set. In FIG. 15, the attribute "bottle shoulder", its attribute value "stroke shoulder", and the priority (weight) "0.4" are set for the feature information "bottle outer shape", and the "attribute" of the feature information "mark" is set. "Layout" and its attribute "left on the label" and priority "0.3" are set for, and the attribute "base color" and its attribute value "white" and priority "0." for the feature information "mark". 1 ”is set, the attribute“ color ”and its attribute value“ white ”, and the priority“ 0.1 ”are set for the feature information“ cap ”, and the attribute“ shape ”is set for the feature information“ cap ”. The attribute value "standard" and priority "0.1" are set.

The verification descriptor is stored in association with the sample information of the object stored in the sample information storage unit 22. In the verification descriptor, the same attributes should be set for the same type of product, for example, PET bottles for beverages, as the attributes in each characteristic information.

An example of the processing process of the verification process of this embodiment will be described with reference to the flowcharts of FIGS. 8, 10 and 16. Since the processing up to S210 is the same as that in the first embodiment, the description thereof will be omitted. Further, the verification process of this embodiment is schematically shown in FIG.

First, unnecessary information, for example, a background portion is removed from the image information of the face area to be processed, and the image information to be processed is cut out (S400). Further, the parameter ACC is set to the initial value 0 (S410).

Then, based on the recognition result of the image recognition processing in the image recognition processing unit 24 of S210, the verification that the product identification information of the product similar to the product shown in the face area to be processed is associated with the corresponding sample information. The descriptor is specified (S420).

The image information cut out from the image in the face area in S400 and the feature information having the highest superiority among the verification descriptors specified in S420 are compared in order (S430), and the attributes of each target in the feature information and their attribute values are compared. , It is determined whether the image information is satisfied (S440). Then, when the attribute and the attribute value are satisfied, the superiority corresponding to the feature information is added to the parameter ACC (S450).

If the parameter ACC is equal to or higher than the preset threshold value, the verification processing unit 25 can obtain the product identification information of the product specified by the image recognition processing in the image recognition processing unit 24 for the product shown in the face area to be processed. It is determined as "correct answer" (S470). On the other hand, when the parameter ACC is less than the threshold value and the verification descriptor still has other feature information (S480), the feature information having the next highest priority among the verification descriptors and the face region in S400 are used. The image information cut out from the image is compared (S430), and the processing after S430 is repeated.

Then, as a result of executing the determination processing of S460 up to the feature information having the lowest superiority in the verification descriptor, if the parameter ACC is not equal to or higher than the threshold value, the processing for all the feature information in the verification descriptor is executed. Therefore, the verification processing unit 25 determines that the product identification information of the product specified by the image recognition processing in the image recognition processing unit 24 is "error" for the product reflected in the face area to be processed (S490). ..

By executing the above verification processing for each face area in the captured image information, the result of the image recognition processing in the image recognition processing unit 24 is verified.

The verification process can be executed as described above. Since the processing after the verification processing (S220) is the same as that after S230, the description thereof will be omitted.

In the above explanation, the superiority corresponding to the feature information satisfying the condition is added to the parameter ACC, and when the parameter ACC exceeds the preset threshold value, the process is determined as "correct answer". Instead of adding the superiority to the ACC, the number of feature information satisfying the conditions may be added. In this case, when the parameter ACC becomes equal to or more than a preset threshold value (threshold value for the number of feature information satisfying the condition), it may be configured to be determined as "correct answer".

Further, in the above-mentioned S440, when determining whether the attribute and the attribute value in the feature information of the verification descriptor are satisfied, the superiority is added if the image information is satisfied, and if the image information is not satisfied, the superiority is added. The processing of the next feature information was performed as it was, but it is not always possible to find the feature information in the image information. Therefore, for the attributes and their attribute values in the feature information of the verification descriptor, the superiority is added if the image information is satisfied, the superiority is subtracted if the image information is not satisfied, and the feature information itself is used in the image information. If it cannot be found, the next feature information may be processed as it is, and so on, in three steps.

Any calculation can be adopted for the calculation regarding whether or not the condition is satisfied and whether or not the feature information is found in S440, and the calculation is not limited to the above.

For the verification descriptor in this embodiment, the target, the attribute, and the attribute value may be set to the extent that the verification descriptor is classified from another product of the same series.

Next, a case where a verification process different from the verification processes of the first and second embodiments is used in the verification system 1 of the present invention will be described. In the verification process in the third embodiment, the process when the profile data is used will be described. Profile data is data that expresses a region constituting an object in image information and a portion that represents the shape. As shown in FIG. 18, the profile data is data showing what kind of shape exists at which position on the image.

The profile data is stored in the sample information storage unit 22 in association with the sample information. The profile data has at least one component, which is an "original file" showing sample information or image information of the face area. The components include the position of a predetermined vertex (for example, upper left) of the closed area, the position and area of the center of gravity of the closed area, the file name of the binary image of the closed area, and the like.

First, an example of a processing process for generating profile data corresponding to sample information will be described with reference to the flowchart of FIG.

The operator photographs the product as sample information from the opposite position using the input terminal 3, receives the input of the captured object image information, identification information, product name, and product code on the management terminal 2, and associates them. It is stored in the sample information storage unit 22 (S500). Then, the sample information to be used in the image recognition processing in the image recognition processing unit 24 is extracted and stored. For example, a local feature amount and an EMD are extracted as an image feature amount and stored in the sample information storage unit 22.

Processing for such sample information is performed in advance for a plurality of products, preferably all products that may be displayed on a display shelf.

Further, the image information or the sample information of the object is reduced to a predetermined number of colors, for example, 5 colors or 10 colors or less (S510). This can be done, for example, by specifying dominant colors (dominant colors) of a predetermined number of colors for the image information and then replacing each pixel with those colors.

The color-reduced image information is divided into continuous regions of the same color to divide the image information into regions for each color (S520). Then, an area where the area is smaller than a predetermined size (area that is too small), an area where the total area is too large (such as the ground color of the object), an area where the degree of adhesion to the edge of the image information is high (background part), and the center of gravity. Excludes areas such as areas contained inside other elements (enclosed figures with an internal design) and areas that are thin and severely bent (delicate character strings), and the remaining areas are specified as components. (S530). Then, profile data is generated by adding the shape information (position of a predetermined vertex of the closed region, the position of the center of gravity, the area) and the binary image to the profile data for each component that is the remaining region. (S540).

FIG. 20 schematically shows an example of the image information (object image information) of the product as the sample information and the corresponding profile. FIG. 20A is an example of object image information, and FIG. 20B is an example of a profile corresponding to the sample information. The profile data shows the positions of each component in the profile of FIG. 20B as text data.

The verification process in the verification process unit 25 is executed using the profile data generated as described above.

An example of the processing process of the verification process of this embodiment will be described with reference to the flowcharts of FIGS. 8, 10, 21, and 22. Since the processing up to S210 is the same as that in the first embodiment, the description thereof will be omitted.

Profile data is generated by executing the same processing as in FIG. 19 from the image information of the face area to be processed. FIG. 21 shows a flowchart showing an example of a processing process for generating profile data from image information in the face region.

First, the image information in the face region is reduced to a predetermined number of colors, for example, 5 colors or less (S600). An example of the image information of the face region is shown in FIG. 23 (a), and an example of the image information of the reduced color of the face region is shown in FIG. 23 (b).

The color-reduced image information is divided into continuous regions of the same color to divide the image information into regions for each color (S610). Then, an area where the area is smaller than a predetermined size (area that is too small), an area where the total area is too large (such as the ground color of the object), an area where the degree of adhesion to the edge of the image information is high (background part), and the center of gravity. Excludes areas such as areas that are contained inside other elements (enclosed figures with an internal design) and areas that are thin and severely bent (delicate character strings), and specify the remaining areas as components. (S620). Then, profile data is generated by adding the shape information (position of a predetermined vertex of the closed region, the position of the center of gravity, the area) and the binary image to the profile data for each component that is the remaining region. (S630). FIG. 24 shows an example of the profile generated for the image information in the face region of FIG. 23 (a).

Using the profile based on the image information of the face area generated as described above, and the profile in which the product identification information of the product similar to the product reflected in the face area to be processed is associated with the corresponding sample information. Then, by evaluating the similarity, the verification processing unit 25 executes the verification processing (S230). This process is schematically shown in FIG. In FIG. 25, the left side is an example of a profile associated with the sample information, and the right side is an example of a profile based on the image information of the face area. In FIG. 25, the components are shown by circles for the sake of simplification of the explanation, but in reality, a closed region is sufficient and the shape is indeterminate.

First, among the components in each of the sample information profile and the face area profile, two or more components are specified from the components having a large area. For example, the components having the top four areas are specified (S700). The component specified here is referred to as the "core". FIG. 25 (a) schematically shows this state.

Then, for each profile, among the core components specified in S700, adjacent components are connected by the centers of gravity. At this time, concatenation is performed according to predetermined rules such as counterclockwise rotation and descending priority so that the rules are fixed. FIG. 25 (b) schematically shows this state. A constellation is a combination of adjacent components connected by their centers of gravity.

Consistency is evaluated to indicate whether the patterns of adjacent components (constellation patterns) match the profile of the sample information and the profile of the face region (S720). That is, in each profile, the similarity of the polygonal line generated by connecting the centers of gravity of the connected components is evaluated. This can be calculated by integrating the similarity of the length ratio with the previous line segment and the similarity of the angle each time the polygonal line bends. If the similarity is equal to or higher than the preset threshold value, it is determined that they match, and if it is less than the threshold value, it is determined that they are not similar.

If it is determined that they are similar, one of the profiles is scaled up or down so that both profiles are overlapped, and the scale and phase are matched (S730). This may be expanded or contracted so that the total extension distance of the polygonal lines in each profile is the same, or the centers of gravity of the polygonal lines may be translated so as to coincide with each other.

As described above, for the components other than the core profile specified in S700, it is specified whether or not there is a corresponding component in each profile, and after the specification, the component is added to the constellation (S740). That is, among the one profile, the components that are not yet in the constellation are identified, the one with the closest center of gravity of the element of the other profile is searched within a certain distance from the center of gravity, and if there is the closest one. ， Identify as having a corresponding component. FIG. 25 (c) schematically shows this state. The components added to the constellation are connected to the centers of gravity of adjacent components. Further, FIG. 26 schematically shows the correspondence between the sample information of FIG. 20 and its profile, the image of the face region of FIG. 23, and the respective components in the profile of FIG. 24.

Then, a process of calculating the similarity from the correspondence between the components of each profile is executed (S750). The similarity calculation process can be calculated by calculating the following formula.
Similarity = w1 x Σ (matching degree between corresponding elements) -w2 x Σ (area of non-corresponding elements)

Here, w1 and w2 are predetermined weights and can be set as arbitrary values. W2 can also be set as 0. That is, when it is not appropriate to consider the area of the uncorresponding element, such as when an unrelated object is reflected, the influence can be eliminated by setting w2 as 0.

Further, in the above-mentioned formula for calculating the similarity, the degree of matching between the corresponding elements is calculated by matching the scales of the sizes of the two profiles and the centers of gravity of the components in the two profiles. It can be calculated by doing.
Matching degree between corresponding elements = (Area of common part of component of face profile and sample information profile- (Area of component of face profile but no sample information profile + sample information profile Area of components that are not the profile of the face) / 2) / Area of the profile of the face

At this time, in order to correspond to the inclination of the product (the inclination of the element of the image information), the above formula is calculated while rotating one of them at a minute angle, and the one with the largest value is adopted. good. Further, the two profiles may be emphasized when the number of corresponding component sets is large.

After calculating the similarity as described above, if the similarity is equal to or higher than a predetermined value (S760), the verification processing unit 25 determines that the recognition result in the image recognition processing in the image recognition processing unit 24 is "correct". (S770). On the other hand, if the calculated similarity is less than a predetermined value (S760), the verification processing unit 25 determines that the recognition result in the image recognition processing in the image recognition processing unit 24 is “error” (S780).

By executing the above verification processing for each face area in the captured image information, the result of the image recognition processing in the image recognition processing unit 24 is verified.

The verification process can be executed as described above. Since the processing after the verification processing (S220) is the same as that after S230, the description thereof will be omitted.

The processing in each of the above-mentioned examples is not limited to the order described in the specification of the present invention, and can be appropriately changed as long as the object is achieved.

Further, when comparing each value, even if "greater than or equal to" and "less than" are described in the examples, the comparison conditions may be appropriately changed to "greater than or equal to" and "less than or equal to".

Further, the verification system 1 of the present invention is effective when specifying the display status of a product displayed on the display shelf from the photographed image information obtained by photographing the display shelf of the store, but the present invention is limited to this. It's not something to do. That is, when a certain object to be photographed is photographed, it can be widely used when specifying the area in which the desired object is captured from the photographed image information.

By using the verification system 1 of the present invention, it is possible to save labor in the verification of the recognition result in the image recognition processing system. Therefore, the conventional manual work can be significantly reduced or eliminated.

1: Verification system 2: Management terminal 3: Input terminal 20: Image information input reception processing unit 21: Image information storage unit 22: Sample information storage unit 23: Image information processing unit 24: Image recognition processing unit 25: Verification processing unit 26 : Verification result processing unit 70: Arithmetic device 71: Storage device 72: Display device 73: Input device 74: Communication device

The first invention is a second aspect of an image recognition processing unit that executes a first comparison process for a part or all areas of image information, and a part or all of recognition results by the image recognition processing unit. It is a verification system having a verification processing unit that executes comparison processing and a sample information storage unit that stores sample information of an object, and the sample information storage unit is information indicating a verification area that characterizes an object. The verification processing unit stores at least one verification descriptor having at least one verification key including the information of the type of the image filter for extracting the verification area and the image information of the verification area for each object. The verification descriptor corresponding to a part or all of the objects specified in the first comparison process is specified, and each verification key in the specified verification descriptor is processed by using the information of the verification key. It is determined whether the area corresponding to the verification area can be extracted from the target area, and if the area can be extracted, the degree of matching between the image information of the extracted area and the image information of the verification key is determined. Calculated, the calculated degree of agreement is compared with a predetermined threshold, and if the comparison result satisfies the predetermined condition, the result of the first comparison process is regarded as correct and the object is reflected in the area to be processed. It is a verification system that identifies things as objects.

The seventh invention is a second aspect of an image recognition processing unit that executes a first comparison process for a part or all areas of image information, and a part or all of recognition results by the image recognition processing unit. It is a verification system having a verification processing unit that executes the comparison processing of the object and a sample information storage unit that stores the sample information of the object, and the sample information storage unit categorizes the part that characterizes the object. At least one or more feature information and superiority are stored as a verification descriptor for each object, and as the feature information, the target that is the part that characterizes the object, the attribute of the target, and the attribute value for the attribute are stored. The verification processing unit has specified the verification descriptor corresponding to a part or all of the objects specified in the first comparison process, and each feature information in the specified verification descriptor. It is determined whether the area to be processed satisfies the attributes and attribute values in the feature information in the order of the feature information having the highest degree of superiority, and if the predetermined conditions are satisfied, the first comparison is made. It is a verification system that identifies the object in the area to be processed as the result of processing is correct.

The eighth invention is an image recognition processing unit that executes a first comparison process for a part or all areas of image information, and a second aspect for a part or all of recognition results by the image recognition processing unit. It is a verification system having a verification processing unit that executes comparison processing and a sample information storage unit that stores sample information of an object, and the sample information storage unit represents the shape of an object in image information. At least one profile data including information on a partial area is stored for each object, and the verification processing unit reduces the color of the area to be processed to a predetermined number of colors and colors the reduced image information. By dividing into each area and specifying the specified area as a component, the profile data of the area to be processed is generated, and the profile data of the area to be processed is compared with the first comparison process. The similarity is compared with the profile data of one or all of the objects specified in the above, and if the similarity satisfies a predetermined condition, the result of the first comparison process is assumed to be correct. This is a verification system that identifies an object in the area to be processed.

The first invention can be realized by loading and executing the program of the present invention in a computer. That is, at least one verification descriptor having at least one verification key including information indicating the verification area that characterizes the object, information on the type of image filter that extracts the verification area, and image information of the verification area is provided for each object. The computer that stores the above is the first for an image recognition processing unit that executes the first comparison processing for a part or all areas of image information, and for a part or all of the recognition results by the image recognition processing unit. A verification program that functions as a verification processing unit that executes the comparison processing of 2, wherein the verification processing unit corresponds to a part or all of the objects specified in the first comparison processing. The verification descriptor is specified, and for each verification key in the specified verification descriptor, it is determined whether the area corresponding to the verification area can be extracted from the area to be processed by using the information of the verification key, and the area is extracted. If it can be done, the degree of matching between the image information of the extracted area and the image information of the verification key is calculated, the calculated degree of matching is compared with a predetermined threshold, and the comparison result satisfies the predetermined condition. In this case, it is a verification program that identifies the object as being reflected in the area to be processed, assuming that the result of the first comparison process is correct.

The seventh invention can be realized by loading and executing the program of the present invention in a computer. That is, a computer that stores at least one or more objects for each object is used as a verification descriptor using feature information that categorizes the parts that characterize the object and the superiority, and is the first for a part or all of the area in the image information. It is a verification program that functions as an image recognition processing unit that executes the comparison processing of the above, and a verification processing unit that executes a second comparison processing for a part or all of the recognition results by the image recognition processing unit. As information, it has an object that is a part that characterizes the object, the attribute of the object, and the attribute value for the attribute, and the verification processing unit is a part of the object specified in the first comparison process. Alternatively, the verification descriptor corresponding to all the objects is specified, and for each feature information in the specified verification descriptor, the area to be processed is the attribute and attribute in the feature information in the order of the feature information having the highest degree of superiority. It is a verification program that determines whether a value is satisfied, and if a predetermined condition is satisfied, identifies the object as being reflected in the area to be processed, assuming that the result of the first comparison process is correct. ..
The eighth invention can be realized by loading and executing the program of the present invention in a computer. That is, a computer that stores at least one profile data including information of a partial region representing the shape of the object in the image information for each object is first for a part or all of the regions in the image information. It is a verification program that functions as an image recognition processing unit that executes the comparison processing of the above, and a verification processing unit that executes a second comparison processing for a part or all of the recognition results by the image recognition processing unit. The processing unit reduces the color of the area to be processed to a predetermined number of colors, divides the reduced image information into areas for each color, and specifies the specified area as a component, thereby making it a processing target. The profile data of the area to be processed is generated, and the similarity is compared between the profile data of the area to be processed and the profile data of one or all the objects specified in the first comparison process. However, when the similarity satisfies a predetermined condition, it is a verification program that identifies the object as being reflected in the area to be processed, assuming that the result of the first comparison processing is correct.

Claims (18)

An image recognition processing unit that executes the first comparison processing for a part or all of the image information, and
A verification system having a verification processing unit that executes a second comparison processing on a part or all of the recognition results by the image recognition processing unit.
The verification processing unit
As the second comparison process, a processing method that is not the same as the first comparison process, or a process in which all the elements used in the first comparison process and the elements used in the second comparison process are not common. Execute processing by method,
A verification system characterized by that. The verification processing unit
As the second comparison process, a process method different from that of the first comparison process, or a process in which all the elements used in the first comparison process and the elements used in the second comparison process are different. Run,
The verification system according to claim 1, wherein the verification system is characterized in that. The verification system is
A sample information storage unit that stores sample information of an object,
It has a verification result processing unit that executes processing for the determination result of the second comparison processing in the verification processing unit.
The first comparison process in the image recognition processing unit is
By comparing the area to be processed in the image information with the sample information in the sample information, the object reflected in the area to be processed is identified.
The second comparison processing in the verification processing unit is
When the comparison result between the corresponding information of a part or all of the objects specified in the first comparison process and the area to be processed satisfies the predetermined condition, the first comparison is performed. Judging that the processing result is correct,
The verification result processing unit
If the verification processing unit determines that the result is correct, the result of the first comparison processing is specified as an object reflected in the area to be processed.
The verification system according to claim 1 or 2, wherein the verification system is characterized in that. The second comparison processing in the verification processing unit is
If the comparison result does not satisfy the predetermined conditions, it is determined that the result of the first comparison process is an error.
The verification result processing unit
If the verification processing unit determines that an error has occurred, a predetermined notification is given.
The verification system according to claim 3, wherein the verification system is characterized in that. The sample information storage unit is
At least one verification descriptor containing at least one verification key including information indicating the verification area that characterizes the object, information on the type of image filter that extracts the verification area, and image information of the verification area is stored. And
The verification processing unit
The verification descriptor corresponding to a part or all of the objects specified in the first comparison process is specified.
For each verification key in the specified verification descriptor, it is determined whether the area corresponding to the verification area can be extracted from the area to be processed by using the information of the verification key, and if the area can be extracted, the area can be extracted. , Calculate the degree of matching between the image information of the extracted area and the image information of the verification key.
The calculated degree of agreement is compared with a predetermined threshold value, and if the comparison result satisfies the predetermined condition, the result of the first comparison process is regarded as correct and the object is specified as an object reflected in the area to be processed. do,
The verification system according to any one of claims 1 to 4, wherein the verification system is characterized in that. The verification processing unit
The verification descriptor corresponding to a part or all of the objects specified in the first comparison process is specified.
For each verification key in the specified verification descriptor, it is determined whether or not the area corresponding to the verification area in the verification key can be extracted from the area to be processed by using the image filter in the verification key. If the area can be extracted, the degree of matching between the extracted area and the image information of the verification key is calculated.
The calculated degree of agreement is compared with a predetermined threshold value, and if the comparison result satisfies the predetermined condition, the result of the first comparison process is regarded as correct and the object is specified as an object reflected in the area to be processed. do,
The verification system according to claim 5, wherein the verification system is characterized in that. The verification processing unit
From the image information or sample information of the object to be the sample information, the closed area that characterizes the object is accepted as the verification area.
Select an image filter that distinguishes between the verification area and the peripheral area for the verification area.
The image information in the verification area is stored in the sample information storage unit as information in the verification key.
The verification system according to claim 5 or 6, wherein the verification system is characterized in that. The verification processing unit is one of the image filters registered in advance.
An image filter is selected by searching for an image filter that fits in the widest area inside the verification area and does not fit in the widest area of the peripheral area.
The verification system according to claim 7, wherein the verification system is characterized in that. The verification processing unit is one of the image filters registered in advance.
An image filter that maximizes the harmonic mean of AT and BF when the number of conforming pixels in the verification area / the number of pixels in the verification area is AT and the number of non-conforming pixels in the peripheral area / the number of pixels in the peripheral area is BF. To select,
The verification system according to claim 7, wherein the verification system is characterized in that. After the binary image is expanded or contracted for the verification region and the peripheral region, an image filter that distinguishes the verification region from the peripheral region with respect to the verification region is selected.
The verification system according to any one of claims 7 to 9, wherein the verification system is characterized in that. The sample information storage unit is
At least one or more of the characteristic information categorizing the parts that characterize the object and the superiority are stored as verification descriptors for each object.
The feature information includes an object that is a part that characterizes the object, an attribute of the object, and an attribute value for the attribute.
The verification processing unit identifies the corresponding verification descriptor of a part or all of the objects specified in the first comparison process.
For each feature information in the specified verification descriptor, it is determined whether the area to be processed satisfies the attributes and attribute values in the feature information in the order of the feature information having the highest degree of superiority.
When a predetermined condition is satisfied, the result of the first comparison process is regarded as correct and the object is specified as an object reflected in the area to be processed.
The verification system according to any one of claims 1 to 4, wherein the verification system is characterized in that. The sample information storage unit is
At least one profile data including information on a partial area representing the shape of the object in the image information is stored for each object.
The verification processing unit
The area to be processed is reduced to a predetermined number of colors, and the color is reduced.
By dividing the color-reduced image information into areas for each color and specifying the specified area as a component, profile data of the area to be processed is generated.
Similarities are compared using the profile data of the region to be processed and the profile data of one or all of the objects specified in the first comparison process.
When the similarity satisfies a predetermined condition, the result of the first comparison process is regarded as correct and the object is specified as an object reflected in the area to be processed.
The verification system according to any one of claims 1 to 4, wherein the verification system is characterized in that. The verification processing unit
Of the components of the profile data of the area to be processed, a predetermined number of components having a large area and the components of the profile data of one or all of the objects specified in the first comparison process. Identify a predetermined number of components with a large area as the core, and identify them as the core.
In each of the profile data, the similarity was calculated using the pattern in which the adjacent components of the core components were connected by the centers of gravity.
When the calculated similarity satisfies a predetermined condition, among the components not included in the pattern, the corresponding component is specified by each of the profile data and added to the pattern.
When the similarity is calculated from the correspondence between the components in the pattern of the profile data and the similarity satisfies a predetermined condition, the result of the first comparison process is regarded as correct and the processing target is Identify as an object in the area to be
The verification system according to claim 12, wherein the verification system is characterized in that. The verification processing unit
The image information or sample information of the object is reduced to the specified number of colors,
The color-reduced image information is divided into areas for each color, and the specified area is specified as a component.
For each of the specified components, the information indicating the region of the shape and the image information of the region are stored in the sample information storage unit as profile data.
12. The verification system according to claim 12 or 13. By executing the first comparison process using the sample information of the product stored in the sample information storage unit for the face area specified from the image information obtained by photographing the display shelf, the object reflected in the face area is executed. Image recognition processing unit that identifies
A verification system including a verification processing unit that executes a second comparison processing for a part or all of the recognition results by the image recognition processing unit.
The verification processing unit
As the second comparison process, a process that is not the same as the image recognition process, or a process in which all the elements used in the first comparison process and the elements used in the second comparison process are not common is executed. ， ，
A verification system characterized by that. The second comparison processing in the verification processing unit is
The information of a part or all of the objects specified in the first comparison process is compared with the face area.
When the comparison result satisfies a predetermined condition, the result of the first comparison process is regarded as correct and the object is specified as an object reflected in the face area.
The verification system according to claim 15, wherein the verification system is characterized in that. Computer,
Image recognition processing unit that executes the first comparison processing for a part or all of the area in the image information,
It is a verification program that functions as a verification processing unit that executes a second comparison processing on a part or all of the recognition results by the image recognition processing unit.
The verification processing unit
As the second comparison process, a process that is not the same as the image recognition process, or a process in which all the elements used in the first comparison process and the elements used in the second comparison process are not common is executed. ， ，
A verification program characterized by that. Computer,
By executing the first comparison process using the sample information of the product stored in the predetermined storage area for the face area specified from the image information obtained by photographing the display shelf, the object reflected in the face area. Image recognition processing unit, which identifies
It is a verification program that functions as a verification processing unit that executes a second comparison processing for a part or all of the recognition results by the image recognition processing unit.
The verification processing unit
As the second comparison process, a process that is not the same as the image recognition process, or a process in which all the elements used in the first comparison process and the elements used in the second comparison process are not common is executed. ， ，
A verification program characterized by that.

Images