JP6966749B2 - Image processing system - Google Patents

Description

The present invention relates to an image processing system.

It may be required to determine whether or not an object to be determined is reflected in a certain image information. For example, there is a case where it is determined whether or not the product is displayed by determining whether or not the product to be determined is reflected in the image information obtained by photographing the display shelf of the product.

In this way, when determining whether or not an object is reflected in the image information, it is common to execute an image matching process between the image information to be processed and the image information of the object. For example, Patent Document 1 below discloses a system for grasping products handled by vending machines by using image recognition technology for image information taken by vending machines based on registered images for each product. Has been done.

Japanese Unexamined Patent Publication No. 2014-191423

The specific processing of Patent Document 1 is to shoot a vending machine from a plurality of directions, align the positional relationships of the shot images, and then superimpose the shot images to generate a composite image. Then, the product handled by the vending machine is specified by collating the generated composite image with the registered image representing the product that may be displayed on the vending machine.

In the collation process (matching process) between the composite image and the registered image representing the product, it is common to execute the process by using each feature amount. However, when the number of products to be compared increases, the number of registered images increases, and the accuracy of the collation process (matching process) cannot be improved. In addition, there is a problem that the calculation time is enormous.

Therefore, in view of the above problems, the present inventor improves the accuracy of the matching process without requiring more calculation time than before when identifying an object such as a product shown in image information obtained by photographing a display shelf or the like. Invented an image processing system.

The first invention is an image processing system for identifying the product that is reflected in the photographed image information display shelf, the image processing system is an area satisfying a predetermined condition in the face region in the image information block by using the layout information, and layout information of the block in the item to the sampling of the a judgment of similarity of layout between blocks, and the candidate processor for identifying a product to be a candidate of the comparison process, the image information a face region of the, by the comparison processing using the sample information items identified as candidates by the candidate processor has a comparison unit for identifying the product that is reflected in the face region , The candidate processing unit calculates a predetermined evaluation value using the area of the common part and / or the difference between the block layout information in the face area of the image information and the layout information of the block of the sampled product. This is an image processing system that specifies the sampled product as a candidate when the calculated evaluation value is equal to or higher than the threshold value.

By configuring as in the present invention, the number of objects to be compared in the comparison process (matching process) can be reduced, so that the calculation time can be reduced as compared with the conventional case.

Further, in order to narrow down the objects to be candidates for the comparison processing, it is preferable to execute the processing as in the present invention. Moreover, since the layout information is not image information but coordinate information, high speed can be realized even if these processes are executed.

In the invention described above, the candidate processing unit, the identifies blocks that satisfy a predetermined condition in the face region in the image information, and the block layout information in a predetermined range of the specified face area, the image of the product to the specimen The information can be configured like an image processing system that determines the similarity of layouts by using the layout information of the blocks of the area corresponding to the size of the predetermined range of the face area.

The surface of the object is not limited to a flat surface, but may be a curved surface such as a cylinder. When the surface of the object is a curved surface, the image information obtained by photographing the object becomes more distorted as it moves away from the center in the left-right direction. Therefore, by cutting out the vicinity of the central part of the first region and performing the matching processing of the layout information with the size corresponding to the vicinity of the cut out central part in the image information of the object to be sampled, the surface of the object can be obtained. The effect of distortion due to the curved surface can be reduced.

In the above-mentioned invention, the candidate processing unit performs the process of determining the similarity of the layout while shifting the area corresponding to the size of the predetermined range of the face area in the image information of the product as the sample. The comparison processing unit performs comparison processing using the sample information of the region determined to have the highest similarity in the candidate processing unit and the face region of the image information in the image information of the product. It can be configured like an image processing system that identifies the product in the face area.

When cutting out the vicinity of the central part, the accuracy of the comparison processing can be improved by performing the comparison processing while shifting it and then using the sample information of the region judged to have the highest similarity.

In the above invention, the comparison processing unit, said the block of image information, correspondence and a block of the specified item as a candidate in the candidate processing unit, by using the correspondence block, image information and / or character By determining the similarity of information, it can be configured like an image processing system that executes the comparison processing.

In the comparison process, blocks may be compared with each other as in the present invention.

A fifth invention is an image processing system that identifies a product shown in image information obtained by photographing a display shelf, and the image processing system obtains image information obtained by photographing the display shelf and corresponding depth information. Image information processing that generates plane-expanded image information to be expanded on a plane for a part or all of the three-dimensional model generated in use, and uses part or all of the generated plane-expanded image information as image information to be processed. parts and the layout information of the block is an area satisfying a predetermined condition in the face region in the image information of the processing target, by using the block layout information in the object to be sampled, the layout of the similarity of the blocks to each other Comparison processing is performed using the candidate processing unit that identifies the product as a candidate for comparison processing, the face area of the image information to be processed, and the sample information of the product specified as a candidate by the candidate processing unit. This is an image processing system having a comparison processing unit for identifying a product appearing in the face area.

The image information to be processed may be image information in which the image information obtained by photographing a display shelf or the like is handled as it is in two dimensions, or image information in which a three-dimensional model is once generated and expanded in a plane. As a result, the processing can be executed more accurately.

The first invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer uses the layout information of the block, which is an area satisfying a predetermined condition in the face area of the image information obtained by photographing the display shelf, and the layout information of the block in the sample product , to form the layout of the blocks. a determination is similarity, candidate processing unit for identifying the product to be a candidate of the comparison process, the comparison process using the face region of the image information, and sample information items identified as candidates by the candidate processing unit This is an image processing program that functions as a comparison processing unit that identifies the products in the face area , and the candidate processing unit uses the layout information of the blocks in the face area of the image information to be processed and the layout information of the blocks. A predetermined evaluation value is calculated using the area common to the layout information of the block of the sampled product and / or the difference area, and if the calculated evaluation value is equal to or greater than the threshold value, the sampled product is specified as a candidate. This is an image processing program.

The fifth invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer generates plane-expanded image information that expands to a plane for a part or all of the three-dimensional model generated by using the image information obtained by photographing the display shelf and the corresponding depth information, and the generated plane. The image information processing unit that uses part or all of the developed image information as the image information to be processed, the layout information of the block that is the area that satisfies the predetermined conditions in the face area of the image information to be processed, and the target to be sampled. The candidate processing unit that determines the layout similarity between the blocks using the layout information of the blocks in the object and identifies the products that are candidates for the comparison processing, the face area of the image information to be processed, and the above. This is an image processing program that functions as a comparison processing unit that identifies products in the face area by performing comparison processing using sample information of products specified as candidates in the candidate processing unit.

By using the image processing system of the present invention, the accuracy of matching processing is improved without requiring more calculation time than before when identifying an object such as a product shown in image information taken of a display shelf or the like. It is possible to make it.

It is a block diagram schematically showing an example of the whole processing function of the image processing system of this invention. It is a block diagram schematically showing an example of the hardware composition of the computer used in the image processing system of this invention. It is a flowchart which shows an example of the processing process of the whole processing in the image processing system of this invention. It is a flowchart which shows an example of the processing process of the process of specifying a block in the image processing system of this invention. It is a flowchart which shows an example of the processing process of the candidate identification processing in the image processing system of this invention. It is a figure which shows an example of photographed image information. It is a figure which shows an example of the image information which performed the orthotopic process with respect to the photographed image information of FIG. It is a figure which shows an example of the image information of a face area. It is a figure which shows an example of the state which specified the block in the image information of a face area. It is the figure which showed typically the layout information of the block specified in the image information of a face area, and the case of interpreting it on the image information. It is a figure which shows typically the image information corresponding to the sample information, the layout information of a block in the image information, and the case of interpreting it on the image information. It is a figure which shows an example of the layout matching process schematically. It is a figure which shows typically the combination of the part which has the area of a corresponding block, and the part which has the area of a corresponding block. It is a figure which shows an example about FIG. It is a figure which shows the other example of the layout matching process schematically. It is a figure which shows an example of the tag information which shows the type of a block. It is a figure which shows an example of the correspondence relation between blocks. It is a figure which shows typically the process in Example 3. FIG. It is a figure which shows typically an example of the type of a block and a tag information, and a narrowing down table when the appearance frequency is used as a comprehensive index value. It is a figure which shows typically the correspondence relationship between the photographed image information and the depth map by depth information. It is a figure which shows an example of the process of 3D modeling based on the photographed image information and the depth map. It is a figure which shows an example of the process of plane expansion schematically. It is a figure which shows typically the stitching process which is executed after 3D modeling. It is a figure which shows typically the stitching process which is executed after 3D modeling. It is a figure which shows typically the process of specifying a reference in a process of determining a viewpoint direction. It is a figure which shows an example of the process of labeling each surface schematically. It is an image diagram which specifies the face when the package type is a can. It is an image diagram that identifies the face when the package type is a bottle. It is an image diagram that identifies the face when the package type is a box. It is an image diagram that identifies the face when the package type is a hanging product.

FIG. 1 shows a block diagram of an example of the overall processing function of the image processing system 1 of the present invention. The image processing system 1 uses a management terminal 2 and an input terminal 3.

The management terminal 2 is a computer that realizes the core processing function of the image processing system 1. Further, the input terminal 3 is a terminal for acquiring image information obtained by photographing a display shelf of a store or the like. In addition, an object such as a product used as a sample used in the comparison process (matching process) described later may be photographed and acquired.

The management terminal 2 and the input terminal 3 in the image processing system 1 are realized by using a computer. FIG. 2 schematically shows an example of a computer hardware configuration. The computer can input information into an arithmetic unit 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk for storing information, a display device 72 such as a display for displaying information, and information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits and receives processing results of the arithmetic unit 70 and information stored in the storage device 71 via a network such as the Internet or LAN.

When the computer is provided with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited thereto.

The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be performed directly on the display with a predetermined input device (such as a pen for a touch panel) or a finger.

The input terminal 3 may include a photographing device such as a camera in addition to the above-mentioned devices. As the input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used. The input terminal 3 captures image information (photographed image information or object image information described later) by visible light or the like with a photographing device.

Each means in the present invention has only a logical distinction in its function, and may form the same area physically or substantially. The processing in each means of the present invention may be appropriately changed in the processing order. In addition, a part of the processing may be omitted. For example, it is possible to omit the process of determining the viewpoint direction, which will be described later. In that case, it is possible to execute processing on the image information that has not been processed to determine the viewpoint direction. Further, a part or all of the functions in the management terminal 2 may be executed in the input terminal 3.

The image processing system 1 includes an image information input reception processing unit 20, an image information storage unit 21, an image information processing unit 22, a candidate processing unit 23, a comparison processing unit 24, a sample information processing unit 25, and a sample information storage unit 26. ..

The image information input reception processing unit 20 receives input of image information (photographed image information) taken by the input terminal 3 and stores it in the image information storage unit 21 described later. For example, the input of the photographed image information of the display shelf of the store is accepted and stored in the image information storage unit 21. In addition to the captured image information, the input terminal 3 may accept input together with information indicating the shooting date and time and the shooting target, for example, store identification information such as a store name, image information identification information for identifying image information, and the like. FIG. 6 shows an example of captured image information. FIG. 6 schematically shows photographed image information obtained by photographing a state in which products are displayed on three shelves on the display shelf.

The image information storage unit 21 stores the captured image information received from the input terminal 3, the shooting date and time, the image information identification information, and the like in association with each other.

The image information processing unit 22 makes the captured image information received by the image information input reception processing unit 20 upright so that the captured image information faces each other, and compares the captured image information with the sample information (matching processing). Performs face processing to specify the area (face area) to execute, and block specifying processing to specify the feature area (block) that satisfies a predetermined condition in the face area.

In general, when the object to be photographed is simply photographed, it is difficult to shoot the object in the orthotopic position in the face-to-face state. This is a process of transforming the image information so that the optical axis of the lens of the photographing device is the same as when the image is taken from a sufficient distance along the perpendicular direction of the plane to be photographed. An example of such correction processing is trapezoidal correction processing. If the image information is distorted, distortion correction processing may be added. FIG. 7 shows the image information in a state in which the orthostatic processing is executed on the captured image information of FIG.

If the captured image information is the image information captured in a facing state, or if there is no distortion, it is not necessary to execute the orthostatic processing and the distortion correction processing.

The captured image information may be any image information to be processed by the present invention. The captured image information also includes the image information after the correction processing for the captured image information such as the orthostatic processing is executed.

Further, when the image information processing unit 22 shoots a plurality of images, the image information processing unit 22 executes a process of synthesizing the images into one image information, and the image information obtained by the synthesizing process is positive. Placement processing, face processing, and block identification processing may be executed. A known method can also be used as a process of synthesizing a plurality of image information into one image information.

The face processing specifies an area (face area) for executing comparison processing with sample information described later in the captured image information. For example, in the case of photographed image information obtained by photographing a product display shelf and the sample information is a product, the area of the product displayed on the display shelf or the area of the product label is specified as the face area. If the product is a PET bottle beverage, the area of the product label is the face area, and if the product is a boxed product (for example, sweets), the entire product package is the face area. Depending on the situation, the face area can be set as appropriate. Even when the product is a PET bottle beverage, the product area may be the face area.

There are various methods for specifying the face area, and it can be set arbitrarily according to the characteristics of the sample information. When the sample information is a product (for example, a PET bottle beverage) and the area of the product label is specified as the face area from the photographed image information of the display shelf, for example, between the shelves of the display shelf. Identify the vertical thin and narrow shadows that occur between products in the area (shelf area), identify the repeating pattern of the image, identify the step on the upper side of the package, the product width is the same, etc. Specify the product area by specifying the delimiter position based on the constraint. Then, from the area of the product, a predetermined rectangular area is specified as a label area, and that area is specified as a face area.

As the method for specifying the face area, any method can be adopted depending on the product category and the product form, and the method is not limited to the above. Further, the correction input by the operator may be accepted for the automatically specified face area. Further, the input of the position of the face area may be accepted from the operator.

Further, the image information processing unit 22 may specify the face region by using deep learning. In this case, the above captured image information is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the face area is set based on the output value. It may be specified. Further, as the learning model, it is possible to use various captured image information in which the face region is given as correct answer data.

The image information processing unit 22 cuts out the face region specified as described above. Cutting out the face area may actually cut out the face area specified from the captured image information, and also includes setting the area as the processing target of the processing described later.

When cutting out the face area, the face area specified in each of the above processes may be cut out as it is, or the face area may be specified by a plurality of methods and the target to be cut out using the result of the face area specified by each method. You may specify the face area to be. For example, the area of a product is specified by identifying the vertical thin and narrow shadows that occur between the products in the area between the shelves of the display shelf (shelf area), and a predetermined area is specified from that area. A method of specifying a rectangular area and a method of specifying a face area by deep learning are performed, and the results of the face areas specified by each of these methods are finally cut out by a predetermined calculation. May be specified. The method for specifying the face area to be cut out is not limited to the above, and can be set arbitrarily.

FIG. 8 shows an example of a case where a product (label portion of a PET bottle beverage) is cut out as a face area from the photographed image information.

When the image information processing unit 22 specifies the face area, the image information processing unit 22 specifies a rectangular area having information that can identify the product as a block in the face area. The area specified as a block is an area of the image information of the face area that is concentrated inside and has common features, and is a characteristic area that characterizes the image information of the face area. The area specified as a block is the area where the product name, product logo, etc. are displayed.

The image information processing unit 22 has, for example, three methods for specifying a block from the face area, but the method is not limited to the three methods, and a plurality of methods may be combined.

The first method is a method of extracting and determining image features common to blocks of an object. That is, in the image information of the face region, a rectangular region having a feature of being composed of a limited number of colors that can be clearly distinguished from each other and having a clear color-to-color boundary (clear edges) is blocked. It is a method to specify as.

First, the color of each pixel constituting the image information in the face region is detected by reducing the number of colors to a palette color having a predetermined number of colors. In addition, the image information in the face area is divided into meshes of a predetermined size, and a distribution map for each palette color is generated. Then, in each pallet, a rectangular area is specified by detecting a local group that is distributed at a certain density or higher and whose vertical and horizontal sizes fall within a predetermined threshold value.

Then, for each group, the distribution map of all palette colors is referred to, and it is determined whether the number of palettes contained in the rectangular area is within a predetermined number, for example, 3, and if it is within the predetermined number, the group area. It is determined whether or not there is an edge within a predetermined threshold value or more. For edge determination, edge detection processing is performed on the image information in the face area, and it is possible to determine whether or not the image information is equal to or greater than the threshold value. Then, the rectangular area surrounding the group that satisfies these conditions is specified as a block.

By the above method, the block can be specified from the face area.

As a second method for specifying a block from the face area, a feature amount such as a local feature amount is extracted from the image information of the face area. Then, a region in which common features are distributed over a wide range (a range of a predetermined size or more) is detected, and that region is erased as a "ground" part of the face region. Then, for the remaining areas, the areas where the features common to each other are distributed are specified, and the rectangular area surrounding them is specified as a block.

As a third method for identifying a block from the face region, the block can be detected from the image information of the face region by AI technology such as machine learning or deep learning. In this case, the image information of the face region is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the block is created based on the output value. It may be specified. In addition, as a learning model, it is possible to use image information of various face regions in which blocks are given as correct answer data. The face area and the block may be specified at the same time from the captured image information.

As another example of using AI technology such as machine learning or deep learning in the third method, there are the following. In advance, the feature amount in the block area (the feature amount may be a plurality of colors, local feature amounts, etc.) in the correct answer data (a block is given to the image information of various face areas) given as the above-mentioned learning model, and the block. A judgment module (judgment machine) such as an SVM (support vector machine) is configured by extracting the features outside the region and using each as a positive example and a negative example. Then, for this determination module such as SVM, each point of the image information in the face area, for example, the image information in the face area is divided into a mesh of a predetermined size, points in the mesh are input, and each point is blocked. It is made to judge whether or not it belongs to the area of. In this determination result, the area surrounding the mesh area determined to belong to the block area is specified as a block.

The process of specifying the block from the face area is not limited to the above three methods, and other methods may be used. Further, a plurality of the above three methods or a plurality of other methods may be combined and specified.

FIG. 9 schematically shows an example when a block is specified from the face area. FIG. 9 shows a case where a block is specified from the face area of FIG. The rectangular area in FIG. 9 is the specified block.

When the block is specified from the image information in the face area by the image information processing unit 22 as described above, the candidate processing unit 23 includes the specified block and the block corresponding to the sample information stored in the sample information storage unit 26 described later. The layout matching process, which is the process of determining the similarity of the layouts between the two, is executed. As the layout matching process, for example, a block is extracted from the image information in the face area. Extracting the block here means, for example, extracting the layout information (position information) of the block in the image information of the face area. FIG. 10 schematically shows this. FIG. 10A shows a block in the image information of the face area, FIG. 10B shows the layout information in coordinates, and FIG. 10C interprets the layout information on the image information. This is a schematic diagram of the case of Each block in FIG. 10A is named "region" to identify each block, and FIG. 10A shows a case where there are five blocks from "region1" to "region5". ing. Further, the layout information of each block is represented by the coordinates of two points in the rectangular area, for example, the upper left and the lower right, where "a" indicates the x coordinate and "b" indicates the y coordinate. It should be noted that the layout information may be indicated by four points instead of two points, or various methods can be used as long as the position information of each block can be specified, such as the distance from one point. In FIG. 10 (c), the layout information of FIG. 10 (b) is shown by the image information of the face area, and the area of each block is shown by the relative coordinates when the upper left of the image information of the face area is the origin. There is.

Similarly, the candidate processing unit 23 extracts a block corresponding to the sample information of the object stored in the sample information storage unit 26. FIG. 11 schematically shows a block corresponding to the extracted object. FIG. 11A shows the extracted object or the image information corresponding thereto, FIG. 11B shows the block in the image information corresponding to the extracted object, and FIG. 11C shows the layout information. Is shown in coordinates, and FIG. 11D schematically shows a case where layout information is interpreted on image information. Since FIG. 11A shows the sample information of the object or the image information corresponding thereto, it is used in the comparison process for identifying the object reflected in the photographed image information, as will be described later. Therefore, there is less noise than the face area cut out from the captured image information, and it is often clear. In addition, it is common that there are no unnecessary items such as stickers. As will be described later, the block in the image information of FIG. 11B may be specified by the operator, or may be automatically specified by the same processing as the image information processing unit 22 described above. Alternatively, the block may be automatically specified and modified by the operator as appropriate, such as selection and addition. Each block in FIG. 11B is named "REGION" to identify each block, and FIG. 11B shows a case where there are six blocks from "REGION1" to "REGION6". ing. Further, the layout information of each block is represented by the coordinates of two points in the rectangular area, for example, the upper left and the lower right, where "x" indicates the x coordinate and "y" indicates the y coordinate. It should be noted that the layout information may be indicated by four points instead of two points, or various methods can be used as long as the position information of each block can be specified, such as the distance from one point. In FIG. 11D, the layout information of FIG. 11C is shown by the image information of the face area, and the area of each block is shown by the relative coordinates when the upper left of the image information of the face area is the origin. There is.

The block in FIG. 10 (a) and the block in FIG. 11 (b) do not always match. For example, in FIG. 10 (a), “REGION 3” and “REGION 4” in FIG. 11 (b) are recognized as one block “region 2”, and in FIG. 10 (a), “REGION 2” in FIG. 11 (b) is recognized. Not recognized. Further, in FIG. 10A, “region 5” is excessively recognized because the label portion has a sticker.

The candidate processing unit 23 uses the layout information of the block extracted from the image information of the face area and the layout information of the block corresponding to the object stored in the sample information storage unit 26 to determine the similarity of the layout information. Calculate the indicated evaluation value, for example, the degree of similarity. The evaluation value indicating the similarity of the layout information is not limited to the similarity of the layout information, and may be any information indicating the similarity. For example, when similarity is used as an evaluation value, various methods can be used to calculate the similarity. For example, the following processing can be used.

First, the size of the image information in the face region and the sample information corresponding to the object stored in the sample information storage unit 26 or the image information corresponding to the sample information or the corresponding image information are matched (the area in this case is the common S). Then, based on the block layout information in the image information of the face region and the sample information of the object stored in the sample information storage unit 26 or the layout information of the block corresponding to the corresponding image information, those intersections (common). Part) and the difference are calculated. Then, logical operations such as AND, OR, and NOT between blocks are performed, and the areas thereof are obtained. An example of this process is schematically shown in FIG. In FIG. 12, a part having a corresponding block area is shown in black, and a part having no corresponding block area is shown in white. FIG. 13 schematically shows this combination. FIG. 13 (a) shows a combination of areas calculated between blocks, and FIG. 13 (b) shows TP (True Positive) and FN (False Negative) for the four combinations of FIG. 13 (a). ), FP (False Positive), TN (True Negative), and the same names as the relationship between the prediction result and the true value. Here, the layout information of the image information corresponding to the object to be sampled is regarded as a true value, and the layout information of the image information in the face area is regarded as the prediction result for it. For example, TP is interpreted to mean that the location where the layout information of the image information in the face area is considered to exist actually matches the layout information in the image information corresponding to the object to be sampled.

ここで，Precisionは適合率（正と予測したデータのうち，実際に正であるものの割合），Recallは再現率（実際に正であるもののうち，正であると予測されたものの割合）である。 Then, the candidate processing unit 23 calculates the similarity using each area calculated as shown in FIG. 13 (a). For example, when the F value is used as the similarity, it is calculated as the following equation 1.
(Number 1)

Here, Precision is the precision rate (the ratio of data predicted to be positive that is actually positive), and Recall is the recall rate (the ratio of data that is actually positive and predicted to be positive). ..

For example, using the layout information of the block extracted from the image information of a certain face area and the layout information of the block corresponding to a certain object stored in the sample information storage unit 26, as shown in FIG. It is assumed that the area of the result of calculating the section, the difference, etc. and performing the logical operation between the blocks is shown in FIG. At this time, the precision is 0.86 (= 0.3 / (0.3 + 0.05)) and the recall is 0.75 (= 0.3 / (0.3 + 0.1)). Therefore, the F value can be calculated as 0.80 (= 2 × 0.75 × 0.86 / (0.75 + 0.86)).

If the F value calculated here is equal to or higher than the predetermined threshold value TH, the candidate processing unit 23 adds the identification information of the object to the candidate list.

In addition, the candidate processing unit 23 may use an index other than the F value in order to calculate the specificity, and the accuracy rate (Accuracy) is actually the case among the data predicted to be positive or negative. You may use (ratio of things) or specificity (specificity, which is the ratio of things that are actually negative and that are predicted to be negative). The correct answer rate and specificity can be calculated using Equation 2.
(Number 2)

The candidate processing unit 23 executes this processing for the object to be compared stored in the sample information storage unit 26. The objects to be compared may be all objects stored in the sample information storage unit 26, or may be some objects narrowed down by some method.

Then, the candidate processing unit 23 sorts the objects added to the candidate list in descending order of similarity, and identifies the upper predetermined number, for example, the upper 10 objects as candidates.

The comparison processing unit 24 executes a comparison process (matching process) between the sample information of the target object specified by the candidate processing unit 23 and the image information of the face region. For example, by extracting the feature amount of the image information in the face area and executing the comparison processing between the extracted feature amount and the sample information such as the feature amount of the candidate object, the image information and the sample information in the face area can be obtained. The similarity is determined, and the identification information of the object contained in the face area, for example, the product name, is determined.

As the comparison process, not only the whole image information in the face area and the sample information are matched, but also a part of the image information in the face area, for example, the feature amount of the image information in the feature area (block) is extracted and sampled. Information feature quantities and comparison processing may be executed.

Further, the comparison process may be performed by a method other than the feature amount.

Further, the comparison processing unit 24 may execute the comparison processing with the sample information by using deep learning. In this case, all or part of the image information in the face region is input to the learning model in which the weighting coefficient between neurons in each layer of the neural network consisting of many layers is optimized, and the output value thereof. Similar sample information may be identified based on. As the learning model, it is possible to use image information of various face regions with sample information given as correct answer data.

The sample information processing unit 25 generates sample information of the object used in the comparison process for identifying the object reflected in the photographed image information, and stores it in the sample information storage unit 26 described later. The sample information is image information (object image information) obtained by photographing an object and / or information on a feature amount thereof. The sample information stores identification information of the object, for example, a product code such as a product name, a model number, and a JAN code, a block in the image information of the object, and its layout information. The image information of the object may be the image information of the entire object, or may be the image information of a part of the object, for example, the label portion when the object is a PET bottle beverage. When the feature amount information of the object image information is used as the sample information, it is not necessary to extract the feature amount each time the processing is performed. The block is a characteristic area that characterizes the image information of the object, and corresponds to, for example, a rectangular area surrounding a product name, a logo, or the like.

The sample information processing unit 25 may generate one sample information for one object, or is in a position where it cannot be photographed when the product is photographed at a plurality of angles for one object, for example, when the product is face-to-face. In order to capture the surface, sample information may be generated from angles such as front, back, top, bottom, and both sides. Further, when one object has a plurality of exteriors (packages, etc.), the sample information and blocks for each exterior may be associated with one object. Moreover, although one block may be associated with the sample information, it is preferable that a plurality of blocks are associated with each other.

For the sample information, a block may be extracted by executing the same processing as the above-mentioned image information processing unit 22 on the object image information obtained by photographing the object to be a sample, or the sample information may be extracted from the extracted block. The operator may select or add only the blocks necessary for specifying the product. Further, instead of automatically executing the extraction process as in the image information processing unit 22, the operator may specify a block from the image information of the read object.

The sample information storage unit 26 uses the sample information generated by the sample information processing unit 25 as object identification information, for example, a product code such as a product name, a model number, and a JAN code, a block in the object image information, and its layout information. Correspond and memorize.

Next, an example of the processing process using the image processing system 1 of the present invention will be described with reference to the flowcharts of FIGS. 3 to 5. In this embodiment, a case where a display shelf of a store is photographed and a product on the display shelf is specified will be described. Therefore, the photographed image information is the display shelf of the store where the product is displayed, and the product may be displayed as the object in the sample information.

First, the operator photographs the product as the sample information from the opposite position, and stores the photographed object image information, identification information, product name, and product code in the sample information storage unit 26 in association with each other. Further, the sample information processing unit 25 executes the same processing as the processing in the image information processing unit 22 for the object image information, and extracts the block. Then, among the extracted blocks, the block and its position information (layout information) are stored in the sample information storage unit 26 as a block used as sample information, for example, a block containing only a product name, a logo, etc. is associated with the sample information. Remember it. In addition, by executing a predetermined process from the object image information, a feature amount such as a local feature amount is extracted and stored in association with the object image information.

Processing for such sample information is performed in advance for a plurality of products, preferably all products that may be displayed on a display shelf.

Then, when the image information input reception processing unit 20 of the management terminal 2 receives the photographed image information obtained by photographing the display shelf of the store from the input terminal 3 (S100), the image information storage unit 21 receives the photographed date and time and the identification information of the photographed image information. And memorize it in association with.

The image information processing unit 22 executes an orthostatic processing for correcting the received captured image information so that the captured image information faces each other (S110). Then, a face process for specifying the face area from the embossed captured image information is executed (S120), and the block is specified from the specified face area (S130).

As described above, in order to identify the block from the face area, there are various methods such as the first method to the third method as described above, but in the case of the first method, for example, Execute the following processing. An example of the processing process in this case will be described with reference to the flowchart of FIG.

First, for the color of each pixel constituting the image information in the face region, the number of colors is reduced to a predetermined number of palette colors such as 16 colors representing the whole, and the palette color is detected (S200). Further, the short side of the face region is divided into, for example, 16 divided square meshes, which palette color is used for each mesh, and a distribution map of the presence or absence of that color is generated for each palette color (S210).

Then, for each mesh of the image information in the face area, expansion processing is performed on the distribution map so that the enlave at a certain distance is a candidate, and the group in which the distribution map falls within a predetermined range is detected (). S220). The predetermined range is, for example, a threshold condition in which the width is 1/10 to 2/3 of the whole, the height is 1/20 to 1/2 of the whole, and the area is 1/100 to 1/10 of the whole. Detects a group of distribution maps (distribution maps that have undergone expansion processing) that fit in.

Then, for each group, the distribution map of all palette colors is referred to, and it is determined whether the number of palettes included in the area is a predetermined number, for example, 3 or less (S230). That is, referring to the distribution map of all palette colors in the area of the group, if the area of the group contains a large number of colors, it is highly likely that it is an image illustration, and it is a characteristic area that identifies the product. Exclude such groups as they are not considered to be certain logos.

In S230, when the number of pallets contained in the group area is within a predetermined number, it is determined whether or not the edge is equal to or more than a predetermined threshold value in the group area (S240). Regarding the edge determination, it is possible to perform edge detection processing on the image information in the face area in advance and determine whether or not the image information is equal to or greater than the threshold value. If the edge does not exist above the threshold value in the group area, it is highly possible that it is not a logo, so that group is excluded.

Then, the rectangular area surrounding the group satisfying these conditions is specified as a block (S250).

This process is executed for the group grouped in S220 (S260).

By the above method, the block can be specified from the face area.

When the image information processing unit 22 specifies a block from the image information in the face area, the candidate processing unit 23 arranges between the specified block and the block corresponding to the object stored in the sample information storage unit 26 described later. By executing the matching process, a candidate product is specified (S140). An example of the processing process of the specific processing of the candidate product will be described with reference to the flowchart of FIG.

Specifically, the candidate processing unit 23 uses the block layout information extracted from the image information in the face area and the block layout information corresponding to the products for all the products to be compared. Intersections, differences, etc. are calculated, logical operations such as AND, OR, and NOT between blocks are performed, and the areas thereof are obtained for layout matching (S310). Then, for example, the similarity such as the F value is calculated, and if it is equal to or higher than the predetermined threshold value TH, the product is added to the candidate list (S320). Then, the same processing is executed for the next product (S330).

Then, when layout matching processing is performed for all the products to be compared, the candidate processing unit 23 sorts the similarities added to the candidate list in descending order (S340), and among them, the top predetermined number, for example, the top 10 products. Is specified as a candidate.

When the candidate processing unit 23 specifies a product that is a candidate for comparison processing in S140, the comparison processing unit 24 extracts the feature amount of the image information in the face region, and the extracted feature amount and the sample of the product specified as the candidate. The comparison process is performed by comparing with the feature amount of the information (S150). At the time of this comparison, the comparison process is performed after the size of the image information in the face area and the size, scale, rotation, etc. of the image information in the sample information are as close as possible. This comparison process is performed for each product specified as a candidate.

Then, as a result of this comparison processing, among the products identified as candidates, the product determined to have the highest degree of similarity is specified as a product in the face area, and the face area, product identification information, product name, product code, etc. are specified. (S160).

Since a plurality of products are generally displayed on the display shelf shown in the photographed image information, it is common that a face area corresponding to the displayed products is cut out. Therefore, by executing each of the above-mentioned processes for each face area, it is possible to specify the products displayed on the display shelf and the display position. The display position of the product can be specified by the order of the vertical and horizontal directions of the face area, and can also be specified as the coordinates in the photographed image information.

When the surface of the object is a curved surface, image distortion occurs as it shifts in the left-right direction. Therefore, it is advisable to execute the following processing.

That is, when the candidate processing unit 23 calculates the similarity of the layout information, as shown in FIG. 12, the candidate processing unit 23 blocks the layout information of the block in the entire image information of the face region and the block in the entire image information corresponding to the object. In addition to executing the process of calculating the similarity using the layout information of, the layout information of the block in a part of the image information of the face area and the layout information of the block in a part of the image information corresponding to the object are used. The degree of similarity may be calculated. An example of this process is schematically shown in FIG.

As shown in FIG. 15, in the case of this embodiment, a certain ratio on the left and right of the image information in the face region, for example, about 1/4 is excluded, and about 1/4 each from the center line of the image information in the face region (center). About the block layout information in the image information cut out (about 1/2 part) and the image information corresponding to the object, after cutting out and cutting out the same or almost the same width as the width cut out from the image information of the above-mentioned face area. Use the layout information of to calculate intersections and differences. Then, logical operations such as AND, OR, and NOT between blocks are performed, and the areas thereof are obtained. Then, the similarity is calculated in the same manner as described above.

Next, the same process is executed by shifting the cutout position of the image information corresponding to the object by a predetermined amount. The layout information of the block in which the cutout position is shifted may be prepared in advance.

Then, the highest similarity is compared with the threshold value TH, and if it is equal to or higher than the threshold value TH, it is added to the candidate list. Further, the cutout position of the image information corresponding to the sample information at the time of the highest similarity when the threshold value TH or more may be stored may be stored. In this case, when the comparison processing unit 24 compares the feature amount of the image information in the face region with the feature amount of the image information corresponding to the sample information, the image information corresponding to the sample information is cut out at the stored cutout position. ， It is preferable to compare the features.

The candidate processing unit 23 does not compare the highest similarity with the threshold TH, but compares it with the threshold TH each time the similarity is calculated, and if it is equal to or higher than the threshold TH, adds it to the candidate list. The process of shifting the cutout position may be completed, and the process of blocking the image information corresponding to the next sample information may be performed. As a result, the process of cutting out the image information corresponding to the object by shifting it by a predetermined amount can be completed when the similarity becomes equal to or higher than the threshold value TH, so that the processing speed can be increased.

In the above description, the layout matching process is performed while shifting the layout information of the block in the image information obtained by cutting out about 1/2 of the central part of the image information in the face area in the same range in the image information of the object by a predetermined amount. However, the opposite processing may be performed. That is, the layout matching process is performed by cutting out the layout information of the block in the image information obtained by cutting out about 1/2 of the central portion of the image information of the object and cutting out the same or almost the same width in the image information of the face area. It may be repeated by shifting by a predetermined amount.

In the first and second embodiments, the comparison processing unit 24 has a feature amount of a part or all of the corresponding image information of the sample information of the object specified as a candidate by the candidate processing unit 23, and a face area. By comparing the feature amount of a part or all of the image information of, the similarity between the sample information and the image information of the face area is determined, and the identification information of the object included in the face area, for example, the product name, etc. Was judged.

In this embodiment, the similarity between the sample information and the image information in the face area is determined by comparing the block of the object specified as a candidate by the candidate processing unit 23 with a part or all of the blocks in the face area. Then, the case of determining the identification information of the object included in the face area, for example, the product name, etc. will be described.

In this case, the sample information storage unit 26 may store tag information indicating the type of the block in association with the block. An example of tag information is shown in FIG. Then, each tag is subjected to predetermined preprocessing, and the processing result is stored in association with the sample information storage unit 26. For example, if the tag information is an image image, the image information is cut out and registered, if the tag information is a product name logo or copy, or if it is rated, OCR processing is performed and stored as text information.

Then, the comparison processing unit 24 first identifies the correspondence between the blocks when comparing the block of the object specified as a candidate by the candidate processing unit 23 with a part or all of the blocks in the face area. In this case, after matching the position and scale of the image information of the face area and the image information of the sample information, the block having the largest intersection of the image information of the face area when viewed from the block corresponding to the sample information is obtained, or The correspondence between blocks can be specified by specifying the center of the layout information of the block corresponding to the sample information, finding the block in the image information of the face area, which is the closest to the center, and so on. FIG. 17 shows an example of the correspondence between blocks.

Then, when the correspondence between the blocks is specified, the preprocessing for the block corresponding to the sample information is executed for the image information block in the corresponding face area, and the result is calculated. Then, the results of the preprocessing are compared with each other. For example, in the case of image information, similarity is evaluated using features, and in the case of text information, text matching is performed. An example of this process is shown in FIG.

The comparison processing unit 24 calculates a comprehensive index value of the degree of agreement with the sample information using the result of comparing the blocks as described above. For example, a setting is made in advance so that the degree of matching of product names is highly evaluated and the degree of matching of manufacturer names is low, and a comprehensive index value is calculated based on the setting. In the evaluation of the degree of matching, for example, a numerical value that determines the weight at the time of matching is related according to the type of tag information corresponding to the block. The weighting value may be determined by the type of tag information corresponding to the block, or may be, for example, the frequency of appearance in the product category when the object is a product. When using the frequency of appearance, it is assumed that the object can be specified by making a table of what and what matches and how far it can be narrowed down, and by narrowing down the object such as a product to one. FIG. 19A shows an example of the correspondence between the block and the type of tag information, and FIG. 19B shows an example of the narrowing table.

The comparison processing unit 24 performs the above comparison processing on an object such as a product specified as a candidate, and identifies the object having the highest overall index value as a product in the face region.

In the first to third embodiments, the case where the image is captured with the two-dimensional image information as the captured image information has been described, but the three-dimensional model is taken from the captured image information in consideration of the distance information between the imaging target and the imaging device. The image information when the image is converted and then projected onto the two-dimensional image information may be used as the captured image information to be processed by the image processing system 1 of the present invention. In this case, the same processing is performed for the image information of the object in the sample information. The processing in this case will be described below.

In the input terminal 3 of the present embodiment, in addition to photographing image information by visible light or the like with a photographing device such as a camera, distance information (depth information) between the imaged object and the photographing device is acquired as two-dimensional information. The information to which the depth information is mapped is referred to as a depth map in this specification. The depth map corresponds to the image information taken by the photographing device, and the depth information up to the object reflected in the photographed image information is mapped to the position corresponding to the photographed image information. In the depth map, at least the range corresponding to the captured image information is divided into meshes, and the depth information is given to each mesh. The vertical and horizontal sizes of the mesh depend on the accuracy of the depth information that can be acquired, but the smaller the vertical and horizontal sizes of the mesh, the higher the accuracy. Usually, it is in the range of 1 mm to several mm, but it may be less than 1 mm or 1 cm or more.

When acquiring depth information, the amount of reflected light from each direction and the time until the reflected light arrives are measured by irradiating light rays of a specific wavelength, such as infrared rays, when shooting the subject. Acquires distance information (depth information) to the shooting target, or irradiates a light beam of a specific wavelength, for example, an infrared dot pattern, and measures the distance information (depth information) from the reflection pattern to the shooting target. In addition to the above, there are methods that utilize the parallax of the stereo camera, but the method is not limited to these. If the depth information is given by the distance from the photographing device (one point on three dimensions), the depth information is converted into the depth information from the photographing surface (plane) of the photographing device. .. The input terminal 3 may be provided with a device (depth detection device) for acquiring depth information in this way.

The positional relationship between the image information and the depth information corresponds to each other. FIG. 20 shows the correspondence between the image information taken with the display shelf as the shooting target and the depth map based on the depth information. FIG. 20 (a) is a diagram schematically showing image information taken, and FIG. 20 (b) is a diagram schematically showing a corresponding depth map.

The image information input reception processing unit 20 receives input of image information to be photographed such as a display shelf taken by the input terminal 3 and stores it in the image information storage unit 21. Further, the image information input reception processing unit 20 receives the input of the depth information and the depth map for the image information captured by the input terminal 3, and stores the captured image information in the image information storage unit 21 in association with the captured image information. When the depth information is received from the input terminal 3 instead of the depth map, the depth map is generated by mapping it.

It is preferable that the image information, the depth information, and the depth map of the photographed object are photographed and the information is acquired at the same time when the image information is photographed, and the input of the information is accepted.

The image information processing unit 22 converts the display shelf or the like into a three-dimensional model based on the photographed image information and the depth map. That is, since the captured image information and the depth map correspond to each other, first convert to a 3D model based on the depth information in the depth map, and then associate the captured image information with the texture mapping. do. FIG. 21 shows an example of a three-dimensional modeling process based on captured image information and a depth map. Note that FIG. 21 shows a case where a PET bottle of a beverage is modeled in three dimensions. FIG. 21A is image information taken. Since the image information processing unit 22 acquires the depth map at the same time, it converts it into a three-dimensional model based on the depth information of each mesh in the depth map. Then, by performing texture mapping in which the color information of the corresponding pixel of the captured image information is pasted at the position of each mesh in the depth map, three-dimensional modeling is performed as shown in FIGS. 21 (b) to 21 (d). can do. 21 (b) shows a viewpoint from the left, FIG. 21 (c) shows a viewpoint from the front, and FIG. 21 (d) shows a viewpoint from the right.

When the image information processing unit 22 generates a three-dimensional model based on the image information obtained by photographing the display shelf or the like and the depth map, the image information processing unit 22 executes a plane expansion process for expanding the three-dimensional model. Plane expansion means expanding a three-dimensional model into a plane (two-dimensional). There are various methods for expanding the three-dimensional model in a plane, and the method is not limited.

As an example of the method of plane expansion, there is a method of expanding to a plane by defining the center of projection inside the 3D model and projecting from there. An example of the plane expansion process in the image information processing unit 22 will be described with reference to FIG. FIG. 22A is image information obtained by photographing a PET bottle, and the top view thereof has a substantially hexagonal shape. The actual shape is finer and uneven, but for the sake of explanation, it is shown here as a hexagon. When the image information of FIG. 22 (a) is three-dimensionally modeled using the depth map, the positional relationship when viewed from above is shown in FIG. 22 (b). In this case, since there is no depth information of the part behind the photographing device (camera), there is a blank when viewed from above as shown in FIG. 22 (b). Then, the central axis of the projection is specified at an arbitrary point of the object to be developed in a plane. For example, if a product is displayed on a display shelf and its shape is not a box-shaped or bag-shaped product but a cylindrical or bottle-shaped surface, the surface is not a flat surface but a regular curved surface, the width and surface of the product. From the shape, find the center point of the rotation axis and specify it as the center axis of the projection. In addition, in order to specify the central axis of projection, any method may be used, not limited to the above. FIG. 22 (c) shows a state in which the central axis of projection is specified. Then, the surface of the object is projected onto an arbitrarily set projection surface from the central axis of the specified projection. This is schematically shown in FIG. 22 (d). As shown in FIG. 22D, when the positions of the surface of the 3D model and the projection surface do not match, the vertical and horizontal dimensions of the image information of the projection result according to the distance between them are determined by an appropriate scale ratio. Adjust by. On the other hand, when the positions of the plane of the three-dimensional model and the projection plane are matched as shown in FIG. 2 (e), it is not necessary to provide the adjustment process of the scale ratio. Note that FIGS. 22 (b) to 22 (e) are views from above.

By performing such a plane expansion process, the image information processing unit 22 can expand the three-dimensional model into two dimensions (plane) and convert it into two-dimensional image information (plane expansion image information). In the above, the process of converting the 3D model to the plane-expanded image information is performed by the process of projection, but in the plane-expanded process, the process of converting the 3D model to the plane-expanded image information may be performed by another process. ..

If the image information processing unit 22 handles the plane-expanded image information generated in this way as the captured image information in the above-mentioned Examples 1 and 2, and executes the processing after the face processing, the first and second embodiments are performed. The same process as in 2 can be executed.

Further, the sample information processing unit 25 generates a three-dimensional model from the image information obtained by photographing the object and the depth map, and the plane development image information is based on the three-dimensional model, as in the image information processing unit 22 described above. To generate. Then, by storing all or part of the plane-developed image information (for example, the label portion) in the sample information storage unit 26 as sample information, the same processing as in Example 1 and Example 2 can be executed.

As a modification of the process of the fourth embodiment, a case where the stitching process is executed will be described.

When a plurality of images are photographed, the image information processing unit 22 may execute a stitching process for synthesizing the images into one image information. The stitching process may be performed before the 3D modeling or after the 3D modeling. A known method can be used for both the stitching process before the 3D modeling and the stitching process after the 3D modeling.

If it is performed before the three-dimensional modeling, the stitching process is executed by detecting the corresponding points between the captured image information. At this time, since the corresponding points can be specified between the captured image information, the depth maps corresponding to the positions of the specified corresponding points are also associated with each other in the same manner. This makes it possible to execute stitching processing before 3D modeling.

In addition, when performing after 3D modeling, the similarity of the uneven shape in the two 3D models is specified from the depth information, and the shapes are similar (the pattern of the unevenness based on the depth information is included in the predetermined range. The stitching process is executed after the 3D modeling by searching for the corresponding points between the captured image information that is the source of the 3D model that corresponds to that area. can. FIG. 23 schematically shows this. 23 (a) and 23 (b) are photographed image information obtained by photographing the shelves arranged side by side, FIG. 23 (c) is a three-dimensional model based on FIG. 23 (a), and FIG. 23 (d) is FIG. 23 (d). It is a three-dimensional model based on b). In this case, as shown in FIG. 24, for example, in the photographed image information, if the reference 101 for stitching processing, for example, the shelf step is photographed so as to be reflected, the reference 101 is in the same viewpoint direction. Rotate the 3D model as in. Then, the similarity of the uneven shape is determined based on the depth information. Then, the stitching process can be executed by searching for the corresponding points in the specified similar area 100. FIG. 24A shows a state in which the three-dimensional model of FIG. 19A is rotated so as to have the same viewpoint direction as that of FIG. 24B. The broken line region 100 in FIGS. 24 (a) and 24 (b) is a region 100 having a concave-convex shape determined based on the depth information. Further, the alternate long and short dash line region 101 is a shelf stage used as a reference for rotating the three-dimensional model so that the three-dimensional model is oriented in the same viewpoint direction. The reference 101 can be arbitrarily specified depending on the object to be photographed, but it is preferably composed of two or more straight lines.

By executing the above processing, the stitching processing can be executed either before or after the 3D modeling. Further, when the information of a plurality of captured images is not used, the stitching process may not be executed.

As a modification of the process of the fourth embodiment, the image information processing unit 22 determines the viewpoint direction so that the three-dimensional model to be photographed faces the position after the three-dimensional modeling process is executed. May be executed, and then the process of plane expansion may be executed. For example, when the object to be photographed is a product display shelf or a product located there, the viewpoint direction is determined so that the display shelf or the product faces the position (front position).

As a process for determining the viewpoint direction, the viewpoint direction may be determined so that the reference in the three-dimensional model satisfies a predetermined condition for the position to face each other. The standard and conditions can be arbitrarily set according to the shooting target and the like. For example, when taking a picture of a display shelf, the front 102 of the shelf of the display shelf is used as a reference, and the condition is that the front 102 of the plurality of shelves of the display shelf are all horizontal and perpendicular to each other. This can be achieved by moving the viewpoint to the center in the vertical and horizontal directions of the shooting range. Normally, when the front side of the shelf is photographed, the front of the display shelf is located in the foreground (the frontmost position), so that the depth information of the 3D model indicates that the front 102 of the shelf is on the shelf. The front surface 102 can be specified. FIG. 25 schematically shows this process. FIG. 25 (a) is photographed image information of the display shelf, and FIG. 25 (b) is a diagram expressing the depth information in black and white density in the depth map of the display shelf (black case is the case). Located in the foreground (the value of the depth information is small), FIG. 25 (c) is a three-dimensional model based on the depth map, in which the depth information is converted into colors. Then, the image information processing unit 22 identifies the front surface of the shelf of the display shelf based on the depth information, and determines the viewpoint direction so that the shelf satisfies the above-mentioned conditions. Further, the same standard as the standard used in the determination process of the viewpoint direction and the standard in the stitching process after the above-mentioned three-dimensional modeling may be used. For example, in either case, the front of the shelf of the display shelf can be used as a reference.

In FIG. 25 (c), for the sake of clarity, the case where the captured image information is processed in the state before the texture mapping is shown, but the process of determining the viewpoint direction is executed for the 3D model after the texture mapping. You may. The process of determining the viewpoint direction is not limited to the above, and can be realized by any process.

If the captured image information is originally captured from a position facing it, the viewpoint direction determination process does not have to be executed.

In the processing of Examples 4 to 6, the image information of the face region was specified by executing the face processing on the plane development image information, but the face region was specified and specified from the three-dimensional model. The plane expansion process may be executed for the face region to obtain the plane expansion image information.

There are various methods for specifying the face area from the 3D model, and it can be set arbitrarily according to the characteristics of the sample information. For example, in a 3D model, the position and range of a surface that satisfies a predetermined condition are specified, and the surface is matched with whether the condition corresponding to the type of the surface model of the predetermined sample information is satisfied, and the condition is satisfied. Specify the face as a face area. For example, as described above, in the case where the sample information is a product, a three-dimensional model is generated from the photographed image information obtained by photographing the display shelf, and the face area of the product is specified, the following processing can be executed. ..

First, after filtering out the high frequency to exclude fine irregularities, a surface that is close to vertical and has a certain area facing directly, an upright cylindrical surface, and a surface of a polyhedral region that is convex and distorted as a whole. Is detected, and the position and range in the three-dimensional space are specified. For example, if the depth information is within a certain range and the normal is stable and facing the front, the vertical correlation is strong with respect to the plane and depth information, and if it is convex in the horizontal direction, the cylinder is in the horizontal direction. And both in the vertical direction are convex, or if the normal is not stable, it is specified as a plane in the region of a convex distorted polyhedron. Then, each detected surface is labeled. FIG. 26 shows a state in which five surfaces are detected and labeled from 1 to 5 respectively. Then, for the detected surface, the distance from the foreground (this is used as the level) is calculated in order from the foreground based on, for example, the average value of the depth information. Then, based on the calculated level, the context of each surface is specified. In FIG. 26, the levels of the surfaces 1 and 2 are calculated as 0 cm, the levels of the surfaces 3 and 4 are calculated as 3 cm, and the level of the surface 5 is calculated as 20 cm. Then, the front-back relationship of the five faces is specified, with the frontmost faces being the faces 1 and 2, the next being located in the back being the faces 3 and 4, and the backmost face being the face 5.

Then, the image information processing unit 22 specifies the type of the surface by using the context of the surface specified in this way. That is, the depth information is the same as the front surface of the shelf, the product tag surface, the rear surface of the shelf, and the front surface of the shelf. Or, a vertical rectangle that is in close proximity (within a certain range, for example, within a few centimeters) and is hollow is specified as the surface of the product tag. Then, preferably, the other side is specified as a product side (a side on which the product may be displayed). Therefore, in the example of FIG. 26, the surfaces 1 and 2 are specified as the front surface and the product tag surface of the shelf board, the surfaces 3 and 4 are specified as the product surface, and the surface 5 is specified as the rear plate surface of the shelf body.

On the product side specified in this way, the corresponding processing is executed based on the package type of the product for which the input is accepted at any timing. Package types include cans, bottles (including PET bottles; the same applies hereinafter), coexistence of cans and bottles, boxes, hanging products, etc., but are not limited to these.

If the package type is a can or bottle, identify the area of the cylinder within a certain size as the face area of the can or bottle. This is schematically shown in FIGS. 27 and 28. FIG. 27 is an image diagram showing the case where the package type is a can, and FIG. 28 is an image diagram showing the case where the package type is a bottle.

When the package type is a coexistence of a can and a bottle, there is a cylinder within a certain size range, the area where there is no small cylinder above is the face area of the can, and there is a cylinder within a certain size range. The area with a small cylinder above is specified as the face area of the bottle.

If the package type is a box, the area of the vertical rectangular plane within a certain size range is specified as the face area of the box. FIG. 29 schematically shows this.

If the package type is a hanging product, the area of a rectangular flat surface or uneven surface that is within a certain size range and floats in the air is specified as the face area of the hanging product. FIG. 30 schematically shows this.

The image information processing unit 22 may specify the face area without accepting the selection of the package type. In this case, it is determined whether the condition as the face area is satisfied by executing the determination process for each of the product surface area and the package type. If there is a face that constitutes the face area, specify the face area that the face constitutes, and what kind of package type the face area is based on the predetermined package type priority conditions. To identify.

The process of specifying the face area from the three-dimensional model can adopt any method depending on the category and shape of the product, and is not limited to the above. Further, the correction input by the operator may be accepted for the automatically specified face area. Further, the input of the position of the face area may be accepted from the operator.

Further, the image information processing unit 22 may specify the face region by using deep learning. In this case, the above 3D model is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the face region is set based on the output value. It may be specified. Further, as a learning model, various three-dimensional models with face regions can be used as correct answer data.

The image information processing unit 22 cuts out the face region specified as described above. Cutting out the face area may actually cut out the face area specified from the three-dimensional model, and also includes setting the area as the processing target of the processing described later.

When cutting out the face area, the face area specified in each of the above processes may be cut out as it is, or the face area may be specified by a plurality of methods and the target to be cut out using the result of the face area specified by each method. You may specify the face area to be. For example, a method of specifying the position and range of a surface that satisfies a predetermined condition from a 3D model and matching whether the condition is satisfied according to the type of the surface model to specify the face region, and a method of specifying the face region by deep learning. The face area to be finally cut out may be specified by a method of specifying and a predetermined operation. The method for specifying the face area to be cut out is not limited to the above, and can be set arbitrarily.

The image information processing unit 22 executes the same plane expansion processing as in the third embodiment for the face region cut out by the image information processing unit 22, and generates plane expansion image information for the face region. Then, if the plane-expanded image information generated in this way is treated as the captured image information in the above-mentioned Examples 1 and 2, and the processing after the block specifying process is executed, the same as in Example 1 and Example 2. Processing can be executed.

Further, the sample information processing unit 25 generates a three-dimensional model from the image information obtained by photographing the object and the depth map, and cuts out the face region based on the three-dimensional model, similarly to the image information processing unit 22 described above. .. Then, by generating the plane-expanded image information from the cut-out face region and storing the plane-expanded image information as sample information in the sample information storage unit 26, the same processing as in the first and second embodiments can be executed.

The processing in each of the above-mentioned examples is not limited to the order described in the specification of the present invention, and can be appropriately changed as long as the object is achieved.

Further, the image processing system 1 of the present invention is effective when specifying the display status of a product displayed on the display shelf from the photographed image information obtained by photographing the display shelf of the store. It is not limited. That is, when a certain object to be photographed is photographed, it can be widely used when specifying the area in which the desired object is captured from the photographed image information.

By using the image processing system 1 of the present invention, the accuracy of comparison processing can be improved without requiring more calculation time than before when identifying an object such as a product shown in image information obtained by photographing a display shelf or the like. It will be possible to improve.

1: Image processing system 2: Management terminal 3: Input terminal 20: Image information input reception processing unit 21: Image information storage unit 22: Image information processing unit 23: Candidate processing unit 24: Comparison processing unit 25: Sample information processing unit 26 : Sample information storage unit 70: Arithmetic device 71: Storage device 72: Display device 73: Input device 74: Communication device

Claims (7)

It is an image processing system that identifies the products shown in the image information of the display shelves.
The image processing system is
Using the layout information of the block , which is an area satisfying a predetermined condition in the face area in the image information, and the layout information of the block in the sample product , the similarity of the layout between the blocks is determined, and the comparison process is performed. The candidate processing unit that identifies the products that are candidates for
Yes a face region of the image information, by the comparison processing using the sample information items identified as candidates by the candidate processing unit, and a comparison processing unit for identifying the product that is reflected in the face region And
The candidate processing unit
A predetermined evaluation value is calculated using the area of the intersection and / or the difference between the block layout information in the face area of the image information and the layout information of the block of the sampled product.
If the calculated evaluation value is equal to or greater than the threshold value, the sampled product is specified as a candidate.
An image processing system characterized by that. The candidate processing unit
Identifies blocks that satisfy a predetermined condition in the face region in the image information,
The identified and layout information of the block in a predetermined range of the face area, by using the layout information of the block of the region corresponding to the size of the predetermined range of the face area in the image information of the commodity to the specimen, the layout similarities Judgment of
The image processing system according to claim 1. The candidate processing unit
The process of determining the similarity of the layout is performed while shifting the area corresponding to the size of the predetermined range of the face area in the image information of the product to be the sample.
The comparison processing unit
In the image information of the product , the sample information of the region determined to have the highest similarity by the candidate processing unit and the face region of the image information are compared and reflected in the face region. Identify the product,
The image processing system according to claim 2, wherein the image processing system is characterized by the above. The comparison processing unit
Correspondence with the block of the image information, and a block of the product was identified as a candidate in the candidate processing unit,
The comparison process is executed by determining the similarity of the image information and / or the character information using the associated block.
The image processing system according to any one of claims 1 to 3 , wherein the image processing system is characterized in that. It is an image processing system that identifies the products shown in the image information of the display shelves.
The image processing system is
For a part or all of the 3D model generated by using the image information obtained by photographing the display shelf and the corresponding depth information, the plane development image information developed on a plane is generated, and the generated plane development image information is obtained. An image information processing unit that uses part or all of the image information to be processed,
Using the layout information of the block, which is an area satisfying a predetermined condition in the face area of the image information to be processed, and the layout information of the block in the sample object, the similarity of the layout between the blocks is determined. A candidate processing unit that identifies products that are candidates for comparison processing,
By performing comparison processing using the face area of the image information to be processed and the sample information of the product specified as a candidate in the candidate processing unit, the comparison processing unit that identifies the product reflected in the face area and the product. ， ，
An image processing system characterized by having. Computer,
Using the layout information of the block, which is an area satisfying a predetermined condition in the face area in the image information obtained by photographing the display shelf, and the layout information of the block in the sample product , the similarity of the layout between the blocks is determined. Candidate processing unit that performs and identifies products that are candidates for comparison processing,
By performing comparison processing using the face area of the image information and the sample information of the product specified as a candidate in the candidate processing unit, the product functions as a comparison processing unit for specifying the product reflected in the face area. It is an image processing program
The candidate processing unit
A predetermined evaluation value is calculated using the area of the intersection and / or the difference between the block layout information in the face area of the image information to be processed and the layout information of the block of the sampled product.
If the calculated evaluation value is equal to or greater than the threshold value, the sampled product is specified as a candidate.
An image processing program characterized by this. Computer,
For a part or all of the 3D model generated by using the image information obtained by photographing the display shelf and the corresponding depth information, the plane development image information developed on a plane is generated, and one of the generated plane development image information. Image information processing unit, whose part or all is the image information to be processed,
Using the layout information of the block, which is an area satisfying a predetermined condition in the face area of the image information to be processed, and the layout information of the block in the sample object, the similarity of the layout between the blocks is determined. Candidate processing unit that performs and identifies products that are candidates for comparison processing,
A comparison processing unit that identifies products in the face area by performing comparison processing using the face area of the image information to be processed and the sample information of the product specified as a candidate in the candidate processing unit.
An image processing program characterized by functioning as.

Images