JP2020194443A - Image processing system - Google Patents

Abstract

To provide an image processing system.SOLUTION: An image processing system comprises: a candidate processing unit for identifying an object serving as a candidate for comparison processing by determining a similarity in a layout using layout information in a second region that satisfies a predetermined condition in a first region in image information subject to processing and layout information in a block of an object serving as a sample; and a comparison processing unit for identifying an object captured in the first region by performing comparison processing using the first region or the second region of the image information subject to the processing and sample information or the block of the object identified as the candidate by the candidate processing unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing system.

It may be required to determine whether or not an object to be determined is reflected in a certain image information. For example, there is a case where it is determined whether or not the product to be determined is displayed in the image information obtained by photographing the display shelf of the product.

In this way, when determining whether or not an object is reflected in the image information, it is common to execute an image matching process between the image information to be processed and the image information of the object. For example, Patent Document 1 below discloses a system for grasping products handled by vending machines by using image recognition technology for image information taken by vending machines based on registered images for each product. Has been done.

Japanese Unexamined Patent Publication No. 2014-191423

The specific processing of Patent Document 1 is to photograph a vending machine from a plurality of directions, align the positional relationships of the photographed images, and then superimpose the photographed images to generate a composite image. Then, the products handled by the vending machine are specified by collating the generated composite image with the registered image representing the products that may be displayed on the vending machine.

In the collation process (matching process) between the composite image and the registered image representing the product, it is common to execute the process by using each feature amount. However, when the number of products to be compared increases, the number of registered images increases, and the accuracy of the matching process (matching process) cannot be improved. In addition, there is a problem that the calculation time is enormous.

Therefore, in view of the above problems, the present inventor improves the accuracy of the matching process without requiring more calculation time than before when identifying an object such as a product shown in image information obtained by photographing a display shelf or the like. Invented an image processing system.

The first invention is an image processing system that identifies an object reflected in image information, and the image processing system is a second region that satisfies a predetermined condition in a first region of the image information to be processed. The candidate processing unit that determines the layout similarity by using the layout information of the above and the layout information of the blocks in the sampled object to identify the target object to be the candidate for the comparison process, and the image of the processed object. By performing comparison processing using the first area or the second area of information and the sample information or block of the object specified as a candidate by the candidate processing unit, the object reflected in the first area is performed. It is an image processing system having a comparison processing unit for specifying an object.

With the configuration as in the present invention, the number of objects to be compared in the comparison process (matching process) can be reduced, so that the calculation time can be reduced as compared with the conventional case.

In the above-described invention, the candidate processing unit calculates the area of the intersection and / or the difference between the layout information of the second region and the layout information of the block of the object to be sampled, and uses the calculation result. A predetermined evaluation value is calculated, and when the calculated evaluation value is equal to or larger than the threshold value, the object can be added to the candidate list, and can be configured like an image processing system.

In order to narrow down the objects to be candidates for the comparison process, it is preferable to execute the process as in the present invention. Moreover, since the layout information is not image information but coordinate information, high speed can be realized even if these processes are executed.

In the above-described invention, the candidate processing unit specifies a second region satisfying a predetermined condition in the first region of the image information to be processed, and the second region in the predetermined range of the specified first region. Image processing that determines the similarity of layouts by using the layout information of the above and the layout information of the blocks of the area corresponding to the size of the predetermined range of the first area in the image information of the object to be the sample. It can be configured like a system.

The surface of the object is not necessarily a flat surface, but may be a curved surface such as a cylinder. When the surface of the object is a curved surface, the image information obtained by photographing the object becomes more distorted as it moves away from the center in the left-right direction. Therefore, by cutting out the vicinity of the central part of the first region and performing matching processing of the layout information in the image information of the object to be sampled with a size corresponding to the vicinity of the cut out central part, the surface of the object can be made. The effect of distortion due to the curved surface can be reduced.

In the above-described invention, the candidate processing unit performs a process of determining the similarity of the layout by performing a process of determining the similarity of the layout with a region corresponding to the size of a predetermined range of the first region in the image information of the object to be the sample. The comparison processing unit shifts the sample information or block of the region determined to have the highest similarity in the image information of the target object by the candidate processing unit, and the first region of the image information of the processing target. Alternatively, by performing comparison processing using the second region, it is possible to configure the image processing system to identify the object reflected in the first region.

When cutting out the vicinity of the central part, the accuracy of the comparison processing can be improved by performing the comparison processing while shifting it and then using the sample information of the region judged to have the highest similarity.

In the above-described invention, the comparison processing unit associates the second region of the image information of the processing target with the block of the object specified as a candidate by the candidate processing unit, and the associated second region. By determining the similarity of image information and / or character information using the block and the block, it can be configured like an image processing system that executes the comparison process.

In the comparison process, blocks may be compared as in the present invention.

In the above-described invention, the image processing system generates plane-expanded image information developed on a plane for a part or all of a three-dimensional model generated by using the image information and the corresponding depth information, and the generated image information is generated. It can be configured like an image processing system having an image information processing unit that uses a part or all of the plane-expanded image information as the image information to be processed.

The image information to be processed may be image information in which image information obtained by photographing a display shelf or the like is handled as it is in two dimensions, or image information in which a three-dimensional model is once generated and the image information is expanded in a plane. As a result, the processing can be executed more accurately.

A seventh invention is an image processing system that identifies an object reflected in image information, and the image processing system includes layout information of a block that satisfies a predetermined condition in a face region of the image information to be processed. Using the layout information of the block of the object to be sampled, the similarity of the layout is judged, and the candidate processing unit that identifies the object to be the candidate for the comparison processing and the face area or the face area in the image information of the processing object or By performing comparison processing using the image information and / or feature amount of the block area, the sample information of the object specified as a candidate by the candidate processing unit, the image information and / or feature amount of the block area, and so on. It is an image processing system having a comparison processing unit for identifying an object reflected in the face region.

Even with the configuration as in the present invention, the same technical effect as in the first invention can be obtained.

The first invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer uses the layout information of the second area that satisfies the predetermined condition in the first area of the image information to be processed and the layout information of the block in the sample object to obtain the similarity of the layout. A candidate processing unit that makes a determination and identifies an object to be a candidate for comparison processing, a first region or a second region of image information of the processing target, and a sample of the object specified as a candidate by the candidate processing unit. This is an image processing program that functions as a comparison processing unit that identifies an object in the first area by performing comparison processing using information or blocks.

The seventh invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer determines the layout similarity using the layout information of the block that satisfies the predetermined condition in the face area of the image information to be processed and the layout information of the block of the sampled object, and compares them. A candidate processing unit that specifies an object to be processed, an image information and / or a feature amount of a face area or a block area in the image information of the processing target, and a sample of the object specified as a candidate by the candidate processing unit. This is an image processing program that functions as a comparison processing unit that identifies an object reflected in the face area by performing comparison processing using information, image information in a block area, and / or a feature amount.

By using the image processing system of the present invention, the accuracy of matching processing is improved without requiring more calculation time than before when identifying an object such as a product shown in image information obtained by photographing a display shelf or the like. It becomes possible to make it.

It is a block diagram which shows typically an example of the whole processing function of the image processing system of this invention. It is a block diagram which shows typically an example of the hardware composition of the computer used in the image processing system of this invention. It is a flowchart which shows an example of the processing process of the whole processing in the image processing system of this invention. It is a flowchart which shows an example of the processing process of the process of specifying a block in the image processing system of this invention. It is a flowchart which shows an example of the processing process of the candidate identification processing in the image processing system of this invention. It is a figure which shows an example of photographed image information. It is a figure which shows an example of the image information which performed the orthotopic processing on the photographed image information of FIG. It is a figure which shows an example of the image information of a face area. It is a figure which shows an example of the state which specified the block in the image information of a face area. It is the figure which showed typically the layout information of the block specified in the image information of a face area, and the case of interpreting it on the image information. It is a figure which shows typically the image information corresponding to the sample information, the layout information of a block in the image information, and the case of interpreting it on the image information. It is a figure which shows an example of the layout matching process schematically. It is a figure which shows typically the combination of the part which has the area of a corresponding block, and the part which has the area of a corresponding block. It is a figure which shows an example about FIG. It is a figure which shows the other example of the layout matching processing schematically. It is a figure which shows an example of tag information which shows the type of a block. It is a figure which shows an example of the correspondence relation between blocks. It is a figure which shows typically the process in Example 3. It is a figure which shows typically an example of the block and tag information type, and the narrowing-down table when the appearance frequency is used as a comprehensive index value. It is a figure which shows typically the correspondence relationship between the photographed image information and the depth map by depth information. It is a figure which shows an example of the process of 3D modeling based on photographed image information and a depth map. It is a figure which shows an example of the plane development process typically. It is a figure which shows typically the stitching process which is executed after 3D modeling. It is a figure which shows typically the stitching process which is executed after 3D modeling. It is a figure which shows typically the process of specifying a reference in the viewpoint direction determination process. It is a figure which shows typically an example of the process of labeling each surface. It is an image diagram which identifies the face when the package type is a can. It is an image diagram which identifies the face when the package type is a bottle. It is an image diagram which identifies the face when the package type is a box. It is an image diagram which identifies the face when the package type is a hanging product.

FIG. 1 shows a block diagram of an example of the overall processing function of the image processing system 1 of the present invention. The image processing system 1 uses a management terminal 2 and an input terminal 3.

The management terminal 2 is a computer that realizes the core processing function of the image processing system 1. Further, the input terminal 3 is a terminal for acquiring image information obtained by photographing a display shelf of a store or the like. In addition, an object such as a product used as a sample used in the comparison process (matching process) described later may be photographed and acquired.

The management terminal 2 and the input terminal 3 in the image processing system 1 are realized by using a computer. FIG. 2 schematically shows an example of the hardware configuration of the computer. The computer can input information from an arithmetic unit 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, and a display device 72 such as a display that displays information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits and receives the processing result of the arithmetic unit 70 and the information stored in the storage device 71 via a network such as the Internet or LAN.

When the computer includes a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used, for example, in portable communication terminals such as tablet computers and smartphones, but are not limited thereto.

The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be performed directly on the display with a predetermined input device (such as a pen for a touch panel) or a finger.

The input terminal 3 may include a photographing device such as a camera in addition to the above-mentioned devices. As the input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used. The input terminal 3 captures image information (photographed image information or object image information described later) by visible light or the like with a photographing device.

Each means in the present invention has only a logical distinction in its function, and may form the same area physically or substantially. The processing order of the processing in each means of the present invention may be changed as appropriate. In addition, a part of the processing may be omitted. For example, the process of determining the viewpoint direction, which will be described later, can be omitted. In that case, it is possible to execute processing on the image information that has not been processed to determine the viewpoint direction. Further, a part or all of the functions in the management terminal 2 may be executed in the input terminal 3.

The image processing system 1 includes an image information input reception processing unit 20, an image information storage unit 21, an image information processing unit 22, a candidate processing unit 23, a comparison processing unit 24, a sample information processing unit 25, and a sample information storage unit 26. ..

The image information input reception processing unit 20 receives input of image information (captured image information) photographed by the input terminal 3 and stores it in the image information storage unit 21 described later. For example, the input of the photographed image information of the display shelf of the store is accepted and stored in the image information storage unit 21. In addition to the captured image information, the input terminal 3 may accept input including information indicating the shooting date and time and the shooting target, for example, store identification information such as a store name, and image information identification information for identifying the image information. FIG. 6 shows an example of captured image information. FIG. 6 schematically shows photographed image information obtained by photographing a state in which a display shelf has three shelves and products are displayed there.

The image information storage unit 21 stores the captured image information received from the input terminal 3, the shooting date and time, the image information identification information, and the like in association with each other.

The image information processing unit 22 sets the captured image information received by the image information input reception processing unit 20 in a state of facing each other, and compares the captured image information with the sample information (matching processing). Performs face processing to specify the area (face area) to execute, and block identification processing to specify the feature area (block) that satisfies a predetermined condition in the face area.

In general, when the object to be photographed is simply photographed, it is difficult to take the image in the state of facing the object to be photographed, so the erecting process is corrected to the state of facing the object to be photographed. This is a process of transforming the image information so that the optical axis of the lens of the photographing device is the same as when the image is photographed from a sufficient distance along the perpendicular direction of the plane to be photographed. An example of such correction processing is trapezoidal correction processing. If the image information is distorted, distortion correction processing may be added. FIG. 7 shows the image information in a state in which the emplacement processing is executed on the captured image information of FIG.

If the captured image information is the image information captured in the face-to-face state, or if there is no distortion, it is not necessary to execute the emplacement processing and the distortion correction processing.

The captured image information may be any image information to be processed according to the present invention. The captured image information also includes the image information after the correction processing for the captured image information such as the emplacement processing is executed.

Further, when the image information processing unit 22 shoots a shooting target, when a plurality of images are shot, the image information processing unit 22 executes a process of synthesizing the images into one image information, and the image information obtained by the synthesizing process is positive. Placement processing, face processing, and block identification processing may be executed. A known method can also be used as a process of synthesizing a plurality of image information into one image information.

The face processing specifies an area (face area) for executing comparison processing with sample information described later in the captured image information. For example, in the case of photographed image information obtained by photographing a product display shelf and the sample information is a product, the area of the product displayed on the display shelf or the area of the product label is specified as the face area. When the product is a PET bottle beverage, the area of the product label is the face area, and when the product is a boxed product (for example, sweets), the entire product package is the face area. The face area can be set as appropriate. Even when the product is a PET bottle beverage, the product area may be the face area.

There are various methods for specifying the face area, and it can be arbitrarily set according to the characteristics of the sample information. When the sample information is a product (for example, a PET bottle beverage) and the area of the product label is specified as the face area from the photographed image information of the display shelf, for example, between the shelves of the display shelf. Identify the vertical thin and narrow shadows that occur between products in the area (shelf area), identify the repeating pattern of the image, identify the step on the upper side of the package, the product width is the same, etc. Specify the product area by specifying the delimiter position based on the constraint. Then, from the product area, a predetermined rectangular area is specified as a label area, and that area is specified as a face area.

As the method for specifying the face area, any method can be adopted depending on the product category and the product form, and the method is not limited to the above. Further, the correction input by the operator may be accepted for the automatically specified face area. Further, the input of the position of the face area may be accepted from the operator.

Further, the image information processing unit 22 may specify the face region by using deep learning. In this case, the above captured image information is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the face region is set based on the output value. It may be specified. Further, as the learning model, it is possible to use various captured image information in which the face region is given as correct answer data.

The image information processing unit 22 cuts out the face region specified as described above. Cutting out the face area may actually cut out the face area specified from the captured image information, and also includes setting the area as the processing target of the processing described later.

When cutting out the face area, the face area specified by each of the above processes may be cut out as it is, or the face area may be specified by a plurality of methods and the target to be cut out using the result of the face area specified by each method. The face area to be used may be specified. For example, the area of a product is specified by specifying the narrow vertical shadow generated between the products in the area between the shelves of the display shelf (shelf area), and a predetermined area is specified from that area. A method of specifying a rectangular area and a method of specifying a face area by deep learning are performed, and the results of the face areas specified by each of these methods are finally cut out by a predetermined calculation. May be specified. The method for specifying the face area to be cut out is not limited to the above, and can be set arbitrarily.

FIG. 8 shows an example of a case where a product (label portion of a PET bottle beverage) is cut out as a face area from the photographed image information.

When the image information processing unit 22 specifies the face area, the image information processing unit 22 specifies a rectangular area having information that can identify the product as a block in the face area. The area specified as a block is an area of the image information of the face area that is concentrated inside and has common features, and is a characteristic area that characterizes the image information of the face area. The area specified as a block is the area where the product name, product logo, etc. are displayed.

There are, for example, three methods for the image information processing unit 22 to identify a block from the face area, but the method is not limited to these, and a plurality of methods may be combined.

The first method is a method of extracting and determining image features common to blocks of an object. That is, in the image information of the face region, a rectangular region having a feature of being composed of a limited number of colors that can be clearly distinguished from each other and having a clear color-to-color boundary (clear edges) is blocked. It is a method to specify as.

First, the colors of each pixel constituting the image information in the face region are detected by reducing the number of colors to a predetermined number of palette colors. In addition, the image information of the face area is divided into meshes of a predetermined size, and a distribution map for each palette color is generated. Then, in each pallet, a rectangular area is specified by detecting a local group that is distributed at a certain density or higher and whose vertical and horizontal sizes fall within a predetermined threshold value.

Then, for each group, the distribution map of all palette colors is referred to, and it is determined whether the number of palettes contained in the rectangular area is within a predetermined number, for example, 3, and if it is within the predetermined number, the group area. It is determined whether or not there is an edge inside a predetermined threshold value or more. For edge determination, edge detection processing is performed on the image information in the face area, and it is possible to determine whether or not the image information is equal to or greater than the threshold value. Then, the rectangular area surrounding the group that satisfies these conditions is specified as a block.

The block can be identified from the face area by the above method.

As a second method for identifying a block from the face area, a feature amount such as a local feature amount is extracted from the image information of the face area. Then, a region in which common features are distributed over a wide range (a range of a predetermined size or more) is detected, and that region is erased as the "ground" part of the face region. Then, for the remaining areas, the areas where the features common to each other are distributed are specified, and the rectangular area surrounding them is specified as a block.

As a third method for identifying a block from the face region, the block can be detected from the image information of the face region by AI technology such as machine learning or deep learning. In this case, the image information of the face region is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the block is divided based on the output value. It may be specified. As the learning model, it is possible to use image information of various face regions in which blocks are given as correct answer data. Note that the face area and the block may be specified at the same time from the captured image information.

In addition, as another example of using AI technology such as machine learning or deep learning in the third method, there are the following. In advance, the feature amount in the block area (the feature amount may be a plurality of colors, local feature amounts, etc.) in the correct answer data (a block is given to the image information of various face areas) given as the above-mentioned learning model, and the block. A judgment module (judgment machine) such as SVM (support vector machine) is constructed by extracting the features outside the region and using each as a positive example and a negative example. Then, for this judgment module such as SVM, each point of the image information in the face area, for example, the image information in the face area is divided into a mesh of a predetermined size, points in the mesh are input, and each point is blocked. It is made to judge whether or not it belongs to the area of. In this determination result, the area surrounding the mesh area determined to belong to the block area is specified as a block.

The process of identifying the block from the face area is not limited to the above three methods, and other methods may be used. Further, a plurality of the above three methods or a plurality of other methods may be combined and specified.

FIG. 9 schematically shows an example when a block is specified from the face area. FIG. 9 shows a case where the block is specified from the face area of FIG. The rectangular area in FIG. 9 is the specified block.

When the image information processing unit 22 specifies a block from the image information in the face area as described above, the candidate processing unit 23 includes the specified block and a block corresponding to the sample information stored in the sample information storage unit 26 described later. The layout matching process, which is the process of determining the similarity of the layouts between the two, is executed. As the layout matching process, for example, blocks are extracted from the image information in the face area. Extracting the block here means, for example, extracting the layout information (position information) of the block in the image information of the face area. This is schematically shown in FIG. FIG. 10 (a) shows a block in the image information of the face area, FIG. 10 (b) shows the layout information in coordinates, and FIG. 10 (c) interprets the layout information on the image information. This is a schematic representation of the case of Each block in FIG. 10A is named "region" to identify each block, and FIG. 10A shows a case where there are five blocks from "region1" to "region5". ing. The layout information of each block is represented by the coordinates of two points in the rectangular area, for example, the upper left and the lower right, where "a" indicates the x coordinate and "b" indicates the y coordinate. Note that the layout information may be indicated by four points instead of two points, or various methods can be used as long as the position information of each block can be specified, such as indicating the distance from one point. In FIG. 10 (c), the layout information of FIG. 10 (b) is shown by the image information of the face area, and the area of each block is shown by the relative coordinates when the upper left of the image information of the face area is the origin. There is.

Similarly, the candidate processing unit 23 extracts a block corresponding to the sample information of the object stored in the sample information storage unit 26. FIG. 11 schematically shows a block corresponding to the extracted object. FIG. 11A shows the extracted object or the image information corresponding to the extracted object, FIG. 11B shows the block in the image information corresponding to the extracted object, and FIG. 11C shows the layout information. Is shown in coordinates, and FIG. 11D schematically shows a case where layout information is interpreted on image information. Since FIG. 11A shows the sample information of the object or the image information corresponding thereto, it is used in the comparison process for identifying the object reflected in the captured image information, as will be described later. Therefore, there is less noise than the face area cut out from the captured image information, and it is often clear. In addition, it is common that there are no unnecessary items such as stickers. As will be described later, the block in the image information of FIG. 11B may be specified by the operator, or may be automatically specified by the same process as the image information processing unit 22 described above. Alternatively, the automatically specified block may be modified by the operator as appropriate, such as selection or addition. Each block in FIG. 11B is named "REGION" to identify each block, and FIG. 11B shows a case where there are 6 blocks from "REGION1" to "REGION6". ing. Further, the layout information of each block is represented by the coordinates of two points in the rectangular area, for example, the upper left and the lower right, where "x" indicates the x coordinate and "y" indicates the y coordinate. Note that the layout information may be indicated by four points instead of two points, or various methods can be used as long as the position information of each block can be specified, such as indicating the distance from one point. In FIG. 11D, the layout information of FIG. 11C is shown by the image information of the face area, and the area of each block is shown by the relative coordinates when the upper left of the image information of the face area is the origin. There is.

The block in FIG. 10 (a) and the block in FIG. 11 (b) do not always match. For example, in FIG. 10 (a), “REGION 3” and “REGION 4” in FIG. 11 (b) are recognized as one block “region 2”, and in FIG. 10 (a), “REGION 2” in FIG. 11 (b) is recognized. Not recognized. Further, in FIG. 10A, “region 5” is excessively recognized because the label portion has a sticker.

The candidate processing unit 23 uses the layout information of the blocks extracted from the image information in the face area and the layout information of the blocks corresponding to the objects stored in the sample information storage unit 26 to determine the similarity of the layout information. Calculate the indicated evaluation value, for example, the degree of similarity. The evaluation value indicating the similarity of the layout information is not limited to the similarity of the layout information, and may be any information indicating the similarity. For example, when similarity is used as an evaluation value, various methods can be used to calculate the similarity. For example, the following processing can be used.

First, the size of the image information in the face region and the sample information corresponding to the object to be stored in the sample information storage unit 26 or the image information corresponding thereto are matched (the area in this case is the common S). Then, based on the block layout information in the image information of the face area and the sample information of the object stored in the sample information storage unit 26 or the layout information of the block corresponding to the corresponding image information, those intersections (common). Part) and the difference are calculated. Then, logical operations such as AND, OR, and NOT between blocks are performed, and the areas thereof are obtained. An example of this process is schematically shown in FIG. In FIG. 12, a part having a corresponding block area is shown in black, and a part having no corresponding block area is shown in white. FIG. 13 schematically shows this combination. FIG. 13 (a) shows a combination of areas calculated between blocks, and FIG. 13 (b) shows TP (True Positive) and FN (False Negative) for the four combinations of FIG. 13 (a). ), FP (False Positive), TN (True Negative), and the same names as the relationship between the prediction result and the true value. Here, the layout information of the image information corresponding to the object to be sampled is regarded as a true value, and the layout information of the image information in the face area is regarded as the prediction result for it. For example, TP is interpreted to mean that the location where the layout information of the image information in the face area is considered to be present actually matches the layout information in the image information corresponding to the sampled object.

Then, the candidate processing unit 23 calculates the similarity using each area calculated as shown in FIG. 13A. For example, when the F value is used as the similarity, it is calculated as the following equation 1.
(Number 1)
Here, Precision is the precision rate (the ratio of the data predicted to be positive that is actually positive), and Recall is the recall rate (the ratio of the data that is actually positive that is predicted to be positive). ..

For example, using the layout information of the block extracted from the image information of a certain face area and the layout information of the block corresponding to a certain object stored in the sample information storage unit 26, as shown in FIG. It is assumed that the area of the result of calculating the section, the difference, etc. and performing the logical operation between the blocks is shown in FIG. At this time, the precision rate is 0.86 (= 0.3 / (0.3 + 0.05)), and the recall rate (Recall) is 0.75 (= 0.3 / (0.3 + 0.1)). Therefore, the F value can be calculated as 0.80 (= 2 × 0.75 × 0.86 / (0.75 + 0.86)).

Then, if the F value calculated here is equal to or higher than the predetermined threshold value TH, the candidate processing unit 23 adds the identification information of the object to the candidate list.

In addition, the candidate processing unit 23 may use an index other than the F value in order to calculate the specificity, and the accuracy rate (Accuracy) is actually the case among the data predicted to be positive or negative. You may use (ratio of things) or specificity (specificity, which is the ratio of things that are actually negative and are predicted to be negative). The correct answer rate and specificity can be calculated using Equation 2.
(Number 2)

The candidate processing unit 23 executes this processing on the object to be compared stored in the sample information storage unit 26. The objects to be compared may be all the objects stored in the sample information storage unit 26, or may be some objects narrowed down by some method.

Then, the candidate processing unit 23 sorts in descending order of similarity added to the candidate list, and identifies the top predetermined number, for example, the top 10 objects as candidates.

The comparison processing unit 24 executes a comparison process (matching process) between the sample information of the target object specified by the candidate processing unit 23 and the image information of the face region. For example, by extracting the feature amount of the image information of the face area and executing the comparison processing between the extracted feature amount and the sample information such as the feature amount of the candidate object, the image information and the sample information of the face area can be obtained. Judges the similarity of the objects, and determines the identification information of the object included in the face area, for example, the product name.

As the comparison process, not only the entire image information in the face area is matched with the sample information, but also a part of the image information in the face area, for example, the feature amount of the image information in the feature area (block) is extracted and sampled. Information feature quantity and comparison processing may be executed.

In addition, the comparison process may be performed by a method other than the feature amount.

Further, the comparison processing unit 24 may execute the comparison processing with the sample information by using deep learning. In this case, all or part of the image information of the face region is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the output value thereof. Similar sample information may be identified based on. As the learning model, it is possible to use image information of various face regions in which sample information is given as correct answer data.

The sample information processing unit 25 generates sample information of the object used in the comparison process for identifying the object reflected in the captured image information, and stores it in the sample information storage unit 26 described later. The sample information is image information (object image information) obtained by photographing an object and / or information on a feature amount thereof. In the sample information, identification information of the object, for example, a product code such as a product name, a model number, and a JAN code, a block in the object image information, and its layout information are stored in association with each other. The image information of the object may be the image information of the entire object, or may be the image information of a part of the object, for example, the label portion when the object is a PET bottle beverage. When the feature amount information of the object image information is used as the sample information, it is not necessary to extract the feature amount each time the processing is performed. The block is a characteristic area that characterizes the image information of the object, and corresponds to, for example, a rectangular area that surrounds a product name, a logo, or the like.

The sample information processing unit 25 may generate one sample information for one object, or is in a position where the sample information is not captured when the object is photographed at a plurality of angles, for example, when the product is face-to-face. In order to capture the surface, sample information may be generated from angles such as front, back, top, bottom, and both sides. In addition, when one object has a plurality of exteriors (packages, etc.), the sample information and blocks for each exterior may be associated with one object. Moreover, although one block may be associated with the sample information, it is preferable that a plurality of blocks are associated with each other.

The sample information may be extracted from the extracted block by executing the same processing as the above-mentioned image information processing unit 22 on the object image information obtained by photographing the object to be sampled. The operator may select or add only the blocks necessary for identifying the product. Further, the operator may specify a block from the image information of the read object, instead of automatically executing the extraction process as in the image information processing unit 22.

The sample information storage unit 26 uses the sample information generated by the sample information processing unit 25 as object identification information, for example, a product code such as a product name, a model number, and a JAN code, a block in the object image information, and its layout information. Correspond and memorize.

Next, an example of the processing process using the image processing system 1 of the present invention will be described with reference to the flowcharts of FIGS. 3 to 5. In this embodiment, a case where a display shelf of a store is photographed and a product on the display shelf is specified will be described. Therefore, the photographed image information is the display shelf of the store where the product is displayed, and the product may be displayed as the object in the sample information.

First, the operator photographs the product as the sample information from the opposite position, and stores the photographed object image information, identification information, product name, and product code in the sample information storage unit 26 in association with each other. Further, the sample information processing unit 25 executes the same processing as the processing in the image information processing unit 22 on the object image information, and extracts blocks. Then, among the extracted blocks, the blocks and their position information (layout information) are stored in the sample information storage unit 26 as blocks used as sample information, for example, only blocks containing product names, logos, etc. are associated with sample information. I will remember it. In addition, by executing a predetermined process from the object image information, feature quantities such as local feature quantities are extracted and stored in association with the object image information.

Processing for such sample information is performed in advance for a plurality of products, preferably all products that may be displayed on the display shelf.

Then, when the image information input reception processing unit 20 of the management terminal 2 receives the photographed image information obtained by photographing the display shelf of the store from the input terminal 3 (S100), the image information storage unit 21 receives the photographed date and time and the identification information of the photographed image information. And memorize it in association with.

The image information processing unit 22 executes the orthostatic processing for correcting the received captured image information so that the captured image information faces each other (S110). Then, face processing for specifying the face area from the emplaced captured image information is executed (S120), and the block is specified from the specified face area (S130).

As described above, in order to identify the block from the face area, there are various methods such as the first method to the third method as described above, but in the case of the first method, for example, Execute the following processing. An example of the processing process in this case will be described with reference to the flowchart of FIG.

First, with respect to the color of each pixel constituting the image information in the face region, the number of colors is reduced to a predetermined number of palette colors such as 16 colors representing the whole, and the palette color is detected (S200). Further, the short side of the face region is divided into, for example, 16 divided square meshes, which palette color is used for each mesh, and a distribution map of the presence or absence of that color is generated for each palette color (S210).

Then, for each mesh of the image information in the face region, expansion processing is performed on the distribution map so as to select excursions at a certain distance as candidates, and the group in which the distribution map falls within a predetermined range is detected ( S220). The predetermined range is, for example, a threshold condition in which the width is 1/10 to 2/3 of the whole, the height is 1/2 to 1/2 of the whole, and the area is 1/100 to 1/10 of the whole. Detects a group of distribution maps (distribution maps that have undergone expansion processing) that fit in.

Then, for each group, the distribution map of all palette colors is referred to, and it is determined whether the number of palettes included in the area is a predetermined number, for example, 3 or less (S230). That is, referring to the distribution map of all palette colors in the group area, if a large number of colors are included in the group area, it is highly likely that it is an image illustration, and it is a characteristic area that identifies the product. Exclude such groups as they are not considered to be certain logos.

In S230, when the number of pallets included in the group area is within a predetermined number, it is determined whether or not the edge is equal to or more than a predetermined threshold value in the group area (S240). Regarding the edge determination, it is possible to perform edge detection processing on the image information in the face area in advance and determine whether or not the image information is equal to or greater than the threshold value. If the edge does not exist above the threshold value in the group area, it is highly possible that it is not a logo, so that group is excluded.

Then, the rectangular area surrounding the group satisfying these conditions is specified as a block (S250).

This process is executed for the group grouped in S220 (S260).

The block can be identified from the face area by the above method.

When the image information processing unit 22 specifies a block from the image information in the face area, the candidate processing unit 23 arranges between the specified block and the block corresponding to the object stored in the sample information storage unit 26 described later. By executing the matching process, a candidate product is specified (S140). An example of the processing process of the specific processing of the candidate product will be described with reference to the flowchart of FIG.

Specifically, the candidate processing unit 23 bases the block layout information extracted from the image information of the face area and the block layout information corresponding to the products for all the products to be compared. Intersections, differences, etc. are calculated, logical operations such as AND, OR, and NOT between blocks are performed, and the areas are obtained for each to perform layout matching (S310). Then, for example, the similarity such as the F value is calculated, and if it is equal to or higher than the predetermined threshold value TH, the product is added to the candidate list (S320). Then, the same processing is executed for the next product (S330).

Then, when layout matching processing is performed on all the products to be compared, the candidate processing unit 23 sorts the similarities added to the candidate list in descending order (S340), and among them, the top predetermined number, for example, the top 10 products. Is specified as a candidate.

When the candidate processing unit 23 specifies a product that is a candidate for comparison processing in S140, the comparison processing unit 24 extracts the feature amount of the image information in the face region, and the extracted feature amount and the sample of the product specified as the candidate. The comparison process is performed by comparing with the feature amount of the information (S150). In this comparison, the size of the image information in the face area and the size, scale, rotation, etc. of the image information in the sample information are made as close as possible before the comparison process is performed. This comparison process is performed for each product specified as a candidate.

Then, as a result of this comparison processing, among the products identified as candidates, the product determined to have the highest degree of similarity is identified as a product in the face area, and the face area, product identification information, product name, product code, etc. (S160).

Since a plurality of products are generally displayed on the display shelf shown in the photographed image information, it is common that a face area corresponding to the displayed products is cut out. Therefore, by executing each of the above-mentioned processes for each face area, it is possible to identify the products displayed on the display shelf and the display position. The display position of the product can be specified by the order of the vertical and horizontal directions of the face area, and can also be specified as the coordinates in the captured image information.

When the surface of the object is a curved surface, image distortion occurs as it shifts in the left-right direction. Therefore, it is advisable to execute the following processing.

That is, when calculating the similarity of the layout information, the candidate processing unit 23 determines the layout information of the block in the entire image information of the face region and the block in the entire image information corresponding to the object, as shown in FIG. In addition to executing the process of calculating the similarity using the layout information of, the layout information of the block in a part of the image information of the face area and the layout information of the block in a part of the image information corresponding to the object are used. The similarity may be calculated. An example of this process is schematically shown in FIG.

As shown in FIG. 15, in the case of this embodiment, a certain ratio on the left and right of the image information in the face area, for example, about 1/4 is excluded, and about 1/4 each from the center line of the image information in the face area (center). About the layout information of the block in the image information cut out (about 1/2 part) and the image information corresponding to the object, the width equal to or almost the same as the width cut out from the image information of the face area described above is cut out and cut out. Use the layout information of to calculate intersections and differences. Then, logical operations such as AND, OR, and NOT between blocks are performed, and the areas thereof are obtained. Then, the similarity is calculated in the same manner as described above.

Next, the same process is executed by shifting the cutout position of the image information corresponding to the object by a predetermined amount. Note that the layout information of the block in which the cutout position is shifted may be prepared in advance.

Then, the highest similarity is compared with the threshold value TH, and if it is equal to or higher than the threshold value TH, it is added to the candidate list. Further, the cutout position of the image information corresponding to the sample information at the time of the highest similarity when the threshold value TH or more is obtained may be stored. In this case, when the comparison processing unit 24 compares the feature amount of the image information in the face area with the feature amount of the image information corresponding to the sample information, the image information corresponding to the sample information is cut out at the stored cutout position. ， It is preferable to compare the features.

The candidate processing unit 23 does not compare the highest similarity with the threshold TH, but compares it with the threshold TH each time the similarity is calculated, and if it is equal to or higher than the threshold TH, adds it to the candidate list. The process of shifting the cutout position may be completed, and the process of blocking the image information corresponding to the next sample information may be performed. As a result, the process of cutting out the image information corresponding to the object by shifting it by a predetermined amount can be completed when the similarity becomes equal to or higher than the threshold value TH, so that the processing speed can be increased.

In the above description, the layout matching process is performed by shifting the layout information of the block in the image information obtained by cutting out about 1/2 of the central part of the image information in the face area in the same range in the image information of the object by a predetermined amount. However, the opposite processing may be performed. That is, the layout matching process is performed by cutting out the block layout information in the image information obtained by cutting out about 1/2 of the central portion of the image information of the object, and cutting out the same or almost the same width in the image information of the face area. It may be repeated by shifting by a predetermined amount.

In the first and second embodiments, the comparison processing unit 24 includes a feature amount of a part or all of the corresponding image information of the sample information of the object specified as a candidate by the candidate processing unit 23, and a face area. By comparing the feature amount of a part or all of the image information of, the similarity between the sample information and the image information of the face area is determined, and the identification information of the object included in the face area, for example, the product name, etc. Was judged.

In this embodiment, the similarity between the sample information and the image information of the face area is determined by comparing the block of the object specified as a candidate by the candidate processing unit 23 with a part or all the blocks of the face area. Then, the case of determining the identification information of the object included in the face area, for example, the product name, etc. will be described.

In this case, the sample information storage unit 26 may store tag information indicating the type of the block in association with the block. An example of tag information is shown in FIG. Then, each tag is subjected to predetermined preprocessing, and the processing result is stored in association with the sample information storage unit 26. For example, if the tag information is an image image, the image information is cut out and registered, if the tag information is a product name logo or copy, or if it is rated, OCR processing is performed and stored as text information.

Then, the comparison processing unit 24 first identifies the correspondence between the blocks when comparing the block of the object specified as a candidate by the candidate processing unit 23 with a part or all of the blocks in the face area. In this case, after matching the position and scale of the image information of the face area and the image information of the sample information, the block having the largest intersection of the image information of the face area when viewed from the block corresponding to the sample information is obtained, or The correspondence between blocks can be specified by specifying the center of the layout information of the block corresponding to the sample information, finding the block in the image information of the face area that is closest to the center, and so on. An example of the correspondence between blocks is shown in FIG.

Then, when the correspondence between the blocks is specified, the preprocessing for the block corresponding to the sample information is executed for the image information block in the corresponding face area, and the result is calculated. Then, the results of the preprocessing are compared. For example, in the case of image information, similarity is evaluated using features, and in the case of text information, text matching is performed. An example of this process is shown in FIG.

The comparison processing unit 24 calculates a comprehensive index value of the degree of agreement with the sample information by using the result of comparing the blocks as described above. For example, a setting is made in advance so that the degree of matching of product names is highly evaluated and the degree of matching of manufacturer names is low, and a comprehensive index value is calculated based on the setting. In the evaluation of the degree of matching, for example, a numerical value that determines the weight at the time of matching is related according to the type of tag information corresponding to the block. The weighting value may be determined by the type of tag information corresponding to the block, or may be, for example, the frequency of appearance in the product category when the object is a product. When using the frequency of appearance, it is assumed that the target can be identified by making a table of what and what matches and how far it can be narrowed down, and if the target such as a product is narrowed down to one. FIG. 19A shows an example of the correspondence between the block and the type of tag information, and FIG. 19B shows an example of the narrowing table.

The comparison processing unit 24 performs the above comparison processing on an object such as a product specified as a candidate, and identifies the object having the highest overall index value as a product in the face region.

In the first to third embodiments, the case where the image is captured with the two-dimensional image information as the captured image information has been described, but the three-dimensional model is taken from the captured image information in consideration of the distance information between the imaging target and the imaging device. The image information when the image is converted and then projected onto the two-dimensional image information may be used as the captured image information to be processed by the image processing system 1 of the present invention. In this case, the same processing is executed for the image information of the object in the sample information. The processing in this case will be described below.

In the input terminal 3 of this embodiment, in addition to photographing image information by visible light or the like with a photographing device such as a camera, distance information (depth information) between the imaged object and the photographing device is acquired as two-dimensional information. The information to which the depth information is mapped is referred to as a depth map in this specification. The depth map corresponds to the image information taken by the photographing device, and the depth information up to the object reflected in the photographed image information is mapped to the position corresponding to the photographed image information. In the depth map, at least the range corresponding to the captured image information is divided into meshes, and the depth information is given to each mesh. The vertical and horizontal sizes of the mesh depend on the accuracy of the depth information that can be acquired, but the smaller the vertical and horizontal sizes of the mesh, the higher the accuracy. Usually, it is in the range of 1 mm to several mm, but it may be less than 1 mm or 1 cm or more.

When acquiring depth information, when shooting a subject, measure the amount of reflected light from each direction by irradiating light rays of a specific wavelength, such as infrared rays, and the time until the reflected light arrives. Acquires distance information (depth information) to the shooting target, or irradiates a light beam of a specific wavelength, for example, an infrared dot pattern, and measures the distance information (depth information) from the reflection pattern to the shooting target. In addition to the above, there are methods that utilize the parallax of a stereo camera, but the method is not limited to these. If the depth information is given by the distance from the imaging device (one point on three dimensions), the depth information is converted into the depth information from the imaging surface (plane) of the imaging device. .. The input terminal 3 may be provided with a device (depth detection device) for acquiring depth information in this way.

The positional relationship between the image information and the depth information corresponds to each other. FIG. 20 shows the correspondence between the image information taken with the display shelf as the shooting target and the depth map based on the depth information. FIG. 20 (a) is a diagram schematically showing image information taken, and FIG. 20 (b) is a diagram schematically showing a corresponding depth map.

The image information input reception processing unit 20 receives input of image information to be photographed such as a display shelf taken by the input terminal 3 and stores it in the image information storage unit 21. Further, the image information input reception processing unit 20 receives the input of the depth information and the depth map for the image information photographed by the input terminal 3, and stores the image information in the image information storage unit 21 in association with the photographed image information. When the depth information is received from the input terminal 3 instead of the depth map, it is mapped to generate the depth map.

It is preferable that the image information, the depth information, and the depth map of the photographed object are photographed and the information is acquired at the same time when the image information is photographed, and the input of the information is accepted.

The image information processing unit 22 converts the display shelf or the like into a three-dimensional model based on the photographed image information and the depth map. That is, since the captured image information and the depth map correspond to each other, first, the image information captured is converted into a three-dimensional model based on the depth information in the depth map, and the captured image information is associated with the texture mapping. To do. FIG. 21 shows an example of the three-dimensional modeling process based on the captured image information and the depth map. Note that FIG. 21 shows a case where a PET bottle of a beverage is three-dimensionally modeled. FIG. 21A is captured image information. Since the image information processing unit 22 acquires the depth map at the same time, it converts it into a three-dimensional model based on the depth information of each mesh in the depth map. Then, by performing texture mapping in which the color information of the corresponding pixel of the captured image information is pasted at the position of each mesh in the depth map, three-dimensional modeling is performed as shown in FIGS. 21 (b) to 21 (d). can do. 21 (b) shows a viewpoint from the left, FIG. 21 (c) shows a viewpoint from the front, and FIG. 21 (d) shows a viewpoint from the right.

When the image information processing unit 22 generates a three-dimensional model based on the image information obtained by photographing the display shelf or the like and the depth map, the image information processing unit 22 executes a plane expansion process for plane expansion of the three-dimensional model. Planar expansion means expanding a three-dimensional model into a plane (two-dimensional). There are various methods for expanding the three-dimensional model in a plane, and the method is not limited.

As an example of the plane expansion method, there is a method of expanding to a plane by defining the center of projection inside the three-dimensional model and projecting from there. An example of the plane expansion process in the image information processing unit 22 will be described with reference to FIG. FIG. 22A is image information obtained by photographing a PET bottle, and the top view thereof has a substantially hexagonal shape. The actual shape is finer and uneven, but for the sake of explanation, it is shown as a hexagon. Then, when the image information of FIG. 22A is three-dimensionally modeled using the depth map, the positional relationship when viewed from above is shown in FIG. 22B. In this case, since there is no depth information of the part behind the photographing device (camera), there is a blank when viewed from above as shown in FIG. 22B. Then, the central axis of the projection is specified at an arbitrary position of the object to be developed in a plane. For example, if a product is displayed on a display shelf and its shape is not a box shape or a bag shape but a cylindrical shape or a bottle shape and the surface is a regular curved surface instead of a flat surface, its width and its surface From the shape, find the center point of the rotation axis and specify it as the center axis of projection. Note that the central axis of projection may be specified by any method, not limited to the above. FIG. 22C shows a state in which the central axis of projection is specified. Then, from the central axis of the specified projection, the surface of the object is projected onto an arbitrarily set projection surface. This is schematically shown in FIG. 22 (d). As shown in FIG. 22D, when the positions of the plane of the three-dimensional model and the projection plane do not match, the vertical and horizontal dimensions of the image information of the projection result according to the distance between them are set to an appropriate scale ratio. Adjust by. On the other hand, when the positions of the plane and the projection plane of the three-dimensional model are matched as shown in FIG. 2 (e), it is not necessary to provide the adjustment processing of the scale ratio. Note that FIGS. 22 (b) to 22 (e) are views from above.

The image information processing unit 22 can expand the three-dimensional model into two dimensions (plane) and convert it into two-dimensional image information (plane expansion image information) by performing such a plane expansion process. In the above description, the process of converting the 3D model to the plane development image information is performed by the projection process, but in the plane expansion process, the process of converting the 3D model to the plane development image information may be performed by other processing. ..

If the image information processing unit 22 handles the plane-developed image information generated in this way as the captured image information in the above-described first and second embodiments and executes the processing after the face processing, the first and second embodiments are performed. The same process as in 2 can be executed.

Further, the sample information processing unit 25 generates a three-dimensional model from the image information obtained by photographing the object and the depth map, and the plane development image information based on the three-dimensional model, similarly to the image information processing unit 22 described above. To generate. Then, by storing all or a part (for example, a label portion) of the plane-developed image information in the sample information storage unit 26 as sample information, the same processing as in the first and second embodiments can be executed.

A case where the stitching process is executed will be described as a modification of the process of the fourth embodiment.

When a plurality of images are photographed, the image information processing unit 22 may execute a stitching process for synthesizing a plurality of images into one image information. The stitching process may be performed before the 3D modeling or after the 3D modeling. A known method can be used for both the stitching process before the three-dimensional modeling and the stitching process after the three-dimensional modeling.

If it is performed before the three-dimensional modeling, the stitching process is executed by detecting the corresponding points between the captured image information. At this time, since the corresponding points can be specified between the captured image information, the depth maps corresponding to the positions of the specified corresponding points are also associated with each other in the same manner. This allows stitching to be performed before 3D modeling.

In addition, when performing after 3D modeling, the similarity of the uneven shape in the two 3D models is specified from the depth information, and the shapes are similar (the pattern of the unevenness based on the depth information is included in the predetermined range. The stitching process is executed after the 3D modeling by searching for the corresponding points between the captured image information that is the source of the 3D model that corresponds to that area. it can. This is schematically shown in FIG. 23 (a) and 23 (b) are photographed image information obtained by photographing the shelves arranged side by side, FIG. 23 (c) is a three-dimensional model based on FIG. 23 (a), and FIG. 23 (d) is FIG. 23 ( It is a three-dimensional model based on b). In this case, as shown in FIG. 24, for example, in the captured image information, if the reference 101 for stitching processing, for example, the shelf step is captured so as to be reflected, the reference 101 is in the same viewpoint direction. Rotate the 3D model as in. Then, the similarity of the uneven shape is determined based on the depth information. Then, the stitching process can be executed by searching for the corresponding points in the specified similar area 100. FIG. 24A shows a state in which the three-dimensional model of FIG. 19A is rotated so as to have the same viewpoint direction as that of FIG. 24B. The broken line region 100 in FIGS. 24 (a) and 24 (b) is a region 100 having a concave-convex shape determined based on the depth information. Further, the alternate long and short dash line region 101 is a shelf stage that serves as a reference for rotating the three-dimensional model so that the three-dimensional model is oriented in the same viewpoint direction. The reference 101 can be arbitrarily specified depending on the object to be photographed, but it is preferably composed of two or more straight lines.

By executing the above processing, the stitching processing can be executed either before or after the three-dimensional modeling. Further, when the information of a plurality of captured images is not used, the stitching process may not be executed.

As a modification of the process of the fourth embodiment, the image information processing unit 22 executes the three-dimensional modeling process and then determines the viewpoint direction so that the three-dimensional model to be photographed faces the position. May be executed, and then the process of plane expansion may be executed. For example, when the object to be photographed is a product display shelf or a product located there, the viewpoint direction is determined so that the display shelf or the product faces the position (front position).

As a process for determining the viewpoint direction, the viewpoint direction may be determined so that the reference in the three-dimensional model satisfies a predetermined condition for the position to face each other. The criteria and conditions can be arbitrarily set according to the shooting target and the like. For example, when photographing a display shelf, the front 102 of the shelf of the display shelf is used as a reference, and the condition is that the front 102 of the plurality of shelves of the display shelf are all horizontal and perpendicular to each other. This can be achieved by moving the viewpoint to the center in the vertical and horizontal directions of the shooting range. Normally, when the product display shelf is photographed, the front surface 102 of the shelf is located in the foreground (the frontmost position) of the display shelf, so the depth information of the three-dimensional model indicates that the shelf is in front of the shelf. The front surface 102 can be specified. FIG. 25 schematically shows this process. FIG. 25 (a) is photographed image information of the display shelf, and FIG. 25 (b) is a diagram expressing the depth information in black and white density in the depth map of the display shelf (black case). Located in the foreground (the value of the depth information is small), FIG. 25 (c) is a three-dimensional model based on the depth map, in which the depth information is converted into colors. Then, the image information processing unit 22 identifies the front surface of the shelf of the display shelf based on the depth information, and determines the viewpoint direction so that the shelf satisfies the above-mentioned conditions. Further, the same criteria may be used for the reference used in the viewpoint direction determination process and the reference in the stitching process after the above-mentioned three-dimensional modeling. For example, in either case, the front surface of the shelf of the display shelf can be used as a reference.

In FIG. 25 (c), for the sake of clarity, the case where the captured image information is processed in the state before the texture mapping is shown. However, the viewpoint direction determination processing is executed for the three-dimensional model after the texture mapping. You may. The process of determining the viewpoint direction is not limited to the above, and can be realized by any process.

If the captured image information is originally captured from a position facing it, the viewpoint direction determination process does not have to be executed.

In the processing of Examples 4 to 6, the image information of the face region was specified by executing the face processing on the plane development image information, but the face region was specified and specified from the three-dimensional model. The plane expansion process may be executed for the face region to obtain the plane expansion image information.

There are various methods for specifying the face region from the three-dimensional model, and it can be arbitrarily set according to the characteristics of the sample information. For example, in a 3D model, the position and range of a surface that satisfies a predetermined condition are specified, and the surface is matched with whether or not the condition corresponding to the type of the surface model of the predetermined sample information is satisfied, and the condition is satisfied. Specify the face as a face area. For example, as described above, in the case where the sample information is a product, a three-dimensional model is generated from the photographed image information obtained by photographing the display shelf, and the face area of the product is specified, the following processing can be executed. ..

First, after filtering out the high frequency to exclude fine irregularities, a surface that is close to vertical and has an area above a certain level facing directly, an upright cylindrical surface, and a surface of a polyhedral region that is convex and distorted as a whole. Is detected, and the position and range in the three-dimensional space are specified. For example, if the depth information is within a certain range and the normal is stable and facing the front, there is a strong vertical correlation between the plane and depth information, and if it is convex in the horizontal direction, it is a cylinder or horizontal direction. And in either the vertical direction, it is specified as a plane in the region of a convex or distorted polyhedron if the normal is not stable. Then, each detected surface is labeled. FIG. 26 shows a state in which five surfaces are detected and labeled from 1 to 5 respectively. Then, for the detected surface, the distance from the foreground (this is used as the level) is calculated in order from the foreground based on, for example, the average value of the depth information. Then, based on the calculated level, the context of each surface is specified. In FIG. 26, the level of the surfaces 1 and 2 is 0 cm, the level of the surfaces 3 and 4 is 3 cm, and the level of the surface 5 is 20 cm. Then, the front-back relationship of the five faces is specified, with the frontmost faces being the faces 1 and 2, the next being located in the back being the faces 3 and 4, and the backmost being the face 5.

Then, the image information processing unit 22 specifies the type of surface by using the context of the surface specified in this way. That is, the depth information is the same as the front surface of the shelf, the product tag surface, the back surface of the shelf, and the front surface of the shelf on the narrow horizontal plane on the depth map. Alternatively, a vertical rectangle that is close (within a certain range, for example, within a few centimeters) and is hollow is specified as the surface of the product tag. Then, preferably, the other side is specified as a product side (a side on which the product may be displayed). Therefore, in the example of FIG. 26, the surfaces 1 and 2 are specified as the front surface and the product tag surface of the shelf board, the surfaces 3 and 4 are specified as the product surface, and the surface 5 is the rear plate surface of the shelf body.

On the product side specified in this way, the corresponding process is executed based on the package type of the product for which input is accepted at any timing. Package types include cans, bottles (including PET bottles; the same applies hereinafter), coexistence of cans and bottles, boxes, hanging products, etc., but are not limited to these.

If the package type is a can or bottle, the area of the cylinder within a certain size range is specified as the face area of the can or bottle. This is schematically shown in FIGS. 27 and 28. FIG. 27 is an image diagram showing the case where the package type is a can, and FIG. 28 is an image diagram showing the case where the package type is a bottle.

When the package type is a coexistence of a can and a bottle, there is a cylinder within a certain size range, an area without a small cylinder above is the face area of the can, and there is a cylinder within a certain size range. The area with a small cylinder above is specified as the face area of the bottle.

If the package type is a box, the area of the vertical rectangular plane within a certain size range is specified as the face area of the box. This is schematically shown in FIG.

If the package type is a hanging product, the area of a rectangular flat surface or uneven surface that is within a certain size range and floats in the air is specified as the face area of the hanging product. This is schematically shown in FIG.

The image information processing unit 22 may specify the face area without accepting the selection of the package type. In this case, it is determined whether or not the condition as the face area is satisfied by executing the determination process for each of the product surface area and the package type. Then, if there is a surface that constitutes the face area, the face area that the surface constitutes is specified, and what kind of package type the face area is based on the predetermined package type priority condition. To identify.

The process of specifying the face region from the three-dimensional model can adopt any method depending on the category and shape of the product, and is not limited to the above. Further, the correction input by the operator may be accepted for the automatically specified face area. Further, the input of the position of the face area may be accepted from the operator.

Further, the image information processing unit 22 may specify the face region by using deep learning. In this case, the above three-dimensional model is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the face region is set based on the output value. It may be specified. Further, as a learning model, various three-dimensional models with face regions can be used as correct answer data.

The image information processing unit 22 cuts out the face region specified as described above. Cutting out the face area may actually cut out the face area specified from the three-dimensional model, and also includes setting the area as the processing target of the processing described later.

When cutting out the face area, the face area specified by each of the above processes may be cut out as it is, or the face area may be specified by a plurality of methods and the target to be cut out using the result of the face area specified by each method. The face area to be used may be specified. For example, a method of specifying the position and range of a surface that satisfies a predetermined condition from a three-dimensional model and matching whether the condition is satisfied according to the type of the surface model to specify the face region, and a method of specifying the face region by deep learning. The face area to be finally cut out may be specified by a method of specifying and a predetermined calculation. The method for specifying the face area to be cut out is not limited to the above, and can be set arbitrarily.

The image information processing unit 22 executes the same plane expansion processing as in the third embodiment on the face region cut out by the image information processing unit 22, and generates plane expansion image information for the face region. Then, if the plane-expanded image information generated in this way is treated as the captured image information in the above-mentioned Examples 1 and 2 and the processing after the block identification processing is executed, the same as in the first and second embodiments. The process can be executed.

Further, the sample information processing unit 25 generates a three-dimensional model from the image information obtained by photographing the object and the depth map, and cuts out the face region based on the three-dimensional model, similarly to the image information processing unit 22 described above. .. Then, the plane development image information is generated from the cut out face area, and the plane development image information is stored in the sample information storage unit 26 as the sample information, so that the same processing as in the first and second embodiments can be executed.

Each process in each of the above-described examples is not limited to the order described in the specification of the present invention, and can be appropriately changed as long as the object is achieved.

Further, the image processing system 1 of the present invention is effective when specifying the display status of a product displayed on the display shelf from the photographed image information obtained by photographing the display shelf of the store. It is not limited. That is, when a certain object to be photographed is photographed, it can be widely used when specifying the area in which the desired object is captured from the photographed image information.

By using the image processing system 1 of the present invention, the accuracy of comparison processing can be improved without requiring more calculation time than before when identifying an object such as a product shown in image information obtained by photographing a display shelf or the like. It is possible to improve.

1: Image processing system 2: Management terminal 3: Input terminal 20: Image information input reception processing unit 21: Image information storage unit 22: Image information processing unit 23: Candidate processing unit 24: Comparison processing unit 25: Sample information processing unit 26 : Specimen information storage unit 70: Arithmetic device 71: Storage device 72: Display device 73: Input device 74: Communication device

The first invention is an image processing system for identifying the product that is reflected in the photographed image information display shelf, the image processing system is an area satisfying a predetermined condition in the face region in the image information block by using the layout information, and layout information of the block in the item to the sampling of the a judgment of similarity of layout between blocks, and the candidate processor for identifying a product to be a candidate of the comparison process, the image information a face region of the, by the comparison processing using the sample information items identified as candidates by the candidate processor has a comparison unit for identifying the product that is reflected in the face region , The candidate processing unit calculates a predetermined evaluation value using the area of the common part and / or the difference between the block layout information in the face area of the image information and the block layout information of the sample product. This is an image processing system that specifies the sampled product as a candidate when the calculated evaluation value is equal to or higher than the threshold value .

Further, in order to narrow down the objects to be candidates for the comparison processing, it is preferable to execute the processing as in the present invention. Moreover, since the layout information is not image information but coordinate information, high speed can be realized even if these processes are executed.

In the invention described above, the candidate processing unit, the identifies blocks that satisfy a predetermined condition in the face region in the image information, and the block layout information in a predetermined range of the specified face area, the image of the product to the specimen It can be configured like an image processing system that determines the similarity of layouts by using the layout information of the blocks of the area corresponding to the size of the predetermined range of the face area in the information.

In the above-described invention, the candidate processing unit performs the process of determining the similarity of the layout while shifting the area corresponding to the size of the predetermined range of the face area in the image information of the product as the sample. , The comparison processing unit performs comparison processing using the sample information of the region determined to have the highest similarity in the candidate processing unit and the face region of the image information in the image information of the product . It can be configured like an image processing system that identifies the products shown in the face area .

In the above invention, the comparison processing unit, said the block of image information, correspondence and a block of the specified item as a candidate in the candidate processing unit, by using the correspondence block, image information and / or character By determining the similarity of information, it can be configured like an image processing system that executes the comparison processing.

A fifth invention is an image processing system that identifies a product reflected in image information obtained by photographing a display shelf, and the image processing system obtains image information obtained by photographing the display shelf and corresponding depth information. Image information processing that generates plane-expanded image information that expands to a plane for part or all of the three-dimensional model generated in use, and uses part or all of the generated plane-expanded image information as image information to be processed. parts and the layout information of the block is an area satisfying a predetermined condition in the face region in the image information of the processing target, by using the block layout information in the object to be sampled, the layout of the similarity of the blocks to each other Comparison processing is performed using a candidate processing unit that identifies a product that is a candidate for comparison processing, a face area of image information to be processed, and sample information of a product that is specified as a candidate by the candidate processing unit. This is an image processing system having a comparison processing unit for identifying a product appearing in the face area .

The first invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer uses the layout information of the block , which is an area satisfying a predetermined condition in the face area in the image information obtained by photographing the display shelf, and the layout information of the block in the sample product , to layout the blocks . a determination is similarity, candidate processing unit for identifying the product to be a candidate of the comparison process, the comparison process using the face region of the image information, and sample information items identified as candidates by the candidate processing unit This is an image processing program that functions as a comparison processing unit that identifies the products in the face area , and the candidate processing unit uses the layout information of the blocks in the face area of the image information to be processed and the layout information of the blocks. A predetermined evaluation value is calculated using the area of common part and / or difference with the layout information of the block of the sampled product, and if the calculated evaluation value is equal to or greater than the threshold value, the sampled product is specified as a candidate. This is an image processing program.

The fifth invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer generates plane-expanded image information that expands to a plane for a part or all of the three-dimensional model generated by using the image information obtained by photographing the display shelf and the corresponding depth information, and the generated plane. An image information processing unit that uses part or all of the developed image information as image information to be processed, layout information of a block that satisfies a predetermined condition in the face area of the image information to be processed, and a target to be sampled. The candidate processing unit that determines the layout similarity between the blocks using the block layout information of the object and identifies the product as a candidate for the comparison processing, the face area of the image information to be processed, and the above. This is an image processing program that functions as a comparison processing unit that identifies products in the face area by performing comparison processing using sample information of products specified as candidates by the candidate processing unit.

Claims (9)

An image processing system that identifies an object reflected in image information.
The image processing system is
Using the layout information of the second area that satisfies the predetermined condition in the first area of the image information to be processed and the layout information of the block in the sample object, the similarity of the layout is judged and compared. A candidate processing unit that identifies an object to be processed,
By performing comparison processing using the first region or the second region of the image information of the processing target and the sample information or the block of the object specified as a candidate by the candidate processing unit, the first region is performed. A comparison processing unit that identifies the object shown in
An image processing system characterized by having. The candidate processing unit
The area of the intersection and / or difference between the layout information of the second area and the layout information of the block of the object to be sampled is calculated, and a predetermined evaluation value is calculated using the calculation result.
If the calculated evaluation value is equal to or greater than the threshold value, the object is added to the candidate list.
The image processing system according to claim 1. The candidate processing unit
In the first area of the image information to be processed, a second area satisfying a predetermined condition is specified, and a second area is specified.
Layout information of the second region in the predetermined range of the specified first region and layout information of the block of the region corresponding to the size of the predetermined range of the first region in the image information of the object to be sampled. To determine the similarity of layouts using
The image processing system according to claim 1 or 2, wherein the image processing system is characterized in that. The candidate processing unit
The process of determining the similarity of the layout is performed while shifting the area corresponding to the size of the predetermined range of the first area in the image information of the object to be the sample.
The comparison processing unit
In the image information of the object, the sample information or block of the region determined to have the highest similarity by the candidate processing unit is compared with the first region or the second region of the image information of the processing target. By processing, the object reflected in the first area is identified.
The image processing system according to claim 3, wherein the image processing system is characterized by the above. The comparison processing unit
The second region of the image information to be processed is associated with the block of the object specified as a candidate by the candidate processing unit.
The comparison process is executed by determining the similarity of the image information and / or the character information using the associated second region and the block.
The image processing system according to any one of claims 1 to 4, wherein the image processing system is characterized in that. The image processing system is
For a part or all of the three-dimensional model generated by using the image information and the corresponding depth information, the plane-expanded image information developed on a plane is generated, and a part or all of the generated plane-expanded image information is displayed. Image information processing unit, which is the image information to be processed.
The image processing system according to any one of claims 1 to 5, wherein the image processing system has. An image processing system that identifies an object reflected in image information.
The image processing system is
Using the layout information of the block that satisfies the predetermined condition in the face area in the image information of the processing target and the layout information of the block of the target object to be sampled, the similarity of the layout is judged and used as a candidate for comparison processing. Candidate processing unit that identifies the object and
The image information and / or feature amount of the face area or block area in the image information of the processing target, the sample information of the object specified as a candidate by the candidate processing unit, the image information and / or feature amount of the block area, and the like. A comparison processing unit that identifies an object in the face area by performing comparison processing using
An image processing system characterized by having. Computer,
Using the layout information of the second area that satisfies the predetermined condition in the first area of the image information to be processed and the layout information of the block in the sample object, the similarity of the layout is judged and compared. Candidate processing unit that identifies the target object to be processed,
By performing comparison processing using the first region or the second region of the image information of the processing target and the sample information or the block of the object specified as a candidate by the candidate processing unit, the first region is performed. Comparison processing unit that identifies the object shown in
An image processing program characterized by functioning as. Computer,
Using the layout information of the block that satisfies the predetermined condition in the face area in the image information of the processing target and the layout information of the block of the target object to be sampled, the similarity of the layout is judged and used as a candidate for comparison processing. Candidate processing unit that identifies the object,
The image information and / or feature amount of the face area or block area in the image information of the processing target, the sample information of the object specified as a candidate by the candidate processing unit, the image information and / or feature amount of the block area, and the like. A comparison processing unit that identifies an object in the face area by performing comparison processing using
An image processing program characterized by functioning as.