JP6788877B2 - Image processing system - Google Patents

Description

The present invention relates to an image processing system.

It may be required to determine whether or not an object to be determined is reflected in a certain image information. For example, there is a case where it is determined whether or not the product to be determined is displayed in the image information obtained by photographing the display shelf of the product.

In this way, when determining whether or not an object is reflected in the image information, it is common to execute an image matching process between the image information to be processed and the image information of the object. For example, Patent Document 1 below discloses a system for grasping products handled by vending machines by using image recognition technology for image information taken by vending machines based on registered images for each product. Has been done.

Japanese Unexamined Patent Publication No. 2014-191423

The specific processing of Patent Document 1 is to photograph a vending machine from a plurality of directions, align the positional relationships of the photographed images, and then superimpose the photographed images to generate a composite image. Then, the products handled by the vending machine are specified by collating the generated composite image with the registered image representing the products that may be displayed on the vending machine.

In the collation process (matching process) between the composite image and the registered image representing the product, it is common to execute the process by using each feature amount. However, when the number of products to be compared increases, the number of registered images increases, and the accuracy of the matching process (matching process) cannot be improved. In addition, there is a problem that the calculation time is enormous.

Therefore, in view of the above problems, the present inventor improves the accuracy of the matching process without requiring more calculation time than before when identifying an object such as a product shown in image information obtained by photographing a display shelf or the like. Invented an image processing system.

The first invention is an image processing system that identifies an object reflected in image information, and the image processing system is a second region that satisfies a predetermined condition in a first region of the image information to be processed. The similarity search process is executed using the information of the above and the block information stored in the sample information storage unit that stores the sample information and the block information of the object to be sampled, and the target object to be the candidate for the comparison process is specified. By performing comparison processing using the candidate processing unit, the first region of the image information of the processing target, and the sample information of the object specified as a candidate by the candidate processing unit, the first region can be obtained. It has a comparison processing unit that identifies an object in the image, and the candidate processing unit reduces the number of colors of the image information in the first region to a predetermined number in the image information in the first region. Then, the image information of the first region is divided into meshes of a predetermined size to generate a distribution map for each of the reduced colors, and for the distribution map for each color, a group that falls within a predetermined range is detected. This is an image processing system that specifies, as the second region, an region surrounding a group in which the number of colors contained in the region is within a predetermined number and the edges are equal to or greater than a predetermined threshold for each group .

The second invention is an image processing system that identifies an object reflected in image information, and the image processing system is a second region that satisfies a predetermined condition in a first region of the image information to be processed. The similarity search process is executed using the information of the above and the block information stored in the sample information storage unit that stores the sample information and the block information of the object to be sampled, and the object to be a candidate for the comparison process is specified. By performing comparison processing using the candidate processing unit, the first region of the image information of the processing target, and the sample information of the object specified as a candidate by the candidate processing unit, the first region can be obtained. It has a comparison processing unit that identifies the object in the image, and the candidate processing unit extracts the feature amount from the image information in the first region to cover a range of a predetermined size or more. The area where the common features are distributed is detected, the area is erased, the area where the common feature amount is distributed is specified for the remaining area, and the area surrounding the area is the second area. It is an image processing system specified as .

A third invention is an image processing system that identifies an object reflected in image information, and the image processing system is a second region that satisfies a predetermined condition in a first region of the image information to be processed. The similarity search process is executed using the information of the above and the block information stored in the sample information storage unit that stores the sample information and the block information of the object to be sampled, and the object to be the candidate for the comparison process is specified. By performing comparison processing using the candidate processing unit, the first region of the image information of the processing target, and the sample information of the object specified as a candidate by the candidate processing unit, the first region can be obtained. It has a comparison processing unit that identifies the object in the picture, and the candidate processing unit has a large number of intermediate layers in which the second region is given as correct answer data to the image information of various first regions. The image information of the first region in the image information to be processed is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of the above layers is optimized, and based on the output value, the image information is input. This is an image processing system that specifies the second region .

A fourth invention is an image processing system that identifies an object reflected in image information, and the image processing system is a second region that satisfies a predetermined condition in a first region of the image information to be processed. The similarity search process is executed using the information of the above and the block information stored in the sample information storage unit that stores the sample information and the block information of the object to be sampled, and the object to be a candidate for the comparison process is specified. By performing comparison processing using the candidate processing unit, the first region of the image information of the processing target, and the sample information of the object specified as a candidate by the candidate processing unit, the first region can be obtained. It has a comparison processing unit that identifies the object in the image, and the candidate processing unit positively corrects the feature amount in the second region in the image information of various first regions as a learning model in advance. As an example, the image information of the first region in the image information to be processed is divided into the determination processing unit in which the feature amount outside the second region in the image information of various first regions is given as a negative example. Each point in the mesh is input, and the determination processing unit is made to determine whether each point belongs to the second region, and in the determination result, the mesh determined to belong to the second region. This is an image processing system that specifies the area surrounding the area as the second area .

By configuring as in these inventions, it is possible to reduce the objects of the sample to be compared in the comparison process (matching process), so that the calculation time can be reduced as compared with the conventional case. Further, by executing the processes described in these inventions, a region having information capable of identifying an object can be specified as a second region. Then, the target objects can be appropriately narrowed down.

A fifth invention is an image processing system that identifies an object reflected in image information, and the image processing system is a second region that satisfies a predetermined condition in a first region of the image information to be processed. The similarity search process is executed using the information of the above and the block information stored in the sample information storage unit that stores the sample information and the block information of the object to be sampled, and the object to be a candidate for the comparison process is specified. By performing comparison processing using the candidate processing unit, the first region of the image information of the processing target, and the sample information of the object specified as a candidate by the candidate processing unit, the first region can be obtained. It has a comparison processing unit that identifies the object in the image, and the candidate processing unit has a size of each in the similarity search processing between the information in the second region and the block information of the object. The similarity search process is executed as it is for any one or more of the scale, the scale, and the rotation, and the comparison processing unit performs the comparison processing between the information in the first region and the sample information of the object specified as the candidate. ， It is an image processing system that executes comparison processing by aligning any one or more of its size, scale, and rotation .

By configuring as in the present invention, it is possible to reduce the number of objects to be compared in the sample in the comparison process (matching process), so that the calculation time can be reduced as compared with the conventional case.
In the similarity search processing between the information in the second area of the image information to be processed and the block information of the target object, the number of target objects is large and the purpose of the similar search process is to narrow down the candidates. Therefore, it is not necessary to perform strict matching processing with an emphasis on the calculation time. On the other hand, in the comparison process between the information in the first area and the sample information of the object specified as a candidate, the number of sample information of the object to be compared is small, and the information in the first area is reflected. Since it is necessary to identify the target object, it is necessary to perform a stricter matching process than the above-mentioned similar search process. Therefore, it is preferable to configure it as in the present invention.

A sixth invention is an image processing system that identifies an object reflected in image information, and the image processing system is a part of a three-dimensional model generated by using the image information and the corresponding depth information. Alternatively, in an image information processing unit that generates plane-expanded image information that expands on a plane and uses a part or all of the generated plane-expanded image information as image information to be processed, and the image information to be processed. Similar search processing using the information of the second area that satisfies the predetermined condition in the first area and the block information stored in the sample information storage unit that stores the sample information and the block information of the object to be sampled. Is executed, and the candidate processing unit that identifies the target object to be the candidate for the comparison processing, the first area of the image information of the processing target, and the sample information of the object specified as the candidate by the candidate processing unit are used. This is an image processing system having a comparison processing unit for identifying an object in the first region by performing comparison processing.

By configuring as in the present invention, it is possible to reduce the number of objects to be compared in the sample in the comparison process (matching process), so that the calculation time can be reduced as compared with the conventional case.
The image information to be processed may be image information in which image information obtained by photographing a display shelf or the like is handled as it is in two dimensions, or image information in which a three-dimensional model is once generated and the image information is expanded in a plane. As a result, the processing can be executed more accurately.

In the above-described invention, the candidate processing unit can be configured like an image processing system that specifies a region having information capable of identifying an object as a second region in the specified first region. ..

It is preferable to specify the region having the characteristics of the present invention as the second region. As a result, the target objects can be appropriately narrowed down.

The first invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer is stored in the sample information storage unit that stores the information in the second area that satisfies the predetermined condition in the first area in the image information to be processed, and the sample information and the block information of the object to be sampled. A candidate processing unit that executes a similar search process using the block information of the target object and identifies an object as a candidate for comparison processing, a first area of image information of the processing target, and the candidate processing unit. An image processing program that functions as a comparison processing unit that identifies an object in the first region by performing comparison processing using the sample information of the object specified as a candidate in the above. In the image information of the first region, the candidate processing unit reduces the color of the image information of the first region to a predetermined number of colors, and divides the image information of the first region into meshes of a predetermined size. The distribution map for each of the reduced colors is generated, the groups that fall within a predetermined range are detected in the distribution map for each color, and the number of colors included in the area is within the predetermined number for each group. This is an image processing program that specifies a region surrounding a group whose edges are equal to or greater than a predetermined threshold as the second region .
The second invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer is stored in the sample information storage unit that stores the information in the second area that satisfies the predetermined condition in the first area in the image information to be processed, and the sample information and the block information of the object to be sampled. A candidate processing unit that executes a similar search process using the block information of the processed object and identifies an object as a candidate for comparison processing, a first area of image information of the processing target, and the candidate processing unit. An image processing program that functions as a comparison processing unit that identifies an object in the first region by performing comparison processing using the sample information of the object specified as a candidate in the above. By extracting the feature amount from the image information of the first region, the candidate processing unit detects a region in which common features are distributed over a range of a predetermined size or more, erases the region, and remains. This is an image processing program that specifies a region in which a common feature amount is distributed, and specifies a region surrounding the region as the second region .
The third invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer is stored in the sample information storage unit that stores the information in the second area that satisfies the predetermined condition in the first area in the image information to be processed, and the sample information and the block information of the object to be sampled. A candidate processing unit that executes a similar search process using the block information of the target object and identifies an object as a candidate for comparison processing, a first area of the image information of the processing target, and the candidate processing unit. An image processing program that functions as a comparison processing unit that identifies an object appearing in the first region by performing comparison processing using sample information of the object specified as a candidate in the above. The candidate processing unit gives the image information of various first regions the second region as correct answer data, and the learning in which the weighting coefficient between the neurons of each layer of the neural network consisting of many intermediate layers is optimized. This is an image processing program that inputs the image information of the first region in the image information to be processed to the model and specifies the second region based on the output value .
The fourth invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer is stored in the sample information storage unit that stores the information in the second area that satisfies the predetermined condition in the first area in the image information to be processed, and the sample information and the block information of the object to be sampled. A candidate processing unit that executes a similar search process using the block information of the target object and identifies an object as a candidate for comparison processing, a first area of image information of the processing target, and the candidate processing unit. An image processing program that functions as a comparison processing unit that identifies an object in the first region by performing comparison processing using the sample information of the object specified as a candidate in the above. As a learning model in advance, the candidate processing unit takes the feature amount in the second region in the image information of various first regions as a positive example, and features outside the second region in the image information of various first regions. Each point in the mesh obtained by dividing the image information of the first region in the image information to be processed is input to the determination processing unit in which the amount is given as a negative example, and each point is set in the second region. This is an image processing program that causes the determination processing unit to determine whether or not it belongs, and in the determination result, specifies a region surrounding a mesh that is determined to belong to the second region as the second region .

The fifth invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer is stored in the sample information storage unit that stores the information in the second area that satisfies the predetermined condition in the first area in the image information to be processed, and the sample information and the block information of the object to be sampled. A candidate processing unit that executes a similar search process using the block information of the target object and identifies an object as a candidate for comparison processing, a first area of image information of the processing target, and the candidate processing unit. An image processing program that functions as a comparison processing unit that identifies an object in the first region by performing comparison processing using the sample information of the object specified as a candidate in the above. In the similarity search processing between the information in the second area and the block information of the object, the candidate processing unit executes the similarity search processing as it is for any one or more of the respective sizes, scales, and rotations. In the comparison processing between the information in the first region and the sample information of the object specified as the candidate, the comparison processing unit performs the comparison processing by aligning any one or more of the size, scale, and rotation. It is an image processing program to be executed.
The sixth invention can be realized by loading the program of the present invention into a computer and executing it. That is, the computer generates plane-expanded image information that expands on a plane for a part or all of the three-dimensional model generated by using the image information and the corresponding depth information, and one of the generated plane-expanded image information. The image information processing unit whose part or all is the image information to be processed, the information of the second area which satisfies the predetermined condition in the first area of the image information to be processed, and the sample information of the object to be sampled. A candidate processing unit that executes a similarity search process using the block information of the object stored in the sample information storage unit that stores the information and the block information, and identifies the target object to be the candidate for the comparison processing, the processing target Comparison processing to identify the object in the first area by performing comparison processing using the first area of the image information and the sample information of the object specified as a candidate by the candidate processing unit. It is an image processing program that functions as a part.

By using the image processing system of the present invention, the accuracy of matching processing is improved without requiring more calculation time than before when identifying an object such as a product shown in image information obtained by photographing a display shelf or the like. It becomes possible to make it.

It is a block diagram which shows typically an example of the whole processing function of the image processing system of this invention. It is a block diagram which shows typically an example of the hardware composition of the computer used in the image processing system of this invention. It is a flowchart which shows an example of the processing process of the whole processing in the image processing system of this invention. It is a figure which shows an example of photographed image information. It is a figure which shows an example of the image information which performed the orthotopic processing on the photographed image information of FIG. It is a figure which shows an example of the state which specified the face area from the photographed image information. It is a figure which shows an example of the state which specified the block from a face area. The conceptual diagram of the processing in the image processing system of this invention is shown. It is a flowchart which shows an example of the processing process of the process of specifying a block. It is a figure which shows typically the correspondence relationship between the photographed image information and the depth map by depth information. It is a figure which shows an example of the process of 3D modeling based on photographed image information and a depth map. It is a figure which shows an example of the plane development process typically. It is a figure which shows typically the stitching process which is executed after 3D modeling. It is a figure which shows typically the stitching process which is executed after 3D modeling. It is a figure which shows typically the process of specifying a reference in the viewpoint direction determination process. It is a figure which shows typically an example of the process of labeling each surface. It is an image diagram which identifies the face when the package type is a can. It is an image diagram which identifies the face when the package type is a bottle. It is an image diagram which identifies the face when the package type is a box. It is an image diagram which identifies the face when the package type is a hanging product.

FIG. 1 shows a block diagram of an example of the overall processing function of the image processing system 1 of the present invention. The image processing system 1 uses a management terminal 2 and an input terminal 3.

The management terminal 2 is a computer that realizes the core processing function of the image processing system 1. Further, the input terminal 3 is a terminal for acquiring image information obtained by photographing a display shelf of a store or the like. In addition, an object such as a product used as a sample used in the comparison process (matching process) described later may be photographed and acquired.

The management terminal 2 and the input terminal 3 in the image processing system 1 are realized by using a computer. FIG. 2 schematically shows an example of the hardware configuration of the computer. The computer can input information from an arithmetic unit 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, and a display device 72 such as a display that displays information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits and receives processing results of the arithmetic unit 70 and information stored in the storage device 71 via a network such as the Internet or LAN.

When the computer includes a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used, for example, in portable communication terminals such as tablet computers and smartphones, but are not limited thereto.

The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be performed directly on the display with a predetermined input device (such as a pen for a touch panel) or a finger.

The input terminal 3 may include a photographing device such as a camera in addition to the above-mentioned devices. As the input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used. The input terminal 3 captures image information (photographed image information or object image information described later) by visible light or the like with a photographing device.

Each means in the present invention has only a logical distinction in its function, and may form the same area physically or substantially. The processing order of the processing in each means of the present invention may be changed as appropriate. In addition, a part of the processing may be omitted. For example, the process of determining the viewpoint direction, which will be described later, can be omitted. In that case, it is possible to execute processing on the image information that has not been processed to determine the viewpoint direction. Further, a part or all of the functions in the management terminal 2 may be executed in the input terminal 3.

The image processing system 1 includes an image information input reception processing unit 20, an image information storage unit 21, an image information processing unit 22, a candidate processing unit 23, a comparison processing unit 24, a sample information processing unit 25, and a sample information storage unit 26. ..

The image information input reception processing unit 20 receives input of image information (captured image information) photographed by the input terminal 3 and stores it in the image information storage unit 21 described later. For example, the input of the photographed image information of the display shelf of the store is accepted and stored in the image information storage unit 21. In addition to the captured image information, the input terminal 3 may accept input including information indicating the shooting date and time and the shooting target, for example, store identification information such as a store name, and image information identification information for identifying the image information. FIG. 4 shows an example of captured image information. FIG. 4 schematically shows photographed image information obtained by photographing a state in which products are displayed on a display shelf having three shelves.

The image information storage unit 21 stores the captured image information received from the input terminal 3, the shooting date and time, the image information identification information, and the like in association with each other.

The image information processing unit 22 sets the captured image information received by the image information input reception processing unit 20 in a state of facing each other, and compares the captured image information with the sample information (matching processing). The face processing for specifying the area (face area) to execute the above, and the block identification processing for specifying the feature area (block) satisfying a predetermined condition in the face area are executed.

In general, when the object to be photographed is simply photographed, it is difficult to take the image in the state of facing the object to be photographed, so the erecting process is corrected to the state of facing the object to be photographed. This is a process of transforming the image information so that the optical axis of the lens of the photographing device is the same as when the image is photographed from a sufficient distance along the perpendicular direction of the plane to be photographed. An example of such correction processing is trapezoidal correction processing. If the image information is distorted, distortion correction processing may be added. FIG. 5 shows the image information in a state in which the emplacement processing is executed on the captured image information of FIG.

If the captured image information is the image information captured in the face-to-face state, or if there is no distortion, it is not necessary to execute the emplacement processing and the distortion correction processing.

The captured image information may be any image information to be processed according to the present invention. The captured image information also includes the image information after the correction processing for the captured image information such as the emplacement processing is executed.

Further, when the image information processing unit 22 shoots a shooting target, when a plurality of images are shot, the image information processing unit 22 executes a process of synthesizing the images into one image information, and the image information obtained by the synthesizing process is positive. Placement processing, face processing, and block identification processing may be executed. A known method can also be used as a process of synthesizing a plurality of image information into any image information.

The face processing specifies an area (face area) for executing comparison processing with sample information described later in the captured image information. For example, when the sample information is a product in the photographed image information obtained by photographing the product display shelf, the area of the product displayed on the display shelf or the area of the product label is specified as the face area. When the product is a PET bottle beverage, the area of the product label is the face area, and when the product is a boxed product (for example, sweets), the entire product package is the face area. The face area can be set as appropriate. Even when the product is a PET bottle beverage, the product area may be the face area.

There are various methods for specifying the face area, which can be arbitrarily set according to the characteristics of the sample information. When the sample information is a product (for example, a PET bottle beverage) and the area of the product label is specified as the face area from the photographed image information of the display shelf, for example, between the shelves of the display shelf. For restrictions such as identifying narrow and narrow shadows that occur between products in the area (shelf area), identifying repeating patterns of images, identifying steps on the upper side of the package, and having the same product width. The area of the product is specified by specifying the division position based on the above. Then, from the product area, a predetermined rectangular area is specified as a label area, and that area is specified as a face area.

As the method for specifying the face area, any method can be adopted depending on the product category and the product form, and the method is not limited to the above. In addition, the person in charge may accept correction input for the automatically specified face area. Further, the person in charge may input the position of the face area.

Further, the image information processing unit 22 may specify the face region by using deep learning. In this case, the above captured image information is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the face region is set based on the output value. It may be specified. Further, as the learning model, it is possible to use various captured image information in which the face region is given as correct answer data.

The image information processing unit 22 cuts out the face region specified as described above. Cutting out the face area may actually cut out the face area specified from the captured image information, and also includes setting the area as the processing target of the processing described later.

When cutting out the face area, the face area specified by each of the above processes may be cut out as it is, or the face area may be specified by a plurality of methods and the target to be cut out using the result of the face area specified by each method. The face area to be used may be specified. For example, the area of a product is specified by specifying a narrow and narrow shadow generated between the products in the area between the shelves of the display shelf (shelf area), and a predetermined rectangular area is specified from that area. A method of specifying the face area and a method of specifying the face area by deep learning may be performed, and the face area to be finally cut out may be specified by a predetermined calculation. The method for specifying the face area to be cut out is not limited to the above, and can be set arbitrarily.

FIG. 6 shows an example of a case where a product (label portion of a PET bottle beverage) is cut out as a face area from the photographed image information.

When the image information processing unit 22 specifies the face area, the image information processing unit 22 sets a rectangular area having information that can identify the product in the face area as a characteristic area (block) that characterizes the image of the face area. Identify as. The area specified as a block is the area where the product name, product logo, etc. are displayed.

There are, for example, three methods for the image information processing unit 22 to identify a block from the face area, but the method is not limited to these, and a plurality of methods may be combined.

The first method is a method of extracting and determining image features common to blocks of an object. That is, in the image information of the face region, a rectangular region having a feature of being composed of a limited number of colors that can be clearly distinguished from each other and having a clear color-to-color boundary (clear edges) is blocked. It is a method to specify as.

First, the colors of each pixel constituting the image information in the face region are detected by reducing the number of colors to a predetermined number of palette colors. In addition, the image information of the face area is divided into meshes of a predetermined size, and a distribution map for each palette color is generated. Then, in each pallet, a rectangular area is specified by detecting a local group that is distributed at a certain density or higher and whose vertical and horizontal sizes fall within a predetermined threshold value.

Then, for each group, the distribution map of all palette colors is referred to, and it is determined whether the number of palettes contained in the rectangular area is within a predetermined number, for example, 3, and if it is within the predetermined number, the group area. It is determined whether or not there is an edge inside a predetermined threshold value or more. For edge determination, edge detection processing is performed on the image information in the face area, and it is possible to determine whether or not the image information is equal to or greater than the threshold value. Then, the rectangular area surrounding the group that satisfies these conditions is specified as a block.

The block can be identified from the face area by the above method.

As a second method for identifying a block from the face area, a feature amount such as a local feature amount is extracted from the image information of the face area. Then, a region in which common features are distributed over a wide range (a range of a predetermined size or more) is detected, and that region is erased as the "ground" part of the face region. Then, for the remaining areas, the areas where the features common to each other are distributed are specified, and the rectangular area surrounding them is specified as a block.

As a third method for identifying a block from the face region, the block can be detected from the image information of the face region by AI technology such as machine learning or deep learning. In this case, the image information of the face region is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the block is created based on the output value. It may be specified. As the learning model, it is possible to use image information of various face regions in which blocks are given as correct answer data. Note that the face area and the block may be specified at the same time from the captured image information.

In addition, as another example of using AI technology such as machine learning or deep learning in the third method, there are the following. In advance, the feature amount in the block area (the feature amount may be a plurality of colors, local feature amounts, etc.) in the correct answer data (a block is given to the image information of various face areas) given as the above-mentioned learning model, and the block. A judgment module (judgment machine) such as SVM (support vector machine) is constructed by extracting the features outside the region and using each as a positive example and a negative example. Then, for this judgment module such as SVM, each point of the image information in the face area, for example, the image information in the face area is divided into a mesh of a predetermined size, points in the mesh are input, and each point is blocked. It is made to judge whether or not it belongs to the area of. In this determination result, the area surrounding the mesh area determined to belong to the block area is specified as a block.

The process of identifying the block from the face area is not limited to the above three methods, and other methods may be used. Further, a plurality of the above three methods or a plurality of other methods may be combined and specified.

FIG. 7 schematically shows an example when a block is specified from the face area. FIG. 7 shows a case where the block is specified from the face area of FIG. The rectangular area in FIG. 7 is the specified block.

When the image information processing unit 22 specifies a block from the image information in the face area as described above, the candidate processing unit 23 searches for a similarity between the specified block and the block stored in the sample information storage unit 26 described later. Execute the process. As a similar search, for example, the feature amount of the image information of the block specified from the image information of the face area is compared with the feature amount of the image information of the block for each object stored in the sample information storage unit 26, and a determination is made. .. In addition to the local features, the features include the degree of matching of the color scheme of each color, the degree of matching of the edge distribution, and a combination of these. When the number of blocks specified from the image information of the face area is m and the number of blocks per object is n, the number of comparisons between them is m × n in a simple comparison, but LSH (Locality Sensitive) When a high-speed method such as Hashing) is used, the number of comparisons can be reduced and high-speed can be realized.

The candidate processing unit 23 sorts the results of the above-mentioned similar search according to a predetermined criterion, and identifies the upper predetermined number, for example, the upper 10 sample information as a candidate.

The comparison processing unit 24 executes a comparison processing (matching processing) between the sample information as a candidate specified by the candidate processing unit 23 and the image information of the face region. For example, by extracting the feature amount of the image information in the face area and executing the comparison processing between the extracted feature amount and the feature amount of the sample information as a candidate, the similarity between the image information in the face area and the sample information can be obtained. Judgment is made, and the identification information of the object included in the face area, for example, the product name is judged.

As the comparison process, not only the entire image information in the face area is matched with the sample information, but also a part of the image information in the face area, for example, the feature amount of the image information in the feature area (block) is extracted and sampled. Information feature quantities and comparison processing may be executed.

In addition, the comparison process may be performed by a method other than the feature amount.

Further, the comparison processing unit 24 may execute the comparison processing with the sample information by using deep learning. In this case, all or part of the image information of the face region is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the output value thereof. Similar sample information may be identified based on. As the learning model, it is possible to use image information of various face regions in which sample information is given as correct answer data.

The sample information processing unit 25 generates sample information of the object to be used in the comparison process for identifying the object reflected in the captured image information, and stores it in the sample information storage unit 26 described later. The sample information is image information (object image information) obtained by photographing an object and / or information on a feature amount thereof. In the sample information, identification information of the object, for example, a product code such as a product name, a model number, and a JAN code, image information of a block in the image information of the object, and its feature amount (block information) are stored. The image information of the object may be the image information of the entire object, or may be the image information of a part of the object, for example, the label portion when the object is a PET bottle beverage. When the feature amount information of the object image information is used as the sample information, it is not necessary to extract the feature amount each time the processing is performed. The block is a characteristic area that characterizes the image information of the object, and corresponds to, for example, a rectangular area that surrounds a product name, a logo, or the like.

The sample information processing unit 25 may generate one sample information for one object, or is in a position where it cannot be captured when shooting a product at a plurality of angles, for example, facing each other for one object. In order to capture the surface, sample information may be generated from angles such as front, back, top, bottom, and both sides. In addition, when one object has a plurality of exteriors (packages, etc.), the sample information and blocks for each exterior may be associated with one object. Moreover, although one block may be associated with the sample information, it is preferable that a plurality of blocks are associated with each other.

The sample information may be extracted from the extracted block by executing the same processing as the above-mentioned image information processing unit 22 on the object image information obtained by photographing the object to be sampled. The operator may select only the blocks necessary for identifying the product. Further, the operator may specify a block from the image information of the read object, instead of automatically executing the extraction process as in the image information processing unit 22.

The sample information storage unit 26 uses the sample information generated by the sample information processing unit 25 as object identification information, such as a product code such as a product name, model number, and JAN code, block image information in the object image information, and / or. It is stored in association with the feature amount (block information).

Next, an example of the processing process using the image processing system 1 of the present invention will be described with reference to the flowchart of FIG. Further, FIG. 8 shows a conceptual diagram of processing in the image processing system 1 of the present invention. Further, in this embodiment, a case where a display shelf of a store is photographed and a product on the display shelf is specified will be described. Therefore, the photographed image information is the display shelf of the store where the product is displayed, and the product may be displayed as the object in the sample information.

First, the operator photographs the product as the sample information from the opposite position, and stores the photographed object image information, identification information, product name, and product code in the sample information storage unit 26 in association with each other. Further, the sample information processing unit 25 executes the same processing as the processing in the image information processing unit 22 on the object image information, and extracts the block. Then, among the extracted blocks, only the blocks used as the sample information, for example, the blocks including the product name, the logo, etc., are stored in the sample information storage unit 26 as the blocks associated with the sample information. In addition, by executing a predetermined process from the object image information, a feature amount such as a local feature amount is extracted and stored in association with the object image information.

Processing for such sample information is executed in advance for a plurality of products.

Then, when the image information input reception processing unit 20 of the management terminal 2 receives the photographed image information obtained by photographing the display shelf of the store from the input terminal 3 (S100), the image information storage unit 21 receives the photographed date and time and the identification information of the photographed image information. And memorize it in association with.

The image information processing unit 22 executes the orthostatic processing for correcting the received captured image information so that the captured image information faces each other (S110). Then, face processing for specifying the face area from the emplaced captured image information is executed (S120), and the block is specified from the specified face area (S130).

As described above, in order to identify the block from the face area, there are various methods such as the first method to the third method as described above, but in the case of the first method, for example, Execute the following processing. An example of the processing process in this case will be described with reference to the flowchart of FIG.

First, with respect to the color of each pixel constituting the image information of the face region, the number of colors is reduced to a predetermined number of palette colors such as 16 colors representing the whole, and the palette color is detected (S200). Further, the short side of the face region is divided by, for example, a square mesh that divides into 64, which palette color is used for each mesh, and a distribution map of the presence or absence of that color is generated for each palette color (S210).

Then, for each mesh of the image information in the face region, expansion processing is performed on the distribution map so as to select a jumping place at a certain distance as a candidate, and a group in which the distribution map falls within a predetermined range is detected ( S220). The predetermined range is, for example, a threshold condition in which the width is 1/10 to 2/3 of the whole, the height is 1/2 to 1/2 of the whole, and the area is 1/100 to 1/10 of the whole. Detects a group of distribution maps (distribution maps that have undergone expansion processing) that fit in.

Then, for each group, the distribution map of all palette colors is referred to, and it is determined whether the number of palettes included in the area is a predetermined number, for example, 3 or less (S230). That is, referring to the distribution map of all palette colors in the group area, if a large number of colors are included in the group area, it is highly likely that it is an image illustration, and it is a characteristic area that identifies the product. Exclude such groups as they are not considered to be certain logos.

In S230, when the number of pallets included in the group area is within a predetermined number, it is determined whether or not the edge is equal to or more than a predetermined threshold value in the group area (S240). Regarding the edge determination, it is possible to perform edge detection processing on the image information in the face area in advance and determine whether or not the image information is equal to or greater than the threshold value. If the edge does not exist above the threshold value in the group area, it is highly possible that it is not a logo, so that group is excluded.

Then, the rectangular area surrounding the group satisfying these conditions is specified as a block (S250).

This process is executed for the group grouped in S220 (S260).

The block can be identified from the face area by the above method.

When the image information processing unit 22 specifies a block from the image information in the face area, the candidate processing unit 23 extracts the feature amount of the image information in the specified block area, and the extracted feature amount and the sample information storage unit 26 The degree of similarity is determined by comparing with the feature amount of the image information of the area of each block of each product stored in. Since the determination of the degree of similarity at this time is to narrow down the candidates, the size of the image information of the area of the specified block and the image information of the area of each block of each product stored in the sample information storage unit 26. The process may be executed without considering the scale, rotation, etc. (it does not have to be as close as possible and remains as it is). Then, the products are sorted in descending order of similarity, and the top predetermined number of products, for example, about 10 products are specified as candidates (S140).

When the candidate processing unit 23 specifies a product that is a candidate for comparison processing in S140, the comparison processing unit 24 extracts the feature amount of the image information in the face region, and the extracted feature amount and the sample of the product specified as the candidate. The comparison process is performed by comparing with the feature amount of the information (the feature amount of the image information of the area corresponding to the face area, not the feature amount of the block) (S150). In this comparison, the size of the image information in the face area and the size, scale, rotation, etc. of the image information in the sample information are made as close as possible before the comparison process is performed. This comparison process is performed for each product specified as a candidate.

Then, as a result of this comparison processing, among the products identified as candidates, the product determined to have the highest degree of similarity is identified as a product in the face area, and the face area, product identification information, product name, product code, etc. (S160).

Since a plurality of products are generally displayed on the display shelf shown in the photographed image information, it is common that a face area corresponding to the displayed products is cut out. Therefore, by executing each of the above-mentioned processes for each face area, it is possible to identify the products displayed on the display shelf and the display position. The display position of the product can be specified by the order of the vertical and horizontal directions of the face area, and can also be specified as the coordinates in the captured image information.

In the first embodiment, the case where the image is photographed with the two-dimensional image information as the photographed image information has been described, but the three-dimensional model is made from the photographed image information in consideration of the distance information between the photographed object and the photographing device. Further, the image information when projected onto the two-dimensional image information from there may be used as the captured image information to be processed by the image processing system 1 of the present invention. In this case, the same processing is executed for the image information of the object in the sample information. The processing in this case will be described below.

In the input terminal 3 of this embodiment, in addition to photographing image information by visible light or the like with a photographing device such as a camera, distance information (depth information) between the imaged object and the photographing device is acquired as two-dimensional information. The information to which the depth information is mapped is referred to as a depth map in this specification. The depth map corresponds to the image information taken by the photographing device, and the depth information up to the object reflected in the photographed image information is mapped to the position corresponding to the photographed image information. In the depth map, at least the range corresponding to the captured image information is divided into meshes, and the depth information is given to each mesh. The vertical and horizontal sizes of the mesh depend on the accuracy of the depth information that can be acquired, but the smaller the vertical and horizontal sizes of the mesh, the higher the accuracy. Usually, it is in the range of 1 mm to several mm, but it may be less than 1 mm or 1 cm or more.

When acquiring depth information, it is necessary to measure the amount of reflected light from each direction and the time until the reflected light arrives by irradiating light with a specific wavelength, for example, infrared rays, when shooting the subject. Acquires distance information (depth information) to the shooting target, or irradiates a light beam of a specific wavelength, for example, an infrared dot pattern, and measures the distance information (depth information) from the reflection pattern to the shooting target. In addition to the above, there is a method of using the parallax of a stereo camera, but the method is not limited to these. If the depth information is given by the distance from the photographing device (one point in three dimensions), the depth information is converted into the depth information from the photographing surface (plane) of the photographing device. .. The input terminal 3 may be provided with a device (depth detection device) for acquiring depth information in this way.

The positional relationship between the image information and the depth information corresponds to each other. FIG. 10 shows the correspondence between the image information taken with the display shelf as the shooting target and the depth map based on the depth information. FIG. 10 (a) is a diagram schematically showing image information taken, and FIG. 10 (b) is a diagram schematically showing a corresponding depth map.

The image information input reception processing unit 20 receives input of image information to be photographed such as a display shelf taken by the input terminal 3 and stores it in the image information storage unit 21. Further, the image information input reception processing unit 20 receives the input of the depth information and the depth map for the image information photographed by the input terminal 3, and stores the image information in the image information storage unit 21 in association with the photographed image information. When the depth information is received from the input terminal 3 instead of the depth map, it is mapped to generate the depth map.

It is preferable that the image information, the depth information, and the depth map of the photographed object are photographed and the information is acquired at the same time when the image information is photographed, and the input of the information is accepted.

The image information processing unit 22 converts the display shelf or the like into a three-dimensional model based on the photographed image information and the depth map. That is, since the captured image information and the depth map correspond to each other, first, the image information captured is converted into a three-dimensional model based on the depth information in the depth map, and the captured image information is associated with the texture mapping. To do. FIG. 11 shows an example of the three-dimensional modeling process based on the captured image information and the depth map. Note that FIG. 11 shows a case where a PET bottle of a beverage is three-dimensionally modeled. FIG. 11A shows the captured image information. Since the image information processing unit 22 acquires the depth map at the same time, it converts it into a three-dimensional model based on the depth information of each mesh in the depth map. Then, by performing texture mapping in which the color information of the corresponding pixel of the captured image information is pasted at the position of each mesh in the depth map, three-dimensional modeling is performed as shown in FIGS. 11 (b) to 11 (d). can do. FIG. 11B shows a viewpoint from the left, FIG. 11C shows a viewpoint from the front, and FIG. 11D shows a viewpoint from the right.

When the image information processing unit 22 generates a three-dimensional model based on the image information obtained by photographing the display shelf or the like and the depth map, the image information processing unit 22 executes a plane expansion process for deploying the three-dimensional model. Planar expansion means expanding a three-dimensional model into a plane (two-dimensional). There are various methods for expanding the three-dimensional model in a plane, and the method is not limited.

As an example of the plane expansion method, there is a method of expanding to a plane by defining the center of projection inside the three-dimensional model and projecting from there. An example of the plane expansion process in the image information processing unit 22 will be described with reference to FIG. FIG. 12A shows image information obtained by photographing a PET bottle, and the top view thereof has a substantially hexagonal shape. The actual shape is finer and uneven, but for the sake of explanation, it is shown as a hexagon. Then, when the image information of FIG. 12 (a) is three-dimensionally modeled using the depth map, the positional relationship when viewed from above is shown in FIG. 12 (b). In this case, since there is no depth information of the part behind the photographing device (camera), there is a blank when viewed from above as shown in FIG. 12B. Then, the central axis of the projection is specified at an arbitrary position of the object to be developed in a plane. For example, if a product is displayed on a display shelf and its shape is not a box shape or a bag shape but a cylindrical shape or a bottle shape and the surface is a regular curved surface instead of a flat surface, its width and its surface From the shape, find the center point of the rotation axis and specify it as the center axis of the projection. Note that the central axis of projection may be specified by any method, not limited to the above. FIG. 12C shows a state in which the central axis of projection is specified. Then, from the central axis of the specified projection, the surface of the object is projected onto an arbitrarily set projection surface. This is schematically shown in FIG. 12 (d). As shown in FIG. 12D, when the positions of the surface of the three-dimensional model and the projection surface do not match, the vertical and horizontal dimensions of the image information of the projection result according to the distance between them are set to an appropriate scale ratio. Adjust by. On the other hand, when the positions of the plane and the projection plane of the three-dimensional model are matched as shown in FIG. 12 (e), it is not necessary to provide the adjustment processing of the scale ratio. Note that FIGS. 12 (b) to 12 (e) are views from above.

The image information processing unit 22 can expand the three-dimensional model into two dimensions (plane) and convert it into two-dimensional image information (plane expansion image information) by performing such a plane expansion process. In the above description, the process of converting the 3D model to the plane-expanded image information is performed by the process of projection, but in the plane-expanded process, the process of converting the 3D model to the plane-expanded image information may be performed by another process. ..

If the image information processing unit 22 handles the plane-developed image information generated in this way as the captured image information in the above-described first embodiment and executes the processing after the face processing, the same processing as in the first embodiment can be executed. ..

Further, the sample information processing unit 25 generates a three-dimensional model from the image information obtained by photographing the object and the depth map, and the plane development image information based on the three-dimensional model, similarly to the image information processing unit 22 described above. To generate. Then, by storing all or a part (for example, a label portion) of the plane-developed image information as sample information in the sample information storage unit 26, the same processing as in the first embodiment can be executed.

A case where the stitching process is executed will be described as a modification of the process of the second embodiment.

When a plurality of images are photographed, the image information processing unit 22 may execute a stitching process for synthesizing a plurality of images into one image information. The stitching process may be performed before the 3D modeling or after the 3D modeling. A known method can be used for both the stitching process before the three-dimensional modeling and the stitching process after the three-dimensional modeling.

If it is performed before the three-dimensional modeling, the stitching process is executed by detecting the corresponding points between the captured image information. At this time, since the corresponding points can be specified between the captured image information, the depth maps corresponding to the positions of the specified corresponding points are also associated with each other in the same manner. As a result, the stitching process can be executed before the 3D modeling.

In addition, when performing after 3D modeling, the similarity of the uneven shape in the two 3D models is specified from the depth information, and the shapes are similar (the pattern of the unevenness based on the depth information is included in the predetermined range. The stitching process is executed after the 3D modeling by searching for the corresponding points between the captured image information that is the source of the 3D model that corresponds to that area. it can. This is schematically shown in FIG. 13 (a) and 13 (b) are photographed image information obtained by photographing the shelves arranged side by side, FIG. 13 (c) is a three-dimensional model based on FIG. 13 (a), and FIG. 13 (d) is FIG. 13 ( It is a three-dimensional model based on b). In this case, as shown in FIG. 14, for example, in the captured image information, if the reference 101 for stitching processing, for example, the shelf step is captured so as to be reflected, the reference 101 is in the same viewpoint direction. Rotate the 3D model as in. Then, the similarity of the uneven shape is determined based on the depth information. Then, the stitching process can be executed by searching for the corresponding points in the specified similar area 100. FIG. 14A shows a state in which the three-dimensional model of FIG. 13A is rotated so as to have the same viewpoint direction as that of FIG. 13B. The broken line region 100 in FIGS. 14 (a) and 14 (b) is a region 100 having a concave-convex shape determined based on the depth information. Further, the alternate long and short dash line region 101 is a shelf stage that serves as a reference for rotating the three-dimensional model so that the three-dimensional model is oriented in the same viewpoint direction. The reference 101 can be arbitrarily specified depending on the object to be photographed, but it is preferably composed of two or more straight lines.

By executing the above processing, the stitching processing can be executed either before or after the three-dimensional modeling. Further, when the information of a plurality of captured images is not used, the stitching process may not be executed.

As a modification of the process of the second embodiment, the image information processing unit 22 executes the three-dimensional modeling process and then determines the viewpoint direction so that the three-dimensional model to be photographed faces the position. May be executed, and then the process of plane expansion may be executed. For example, when the object to be photographed is a product display shelf or a product located there, the viewpoint direction is determined so that the display shelf or the product faces the position (front position).

As a process for determining the viewpoint direction, the viewpoint direction may be determined so that the reference in the three-dimensional model satisfies a predetermined condition for the position to face each other. The criteria and conditions can be arbitrarily set according to the shooting target and the like. For example, when photographing a display shelf, the front 102 of the shelf of the display shelf is used as a reference, and the condition is that the front 102 of the plurality of shelves of the display shelf are all horizontal and perpendicular to each other. This can be achieved by moving the viewpoint to the center in the vertical and horizontal directions of the shooting range. Normally, when the product display shelf is photographed, the front surface 102 of the shelf is located in the foreground (the frontmost position) of the display shelf, so the depth information of the three-dimensional model indicates that the shelf is in front of the shelf. The front surface 102 can be specified. FIG. 15 schematically shows this process. FIG. 15 (a) is photographed image information obtained by photographing the display shelf, and FIG. 15 (b) is a diagram expressing the depth information in black and white density in the depth map of the display shelf (in the case of black). Located in the foreground (the value of the depth information is small), FIG. 15C is a three-dimensional model based on the depth map, in which the depth information is converted into colors. Then, the image information processing unit 22 identifies the front surface of the shelf of the display shelf based on the depth information, and determines the viewpoint direction so that the shelf satisfies the above-mentioned conditions. Further, the same standard may be used for the standard used in the determination process of the viewpoint direction and the standard in the stitching process after the above-mentioned three-dimensional modeling. For example, in either case, the front surface of the shelf of the display shelf can be used as a reference.

FIG. 15C shows a case where the captured image information is processed in the state before texture mapping for the sake of clarity. However, the viewpoint direction determination processing is executed for the three-dimensional model after the texture mapping. You may. The process of determining the viewpoint direction is not limited to the above, and can be realized by any process.

If the captured image information is originally captured from a position facing it, the viewpoint direction determination process does not have to be executed.

In the processing of Examples 2 to 4, the image information of the face region was specified by executing the face processing on the plane development image information, but the face region was specified and specified from the three-dimensional model. The plane expansion process may be executed for the face region to obtain the plane expansion image information.

There are various methods for specifying the face region from the three-dimensional model, and it can be arbitrarily set according to the characteristics of the sample information. For example, in a 3D model, the position and range of a surface that satisfies a predetermined condition are specified, and the surface is matched with whether or not the condition corresponding to the type of the surface model of the predetermined sample information is satisfied, and the condition is satisfied. Specify the face as a face area. For example, as described above, in the case where the sample information is a product, a three-dimensional model is generated from the photographed image information obtained by photographing the display shelf, and the face area of the product is specified, the following processing can be executed. ..

First, after filtering out the high frequency to exclude fine irregularities, a surface that is close to vertical and has an area above a certain level facing directly, an upright cylindrical surface, and a surface of a polyhedral region that is convex and distorted as a whole. Is detected, and the position and range in the three-dimensional space are specified. For example, if the depth information is within a certain range and the normal is stable and facing the front, there is a strong vertical correlation between the plane and depth information, and if it is convex in the horizontal direction, it is a cylinder or horizontal direction. And in either the vertical direction, it is specified as a plane in the region of a convex or distorted polyhedron if the normal is not stable. Then, each detected surface is labeled. FIG. 16 shows a state in which five surfaces are detected and labeled from 1 to 5 respectively. Then, for the detected surface, the distance from the foreground (this is used as the level) is calculated in order from the foreground based on, for example, the average value of the depth information. Then, based on the calculated level, the context of each surface is specified. In FIG. 16, the level of the surfaces 1 and 2 is 0 cm, the level of the surfaces 3 and 4 is 3 cm, and the level of the surface 5 is 20 cm. Then, the front-back relationship of the five faces is specified, with the frontmost faces being the faces 1 and 2, the next being located in the back being the faces 3 and 4, and the backmost being the face 5.

Then, the image information processing unit 22 specifies the type of surface by using the context of the surface specified in this way. That is, the depth information is the same as the front surface of the shelf, the product tag surface, the back surface of the shelf, and the front surface of the shelf on the narrow horizontal plane on the depth map. Alternatively, a vertical rectangle that is close (within a certain range, for example, within a few centimeters) and is hollow is specified as the surface of the product tag. Then, preferably, the other side is specified as a product side (a side on which the product may be displayed). Therefore, in the example of FIG. 16, the surfaces 1 and 2 are specified as the front surface and the product tag surface of the shelf board, the surfaces 3 and 4 are specified as the product surface, and the surface 5 is specified as the rear plate surface of the shelf body.

On the product side specified in this way, the corresponding process is executed based on the package type of the product for which input is accepted at any timing. Package types include cans, bottles (including PET bottles; the same applies hereinafter), coexistence of cans and bottles, boxes, hanging products, etc., but are not limited to these.

If the package type is a can or bottle, the area of the cylinder within a certain size range is specified as the face area of the can or bottle. This is schematically shown in FIGS. 17 and 18. FIG. 17 is an image diagram showing the case where the package type is a can, and FIG. 18 is an image diagram showing the case where the package type is a bottle.

When the package type is a coexistence of a can and a bottle, there is a cylinder within a certain size range, an area without a small cylinder above is the face area of the can, and there is a cylinder within a certain size range. The area with a small cylinder above is specified as the face area of the bottle.

If the package type is a box, the area of the vertical rectangular plane within a certain size range is specified as the face area of the box. This is schematically shown in FIG.

If the package type is a hanging product, the area of a rectangular flat surface or uneven surface that is within a certain size range and floats in the air is specified as the face area of the hanging product. This is schematically shown in FIG.

The image information processing unit 22 may specify the face area without accepting the selection of the package type. In this case, it is determined whether or not the condition as the face area is satisfied by executing the determination process for each of the product surface area and the package type. Then, if there is a surface that constitutes the face area, the face area that the surface constitutes is specified, and what kind of package type the face area is based on the predetermined package type priority condition. To identify.

The process of specifying the face region from the three-dimensional model can adopt any method depending on the category and shape of the product, and is not limited to the above. In addition, the person in charge may accept correction input for the automatically specified face area. Further, the person in charge may input the position of the face area.

Further, the image information processing unit 22 may specify the face region by using deep learning. In this case, the above three-dimensional model is input to the learning model in which the weighting coefficient between the neurons of each layer of the neural network consisting of many layers is optimized, and the face region is set based on the output value. It may be specified. Further, as a learning model, various three-dimensional models with face regions can be used as correct answer data.

The image information processing unit 22 cuts out the face region specified as described above. Cutting out the face area may actually cut out the face area specified from the three-dimensional model, and also includes setting the area as the processing target of the processing described later.

When cutting out the face area, the face area specified by each of the above processes may be cut out as it is, or the face area may be specified by a plurality of methods and the target to be cut out using the result of the face area specified by each method. The face area to be used may be specified. For example, a method of specifying the position and range of a surface that satisfies a predetermined condition from a three-dimensional model and matching whether the condition is satisfied according to the type of the surface model to specify the face region, and a method of specifying the face region by deep learning. The face area to be finally cut out may be specified by a method of specifying and a predetermined calculation. The method for specifying the face area to be cut out is not limited to the above, and can be set arbitrarily.

The image information processing unit 22 executes the same plane expansion processing as in the second embodiment on the face region cut out by the image information processing unit 22, and generates plane expansion image information for the face region. Then, if the plane development image information generated in this way is treated as the captured image information in the above-described first embodiment and the processing after the block identification processing is executed, the same processing as in the first embodiment can be executed.

Further, the sample information processing unit 25 generates a three-dimensional model from the image information obtained by photographing the object and the depth map, and cuts out the face region based on the three-dimensional model, similarly to the image information processing unit 22 described above. .. Then, the same processing as in the first embodiment can be executed by generating the plane development image information from the cut out face area and storing the plane development image information as the sample information in the sample information storage unit 26.

Each process in each of the above-described examples is not limited to the order described in the specification of the present invention, and can be appropriately changed as long as the object is achieved.

Further, the image processing system 1 of the present invention is effective in specifying the display status of a product displayed on the display shelf from the photographed image information obtained by photographing the display shelf of the store. It is not limited. That is, when a certain object to be photographed is photographed, it can be widely used when specifying the area in which the desired object is captured from the photographed image information.

By using the image processing system 1 of the present invention, when identifying an object such as a product shown in image information obtained by photographing a display shelf or the like, the accuracy of comparison processing can be improved without requiring more calculation time than before. It is possible to improve.

1: Image processing system 2: Management terminal 3: Input terminal 20: Image information input reception processing unit 21: Image information storage unit 22: Image information processing unit 23: Candidate processing unit 24: Comparison processing unit 25: Sample information processing unit 26 : Specimen information storage unit 70: Arithmetic device 71: Storage device 72: Display device 73: Input device 74: Communication device

Claims (13)

An image processing system that identifies an object reflected in image information.
The image processing system is
Information in a second area that satisfies a predetermined condition in the first area in the image information to be processed, and block information stored in a sample information storage unit that stores sample information and block information of the object to be sampled. A candidate processing unit that executes similar search processing using and identifies the target object to be a candidate for comparison processing, and
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. It has a comparison processing unit to specify,
The candidate processing unit
In the image information of the first region,
The image information in the first region is reduced to a predetermined number of colors.
The image information of the first region is divided into meshes of a predetermined size to generate a distribution map for each of the reduced colors.
For the distribution map for each color, a group that fits within a predetermined range is detected.
For each group, the area surrounding the group in which the number of colors included in the area is within a predetermined number and the edge is equal to or more than a predetermined threshold value is specified as the second area.
An image processing system characterized by this. An image processing system that identifies an object reflected in image information.
The image processing system is
Information in a second area that satisfies a predetermined condition in the first area in the image information to be processed, and block information stored in a sample information storage unit that stores sample information and block information of the object to be sampled. A candidate processing unit that executes similar search processing using and identifies the target object to be a candidate for comparison processing, and
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. It has a comparison processing unit to specify,
The candidate processing unit
By extracting the feature amount from the image information of the first region, a region in which common features are distributed over a range of a predetermined size or more is detected and the region is erased.
For the remaining region, a region in which common features are distributed is specified, and a region surrounding the region is specified as the second region.
An image processing system characterized by this. An image processing system that identifies an object reflected in image information.
The image processing system is
Information in a second area that satisfies a predetermined condition in the first area in the image information to be processed, and block information stored in a sample information storage unit that stores sample information and block information of the object to be sampled. A candidate processing unit that executes similar search processing using and identifies the target object to be a candidate for comparison processing, and
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. It has a comparison processing unit to specify,
The candidate processing unit
For a learning model in which the weighting coefficient between neurons in each layer of a neural network consisting of many layers is optimized, in which the second area is given as correct data to the image information of various first areas. The image information of the first region in the image information to be processed is input, and the second region is specified based on the output value.
An image processing system characterized by this. An image processing system that identifies an object reflected in image information.
The image processing system is
Information in a second area that satisfies a predetermined condition in the first area in the image information to be processed, and block information stored in a sample information storage unit that stores sample information and block information of the object to be sampled. A candidate processing unit that executes similar search processing using and identifies the target object to be a candidate for comparison processing, and
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. It has a comparison processing unit to specify,
The candidate processing unit
As a learning model in advance, the feature amount in the second region in the image information of various first regions is used as a positive example, and the feature amount outside the second region in the image information of various first regions is used as a negative example. Each point in the mesh obtained by dividing the image information of the first region in the image information to be processed is input to the given determination processing unit.
For each of the points, the determination processing unit is made to determine whether or not it belongs to the second region.
In the determination result, the region surrounding the mesh determined to belong to the second region is specified as the second region.
An image processing system characterized by this. An image processing system that identifies an object reflected in image information.
The image processing system is
Information in a second area that satisfies a predetermined condition in the first area in the image information to be processed, and block information stored in a sample information storage unit that stores sample information and block information of the object to be sampled. A candidate processing unit that executes similar search processing using and identifies the target object to be a candidate for comparison processing, and
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. It has a comparison processing unit to specify,
The candidate processing unit
In the similarity search process between the information in the second region and the block information of the object, the similarity search process is executed as it is for any one or more of the respective sizes, scales, and rotations.
The comparison processing unit
In the comparison process between the information in the first region and the sample information of the object specified as the candidate, the comparison process is executed by aligning any one or more of the size, scale, and rotation.
An image processing system characterized by this. An image processing system that identifies an object reflected in image information.
The image processing system is
For a part or all of the three-dimensional model generated by using the image information and the corresponding depth information, the plane-expanded image information developed on a plane is generated, and a part or all of the generated plane-developed image information is displayed. The image information processing unit that uses the image information to be processed ,
Block information stored in a sample information storage unit that stores information in a second area that satisfies a predetermined condition in the first area of the image information to be processed, and sample information and block information of the object to be sampled. A candidate processing unit that executes similar search processing using and and identifies objects to be candidates for comparison processing, and
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. The comparison processing unit to be specified and
An image processing system characterized by having. The candidate processing unit
In the specified first region, a region having information capable of identifying an object is specified as a second region.
The image processing system according to any one of claims 1 to 6, wherein the image processing system is characterized in that. Computer,
An object stored in a sample information storage unit that stores information in a second area that satisfies a predetermined condition in the first area of image information to be processed, and sample information and block information of the object to be sampled. Candidate processing unit that executes similar search processing using block information and identifies objects to be candidates for comparison processing,
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. An image processing program that functions as a specific comparison processing unit.
The candidate processing unit
In the image information of the first region,
The image information in the first region is reduced to a predetermined number of colors.
The image information of the first region is divided into meshes of a predetermined size to generate a distribution map for each of the reduced colors.
For the distribution map for each color, a group that fits within a predetermined range is detected.
For each group, the area surrounding the group in which the number of colors contained in the area is within a predetermined number and the edge is equal to or more than a predetermined threshold value is specified as the second area.
An image processing program characterized by this. Computer,
An object stored in a sample information storage unit that stores information in a second area that satisfies a predetermined condition in the first area of image information to be processed, and sample information and block information of the object to be sampled. Candidate processing unit that executes similar search processing using block information and identifies objects to be candidates for comparison processing,
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. An image processing program that functions as a specific comparison processing unit.
The candidate processing unit
By extracting the feature amount from the image information of the first region, a region in which common features are distributed over a range of a predetermined size or more is detected and the region is erased.
For the remaining region, a region in which common features are distributed is specified, and a region surrounding the region is specified as the second region.
An image processing program characterized by this. Computer,
An object stored in a sample information storage unit that stores information in a second area that satisfies a predetermined condition in the first area of image information to be processed, and sample information and block information of the object to be sampled. Candidate processing unit that executes similar search processing using block information and identifies objects to be candidates for comparison processing,
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. An image processing program that functions as a specific comparison processing unit.
The candidate processing unit
For a learning model in which the weighting coefficient between neurons in each layer of a neural network consisting of many layers is optimized, in which the second area is given as correct data to the image information of various first areas. The image information of the first region in the image information to be processed is input, and the second region is specified based on the output value.
An image processing program characterized by this. Computer,
An object stored in a sample information storage unit that stores information in a second area that satisfies a predetermined condition in the first area of image information to be processed, and sample information and block information of the object to be sampled. Candidate processing unit that executes similar search processing using block information and identifies objects to be candidates for comparison processing,
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. An image processing program that functions as a specific comparison processing unit.
The candidate processing unit
As a learning model in advance, the feature amount in the second region in the image information of various first regions is used as a positive example, and the feature amount outside the second region in the image information of various first regions is used as a negative example. Each point in the mesh obtained by dividing the image information of the first region in the image information to be processed is input to the given determination processing unit.
For each of the points, the determination processing unit is made to determine whether or not it belongs to the second region.
In the determination result, the region surrounding the mesh determined to belong to the second region is specified as the second region.
An image processing program characterized by this. Computer,
An object stored in a sample information storage unit that stores information in a second area that satisfies a predetermined condition in the first area of image information to be processed, and sample information and block information of the object to be sampled. Candidate processing unit that executes similar search processing using block information and identifies objects to be candidates for comparison processing,
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. An image processing program that functions as a specific comparison processing unit.
The candidate processing unit
In the similarity search process between the information in the second region and the block information of the object, the similarity search process is executed as it is for any one or more of the respective sizes, scales, and rotations.
The comparison processing unit
In the comparison process between the information in the first region and the sample information of the object specified as the candidate, the comparison process is executed by aligning any one or more of the size, scale, and rotation.
An image processing program characterized by this. Computer,
For a part or all of the three-dimensional model generated by using the image information and the corresponding depth information, the plane-expanded image information developed on a plane is generated, and a part or all of the generated plane-developed image information is displayed. Image information processing unit, which is the image information to be processed,
An object stored in a sample information storage unit that stores information in a second area that satisfies a predetermined condition in the first area of the image information to be processed, and sample information and block information of the object to be sampled. Candidate processing unit that executes similar search processing using the block information of and identifies the target object to be the candidate for comparison processing.
By performing comparison processing using the first region of the image information of the processing target and the sample information of the object specified as a candidate by the candidate processing unit, the object reflected in the first region can be obtained. Comparison processing unit to identify,
An image processing program characterized by functioning as.