JP2023170655A - information processing system - Google Patents

Abstract

To provide an information processing system for improving recognition accuracy in the case of identifying a product from image information.SOLUTION: An information processing system for identifying a product shown in image information includes a first learning processing part for performing machine learning by using first annotation data obtained by associating data from masking the inner side of the closed area delineated by the product's outer shape with attributes to create a first learning model, and a recognition processing part for identifying product identification information of the product shown in the image information obtained by photographing a display shelf. In the information processing system, the recognition processing part identifies the outer shape of the product shown in the image information as a face area by using the image information and the first learning model, and identifies the product identification information of the product shown in the identified face area.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing system that does not reduce recognition accuracy when identifying products from image information.

BACKGROUND ART In various stores such as convenience stores and supermarkets, it is common for products to be sold on display shelves. This display method involves displaying multiple products horizontally to make them more visible to buyers, or allowing other people to purchase the same product even if one product is purchased. In some cases, products are displayed vertically. Managing where and how many products are displayed on a display shelf is important in terms of product sales strategy.

Therefore, in order to understand the actual display status of products in a store, there is a method of photographing the display shelves with a photographing device and automatically identifying the displayed products from the photographed image information. For example, there is a method of using image recognition technology on images of store display shelves based on sample images of each product. These conventional techniques include, for example, Patent Document 1 and Patent Document 2 below.

Japanese Patent Application Publication No. 5-342230 Japanese Patent Application Publication No. 5-334409

The invention of Patent Document 1 is a system that helps even those without knowledge to decide on which display shelves products should be displayed. Therefore, although it is possible to grasp where products are displayed, it does not specify the products that are displayed. Further, Patent Document 2 is a system that supports input of product images in a planogram support system that supports display of products. However, the system of Patent Document 2 only supports the input of product images when using the planogram support system, and even if this system is used, it is not possible to grasp the specific display status of products. .

Furthermore, in addition to Patent Document 1 and Patent Document 2, there is also a technology that uses image recognition processing technology to identify displayed products from image information obtained by photographing display shelves. This is useful in that it is possible to grasp the actual display status at the store.

In conventional technology, when identifying a product using image recognition processing technology, a rectangular area where the product is thought to be is detected from image information taken of a display shelf, and matching processing is performed for that rectangular area with a sample image of the product. Alternatively, the product can be identified by performing deep learning processing using the rectangular area as an input value.

However, the shape (outline) of the product is not necessarily rectangular. When a rectangular area is detected that is thought to contain a product, part of another product may be reflected in the rectangular area. Further, the background may be reflected in the rectangular area. Therefore, if such a rectangular area is used as training data for deep learning, an image to be processed, or image matching processing, there is a problem that it causes a decrease in the accuracy of product identification.

In view of the above problems, the present inventors have invented an information processing system that improves the accuracy of product identification when identifying products on display from image information.

A first invention is an information processing system for identifying a product shown in image information, and a first annotation that associates data obtained by masking the inside of a closed area with the outline of the product as an external shape and an attribute. a first learning processing unit that performs machine learning using data to create a first learning model; and a recognition processing unit that identifies product identification information of products shown in image information taken of display shelves. The recognition processing unit uses the image information and the first learning model to identify the external shape of the product shown in the image information as a face area, and identifies the external shape of the product shown in the identified face area. This is an information processing system that identifies product identification information for products that are currently being sold.

As in the present invention, by identifying product identification information using a learning model that is machine learned using first annotation data based on mask-processed data with the outline of the product as the external shape, an image of a display shelf is created. The face area extracted from the information can be changed from a conventional rectangular area to an area that follows the outer shape of the product. This makes it possible to reduce the reflection of other products and the background, and improve the accuracy when identifying products.

In the above invention, the first learning processing unit may be configured as an information processing system that creates the first learning model by performing machine learning by image segmentation using the first annotation data. Can be done.

When performing machine learning, machine learning using an image segmentation method is preferable.

In the above invention, the information processing system performs a second learning process of creating a second learning model by performing machine learning using second annotation data that associates product image data with product identification information. The recognition processing unit uses the image information of the identified face area and the second learning model to identify product identification information of the product shown in the face area. It can be configured like an information processing system.

In the above invention, the recognition processing unit performs image matching processing on the image information of the identified face area and the sample information of the product stored in the sample information storage unit, thereby identifying the product appearing in the face area. It can be configured like an information processing system that identifies product identification information.

Since the face area is composed of an area whose outer shape is the outline of the product, it is preferable to identify the product identification information by performing processing such as those disclosed in these inventions.

A fifth invention is an information processing system for identifying a product shown in image information, and a first annotation that associates data obtained by masking the inside of a closed area with the outline of the product as an external shape and an attribute. It has a first learning processing unit that performs machine learning using data to create a first learning model, and uses the first learning model and image information obtained by photographing the display shelf to This information processing system specifies the external shape of a product shown in image information as a face area, and identifies product identification information of the product shown in the specified face area.

A sixth invention is an information processing system for identifying products shown in image information, which includes an image information input reception processing unit that receives input of image information obtained by photographing a display shelf; It has a displayed product recognition processing unit that identifies product identification information of the product in the image from the image information that has been normalized, and the displayed product recognition processing unit identifies the outline of the product as an external shape. A first learning model created by machine learning using first annotation data in which data obtained by masking the inside of the closed region and attributes are associated with each other; An information processing system that uses image information in which image information is normalized to identify the external shape of the product shown as a face area, and identifies product identification information of the product shown in the specified face area. be.

Even with the configurations of these inventions, the same technical effects as the first invention can be obtained.

The first invention can be realized by loading and executing the program of the invention into a computer. That is, a computer performs machine learning to create a first learning model using first annotation data in which attributes are associated with data obtained by masking the inside of a closed area using the outline of the product as an external shape. an information processing program that functions as a learning processing section of No. 1 and a recognition processing section that identifies product identification information of a product shown in image information obtained by photographing a display shelf; The information processing program is an information processing program that uses a first learning model to identify an external shape of a product shown in the image information as a face area, and identifies product identification information of a product shown in the identified face area. .

The fifth invention can be realized by loading and executing the program of the invention into a computer. That is, a computer performs machine learning to create a first learning model using first annotation data in which attributes are associated with data obtained by masking the inside of a closed area using the outline of the product as an external shape. 1, the information processing program is configured to function as a learning processing unit according to the first aspect of the present invention, which uses the first learning model and image information obtained by photographing a display shelf to identify an external shape of a product shown in the image information as a face area. This information processing program causes the user to identify the product identification information of the product shown in the specified face area.

The sixth invention can be realized by loading and executing the program of the invention into a computer. That is, the computer is operated by an image information input reception processing unit that receives input of image information of a photograph of a display shelf, and identifies the product in the photograph based on the input image information or the image information obtained by normalizing the image information. An information processing program that functions as a displayed product recognition processing unit that identifies information, wherein the displayed product recognition processing unit associates data obtained by masking the inside of a closed area with the outline of the product as an external shape, and attributes. The first learning model created by machine learning using the first annotation data and the image information that received the input or the image information that normalized the image information, This information processing program specifies the external shape of a product shown in the image as a face area, and identifies product identification information of the product shown in the specified face area.

By using the information processing system of the present invention, it is possible to improve the accuracy of identification when identifying products on display from image information.

1 is a block diagram schematically showing an example of the configuration of an information processing system of the present invention. FIG. 2 is a block diagram schematically showing an example of the configuration of a displayed product recognition processing section in the information processing system of the present invention. FIG. 1 is a block diagram schematically showing an example of the hardware configuration of a computer used in the information processing system of the present invention. It is a flowchart which shows an example of the processing process of the learning process in the information processing system of this invention. 3 is a flowchart illustrating an example of a recognition processing process in the information processing system of the present invention. FIG. 3 is a diagram schematically showing an example of first annotation data. It is a figure which shows typically another example of 1st annotation data. It is a figure which shows typically an example of 2nd annotation data. It is a figure which shows typically another example of 2nd annotation data. FIG. 3 is a diagram showing an example of photographed image information. FIG. 7 is a diagram showing another example of photographed image information. 11 is a diagram showing an example of image information obtained by normalizing the captured image information of FIG. 10. FIG. 12 is a diagram illustrating an example of image information obtained by normalizing the photographed image information of FIG. 11. FIG. FIG. 6 is a diagram schematically showing a state in which an input for specifying a shelf area has been received for normalized image information obtained by normalizing image information obtained by photographing display shelves on which products are displayed. FIG. 6 is a diagram schematically showing a state in which an input for specifying a shelf area has been received with respect to normal image information obtained by normalizing image information obtained by photographing a display shelf on which products are hung and displayed. FIG. 7 is a diagram showing an example of a case where a face area is specified from image information of a shelf area. FIG. 3 is a diagram showing an example of photographed image information. 18 is a diagram showing an example of normalized image information obtained by performing normalization processing on the captured image information of FIG. 17. FIG. 2 is a block diagram schematically showing an example of the configuration of an information processing system in Example 2. FIG. It is a figure showing an example of specimen information stored in a specimen information storage part. FIG. 7 is a block diagram schematically showing an example of a displayed product recognition processing section in Example 3. FIG.

An example of the processing functions of the information processing system 1 of the present invention is shown in block diagrams in FIGS. 1 and 2. The information processing system 1 uses a management terminal 2 and an image information input terminal 3. FIG. 1 is a block diagram showing the overall functions of the information processing system 1, and FIG. 2 is a block diagram showing the functions of a displayed product recognition processing section 214, which will be described later.

The management terminal 2 is a computer used by an organization such as a company that operates the information processing system 1. Further, the image information input terminal 3 is a terminal for inputting image information obtained by photographing display shelves in a store.

The management terminal 2 and image information input terminal 3 in the information processing system 1 are realized using a computer. FIG. 3 schematically shows an example of the hardware configuration of a computer. A computer has an arithmetic unit 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or hard disk that stores information, and a display device 72 such as a display that displays information, and is capable of inputting information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits and receives processing results of the arithmetic device 70 and information stored in the storage device 71 via a network such as the Internet or a LAN.

When the computer is equipped with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used, for example, in portable communication terminals such as tablet computers and smartphones, but are not limited thereto.

The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be performed directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

The image information input terminal 3 may include a photographing device such as a camera in addition to the above devices. As the image information input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can also be used.

The respective means in the present invention are only logically distinguished in their functions, and may physically or practically form the same area. The order of the processing in each means of the present invention can be changed as appropriate. Also, part of the processing may be omitted. For example, it is also possible to omit the perpendicularization process, which will be described later. In that case, it is possible to perform processing on image information that has not been subjected to normalization processing.

The information processing system 1 includes a learning processing section 20 and a recognition processing section 21. The learning processing section 20 includes a first learning processing section 201 and a second learning processing section 202.

The first learning processing unit 201 uses the first annotation data to perform a learning process using machine learning on image information obtained by photographing a display shelf, preferably a learning process using an image segmentation method. This learning process is a learning process in machine learning, and for example, in order to create a learning model using deep learning, a learning process using image segmentation is executed.

The first annotation data consists of data in which the outline of a product that is likely to be displayed on a display shelf is used as an external shape, and the inside of a closed area formed by the outline is masked, and tags that classify the attributes of the product's outline ( This is data that corresponds to the label. The first annotation data does not need to be one for one product, and may be plural for one product. That is, data obtained by masking the inside of the outline of a product from a plurality of directions as an external shape may be associated with an attribute, and each may be used as the first annotation data of the product. The attributes include, for example, the classification of the product's container and the product identification information (JAN code, etc.) of the product. Container classification may be by type of container, such as cans, bottles, boxes, and pouch containers, or may be further subdivided according to usage, such as detergent containers. That is, the attribute may be anything that indicates how the closed area based on the outline of the product is classified. An example of the first annotation data is shown in FIGS. 6 and 7. The product identification information is not limited to the JAN code, but may be any information that can uniquely identify the product.

Figure 6 shows the case where the outline of a can and the data obtained by masking the inside of the closed area formed by the outline are associated with "can" as an attribute to form the first annotation data. A case is shown in which first annotation data is created by associating data obtained by masking the outline of shampoo and the inside of a closed region formed by the outline with "pouch container" as an attribute. As the attribute in the first annotation data, if the product can be identified from the outline of the product itself, product identification information such as a JAN code may be used instead of the container classification.

By executing a learning process for machine learning using the first annotation data in the first learning processing unit 201, a learning model for identifying the outline area of a product from image information obtained by photographing a display shelf is created. (first learning model).

The second learning processing unit 202 executes a machine learning learning process using the second annotation data to determine the products in the area based on predetermined image information, preferably image information of a face area (described later). A learning model (second learning model) for identifying product identification information is created. In this case, it is preferable to perform a learning process using an image classification method, but it is also recommended to perform a learning process using methods such as object detection, image classification/localization, etc. There may be.

The second annotation data is data in which image information of a product that may be displayed on a display shelf is associated with product identification information of the product as a tag (label). Examples of the second annotation data are shown in FIGS. 8 and 9. Similarly to the first annotation data, the second annotation data does not need to be one per product, and may be plural. That is, the product may be photographed from a plurality of directions, and image information of the product taken from each direction may be associated with product identification information to form the second annotation data.

Figure 8 shows a case where image information of a can is associated with product identification information to form second annotation data, and Figure 9 shows a case where image information of a refillable shampoo is associated with product identification information. This shows the case where the annotation data is used as the second annotation data. The second annotation data in FIG. 8 corresponds to the first annotation data in FIG. 6, and the second annotation data in FIG. 8 corresponds to the second annotation data in FIG.

The recognition processing section 21 includes an image information input reception processing section 210, an image information storage section 211, an image information uprighting processing section 212, a shelf identification processing section 213, and a displayed product recognition processing section 214.

The image information input reception processing section 210 receives input of image information (photographed image information) of a store display shelf photographed by the image information input terminal 3, and stores it in the image information storage section 211, which will be described later. It is preferable to receive input from the image information input terminal 3, in addition to photographed image information, store identification information such as photographing date and time, store name, and image information identification information for identifying image information. Examples of captured image information are shown in FIGS. 10 and 11. In FIGS. 10 and 11, the photographed image information shows that there are three shelves on the display shelf, and products are displayed on the shelves. Although the present invention does not specifically specify the processing, display shelves and shelves are often long in the horizontal direction. Therefore, in the process, the data may be divided into sections of a certain width and used as processing targets for each process.

The image information storage unit 211 stores photographed image information, photographed date and time, store identification information, image information identification information, etc. received from the image information input terminal 3 in association with each other. The photographed image information may be any image information to be processed by the present invention. Generally, when simply taking a picture, it is difficult to take a picture with the subject facing directly, so correction processing such as keystone correction processing is performed to correct the subject so that it is facing directly. Good. It also includes image information obtained by combining multiple images of one display shelf into one image. Further, image information after the distortion correction process has been executed is also included in the photographed image information.

The image information normalization processing section 212 performs a process (orientation processing) of correcting the photographed image information stored in the image information storage section 211 so that the object to be photographed faces directly, for example, trapezoidal correction. Generate orthogonal image information that has been processed. The trapezoidal correction process is a correction process performed so that the shelves of the display shelf shown in the photographed image information are horizontal. Orthogonalization means transforming the image information so that the optical axis of the lens of the imaging device is perpendicular to the plane to be photographed, and the image information is the same as when photographed from a sufficiently far distance. Although there is a correction process, it is not limited thereto.

The trapezoidal correction process executed by the image information uprighting processing unit 212 receives input specifying four vertices in the captured image information, and executes the trapezoidal correction process using each of the vertices. The four vertices that accept the designation may be the four vertices of the shelves of the display shelf, or the four vertices of the shelf positions of the display shelf. Alternatively, it may be four vertices of a group of two or three shelves. Any four points can be specified as the four vertices. FIG. 12 shows an example of the photographed image information of FIG. 10, and FIG. 13 shows an example of the photographed image information (original image information) obtained by normalizing the photographed image information of FIG. 11.

The shelf identification processing unit 213 selects a shelf area where a product may be placed ( (shelf area). Although a display shelf is shown in the photographed image information and the normal position image information, the display shelf includes a shelf area where products are displayed. Therefore, the shelf area is specified from the normal position image information. To specify the shelf area, the operator of the management terminal 2 may manually specify the shelf area, and the shelf specification processing unit 213 may accept it, or the shelf area for which the input was manually received for the first time may be specified. Based on the information, the shelf area may be automatically specified from the second time onward.

FIG. 14 schematically shows a state in which an input for specifying a shelf area has been received for normalized image information that is a normalized image of a display shelf on which products such as beverage cans are displayed. . In addition, FIG. 15 shows a state in which an input for specifying a shelf area is accepted for the upright image information, which is an upright position of image information taken of a display shelf on which products such as toothbrushes are hung and displayed. is schematically shown.

Note that when identifying the shelf area, the shelf identification processing unit 213 may use deep learning to specify the shelf area. In this case, the above-mentioned orthogonal image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and based on the output value, the shelf A region may also be specified. Further, as a learning model, it is possible to use various types of orthographic image information in which shelf areas are given as correct data.

The shelf area identified by the shelf identification processing unit 213 specifies its image information as shelf area image information. The shelf identification processing unit 213 may actually cut out the image information, or may virtually cut it out by specifying the image information of the area by coordinates, etc., without actually cutting out the image information. good. Note that if there are multiple shelves on the display shelf, each shelf is cut out as shelf area image information. Further, the coordinates indicating the area of the shelf are the coordinates of the vertices for specifying the area, and may be the coordinates of, for example, four points, the upper right and lower left, and the upper left and lower right, in the orthogonal image information. Further, it is a relative coordinate based on a predetermined location in the image information (for example, the top left vertex of the display shelf) such as a display shelf in the normal image information. Note that in this specification, cutting out image information may mean actually cutting out image information, similar to cutting out in the shelf identification processing unit 213, or actually cutting out image information without cutting out image information. It is also possible to virtually cut out the image information by specifying the image information using coordinates or the like.

The displayed product recognition processing unit 214 executes a process of recognizing displayed products from the display shelf shown in the image information, preferably photographed image information or normal orientation image information.

The displayed product recognition processing section 214 includes a face identification processing section 2141 and a product identification processing section 2142.

The face identification processing unit 2141 identifies a face area (face area) for each shelf in the shelf area in the normal image information. A face is an area where a product is placed, and it does not matter whether the product is placed there or not. The size of the face area is the same or approximately the same size as the product to be placed there.

The face identification processing unit 2141 inputs photographed image information, normal orientation image information, or image information of the shelf region as an input value to the learning model (first learning model) learned by the first learning processing unit 201. , Identify the face area from the input image information. That is, the face identification processing unit 2141 uses the learning model trained in the first learning processing unit 201 (a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized). On the other hand, image information of a region to be processed, for example, a shelf region, is input, and the face region is specified based on the output value. For the identified face area, face identification information that identifies the face area is assigned, and product identification information is stored together with photographed image information, normal position image information, or position information in the image information of the shelf area (for example, coordinates in the image information). 22.

FIG. 16 shows an example of a case where the face area is identified from the image information of the shelf area. FIG. 16(a) is an example of image information of the shelf region input to the learning model trained by the first learning processing unit 201, and FIG. 16(b) is an example of the input value and the input value in FIG. 16(a). FIG. 7 is a diagram showing an example of a state in which a face region is identified using the learning model in the image information of the shelf region. Although FIG. 16(b) shows a superimposed state in which the face area is identified in the shelf area, the image information of the identified face area may be cut out and output as is.

The product identification processing unit 2142 inputs the image information of the face area into a learning model (second learning model) that has been trained by the second learning processing unit 202 to learn the product identification information of the product displayed in the face area. It is input as a value, and the identification information of the product in that area is identified from the input image information. That is, the product identification processing unit 2142 uses the learning model trained in the second learning processing unit 202 (a learning model in which the weighting coefficients between neurons in each layer of a neural network whose intermediate layer is composed of many layers are optimized). On the other hand, image information of the face area to be processed is input, and product identification information of the product in the face area is identified based on the output value.

The displayed product recognition processing section 214 may perform the processing of the face identification processing section 2141 and the product identification processing section 2142 together using deep learning or the like.

The product identification information storage unit 22 stores information indicating product identification information of products displayed on each face of the shelves of the display shelf. For example, product identification can be performed in association with product identification information, photographing date and time information, store information, image information identification information of photographed image information, image identification information of normal position image information, and face identification information for identifying faces. The information is stored in the information storage section 22.

Next, an example of the processing process of the information processing system 1 of the present invention will be explained using the flowcharts of FIGS. 4 and 5. In the following description, a case will be described in which product identification information of a displayed product is identified from photographed image information.

First, a learning process for learning a learning model used in the recognition processing unit 21 of the information processing system 1 of the present invention will be described using the flowchart of FIG. 4.

First annotation data is created as training data for the learning model in the first learning processing unit 201 (S100). The first annotation data is image data in which an outer shape is formed by the outline of a product that may be displayed on a display shelf, and a closed area inside the outer shape is masked. This image data is created by associating attributes with tags (see FIGS. 6 and 7).

Similarly, second annotation data is created as training data for the learning model in the second learning processing unit 202 (S110). The second annotation data is created by associating product identification information of the product as a tag with an image of the product that may be displayed on the display shelf. The image information of the product at this time preferably corresponds to the first annotation data, and is preferably image information whose outer shape is the outline of the product.

Then, the created first annotation data is input as training data, the first learning processing unit 201 executes learning processing for machine learning, and identifies the contour area of the product from the image information obtained by photographing the display shelf. A learning model (first learning model) is created (S120).

In addition, the created second annotation data is input as training data, and the second learning processing unit 202 executes learning processing for machine learning to identify the product in the area from the image information, preferably the face area. A learning model (second learning model) for identifying information is created (S130).

By performing the above processing, each learning model can be created.

Next, a recognition process for identifying product identification information of products displayed on a display shelf from image information obtained by photographing the display shelf will be explained using the flowchart of FIG.

Photographed image information of a store display shelf is input from the image information input terminal 3, and the input is accepted by the image information input reception processing unit 210 of the management terminal 2 (S200). FIG. 17 shows an example of captured image information. It also accepts input of photographing date and time, store identification information, and image information identification information of photographed image information. Then, the image information input reception processing section 210 associates and stores the input photographed image information, photographing date and time, store identification information, and image information identification information of the photographed image information in the image information storage section 211.

When a predetermined operation input is received on the management terminal 2, the orthogonal image information orthogonalization processing unit 212 extracts the photographed image information stored in the image information storage unit 211, and performs orthogonalization processing such as trapezoidal correction processing. The four points of shelf positions (positions of display shelves) that are the vertices of the display are received, and a normalization process is executed (S210). FIG. 18 shows an example of photographed image information (normalized image information) on which the normalization process has been performed in this manner.

Then, by receiving a predetermined operation input on the management terminal 2 with respect to the normal position image information, the shelf identification processing unit 213 identifies the shelf position area (S220). That is, input of the shelf area in the normal position image information is accepted. FIGS. 14 and 15 are diagrams showing a state in which the shelf area is specified from the normal position image information.

Once the shelf area is identified as described above, image information of the shelf area is extracted from the normal position image information. Then, a process of identifying a face is executed for each shelf in the shelf area image information (S230). That is, the face identification processing unit 2141 inputs the image information of the shelf area as an input value to the learning model trained in the first learning processing unit 201 (first learning model), and determines the face from the input image information. Identify the area.

After specifying each face area of each shelf in the shelf position area image information of the normal position image information as described above, the product identification processing unit 2142 uses the learning model learned by the second learning processing unit 202 (second The image information of the face area is input as an input value to the learning model (learning model), and the product identification information of the product shown in the face area is identified (S240). The identified product identification information is stored in the product identification information storage unit 22 in association with the photographing date and time, store identification information, image information identification information of the photographed image information, image information identification information of the normal position image information, and face identification information.

Note that it is not always possible to identify product identification information in all face areas. Therefore, for face areas that cannot be identified, input of product identification information is accepted, and the input product identification information is converted into the photographing date and time, store identification information, image information identification information of the photographed image information, and image information identification information of the upright image information. The product identification information is stored in the product identification information storage unit 22 in association with the face identification information. Further, input may be similarly accepted for the correction process of the identified product identification information.

By performing the above processing, it is possible to identify the product identification information of the product displayed on the shelf of the display shelf shown in the photographed image information. In addition, unlike conventional systems, the face area is not a rectangular area, but the product is identified along the outline of the product, so unnecessary information contained in the face area, such as noise from other products, is excluded. Therefore, recognition accuracy improves.

Note that if product identification information is used as an attribute in the first annotation data (if the product can be identified from the product's external shape), the image information may be input to the first learning model to identify the face area. You can identify the product identification information of the product. In that case, it is not necessary to execute the processing by the second learning processing unit 202 and the product identification processing unit 2142, and once the face area is identified by the face identification processing unit 2141, the product identified in the face area can be identified. The information may be identified using attribute product identification information as an output result of the first learning model.

In Example 1, we explained a configuration that uses machine learning in the two processes of identifying the face area and identifying product identification information from the face area.However, regarding the process of identifying product identification information from the face area, image matching processing may also be used. FIG. 19 shows an example of the configuration of the information processing system 1 in this case.

In the information processing system 1 in this embodiment, the second learning processing section 202 does not need to be provided in the learning processing section 20. The recognition processing unit 21 also includes a specimen information storage unit that stores specimen information used in image matching processing.

The specimen information storage unit stores specimen information for identifying which product is displayed on the shelf of the display shelf shown in the image information. Specimen information is image information of products that may be displayed on display shelves, taken from multiple angles such as top and bottom, left and right, and diagonally. FIG. 20 shows an example of specimen information stored in the specimen information storage section. Although FIG. 20 shows the case where canned beer is photographed from various angles as sample information, it is not limited to canned beer. The specimen information storage unit stores specimen information and product identification information in association with each other.

The sample information storage unit stores information extracted from the sample information and necessary for calculating similarity, such as information on pairs of image feature amounts and their positions, along with or in place of the sample information. You may do so. The sample information also includes information necessary for calculating similarity. In this case, when the displayed product recognition processing unit 214 performs matching processing between the image information of the face area and the specimen information, which will be described later, it is not necessary to calculate the image feature amount for the specimen information every time, and the calculation time is reduced. Can be shortened.

Furthermore, the sample information stored in the sample information storage unit may be image information of the product obtained by masking the outline of the product in the first annotation data used during the learning process of the first learning processing unit 201. good. That is, when creating the first annotation data, image information of the product taken from one or more directions or its image feature amount is used as sample information. Then, among the image information of the photographed product, the inside of the closed area is masked using the outline as the external shape, and the attributes are tagged to create first annotation data. Through such processing, specimen information and first annotation data can be created together.

The product identification processing unit 2142 in this embodiment executes matching processing between the image information of the face area identified by the face identification processing unit 2141 and the specimen information stored in the specimen information storage unit, and performs matching processing between the image information of the face area identified by the face identification processing unit 2141 and the specimen information stored in the specimen information storage unit, Identify the product identification information of the product. That is, each image feature amount is calculated from the image information in the face area of a certain shelf (the face identification information of this face area is X) and each specimen information stored in the specimen information storage unit, and the characteristics are Similarity is determined by finding pairs of points. Then, the specimen information with the highest similarity is identified, and if the similarity at that time is equal to or greater than a predetermined threshold, the product identification information corresponding to the specimen information is identified based on the specimen information storage section. The identified product identification information is then used as the product identification information of the product displayed on the face of the face identification information X. Note that for faces that are determined to be not similar to any sample information, information indicating that the face identification information is "empty" (information indicating that there is no product) is attached to the face identification information. The product identification processing unit 2142 converts the identified product identification information or information indicating that it is "empty" into the photographing date and time, store identification information, image information identification information of the photographed image information, image information identification information of the normal position image information, It is stored in the product identification information storage unit 22 in association with the face identification information.

As an example, the product identification processing unit 2142 specifically executes the following processing. First, the similarity between the image information consisting of the coordinates of the face area to be processed and the specimen information stored in the specimen information storage unit is determined, and the product identification information corresponding to the specimen information with the highest similarity is determined. If the identified similarity is greater than or equal to a predetermined threshold, the product is identified as product identification information of the product displayed in the face area formed by the coordinates.

Here, in order to determine the similarity between face image information and sample information, the following processing is performed. First, in the processing before the product identification information identification processing in the product identification processing unit 2142, the image information of the face area on the shelf in the normal orientation image information and the specimen information are in the same direction (overturned or inverted). In addition, the size of each image information is approximately the same (if the size of image information differs over a predetermined range, each image information is Adjust the size so that the size is within the specified range).

In order to determine the similarity between the image information of the face region and the specimen information, the product identification processing unit 2142 uses an image of the specimen information and feature points based on the image feature amount (for example, local feature amount) of the image information of the face. Feature points based on feature amounts (for example, local feature amounts) are each extracted. Then, a pair of the most similar feature points of the image information of the face and feature points of the sample information is detected, and the difference in the coordinates of the corresponding points is determined. Then, calculate the average value of the differences. The average value of the differences indicates the overall average movement amount between the image information of the face area and the sample information. Then, the coordinate differences between all pairs of feature points are compared with the average coordinate difference, and pairs with a large degree of deviation are excluded. Then, the similarity is ranked based on the number of remaining corresponding points.

The similarity between the image information of the face area and the sample information can be calculated using the method described above. Furthermore, in order to improve the accuracy, the EMD (Earth Movers Distance) between color histograms may be determined and used as a measure of similarity. As a result, it is possible to compare similarities that are relatively strong against environmental changes such as brightness information of photographed image information, and it is possible to perform identification with high accuracy.

Alternatively, the similarity may be determined by determining the EMD between signatures (sets of image features and weights) of image information of each face region, and using this as a measure of similarity. For example, the image feature amount of the signature is determined by calculating the frequency distribution of the image information of the face area in the HSV color space, grouping it with respect to hue and saturation, and calculating the image feature amount based on the number of features and the region in the HSV color space. can do. The reason for grouping based on hue and saturation is to reduce dependence on brightness so that it is not greatly influenced by shooting conditions.

Further, in order to speed up processing, similarity such as L2 distance between image feature quantities such as a color correlogram or a color histogram of image information in an appropriate color space can be used instead of the signature and EMD.

The determination of similarity is not limited to the above. The identified product identification information is stored in the product identification information storage unit 22 in association with photographing date and time information, store information, image information identification information of photographed image information, image identification information of normal position image information, and face identification information.

For faces whose product identification information could not be identified, information indicating that the face area is "empty" (information indicating that the product is not out of stock) is stored in the product identification information storage unit 22.

As described above, even when image matching processing is used to identify a product from the image information of the face area, since the face area is not a rectangular area, the image matching process can be executed with high accuracy.

As a modification of the above-mentioned embodiments 1 and 2, a shelf comparison processing unit 2143 is provided to detect changes in each shelf, and if there is no change in each shelf, the previous recognition result is used as is. You can also do that. An example of the displayed product recognition processing section 214 in this case is shown in FIG.

The shelf comparison processing unit 2143 compares the image information of the shelf area in the previous (N-1st) normal position image information with the image information of the shelf area in the current (Nth) normal position image information. Based on this, if the similarity is high, it is determined that the product identification information or "empty" of each face on that shelf is the same. As mentioned above, this similarity determination process uses the image feature amount of the image information of the shelf area in the previous (N-1st) normal orientation image information and the current (Nth) orientation image information. The similarity may be determined based on image information of the region of the shelf, or may be determined using EMD between color histograms. Moreover, it is not limited to these. Then, instead of performing identification processing for each face in the product identification processing section 2142, the product identification processing section 2142 uses the product identification information of each face on the shelf in the N-th normal orientation image information for the N-1st identical This information is stored in the product identification information storage unit 22 as the same product identification information for each face on the shelf. As a result, it is possible to omit processing for shelves that rarely change, such as shelves where products do not move much or shelves that are managed in extremely short cycles.

The processes of the first to third embodiments described above can also be combined as appropriate. Further, each process is not limited to the order described in the specification of the present invention, and can be changed as appropriate as long as the purpose is achieved. Further, although the processing in the displayed product recognition processing unit 214 is performed on the normalized image information obtained by performing the normalization process on the photographed image information, the processing may be performed on the photographed image information. In that case, normal image information may be read as photographed image information.

Alternatively, instead of specifying the shelf area in the recognition processing unit 21 and specifying a face area, which will be described later, from there, the photographed image information, the normal position image information, or the image of the shelf area can be obtained without specifying the shelf area. It can also be configured to specify a face area, which will be described later, from the entire information. In that case, the shelf identification processing section 213 may not be provided, and the configuration may be such that the processing is not executed.

In the above-mentioned Examples 1 to 4, explanations have been made by exemplifying display shelves in convenience stores, supermarkets, etc., but the present invention is not limited thereto, and as long as some kind of product is displayed on the display shelves, It can be applied to any genre. For example, it can be applied to medicines (commodities) displayed on display shelves (medicine shelves) for displaying medicines in dispensing pharmacies. Similarly, it can also be applied to products displayed on display shelves in a warehouse.

By using the information processing system 1 of the present invention, it is possible to improve the accuracy of product identification when identifying products on display from image information.

1: Information processing system 2: Management terminal 3: Image information input terminal 20: Learning processing section 21: Recognition processing section 22: Product identification information storage section 70: Arithmetic device 71: Storage device 72: Display device 73: Input device 74: Communication device 201: First learning processing section 202: Second learning processing section 210: Image information input reception processing section 211: Image information storage section 212: Image information uprighting processing section 213: Shelf specification processing section 214: Display product recognition processing unit 2141: Face identification processing unit 2142: Product identification processing unit 2143: Shelf comparison processing unit

Claims (9)

An information processing system that identifies products shown in image information,
a first learning processing unit that performs machine learning to create a first learning model using first annotation data in which attributes are associated with data obtained by masking the inside of the closed region using the outline of the product as the external shape; and,
a recognition processing unit that identifies product identification information of a product shown in image information taken of a display shelf;
It has
The recognition processing unit is
Using the image information and the first learning model, specifying the external shape of the product shown in the image information as a face area, and identifying product identification information of the product shown in the specified face area.
An information processing system characterized by: The first learning processing unit includes:
creating the first learning model by performing machine learning by image segmentation using the first annotation data;
The information processing system according to claim 1, characterized in that: The information processing system includes:
a second learning processing unit that performs machine learning using second annotation data that associates product image data and product identification information to create a second learning model;
The recognition processing unit is
identifying product identification information of a product appearing in the face area using the image information of the identified face area and the second learning model;
The information processing system according to claim 1 or claim 2, characterized in that: The recognition processing unit is
identifying product identification information of the product appearing in the face area by performing image matching processing on the image information of the identified face area and sample information of the product stored in the sample information storage unit;
The information processing system according to claim 1 or claim 2, characterized in that: An information processing system that identifies products shown in image information,
a first learning processing unit that performs machine learning to create a first learning model using first annotation data in which attributes are associated with data obtained by masking the inside of the closed region using the outline of the product as the external shape; ,
It has
Using the first learning model and image information taken of a display shelf, the external shape of the product shown in the image information is identified as a face area, and product identification information of the product shown in the identified face area is obtained. identify,
An information processing system characterized by: An information processing system that identifies products shown in image information,
an image information input reception processing unit that accepts input of image information obtained by photographing a display shelf;
a displayed product recognition processing unit that identifies product identification information of the product in the photograph from the image information that has received the input or the image information that has been normalized;
It has
The displayed product recognition processing unit is
A first learning model created by machine learning using first annotation data in which attributes are associated with data obtained by masking the inside of the closed region using the outline of the product as the outer shape, and the input. Using the received image information or image information obtained by normalizing the image information, the external shape of the product shown is specified as a face area, and product identification information of the product shown in the specified face area is identified. do,
An information processing system characterized by: computer,
a first learning processing unit that performs machine learning to create a first learning model using first annotation data in which attributes are associated with data obtained by masking the inside of the closed region using the outline of the product as the external shape; ,
a recognition processing unit that identifies the product identification information of the product shown in the image information of the display shelf;
An information processing program that functions as a
The recognition processing unit is
Using the image information and the first learning model, specifying the external shape of the product shown in the image information as a face area, and identifying product identification information of the product shown in the specified face area.
An information processing program characterized by: computer,
a first learning processing unit that performs machine learning to create a first learning model using first annotation data in which attributes are associated with data obtained by masking the inside of the closed region using the outline of the product as the external shape; ,
An information processing program that functions as a
Using the first learning model and image information taken of a display shelf, the external shape of the product shown in the image information is identified as a face area, and product identification information of the product shown in the identified face area is obtained. identify,
An information processing program characterized by: computer,
an image information input reception processing unit that accepts input of image information obtained by photographing a display shelf;
a displayed product recognition processing unit that identifies product identification information of the product in the image from the image information that has received the input or the image information that has been normalized;
An information processing program that functions as a
The displayed product recognition processing unit is
A first learning model created by machine learning using first annotation data in which attributes are associated with data obtained by masking the inside of the closed region using the outline of the product as the outer shape, and the input. Using the received image information or image information obtained by normalizing the image information, the external shape of the product shown is specified as a face area, and product identification information of the product shown in the specified face area is identified. do,
An information processing program characterized by:

Images