JP2021096635A - Image processing system, image processing method, and program - Google Patents

Abstract

To provide a technique that accurately identifies variations of a product having various variations.SOLUTION: An image processing system 10 includes an object detection unit 110, a category specification unit 120, a product identification unit 130, and a display processing unit 140. The object detection unit 110 detects a region in which an object is present in an image. The category specification unit 120 specifies a category to which the object belongs by using a shape of the detected region. The product identification unit 130 identifies a product based on reference data on the product corresponding to the specified category and a feature quantity obtained from the region. The display processing unit 140 outputs an output image including information indicating a position of the detected region and a product identification result to a display device.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for supporting shelving allocation management in a store.

Research is being conducted on a technique for efficiently performing shelving allocation work by analyzing images of products placed on display shelves or the like using a computer.

An example of the above-mentioned technique is disclosed in, for example, Patent Document 1 below. Specifically, in Patent Document 1 below, there is a technique for narrowing down the data to be collated based on the character information of each article in the image and the position information of the character information, and improving the image processing speed in the shelving allocation management work. It is disclosed.

Japanese Unexamined Patent Publication No. 2014-044481

When recognizing a product using a computer, the product is recognized using the image feature amount of the product. Among the products handled in the store, there may be products having various variations such as different capacities (sizes). Computers need to distinguish between variations of such products and recognize individual products. However, the package design often has little change depending on the variation, and the features on the image tend to be similar. As a result, the computer may confuse the variation of the product, and the accuracy of product recognition (distinguishing) may decrease.

The present invention has been made in view of the above problems. One of the objects of the present invention is to provide a technique for accurately identifying the variations of products having various variations.

The image processing apparatus of the present invention
An object detection means that detects an area where an object exists in an image,
A category specifying means for specifying the category to which the object belongs using the detected shape of the region, and
A product identification means for identifying a product based on the reference data of the product corresponding to the specified category and the feature amount obtained from the area.
A display processing means for outputting an output image including information indicating the detected position of the region and the product identification result to the display device, and
To be equipped.

The image processing method of the present invention
The computer
Detects the area where the object exists in the image,
The category to which the object belongs is identified by using the detected shape of the region.
The product is identified based on the reference data of the product corresponding to the specified category and the feature amount obtained from the area.
An output image including information indicating the detected position of the region and the product identification result is output to the display device.
Including that.

The program of the present invention
Computer,
An object detection means that detects an area where an object exists in an image,
A category identifying means for identifying the category to which the object belongs using the detected shape of the region,
A product identification means for identifying a product based on the reference data of the product corresponding to the specified category and the feature amount obtained from the area.
A display output means for outputting an output image including information indicating the detected position of the region and the product identification result to the display device.
To function as.

According to the present invention, there is provided a technique for accurately identifying variations of products having various variations.

It is a figure which illustrates the functional structure of the image processing apparatus in 1st Embodiment. It is a block diagram which illustrates the hardware structure of an image processing apparatus. It is a flowchart which illustrates the flow of the process executed by the image processing apparatus of 1st Embodiment. It is a figure which shows an example of the image acquired by the object detection part. It is a figure which shows an example of the processing result information stored in the storage area by the processing of an object detection part. It is a figure which shows an example of the processing result information updated by the processing of the category identification part. It is a figure which shows an example of the product information including the reference data for product identification. It is a figure which shows an example of the processing result information updated by the processing of a product identification part. It is a figure which shows an example of the output image which the display processing unit of 1st Embodiment outputs to a display device. It is a figure which shows an example of the output image which the display processing unit of 1st Embodiment outputs to a display device. It is a figure which shows an example of the output image which the display processing unit of 1st Embodiment outputs to a display device. It is a figure which illustrates the function of the category identification part of 2nd Embodiment. It is a figure which shows an example of the screen which accepts the input which shows to use the mode which uses the category specified for two other objects. It is a figure which illustrates the functional structure of the image processing apparatus in 3rd Embodiment. It is a flowchart which illustrates the flow of the process executed by the image processing apparatus of 3rd Embodiment. It is a figure which shows an example of the output image which the display processing part of 2nd Embodiment outputs to a display device. It is a figure which shows an example of the information which shows the processing result of each processing part of an image processing apparatus.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings, similar components are designated by the same reference numerals, and description thereof will be omitted as appropriate. Further, unless otherwise specified, in each block diagram, each block represents a configuration of a functional unit, not a configuration of a hardware unit. Further, the direction of the arrow in the figure is for making the flow of information easy to understand, and does not limit the direction of communication (one-way communication / two-way communication) unless otherwise specified.

[First Embodiment]
<Functional configuration>
FIG. 1 is a diagram illustrating a functional configuration of the image processing device 10 according to the first embodiment. As shown in FIG. 1, the image processing device 10 of the present embodiment includes an object detection unit 110, a category identification unit 120, a product identification unit 130, and a display processing unit 140.

The object detection unit 110 acquires an image to be processed (for example, an image of a shelf on which products are displayed), and detects an area in which the object (product) exists in the image. The object detection unit 110 can detect a region in which an object exists in an image by using, for example, a detector (learning model) constructed by repeating machine learning using a plurality of teacher data. Here, the teacher data is configured by combining a learning image in which a plurality of objects are captured and label information indicating the position of the objects in the image. The detector learns parameters (features) for discriminating the boundaries of individual objects in the repetition of machine learning using such teacher data.

The category identification unit 120 identifies the category to which the object located in the region belongs by using the shape of the region detected by the object detection unit 110. Here, the "category" means a variation in capacity and quantity (number of pieces per package) when sold as one product. For example, for beverage products such as beer, there are variations in capacity (350 ml / 500 ml, etc.) and variations in quantity (single item / multi-pack). Although not limited to the beer examples given here, in the present invention, variations of such capacity and quantity can be set as "categories".

The product identification unit 130 identifies an object (product) located in the region based on the reference data for product identification and the image feature amount obtained from the region detected by the object detection unit 110. As an example, the product identification unit 130 matches the image feature amount obtained from the region detected by the object detection unit 110 with the reference data (reference image feature amount) of each product prepared for identifying the product. To identify an object (commodity) located in the area. In this case, the reference data for product identification is stored in, for example, a storage device of the image processing device 10 or an external database. Here, the product identification unit 130 narrows down the reference data of the product used in the matching process with the image feature amount obtained from the area detected by the object detection unit 110 based on the category of the object located in the area. For example, when the category "X" is specified as the category of the object located in a certain area, the product identification unit 130 selects the reference data of the product corresponding to the specified category "X" as the target data of the matching process. To do. In this example, other reference data corresponding to a category different from the specified category "X" is not used in the matching process.

The display processing unit 140 outputs an output image to the display device 20 as shown in the figure. The output image output by the display processing unit 140 includes at least information indicating the position of the region detected by the object detection unit 110 and the product identification result by the product identification unit 130.

<Action / effect>
In the present embodiment, the category is specified in advance using the shape of the detected object before the product identification process (matching process) is performed. Here, it is common that a unified (similar) design is adopted for products having variations in capacity and quantity. That is, when a product is identified using the feature amount obtained from the package design, there is a possibility that variations such as the capacity and quantity of the product may be erroneously determined. On the other hand, if the capacity and quantity are different, there will be a clear difference in the shape (size) of the appearance. Therefore, according to the configuration of the present embodiment, it is possible to accurately discriminate the product category (variation). Further, according to the configuration of the present embodiment, the number of reference data of the products used in the matching process for identifying the products can be narrowed down based on the category identification result, so that the time required for the process can be shortened. You can expect it.

<Hardware configuration example>
The image processing device 10 may be realized by hardware that realizes each functional component (eg, a hard-wired electronic circuit, etc.), or a combination of hardware and software (eg, an electronic circuit and it). It may be realized by a combination of programs to be controlled, etc.). Hereinafter, a case where the image processing device 10 is realized by a combination of hardware and software will be further described.

FIG. 2 is a block diagram illustrating a hardware configuration of the image processing device 10.

The image processing device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

The bus 1010 is a data transmission line for the processor 1020, the memory 1030, the storage device 1040, the input / output interface 1050, and the network interface 1060 to transmit and receive data to and from each other. However, the method of connecting the processors 1020 and the like to each other is not limited to the bus connection.

The processor 1020 is a processor realized by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

The storage device 1040 is an auxiliary storage device realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, a ROM (Read Only Memory), or the like. The storage device 1040 stores a program module that realizes each function of the image processing device 10 (object detection unit 110, category identification unit 120, product identification unit 130, display processing unit 140, etc.). When the processor 1020 reads each of these program modules into the memory 1030 and executes them, each function corresponding to each program module is realized.

The input / output interface 1050 is an interface for connecting the image processing device 10 and peripheral devices. Peripheral devices include, for example, input devices such as keyboards and mice, and output devices such as displays (touch panel displays) and speakers.

The network interface 1060 is an interface for connecting the image processing device 10 to the network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method of connecting the network interface 1060 to the network may be a wireless connection or a wired connection. The image processing device 10 is communicably connected to, for example, the user terminal 2 via the network interface 1060. Here, the user terminal 2 has a camera module having a function of capturing an image to be processed, and a display device 20 for displaying an output image of the display processing unit 140. The user terminal 2 is, for example, a smartphone, a tablet terminal, a mobile PC, or the like. The object detection unit 110 can acquire the image to be processed generated by the image pickup device 30 of the user terminal 2 via the network interface 1060. Further, the display processing unit 140 can transmit drawing data of the output image to the display device 20 of the user terminal 2 via the network interface 1060.

The configuration illustrated in FIG. 2 is merely an example, and the hardware configuration of the image processing device 10 is not limited to the example in FIG. For example, each function of the image processing device 10 may be provided in a user terminal 2 such as a smartphone or a tablet terminal. In this case, an application (program) that executes each function of the image processing device 10 is installed in the user terminal 2. As a result, the user terminal 2 becomes a device substantially equivalent to the image processing device 10.

<Processing flow>
FIG. 3 is a flowchart illustrating the flow of processing executed by the image processing apparatus 10 of the first embodiment.

The object detection unit 110 acquires an image showing all or a part of the products arranged in the display space such as the product display shelf (S102). The object detection unit 110 acquires, for example, an image as illustrated in FIG. FIG. 4 is a diagram showing an example of an image acquired by the object detection unit 110. FIG. 4 shows an example of an image when a part of the product display shelf is photographed. Here, when the image processing device 10 is a device having a camera function (for example, a smartphone or a tablet terminal), the object detection unit 110 activates the camera function to acquire the generated image as an image to be processed. Can be done. Further, when an external device having a camera function (for example, the user terminal 2 in the configuration of FIG. 2) provided separately from the image processing device 10 generates an image, the object detection unit 110 uses the external device via a network. You can get the image generated by.

Next, the object detection unit 110 detects a region in which the object exists in the acquired image (S104). The object detection unit 110 detects the region where each object is located, for example, by using a learning model constructed so that the region of the object can be discriminated by machine learning. As illustrated in FIG. 4, there may basically be a plurality of objects (individual products) in the image obtained by capturing the display space of the products. Therefore, in the process of S104, basically, a plurality of regions can be detected from one image. The object detection unit 110 stores information (processing result information, eg, FIG. 5) obtained as a result of analyzing an image as illustrated in FIG. 4 in a storage area such as a memory 1030 or a storage device 1040. FIG. 5 is a diagram showing an example of processing result information stored in the storage area by the processing of the object detection unit 110. The information exemplified in FIG. 5 includes information for identifying individual regions detected in the image (region ID) and information indicating the position of the region in the image coordinate system (region information). There is. In the columns of "category identification result" and "product identification result" in FIG. 5, information indicating the processing result of the category identification unit 120 and information indicating the processing result of the product identification unit 130, which will be described later, are stored, respectively.

Next, the category specifying unit 120 selects one of the regions detected in the processing of S104 as the processing target region (S106). Then, the category specifying unit 120 identifies the category to which the object located in the processing target area belongs based on the shape of the selected processing target area (S108). The category specifying unit 120 updates the processing result information stored in the storage area by using the specifying result of the category of the object located in the processing target area (example: FIG. 6). FIG. 6 is a diagram showing an example of processing result information updated by the processing of the category specifying unit 120. As illustrated in FIG. 6, the category specifying unit 120 stores information indicating the category specified as the category of the object located in the processing target area in the “category specifying result” column.

As an example, the category specifying unit 120 can specify the category of the object located in the processing target area by using the size of the processing target area in the height direction. For example, the category specifying unit 120 performs the following processing. First, the category specifying unit 120 calculates the size (number of pixels) of the processing target area in the height direction based on the area information indicating the position of the processing target area in the image. Next, the category specifying unit 120 identifies the category corresponding to the size of the processing target area in the height direction based on the information indicating the correspondence relationship between the size of the area in the height direction and the category. Here, the correspondence between the size of the area in the height direction in the image and the category is, for example, the display space in which the product (object) is arranged is actually photographed while changing the shooting position within an assumed range. It can be determined based on the result. Information indicating the correspondence between the size of the area in the height direction and the category is stored in advance in a storage area (for example, memory 1030, storage device 1040, etc.) accessible to the category identification unit 120.

As another example, the category identification unit 120 uses the ratio (aspect ratio) of the size of the processing target area in the height direction and the size of the processing target area in the width direction to classify the object located in the processing target area. Can also be specified. For example, the category specifying unit 120 calculates the size of the processing target area in the height direction and the size in the width direction, respectively, based on the area information indicating the position of the processing target area in the image. Further, the category specifying unit 120 calculates the aspect ratio using the size in the height direction and the size in the width direction. Then, the category specifying unit 120 identifies the category corresponding to the calculated aspect ratio based on the information indicating the correspondence relationship between the aspect ratio and the category. Here, the correspondence between the aspect ratio and the category can be defined by using the information of the product size (for example, the size of a 350 ml can or the packing size of a multi-pack). Information indicating the correspondence between the aspect ratio and the category is stored in advance in a storage area (for example, memory 1030, storage device 1040, etc.) accessible to the category identification unit 120.

After the category of the object located in the processing target area is specified, the product identification unit 130 extracts the image feature amount to be compared with the reference data for product identification prepared in advance from the processing target area (S110). Here, the reference data for product identification is prepared in advance in a storage area such as a memory 1030 or a storage device 1040 in a format as shown in FIG. 7, for example. FIG. 7 is a diagram showing an example of product information including reference data for product identification. The reference data is a product image or an image feature amount extracted from the product image.

Further, the product identification unit 130 selects the reference data corresponding to the category from all the reference data for product identification based on the category of the object located in the processing target area specified in the process of S108 (S112). ). For example, suppose that the category of the object located in the processing target area is specified as "500 ml" in the processing of S108. In this case, the product identification unit 130 uses the reference data illustrated in FIG. 7 as reference data (references stored in the records on the 2, 6, 10 and 14th rows) corresponding to the category of "500 ml". Data) is selected as the reference data used in the matching process.

Then, the product identification unit 130 compares the reference data of the product selected based on the category with the image feature amount obtained from the processing target area, and based on the similarity (score calculated by the matching process). An object (commodity) located in the region is identified (S114). For example, the product identification unit 130 acquires the product ID associated with the reference data having the highest degree of similarity as the product identification result. Further, the product identification unit 130 identifies reference data indicating the degree of similarity equal to or higher than a predetermined threshold value (when a plurality of data are present, the reference data having the highest degree of similarity among them), and assigns a product ID associated with the reference data. It may be configured to be acquired as a product identification result. If there is no reference data indicating the degree of similarity equal to or higher than a predetermined threshold value, the product identification unit 130 may simply acquire the product ID associated with the reference data having the highest degree of similarity, or the product may obtain the product ID. Information indicating that the identification could not be made may be generated. Then, the product identification unit 130 updates the processing result information stored in the storage area by using the product identification result (example: FIG. 8). FIG. 8 is a diagram showing an example of processing result information updated by the processing of the product identification unit 130. As illustrated in FIG. 8, the product identification unit 130 stores information indicating the category specified as the identification result (product identification) by the matching process in the “category identification result” column. In addition, "-" in this figure means information indicating that the product could not be identified.

The processing from S106 to S114 is repeated until all the regions detected in the processing of S104 are selected as the processing target regions. Specifically, when an unselected area remains as the processing target area (S116: YES), a new processing target area is selected from the unselected areas, and the above processing is repeated. When all the areas are selected (S116: NO), the display processing unit 140 generates an output image using the result of the matching process of each area, and outputs the output image to the display device 20 (S118).

<Example of output image>
Hereinafter, the output image output by the display processing unit 140 will be described with some specific examples. 9 to 11 are diagrams showing an example of an output image output to the display device 20 by the display processing unit 140 of the first embodiment.

In the example of FIG. 9, the display processing unit 140 superimposes, for example, a frame-shaped display element A1 indicating an area where an object exists on an image to be processed to generate an output image that visualizes the position where the object exists. ing. At this time, the display processing unit 140 can set the display position (position coordinates on the image) of the display element A1 corresponding to each area based on the area information stored in the storage area. In addition, the display processing unit 140 can set the display mode of the display element A1 indicating the position of each area according to the result of the matching process for each area. In the example of FIG. 9, the display processing unit 140 determines the area where the product can be identified (the area where the frame line is the solid line) and the area where the product cannot be identified (the area where the frame line is the solid line) depending on the type of the frame line (solid line / dotted line) of the display element A1. The border is the area of the dotted line). By doing so, it becomes easier for the person who confirms the output image to grasp the success or failure of the matching process. The display processing unit 140 may be able to distinguish between the area where the product can be identified and the area where the product cannot be identified, not only by the type of the border but also by the color of the border, the type of the fill pattern, and the like. ..

Further, as illustrated in FIG. 9, the display processing unit 140 may include the product list A2 indicating the product identified in the image to be processed in the output image. The display processing unit 140 can generate the product list A2 based on the result of the matching processing of each area by the product identification unit 130. In this case, the display processing unit 140 may be configured to update the output image according to the selection operation on the product list A2. For example, as illustrated in FIG. 10, it may be configured to highlight the display element A1 in the area corresponding to the product selected on the product list A2. In the example of FIG. 10, the state in which the input for selecting "C beer can 350 ml" is made on the product list A2 is shown. In this case, the display processing unit 140 specifies, for example, an area associated with the selected product from the information stored in the storage area (example: FIG. 8) based on the identification information of the product selected on the list. .. Then, the display processing unit 140 changes the color of the border of the display element A1, the thickness of the border, the type of the border, the type of the fill in the frame, and the like, so that the display corresponding to the specified area can be displayed. Highlight element A1. By doing so, the person who confirms the output image can easily grasp the position of each identified product on the image. Further, as illustrated in FIG. 10, the display processing unit 140 may further display the display element A3 indicating the information of the selected product on the image according to the selection state of the product list A2. ..

Further, the display processing unit 140 recognizes an area in which products having the same name and products belonging to the same category are continuously arranged as one continuous area, and displays the display element B1 indicating the continuous area and the product name and category. An output image including the display element B2 shown may be generated (example: FIG. 11). The details of the product arrangement in the display space can be easily grasped from the output image as illustrated in FIG.

[Second Embodiment]
The present embodiment has the same configuration as the first embodiment except for the points described below.

As shown in FIG. 12, the category specifying unit 120 of the present embodiment uses the information of the categories specified for two other objects located on both sides of the object (target object) located in the processing target area. Set the category of the target object. FIG. 12 is a diagram illustrating the function of the category specifying unit 120 of the second embodiment. In FIG. 12, the dotted line frame means the processing target area. When the category specifying unit 120 of the present embodiment specifies the category of the target object 1201 located in the processing target area, the categories of the two other objects (object 1202 and the object 1203) located on both sides of the object 1201 are specified. It is determined whether or not the two other objects are already in the same category. In the example of FIG. 12, the categories of the two other objects (object 1202 and object 1203) have been identified, and the categories are the same. In this case, the category specifying unit 120 sets the category common to the object 1202 and the object 1203 (here, “500 ml, single item”) as the category of the target object 1201 located in the processing target area. When the categories of the object 1202 and the object 1203 are not the same, the category specifying unit 120 is the target object 1201 located in the processing target area based on the shape of the processing target area as described in the first embodiment. Category can be specified.

<Action / effect>
In the present embodiment, when the category of the object located in the processing target area is specified, the categories of the two other objects located on both sides of the object have already been specified, and the categories of the two other objects are the same. In this case, the category specified for other objects is set as the category of the object located in the processing target area. For example, in the present embodiment, the category identification unit 120 performs the category identification process using the shape of the area skipping one area on the area detected by the object detection unit 110, and then performs the above-mentioned process on the remaining area. Configured to run. As a result, for at least a part of the area, the category can be specified by a simpler process than the process using the shape of the area, and the total processing time can be expected to be shortened.

In the present embodiment, the category specifying unit 120 responds to an input indicating that the mode for setting the category of the target object is used by using the categories of two other objects located on both sides of the target object. It may be configured to perform the above-mentioned processing. For example, when an image to be processed is acquired, the category specifying unit 120 causes the display device 20 to display a screen as illustrated in FIG. 13, and identifies two other objects via the screen. It is configured to accept input that specifies the area to use the mode that uses the category. FIG. 13 is a diagram showing an example of a screen that accepts setting input of a region that uses a mode that uses a category specified for two other objects. In the example of FIG. 13, the category specifying unit 120 receives an input for designating an area according to a user's input operation using an input device such as a mouse or a touch panel. The user input operation is, for example, an operation of designating an arbitrary area, an operation of selecting at least one of the partial areas (each stage of the display shelf) specified by the position of the shelf board, and the like. In the case illustrated in FIG. 13, the category specifying unit 120 accepts an input operation for designating an area corresponding to the middle stage of the product display shelf, and recognizes the area as the target area of the above mode.

According to such a configuration, the user can arbitrarily set an area to which the processing of the present embodiment is to be applied to the image to be processed. For example, the user can exclude in advance an area unsuitable for the processing of the present embodiment (such as an area in which three or more products of the same category are not continuously arranged) depending on the arrangement pattern of the products.

[Third Embodiment]
In the image recognition technology by a machine (image processing device 10), it is currently difficult to completely eliminate errors. In order to improve the accuracy of the final output, a method of visually confirming and correcting a part that is difficult to judge by a machine may be adopted, but the image of the display space targeted by the present invention may be adopted. When there are many confirmation targets, it is very troublesome to confirm each one with the human eye. The image processing device 10 of the present embodiment further includes a configuration for solving this problem.

<Functional configuration>
The image processing apparatus 10 of the present embodiment has the same configuration as that of the first embodiment except for the points described below. Further, the image processing device 10 of the present embodiment may further include the functions described in the second embodiment. FIG. 14 is a diagram illustrating the functional configuration of the image processing device 10 according to the third embodiment. As shown in FIG. 14, the image processing apparatus 10 of the present embodiment is different from the configuration of the first embodiment in that the identification error candidate extraction unit 150 is further provided.

The identification error candidate extraction unit 150 identifies an area (hereinafter, also referred to as “candidate area”) in which a product may be erroneously identified. A specific operation example of the identification error candidate extraction unit 150 will be described later. Further, the display processing unit 140 of the present embodiment displays an output image further including a display element that makes the candidate area identifiable based on the identification result of the candidate area by the identification error candidate extraction unit 150.

<Processing flow>
FIG. 15 is a flowchart illustrating a flow of processing executed by the image processing apparatus 10 of the third embodiment. The process shown in the example of FIG. 15 is executed in place of the process of S118 of FIG. 3 after the determination process of S116 of FIG. 3 becomes “NO”.

When all the regions detected by the object detection unit 110 are selected as the processing target regions (S116: NO), the identification error candidate extraction unit 150 identifies the candidate regions from the processing target regions (S202).

As an example, the identification error candidate extraction unit 150 may be configured to specify the candidate area based on the certainty of the matching process (the certainty of the product identification result) by the product identification unit 130. The certainty of the matching process can be determined, for example, based on the magnitude of the score (similarity) obtained as a result of the matching process. Specifically, the identification error candidate extraction unit 150 can determine that the higher the degree of similarity obtained as a result of the matching process, the higher the certainty of the matching process. The identification error candidate extraction unit 150 determines, for example, whether or not the result (similarity) of the matching process of each region is less than a predetermined threshold value, and determines the region in which the similarity degree of less than the predetermined threshold value is obtained as a result of the matching process. Specify as a candidate area.

As another example, the identification error candidate extraction unit 150 may be configured to specify the candidate region based on the array of products obtained from the result of the matching process by the product identification unit 130. For example, the identification error candidate extraction unit 150 can detect the irregular arrangement of the product and specify the region corresponding to the product as the candidate region. As a specific example, the identification error candidate extraction unit 150 corresponds to a specific product when it can detect from the product identification result that the number of faces (the number of rows of the product) is different. The area can be specified as a candidate area. Further, as another specific example, the identification error candidate extraction unit 150 is different from the first category in the area where products belonging to a certain first category (example: 500 ml, single item) are continuously arranged. A state in which products belonging to a different second category (eg, multi-pack) suddenly appears is detected as "irregular arrangement of products", and an area corresponding to the product belonging to the second category is specified as a candidate area. can do.

Then, the display processing unit 140 generates an output image using the result of the matching process of each area by the product identification unit 130 and the identification result of the candidate area in the processing of S202, and outputs the output image to the display device 20 ( S204).

<Example of output image>
FIG. 16 is a diagram showing an example of an output image output to the display device 20 by the display processing unit 140 of the second embodiment. In the example of FIG. 16, the display processing unit 140 makes it possible to distinguish the area where the product may be erroneously identified from the other areas by the display element B (diagonal line filling pattern). The display processing unit 140 can generate an output image using, for example, the information illustrated in FIG.

FIG. 17 is a diagram showing an example of information showing a processing result of each processing unit of the image processing apparatus 10. The information illustrated in FIG. 17 is different from FIGS. 5, 6 and 8 in that it further has a column of "candidate area flags". Here, when a certain area is specified as a candidate area, the identification error candidate extraction unit 150 indicates that the area is a candidate area in the “candidate area flag” column of the area (example of this figure). Then, "1 (candidate area)") is set. Then, the display processing unit 140 changes the display mode of the display element (example: frame-shaped display element) corresponding to each area depending on whether or not "1 (candidate area)" is set in the candidate area flag. .. Specifically, the display processing unit 140 gives a specific mark to the area specified as the candidate area, or the type of the frame line, the color of the frame line, the thickness of the frame line, and the fill pattern in the frame. Set the type and the like to a mode peculiar to the candidate area. As a result, an output image in which the candidate area can be distinguished is generated. Then, the display processing unit 140 outputs the generated output image to the display device 20.

<Action / effect>
In the present embodiment, an output image including information for discriminating an area (candidate area) in which a product may be erroneously identified is displayed on the display device 20. As a result, the person who confirms the output image can easily find the area where the image processing device 10 may make a judgment error. As a result, it is possible to efficiently take measures to improve the accuracy of the final output, such as correcting an error.

Although the embodiments of the present invention have been described above with reference to the drawings, the present invention should not be construed as being limited to these, and are based on the knowledge of those skilled in the art as long as they do not deviate from the gist of the present invention. Therefore, various changes and improvements can be made. The plurality of components disclosed in the embodiments can form various inventions by appropriate combinations. For example, some components may be deleted from all the components shown in the embodiment, or components of different embodiments may be combined as appropriate.

Further, in the plurality of flowcharts used in the above description, a plurality of steps (processes) are described in order, but the execution order of the steps executed in each embodiment is not limited to the order of description. In each embodiment, the order of the illustrated steps can be changed within a range that does not hinder the contents. In addition, the above-described embodiments can be combined as long as the contents do not conflict with each other.

Some or all of the above embodiments may also be described, but not limited to:
1. 1.
An object detection means that detects an area where an object exists in an image,
A category specifying means for specifying the category to which the object belongs using the detected shape of the region, and
A product identification means for identifying a product based on the reference data of the product corresponding to the specified category and the feature amount obtained from the area.
A display processing means for outputting an output image including information indicating the detected position of the region and the product identification result to the display device, and
An image processing device comprising.
2.
The object detection means detects a region where the object exists by using a learning model constructed by machine learning.
1. 1. The image processing apparatus according to.
3. 3.
When the category specified for each of the two other objects located adjacent to the object is the same, the category specifying means sets the category of the object to the same category as the two other objects.
1. 1. Or 2. The image processing apparatus according to.
4.
The category specifying means sets the category of the object by using the category of the two other objects in response to the input using the mode of setting the category of the object by using the category of the two other objects. Execute the process to
3. 3. The image processing apparatus according to.
5.
The category specifying means identifies the category of the object by using at least one of the height direction size of the region and the ratio of the height direction size of the region to the lateral size of the region. To do
1. 1. From 4. The image processing apparatus according to any one of the above.
6.
Further equipped with an identification error candidate extraction means for identifying a candidate area where a product may be erroneously identified,
The display processing means superimposes on the output image further including a display element that makes the candidate region identifiable.
1. 1. From 5. The image processing apparatus according to any one of the above.
7.
The identification error candidate extraction means identifies the candidate region based on the certainty of the identification result.
6. The image processing apparatus according to.
8.
The identification error candidate extraction means identifies the candidate region based on the sequence of products obtained from the identification result.
6. The image processing apparatus according to.
9.
The computer
Detects the area where the object exists in the image,
The category to which the object belongs is identified by using the detected shape of the region.
The product is identified based on the reference data of the product corresponding to the specified category and the feature amount obtained from the area.
An output image including information indicating the detected position of the region and the product identification result is output to the display device.
Image processing methods including that.
10.
The computer detects the region where the object exists by using a learning model constructed by machine learning.
Including that 9. The image processing method described in.
11.
When the computer has the same category specified for each of the two other objects located adjacent to the object, the category of the object is set to the same category as the two other objects.
Including that 9. Or 10. The image processing method described in.
12.
A process in which the computer sets the category of the object using the two other object categories in response to an input using a mode in which the computer sets the object category using the two other object categories. To execute,
Including 11. The image processing method described in.
13.
The computer identifies a category of the object using at least one of the height dimension of the region and the ratio of the height orientation of the region to the lateral magnitude of the region.
Including that 9. From 12. The image processing method according to any one of the above.
14.
The computer
Identify candidate areas that may have misidentified the product
The display processing means superimposes on the output image further including a display element that makes the candidate region identifiable.
Including that 9. To 13. The image processing method according to any one of the above.
15.
The computer identifies the candidate region based on the certainty of the identification result.
Including 14. The image processing method described in.
16.
The computer identifies the candidate region based on the sequence of goods obtained from the identification result.
Including 14. The image processing method described in.
17.
Computer,
An object detection means that detects an area where an object exists in an image,
A category identifying means for identifying the category to which the object belongs using the detected shape of the region,
A product identification means for identifying a product based on the reference data of the product corresponding to the specified category and the feature amount obtained from the area.
A display output means for outputting an output image including information indicating the detected position of the region and the product identification result to the display device.
A program to function as.
18.
The object detection means detects a region where the object exists by using a learning model constructed by machine learning.
17. The program described in.
19.
When the category specified for each of the two other objects located adjacent to the object is the same, the category specifying means sets the category of the object to the same category as the two other objects.
17. Or 18. The program described in.
20.
The category specifying means sets the category of the object by using the category of the two other objects in response to the input using the mode of setting the category of the object by using the category of the two other objects. Execute the process to
19. The program described in.
21.
The category specifying means identifies the category of the object by using at least one of the height direction size of the region and the ratio of the height direction size of the region to the lateral size of the region. To do
17. From 20. The program described in any one of.
22.
The computer
Further function as an identification error candidate extraction means to identify a candidate area where a product may be erroneously identified.
The display processing means superimposes on the output image further including a display element that makes the candidate region identifiable.
17. From 21. The program described in any one of.
23.
The identification error candidate extraction means identifies the candidate region based on the certainty of the identification result.
22. The program described in.
24.
The identification error candidate extraction means identifies the candidate region based on the sequence of products obtained from the identification result.
22. The program described in.

10 Image processing device 1010 Bus 1020 Processor 1030 Memory 1040 Storage device 1050 Input / output interface 1060 Network interface 110 Object detection unit 120 Category identification unit 130 Product identification unit 140 Display processing unit 150 Identification error candidate extraction unit 2 User terminal 20 Display device

Claims (10)

An object detection means that detects an area where an object exists in an image,
A category specifying means for specifying the category to which the object belongs using the detected shape of the region, and
A product identification means for identifying a product based on the reference data of the product corresponding to the specified category and the feature amount obtained from the area.
A display processing means for outputting an output image including information indicating the detected position of the region and the product identification result to the display device, and
An image processing device comprising. The object detection means detects a region where the object exists by using a learning model constructed by machine learning.
The image processing apparatus according to claim 1. When the category specified for each of the two other objects located adjacent to the object is the same, the category specifying means sets the category of the object to the same category as the two other objects.
The image processing apparatus according to claim 1 or 2. The category specifying means sets the category of the object by using the category of the two other objects in response to the input using the mode of setting the category of the object by using the category of the two other objects. Execute the process to
The image processing apparatus according to claim 3. The category specifying means identifies the category of the object by using at least one of the height direction size of the region and the ratio of the height direction size of the region to the lateral size of the region. To do
The image processing apparatus according to any one of claims 1 to 4. Further equipped with an identification error candidate extraction means for identifying a candidate area where a product may be erroneously identified,
The display processing means superimposes on the output image further including a display element that makes the candidate region identifiable.
The image processing apparatus according to any one of claims 1 to 5. The identification error candidate extraction means identifies the candidate region based on the certainty of the identification result.
The image processing apparatus according to claim 6. The identification error candidate extraction means identifies the candidate region based on the sequence of products obtained from the identification result.
The image processing apparatus according to claim 6. The computer
Detects the area where the object exists in the image,
The category to which the object belongs is identified by using the detected shape of the region.
The product is identified based on the reference data of the product corresponding to the specified category and the feature amount obtained from the area.
An output image including information indicating the detected position of the region and the product identification result is output to the display device.
Image processing methods including that. Computer,
An object detection means that detects an area where an object exists in an image,
A category identifying means for identifying the category to which the object belongs using the detected shape of the region,
A product identification means for identifying a product based on the reference data of the product corresponding to the specified category and the feature amount obtained from the area.
A display output means for outputting an output image including information indicating the detected position of the region and the product identification result to the display device.
A program to function as.

Images