JP2023161432A - Image processing device and image processing method - Google Patents

Abstract

To create a learning model that enables a user to easily and visually confirm a collection state of teacher images prior to learning, and is efficient and high in accuracy.SOLUTION: The processor 13 is configured to: generate teacher images including a detection object and a background from an area image associated with a monitor area; set, for each of teacher images, attributes of the detection object included in the teacher image; generate visualized image including visualized gathering states of teacher images at respective positions in the area image targeting a teacher image having attributes that a user specifies; and output display information generated by superposing the visualized images on the area image. The processor further generates, as an area image, a teacher image from a reality area image photographed by a camera or a virtual area image generated through CG. The processor further generates a visualized image including gathering state of teacher images by color kinds associated with a person as an attribute.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing device and an image processing method that visualize the collection status of teacher images for constructing an image recognition model corresponding to a monitoring area.

Systems are in use that use image recognition models (machine learning models) built through machine learning such as deep learning to detect predetermined events, such as a person visiting a store, from images captured by a camera. An image recognition model is constructed by machine learning using a large number of collected teacher images (learning images), but if the teacher images are biased, an image recognition model with stable accuracy cannot be constructed.

In order to avoid a decrease in the accuracy of the image recognition model due to such bias in the teacher image, conventionally, the probability of the entity of a person or store (component) existing in the monitoring area (application environment) that is the processing target of the image recognition model has been calculated. A technique is known that reduces bias in learning data (teacher images) by changing the distribution (see Patent Document 1).

JP 2021-111101 Publication

According to conventional technology, the set of teacher images (learning dataset) is updated to reduce bias in the teacher images, which increases the possibility of constructing a highly accurate machine learning model. In some cases, the accuracy of the learning model is insufficient. Therefore, when evaluating the constructed machine learning model, if it is determined that the accuracy of the machine learning model is insufficient, the set of teacher images is updated by adding missing teacher images, and then the machine learning The steps are repeated and the constructed machine learning model is evaluated. In this way, with conventional technology, it is necessary to repeatedly update the set of teacher images, perform machine learning, and evaluate the machine learning model, which takes a lot of effort to complete a machine learning model with sufficient accuracy. There are cases.

On the other hand, when the teacher image collection status (annotation status), that is, whether or not there are a sufficient number of teacher images with necessary attributes and an appropriate distribution, is visualized and presented to the user, the user You can instantly grasp the image collection status and efficiently perform annotation work to add missing teacher images.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide an image processing device and an image processing method that allow a user to easily visually check the collection status of teacher images prior to learning, and that can efficiently create a highly accurate learning model. Main purpose.

An image processing device of the present invention is an image processing device in which a processor executes a process of visualizing a collection status of teacher images for constructing an image recognition model corresponding to a monitoring area, the processor A teacher image including a detection target object and a background is generated from an area image, attributes related to the characteristics of the detection target included in the teacher image are set for each teacher image, and a teacher image having the attributes specified by the user is set. The present invention is configured to generate a visualized image that visualizes the collection status of the teacher image at each position of the area image with the teacher image as a target, and output display information in which the visualized image is superimposed on the area image.

Further, the image processing method of the present invention is an image processing method in which a processor executes a process of visualizing the collection status of teacher images for constructing an image recognition model corresponding to a monitoring area, the method comprising: , a teacher image including a detection target and a background is generated, attributes related to the characteristics of the detection target included in the teacher image are set for each teacher image, and the teacher image has the attributes specified by the user. The present invention is configured to generate a visualized image that visualizes the acquisition status of the teacher images at each position of the area image, and to output display information in which the visualized image is superimposed on the area image.

According to the present invention, the user can easily check the collection status of teacher images having the attributes specified by the user, that is, whether the teacher images having the necessary attributes are evenly collected. In particular, the user can easily grasp the position on the area image where there is a problem in the collection status of teacher images. Thereby, the user can easily visually check the annotation status of the teacher image prior to learning, and can efficiently create a highly accurate learning model.

Overall configuration diagram of the image recognition model construction system according to this embodiment Block diagram showing a schematic configuration of an image processing device Explanatory diagram showing the detection area and pre-detection area set on the area image in the case of a people counting system Explanatory diagram showing an overview of teacher image generation processing Explanatory diagram showing information about area images registered in the database Explanatory diagram showing a person rectangle Explanatory diagram showing teacher images Explanatory diagram showing information about teacher images registered in the database An explanatory diagram showing deficiencies in the annotation status regarding the extraction position of the teacher image on the area image Explanatory diagram showing the extraction number measurement process Explanatory diagram showing the extraction number measurement process when people overlap Explanatory diagram showing the screen of annotation work mode Explanatory diagram showing the screen when selecting a list in annotation status confirmation mode Explanatory diagram showing the screen when selecting a graph in annotation status confirmation mode Explanatory diagram showing the screen when selecting a graph in annotation status confirmation mode Explanatory diagram showing the screen of annotation status detailed confirmation mode Explanatory diagram showing the screen of annotation status detailed confirmation mode Explanatory diagram showing the screen of annotation status detailed confirmation mode Explanatory diagram showing the screen of annotation status detailed confirmation mode Explanatory diagram showing another example of the screen in the annotation status detailed confirmation mode

A first invention made to solve the above problem is an image processing device in which a processor executes a process of visualizing the collection status of teacher images for constructing an image recognition model corresponding to a monitoring area, The processor generates a teacher image including a detection target object and a background from the area image related to the monitoring area, sets attributes related to the characteristics of the detection target included in the teacher image for each teacher image, and Targeting the teacher image having the specified attribute, generate a visualized image that visualizes the collection status of the teacher image at each position of the area image, and output display information in which the visualized image is superimposed on the area image. The configuration is as follows.

According to this, the user can easily check the collection status of the teacher images having the attributes specified by the user, that is, whether the teacher images having the necessary attributes are evenly collected. In particular, the user can easily grasp the position on the area image where there is a problem in the collection status of teacher images. Thereby, the user can easily visually check the annotation status of the teacher image prior to learning, and can efficiently create a highly accurate learning model.

In a second aspect of the invention, the processor generates, as the visualized image, a heat map image representing a collection status of the teacher images at each position of the area image.

According to this, the user can easily grasp the collection status of teacher images at each position of the area image.

Further, in a third invention, the processor is configured to generate, as the visualized image, a mark image representing a range on the area image where there is a problem in the collection status of the teacher image.

According to this, the user can easily grasp the area on the area image where there is a problem in the collection status of the teacher images.

Further, in a fourth invention, the processor generates the teacher image from a real area image photographed by a camera or a virtual area image created by CG as the area image.

According to this, a teacher image can be efficiently generated.

Further, in a fifth invention, the processor is configured to generate the visualized image that visualizes the collection status of the teacher images for each color type regarding the person as the attribute.

According to this, the accuracy of image recognition models may vary greatly depending on the color type of people, so by presenting the collection status of teacher images for each color type of people to the user, it is possible to easily achieve high-precision image recognition. Models (machine learning models) can be created.

Further, in a sixth invention, the processor generates a statistical graph that visualizes the collection status of the teacher images for each attribute, and outputs the display information including this statistical graph.

According to this, the user can easily understand the collection status of teacher images for each attribute. In this case, a three-dimensional statistical graph may be generated that visualizes the collection status of teacher images for each combination of multiple attributes.

Further, in a seventh aspect of the invention, the processor outputs the display information including a first screen for generating the teacher image and setting attributes for the teacher image in response to a user's operation, and The display information is configured to display the visualized image superimposed thereon and output the display information including a second screen provided with an operation section for returning to the first screen.

According to this, if there is a problem with the collection status of teacher images, you can immediately proceed to the task of adding missing teacher images on the first screen that generates a teacher image and sets attributes for the teacher image. I can do it.

Further, an eighth invention is an image processing method in which a processor executes a process of visualizing the collection status of teacher images for constructing an image recognition model corresponding to a monitoring area, the method comprising: A teacher image including a detection target object and a background is generated, attributes related to the characteristics of the detection target included in the teacher image are set for each teacher image, and the teacher image having the attributes specified by the user is targeted. Then, a visualized image that visualizes the acquisition status of the teacher images at each position of the area image is generated, and display information in which the visualized image is superimposed on the area image is output.

According to this, as in the first invention, the user can easily visually check the annotation status of the teacher image prior to learning, and can efficiently create a highly accurate learning model.

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is an overall configuration diagram of an image recognition model construction system according to this embodiment.

This system includes an image processing device 1 (information processing device), a camera 2, and a recorder 3.

Camera 2 photographs the monitoring area. The recorder 3 stores images taken by the camera 2. Photographed images stored in the recorder 3 are input to the image processing device 1 .

The image processing device 1 is composed of a PC or the like. The image processing device 1 is connected to a display 4 and an input device 5 such as a keyboard and a mouse. Note that a touch panel display in which the display 4 and the input device 5 are integrated may be used.

The image processing device 1 constructs an image recognition model (machine learning model) for detecting a predetermined event from images captured by the camera 2 using machine learning such as deep learning. The image processing device 1 also generates a teacher image (learning image) used for learning to construct an image recognition model. The image processing device 1 also evaluates the performance of the machine learning model using an evaluation image different from the teacher image.

Furthermore, prior to learning, the image processing device 1 visualizes the collection status (annotation status) of teacher images, that is, whether or not there are a sufficient number of teacher images with the necessary attributes and in an appropriate distribution, so that the user can to be presented.

Note that in this embodiment, the image processing device 1 performs a learning process of generating a teacher image and constructing an image recognition model (machine learning model) using the teacher image, but this is different from the image processing device 1. A learning process may be performed in the device.

Next, a schematic configuration of the image processing device 1 will be described. FIG. 2 is a block diagram showing a schematic configuration of the image processing device 1. As shown in FIG.

The image processing device 1 includes a communication section 11, a storage section 12, and a processor 13.

The communication unit 11 communicates with the recorder 3.

The storage unit 12 stores programs executed by the processor 13 and the like. It also stores registration information of a database that manages the teacher images generated by the processor 13 and their attributes.

The processor 13 performs various processes by executing programs stored in the storage unit 12. In this embodiment, the processor 13 performs teacher image generation processing, extraction number measurement processing, visualization processing, output processing, learning processing, and the like.

In the teacher image generation process (annotation process), the processor 13 generates a teacher image (learning image) used for learning to construct an image recognition model. Further, in the teacher image generation process, the processor 13 sets attributes related to the characteristics of the detection target and its background included in the teacher image for each teacher image in accordance with the user's input operation.

In the extraction number measurement process, the processor 13 measures the number of teacher images to be extracted at each position on the area image for each attribute of the teacher image, for example, the attribute of the person included in the teacher image (for example, the color of the person's clothing). .

In the visualization process, the processor 13 visualizes the collection status (annotation status) of teacher images. In the visualization process, a visualized image is generated that visualizes the collection status of teacher images at each position of the area image, targeting teacher images having attributes specified by the user. Specifically, a heat map image representing the teacher image collection status at each position of the area image and a frame image (mark image) representing a range on the area image where there is a problem with the teacher image collection status are generated.

In the output process, the processor 13 displays an annotation work mode screen (see FIG. 12), an annotation status confirmation mode screen (see FIGS. 13 to 15), an annotation status confirmation mode screen (see FIGS. 16 to 19), etc. Output to display 4.

In the learning process, the processor 13 uses machine learning to construct an image recognition model (machine learning model) that detects a predetermined event from images taken by the camera 2. In the learning process, a teacher image generated in the teacher image generation process is used.

Next, an event detection system using an image recognition model will be described. FIG. 3 is an explanatory diagram showing a detection area and a pre-detection area set on an area image in the case of the people counting system.

In this embodiment, an image recognition model (machine learning model) used in a people counting system that counts the number of people passing through a monitoring area is constructed. Specifically, in order to measure the number of people visiting the store (number of customers visiting the store), an image recognition model is used to identify people as a target event from an area image taken by camera 2 of the entrance of the store as a monitoring area. Detect store visits.

In this case, the object to be detected is a person (customer) visiting the store. Further, a detection area and a pre-detection area are set in advance in the area image. The detection area is set at the location of the store entrance in the area image. The pre-detection area is an area through which a person passes before entering the detection area, and is set at a position in a passage adjacent to the detection area.

When the person (detection target) moves from the pre-detection area to the detection area, the image recognition model determines that the person has visited the store, and adds one person to the measurement result (number of customers visiting the store). At this time, based on the position of the representative point of the person, it is determined that the person has passed through the pre-detection area and that the person has entered the detection area. Note that the representative point is the center point of the person's rectangle or the center point of the feet.

The detection area and the pre-detection area are set as polygons on the area image. The coordinates of the vertices of the polygon are registered as information regarding the positions of the detection area and the pre-detection area set on the area image.

In this embodiment, an image recognition model used in a people counting system that measures the number of people passing through a surveillance area will be described, but the image recognition model may also be used in an event detection system that detects various events. .

For example, it may be an image recognition model used in an intrusion detection system that detects that a person (detection target) has entered a prohibited area. In this case, in the area image taken by camera 2 of the monitoring area including the no-entry area, a detection area is set at the position of the no-entry area, and the person is prohibited from entering as the person moves from the pre-detection area to the detection area. It is determined that the person has invaded the area.

Alternatively, it may be an image recognition model used in an abandonment detection system that detects whether a person has left luggage behind. In this case, for example, in an area image taken by camera 2 of a surveillance area that includes a prohibited area such as an emergency entrance (fire brigade entrance), a detection area is set at the position of the prohibited area, and a detection area is set at the position of the prohibited area, such as an emergency entrance (fire brigade entrance). When the person moves from the pre-detection area to the detection area and leaves the detection area while leaving the package in the detection area, it is determined that the person has left the package behind.

Alternatively, the image recognition model may be used in a stay detection system that detects that a person has stayed in a specific place for a long time. In this case, for example, if the subject is related to a customer waiting at the checkout at a retail store, a detection area is set at the position of the checkout waiting area in the area image taken by camera 2 of the monitoring area including the checkout waiting area, and a person is detected. If the person stays in the detection area for a predetermined time or longer after moving from the front area to the detection area, it is determined that the person has stayed in the cashier waiting area for a long time.

Next, an overview of the teacher image generation process performed by the image processing device 1 will be explained. FIG. 4 is an explanatory diagram showing an overview of the teacher image generation process.

The image processing device 1 performs a process of generating a teacher image (learning image) used in machine learning for constructing an image recognition model (machine learning model) (teacher image generation process).

Here, as shown in FIG. 4(A), if a person (detection target object) is included in the live-action area image (real-area image) captured by the camera 2 of the target monitoring area, the person in the live-shot area image is A teacher image is generated by cutting out the area containing the .

Furthermore, as shown in FIG. 4(B), a composite area image including a person is created by superimposing a person image (a live-action person image or a CG person image) on an area image (a live-action area image or a CG area image). A teacher image is generated by cutting out a region including a person from the composite area image. Here, the real area image (actual area image) is an image captured by the camera 2 of the monitoring area. The CG area image (virtual area image) is an image that simulates a real area image using CG (Computer Graphics). A real person image (real person image) is an image generated by cutting out a person area from an image photographed by the camera 2 or the like. A CG person image (virtual person image) is an image created using CG.

Further, the teacher image is created for each frame from the area image. Therefore, as shown in FIG. 4(A), when a teacher image is generated from a live-action area image including a person, each frame on the live-action area image is Since the person moves, the cutting position of the teacher image may be gradually changed in accordance with the movement of the person. Furthermore, as shown in FIG. 4(B), when a teacher image is generated from a composite area image in which a person image is superimposed on an area image, the area is The teacher image may be cut out frame by frame from the area image while moving the person image on the image.

Furthermore, when a teacher image is generated from a composite area image in which a person image (a live-action person image, a CG person image) is superimposed on an area image (a live-action area image, a CG area image), the person in the area image appears. A person image is superimposed on a possible position.

Note that even if the target monitoring area is the same, if images are taken from different directions with different cameras 2, the orientation of the person (detected object) will be different, so a separate teacher image is prepared and a different image recognition model (mechanical learning model) is constructed.

Next, information regarding area images will be explained. FIG. 5 is an explanatory diagram showing information regarding area images registered in the database.

The area image (actual background image, CG background image) is an image representing a monitoring area. The area image includes structures such as pillars, walls, and shutters that exist in the monitoring area. Furthermore, the area image includes people staying in the monitoring area.

The image processing device 1 registers and manages information regarding area images in a database. Information related to area images registered in the database includes structure information and person information. The structure information is information regarding structures included in the area image. The person information is information regarding a person included in the area image as a background.

The structure information includes information regarding the type of structure and information regarding the attributes of the structure. Information regarding the type of structure is information indicating whether the structure is a fixed structure or a mobile structure. For example, pillars and walls are fixed structures, and shutters installed at store entrances are movable structures. The information regarding the attributes of the structure is, for example, the opening/closing time of a shutter as a moving structure.

The person information includes information regarding the color type of the person's clothing (the color of the upper body clothing, the color of the lower body clothing), and information regarding the presence or absence of the person's belongings. Note that belongings include not only luggage but also objects that the person moves, such as strollers and trolleys.

Next, a person rectangle set by the image processing device 1 will be explained. FIG. 6 is an explanatory diagram showing a person rectangle. Note that when a teacher image is cut out from a composite area image on which a person image (a real person image, a CG person image) is superimposed, the person area corresponds to the person image.

As shown in FIG. 6A, in this embodiment, a person rectangle surrounding a person as a detection target is set, and the height H and width W of the person rectangle are registered in the database as information regarding the person rectangle. . The example shown in FIG. 6(B) is a case where a person as a detection target is moving a stroller. In this case as well, the person rectangle is set so as to surround only the person.

Further, as shown in FIG. 6C, in the case of a CG person image, the coordinates of points (contour points) forming the contour are registered in the database as information regarding the contour of the person.

In addition, as shown in FIGS. 6(D) and (E), information regarding the position of the person includes the coordinates of the reference point (the upper left point of the person's rectangle), the coordinates of the center point (the center point of the person's rectangle), The coordinates of the center point of the feet (the intersection of the perpendicular line passing through the center point of the person's rectangle and the base of the person's rectangle) are registered in the database.

Next, the teacher image generated by the image processing device 1 will be explained. FIG. 7 is an explanatory diagram showing a teacher image.

The teacher image is generated by cutting out a region including a person from an area image including the person. The teacher image includes a person area representing a person as a detection target and a background area serving as the background of the person. The background area includes architectural structures such as columns and floors.

As shown in FIGS. 7(A-1) and (A-2), the teacher image is cut out from the area image based on the person rectangle surrounding the person as the detection target. That is, the teacher image is created by cutting out from the area image a rectangular range that is expanded by a predetermined width around the person rectangle. Specifically, in the teacher image, the area of the person rectangle is enlarged horizontally by a predetermined horizontal enlargement width α, and the area of the person rectangle is enlarged vertically by a predetermined vertical enlargement width β. It has a certain quality. The teacher image includes a person area included in the person rectangle, a background area included in the person rectangle, and a background area around the person rectangle. Note that the enlargement widths α and β may be appropriately set, for example, in the range of 0 to 10 pixels.

The example shown in FIGS. 7(B-1) and (B-2) is a case where a person as a detection target is moving a stroller. In this case as well, a teacher image is created by cutting out from the area image a rectangular range that is expanded by a predetermined width around a person rectangle that surrounds only the person.

Furthermore, in the monitoring area during actual operation, a situation may occur in which a plurality of people overlap one another when viewed from the camera 2 (overlapping people). In order to ensure the performance of image recognition models (machine learning models) even in such situations, we have developed a machine that builds image recognition models (machine learning models) using teacher images with overlapping people. Learning takes place.

In addition, when overlapping people occur, as shown in Figures 7 (C-1) and (C-2), another person appears behind the person to be detected, and in Figure 7 (D- As shown in 1) and (D-2), there is a state in which another person appears in front of the person to be detected. In this case, if a teacher image is extracted with another person appearing behind the person to be detected, the teacher image includes a person area as a background. Further, when a teacher image is extracted with another person appearing in front of the person to be detected, the teacher image includes a person area as the foreground.

Note that the teacher image in which persons overlap is generated by extracting the teacher image from a live-action area image in which persons overlap. Additionally, by generating a composite area image in which a person image (real-life person image, CG person image) is superimposed on an area image that includes a person so as to overlap the person included in the area image, overlapping of people can also be prevented. A teacher image is generated in a state where the Furthermore, by generating a composite area image in which multiple person images (real-life person images, CG person images) are superimposed on an area image, a teacher image with overlapping people can be generated. .

Further, when a teacher image is generated from an area image, information regarding the position of the teacher image on the area image (coordinates) and information regarding the size of the teacher image (height, width) are registered in the database.

Next, information regarding teacher images managed by the image processing device 1 will be explained. FIG. 8 is an explanatory diagram showing information regarding teacher images registered in the database.

The image processing device 1 registers and manages information regarding teacher images in a database. Information regarding teacher images registered in the database includes an image number (image identification information), attribute information, and image information.

The attribute information includes information regarding the clothing of the person included in the teacher image (color of upper body clothing, color of lower body clothing), information regarding presence/absence of belongings of the person, and information regarding concealment. In addition to this, the attribute information may include the person's gender, height, body shape, and the like. Note that belongings include not only luggage but also objects that the person moves, such as strollers and trolleys.

Information regarding concealment includes information regarding overlapping of persons and information regarding concealment by objects other than persons. Information regarding the overlap of people indicates whether or not another person exists in the background of the person serving as the detection target, or another person exists in the foreground of the person serving as the detection target. . The information regarding concealment by an object other than a person indicates whether or not a state in which a person as a detection target is partially hidden by an object other than a person has occurred.

The image information includes information about the person rectangle surrounding the person included in the teacher image (height H, width W), information about the contour of the person in the case of a CG person image (coordinates of points making up the contour), and the person Information regarding the location of the area is included.

Information regarding the position of the person area includes the coordinates of the reference point (the upper left point of the person rectangle), the coordinates of the center point (center point of the person rectangle), and the center point of the feet (perpendicular line passing through the center point of the person rectangle). The coordinates of the intersection point with the base of the person rectangle are included. Note that the center point of the feet is used to determine whether a person has entered the detection area and the pre-detection area. Further, the coordinates of each point are coordinates on the area image.

Next, we will explain the procedure for improving deficiencies in the annotation status. FIG. 9 is an explanatory diagram showing deficiencies in the annotation status regarding the extraction position of the teacher image on the area image.

In this embodiment, a teacher image is generated by cutting out a region including a person from an area image including the person. On the other hand, the teacher image includes a person area and a background area, and the person recognition accuracy in the image recognition model (machine learning model) changes depending on the difference in characteristics between the person area and the background area in the teacher image. . That is, even if people have similar characteristics, if the position on the area image from which the teacher image is extracted differs, the recognition accuracy of the person will change. Further, even if the position on the area image from which the teacher image is extracted is the same, if the characteristics of the person, for example, the color of the person's clothing, are different, the recognition accuracy of the person will change. Therefore, if there is a bias in the position where the teacher image is extracted from the area image for each person's characteristics (for example, the color of the person's clothing), it is not possible to construct an image recognition model (machine learning model) with stable accuracy.

Therefore, in the present embodiment, the image processing device 1 measures the number of extracted teacher images at each position on the area image for each attribute of the person included in the teacher image (extracted number measurement process). Next, for each attribute of the person included in the teacher image, the number of teacher images extracted at each position on the area image is compared, and teacher images are evenly extracted from areas in the area image where a person may appear. The user is presented with information representing the annotation status regarding whether or not the annotation is being performed. Specifically, when a position where the number of extracted teacher images is significantly smaller than other positions in the area image is detected, the position where the number of extracted teacher images is significantly smaller is presented to the user.

In response to this, the user performs an annotation work to add teacher images to positions where the number of extracted teacher images is significantly small. As a result, missing teacher images are replenished, and deficiencies in the annotation situation where the teacher images are positionally biased are corrected, and the image recognition model ( It is possible to avoid problems in which the recognition accuracy of machine learning models (machine learning models) changes significantly.

In the example shown in FIG. 9, the number of extracted teacher images at each position on the area image is measured, focusing on the color of the person's clothing as an attribute of the person included in the teacher image. In this example, attention is focused on four colors: yellow, light blue, green, and red. In addition, in Figure 9, the analysis results show areas where there are few teacher images for the specified attribute (the color of the person's clothing is yellow), areas where there are few teacher images without limiting the attribute, and areas where there are sufficient teacher images for all attributes. area is shown on the area image.

Note that the number of teacher images to be extracted may be measured by focusing on other colors such as black and white, or the color system may be changed as necessary. Furthermore, the number of extracted teacher images may be measured by dividing the color of a person's clothing into warm colors, cool colors, and black and white.

In addition, in this example, we are focusing on the color of the person's entire body, and the color of the person's clothes is the same color for the upper and lower body, but the color of the person's clothes is different for the upper and lower body. In this case, the number of extracted teacher images may be measured by paying attention to the combination of the color of the upper body and the color of the lower body.

Next, a process for measuring the number of extracted images performed by the image processing device 1 will be explained. FIG. 10 is an explanatory diagram showing the extraction number measurement process.

In the extraction number measurement process, first, an area attribute value is assigned to each position (pixel) on the teacher image, and then the area attribute value allocated to each position (pixel) on the teacher image is applied to the corresponding area image. Processing (mapping processing) of allocating to positions (pixels) is performed. Specifically, for example, a region attribute value of "1" is assigned to the person region, and a region attribute value of "0" is assigned to the background region. The mapping process is performed on all target teacher images.

Next, at each position (pixel) on the area image, a process (counting process) is performed to count the number of times each area attribute value (for example, 1, 0) is assigned. The count value thus obtained represents the number of extracted teacher images for each attribute at each position (pixel) on the area image.

Here, the number of extracted teacher images at each position (pixel) on the area image can be measured by focusing on teacher images with a specific attribute. Specifically, the number of times an area attribute value of "1" is assigned to each position (pixel) on the area image for a teacher image in which the person's clothes are in a color of interest (for example, yellow) as the attribute of interest. count. This count value represents the number of extracted teacher images in which the color of a person's clothing is the color of interest (for example, yellow). Further, by performing this process similarly for each color of the person's clothing, it is possible to obtain the number of extracted teacher images for each color of the person's clothing. This makes it possible to visualize the position on the area image for which the number of teacher images extracted is small for each color of a person's clothing.

Furthermore, the number of times an area attribute value of "0" is assigned to each position (pixel) on the area image is counted. This count value represents the number of teacher images extracted at each position (pixel) on the area image. Thereby, it is possible to visualize the position on the area image where the number of extracted teacher images is small.

Next, a process for measuring the number of extracted images when overlapping people occur will be described. FIG. 11 is an explanatory diagram showing a process for measuring the number of extracted images when overlapping people occur.

When overlapping people occur, different area attribute values are assigned to the background person area and the foreground person area in a mapping process that assigns area attribute values to each position (pixel) on the teacher image. Specifically, for example, as shown in FIG. 11A, if there is a person serving as the background, an area attribute value of "2" is assigned to the person area serving as the background. As shown in FIG. 11B, if there is a person in the foreground, an area attribute value of "3" is assigned to the foreground person area. Note that this is similar to the example shown in FIG. 10 in that a region attribute value of "1" is assigned to a person region, and a region attribute value of "0" is assigned to a background region that does not include a person.

In addition, in the counting process that counts the number of times an area attribute value is allocated, the number of times each area attribute value (for example, 1, 0, 2, 3) is allocated at each position (pixel) on the area image is counted. do.

Here, as a teacher image of the attribute of interest, the number of extracted teacher images at each position of the area image is measured regarding a teacher image including overlapping people. That is, the number of times that an area attribute value of "2" or "3" is assigned to each position (pixel) on the area image is counted. This count value represents the number of extracted teacher images that include overlapping people. This makes it possible to visualize positions on area images where a small number of extracted teacher images including overlapping people are extracted.

When people overlap in this way, especially when another person appears in front of the person to be detected, the person to be detected will be partially hidden by the other person. . On the other hand, an object other than a person may exist in front of the person serving as the detection target. In this case, the person as the detection target is hidden by an object other than the person. In order to ensure the performance of the image recognition model (machine learning model) even in such situations, we use a teacher image in which occlusion occurs, that is, a teacher image in which the person to be detected is hidden by an object in the foreground. It is preferable to perform machine learning to construct an image recognition model (machine learning model) using .

In this case, a live-action area image in which concealment has occurred, a composite area image in which an object image (live-action object image, CG object image) is superimposed on an area image including a person, or a composite area image in which an object image (live-action object image, CG object image) is superimposed on an area image containing a person, or a composite area image in which an object image (live-action object image, CG object image) is superimposed on an area image containing a person, or a person image (live-action person image, CG person image) ), it is possible to obtain a teacher image in which concealment has occurred by using a composite area image in which an object image is superimposed on the object image.

Next, the screen displayed on the display 4 will be explained. FIG. 12 is an explanatory diagram showing a screen in the annotation work mode. FIG. 13 is an explanatory diagram showing a screen when selecting a list in the annotation status confirmation mode. FIGS. 14 and 15 are explanatory diagrams showing screens when selecting a graph in the annotation status confirmation mode. 16, FIG. 17, FIG. 18, and FIG. 19 are explanatory diagrams showing screens in the annotation status detailed confirmation mode.

As shown in FIGS. 12 to 19, the screens 21, 61, 71, and 81 displayed on the display 4 are provided with tabs 22 (operation sections) for annotation work, annotation status confirmation, and annotation status detailed confirmation. ing. When the user operates the annotation work tab 22, an annotation work mode screen shown in FIG. 12 is displayed. When the user operates the annotation status confirmation tab 22, the screen transitions to an annotation status confirmation mode screen shown in FIGS. 13 to 15. When the user operates the annotation status detailed confirmation tab 22, the screen transitions to an annotation status detailed confirmation mode screen shown in FIGS. 16 to 19.

An image input section 31 is provided on the screen 21 (first screen) in the annotation work mode (teacher image creation mode) shown in FIG. The image input section 31 is provided with tabs 32 for CG and real photography. When the user operates the CG tab 32, a CG image input mode is entered. When the user operates the live-action image tab 32, a real-image input mode is entered. Further, the image input section 31 is provided with a person image input section 33 and an area image input section 34.

In the person image input section 33, when the user operates the input button 35, a list of person images is displayed, and by selecting a person image here, the user can input a person image. At this time, a CG person image is input in the CG image input mode, and a real person image is input in the real image input mode.

In the area image input unit 34, when the user operates the input button 36, a list of area images is displayed, and by selecting an area image here, the user can input an area image. At this time, a CG area image is input in the CG image input mode, and a real area image is input in the real image input mode.

Furthermore, an area image display section 41 is provided on the screen 21 in the annotation work mode. The area image display section 41 displays the input area image (CG area image, real area image). Further, in the area image display section 41, the input person image (CG person image, real photographed person image) is displayed in a superimposed manner on the area image. Furthermore, in the area image display section 41, the user can adjust the position and size of the person image superimposed on the area image by dragging the mouse or the like.

Further, in the area image display section 41, the user can specify a target person on the area image, that is, specify a position on the area image from which the teacher image is to be cut out. Specifically, the user can input the person frame 42 that surrounds the person image (CG person image or live photo area image) on the area image (CG area image or live photo area image) by dragging the mouse or the like. . This person frame 42 becomes a candidate for the range of the teacher image. That is, a person rectangle is set based on the position of the person frame 42, and a teacher image is generated based on the person rectangle. At this time, the teacher image is cut out from the combined area image in which the area image and the person image are combined.

Note that when creating a teacher image for a person included in the live-action area image displayed on the area-image display section 41, a person frame 42 surrounding the person included in the live-action area image may be input. In this case, the user's input operation on the person image input section 33 is not required.

Further, by performing the person detection process, the input operation of the person frame 42 by the user may be omitted. That is, a person detection process is performed on a live-action area image containing a person or a composite area image in which an area image and a person image are combined, and a teacher image is cut out based on the person detection frame obtained by the person detection process. You may be

Further, a frame operation section 45 is provided on the screen 21 in the annotation work mode. The frame operation unit 45 is provided with a button 46 that moves the area image back by one frame, and a button 47 that moves the area image back one frame. By operating the frame operation unit 45, a teacher image can be created for each frame of the area image.

Further, a detection area input section 51 is provided on the screen 21 in the annotation work mode. In the detection area input section 51, when the user operates the input button 52, the transition is made to the detection area input mode, and the user can input the detection area and the area image displayed on the area image display section 41 by dragging the mouse or the like. You can input the range of the pre-detection area.

Further, an attribute input section 53 is provided on the screen 21 in the annotation work mode. In the attribute input section 53, the user can input the color of upper body clothing and the color of lower body clothing as attributes related to the person included in the teacher image. When a CG person image is selected, the color of the clothes is already known, so no input operation is required by the user.

Further, a title input section 55 is provided on the screen 21 in the annotation work mode. In the title input section 55, the user can input the title of the teacher image, specifically, the name given to the group of teacher images. The teacher image title (group name) identifies a collection of teacher images when checking the annotation status of the teacher images.

Further, a save button 56 is provided on the screen 21 in the annotation work mode. When the user operates the save button 56, a teacher image is generated based on the input contents of each section, the teacher image is saved in the storage section 12, and attribute information etc. regarding the teacher image are registered in the database. .

A teacher image selection section 62 is provided on the annotation status confirmation mode screen 61 shown in FIGS. 13, 14, and 15. The teacher image selection section 62 allows the user to select a group of teacher images to be analyzed. For example, when the user operates the teacher image selection section 62, a list of groups of teacher images is displayed, and the user can select a group of teacher images here. In this example, a group of teacher images in which entrance A is the monitoring area is selected.

Further, the annotation status confirmation mode screen 61 is provided with a list button 63 and a graph button 64. When the user operates the list button 63, analysis processing is performed on the group of teacher images selected by the teacher image selection section 62, and a list selection screen 61 shown in FIG. 13 is displayed as the analysis result. be done. Further, when the user operates the graph button 64, analysis processing is performed on the group of teacher images selected by the teacher image selection section 62, and as an analysis result, a screen 71 at the time of graph selection shown in FIG. is displayed.

On the screen 61 when selecting a list in the annotation status confirmation mode shown in FIG. 13, a list 66 is displayed on the visualization result display section 65. The list 66 displays a list of attributes of each teacher image registered in the database. In this example, attributes of teacher images related to entrance/exit A are displayed in a list. Further, as attributes of the teacher image, characteristics of the person included in the teacher image, particularly the color of the person's clothing, and the time period of the area image that is the background of the teacher image are displayed.

The list 66 visualizes the annotation status of the teacher images using text information, and by visually checking the list 66, the user can confirm deficiencies in the annotation status such as a lack of teacher images with a specific attribute. I can do it. Specifically, the user may notice that there are fewer teacher images containing people wearing clothes of a specific color (e.g. yellow) than teacher images of other attributes in area images taken during a specific time period (e.g. 8 o'clock). can be confirmed.

In the example shown in FIG. 13, the list 66 displays the color of the person's clothing and the time period of the area image as attributes related to the teacher image, but other attributes may also be displayed. For example, the presence or absence of overlapping people, that is, whether the teacher image includes a background person area or a foreground person area may be displayed. Furthermore, the image quality (resolution, presence or absence of blur, etc.) of the person area and background area of the teacher image may be displayed. Furthermore, the season of the area image (spring, summer, fall, winter) may be displayed.

On the screen 71 when selecting a graph in the annotation status confirmation mode shown in FIGS. 14 and 15, a statistical graph 72 is displayed on the visualization result display section 65. The statistical graph 72 is a three-dimensional bar graph. In the statistical graph 72, the first axis in the horizontal direction represents the time period of the area image that is the background of the teacher image as the first attribute of the teacher image, and the second axis in the depth direction represents the time period of the area image that is the background of the teacher image. The attribute represents the characteristics of the person included in the teacher image, particularly the color of the person's clothing, and the third axis in the vertical direction represents the number of teacher images. In other words, a bar graph is drawn for each combination of the first attribute (time period of the area image) and second attribute (color of person's clothing) of the teacher image, and the height of the bar graph It represents the number of teacher images having the attribute of 2.

The statistical graph 72 visualizes the annotation status of the teacher image as a bar graph, and by visually observing the statistical graph 72, the user can confirm deficiencies in the annotation status such as a lack of teacher images with a specific attribute. can. Specifically, the user may notice that there are fewer teacher images containing people wearing clothes of a specific color (e.g. yellow) than teacher images of other attributes in area images taken during a specific time period (e.g. 8 o'clock). can be confirmed.

Here, in the screen 71 shown in FIG. 14, there are fewer teacher images in which the person is dressed in yellow than teacher images in which the person is in light blue, green, and red, and a teacher image with this attribute is added. There is a need. On the other hand, on the screen 71 shown in FIG. 15, the number of teacher images is uniform for yellow, light blue, green, and red regarding the clothing color of the person included in the teacher image, and the annotation status of the teacher image is improved. has been done.

Note that an operation section for selecting the first attribute of the teacher image represented by the first axis in the horizontal direction and the second attribute of the teacher image represented by the second axis in the depth direction in the statistical graph 72 is provided on the screen. The attributes of the teacher image represented by each axis in the statistical graph 72 may be specified by the user on the screen.

A teacher image selection section 82 is provided in the annotation status detailed confirmation mode screen 81 (second screen) shown in FIGS. 16 to 19. The teacher image selection section 82 allows the user to select a group of teacher images to be analyzed. For example, when the user operates the teacher image selection section 82, a list of teacher image groups is displayed, from which a teacher image group can be selected. In this example, a group of teacher images in which the monitoring area is entrance A and the time zone is 8 o'clock is selected.

Further, the screen 81 in the annotation status detailed confirmation mode is provided with an analysis button 83 and a visualization result display section 84. When the user operates the analysis button 83, analysis processing is performed on the group of teacher images selected in the teacher image selection section 82, and a heat map image 85 is displayed in the visualization result display section 84 as an analysis result. Is displayed. The heat map image 85 is displayed in a transparent state superimposed on the area image.

In the heat map image 85, the annotation status of the teacher image at each position on the area image is visualized. Specifically, the heat map image 85 is divided into a plurality of cells like a mesh, and in each cell, the number of teacher images whose representative point is located within the cell is expressed by a change in gradation.

Note that the number of teacher images whose representative points are located within a cell may be expressed by a change in hue. In this case, as the number of teacher images increases, the color of the cell may change, for example, in the order of blue, yellow, orange, and red. Further, the number of teacher images whose representative points are located within a cell may be expressed by a change in a pattern (pattern image) or the like.

Further, the screen 81 in the annotation status detailed confirmation mode is provided with an attribute selection section 87. In the attribute selection section 87, the user can select the color of a person's clothing as an attribute of the teacher image, particularly as an attribute of the person included in the teacher image. Specifically, the attribute selection section 87 is provided with buttons 88 for "all", "yellow", "light blue", "green", and "red". By operating the button 88, the user can select the color of the person's clothing. Further, in the attribute selection section 87, the user can select a plurality of colors of clothing of the person.

When the attribute selection unit 87 selects the color of the person's clothing as an attribute of the teacher image, particularly an attribute of the person included in the teacher image, in the heat map image 85, in each cell, the representative point is located within the cell. In addition, the number of teacher images that correspond to the specified attribute is expressed as a change in gradation.

In this example, as attributes of the teacher image, a heat map image 85 is displayed that visualizes the annotation status by focusing on the attributes related to the characteristics of the person included in the teacher image, especially the color of clothes, but other than the teacher image It may be a heat map image 85 that focuses on the attributes of . For example, the heat map image 85 may represent the presence or absence of overlapping people, that is, whether the teacher image includes a background person area or a foreground person area. Further, the image quality (for example, resolution, presence or absence of blur, etc.) of the person area and background area of the teacher image may be expressed in the heat map image 85.

Here, first, as shown in FIG. 16, the user checks the annotation status of the teacher image without limiting the attributes of the teacher image. Specifically, the user operates a button 88 for selecting all colors (yellow, light blue, green, and red) in the attribute selection section 87, so that the color of the clothes of the person included in the teacher image is not limited. Check the annotation status of the teacher image.

In this example, in the heat map image 85, the left cell in the area image has a small number of teacher images with the specified attribute, and the center cell in the area image has a small number of teacher images with the specified attribute. In many cases, the cell on the right side of the area image has no teacher image with the specified attribute.

Next, as shown in FIG. 17, the user checks the annotation status only for teacher images with specific attributes. In this example, the user operates the yellow selection button 88 in the attribute selection section 87 to check the annotation status regarding the teacher image with the attribute of the person's clothing being yellow.

In this example, similar to the state shown in FIG. 16, in the heat map image 85, the left cell in the area image has a small number of teacher images with the specified attribute, and the center cell in the area image has a small number of teacher images. There are a large number of teacher images with the specified attribute, and there are no teacher images with the specified attribute in the right cell in the area image.

Therefore, the user operates the annotation work tab 22 (operation unit) to return to the annotation work mode screen 21 shown in FIG. 12, and performs the annotation work to add the missing teacher image. In this example, a teacher image is added in which the representative points are located in the left and right regions of the area image and the person's clothing is yellow. That is, a teacher image including a person whose clothing color is yellow is added with the left and right regions in the area image as the background. As a result, the shortage of teacher images with attributes such that the representative points are located in the left and right regions of the area image and the color of the person's clothing is yellow can be alleviated.

Next, as shown in FIG. 18, the user selects the group of teacher images to which the teacher image has been added (entrance/exit A_8:00_teacher image_addition 1) in the teacher image selection unit 82 to be analyzed. , check the annotation status. In addition, the user operates a button 88 for selecting a color other than yellow (light blue, green, red) in the attribute selection section 87, and selects a teacher image with an attribute corresponding to a color other than yellow (light blue, green, red). Check the annotation status regarding.

In this example, in the heat map image 85, in the left and center cells in the area image, the number of teacher images that include a teacher image with the specified attribute, that is, a person whose clothes are light blue, green, or red. However, in the cell on the right side of the area image, there is no teacher image with the specified attribute at all.

Therefore, the user operates the annotation work tab 22 to return to the annotation work mode screen 21 shown in FIG. 12, and performs an annotation work to add the missing teacher image. In this example, a teacher image is added in which the representative point is located in the right region of the area image and the color of the person's clothing is light blue, green, or red. That is, a teacher image is added that includes a person whose clothing color is light blue, green, or red, with the right region in the area image as the background. As a result, the lack of teacher images in which the representative point is located in the right region of the area image and in which the color of the person's clothing is light blue, green, or red can be alleviated.

Next, as shown in FIG. 19, in the teacher image selection section 82, the user selects the teacher image group (entrance/exit A_8:00_teacher image_addition 2) to which the teacher image has been added again as an analysis target. to check the annotation status. In addition, the user operates the button 88 for selecting all colors (yellow, light blue, green, red) in the attribute selection section 87 to select the attributes corresponding to all the colors (yellow, light blue, green, red). Check the annotation status regarding teacher images.

In this example, in the heat map image 85, the number of teacher images is uniform in all cells within the range where a person can exist, and the user can confirm that the annotation situation has been improved. can do.

Further, on the screen 81 in the annotation status detailed confirmation mode, when the user specifies a measurement target area 91 on the heat map image 85 by dragging the mouse or the like, a list 92 (frequency distribution table) is displayed. The list 92 represents the distribution of the colors of people's clothes as attributes of teacher images for teacher images whose representative points are located in the designated measurement target area 91. Specifically, the number of teacher images is displayed for each color of the person's clothing (yellow, light blue, green, red). The user can check the details of the annotation status based on the number of teacher images for each color of person's clothing (yellow, light blue, green, red). In this example, the number of teacher images of all colors is equal, and the user can confirm that the teacher images with the necessary attributes are evenly distributed.

Further, a learning button 88 is provided on the annotation status detailed confirmation mode screen 81. When the user confirms that the teacher images with the required attributes are evenly distributed, the user operates the learning button 88. Thereby, learning processing is executed based on the generated teacher image, and an image recognition model (machine learning model) is created.

Next, another example of the screen in the annotation status detailed confirmation mode will be described. FIG. 20 is an explanatory diagram showing another example of the screen in the annotation status detailed confirmation mode.

In the screen 81 shown in FIGS. 16 to 19, a heat map image 85 (visualized image) in which the number of extracted teacher images is expressed by a change in gradation is displayed in the visualization result display section 84, superimposed on the area image. . On the other hand, on the screen 101 shown in FIG. 20, a frame image 102 (mark image) representing a range on the area image where there is a problem in the collection status of teacher images is displayed in the visualization result display section 84 in a superimposed manner on the area image.

In this example, a frame image 102 is superimposed and displayed on the area image so as to surround a range on the area image from which a small number of extracted teacher images corresponding to a specified attribute (for example, a person's clothes are yellow) are extracted. Furthermore, a frame image 102 is superimposed and displayed on the area image so as to surround a range on the area image from which a small number of teacher images have been extracted without limiting the attributes.

Note that a range on the area image where there is no problem in the collection status of teacher images may be displayed as the frame image 102. For example, the frame image 102 may be drawn so as to surround a range on the area image from which a large number of extracted teacher images having a specific attribute are extracted. Further, the frame image 102 may be drawn in a different color depending on the number of extracted teacher images.

Further, in this example, the color of the person's clothing is specified as an attribute of the teacher image, and an area where the number of extracted teacher images in which the person's clothes are a specific color is displayed as the frame image 102. Attributes other than the color of clothing may be specified. For example, overlapping people may be specified as an attribute of the teacher image, and an area where a small number of extracted teacher images in which overlapping people occur may be displayed as the frame image 102.

Further, in this example, a frame image 102 (mark image) representing a range on the area image where there is a problem in the collection status of teacher images is displayed superimposed on the area image, but the mark image is not limited to the frame image 102. For example, as a mark image, a semi-transparent image with a pattern drawn thereon may be superimposed and displayed in a range on an area image where there is a problem in the collection status of teacher images. Furthermore, in addition to the frame image 102 (mark image), a comment (not shown) regarding the addition of the teacher image may be presented.

As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments in which changes, replacements, additions, omissions, etc. are made. Furthermore, it is also possible to create a new embodiment by combining the components described in the above embodiments.

The image processing device and image processing method according to the present invention have the effect that a user can easily visually check the collection status of teacher images prior to learning, and can efficiently create a highly accurate learning model. The present invention is useful as an image processing device and an image processing method for visualizing the collection status of teacher images for constructing an image recognition model corresponding to a monitoring area.

1 Image processing device 13 Processor 21 Annotation work mode screen 22 Tab 61 Annotation status confirmation mode screen 66 List 71 Annotation status confirmation mode screen 72 Statistical graph 81 Annotation status detailed confirmation mode screen 85 Heat map image 101 Annotation status details Confirmation mode screen 102 Frame image (mark image)

Claims (8)

An image processing device that uses a processor to perform a process of visualizing the collection status of teacher images for constructing an image recognition model corresponding to a monitoring area, the image processing device comprising:
The processor includes:
Generating a teacher image including a detection target object and a background from an area image related to the monitoring area,
setting attributes related to the characteristics of the detection target included in the teacher image for each teacher image;
generating a visualized image that visualizes the collection status of the teacher image at each position of the area image, targeting the teacher image having the attribute specified by the user;
An image processing device that outputs display information in which the visualized image is superimposed on the area image. The processor includes:
2. The image processing apparatus according to claim 1, wherein a heat map image representing a collection status of the teacher image at each position of the area image is generated as the visualized image. The processor includes:
2. The image processing apparatus according to claim 1, wherein a mark image representing a range on the area image where there is a problem in the collection status of the teacher image is generated as the visualized image. The processor includes:
The image processing apparatus according to claim 1, wherein the teacher image is generated from a real area image photographed by a camera or a virtual area image created by CG as the area image. The processor includes:
The image processing apparatus according to claim 1, wherein the image processing apparatus generates the visualized image that visualizes the collection status of the teacher images for each color type related to the person as the attribute. The processor includes:
The image processing apparatus according to claim 1, wherein the image processing apparatus generates a statistical graph that visualizes the collection status of the teacher images for each attribute, and outputs the display information including this statistical graph. The processor includes:
outputting the display information including a first screen for generating the teacher image and setting attributes for the teacher image in response to a user's operation;
Claim characterized in that the visualized image is superimposed and displayed on the area image, and the display information including a second screen provided with an operation unit for returning to the first screen is output. 1. The image processing device according to 1. An image processing method in which a processor executes processing for visualizing the collection status of teacher images for constructing an image recognition model corresponding to a monitoring area, the method comprising:
Generating a teacher image including a detection target object and a background from an area image related to the monitoring area,
setting attributes related to the characteristics of the detection target included in the teacher image for each teacher image;
generating a visualized image that visualizes the collection status of the teacher image at each position of the area image, targeting the teacher image having the attribute specified by the user;
An image processing method characterized by outputting display information in which the visualized image is superimposed on the area image.

Images