JP7349290B2 - Object recognition device, object recognition method, and object recognition program - Google Patents

Description

The present invention relates to a technology for recognizing a target object by detecting predetermined information regarding the target object, such as the position of the target object (part, area, etc.) from measurement data such as an image.

There is a lot of research being done on detecting multiple parts of a person that appear in captured images based on machine learning.

For example, in the technology described in Non-Patent Document 1, a large number of learning images depicting people are used as input values, and annotations describing the types and positions of human body parts in the learning images are used as target values for output values. Deep learning the model. Then, by inputting the photographed image to the trained model, the type and position of the body part of the person in the photographed image is output. This annotation is created for the part appearing in the learning image. By the way, the information about each part written in the annotation and the information about each part output by the trained model are called key points.

If the parts necessary for various types of recognition about a person can be detected, it becomes possible to recognize not only the posture of the person, but also the region of existence, and whether the person is an adult or a child (attribute) based on proportions.

“Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.”, Z. Cao, T. Simon, S. Wei and Y. Sheikh (2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1302-1310)

However, with the conventional technology, the accuracy of estimating parts that do not appear in the photographed image is low, so if objects are hidden from each other, it becomes difficult to recognize the posture, area of existence, attributes, etc. of the objects. Ta.

For example, if two people overlap on a photographed image, a partially hidden person may not be detected. Therefore, when recognizing a person's presence area based on the detection result, a partially hidden person will be recognized as not having a presence area. Furthermore, it is difficult to recognize the posture and attributes of a partially hidden person.

That is, in the conventional technology, since the relationship between the learning image and the part that does not appear in the image is not explicitly learned, it is difficult to detect the part that does not appear in the photographed image. Therefore, in the conventional technology, it may be difficult to recognize the posture, area of existence, attributes, etc. of a hidden object.

Further, the above problem occurs not only in two-dimensional measurement data (images) but also in three-dimensional measurement data, and similarly occurs in the time series of two-dimensional measurement data and the time series of three-dimensional measurement data.

The present invention has been made in view of the above-mentioned problems, and provides an object recognition device, an object recognition method, and an object recognition device that can recognize with high accuracy even a measured object that is partially hidden by another object. The purpose is to provide programs.

(1) The object recognition device according to the present invention is a device that recognizes predetermined information regarding each object from measurement data obtained by measuring a plurality of objects, and in which a sample of the object is measured. data obtained by masking a mask region that includes a part of the region of the sample in the sample data that has been obtained, and information on the mask region are input, and the predetermined information regarding the sample is used as the output target value. a detector storage means storing a detector generated in advance through learning; data obtained by masking a detected area in which the object has been detected in the measurement data; and a detection means for inputting the mask processing data into the detector to detect the predetermined information regarding the object in the measurement data. .

(2) In the object recognition device according to (1) above, the detector storage means performs the learning also when the sample data is inputted in a state before the masking process is performed. storing the detector, and the detecting means inputs the measurement data to the detector instead of the mask processing data when there is no detected area, and inputs the predetermined information regarding the object in the measurement data. It can be configured to detect.

(3) In the object recognition device described in (1) and (2) above, the predetermined information may be information regarding a position of a part constituting the object.

(4) The object recognition device described in (1) to (3) above may further include a learning means for generating the detector through the learning based on the sample data.

(5) The object recognition method according to the present invention is a method of recognizing predetermined information regarding each object from measurement data obtained by measuring a plurality of objects, wherein a sample of the object is measured. data obtained by masking a mask region that includes a part of the region of the sample in the sample data that has been obtained, and information on the mask region are input, and the predetermined information regarding the sample is used as the output target value. a step of preparing a detector generated in advance through learning, and data obtained by masking a detected area in which the object has been detected in the measurement data and information on the detected area. and a step of inputting the mask processing data into the detector to detect the predetermined information regarding the object in the measurement data.

(6) The object recognition program according to the present invention is a program that causes a computer to perform a process of recognizing predetermined information regarding each object from measurement data obtained by measuring a plurality of objects, the program comprising: A computer is input with data obtained by masking a mask area including a part of the area of the sample in the sample data in which the sample of the target object is measured, and information on the mask area, and the information about the sample is input. a detector storage means storing a detector generated in advance by learning using the predetermined information as an output target value; a mask unit that generates mask processing data consisting of data obtained by mask processing and information on the detected area; and a mask unit that inputs the mask processing data to the detector to obtain the predetermined information regarding the object in the measurement data. It functions as a detection means for detecting information.

According to the object recognition device, object recognition method, and object recognition program of the present invention, it is possible to recognize with high accuracy even a measured object whose part is hidden by another object.

1 is a block diagram showing a schematic configuration of a target object recognition device according to an embodiment of the present invention. FIG. 2 is a schematic functional block diagram regarding a learning stage of the object recognition device according to the embodiment of the present invention. FIG. 3 is a schematic diagram illustrating an example of learning data regarding a learning stage of the object recognition device according to the embodiment of the present invention. FIG. 2 is a schematic functional block diagram regarding the recognition stage of the object recognition device according to the embodiment of the present invention. FIG. 2 is a schematic flow diagram regarding the operation of the object recognition device according to the embodiment of the present invention in a learning stage. FIG. 2 is a schematic flow diagram regarding the operation of the object recognition device in the recognition stage according to the embodiment of the present invention. FIG. 3 is a schematic diagram for explaining a processing example at a recognition stage of the object recognition device according to the embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An object recognition device 1 that is an embodiment of the present invention (hereinafter referred to as an embodiment) will be described below based on the drawings. The object recognition device according to the present invention obtains predetermined information regarding a predetermined object from measurement data. The positions of a plurality of parts constituting a person appearing in space and the area where the person exists (object area) are detected. That is, in this embodiment, the measurement data is a two-dimensional image, the target object is a person, and the predetermined information regarding the target object is information regarding the position of the target object, specifically, the position of the body part of the person. and the realm of human existence. Incidentally, the object recognition device 1 detects an area surrounding a plurality of body parts detected for each person as the object area of the person. Note that in the measurement data, a part of the body part may be hidden by another object. In the present invention, estimation of a hidden part is referred to as part detection, and estimation of an object area using the estimation result of a hidden part is referred to as detection of an object area.

The plurality of parts used for the target object recognition are referred to as required detection parts, and representative points of the required detection parts are referred to as key points. The key point information is represented by a combination of at least the type and position of the corresponding part, and data including this combination is referred to as part data. By detecting each key point, the position of the corresponding detection target region is detected. Note that the type of the part to be detected is determined in advance depending on the object and the purpose of recognition.

The object recognition device 1 uses learning images and annotations (added data) to the learning images to learn and store a detector for detecting body part data. Here, the given data is part data given to the object appearing in the measurement data for learning. Then, the object recognition device 1 detects body part data in the photographed image using the stored detector. In particular, the object recognition device 1 performs learning by performing mask processing on a part of the learning image. In this way, a detector capable of detecting hidden parts is learned. Then, the object recognition device 1 applies mask processing to a region of the object that may be hiding another object in the photographed image, and applies the detector to the region of the object. This allows detection including hidden parts.

[Configuration of object recognition device 1]
FIG. 1 is a block diagram showing a schematic configuration of an object recognition device 1. As shown in FIG. The object recognition device 1 includes a photographing section 2, a communication section 3, a storage section 4, an image processing section 5, and a display section 6.

The photographing unit 2 is a measurement unit that acquires measurement data, and is a surveillance camera in this embodiment. The photographing section 2 is connected to the image processing section 5 via the communication section 3, photographs the monitoring space at predetermined time intervals to generate photographed images, and sequentially inputs the photographed images to the image processing section 5. For example, the photographing unit 2 is installed on a pole installed in a corner of the event venue, which is the surveillance space, with a predetermined fixed field of view overlooking the surveillance space, and photographs the surveillance space at a frame period of 1 second, coloring it. Generate an image. Note that the photographing unit 2 may generate a monochrome image instead of a color image. Further, the image processing unit 5 is installed at, for example, an image analysis center.

The communication section 3 is a communication circuit, one end of which is connected to the image processing section 5, and the other end connected to the photographing section 2 and the display section 6. The communication section 3 acquires a photographed image from the photographing section 2 and inputs it to the image processing section 5 , and receives the recognition result of the object from the image processing section 5 and outputs it to the display section 6 .

Note that the photographing section 2, the communication section 3, the storage section 4, the image processing section 5, and the display section 6 are connected as appropriate depending on the installation location of each section. For example, when the photographing section 2, the communication section 3, and the image processing section 5 are installed remotely, the photographing section 2 and the communication section 3 can be connected via an Internet line. Further, the communication section 3 and the image processing section 5 may be connected via a bus. In addition, a LAN (Local Area Network), various cables, etc. can be used as the connection means.

The storage unit 4 is a memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), and stores various programs and various data. For example, the storage unit 4 stores learning images, data added to the learning images, and information on a detector that is a trained model. The storage unit 4 is connected to the image processing unit 5 and inputs and outputs this information to and from the image processing unit 5. That is, information necessary for object recognition and information generated in the process of recognition processing are input and output between the storage section 4 and the image processing section 5.

The image processing unit 5 is a measurement data processing unit that processes measurement data, and is a calculation device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an MCU (Micro Control Unit), or a GPU (Graphics Processing Unit). configured. The image processing unit 5 operates as various processing means and control means by reading and executing programs from the storage unit 4, and reads various data from the storage unit 4 and stores generated data in the storage unit 4 as necessary. Make me remember. For example, the image processing unit 5 learns and generates a detector. Further, the image processing unit 5 stores the generated detector in the storage unit 4 via the communication unit 3. The image processing unit 5 also uses a detector to perform a process of recognizing an object in a photographed image.

The display unit 6 is a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like, and displays the recognition results input from the communication unit 3. The monitor determines whether or not a response is necessary based on the displayed recognition result, and takes measures such as dispatching a response person as necessary.

In this embodiment, the storage section 4 and the image processing section 5 are provided on the image analysis center side, but they may be provided on the imaging section 2 side.

Hereinafter, regarding the configuration of the object recognition device 1, first, the configuration related to the learning stage of learning the detector will be explained, and then the configuration related to the recognition stage of recognizing the target object using the detector will be explained.

[Configuration of object recognition device 1 regarding learning stage]
FIG. 2 is a schematic functional block diagram of the object recognition device 1 regarding the learning stage, in which the storage section 4 functions as a learning data storage means 40 and a learning model storage means 41, and the image processing section 5 functions as a learning mask means 50. and functions as a learning means 51.

The learning data storage means 40 is a learning image storage means that stores a large number of learning images, and also serves as a given data storage means that stores given data for the learning images. The learning images and assigned data include those that are stored in advance prior to learning and those that are generated during learning. The learning data storage means 40 stores the learning image and the assigned data of each person photographed in the image in association with each other. Hereinafter, the person photographed in the learning image will be referred to as a sample. Different people are different samples, and even if they are the same person, different images are different samples. Further, the learning image is measurement data (sample data) in which samples are measured. Specifically, each sample is given a sample ID for mutual identification, the learning image is given an image ID, and the learning data storage means 40 stores the correspondence of these IDs. The assigned data includes information about each key point of each sample. That is, based on the provided data, the position of the representative point of each type of a plurality of detection-required parts of each sample can be determined. Further, the learning data storage means 40 stores the learning data generated by the learning mask means 50.

The learning mask means 50 acquires the learning image from the learning data storage means 40 and creates a masked learning image by masking a part of the learning image. Thereafter, information representing the masked area (mask area information) and the masked learning image are output to the learning data storage means 40. The mask area information is, for example, a binary image having the same width and height as the masked learning image, and stores 1 in the masked area and 0 in other areas.

The learning mask means 50 randomly sets the position and size of the mask area. Specifically, the learning mask means 50 takes two random numbers without duplication from integer values within the range from 0 to the number of pixels of the width of the image, sets them as x1 and x2 (x1<x2), and sets them as x1 and x2 (x1<x2), Two random numbers are taken without duplication from the integer values within the range of the number of pixels so far, and are set as y1 and y2 (y1<y2). Then, a region on the image satisfying x1≦x<x2 and y1≦y<y2 is set as a range to be masked, and is filled with a predetermined value (mask value). Here, the mask value was the average color of all pixel values of all learning images.

Note that depending on the random numbers, a mask area that does not overlap with the image of the target object may be set, so the learning masking means 50 repeatedly performs a sufficient number of generation processes on each learning image to This ensures that a masked learning image is generated in which a part of the image of the image overlaps with the mask area. Alternatively, the learning mask means 50 refers to the assigned data and performs a process for each learning image to generate a masked learning image only when it is confirmed that the position of one or more key points is within the mask area. It may be repeated a certain number of times for the image.

FIG. 3 is a schematic diagram illustrating an example of learning data, and shows learning data generated from a learning image 100 in which a sample appears and data 101 attached to the sample.

The learning image 100 is an example of a learning image stored in the learning data storage means 40 in advance. The learning images prepared in advance do not have to be real images actually taken with a camera, and may be images created using computer graphics (CG), for example. The assigned data 101 is assigned data that is stored in advance in the learning data storage means 40 in association with the learning image 100. The assigned data prepared in advance may be created manually, or may be created by a person checking what is extracted by a machine and correcting it as necessary, or may be a mixture of these. . Note that the provided data shown here is an example in which the number of detection parts is 17 and the topology of a person's key points is diagrammed. It is illustrated with 17 white circles representing the positions of key points and 16 line segments representing the connection relationships between key points.

The learning mask means 50 generates a plurality of masked learning images and mask area information by associating them with each other based on the learning image 100 prepared in advance in the learning data storage means 40. In FIG. 3, masked learning images 110, 120, and 130 are shown as three examples of various masked learning images generated from the learning image 100 by the learning masking means 50. In these masked learning images, the shaded area is the masked area, including a masked learning image 110 in which only the lower body of the person is masked, a masked learning image 120 in which most of the central part of the person is masked, and a masked learning image 120 in which most of the middle part of the person is masked. A masked learning image 130 in which most of the left side of the body is masked is shown.

The mask area information 111, 121, 131 is mask area information generated by the learning mask means 50 corresponding to the masked learning images 110, 120, 130, respectively. Furthermore, the attached data 112, 122, and 132 are attached data that are associated with the masked learning images 110, 120, and 130, respectively. Incidentally, these are copies of the assigned data 110.

In the above example, the learning masking means 50 randomly generates mask areas, but the size and position of the mask areas may be set regularly and comprehensively. Further, in the above example, the learning masking means 50 generates a rectangular mask area, but the shape of the mask area may be set to an ellipse, or may be set to the shape of an assumed shielding object, or may be set to an appropriate shape. It can be any shape.

The learning means 51 learns the detector using at least the masked learning image obtained from the learning mask means 50, the mask area information, and the assigned data. That is, the learning means 51 generates a detector through learning using the masked learning image and the mask area information as input and the assigned data as the output target value. Preferably, the learning means 51 also uses the original learning image (learning image without mask) for the learning of the detector. At this time, the learning means 51 generates mask area information in which all elements are 0 corresponding to the unmasked learning image and uses it for learning. In other words, the learning model of the detector stored in the learning model storage means 41 inputs a set of a masked learning image and mask area information, and a set of an unmasked learning image and mask area information, and divides each set into two sets. It is trained to output key points for each included sample. Note that learning here means finding the parameters of the detector from the given data.

In this embodiment, the detector is modeled using convolutional neural networks (CNNs), which are composed of convolutional layers, linear transformation processing, activation functions, and the like. A ReLU function is used as the activation function. The learning means 51 performs learning for the parameters of each element constituting the CNNs by minimizing an error function that quantifies the degree of deviation between the output and the corresponding given data. The error function is determined in advance. Stochastic steepest descent method is used for minimization.

Here, the detector uses the mask area information as a clue and calculates the output using only the unmasked portion of the masked image. In this way, by learning to detect the target value before masking without using the masked area, the detector can detect the surrounding mask even if there is a key point in the masked area. It becomes possible to detect key points within the masked area using information from the unmarked areas.

As a detector, any module may be used as long as it calculates the output using the unmasked portion of the masked training image using the mask area information as a clue, but in this embodiment, the following reference is used. This module is constructed by using Partial Convolution, which was proposed in , as a convolution layer.

Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao and Bryan Catanzaro, “Image Inpainting for Irregular Holes Using Partial Convolutions,” The European Conference on Computer Vision (ECCV), pp. 85-- 100, 2018.

Here, when inputting to CNNs, the masked learning images are normalized by subtracting the average color obtained from all pixel values of all learning images from each pixel value. In other words, the pixel values in the masked area are normalized to zero.

The learning model storage means 41 stores a learning model for the detector. Specifically, the detector parameters obtained by the learning means 51 are stored. Also, the structure of CNNs used as a detector is stored. Along with the learning process by the learning means 51, the learning model stored in the learning model storage means 41 is updated. When the learning is completed, the learning model storage means 41 stores the trained model of the detector and functions as the detector storage means 43.

[Configuration of object recognition device 1 regarding recognition stage]
FIG. 4 is a schematic functional block diagram of the object recognition device 1 regarding the recognition stage, in which the storage section 4 functions as an object area storage means 42 and a detector storage means 43, and the image processing section 5 functions as an already detected area masking means. 53, functions as a key point detection means 54 and an object area calculation means 55, and the communication section 3 cooperates with the image processing section 5, and functions as a photographed image acquisition means 30 and a recognition result output means 31.

The photographed image acquisition means 30 sequentially acquires photographed images from the photographing section 2 and outputs them to the image processing section 5.

As described above, the detector storage means 43 stores the detectors generated in the learning stage.

The object area storage means 42 stores information on the object area calculated by the object area calculation means 55. The information includes part data (detected part data) detected as the source of the target object area.

The detected area masking means 53 masks the detected area in which the object has been detected in the captured image, and generates a masked captured image and information on the detected area (detected area information). ) to generate mask processing data. Specifically, the detected area masking means 53 reads out the object area from the object area storage means 42, defines the detected area, and generates a masked captured image using this as a mask area. Further, the detected area masking means 53 generates detected area information indicating the masked area. The detected area information is a binary image having the same width and height as the photographed image, and for example, 1 is stored in the detected area and 0 is stored elsewhere. The pixel values of the masked area of the masked captured image are replaced with the mask values used by the learning masking means 50 in the learning stage.

Note that if the object area storage means 42 does not store any object area, the detected area masking means 53 sets all elements of the detected area information to 0 and uses the masked image instead of the masked captured image. The original photographed image, which is a blank photographed image, is output.

Note that the detected area is not necessarily the target area itself. For example, in this embodiment, a circumscribed rectangle, which will be described later, is the object area, but focusing on the fact that the corners of the circumscribed rectangle are often the background, an elliptical area inscribed in the circumscribed rectangle is used as the detected area. may also be used. In this embodiment, an elliptical area obtained by multiplying the short axis and long axis of an ellipse inscribed in a rectangular object area by a constant is used as the detected area. Furthermore, as will be described later, when a plurality of target object areas are detected in one captured image, the detected area becomes an area resulting from combining these areas.

The key point detection means 54 reads the detector stored in the detector storage means 43, and inputs the photographed image obtained from the detected area masking means 53, or the masked photographed image and the detected area information to the detector. do. The detector outputs part data (detected part data) for each object shown in the image, and the key point detection means 54 outputs the detected part data to the object area calculation means 55.

The target object area calculating means 55 receives the detected part data from the key point detecting means 54, calculates a predetermined range based on the required detection part indicated by the detected part data as the target object area in the measurement data, and calculates the calculated target object area. information is stored in the object area storage means 42. For example, the object area calculation means 55 can calculate a circumscribed rectangle that circumscribes the key point group of each person as the object area regarding the person.

Alternatively, the circumscribed rectangle may be expanded at a predetermined ratio to form the object area. That is, when setting the target object area, margins are provided on the top, bottom, left, and right sides, taking into consideration the fact that key points are detected slightly inside the true area and detection errors. Furthermore, the ratios for the up, down, left, and right directions may be different depending on the definition of each key point and the estimation of the detection error of each key point.

Alternatively, each detected body part data can be converted into a target area by inputting the detected body part data for each person detected from each input image into a converter that has learned to convert human body part data to a target object area. Good too. In this case, the correct data for the target object area for each sample is included in the assigned data, and the converter inputs the part data of the sample assigned data and outputs the target object area of the assigned sample data. It can be a trained model generated by learning with a target value of . The storage unit 4 also functions as a converter storage means (not shown) and stores converters in advance. Then, the object area calculation means 55 reads out the converter from the converter storage means and uses it. That is, the object area calculation means 55 inputs the detection part data detected by the key point detection means 54 to the converter, and detects the object area in the measurement data.

In the detection part data detected in the above manner, key points hidden in the measurement data are complemented, so there are problems such as detection of extremely small object areas due to occlusion, and problems where one object area is detected due to occlusion. It is possible to significantly reduce the problem that the target object area related to the target object is divided into a plurality of parts and detected.

Here, the part of the object that is hidden may be detected before masking the region of the object that is hiding the object. The region detected in this way is often only a part of the object, and the object region is erroneously recognized to be smaller than it should be. Furthermore, in some cases, an area that is originally a single object may be mistakenly recognized as an area of multiple objects, such as when a part of the upper body and a part of the lower body of a person are detected separately. Ru.

Therefore, the object area calculation means 55 excludes the output of the key point detection means 54 (output of the detector) for which the number of parts is less than a predetermined number (a predetermined lower limit number), and calculates the number of parts. Only the target object areas of the target objects whose number is equal to or greater than the lower limit number are calculated. By doing so, the effect of preventing erroneous recognition can be further enhanced.

Specifically, for example, the object area calculation means 55 calculates the object area only for a person for whom more than half (9 or more) of the 17 detection-required parts are detected. The information is additionally recorded in the storage means 42. Note that setting the lower limit number to half is just an example, and other values may be set as appropriate depending on the purpose of recognition.

Further, conditions other than the number of parts may be used as a condition for calculating the target object area. For example, the object area calculation means 55 calculates the object area only for objects for which a specific detection-required part (for example, a face) has been detected. Further, for example, the degree of importance is determined in advance for each type of part, and the object area calculation means 55 only applies to objects for which the sum of the degrees of importance of the detected parts to be detected is equal to or greater than a predetermined threshold. Calculate the object area.

The recognition result output means 31 outputs the information on the object area outputted by the object area calculation means 55 to the display section 6. For example, the recognition result output means 31 generates an image in which a rectangle representing the object area is drawn on the photographed image and outputs it to the display unit 6. Note that if the target object area is not detected, the recognition result may be that there is no target object, and the captured image may be output as is.

[Operation of object recognition device 1]
Next, the operation of the object recognition device 1 will be explained separately in a learning stage and a recognition stage.

[Operation of object recognition device 1 at learning stage]
FIG. 5 is a schematic flow diagram regarding the operation of the object recognition device 1 in the learning stage.

The object recognition device 1 performs an operation of learning the detector prior to an operation of recognizing an object appearing in a photographed image.

When the learning operation is started, the image processing unit 5 reads the learning model of the detector from the learning model storage means 41 in order to perform the learning of the detector. The parameters of the learning model at this point are initial values (step S10).

Subsequently, the image processing section 5 functions as the learning mask means 50, and reads the learning image and the data assigned to the sample group within the image from the learning data storage means 40 (step S11). The learning masking means 50 randomly sets regions to be masked in the image (step S12). Furthermore, the learning masking means 50 creates a masked image by masking the set area (step S13). The masked image and mask area information generated from the learning image are stored in the learning data storage means 40 in association with the assigned data for the learning image.

Next, the image processing section 5 functions as a learning means 51, receives the masked learning image generated by the learning masking means 50 and the mask area information, and detects key points (step S14). Then, the degree of deviation between the detected key points and the assigned data of the learning image before masking that is associated with the masked learning image is calculated using an error function (step S15), and the The parameters of the learning model are updated using (step S16).

Furthermore, the image processing unit 5 functions as a learning means 51, and determines whether the repetition end condition is satisfied (step S17). If the condition is satisfied ("YES" in step S17), the learned model is stored as a detector in the learned model storage means 41 (step S18). On the other hand, if the condition is not satisfied ("NO" in step S17), the operations from step S11 to step S17 are repeated until the repetition end condition is satisfied. The repetition end condition may be, for example, that the value of the error function or its amount of change has become smaller than a predetermined threshold, or that a predetermined number of repetitions has been reached.

[Operation of object recognition device 1 at recognition stage]
FIG. 6 is a schematic flow diagram regarding the operation of the object recognition device 1 in the recognition stage.

The object recognition device 1 performs an operation of recognizing an object appearing in a photographed image using the detector generated in the above-described learning stage.

When the object recognition device 1 starts the operation, the photographing section 2 photographs the monitoring space at predetermined intervals and sequentially transmits the photographed images to an image analysis center where the image processing section 5 is installed. The image processing section 5 cooperates with the communication section 3 to repeat the operation shown in the flowchart of FIG. 6 every time it receives a photographed image from the photographing section 2.

The communication section 3 functions as a photographed image acquisition means 30, and upon receiving a photographed image, outputs the photographed image to the image processing section 5 (step S20).

The image processing unit 5 functions as a detected area masking unit 53, generates detected area information in which all elements are 0, and inputs it to the key point detection unit 54 together with the captured image (step S21).

Subsequently, the image processing section 5 functions as a key point detection means 54, and the key point detection means 54 reads out the detector stored in the detector storage means 43, and stores the input captured image and detected area information. The key points are input to the detector, detected for each person, and output as detected part data (step S22).

Next, the image processing unit 5 functions as an object area calculation means 55, and performs object recognition processing to calculate the object area of each person's key points using the detected part data detected by the key point detection means 54 as input. (Step S23). Note that when the detected part data is empty, the object area calculation means 55 outputs the object area as also empty.

The object area calculation means 55 determines whether the detection situation by the key point detection means 54 satisfies the repetition end condition (step S24). The object area calculation means 55 determines that the number of repetitions exceeds a predetermined upper limit or that the detection part data detected by the key point detection means 54 includes an object in which half or more of the number of required detection parts is detected. If it does not exist, it is determined that the repetition end condition is satisfied, and if there is an object for which the number of repetitions is less than or equal to the upper limit and half or more of the required detection parts are detected, it is determined that the repetition end condition is not satisfied.

If the repetition end condition is not satisfied ("NO" in step S24), the image processing unit 5 returns the process to step S22 after processing steps S25 to S27 described below.

In the processing from step S25 to S27, first, the image processing unit 5 functions as the object area calculation means 55, and calculates the information of the object area of the object in which half or more of the number of required detection parts are detected. It is stored in the area storage means 42 (step S25). Subsequently, the image processing section 5 functions as an already detected area masking means 53, and uses the object area calculating means 55 to determine the object area detected so far in the photographed image and stored in the object area storage means 42. All are read out and the detected object areas are combined to generate detected area information (step S26). For example, elliptical detected regions set for each of a plurality of object regions are combined, and the sum region of the plurality of ellipses is used as a mask region. Thereafter, the detected area masking means 53 performs mask processing on the area indicated by the detected area information on the photographed image to generate a masked image (step S27), and sends the masked image together with the detected area information to the key point detection means 54. input. That is, what is input from the detected area masking unit 53 to the key point detection unit 54 in step S22 is the unmasked captured image and the detected area information indicating that there is no masked area in the first step S22. However, in step S22 from the second time onward, the masked captured image and already detected area information indicating the masked area are displayed.

The processing in steps S22 and S23 using this masked photographic image is repeated until the repetition termination condition of step S24 is satisfied.

If the repetition end condition is satisfied (“YES” in step S24), the object area calculation means 55 reads out all the object areas stored in the object area storage means 42. The read object area is output to the display unit 6 via the communication unit 3 as a recognition processing result (step S28). Specifically, the image processing section 5 and the communication section 3 cooperate to function as the recognition result output means 31, and create a recognized image from information such as the object area inputted from the object area calculation means 55. , and outputs this to the display section 6. Note that the detected body part data and object area stored in the object area storage means 42 are deleted before processing the next captured image.

FIG. 7 is a schematic diagram for explaining an example of processing at the recognition stage of the object recognition device 1. Using FIG. The operation of 55 will be explained.

A photographed image 200 shown as an example in FIG. 7 includes two people 201 and 202. In the first process of key point detection and target object area calculation for this photographed image 200, the photographed image 200 without a mask is input from the detected area masking means 53 to the key point detection means 54, and the processing result 210 includes a person A key point 211 corresponding to 202 is detected, and a circumscribed rectangle of the key point 211 is calculated as the object region 212.

In other words, since the detector extracts information about body parts from the image and performs detection, in a situation where people are concealed from each other, as in the photographed image 200, the person 202 in the foreground and the person 201 in the back overlap. The characteristics of the person 202 in the foreground are mainly captured with respect to the area in which the image is displayed. As a result, the detector cannot detect the key points of the person 201 in the background for such an image.

In this regard, the object recognition device 1 aims to detect people by performing key point detection multiple times, rather than detecting all people at once. In the second process of key point detection and target object area calculation for the photographed image 200 in the example of FIG. In the image 220, a detected area 221 generated based on the target object area 212 is subjected to mask processing. In the processing result 230 for this image 220, a key point 231 corresponding to the person 201 is detected, and a circumscribed rectangle of the key point 231 is calculated as an object region 232.

In other words, as long as there is a storage device to store the human area that has been detected once, there is no need to perform detection next time on the same captured image. Key point detection may be performed on a masked captured image in which a human region is masked. In the example of FIG. 7, once the person 202 in the foreground is detected from the unmasked photographed image 200, the next detection will be based on the processing result 210 in which the person 202 in the foreground was detected, such as the masked photographed image 220. An image in which the detected area 221 is appropriately masked may be used. By using the masked image 220 and its detected area information, the detector operates to detect parts within the masked area to the extent that it can be inferred and complemented from the unmasked area. It is expected that the person 201 in the background can be detected.

That is, a captured image is input to the key point detection means 54, the object area calculation means 55 calculates the object area from the obtained detected part data, and the object area is inputted to the already detected area masking means 53, thereby detecting the area that has already been detected. Create a masked photographed image with the human area masked. By applying the key point detection means 54 again to the masked photographed image, the part data of the undetected sample is detected. By repeating creating masked captured images and detecting key points until no new detection part data can be obtained, it is possible to cope with concealment by multiple people.

[Modified example]
(1) In the above embodiment, an example is shown in which the whole body of a person is the object, but the object may be a part of the human body such as the upper half of the person, or an object other than the person such as a vehicle or a chair. Good too.

(2) In the above embodiment, the measurement data obtained by measuring the object is a two-dimensional image, and the measurement means for acquiring the measurement data is the camera 2 that takes the two-dimensional image. The measurement data and measurement means are not limited to this example. For example, the measurement data may be data obtained by measuring a three-dimensional space. Examples of 3D measurement data include distance images obtained using a distance image sensor as a measurement means, 3D data constructed from images taken with a multi-view camera, and point clouds measured with LiDAR (Light Detection and Ranging). I can list data. Further, the measurement data can also be a time series of two-dimensional images (time series of two-dimensional measurement data) or a time series of three-dimensional measurement data. Note that mask processing in the case of point cloud data is a process that causes data to be omitted.

(3) In the above embodiment, an example is shown in which the average value of the pixel values of the learning image is used as the mask value when generating the masked learning image and the masked captured image. In another embodiment, the pixel value of the masked area can be a predetermined single value, a random value (random noise), or a mask value suitable for the configuration of the detector.

(4) In the above embodiment, a detector is used in which the target value is body part data for each sample, but the present invention can also be applied to tasks with different recognition targets.

For example, by using an object detector that detects a circumscribed rectangle for each sample from an image as a detector, it is possible to perform object detection that is robust against concealment. In this case, the circumscribed rectangle becomes the target value for detector learning instead of the body part data, and the output of the key point detection means 54 becomes the detected value of the circumscribed rectangle for each sample. The output of 54 is used as it is as the target object region.

Further, the detector may be a detector that detects attributes such as age and gender, or a state such as a smiling face, in addition to information regarding the position of the target object.

(5) In the above embodiment, the same detector is used in the key point detection means 54 during repeated detection, but different detectors may be used for the first and second and subsequent iterations. good. In that case, since the detected area information filled with all zeros is used for the first time, a detector that receives only images as input may be used. The detector used for the first time is trained by inputting only images.

Also, during the second and subsequent iterations, if a plurality of detectors are learned and prepared in advance, the detectors to be used may be switched depending on the number of times. For example, as the number of repetitions increases, by using a trained detector with a larger number of mask areas during learning, more appropriate detection part data can be obtained even when there are many mask areas.

(6) In the above embodiment, when creating a mask area from the target object area by the already detected area masking means 53, a constant multiple of the elliptical area inscribed in the target object area is used. An estimator for estimating a person region (instance mask) for each sample may be learned in advance, and the mask region may be generated using the estimator. In that case, the detected region data is also input from the object region storage means 42 to the detected region masking means 53 via the object region calculation means 55, if necessary. In particular, by estimating the instance mask from the detected part data, positional information of the part that is more detailed than the target area can be obtained, so the instance mask can be estimated more accurately. Furthermore, since a human region can be estimated to take any shape, not just a predetermined shape such as a rectangle or an ellipse, it becomes easier to calculate an appropriate mask region even when concealment is complicated. Further, the estimator at this time may be an estimator that also inputs images.

Further, the instance mask may be detected by the key point detection means 54 instead of the detected area masking means 53 at the same time as the detection of body part data. In that case, the detector receives the image as input and detects part data and an instance mask corresponding to the part data. In other words, part data and an instance mask associated therewith are obtained for each sample. The learning means 51 uses the instance mask as a target value in addition to the imparted data to learn the detector. The learning data storage means 40 stores the instance mask in addition to the learning image and the imparted data.

(7) In the above embodiment, the upper limit of the number of repetitions and the lower limit of the number of detection sites are used as the repetition termination conditions, but the condition of the number of detection sites may be excluded from the repetition termination conditions. In that case, although unnecessary repetition occurs, the amount of calculations is leveled out, making control easier.

(8) In the above embodiment, the ReLU function is used as the activation function in the detector, but a tanh function, a sigmoid function, or the like may be used as the activation function. Further, a configuration having a shortcut structure such as that used in ResNet (residual network) may be used.

1 object recognition device, 2 photographing section, 3 communication section, 4 storage section, 5 image processing section, 6 display section, 7 operation input section, 30 photographed image acquisition means, 31 recognition result output means, 40 learning data storage means , 41 learning model storage means, 42 object area storage means 42, 43 detector storage means, 50 learning mask means, 51 learning means, 53 detected area masking means 53, 54 key point detection means, 55 object area calculation Means 55.

Claims (6)

An object recognition device that recognizes predetermined information that is information regarding the position of each of the objects from measurement data obtained by measuring a plurality of objects,
Masked sample data obtained by masking a mask area that includes a part of the area of the sample in the sample data in which the sample of the target object was measured, and information indicating the position of the mask area on the sample data. a detector storage means storing a detector generated in advance by learning with the predetermined information of the sample in the region including the mask region as an output target value;
The first target object is detected in the measurement data in which a first target object and a second target object partially blocked by the first target object are measured . a masking unit that generates mask processing data consisting of masked measurement data obtained by masking a detected area, which is a region, and information indicating a position of the detected area on the measurement data ;
a detection means for inputting the mask processing data into the detector to detect the predetermined information of the second object in a region including the already detected region in the measurement data ;
An object recognition device comprising: The detector storage means stores the detector that has performed the learning even when the sample data is inputted in a state before the mask processing is performed,
The detection means inputs the measurement data to the detector instead of the mask processing data and detects the predetermined information of the target in the measurement data when there is no detected area;
The object recognition device according to claim 1, characterized in that: 3. The object recognition device according to claim 1, wherein the predetermined information is information regarding a position of a part constituting the object. The object recognition device according to any one of claims 1 to 3, further comprising a learning unit that generates the detector through the learning based on the sample data. An object recognition method that recognizes predetermined information that is information regarding the position of each of the objects from measurement data obtained by measuring a plurality of objects, the method comprising:
Masked sample data obtained by masking a mask area that includes a part of the area of the sample in the sample data in which the sample of the target object was measured, and information indicating the position of the mask area on the sample data. a step of preparing a detector generated in advance by learning with the predetermined information of the sample in the region including the mask region as an output target value;
The first target object is detected in the measurement data in which a first target object and a second target object partially blocked by the first target object are measured . a step of generating mask processing data consisting of masked measurement data obtained by masking a previously detected region , which is a region, and information indicating a position of the detected region on the measurement data ;
inputting the mask processing data into the detector to detect the predetermined information of the second object in a region including the already detected region in the measurement data;
An object recognition method characterized by comprising: A program that causes a computer to perform a process of recognizing predetermined information , which is information regarding the position of each of the objects, from measurement data obtained by measuring a plurality of objects, the program comprising:
the computer,
Masked sample data obtained by masking a mask area that includes a part of the area of the sample in the sample data in which the sample of the target object was measured, and information indicating the position of the mask area on the sample data. Detector storage means storing a detector generated in advance by learning with the predetermined information of the sample in the region including the mask region as an output target value;
The first target object is detected in the measurement data in which a first target object and a second target object partially blocked by the first target object are measured . a masking unit that generates mask processing data consisting of masked measurement data obtained by masking a previously detected region, which is a region, and information indicating a position of the detected region on the measurement data ;
Detection means for inputting the mask processing data into the detector to detect the predetermined information of the second object in a region including the already detected region in the measurement data ;
An object recognition program characterized by functioning as an object recognition program.

Images