JP2021051376A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To enable appropriate collation of a person even when the person is partially masked by another object.SOLUTION: An image processing apparatus includes: detection means of detecting a feature point group in association with regions of an object from an image obtained by imaging the object having a plurality of regions; acquisition means of acquiring reliability which indicates a degree of probability of the region corresponding to the feature point, for each of the detected feature points; extraction means of extracting feature quantity for identifying an object for each of the regions corresponding to the feature points, from the image; and recognition means of recognizing the object by comparing the extracted feature quantity with feature quantity of the object registered in advance, in accordance with the reliability acquired for each of the feature points.SELECTED DRAWING: Figure 1

Description

The present invention relates to the detection of a person in an image.

In a surveillance camera system, there is a technique of detecting an object such as a person from a camera image and determining whether or not it is the same as an object detected by another camera. When the object to be identified is a person, the object is first detected in the camera image. Next, collation features representing the features unique to the object are extracted from the area of the object. Then, by comparing the matching features of the objects detected by different cameras, it is possible to identify whether or not these objects are the same.

In Non-Patent Document 1, joint points are extracted from an image of a person, and image features in the vicinity of the joints are extracted for each joint. Based on the image features extracted for each joint, the matching features of the whole body are generated.

C. Su et al. “Pose-driven Deep Convolutional Model for Person Re-identification,” IEEE, 2017

In the method of Non-Patent Document 1, when a part of a person is shielded and cannot be seen, the image feature extracted from the peripheral area of the shielded joint may not include the image feature used for person matching. high. Therefore, in the method of Non-Patent Document 1, there is a high possibility that the collation of a partially shielded person will fail. The present invention has been made in view of the above problems, and an object of the present invention is to enable appropriate matching of a person even in a situation where a part of the person is shielded by another object.

In order to achieve the object of the present invention, a detection means for detecting a feature point group corresponding to a part of the object from an image obtained by capturing an object having a plurality of parts, and each of the detected feature points. An acquisition means for acquiring the reliability indicating the certainty of the part corresponding to the feature point, and an extraction means for extracting the feature amount for identifying an object for each part corresponding to the feature point from the image. Each feature point has a recognition means for recognizing the object by comparing the extracted feature amount with the feature amount of the object registered in advance according to the acquired reliability. It is characterized by.

According to the present invention, even in a situation where a part of a person is shielded by another object, the person can be appropriately collated.

Block diagram showing a functional configuration example of the image display device of the embodiment Block diagram showing a functional configuration example of the image feature determination unit Block diagram showing a hardware configuration example Embodiment A flowchart showing the flow of processing executed by the image processing apparatus. Flowchart showing the flow of processing executed by the image processing device Flowchart showing the flow of processing executed by the image processing device The figure explaining an example of the correction of the characteristic point of the waist The figure explaining an example of correction of a feature point of a foot The figure explaining the process of determining the area of an object Flowchart showing the flow of processing executed by the image processing device The figure explaining the process of correcting a feature point outside a partial image area. The figure explaining the configuration example of the neural network Flowchart showing the flow of processing to be trained by the neural network The figure explaining the screen display example The figure explaining the example of the part in a face The figure explaining the configuration example of the sub-network The figure explaining the configuration example of the image integration subnetwork The figure explaining an example of the shielding part of a person

Hereinafter, embodiments of the present invention will be described.

<Embodiment 1>
FIG. 3 shows an example of the hardware configuration of the present embodiment. In FIG. 3, 301 is an image pickup device (imaging means) for converting a subject image from light into an electric signal, which is composed of a CCD, CMOS, or the like. Reference numeral 302 denotes a signal processing circuit that processes a time-series signal related to a subject image obtained from the image sensor 301 and converts it into a digital signal. 301 and 302 are connected to the bus as cameras. Reference numeral 303 denotes a CPU, which controls the entire apparatus by executing a control program stored in the ROM 304. Reference numeral 304 denotes a ROM, which stores a control program executed by the CPU 303 and various parameter data. When the control program is executed by the CPU 303, the device is made to function as various means for executing each process shown in the flowchart described later. Reference numeral 305 is RAM, which stores images and various information. Further, the RAM 305 functions as a work area of the CPU 303 and a temporary save area for data. Reference numeral 306 is a display. Reference numeral 307 is a pointing device such as a mouse or an input device such as a keyboard, which receives input from a user. Reference numeral 308 is a communication device such as a network or a bus, which communicates data and control signals with other communication devices. In the present embodiment, the processing corresponding to each step of the flowchart described later is realized by software using the CPU 303, but a part or all of the processing is realized by hardware such as an electronic circuit. It doesn't matter. Further, the image display device of the present invention may be realized by using a general-purpose PC by omitting the image pickup element 301 and the signal processing circuit 302, or may be realized as a dedicated device. Further, software (program) acquired via a network or various storage media may be executed by a processing device (CPU, processor) such as a personal computer.

Prior to the description of the embodiment, the terms will be described. A feature point is a point associated with a structural unit of an object composed of a plurality of parts. In the following description, the feature points are specifically the positions (two-dimensional coordinates) of the joints of a person in the image. The reliability is calculated for each of the detected feature points, and is a real number from 0 to 1 indicating the likelihood that the portion corresponding to the feature point exists on the image. For example, when detecting the position of a person's head as a feature point, the reliability increases if the person's head is clearly shown in the image. Conversely, if the head appears hazy, or if the head is shielded by some other object, the reliability of the feature points corresponding to the head is reduced. That is, it indicates the certainty that the position indicated by the feature point is the portion corresponding to the feature point. The present embodiment will be described by taking a person as an example of the object to be monitored, but the present embodiment is not limited to this, and other objects such as animals and cars may be used. That is, any structure composed of a plurality of parts can be applied. In this embodiment, a person is identified by using the feature amount of the whole body of the person. On the other hand, a person may be identified using a face, and in this case, it is known by names such as "face recognition", "face matching", and "face search".

The configuration of this embodiment is shown in FIG. In this embodiment, the image acquisition unit 101, the first detection unit 102, the feature group unit 103, the second detection unit 104, the feature point storage unit 105, the area determination unit 106, the image extraction unit 107, and the image feature extraction unit 108 , A recognition unit 109, a display unit 110, a learning unit 111, and an object storage unit 112.

The image acquisition unit 101 acquires an image frame obtained by capturing an image of an object having a plurality of parts from a camera. The first detection unit 102 detects the position of the feature point of the object and its reliability from the image frame. Details of the method for detecting the position of a person's joint in an image and its reliability will be described later. The feature group determination unit 103 determines a feature group for detecting feature points whose reliability is less than a predetermined value, based on the position and reliability of the feature points detected by the first detection unit 102. The combination of the feature point groups is prepared in advance, and is determined from among them according to the condition of the reliability of the feature points. The specific method will be described later. When the reliability of the predetermined feature points among the feature points detected by the first detection unit is smaller than the predetermined value, the second detection unit 104 uses a method different from that of the first detection means from the image. The predetermined feature point is detected. The feature points are detected by using the relative positional relationship between the feature points. The specific method will be described later. The feature point storage unit 105 stores the detected feature points. The area determination unit 106 determines the area where the object exists from the feature points. Among the detected feature points, a region in which the object to be extracted for image feature exists is determined by using a combination of specific feature points determined in advance. The image extraction unit 107 cuts out a region determined by the region determination unit from the image frame. The image feature extraction unit 108 extracts an image feature for identifying a person from the cut out partial image by using a neural network or the like. The recognition unit 109 performs image recognition using the extracted image features. In this embodiment, a person is identified as image recognition. Specifically, by comparing the extracted image features with each other, it is determined whether or not the feature amounts belong to the same person. Details will be described later. The display unit 110 displays the result of image recognition on the screen. The learning unit 111 learns a neural network or the like used for image feature extraction by the image feature extraction unit 108. The object storage means 112 stores information on the object used by the recognition unit 109.

A configuration example of the image feature extraction unit 108 of FIG. 1 is shown in FIG. The image feature extraction unit 108 is composed of an out-of-area feature point correction unit 202, an object part extraction unit 203, an intermediate image feature extraction unit 204, a reliability conversion unit 205, a feature integration unit 206, and an image feature output unit 207.

The out-of-area feature point correction unit 202 corrects the feature points outside the partial image area among the feature points extracted by the feature point extraction unit 102 in FIG. The object part extraction unit 203 extracts an object part (part) from the image. The intermediate image feature extraction unit 204 extracts the first image feature (intermediate image feature) from the image and the part of the object. The reliability conversion unit 205 applies the conversion process to the reliability of the feature points extracted by the feature point extraction unit 102. The feature integration unit 206 integrates the output of the intermediate image feature extraction unit 204 and the output of the reliability conversion unit 205. The image feature output unit 207 generates an image feature from the output of the feature integration unit 206.

The operation of this image processing apparatus will be described with reference to the flowchart of FIG. The process shown in the flowchart of FIG. 4 is executed by the CPU 303 of FIG. 3, which is a computer, according to a computer program stored in the ROM 304.

Step 401 acquires an image frame from the camera. This step corresponds to the operation of the image acquisition unit 101 of FIG.

Step 402 detects a plurality of feature points associated with the parts of the object from the image of the object having the plurality of parts acquired in step 401 (first detection method). This step corresponds to the operation of the first detection unit 102 in FIG. Further, in step 402, the image frame is input, and a plurality of feature points of the person existing in the image and their reliabilitys are extracted. For each of the detected feature points, the reliability indicating the certainty that the feature points are reflected in the image is acquired. If the image processing target is a person, the joint position of the human body can be used as a feature point. The five feature points detected in this step are the crown, neck, hips, right ankle, and left ankle. Convolutional Pose Machines are used to detect feature points. (Shih-En Wei et al., "Convolutional Pose Machines," IEEE, 2016.). In this method, a trained model (neural network) is used to calculate a reliability map showing where each joint position is located on the image. The reliability map is a two-dimensional map, and if the number of joint points is P, there are P + 1 (one is a map corresponding to the background). In the reliability map of a certain joint point, the position with high reliability is regarded as the position where the joint point exists. The reliability is a real number from 0 to 1 indicating the likelihood that the feature point exists. The closer it is to 1, the higher the probability that the joint point exists. Since the joint points shielded by other objects are extracted from the non-human object, the plausibility as a human joint is reduced. Therefore, the reliability of the joint position is lower than that of a joint that is not shielded by other objects. On the other hand, joints that are not hidden by other objects are well extracted from the person, so that the reliability of the joints is high.

As a method for detecting the feature points of the object and its reliability, a method other than Convolutional Pose Machines may be used. For example, a rule-based method may be used to identify each breakpoint using image features extracted for each breakpoint of the human body. Alternatively, the image feature of the head may be extracted from the image, and the position of the body may be estimated from the position where the head is extracted. Further, in the present embodiment, the joint points of the human body are used as the feature points, but if the image processing target is a face, the face feature points can be used. As the facial feature points, the center and end points of parts such as eyes, eyebrows, nose, mouth, and ears, points on the contour line, points on the contour line of the entire face shape, and the like can be used.

Step 403 determines a feature point cloud to be used in the second detection method. Step 403 corresponds to the operation of the feature group determination unit 103 of FIG. The feature point group determined in step 403 is used in the second detection method. A plurality of combinations of patterns are prepared for the feature point group, and the feature point group is selected and determined according to the reliability condition of the feature point. It will be used in the second detection method in a later step 404. The feature point group includes feature points (here, head, neck or waist) used to determine the corrected position. In the present embodiment, the feature points to be corrected as predetermined feature points are the waist, the right ankle, and the left ankle. Since the correction of the right ankle and the correction of the left ankle are performed in the same procedure, only the correction of the right ankle will be described. Hereinafter, the ankle on one side to be processed is simply referred to as "ankle".

The operation of step 403 will be described with reference to the flowchart of FIG. As will be described later, six types of feature point groups A1, A2, A3, B1, B2, and B3 are prepared in advance as candidates for the feature point group used for correction. One of the feature point groups A1, A2, and A3 related to the correction of the waist and one of the feature point groups B1, B2, and B3 related to the detection of the ankle in the second detection method are determined according to the conditions.

Although the details will be described later, the feature point group A1 is an empty set, and the detection result of the first detection unit is adopted as it is. Using the feature point group A2, the position of the waist is detected from the positions of the head and neck on the current frame. Using the feature point group A3, the current waist position is detected from the head and waist positions in the past frame. The feature point group B1 is an empty set, and the detection result of the first detection unit is adopted as it is. Using the feature point group B2, the position of the ankle is detected from the positions of the neck and waist on the current frame. Using the feature point group B3, the position of the ankle in the current frame is detected from the positions of the neck and ankle in the past frame.

Step 501 of FIG. 5 evaluates whether or not the reliability of the waist in the current frame determined in step 402 is equal to or higher than a predetermined threshold value. If it is above the threshold, the process proceeds to step 503, and if not, the process proceeds to step 502.

In step 502, it is evaluated whether or not the reliability of the waist in the past frame stored in the feature point storage unit 105 is equal to or higher than the threshold value. If it is above the threshold, the process proceeds to step 505, and if not, the process proceeds to 504. The past frame is an image frame acquired in step 401 of the previous repeating loop in the repeating loop of the flowchart of FIG. However, when the feature points in the past frame are not stored in the feature point storage unit 105, that is, when step 403 of FIG. 4 is executed for the first time, the process proceeds to step 504.

In step 503, the feature point group A1 is determined as the feature point group used in the second detection method, and the process proceeds to step 506. When the feature point group A1 is determined, it is a case where the waist feature points of the current frame are reliable, and it is not necessary to detect the waist feature points again in the subsequent processing.

In step 504, the feature point group A2 is determined as the feature point group used in the second detection method, and the process proceeds to step 506. When the feature point group A2 is determined, the hip joint points of both the current frame and the past frame are unreliable, and the position of the waist of the current frame is changed from the position of the head and neck of the current frame. Detect by processing.

In step 505, the feature point group A3 is selected as the feature point group used for the correction, and the process proceeds to step 506. When the feature point group A3 is selected, the waist feature points of the current frame are unreliable, but the waist feature points of the past frame are reliable, and the current frame is from the head and waist positions of the past frame. The waist position is corrected in the subsequent processing.

Step 506 evaluates whether the reliability of the ankle in the current frame determined in step 402 is greater than or equal to a predetermined threshold. If it is above the threshold, the process proceeds to step 508, and if not, the process proceeds to step 507.

In step 507, it is evaluated whether or not the reliability of the ankle in the past frame stored in the feature point storage unit 105 is equal to or higher than a predetermined threshold value. If it is above the threshold, the process proceeds to step 510, and if not, the process proceeds to 509. However, when the feature points in the past frame are not stored in the feature point storage unit 105, that is, when step 403 of FIG. 4 is executed for the first time, the process proceeds to step 509.

Here, the threshold values used in S501, S502, S506, and S507 are different values in this embodiment, but may be the same value.

In step 508, the feature point group B1 is selected as the feature point group used for the correction, and the processing of the flowchart of FIG. 5 is completed. When the feature point group B1 is selected, it means that the feature points of the foot in the current frame are reliable, and it is not necessary to detect the position of the foot in a later process.

In step 509, the feature point group B2 is selected as the feature point group used for the correction, and the processing of the flowchart of FIG. 5 is completed. When the feature point group B2 is selected, the foot position is unreliable in both the current frame and the past frame, and the foot position of the current frame is detected from the foot and waist positions of the current frame in the subsequent processing. To do.

In step 510, the feature point group B3 is selected as the feature point group used for the correction, and the processing of the flowchart of FIG. 5 is completed. When the feature point group B3 is selected, the feature points of the foot are unreliable in the current frame, but the feature points of the foot are reliable in the past frame, and the position of the current frame is determined from the positions of the neck and foot of the past frame. It will be detected in the subsequent processing.

In the above description of steps 506, 507, 508, 509, and 510, only one ankle (right ankle) was targeted, but the other ankle (left ankle) is also a feature point used in the second detection method. Determine the group. In order to detect the position of the ankle, it is preferable that the position of the ankle can be estimated from a feature point as close to the position of the ankle as possible. Therefore, when the waist position can be adopted (the waist position is highly reliable), the waist position is used to detect the ankle position. If the position of the waist is unknown (the position of the waist is unreliable), the position of the ankle is detected by using the position of the neck next to the waist and the position of the neck closest to the ankle. The following processing order is based on the above intention, but the order may be changed. Further, the feature group may be determined so as to detect only the position of the ankle without detecting the position of the waist.

In step 404 of FIG. 4, a predetermined feature point is detected by the second detection method using the feature point group determined in step 403. The process of step 404 corresponds to the second detection unit 104 of FIG. The operation of step 404 will be described with reference to the flowchart of FIG. In the process of FIG. 6, a predetermined feature point (position of the ankle) is detected based on the feature point groups A1, A2, A3, B1, B2, and B3 determined by the process of the flowchart of FIG.

Since the correction of the right ankle and the left ankle is performed in the same procedure as in step 403 of FIG. 4, only the detection of the right ankle will be described. Hereinafter, the ankle on one side to be processed is simply referred to as "ankle".

In step 601 of FIG. 6, it is determined which of the feature point groups A1, A2, and A3 related to the waist is selected. If the feature point group A1 is selected, the process proceeds to step 602, if the feature point group A2 is selected, the process proceeds to step 603, and if the feature point group A3 is selected, the process proceeds to step 604. In step 602, step 603, and step 604, the position of the feature point of the waist is detected by the second detection method.

Step 602 does not detect the position of the waist feature point. This is because the reliability of the waist feature points in the previous process is larger than a certain threshold value and is considered to be reliable.

Step 603 detects the position of the waist from the positions of the head and neck detected in the current image frame. The process will be described with reference to FIG. 7. As shown in FIG. 7A, the feature points of the crown 701, the neck 702, the waist 703, the right ankle 704, and the left ankle 705 are detected by step 402 of FIG. First, as shown in FIG. 7B, a straight line 706 connecting the head and the neck is calculated. Also, the distance between the head and neck is calculated from the respective position coordinates. Here, it can be assumed that the ratio of the distance between the head and the neck of the human body and the distance between the head and the waist is approximately the same, although there are individual differences. Therefore, the position of the waist is on a straight line connecting the head and the neck, and the ratio of the distance between the head and the neck and the distance between the head and the waist is detected so as to be a predetermined value. FIG. 7C shows an example of the corrected waist feature point 707. This predetermined ratio can be determined, for example, from the ratio of the average adult human body part.

Step 604 detects the current hip position from the head and hip positions in the past frame. First, the distance between the head and the waist is calculated from the feature points of the past frame stored in the feature point storage unit 105. Next, in the current frame, a straight line connecting the head and the neck is calculated in the same manner as in FIG. 7 (b). Here, it is assumed that the distance between the head and the waist in the past frame and the distance between the head and the waist in the current frame are approximately the same. Then, the position of the waist in the current frame is detected so that the position of the waist is on the straight line connecting the head and the neck and the distance between the head and the waist in the current frame is equal to the distance between the head and the waist in the past frame. ..

In step 605 of FIG. 6, it is determined which of the feature point groups B1, B2, and B3 related to the ankle is selected. If the feature point group B1 is selected, the process proceeds to step 606, if the feature point group B2 is selected, the process proceeds to step 607, and if the feature point group B3 is selected, the process proceeds to step 608. In step 607 and step 608, the position of the feature point of the ankle is detected. Step 606 does not detect the position of the ankle feature point.

Step 607 detects the position of the ankle from the position of the neck and hips on the current frame. The process will be described with reference to FIG. As shown in FIG. 8A, the feature points of the crown 801 and the neck 802, the waist 803, the right ankle 804, and the left ankle 805 are detected by step 402 of FIG. First, as shown in FIG. 8B, a straight line 806 (body axis) connecting the neck and the waist is calculated. Also, the distance between the neck and hips is calculated from the respective position coordinates. Here, it can be assumed that the ratio of the distance between the neck and the waist of the human body and the distance between the neck and the right ankle is approximately the same, although there are individual differences. Therefore, the position of the ankle is on a straight line connecting the neck and the waist, and the ratio of the distance between the neck and the waist and the distance between the neck and the ankle is detected to be a predetermined value. Figure 8 (c) on the left shows an example after detecting the feature points of the ankle 807.

Step 604 detects the position of the ankle in the current frame from the position of the neck and ankle in the past frame. First, the distance between the neck and the waist is calculated from the feature points of the past frame stored in the feature point storage unit 105. Next, in the current frame, a straight line (body axis) connecting the neck and the waist is calculated in the same manner as in FIG. 8 (b). Here, it is assumed that the distance between the neck and the ankle in the past frame and the distance between the neck and the ankle in the current frame are approximately the same. Then, the position of the ankle in the current frame is detected so that the position of the ankle is on the body axis and the distance between the neck and the ankle in the current frame is equal to the distance between the neck and the ankle in the past frame.

In the above description of steps 605, 606, 607, and 608, only the right ankle is targeted, but the left ankle is also detected in the same manner. By this process, even if the ankle portion is not detected well by the first detection unit due to occlusion or noise, a more probable ankle position can be detected.

In step 405 of FIG. 4, the region where the object exists is determined based on the detected feature points. This partial image area indicates an area in which a person exists in the captured image, and is used to specify an area for extracting the person image from the image frame in a later process. The operation of step 405 corresponds to the area determination unit 106 of FIG. The process of step 405 will be described with reference to FIG. 9A. As shown in FIG. 9A, the feature points of the crown, neck, waist, right ankle, and left ankle are present in the image frame 903. First, calculate the midpoint between the right and left ankles. Then, the straight line 901 (body axis) connecting the head and its midpoint is calculated. In the present embodiment, the partial image area is rectangular and the aspect ratio is predetermined. The rectangle 902 is determined so that the vertical direction of the rectangle is parallel to the body axis, the central axis of the rectangle is equal to the body axis, the upper side of the rectangle is in contact with the head, and the lower side of the rectangle is in contact with the ankle. At this time, a margin may be provided between the upper side of the rectangle and the head, and between the lower side of the rectangle and the ankle. For example, a margin having a size obtained by multiplying the distance (height) between the head and ankle by a certain coefficient may be provided. That is, the partial image area is determined based on the circumscribed rectangle of the feature point. In the present embodiment, the aspect ratio of the rectangle is fixed in order to facilitate input to the later neural network, but it may not be fixed depending on the configuration of the later processing. When the corrected joint position is used, it is possible that the part of the person is shielded or a lot of noise is generated in the region determined here. For example, as shown in FIG. 18, even when the part of the ankle is hidden by the shield 1803, it is determined as the area including the part of the person. By determining the region in this way, it is possible to determine the partial image region in which the arrangement of the parts of the human body in the rectangle is consistent. By making the arrangement of the parts consistent, there is an effect that the feature amount of each part that more reflects the characteristics of each part can be extracted in the feature amount extraction process performed in the subsequent stage.

In step 406 of FIG. 4, the partial image area determined in step 405 is cut out from the image frame as a person image. If the rectangle of the partial image area determined in step 405 is tilted, the image is rotated so that the rectangle stands upright. An example cut out from FIG. 9 (a) is shown in FIG. 9 (b). The operation of step 406 corresponds to the image extraction unit 107 of FIG.

In step 407, the corrected portion in the current frame is stored. The operation of step 407 corresponds to the feature point storage unit 105 of FIG.

In step 408, the feature amount is extracted from the partial image area (personal image). The operation of step 408 corresponds to the image feature extraction unit 108 of FIGS. 1 and 2. The operation of step 408 will be described with reference to the flowchart of FIG.

In step 1001 of FIG. 10, the out-of-area feature point correction unit 202 corrects the reliability of the feature points outside the partial image area based on the coordinates of the partial image area and the feature points. Step 1001 corresponds to the out-of-area feature point correction unit 202 of FIG. When the aspect ratio of the rectangle in the partial image area is fixed, the feature points may not be included in the partial image area, such as when the limbs are spread out. The human body part outside the partial image area is outside the range of feature extraction, and there is a problem that the accuracy of feature extraction in this part is lowered. Therefore, in order to mitigate the influence in a later step, an adjustment is made to reduce the reliability of the feature points outside the partial region. For example, in FIG. 11, the right ankle 1104 is outside the range of the rectangle 1106, which reduces the reliability of the feature points of the right ankle. In the present embodiment, the value obtained by multiplying the original reliability by a predetermined real value smaller than 1 is used as the corrected reliability. In this way, by reducing the reliability of the feature points outside the partial region, the problem of the decrease in the accuracy of feature extraction due to the placement of human body parts outside the partial region and the decrease in the accuracy of feature extraction due to shielding are reduced. The problem can be dealt with by a common process thereafter.

In step 1002, the feature amount is extracted from the reliability of the partial image area and the feature point. A neural network can be used to extract the features. FIG. 12 shows a configuration example of the neural network. The neural network of FIG. 12 inputs the image 1201 and the feature point reliability 1206, and outputs the image feature 1210. The neural network is composed of an image conversion subnet 1202, a reliability transformation subnet 1207, an integrated subnet 1208, and a feature output subnet 1209. The image conversion sub-network 1202 corresponds to the intermediate image feature extraction unit 204 of FIG. The reliability conversion sub-network 1207 corresponds to the reliability conversion unit 205 of FIG. The integrated subnetwork 1208 corresponds to the feature integrated section 206 of FIG. The feature output sub-network 1209 corresponds to the image feature output unit 207 of FIG.

Input data, intermediate data, and output data handled by the neural network are treated as tensors. A tensor is data expressed as a multidimensional array, and the number of dimensions is called the rank. A tensor with a rank of 0 is called a scalar, a tensor with a rank of 1 is called a vector, and a tensor with a rank of 2 is called a matrix. For example, an image having one channel (such as a grayscale image) can be treated as a size H × W rank 2 tensor or a size H × W × 1 rank 3 tensor. Further, an image having an RGB component can be treated as a tensor having a size of H × W × 3 and a rank of 3.

The data obtained by extracting the surface of the tensor cut at a certain position in a certain dimension and its operation are called slices. For example, by slicing a rank 3 tensor of size H × W × C at the cth position of the third dimension, a rank 2 tensor of H × W or a rank 3 tensor of H × W × 1 can be obtained. Be done.

A layer that performs a convolution operation on a certain tensor is called a convolution layer (abbreviated as Conv). The coefficient of the filter used for the convolution operation is called "weight". As an example, the convolution layer generates an H × W × D output tensor from an H × W × C input tensor.

A layer that multiplies a vector by a weight matrix and adds a bias vector is called a fully connected layer (abbreviated as FC). As an example, a vector of length D is generated from a vector of length C by applying a fully connected layer.

The operation of reducing the size of a tensor by dividing a tensor into intervals and taking the maximum value of that interval is called maximum pooling. When the average value of the section is taken instead of the maximum value, it is called average pooling. In the present embodiment, the maximum pooling is used, and the layer of the neural network that performs this is simply referred to as a pooling layer (abbreviated as Polling). In the present embodiment, the pooling layer outputs a tensor in which the size of the first dimension and the second dimension is half of the input. Specifically, an H / 2 × W / 2 × C output tensor is generated from an H × W × C input tensor.

In a neural network, the nonlinear function that is usually applied after the convolution layer is called the activation function. The activation function includes a rectified linear function (abbreviated as ReLU) and a sigmoid function. In particular, the sigmoid function has the property that the range of output values is 0 to 1. In this embodiment, ReLU is used as an activation function unless otherwise specified.

In a neural network, the operation of arranging and connecting tensors in a certain dimensional direction is called "connection".

Global average reporting will be described. In a rank 3 size H × W × C tensor, the average value of all the elements contained in each slice is taken for the slices at all positions in the third dimension. Then, by arranging the average values of C pieces, a vector of length C is generated. This operation is called Global average reporting.

In FIG. 12, the size of the image 1201 that is the input of the neural network is width W1, height H1, and the number of channels 3. That is, the image can be regarded as a H1 × W1 × 3 tensor.

The image conversion subnet 1202 converts the image 1201 into a feature map. The image conversion subnet 1202 is further composed of a preprocessing subnet 1203, a parts estimation subnet 1204, and an image integration subnet 1205.

The image conversion sub-network 1202 extracts a feature amount for identifying an object for each part corresponding to the detected feature point. Specifically, L. Like the paper by Zhao et al., It includes a module that estimates parts and extracts the features of the parts. The image conversion sub-network 1202 corresponds to the object part extraction unit 203 of FIG. (. The object part extraction unit 203 may be operated outside the neural network to provide information on the position and size of the part from the outside. Further, the object part extraction unit 203 and the first detection unit 102 in FIG. 1 may be used for each other, and the information derived from the output of the first detection unit 102 may be used as the output of the object part extraction unit 203. Well, the reverse may be done. The feature amount for each part extracted here is integrated as the total feature amount in the later processing. At that time, weighting is performed so that the feature amount of each part is reflected in the overall feature amount according to the reliability of each feature point. That is, it is suppressed that the feature amount extracted from the part corresponding to the feature point having low reliability contributes to the final recognition result. Feature points with low reliability may be obstructed by an object or have a lot of noise, and the features extracted from that part do not always indicate the characteristics of the part of the object. Because there is no such thing. By performing such processing, a feature amount that more reflects the features of the object can be generated, and the effect of improving the recognition accuracy of the object can be expected.

The image conversion subnet line 1202 can be composed of a sequence of one or more convolution layers (Conv) and a maximum pooling layer (Pooling). In the present embodiment, it is composed of a sequence such as "Conv, Conv, Polling, Conv, Polling, Conv, Polling, Conv". The outline of the configuration is shown in FIG. 16 (a). As a result of applying the image conversion subnetwork to the image, a tensor of H2 × W2 × C2 is obtained.

The parts estimation sub-network 1204 takes the output of the image conversion sub-network 1202 as an input, and outputs a tensor of H2 × W2 × P1 which is a feature map. Here, P1 is the number of parts to be estimated, and may be an arbitrary number determined in advance. The slice (tensor having a size of H2 × W2 × 1) at the position p in the third dimension of this tensor is a mask image showing the existence position of the p-th part. Each pixel takes a value from 0 to 1, and the closer it is to 1, the higher the probability that the part exists at that position. The parts estimation subnet 1204 is composed of one convolution layer and a sigmoid function. The outline of the configuration is shown in FIG. 16 (b). The configuration of the parts estimation network is not limited to this, and may be configured by a plurality of convolution layers.

The image integration subnet 1205 integrates the output of the image transformation subnet 1202 with the parts estimation subnet 1204. FIG. 17 shows the processing flow. First, C2 slices 1702 (tensors of size H2 × W2 × 1) at position p in the third dimension of the output tensor 1701 of the parts estimation subnetwork are copied and connected in the third dimension, and the size is increased. Extend to H2 x W2 x C2 tensor 1703. Then, each element of this tensor is multiplied by each element of the output tensor 1704 of the image conversion subnet 1202 to generate a new tensor 1705 (size H2 × W2 × C2). Then, by applying global average reporting to this tensor, a vector 1706 of length C2 is generated, and by further applying a fully connected layer, a vector 1707 of length C3 is generated. This process is applied to the channel p of all parts to generate a vector 1708 that concatenates each generated vector. That is, the length of the vector 1708 generated by the image integration subnet is (C3) P1. In the present embodiment, the data to be integrated is a vector, but the vector is a kind of tensor, and even when the data to be integrated is a tensor of the second or higher order, it may be integrated by combining in the same manner.

The feature point reliability 1206 is a vector of length C4. In this embodiment, since the number of feature points detected in step 402 in FIG. 4 is 5, C4 = 5.

The reliability conversion subnet 1207 converts the feature point reliability 1206 into a vector of length C5. The reliability conversion subnetwork 1207 can be composed of 0 or more fully connected layers. In this embodiment, there is one fully connected layer. The outline of the configuration is shown in FIG. 16 (c).

The integrated subnet 1208 integrates the output vector of the image integrated subnet 1205 with the output vector of the reliability transformation subnet 1207. The integrated subnet network 1208 outputs a vector of length C6. In this embodiment, these two vectors are connected. The outline of the configuration is shown in FIG. 16 (d). Therefore, C6 = (C3) P1 + C5.

The feature output subnet 1209 takes the output vector of the integrated subnet 1208 as an input and outputs the image feature 1210 which is a vector of length C7. Features The output subnetwork 1209 can consist of one or more fully coupled layers. In this embodiment, it is composed of two fully connected layers. The outline of the configuration is shown in FIG. 16 (e). This image feature is also referred to as a "matching feature", a "personal feature", a "descriptor", or an "embedding".

In step 409 of FIG. 4, the feature amount of the person image extracted in step 408 is compared with the feature amount stored in the person database. The person database is a storage means in which cut-out images and feature amounts of N people to be identified are registered in advance. An image of the person to be identified is taken in advance, and the image is cut out and the feature amount is extracted by the same method as in steps 402 to 408 and saved. The person database corresponds to the object storage means 112 of FIG. In step 409, the distance between the feature amount of the person in the person database and the feature amount of the person image extracted in step 408 is calculated. Then, the people in the person database are sorted in order of distance, and the person with the shortest distance is placed at the beginning of the line. Step 409 corresponds to the processing of the recognition unit 109 in FIG. In this embodiment, the Euclidean distance is used for comparison of features. The feature quantities may be compared by other methods, other distance indexes such as L1 distance and cosine distance, and may be compared by using machine learning such as metric learning or neural network.

In step 410 of FIG. 4, the person corresponding to the person in step 409 is displayed on the screen. Step 410 corresponds to the processing of the image display unit 110 of FIG. An example of the display screen is shown in FIG. The display screen 1401 is composed of a query 1402 and a gallery 1403. Query 1402 is an image of the person to be searched, and displays the person image cut out in step 406. The gallery 1403 is a list of search results, and displays the top five images in order of the images in the person database sorted in order of distance in step 409. At this time, the top five people may be displayed, or only the people whose distance is less than or equal to a predetermined threshold value may be displayed from among the five people. The image displayed in the gallery may be cut out by the same method as in step 407 from step 401 of FIG. 4, or may be cut out by another method. As shown in FIG. 14, a marker indicating the position of the detected feature point may be superimposed and displayed on the image of the person in the query and the gallery.

Step 411 of FIG. 4 determines whether or not to end the processing of the flowchart. In the present embodiment, when the number of times of execution of step 411 exceeds the specified number of times, it is determined that the process ends. If not, the process proceeds to step 401 and the flowchart processing is continued.

<Neural network learning>
The method of learning the neural network used in the image feature extraction unit 108 of FIG. 1 will be described with reference to the flowchart of FIG. The processing of the flowchart of FIG. 13 corresponds to the learning means 111 of FIG.

The structure of the neural network is shown in FIG. 12 as described above. The neural network inputs the image 1201 and the feature point reliability 1206, and outputs the image feature 1210.

Neural networks are learned by triple loss. (F. Shroff et al. "Face Net: A United Embedding for Face Recognition and Clustering," arXiv: 1503.03832). In triple loss, a triplet consisting of a sample called an anchor sample, a sample of the same person as the anchor called a positive sample, and a sample of a person different from the anchor called a negative sample is used. The network is updated by calculating the loss function by comparing the features obtained from the anchor sample, positive sample, and negative sample.

Step 1301 of FIG. 13 initializes the weights of the convolution layer and the fully connected layer constituting the network. In this embodiment, a random number is used as the initial value of the weight.

In step 1302, learning data is randomly acquired from the learning data group. One training data is a triplet and includes one anchor sample, one positive sample, and one negative sample. Anchor sample, positive sample, and negative sample are each composed of an image and feature point reliability. The image and the feature point reliability are generated by the same procedure as that input to the neural network used in the flowchart of FIG.

Step 1303 updates the network with the training data. First, the current state network is applied to the anchor sample, the positive sample, and the negative sample, and the features are calculated for each. For these three features, the loss is calculated by triple loss. Then, the weight in the network is updated by the backpropagation method.

It is determined in step 1304 whether to end learning. When step 1304 is executed a predetermined number of times, it is determined that the step 1304 is finished, and the series of processes of the flowchart of FIG. 13 is finished. If it is determined that the process does not end, the process proceeds to step 1302.

According to the present embodiment, the feature group determination unit 103 and the second detection unit 104 can detect the bad feature points from the good feature points again. Therefore, even in a situation where a part of the object is shielded by another object or is subjected to disturbance, the effect of reducing the error of the object area determination by the area determination unit 106 can be expected.

It is assumed that the reliability of the feature points acquired by the first detection unit 102 is lower than that in the normal state in the region where a part of the object is shielded by another object or the region is disturbed. it can. At this time, it is considered that the quality of the image features for image recognition extracted from these local regions also deteriorates at the same time. Therefore, by using the information on the reliability of the feature points as an index showing the reliability of a certain local region in the image feature extraction unit 108, the effect of reducing the deterioration of the quality of the image features can be expected. Therefore, the effect of improving the accuracy of image recognition can be expected.

Step 1001 in FIG. 10 reduces the reliability of feature points outside the partial image area. The human body part outside the partial image area is outside the range of feature extraction, and there is a problem that the accuracy of feature extraction in this part is lowered. Therefore, in order to reduce the influence in a later step, the effect of reducing the deterioration of the quality of the image feature can be expected by reducing the reliability of the feature points outside the partial region.

In steps 403 and 404, the feature points used for the correction are selected and the feature points are corrected by using not only the feature points of the current frame but also the feature points of the past frame. By using the feature points of the past frame, the effect of improving the correction accuracy of the feature points can be expected even when the reliability of the feature points is low in the current frame.

In step 403, the feature points are selected in a predetermined order. By preferentially selecting the feature points that are expected to have better accuracy in the correction of the feature point positions in step 404, the effect of correcting the feature point positions more accurately can be expected.

In step 404, the feature points are corrected in a predetermined order. Here, the feature points are corrected in the order of waist and legs. This is because the body parts of a person are connected in the order of neck, waist, and legs. First, the hip position can be corrected, and then the legs can be corrected using the more correct hip position. By comparing the feature points in a predetermined order in this way, the effect of being able to correct the feature point positions more accurately can be expected.

In step 404, the position of the feature point is corrected from the relative positional relationship between the feature points. In the embodiment, the feature points are corrected based on the ratio of the distances between the feature points and the straight line (body axis) obtained from the feature points. In this way, by using the prior knowledge about the structure of the object, the effect of being able to correct the position of the feature point more correctly can be expected.

<Modified Example of Embodiment 1>
The feature points extracted in step 402 are not limited to the crown, neck, hips, right ankle, and left ankle, but may be other parts such as wrists, elbows, and knees. In addition, it does not necessarily have to be on the body part, and other points determined by the positional relationship of the body part such as the midpoint between the right ankle and the left ankle and the intersection of the lines connecting the body axis and the left ankle / right ankle may be used. Absent.

In step 604, the position of the waist in the current frame is corrected from the distance between the head and the waist in the past frame, but other methods may be used. The position of the waist of the current frame may be corrected from the difference in the position coordinates of the head and the waist in the past frame. For example, as a difference between the position coordinates of the head and the waist in the past frame, it is assumed that the x-coordinate and the y-coordinate of the waist are X-pixel and Y-pixel larger than the x-coordinate and the y-coordinate of the head, respectively. The position of the waist may be corrected in the current frame so as to be equal to the difference in the position coordinates between the head and the waist in the past frame. Further, instead of the difference in the position coordinates of the head and the waist, the difference in the position coordinates of the neck and the waist may be used.

In step 607, the ratio of the distance between the neck and the waist of the human body and the distance between the neck and the right ankle (or the left ankle) is used, but the ratio is not limited to this, and the ratio between other feature points may be used. As an example, the head may be used instead of the neck, such as the ratio of the head-to-waist distance to the head-to-right ankle (or left ankle) distance. As another example, the ratio of the distance between the head and the neck and the distance between the waist and the right ankle (or left ankle) may be used. The same applies to step 608.

In step 607, the right ankle and the left ankle were corrected so as to be on the body axis. Not limited to this, correction may be made by moving the right ankle (or left ankle) in the body axis direction so that the ratio between the feature points becomes a predetermined value. The same applies to step 608.

In the area determination unit 106, the partial image area is rectangular, but other shapes may be used. For example, it may be a polygon or may be surrounded by a curve. Instead of a figure, a mask image that distinguishes an object area from another area may be used.

The structure of the neural network of the first embodiment is not limited to this. For example, another subnet may be inserted between subnets. Moreover, the branch structure of the network may be different. Regarding the configuration of the sub-network, the types and numbers of components such as the convolution layer, the pooling layer, and the fully connected layer may be different.

In the integrated subnet 1208 of FIG. 12, the two vectors are integrated by combining the two vectors, but other calculation methods may be used. For example, if the sizes of the two vectors are the same, multiplication or addition between the elements of the vectors may be used instead.

Although the reliability conversion unit 205 of FIG. 2 is implemented as the reliability conversion sub-network 1207 as shown in FIG. 12, the reliability conversion unit 205 may be provided outside the neural network. For example, processing such as normalization processing or conversion processing may be performed on the reliability of the feature points outside the neural network, and the processing result may be used as one of the inputs of the neural network.

In step 403 and step feature points in FIG. 4, the feature point group to be used for correction was selected from the current frame and the previous frame, and the feature points were corrected. Not only the previous frame but also the previous frame may be used to select the feature point group and correct the feature points. Further, a frame of 3 or more may be used in combination with the current frame.

Although the image feature extraction unit 108 is composed of a neural network, a method other than the neural network may be used. For example, HOG (Histogram of Oriented Gradients) features and LBP (Local Binary Pattern) features may be extracted, and image features may be determined based on these. Alternatively, parts may be estimated from HOG features and LBP features.

Although the straight line 706 of FIG. 7 was calculated from the head and neck in step 603 of FIG. 6, a straight line may be calculated from only the head or neck. For example, if it can be assumed that the body axis of a person is parallel to the y-axis of the image frame, then the straight line can be assumed to be parallel to the y-axis of the image frame, starting from either the neck or the head. You can calculate a straight line. Similarly, in step 405 of FIG. 4, the straight line 901 of FIG. 9 is calculated from a plurality of points, but it may be calculated from one point.

In S1001 of FIG. 10, the value obtained by multiplying the original reliability by a predetermined real value smaller than 1 is used as the corrected reliability, but other methods may be used. The method for updating the reliability is not limited to this, and the reliability may be set to 0, a predetermined real value may be subtracted from the reliability, or another method may be used.

As described above, by the process described in the first embodiment, the person can be appropriately collated even in a situation where a part of the person is shielded by another object.

<Embodiment 2>
In the first embodiment, the whole body of the person is targeted for image processing, but the face may be targeted for image processing instead. In the second embodiment, only the difference from the first embodiment will be described.

When the face is targeted, the facial feature points are detected in step 402 of FIG. It is illustrated in FIG. Here, it is assumed that the right eye 1501, the left eye 1502, the nose 1503, the right end 1504 of the mouth, and the left end 1505 of the mouth are detected as feature points.

In the second embodiment, consider the case where the feature points of the right eye are corrected from the nose and the mouth in steps 403 and 404. The process for the left eye is the same as that for the right eye.

The process of step 403 will be described. First, the reliability of the feature points of the right eye is evaluated. If the reliability is equal to or higher than the threshold value, the feature point group C1 is selected. When the reliability is smaller than the threshold value, the feature point group C2 is selected if the reliability of the right eye in the past frame is not equal to or higher than the threshold value, and the feature point group C3 is selected if the reliability is higher than the threshold value.

The process of step 404 will be described. If the feature point group used for correction is the feature point group C1, the position of the right eye is not corrected. If it is the feature point group C2, the position of the right eye of the current frame is corrected so as to be close to the arrangement of the facial parts of the average person from the positional relationship between the nose and the right end of the mouth and the left end of the mouth of the current frame. .. If it is the feature point group C3, the position of the right eye of the current frame is corrected so as to be close to the arrangement of the right eye, the nose, the right end of the mouth, and the left end of the mouth of the past frame.

The processing of the other steps is the same as the processing of the first embodiment if the feature points extracted from the whole body are replaced with the facial feature points.

In the second embodiment, the facial feature points are the right eye, the left eye, the nose, the right end of the mouth, and the left end of the mouth. It may be a feature point. Then, the processing of step 403 and step 404 may be changed accordingly.

According to the second embodiment, the effect of cutting out the face image from the image frame and improving the performance of face recognition can be expected. For example, it is effective in a case where the face is partially covered with accessories such as sunglasses and a mask, and a case where a part of the face is temporarily hidden by a hand or the like.

The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network for data communication or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. Further, the program may be recorded and provided on a computer-readable recording medium.

101 Image acquisition unit 102 First detection unit 103 Feature group determination unit 104 Second detection unit 105 Feature point storage unit 106 Area determination unit 107 Image extraction unit 108 Image feature extraction unit 109 Recognition unit 110 Display unit 111 Learning unit 112 Object Memory

Claims (10)

A detection means for detecting a feature point cloud corresponding to a part of the object from an image obtained by capturing an object having a plurality of parts.
For each of the detected feature points, an acquisition means for acquiring the reliability indicating the certainty of the portion corresponding to the feature point, and
An extraction means for extracting the feature amount for identifying an object for each part corresponding to the feature point from the image, and
Each feature point has a recognition means for recognizing the object by comparing the extracted feature amount with the feature amount of the object registered in advance according to the acquired reliability. An image processing device characterized by. The image processing apparatus according to claim 1, wherein the extraction means extracts the feature amount by a neural network that outputs the feature amount by inputting a partial image including the person and the reliability. Claim 1 is characterized in that, when the feature point is located outside the range of a predetermined region, the acquisition means acquires the reliability so that the reliability of the feature point is smaller than the predetermined value. Or the image processing apparatus according to 2. The object is a human body
The image according to any one of claims 1 to 3, wherein the recognition means identifies the person by comparing the feature amount of the person registered in advance with the extracted feature amount. Processing equipment. The image processing apparatus according to claim 4, wherein the feature point is a position of a joint of a person. The image processing apparatus according to claim 4, wherein the feature point is a position of a facial portion of a person. Further having an integration means for integrating the reliability and the extracted features.
The recognition means according to any one of claims 1 to 6, wherein the recognition means recognizes the object by comparing the integrated feature amount with the feature amount of the object registered in advance. Image processing equipment. An image processing device that extracts image features from an image.
A feature point extraction means for extracting a feature point cloud of an object having a plurality of parts from the image and a reliability indicating the certainty of the part corresponding to the feature point.
An image feature extraction means for extracting a first image feature from the image for each part corresponding to the feature point, and an image feature extraction means.
A conversion means for converting the reliability and the first image feature into a form that can be integrated.
An image processing apparatus having an output means for outputting a second image feature of the object in which the intermediate image feature converted by the conversion means and the reliability are integrated. A program for causing a computer to function as each means included in the image processing apparatus according to any one of claims 1 to 8. A detection step of detecting a feature point cloud corresponding to a part of the object from an image obtained by capturing an object having a plurality of parts.
For each of the detected feature points, an acquisition step of acquiring the reliability indicating the certainty of the part corresponding to the feature point, and
An extraction step of extracting the feature amount for identifying an object for each part corresponding to the feature point from the image, and
Each feature point has a recognition step of recognizing the object by comparing the extracted feature amount with the feature amount of the object registered in advance according to the acquired reliability. An image processing method characterized by.

Images