JP7479809B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to detecting people in images.

In surveillance camera systems, there is technology that detects objects such as people from camera images and determines whether they are the same as objects detected by other cameras. When the object to be identified is a person, the object is first detected from the camera image. Next, matching features that represent the object's unique characteristics are extracted from the object's area. Then, by comparing the matching features of objects detected by different cameras, it is possible to identify whether these objects are the same.

In Non-Patent Document 1, joint points are extracted from an image of a person, and then image features near each joint are extracted. Matching features for the entire body are generated based on the image features extracted for each joint.

C. Su et al. "Pose-driven Deep Convolutional Model for Person Re-identification," IEEE, 2017

In the method of Non-Patent Document 1, if a person is partially occluded and cannot be seen, the image features extracted from the area surrounding the occluded joint are likely to not include the image features used for person matching. Therefore, the method of Non-Patent Document 1 is likely to fail to match a person who is partially occluded. The present invention has been made in consideration of the above problems, and aims to make it possible to properly match people even in situations where a person is partially occluded by another object.

In order to achieve the object of the present invention, the image processing device of the present invention is characterized in having an acquisition means for acquiring from an image of a person a plurality of feature points corresponding to a plurality of body parts of the person and a reliability indicating the likelihood that each feature point is the said body part, an extraction means for extracting a feature amount obtained by weighting and integrating the feature amounts for each of the plurality of body parts corresponding to the plurality of feature points in accordance with the reliability acquired by the acquisition means, as a matching feature amount for comparison with a feature amount of a person registered in advance, and a recognition means for determining that the person in the image is the same person as the person registered in advance if the feature amount of the person registered in advance and the matching feature amount extracted by the extraction means match .

The present invention allows for proper matching of people even when part of a person is occluded by another object.

FIG. 1 is a block diagram showing an example of a functional configuration of an image display device according to an embodiment of the present invention; A block diagram showing an example of a functional configuration of an image feature determination unit. Block diagram showing an example of hardware configuration 1 is a flowchart showing a flow of processing executed by an image processing apparatus according to an embodiment of the present invention. 1 is a flowchart showing a flow of processing executed by an image processing apparatus; 1 is a flowchart showing a flow of processing executed by an image processing apparatus; FIG. 1 is a diagram illustrating an example of correction of waist feature points. FIG. 1 is a diagram for explaining an example of correction of foot feature points; A diagram explaining the process of determining the area of an object. 1 is a flowchart showing a flow of processing executed by an image processing apparatus; FIG. 13 is a diagram for explaining a process for correcting feature points outside a partial image area. A diagram explaining an example of a neural network configuration. Flowchart showing the process of training a neural network Diagram explaining screen display examples A diagram explaining examples of parts of the face A diagram explaining an example of a subnetwork configuration A diagram illustrating an example of the configuration of an image integration sub-network. FIG. 1 is a diagram illustrating an example of an occluded portion of a person.

The following describes an embodiment of the present invention.

<Embodiment 1>
FIG. 3 shows an example of the hardware configuration of this embodiment. In FIG. 3, 301 is an image sensor (imaging means) that is composed of a CCD, a CMOS, or the like, and converts a subject image from light into an electric signal. 302 is a signal processing circuit that processes a time series signal related to the subject image obtained from the image sensor 301 and converts it into a digital signal. 301 and 302 are connected to a bus as a camera. 303 is a CPU that controls the entire device by executing a control program stored in a ROM 304. 304 is a ROM that stores the control program executed by the CPU 303 and various parameter data. The control program is executed by the CPU 303 to make the device function as various means for executing each process shown in the flowchart described later. 305 is a RAM that stores images and various information. In addition, the RAM 305 functions as a work area for the CPU 303 and a temporary save area for data. 306 is a display. 307 is an input device such as a pointing device such as a mouse and a keyboard, which accepts input from a user. Reference numeral 308 denotes a communication device such as a network or a bus, which communicates data and control signals with other communication devices. In this embodiment, the processes corresponding to the steps of the flowchart described later are realized by software using the CPU 303, but part or all of the processes may be realized by hardware such as electronic circuits. The image display device of the present invention may be realized by using a general-purpose PC without the image sensor 301 and the signal processing circuit 302, or may be realized as a dedicated device. Software (programs) acquired via a network or various storage media may be executed by a processing device (CPU, processor) such as a personal computer.

Prior to the description of the embodiment, the terminology will be explained. A feature point is a point associated with a constituent unit of an object composed of multiple parts. In the following description, the feature point is specifically the position (two-dimensional coordinates) of a person's joints in an image. The reliability is calculated for each detected feature point, and is a real number from 0 to 1 indicating the likelihood that a part corresponding to the feature point exists in the image. For example, when detecting the position of a person's head as a feature point, if the head of a person is clearly captured in the image, the reliability is high. Conversely, if the head is blurred or is occluded by some other object, the reliability of the feature point corresponding to the head is low. In other words, it indicates the likelihood that the position indicated by the feature point is the part corresponding to the feature point. In this embodiment, a person is used as an example of a monitored object, but this is not limited to this, and other objects such as animals and cars may also be used. In other words, the present invention is applicable to any structure composed of multiple parts. In this embodiment, a person is identified using the features of the entire body of the person. On the other hand, a person may be identified using a face, in which case it is known under the names of "face recognition," "face matching," "face search," etc.

The configuration of this embodiment is shown in Figure 1. This embodiment is composed of an image acquisition unit 101, a first detection unit 102, a feature group unit 103, a second detection unit 104, a feature point storage unit 105, an area determination unit 106, an image extraction unit 107, an image feature extraction unit 108, a recognition unit 109, a display unit 110, a learning unit 111, and an object storage unit 112.

The image acquisition unit 101 acquires an image frame of an object having multiple parts from a camera. The first detection unit 102 detects the positions of the feature points of the object from the image frame and their reliability. The method of detecting the positions of the joints of a person in an image and their reliability will be described in detail later. The feature group determination unit 103 determines a feature group for detecting feature points whose reliability is smaller than a predetermined value based on the positions and reliability of the feature points detected by the first detection unit 102. This combination of feature points is prepared in advance, and is determined from among them according to the condition of the reliability of the feature points. The specific method will be described later. If the reliability of a specific feature point among the feature points detected by the first detection unit is smaller than a predetermined value, the second detection unit 104 detects the specific feature point from the image by a method different from the first detection means. The feature points are detected using the relative positional relationship between the feature points. The specific method will be described later. The feature point storage unit 105 stores the detected feature points. The area determination unit 106 determines the area where the object exists from the feature points. The area where the object to be subjected to image feature extraction exists is determined using a combination of specific feature points determined in advance among the detected feature points. The image extraction unit 107 cuts out the area determined by the area determination unit from the image frame. The image feature extraction unit 108 extracts image features for identifying a person from the cut-out partial image using a neural network or the like. The recognition unit 109 performs image recognition using the extracted image features. In this embodiment, image recognition involves identifying a person. Specifically, by comparing extracted image features with each other, it is determined whether or not these features belong to the same person. Details will be described later. The display unit 110 displays the results of the image recognition on a screen. The learning unit 111 learns the neural network and the like used by the image feature extraction unit 108 to extract image features. The object storage means 112 stores information about objects used by the recognition unit 109.

Figure 2 shows an example of the configuration of the image feature extraction unit 108 in Figure 1. The image feature extraction unit 108 is composed of an outside-area feature point correction unit 202, an object part extraction unit 203, an intermediate image feature extraction unit 204, a reliability conversion unit 205, a feature integration unit 206, and an image feature output unit 207.

The outside-area feature point correction unit 202 corrects feature points outside the partial image area, among the feature points extracted by the feature point extraction unit 102 in Figure 1. The object part extraction unit 203 extracts object parts (parts) from the image. The intermediate image feature extraction unit 204 extracts a first image feature (intermediate image feature) from the image and the object part. The reliability conversion unit 205 applies conversion processing to the reliability of the feature points extracted by the feature point extraction unit 102. The feature integration unit 206 integrates the output of the intermediate image feature extraction unit 204 and the output of the reliability conversion unit 205. The image feature output unit 207 generates image features from the output of the feature integration unit 206.

The operation of this image processing device will be described with reference to the flowchart in FIG. 4. The process shown in the flowchart in FIG. 4 is executed by the CPU 303 in FIG. 3, which is a computer, in accordance with a computer program stored in the ROM 304.

Step 401 acquires an image frame from the camera. This step corresponds to the operation of the image acquisition unit 101 in Figure 1.

Step 402 detects multiple feature points associated with the parts of an object from an image of the object having multiple parts obtained in step 401 (first detection method). This step corresponds to the operation of the first detection unit 102 in FIG. 1. In addition, in step 402, an image frame is input, and multiple feature points of a person present in the image and their reliability are extracted. For each detected feature point, a reliability indicating the likelihood that the feature point appears in the image is obtained. If the image processing target is a person, the joint positions of the human body can be used as feature points. The feature points detected in this step are five points: the top of the head, the neck, the waist, the right ankle, and the left ankle. Convolutional Pose Machines are used to detect the feature points. (Shih-En Wei et al., "Convolutional Pose Machines," IEEE, 2016.) In this method, a trained model (neural network) is used to calculate a reliability map that indicates where each joint position is located on the image. The reliability map is a two-dimensional map, and if the number of joint points is P, there are P+1 reliability maps (one map corresponds to the background). In the reliability map for a certain joint point, the position with the highest reliability is considered to be the position where that joint point exists. The reliability is a real number between 0 and 1 that indicates the likelihood that the feature point exists. The closer it is to 1, the higher the probability that the joint point exists. Joint points that are occluded by other objects are extracted from objects that are not people, so the likelihood that they are human joints decreases. Therefore, the reliability of the joint position is lower than that of joints that are not occluded by other objects. On the other hand, joints that are not hidden by other objects are well extracted from people, so the reliability of the joints is high.

The method of detecting feature points of an object and their reliability may be a method other than Convolutional Pose Machines. For example, a rule-based method may be used to identify each joint point of the human body using image features extracted from the joint points. Alternatively, image features of the head may be extracted from the image, and the position of the torso may be estimated from the position where the head is extracted. In addition, in this embodiment, the joint points of the human body are used as feature points, but if the image processing target is a face, facial feature points can be used. As facial feature points, the centers and end points of parts such as the eyes, eyebrows, nose, mouth, and ears, points on the contour line, points on the contour line of the entire face shape, etc. can be used.

Step 403 determines the feature point group to be used in the second detection method. Step 403 corresponds to the operation of the feature point determination unit 103 in FIG. 1. The feature point group determined in step 403 is used in the second detection method. A number of combination patterns of feature points are prepared, and one is selected and determined according to the condition of the reliability of the feature points. It is used in the second detection method in the subsequent step 404. The feature point group includes feature points (here, head, neck, or waist) used to determine the position after correction. In this embodiment, the feature points to be corrected as the specified feature points are the waist, right ankle, and left ankle. The correction of the right ankle and the left ankle is performed using the same procedure, so only the correction of the right ankle will be explained. Hereinafter, the ankle on one side to be processed will be simply referred to as "ankle".

The operation of step 403 is explained using the flowchart in Figure 5. As described later, six types of feature point groups, A1, A2, A3, B1, B2, and B3, are prepared in advance as candidates for feature point groups to be used for correction. One of the feature point groups A1, A2, and A3 related to correction of the waist and one of the feature point groups B1, B2, and B3 related to detection of the ankles in the second detection method are determined according to conditions.

Details will be described later, but feature point group A1 is an empty set, and the detection results of the first detection unit are used as is. Feature point group A2 is used to detect the position of the waist from the positions of the head and neck in the current frame. Feature point group A3 is used to detect the current position of the waist from the positions of the head and waist in previous frames. Feature point group B1 is an empty set, and the detection results of the first detection unit are used as is. Feature point group B2 is used to detect the position of the ankles from the positions of the neck and waist in the current frame. Feature point group B3 is used to detect the position of the ankles in the current frame from the positions of the neck and ankles in previous frames.

In step 501 of FIG. 5, it is evaluated whether the reliability of the waist in the current frame determined in step 402 is equal to or greater than a predefined threshold. If it is equal to or greater than the threshold, the process proceeds to step 503; if not, the process proceeds to step 502.

In step 502, it is evaluated whether the reliability of the waist in the past frame stored in the feature point storage unit 105 is equal to or greater than a threshold value. If it is equal to or greater than the threshold value, the process proceeds to step 505, and if it is not, the process proceeds to step 504. A past frame is the image frame acquired in step 401 of the previous repeat loop in the repeat loop of the flowchart in FIG. 4. However, if the feature points in the past frame are not stored in the feature point storage unit 105, that is, if step 403 in FIG. 4 is being executed for the first time, the process proceeds to step 504.

In step 503, feature point group A1 is determined as the feature point group to be used in the second detection method, and the process proceeds to step 506. When feature point group A1 is determined, it means that the waist feature points in the current frame are reliable, and there is no need to redetect the waist feature points in the subsequent processing.

In step 504, feature point group A2 is determined as the feature point group to be used in the second detection method, and the process proceeds to step 506. If feature point group A2 is determined, the waist joint points in both the current frame and the previous frame are unreliable, and the position of the waist in the current frame is detected in the subsequent process from the positions of the head and neck in the current frame.

In step 505, feature point group A3 is selected as the feature point group to be used for correction, and the process proceeds to step 506. When feature point group A3 is selected, the waist feature points in the current frame are unreliable, but the waist feature points in the previous frame are reliable, and the current waist position is corrected in the subsequent processing from the head and waist positions in the previous frame.

Step 506 evaluates whether the confidence of the ankle in the current frame determined in step 402 is equal to or greater than a predefined threshold. If it is equal to or greater than the threshold, proceed to step 508; if not, proceed to step 507.

In step 507, it is evaluated whether the reliability of the ankle in the past frame stored in the feature point storage unit 105 is equal to or greater than a predetermined threshold value. If it is equal to or greater than the threshold value, the process proceeds to step 510, and if it is not, the process proceeds to step 509. However, if the feature point storage unit 105 has not stored any feature points in the past frame, i.e., if this is the first time step 403 in FIG. 4 is executed, the process proceeds to step 509.

In this embodiment, the threshold values used in S501, S502, S506, and S507 are different values, but they may be the same value.

In step 508, feature point group B1 is selected as the feature point group to be used for correction, and the process of the flowchart in FIG. 5 ends. When feature point group B1 is selected, the foot feature points in the current frame are reliable, and there is no need to detect the foot positions in subsequent processing.

In step 509, feature point group B2 is selected as the feature point group to be used for correction, and the processing of the flowchart in FIG. 5 is terminated. When feature point group B2 is selected, the foot positions are unreliable in both the current frame and previous frames, and the foot positions in the current frame are detected in subsequent processing from the foot and hip positions in the current frame.

In step 510, feature point group B3 is selected as the feature point group to be used for correction, and the processing of the flowchart in FIG. 5 ends. When feature point group B3 is selected, the foot feature points in the current frame are unreliable, but the foot feature points in the previous frame are reliable, and the position of the current frame is detected from the positions of the neck and feet in the previous frame in the subsequent processing.

Although the above explanation of steps 506, 507, 508, 509, and 510 has been directed to only one ankle (right ankle), the feature point group to be used in the second detection method is determined for the other ankle (left ankle) in the same manner. Note that, in order to detect the position of the ankle, it is preferable to estimate the position of the ankle from a feature point as close as possible to the position of the ankle. Therefore, when the position of the waist can be used (high reliability of the waist position), the position of the ankle is detected using the position of the waist. When the position of the waist is unknown (low reliability of the waist position), the position of the neck, which is the second closest to the ankle after the waist, is used to detect the position of the ankle. The following processing order is based on the above intention, but the order may be changed. Also, the feature group may be determined so as to detect only the position of the ankle without detecting the position of the waist.

In step 404 in FIG. 4, the group of feature points determined in step 403 is used to detect a predetermined feature point by a second detection method. The processing in step 404 corresponds to the second detection unit 104 in FIG. 1. The operation of step 404 will be explained using the flowchart in FIG. 6. In the processing in FIG. 6, a predetermined feature point (ankle position) is detected based on the group of feature points A1, A2, A3, B1, B2, and B3 determined in the processing in the flowchart in FIG. 5.

As with step 403 in FIG. 4, the correction of the right and left ankles is performed using the same procedure, so only the detection of the right ankle will be described. Hereinafter, the ankle on one side being processed will simply be referred to as "ankle."

In step 601 in FIG. 6, it is determined which of the feature point groups A1, A2, and A3 relating to the waist has been selected. If feature point group A1 has been selected, the process proceeds to step 602; if feature point group A2 has been selected, the process proceeds to step 603; if feature point group A3 has been selected, the process proceeds to step 604. In steps 602, 603, and 604, the positions of the waist feature points are detected using the second detection method.

Step 602 does not detect the location of the waist feature points because the reliability of the waist feature points in previous processing is greater than a certain threshold and is therefore considered reliable.

In step 603, the position of the waist is detected from the positions of the head and neck detected in the current image frame. The process will be described with reference to FIG. 7. As shown in FIG. 7(a), the feature points of the top of the head 701, the neck 702, the waist 703, the right ankle 704, and the left ankle 705 are detected by step 402 in FIG. 4. First, as shown in FIG. 7(b), a straight line 706 connecting the head and neck is calculated. Also, the distance between the head and neck is calculated from the respective position coordinates. Here, it can be assumed that the ratio of the head-neck distance and the head-waist distance of the human body is approximately the same, although there are individual differences. For this reason, the position of the waist is detected so that it is on the straight line connecting the head and neck, and the ratio of the head-neck distance and the head-waist distance is a predetermined value. An example of the feature point 707 of the waist after correction is shown in FIG. 7(c). This predetermined ratio can be determined, for example, from the ratio of the parts of the average adult human body.

In step 604, the current waist position is detected from the head and waist positions in the previous frame. First, the distance between the head and waist is calculated from the feature points of the previous frame stored in the feature point storage unit 105. Next, in the current frame, a straight line connecting the head and neck is calculated as in FIG. 7(b). Here, it is assumed that the distance between the head and waist in the previous frame and the distance between the head and waist in the current frame are approximately the same. Then, the position of the waist in the current frame is detected so that the position of the waist is on the straight line connecting the head and neck, and the distance between the head and waist in the current frame is equal to the distance between the head and waist in the previous frame.

In step 605 of FIG. 6, it is determined which of the ankle-related feature point groups B1, B2, or B3 has been selected. If feature point group B1 has been selected, the process proceeds to step 606; if feature point group B2 has been selected, the process proceeds to step 607; if feature point group B3 has been selected, the process proceeds to step 608. In steps 607 and 608, the positions of the ankle feature points are detected. In step 606, the positions of the ankle feature points are not detected.

In step 607, the position of the ankle is detected from the position of the neck and waist in the current frame. The process will be described with reference to FIG. 8. As shown in FIG. 8(a), the feature points of the top of the head 801, neck 802, waist 803, right ankle 804, and left ankle 805 are detected by step 402 in FIG. 4. First, as shown in FIG. 8(b), a straight line 806 (body axis) connecting the neck and waist is calculated. Also, the distance between the neck and waist is calculated from the respective position coordinates. Here, it can be assumed that the ratio of the distance between the neck and waist and the distance between the neck and the right ankle of the human body is approximately the same, although there are individual differences. For this reason, the position of the ankle is detected so that it is on the straight line connecting the neck and waist, and the ratio of the distance between the neck and waist and the distance between the neck and ankle is a predetermined value. An example after the feature point of the ankle 807 is detected is shown in FIG. 8(c) on the left.

In step 604, the position of the ankle in the current frame is detected from the positions of the neck and ankle in the previous frame. First, the distance between the neck and waist is calculated from the feature points of the previous frame stored in the feature point storage unit 105. Next, in the current frame, a straight line (body axis) connecting the neck and waist is calculated as in FIG. 8(b). Here, it is assumed that the distance between the neck and ankle in the previous frame is approximately the same as the distance between the neck and ankle in the current frame. Then, the position of the ankle in the current frame is detected so that the position of the ankle is on the body axis and the distance between the neck and ankle in the current frame is equal to the distance between the neck and ankle in the previous frame.

In the above explanation of steps 605, 606, 607, and 608, only the right ankle was targeted, but the left ankle is also detected in the same way. This process makes it possible to detect a more accurate position of the ankle even if the ankle is not properly detected by the first detection unit due to occlusion or noise.

In step 405 in FIG. 4, the area in which the object exists is determined based on the detected feature points. This partial image area indicates the area in the captured image in which a person exists, and is used to specify the area in which a person image is extracted from the image frame in a later process. The operation of step 405 corresponds to the area determination unit 106 in FIG. 1. The processing of step 405 will be explained using FIG. 9(a). As shown in FIG. 9(a), the image frame 903 has feature points for the top of the head, the neck, the waist, the right ankle, and the left ankle. First, the midpoint of the right ankle and the left ankle is calculated. Then, a straight line 901 (body axis) connecting the head and the midpoint is calculated. In this embodiment, the partial image area is a rectangle, and the aspect ratio is determined in advance. A rectangle 902 is determined so that the vertical direction of the rectangle is parallel to the body axis, the central axis of the rectangle is equal to the body axis, the upper side of the rectangle is in contact with the head, and the lower side of the rectangle is in contact with the ankles. At this time, a margin may be provided between the upper side of the rectangle and the head, and between the lower side of the rectangle and the ankles. For example, a margin of a size obtained by multiplying the distance between the head and the ankle (height) by a certain coefficient may be provided. That is, the partial image area is determined based on the circumscribing rectangle of the feature points. In this embodiment, the aspect ratio of the rectangle is fixed to facilitate subsequent input to the neural network, but it may not be fixed depending on the configuration of subsequent processing. When using corrected joint positions, the area determined here may have a person's body parts obscured or may contain a lot of noise. For example, as shown in FIG. 18, even if the ankle part is hidden by an obstruction 1803, the area is determined as including the person's body parts. By determining the area in this way, a partial image area in which the arrangement of the parts of the human body within the rectangle is consistent can be determined. By making the arrangement of the parts consistent, there is an effect that the feature amount of each part that is more accurately reflected in the feature amount extraction process performed later can be extracted.

In step 406 in FIG. 4, the partial image area determined in step 405 is cut out from the image frame as a person image. If the rectangle of the partial image area determined in step 405 is tilted, the image is rotated so that the rectangle stands upright. An example of the cutout from FIG. 9(a) is shown in FIG. 9(b). The operation of step 406 corresponds to the image extraction unit 107 in FIG. 1.

In step 407, the corrected part in the current frame is stored. The operation of step 407 corresponds to the feature point storage unit 105 in FIG. 1.

Step 408 extracts features from the partial image region (person image). The operation of step 408 corresponds to the image feature extraction unit 108 in Figs. 1 and 2. The operation of step 408 will be explained using the flowchart in Fig. 10.

In step 1001 in FIG. 10, the outside-area feature point correction unit 202 corrects the reliability of the feature points outside the partial image area based on the coordinates of the partial image area and the feature points. Step 1001 corresponds to the outside-area feature point correction unit 202 in FIG. 2. When the aspect ratio of the rectangle of the partial image area is fixed, the feature points may not be included in the partial image area, such as when the arms and legs are spread apart. The human body parts outside the partial image area are outside the range of feature extraction, and there is a problem that the accuracy of feature extraction in this part decreases. For this reason, in order to reduce the influence in a later step, an adjustment is made to reduce the reliability of the feature points outside the partial area. For example, in FIG. 11, the right ankle 1104 is outside the range of the rectangle 1106, and the reliability of the feature point of this right ankle is reduced. In this embodiment, the value obtained by multiplying the original reliability by a predetermined real value smaller than 1 is set as the reliability after correction. In this way, by reducing the reliability of feature points outside the partial region, the problem of reduced accuracy of feature extraction due to the placement of human body parts outside the partial region and the problem of reduced accuracy of feature extraction due to occlusion can be addressed by using a common process described below.

In step 1002, features are extracted from the partial image region and the reliability of the feature points. A neural network can be used to extract the features. FIG. 12 shows an example of the configuration of a neural network. The neural network in FIG. 12 receives an image 1201 and feature point reliability 1206 as input, and outputs an image feature 1210. The neural network is composed of an image conversion sub-network 1202, a reliability conversion sub-network 1207, an integration sub-network 1208, and a feature output sub-network 1209. The image conversion sub-network 1202 corresponds to the intermediate image feature extraction unit 204 in FIG. 2. The reliability conversion sub-network 1207 corresponds to the reliability conversion unit 205 in FIG. 2. The integration sub-network 1208 corresponds to the feature integration unit 206 in FIG. 2. The feature output sub-network 1209 corresponds to the image feature output unit 207 in FIG. 2.

The input data, intermediate data, and output data handled by neural networks are treated as tensors. Tensors are data represented as multidimensional arrays, and their number of dimensions is called the rank. A tensor with rank 0 is called a scalar, a tensor with rank 1 is called a vector, and a tensor with rank 2 is called a matrix. For example, an image with one channel (such as a grayscale image) can be treated as a rank 2 tensor of size H x W, or a rank 3 tensor of size H x W x 1. Additionally, an image with RGB components can be treated as a rank 3 tensor of size H x W x 3.

The data obtained by cutting a tensor at a certain position in a certain dimension and the operations performed on it are called slices. For example, slicing a rank-3 tensor of size H x W x C at the cth position in the third dimension gives a rank-2 tensor of size H x W or a rank-3 tensor of size H x W x 1.

A layer that performs a convolution operation on a tensor is called a convolution layer (abbreviated as Conv). The filter coefficients used in the convolution operation are called "weights." As an example, a convolution layer generates an output tensor of HxWxD from an input tensor of HxWxC.

A layer that multiplies a vector by a weight matrix and adds a bias vector is called a fully connected layer (abbreviated as FC). As an example, a vector of length D is generated from a vector of length C by applying a fully connected layer.

The operation of dividing a tensor into intervals and taking the maximum value of the interval to reduce the size of the tensor is called max pooling. When the average value of the interval is taken instead of the maximum value, it is called average pooling. In this embodiment, max pooling is used, and the layer of the neural network that performs this is simply called the pooling layer (abbreviated as Pooling). In this embodiment, the pooling layer outputs a tensor whose first and second dimensions are half the size of the input. Specifically, an output tensor of H/2×W/2×C is generated from an input tensor of H×W×C.

In neural networks, a nonlinear function that is usually applied after the convolution layer is called an activation function. Activation functions include the normalized linear function (abbreviated as ReLU) and the sigmoid function. In particular, the sigmoid function has the property that the output value ranges from 0 to 1. In this embodiment, ReLU is used as the activation function unless otherwise specified.

In neural networks, the operation of arranging and connecting tensors in a certain dimensional direction is called "concatenation."

Let us explain Global average pooling. In a rank-3 tensor of size H x W x C, for every slice in the third dimension, we take the average of all elements contained in each slice. Then, by arranging these C average values, we generate a vector of length C. This operation is called Global average pooling.

In Figure 12, the size of the image 1201 that is the input to the neural network is width W1, height H1, and the number of channels is 3. In other words, the image can be considered as a tensor of H1 x W1 x 3.

The image transformation subnetwork 1202 transforms the image 1201 into a feature map. The image transformation subnetwork 1202 is further composed of a preprocessing subnetwork 1203, a part estimation subnetwork 1204, and an image synthesis subnetwork 1205.

The image conversion sub-network 1202 extracts features for identifying an object for each part corresponding to the detected feature points. Specifically, as in the paper by L. Zhao et al., it includes a module that estimates parts and extracts the features of the parts. The image conversion sub-network 1202 corresponds to the object part extraction unit 203 in Figure 2. (L. Zhao et al. "Deeply-Learned Part-Aligned Representations for Person Re-Identification," IEEE, 2017.) In this embodiment, the object part extraction unit 203 is operated within the neural network that performs feature extraction, but the object part extraction unit 203 may be operated outside this neural network and information regarding the position and size of the parts may be provided from outside. In addition, the object part extraction unit 203 and the first detection unit 102 in FIG. 1 may share the same purpose, and information derived from the output of the first detection unit 102 may be used as the output of the object part extraction unit 203, or vice versa. The feature amounts for each part extracted here are integrated as an overall feature amount in a later process. At that time, weighting is performed so that the feature amount for each part is reflected in the overall feature amount according to the reliability of each feature point. In other words, the feature amount extracted from a part corresponding to a feature point with low reliability is suppressed from contributing to the final recognition result. This is because a feature point with low reliability may indicate that the object is occluded or there is a lot of noise, and the feature amount extracted from that part does not necessarily indicate the feature of that part of the object. By performing such processing, a feature amount that better reflects the features of the object can be generated, and the effect of improving the recognition accuracy of the object can be expected.

The image conversion sub-network 1202 can be configured with a sequence of one or more convolution layers (Conv) and max pooling layers (Pooling). In this embodiment, it is configured with a sequence of "Conv, Conv, Pooling, Conv, Pooling, Conv, Pooling, Conv". The outline of the configuration is shown in Figure 16 (a). As a result of applying the image conversion sub-network to an image, a tensor of H2 x W2 x C2 is obtained.

The part estimation sub-network 1204 takes the output of the image conversion sub-network 1202 as input, and outputs a tensor of H2 x W2 x P1, which is a feature map. Here, P1 is the number of parts to be estimated, and may be any number determined in advance. A slice of this tensor at position p in the third dimension (a tensor of size H2 x W2 x 1) is a mask image indicating the location of the p-th part. Each pixel takes a value between 0 and 1, and the closer to 1 the pixel is, the higher the probability that the part is present at that position. The part estimation sub-network 1204 is composed of one convolution layer and a sigmoid function. An outline of the configuration is shown in Figure 16 (b). The configuration of the part estimation network is not limited to this, and it may be composed of multiple convolution layers.

The image integration sub-network 1205 integrates the outputs of the image conversion sub-network 1202 and the part estimation sub-network 1204. The process flow is shown in FIG. 17. First, the slice 1702 (tensor with size H2×W2×1) at the position p in the third dimension of the output tensor 1701 of the part estimation sub-network is copied C2 times and concatenated in the third dimension direction to expand it to a tensor 1703 of size H2×W2×C2. Then, each element of this tensor is multiplied by each element of the output tensor 1704 of the image conversion sub-network 1202 to generate a new tensor 1705 (size H2×W2×C2). Then, global average pooling is applied to this tensor to generate a vector 1706 of length C2, and a fully connected layer is further applied to generate a vector 1707 of length C3. This process is applied to the channel p of all parts, and a vector 1708 is generated by concatenating each generated vector. That is, the length of vector 1708 generated by the image integration sub-network is (C3)P1. In this embodiment, the data to be integrated is a vector, but a vector is a type of tensor, and even if the data to be integrated is a second-order or higher tensor, it may be integrated by combining in the same manner.

The feature point reliability 1206 is a vector of length C4. In this embodiment, the number of feature points detected in step 402 of FIG. 4 is 5, so C4=5.

The confidence conversion sub-network 1207 converts the feature point confidence 1206 into a vector of length C5. The confidence conversion sub-network 1207 can be configured with zero or more fully connected layers. In this embodiment, it has one fully connected layer. An outline of the configuration is shown in Figure 16 (c).

The merging subnetwork 1208 merges the output vector of the image merging subnetwork 1205 and the output vector of the confidence conversion subnetwork 1207. The merging subnetwork 1208 outputs a vector of length C6. In this embodiment, these two vectors are concatenated. The outline of the configuration is shown in FIG. 16(d). Therefore, C6 = (C3)P1 + C5.

The feature output subnetwork 1209 receives the output vector of the integrating subnetwork 1208 as input, and outputs image features 1210, which are vectors of length C7. The feature output subnetwork 1209 can be composed of one or more fully connected layers. In this embodiment, it is composed of two fully connected layers. An outline of the configuration is shown in Figure 16(e). These image features are also called "matching features", "person features", "descriptors", and "embedding".

In step 409 of FIG. 4, the feature amount of the person image extracted in step 408 is compared with the feature amount stored in the person database. The person database is a storage means in which cut-out images and feature amounts of N people to be identified are registered in advance. An image of the person to be identified is photographed in advance, and the image is cut out and feature amounts are extracted and stored in the same manner as in steps 402 to 408. The person database corresponds to the object storage means 112 in FIG. 1. In step 409, the distance between the feature amount of the person in the person database and the feature amount of the person image extracted in step 408 is calculated. Then, the people in the person database are sorted in order of distance, and the person with the smallest distance is placed at the top of the order. Step 409 corresponds to the processing of the recognition unit 109 in FIG. 1. In this embodiment, Euclidean distance is used to compare the feature amounts. Other methods for comparing the feature amounts may be used, such as other distance indices such as L1 distance and cosine distance, and comparison may be made using machine learning such as metrics learning and neural networks.

Step 410 in FIG. 4 displays the person found in step 409 on the screen. Step 410 corresponds to the processing of the image display unit 110 in FIG. 1. An example of the display screen is shown in FIG. 14. The display screen 1401 is composed of a query 1402 and a gallery 1403. The query 1402 is an image of the person to be searched, and displays the person image extracted in step 406. The gallery 1403 is a list of search results, and displays the top five people in the images in the person database sorted in order of distance in step 409. In this case, the top five people may be displayed, or only people whose distance is less than a predetermined threshold may be displayed from among the five people. The image displayed in the gallery may be extracted in the same manner as in steps 401 to 407 in FIG. 4, or may be extracted in another manner. Markers indicating the positions of detected feature points may be superimposed on the images of the person in the query and gallery, as shown in FIG. 14.

Step 411 in FIG. 4 determines whether or not to end the processing of the flowchart. In this embodiment, if the number of times step 411 is executed reaches or exceeds a specified number, it is determined that the processing should end. If not, the process proceeds to step 401 and continues the processing of the flowchart.

<Neural network training>
A method of learning the neural network used in the image feature extraction unit 108 in Fig. 1 will be described with reference to the flowchart in Fig. 13. The process in the flowchart in Fig. 13 corresponds to the learning means 111 in Fig. 1.

The structure of the neural network is shown in Figure 12, as described above. The neural network takes an image 1201 and feature point reliability 1206 as input, and outputs image features 1210.

The neural network is trained using triple loss. (F. Shroff et al. "Face Net: A Unified Embedding for Face Recognition and Clustering," arXiv:1503.03832). Triplet loss uses triplets consisting of a sample called an anchor sample, a sample of the same person as the anchor called a positive sample, and a sample of a different person than the anchor called a negative sample. The network is updated by comparing the features obtained from the anchor sample, positive sample, and negative sample to calculate a loss function.

Step 1301 in FIG. 13 initializes the weights of the convolution layer and the fully connected layer that make up the network. In this embodiment, random numbers are used as the initial values of the weights.

In step 1302, training data is randomly acquired from the training data group. One training data is a triplet, including one anchor sample, one positive sample, and one negative sample. The anchor sample, the positive sample, and the negative sample each consist of an image and feature point reliability. The images and feature point reliability are generated using a procedure similar to that for inputting them to the neural network used in the flowchart of Figure 4.

Step 1303 updates the network with the training data. First, the current state of the network is applied to the anchor sample, positive sample, and negative sample to calculate the features for each. Losses are calculated for these three features using triplet loss. Then, the weights in the network are updated using the backpropagation method.

In step 1304, it is determined whether to end the learning. If step 1304 has been executed a specified number of times, it is determined that the learning has ended, and the series of processes in the flowchart of FIG. 13 is terminated. If it is determined that the learning has not ended, the process proceeds to step 1302.

According to this embodiment, the feature group determination unit 103 and the second detection unit 104 can detect unsatisfactory feature points from good feature points again. Therefore, even in a situation where part of the object is occluded by another object or is subject to disturbance, it is expected to have the effect of reducing errors in object region determination by the region determination unit 106.

In areas where an object is partially occluded by other objects or is subject to disturbance, it can be assumed that the reliability of the feature points acquired by the first detection unit 102 will be lower than normal and will be output. At this time, it is believed that the quality of the image features for image recognition extracted from these local areas will also decrease at the same time. Therefore, by using information on the reliability of the feature points as an index representing the reliability of a certain local area in the image feature extraction unit 108, it is expected that the effect of reducing the decrease in the quality of the image features can be expected. Therefore, it is expected that the effect of improving the accuracy of image recognition can be expected.

Step 1001 in Figure 10 reduces the reliability of feature points outside the partial image region. Body parts outside the partial image region are outside the range of feature extraction, and there is a problem that the accuracy of feature extraction in this part decreases. Therefore, in order to reduce the impact in a later step, the reliability of feature points outside the partial region is reduced, which is expected to have the effect of reducing the deterioration of the quality of image features.

In steps 403 and 404, feature points from past frames as well as the current frame are used to select the group of feature points to be used for correction and to correct the feature points. By using feature points from past frames, it is expected that the accuracy of correction of feature points can be improved even when the reliability of the feature points in the current frame is low.

In step 403, feature points are selected in a predetermined order. By preferentially selecting feature points that are expected to have better accuracy in correcting the feature point positions in step 404, it is expected that the feature point positions can be corrected more accurately.

In step 404, the feature points are corrected in a specified order. Here, the feature points are corrected in the order of the waist, then the legs. This is because a person's body parts are connected in the order of neck, waist, legs. First, the position of the waist is corrected, and then the legs can be corrected using this more accurate position of the waist. In this way, by comparing feature points in a specified order, it is expected that the feature point positions can be corrected more accurately.

In step 404, the positions of the feature points are corrected based on the relative positional relationship between the feature points. In this embodiment, the feature points are corrected based on the ratio of the distances between the feature points and the straight line (body axis) obtained from the feature points. In this way, by using prior knowledge about the structure of the object, it is expected that the positions of the feature points can be corrected more accurately.

<Modification of the First Embodiment>
The feature points extracted in step 402 are not limited to the top of the head, neck, waist, right ankle, and left ankle, but may be other parts such as wrists, elbows, knees, etc. Also, they do not necessarily have to be on body parts, but may be other points determined based on the positional relationship of body parts, such as the midpoint between the right ankle and the left ankle, or the intersection of the line connecting the body axis with the left ankle and the right ankle.

In step 604, the position of the waist in the current frame is corrected based on the distance between the head and waist in the previous frame, but other methods may be used. The position of the waist in the current frame may be corrected based on the difference in the position coordinates of the head and waist in the previous frame. For example, the difference in the position coordinates of the head and waist in the previous frame may be set to X pixels and Y pixels larger than the x and y coordinates of the head. The position of the waist in the current frame may be corrected to be equal to the difference in the position coordinates of the head and waist in the previous frame. Also, the difference in the position coordinates of the neck and waist may be used instead of the difference in the position coordinates of the head and waist.

In step 607, the ratio of the distance between the neck and waist of the human body and the distance between the neck and the right ankle (or the left ankle) is used, but this is not limiting and other ratios between feature points may be used. As one example, the head may be used instead of the neck, such as the ratio of the distance between the head and waist and the distance between the head and the right ankle (or the left ankle). As another example, the ratio of the distance between the head and neck and the distance between the waist and the right ankle (or the left ankle) may be used. The same applies to step 608.

In step 607, the right ankle and the left ankle are corrected so that they are on the body axis. This is not limiting, and correction may be made by moving the right ankle (or the left ankle) in the direction of the body axis so that the ratio between the feature points becomes a predetermined value. The same is true for step 608.

In the region determination unit 106, the partial image region is rectangular, but other shapes are also acceptable. For example, it may be polygonal, or may be surrounded by curves. Instead of a figure, a mask image that distinguishes the object region from other regions may also be acceptable.

The structure of the neural network in the first embodiment is not limited to this. For example, another sub-network may be inserted between the sub-networks. The branching structure of the network may be different. The configuration of the sub-network may have different types and numbers of components such as convolution layers, pooling layers, and fully connected layers.

In the integration subnetwork 1208 of FIG. 12, two vectors are integrated by combining them, but other calculation methods may be used. For example, if the two vectors are the same size, multiplication or addition of the elements of the vectors may be used instead.

The reliability conversion unit 205 in FIG. 2 is implemented as a reliability conversion sub-network 1207 as in FIG. 12, but the reliability conversion unit 205 may be provided outside the neural network. For example, normalization processing, conversion processing, and other processing may be performed on the reliability of feature points outside the neural network, and the processing result may be one of the inputs to the neural network.

In step 403 and step Feature point correction in FIG. 4, the feature points to be used for correction are selected from the current frame and the previous frame, and the feature points are corrected. The feature points may be selected and the feature points may be corrected using not only the previous frame, but also the frame before that. Furthermore, three or more frames may be used in combination with the current frame.

Although the image feature extraction unit 108 is configured with a neural network, methods other than a neural network may be used. For example, HOG (Histogram of Oriented Gradients) features or LBP (Local Binary Pattern) features may be extracted and image features may be determined based on these. Alternatively, parts may be estimated from HOG features or LBP features.

In step 603 of Figure 6, straight line 706 in Figure 7 was calculated from the head and neck, but it is also possible to calculate the straight line from just the head or neck. For example, if it can be assumed that the axis of a person's body is parallel to the y-axis of the image frame, then it can be assumed that the straight line is parallel to the y-axis of the image frame, and the straight line can be calculated from a single point on either the neck or the head. Similarly, in step 405 of Figure 4, straight line 901 in Figure 9 was calculated from multiple points, but it is also possible to calculate it from a single point.

In S1001 of FIG. 10, the corrected reliability is determined by multiplying the original reliability by a predetermined real value smaller than 1, but other methods may be used. The method of updating the reliability is not limited to this, and the reliability may be set to 0, a predetermined real value may be subtracted from the reliability, or other methods may be used.

As described above, the process described in embodiment 1 allows appropriate matching of a person even in a situation where part of the person is occluded by another object.

<Embodiment 2>
In the first embodiment, the whole body of a person is the subject of image processing, but the face may be the subject of image processing instead. In the second embodiment, only the differences from the first embodiment will be described.

When the target is a face, facial feature points are detected in step 402 in FIG. 4. This is illustrated in FIG. 15. Here, the right eye 1501, left eye 1502, nose 1503, right edge of the mouth 1504, and left edge of the mouth 1505 are detected as feature points.

In the second embodiment, we consider a case where the feature points of the right eye are corrected from the nose and mouth in steps 403 and 404. The left eye is processed in the same way as the right eye.

The process of step 403 will now be described. First, the reliability of the feature points of the right eye is evaluated. If the reliability is equal to or greater than a threshold, feature point group C1 is selected. If the reliability is less than the threshold, and the reliability of the right eye in the previous frame was not equal to or greater than the threshold, feature point group C2 is selected, and if it is equal to or greater than the threshold, feature point group C3 is selected.

The processing of step 404 will be explained. If the feature point group used for correction is feature point group C1, the position of the right eye is not corrected. If it is feature point group C2, the position of the right eye in the current frame is corrected so that it is closer to the arrangement of the facial features of an average person, based on the positional relationship of the right edge of the nose and the right edge of the mouth, and the left edge of the mouth in the current frame. If it is feature point group C3, the position of the right eye in the current frame is corrected so that it is closer to the arrangement of the right eye, nose, right edge of the mouth, and left edge of the mouth in the previous frame.

The processing in the other steps is similar to that in embodiment 1 if the feature points extracted from the whole body are replaced with facial feature points.

In the second embodiment, the facial feature points are the right eye, the left eye, the nose, the right edge of the mouth, and the left edge of the mouth, but other parts such as the corners of the eyes, the inner corners of the eyes, the pupils, the right edge of the nose, the bottom edge of the nose, the eyebrows, and the facial contour may be used as feature points. The processing in steps 403 and 404 may be changed accordingly.

According to the second embodiment, it is expected that the performance of cutting out a face image from an image frame and of face recognition can be improved. For example, this is effective in cases where the face is partially covered by an accessory such as sunglasses or a mask, or where part of the face is temporarily hidden by a hand or the like.

The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. The computer (or CPU, MPU, etc.) of the system or device then reads and executes the program. The program may also be provided by recording it on a computer-readable recording medium.

Reference Signs List 101 Image acquisition unit 102 First detection unit 103 Feature group determination unit 104 Second detection unit 105 Feature point storage unit 106 Area determination unit 107 Image extraction unit 108 Image feature extraction unit 109 Recognition unit 110 Display unit 111 Learning unit 112 Object storage unit

Claims (9)

an acquisition means for acquiring, from an image of a person, a plurality of feature points corresponding to a plurality of body parts of the person and a reliability indicating a likelihood that each of the feature points corresponds to the body parts;
an extraction means for extracting a feature amount obtained by weighting and integrating the feature amounts for the plurality of body parts corresponding to the plurality of feature points in accordance with the reliability acquired by the acquisition means, as a comparison feature amount to be compared with a feature amount of a person registered in advance;
a recognition means for determining that the person in the image is the same person as the person registered in advance when the feature amount of the person registered in advance matches the feature amount for comparison extracted by the extraction means;
13. An image processing device comprising: The image processing device according to claim 1, characterized in that the extraction means extracts the features for matching using a neural network that receives as input a partial image including the person and the reliability and outputs the features for matching. The image processing device according to claim 1 or 2, characterized in that one or more of the plurality of feature points corresponds to the position of a joint of a person. 4. The image processing apparatus according to claim 1, wherein one or more of the plurality of feature points correspond to positions of facial features of a person. Further comprising a control means,
The recognition means compares the feature for matching extracted by the extraction means with feature amounts of the plurality of people registered in advance,
5. The image processing device according to claim 1, wherein the control means causes a display unit to display information representing a person among the plurality of persons, the person corresponding to a feature that has the shortest distance from the feature for matching. 6. The image processing device according to claim 5, wherein the control means identifies a predetermined number of people from among the plurality of people in order of shortest distance from the feature for matching, and causes information representing the predetermined number of people to be displayed on the display unit. A program for causing a computer to function as each of the means included in the image processing apparatus according to any one of claims 1 to 6 . an acquisition step of acquiring, from an image of a person, a plurality of feature points corresponding to a plurality of body parts of the person and a reliability indicating a likelihood that each of the feature points is the body part;
an extraction step of extracting a feature amount obtained by weighting and integrating the feature amounts for the plurality of body parts corresponding to the plurality of feature points in accordance with the reliability acquired in the acquisition step, as a matching feature amount to be compared with a feature amount of a person registered in advance;
a recognition step of determining that the person in the image is the same person as the person registered in advance when the feature amount of the person registered in advance matches the feature amount for comparison extracted in the extraction step;
13. An image processing method comprising: 9. The image processing method according to claim 8, wherein the extraction step extracts the feature for matching by a neural network that receives as input a partial image including the person and the reliability and outputs the feature for matching.

Images