JP2018088057A - Image recognition device and image recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten time needed for image recognition processing upon performing image recognition on time-series images through combining a cascade-type processing part and deep learning.SOLUTION: A cascade-type processing part 15 performs image recognition processing using deep learning at respective stages in a cascade structure. A stage number determination part 19 determines one or more stages which perform image recognition processing among respective stages in the cascade structure. The one or more stages determined by the stage number determination part 19 perform image recognition processing on time-series images (video for example).SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for performing image recognition using deep learning (deep learning).

  Applications of deep learning have been studied in various fields. For example, Non-Patent Document 1 discloses a technique for estimating the position of a human joint by analyzing a human image using deep learning and detecting the posture of the human from the position of the joint. The technique of Non-Patent Document 1 estimates a joint position by combining a cascade detector and deep learning. The reason why the detector has a cascade structure is to improve the estimation accuracy of the joint position. The first stage of the cascade detector roughly estimates the joint position, and the next stage estimates the joint position based on the joint position estimated by the previous stage. In this way, each stage of the cascade detector performs image recognition processing for estimating the joint position, and deep learning is used for this processing.

  In addition to the cascade detector, a cascade categorizer can be combined with deep learning. For example, when a specific person is detected, the cascade categorizer classifies the specific person and the other person. In this specification, these are collectively referred to as a cascade processing unit.

"DeepPose: Human Position Estimate via Deep Neural Networks", [online], [November 17, 2016 search], Internet <URL: http://www.cv-foundation.org/openaccess/content_cvpr_C_Purse_P_P_P___ .pdf>

  Since deep learning requires a large amount of processing, deep learning requires time. The cascade processing unit performs image recognition processing in a plurality of stages. For this reason, in the image recognition apparatus that performs the image recognition process by combining the cascade type processing unit and the deep learning, the time of the image recognition process becomes long. In the case of a time-series image (for example, a moving image), it takes a long time to perform image recognition processing. For example, when detecting the posture of a moving human, there is a possibility that a large shift occurs between the detected posture and the current posture.

  An object of the present invention is to provide an image recognition apparatus and an image recognition method capable of shortening the time required for image recognition processing when image recognition is performed on a time-series image by combining a cascade processing unit and deep learning. That is.

  An image recognition apparatus according to a first aspect of the present invention has a cascade structure, and in each stage of the cascade structure, a processing unit that performs image recognition processing using deep learning, and each stage of the cascade structure A determination unit that determines one or more stages for performing the image recognition process, and the one or more stages determined by the determination unit perform the image recognition process on a time-series image.

  The determination unit determines one or more stages for performing image recognition processing using deep learning. For this reason, not all stages of the cascade structure always perform image recognition processing using deep learning, but a smaller number of stages can perform image recognition processing using deep learning. Therefore, according to the image recognition apparatus according to the first aspect of the present invention, the time required for the image recognition process when the image recognition is performed on the time-series image by combining the cascade processing unit and the deep learning. Can be shortened.

  A time-series image is an image arranged in the order of captured time, for example, a moving image or an image captured at a certain time interval.

  The image recognition process is, for example, an image recognition process for estimating the position of an object or an image recognition process for detecting an object. The image recognition process for estimating the position of an object is, for example, an image recognition process for estimating the position of a human joint in order to detect a human posture. The image recognition process for detecting an object is, for example, an image recognition process for detecting a specific person from a large number of persons or an image for detecting a person from objects existing in front of a road for automatic driving. Recognition process.

  In the above configuration, the processing unit outputs a result of the image recognition processing performed on the time-series image in the last stage among the one or more stages, and the determination unit is based on the result. And determining the one or more stages.

  The last stage will be described by taking a three-stage cascade structure as an example. When the first stage, the second stage, and the third stage are selected, the last stage is the third stage. When only the third level is selected, the last level is the third level. When the first stage and the second stage are selected, the last stage is the second stage. When only the first stage is selected, the last stage is the first stage.

  Based on the result of the image recognition process, for example, when emphasizing real-time property, it is whether or not the processing unit can output the result in real time. When the processing unit cannot output the result in real time, the determination unit determines to reduce the number of stages used for the image recognition processing so that the processing unit can output the result in real time. When the processing unit can output the result in real time, the determination unit determines to increase the number of stages used for the image recognition processing in order to improve the accuracy of image recognition in addition to the real time property.

  In the above configuration, the processing unit outputs a result of the image recognition processing performed on the time-series image in the last stage among the one or more stages, and the image recognition apparatus is based on the result. The determination unit further determines a reliability of the result, and the determination unit determines the one or more stages based on the reliability.

  When there are a large number of stages for one or more stages determined by the determination unit, the reliability of the result of image recognition processing is improved. When the number of stages is small, the calculation speed of the result is improved, so that the processing unit can output the result in real time.

  Based on the reliability, for example, the determination unit determines to decrease the number of stages as the reliability increases, and determines to increase the number of stages as the reliability decreases. This makes it possible to balance the calculation speed of the result and the reliability of the result.

  The degree of reliability can be determined by a threshold value. When there is one threshold, the degree of reliability can be divided into a case where the reliability is high and a case where the reliability is not high. Increasing the number of thresholds can finely divide the degree of reliability. For example, when there are two thresholds, the degree of reliability can be divided into a case where the reliability is high, a case where the reliability is medium, and a case where the reliability is low.

  In the above configuration, the processing unit calculates an estimated position of the object by the image recognition processing, and the determination unit is currently performing the image recognition by the processing unit among a plurality of images constituting the time-series image. The processed image is defined as a first image, the image subjected to the image recognition processing by the processing unit prior to the first image is defined as a second image, and the object in the first image is defined as the second image. A distance between the estimated position and the estimated position of the object in the second image is calculated, and the reliability is determined according to the magnitude of the distance.

  Assume that the time when the first image is captured is close to the time when the second image is captured. If the accuracy of the estimated position of the object in the first image and the accuracy of the estimated position of the object in the second image are high, these estimated positions should be very close (or the same). Therefore, the determination unit determines that the reliability of the estimated position of the object in the first image is high, for example, if the distance between these estimated positions is smaller than a predetermined threshold value. On the other hand, the determination unit determines that the reliability of the estimated position of the object in the first image is not high if the distance is equal to or greater than a predetermined threshold value.

  In the above configuration, when the first stage is not the first stage among the one or more stages, the stage located immediately before is output based on the past result output by the stage located immediately before the first stage. An estimation unit for estimating a result to be performed, and the first stage performs the image recognition process using a result estimated by the estimation unit.

  In the cascade structure, each stage performs image recognition processing using the result output by the stage positioned immediately before. For example, the second stage performs image recognition processing using the result output from the first stage, and the third stage performs image recognition processing using the result output from the second stage. Among the one or more stages determined by the determination unit, when the first stage is not the first stage, there is no result output by the stage positioned immediately before. For example, assume that only the third stage is selected in a three-stage cascade structure. Since the third stage performs image recognition processing using the result output from the second stage, the result output from the second stage is required. However, since only the third stage is selected, the second stage does not output the result. Therefore, the estimation unit estimates the result output from the second stage based on the past result output from the second stage. For example, when the result is the estimated position of the object, the estimation unit estimates the result using extrapolation, object tracking, deep learning, or the like.

  An image recognition method according to a second aspect of the present invention is an image recognition method having a cascade structure, and using a processing unit that performs image recognition processing using deep learning at each stage of the cascade structure, Among the stages of the cascade structure, a first step for determining one or more stages for performing the image recognition processing, and the one or more stages determined in the first step are performed on a time-series image, And a second step of performing the image recognition process.

  The image recognition method according to the second aspect of the present invention defines the image recognition device according to the first aspect of the present invention from the viewpoint of the method, and is the same as the image recognition device according to the first aspect of the present invention. It has the following effects.

  According to the present invention, it is possible to shorten the time required for image recognition processing when image recognition is performed on a time-series image by combining a cascade processing unit and deep learning.

It is a block diagram which shows the structure of the image recognition apparatus which concerns on embodiment. It is a schematic diagram explaining all the stage selection modes. It is a schematic diagram explaining the last stage selection mode. It is the flowchart 1 explaining operation | movement of the image recognition apparatus which concerns on embodiment. It is the flowchart 2 explaining operation | movement of the image recognition apparatus which concerns on embodiment. It is a schematic diagram which shows an example of a kth frame. It is a schematic diagram which shows an example of input image Im-1. It is a schematic diagram which shows an example of the relationship between the estimated position P1-1 of a left shoulder joint, and input image Im-1. It is a schematic diagram which shows an example of input image Im-1 to which the rectangular area was set. It is a schematic diagram which shows an example of the cut-out image Im2-1 for 2 steps | paragraphs. It is a schematic diagram which shows an example of the relationship between the estimated position P2-1 of the left shoulder joint and the cut-out image Im2-1 for the second stage. It is a schematic diagram which shows an example of input image Im-1 to which the rectangular area was set. It is a schematic diagram which shows an example of the 3rd-stage cut-out image Im3-1. It is a schematic diagram which shows an example of the relationship between the estimated position P3-1 of the left shoulder joint and the third-stage cutout image Im3-1. It is a schematic diagram which shows an example of the k + 10th frame. It is a schematic diagram which shows an example of input image Im-2. It is a schematic diagram which shows the estimated positions P2-2 to P2-5 of the left shoulder joint calculated by the second-stage processing unit using the second-stage cut-out images Im2-2 to Im2-5. It is a schematic diagram which shows an example of input image Im-2 to which the rectangular area was set. It is a schematic diagram which shows an example of the 3rd step | paragraph cut-out image Im3-2. It is a schematic diagram which shows an example of the relationship between the estimated position P3-2 of a left shoulder joint, and the 3rd step | paragraph cut-out image Im3-2. It is a schematic diagram explaining 2 step | paragraph selection mode. It is a flowchart explaining operation | movement in 2 step | paragraph selection mode. It is a schematic diagram which shows an example of input image Im-2 to which the rectangular area was set. It is a schematic diagram which shows an example of the cut-out image Im2-6 for the second stage. It is a schematic diagram which shows an example of the relationship between the estimated position P2-7 of the left shoulder joint and the second-stage cutout image Im2-6. It is explanatory drawing explaining the 1st mode performed in the 2nd modification. It is explanatory drawing explaining the 2nd mode performed in the 2nd modification. It is explanatory drawing explaining the 3rd mode performed in the 2nd modification.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the structure which attached | subjected the same code | symbol shows that it is the same structure, The description is abbreviate | omitted about the content which has already demonstrated the structure. In the present specification, when referring generically, it is indicated by a reference symbol without a suffix (for example, input image Im), and when referring to an individual configuration, it is indicated by a reference symbol with a suffix (for example, input image Im). -1).

  FIG. 1 is a block diagram illustrating a configuration of an image recognition apparatus 1 according to the embodiment. The image recognition device 1 includes a person detection unit 11, an input image generation unit 13, a cascade processing unit 15, a reliability determination unit 17, a stage number determination unit 19, and an estimation unit 12 as functional blocks. The image recognition apparatus 1 is realized by hardware (CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), and the like, and software.

  The person detection unit 11 performs image processing on the frame of the moving image captured by the imaging unit 3, and detects a person if a person is captured in the frame. A moving image is a specific example of a time-series image. A time-series image is an image arranged in the order of captured time. For example, images taken at regular time intervals are time-series images. The person detection unit 11 can detect a person using a known person detection technique such as person detection using a HOG (Histogram of Oriented Gradients) feature. The imaging unit 3 is, for example, a digital visible light camera or a digital infrared camera.

  The input image generation unit 13 generates an input image Im. The input image Im is a cutout image obtained by cutting out a rectangular area including a person from a frame. The input image generation unit 13 inputs the input image Im to the cascade processing unit 15.

  The cascade processing unit 15 has a cascade structure, and performs image recognition processing using deep learning at each stage of the cascade structure. The cascade processing unit 15 is a general term for a cascade detector and a cascade classifier. In the embodiment, a predetermined human joint (for example, a left shoulder joint, a left elbow joint, a left wrist joint, a left hip joint, a left knee joint, a left ankle joint, a right shoulder joint, a right elbow joint) using the cascade processing unit 15. , Right wrist joint, right hip joint, right knee joint, right ankle joint).

  The calculation of the estimated joint position using deep learning will be briefly described. Since it is the same in any of the predetermined joints, the calculation of the estimated position of the left shoulder joint will be described as an example. The image recognition apparatus 1 includes a large number of images of a person whose left shoulder joint is photographed (including an image of a left shoulder joint covered with clothes and an image of a left shoulder joint not covered with clothes) and a left shoulder joint. Learning to recognize the left shoulder joint using a number of images that have not been copied. Based on this learning, the cascade processing unit 15 estimates the position of the left shoulder joint of the person captured in the frame of the moving image. In deep learning, a multilayer neural network (DNN) is used. As a multi-layer neural network, for example, there is CNN (Convolutional Neural Network).

  The cascade processing unit 15 includes a first stage processing unit 151, a second stage processing unit 153, and a third stage processing unit 155. In the embodiment, the number of stages will be described as an example, but the number of stages may be plural.

  The first-stage processing unit 151 performs an image recognition process using deep learning, and roughly calculates an estimated joint position. The second-stage processing unit 153 performs image recognition processing using deep learning based on the result output by the first-stage processing unit 151 (here, the estimated joint position calculated by the first-stage processing unit 151). Then, the estimated position of the joint is calculated. Thereby, the estimated position of the joint has higher accuracy than the estimated position of the joint calculated by the first stage processing unit 151. The third-stage processing unit 155 performs image recognition processing using deep learning based on the result output by the second-stage processing unit 153 (here, the estimated joint position calculated by the second-stage processing unit 153). Then, the estimated position of the joint is calculated. Thereby, the estimated position of the joint has higher accuracy than the estimated position of the joint calculated by the second stage processing unit 153. The calculation of the estimated joint position using the cascade processing unit 15 described above is disclosed in Non-Patent Document 1, for example.

  The image recognition apparatus 1 uses the estimated position of the predetermined joint calculated by the third-stage processing unit 155 as the current estimated position of the predetermined joint, and determines the posture of the person based on these. For this determination, a known algorithm for determining the posture of the person using the joint position can be used, and therefore description of determining the posture of the person is omitted. The image recognition device 1 causes the display unit 5 to display the determination result of the posture of the person.

  The reliability determination unit 17 determines the reliability of the estimated joint position. Since it is the same in any of the predetermined joints, the reliability of the estimated position of the left shoulder joint will be described as an example. The kth frame is an example of a first image, and the (k-1) th frame is an example of a second image. If the accuracy of the estimated position of the left shoulder joint in the k-th frame (current estimated position of the left shoulder joint) and the accuracy of the estimated position of the left shoulder joint in the k-1 frame (the past estimated position of the left shoulder joint) are high These estimated positions should be very close (or the same). Accordingly, the reliability determination unit 17 determines that the reliability of the estimated position of the left shoulder joint in the k-th frame is high if the distance between these estimated positions is smaller than a predetermined threshold value. On the other hand, the reliability determination unit 17 determines that the reliability of the estimated position of the left shoulder joint in the k-th frame is not high if the distance is equal to or greater than a predetermined threshold value.

  If the reliability is not high, the cascade processing unit 15 needs to calculate the estimated position of the left shoulder joint using the first-stage processing unit 151 to the third-stage processing unit 155. Therefore, when the reliability is not high, the stage number determination unit 19 instructs the cascade processing unit 15 to execute the all-stage selection mode. The all-stage selection mode is a mode in which the joint position is calculated using all of the first-stage processing unit 151, the second-stage processing unit 153, and the third-stage processing unit 155. FIG. 2 is a schematic diagram for explaining the all-stage selection mode. The first-stage processing unit 151, the second-stage processing unit 153, and the third-stage processing unit 155 calculate joint estimation positions for the same input image Im.

  On the other hand, if the reliability is high, the cascade processing unit 15 can calculate the estimated position of the left shoulder joint with high accuracy even if the first-stage processing unit 151 and the second-stage processing unit 153 are omitted. it can. Therefore, the stage number determination unit 19 instructs the cascade processing unit 15 to execute the final stage selection mode when the reliability is high. The final stage selection mode is a mode in which the first stage processing unit 151 and the second stage processing unit 153 are omitted, and the joint position is calculated using the third stage processing unit 155. FIG. 3 is a schematic diagram for explaining the final stage selection mode. Since the first stage processing unit 151 and the second stage processing unit 153 are omitted in the final stage selection mode, the input image Im is input only to the third stage processing unit 155.

  As described above, the stage number determination unit 19 selects the entire stage selection mode shown in FIG. 2 or the final stage selection mode shown in FIG. That is, the stage number determination unit 19 determines one or more stages for performing the image recognition process for calculating the estimated position of the joint among the stages of the cascade processing unit 15.

  Note that the reliability is not limited to the above distance. According to deep learning, it is known that a probability that an estimated position calculated by deep learning is an actual position can be calculated. Using this, the reliability determination unit 17 calculates the reliability. For example, as described above, the first-stage processing unit 151, the second-stage processing unit 153, and the third-stage processing unit 155 each calculate the estimated position of the left shoulder joint in the k-th frame using deep learning. . At this time, the first-stage processing unit 151 calculates, using deep learning, the probability that the estimated position of the left shoulder joint calculated by the first-stage processing unit 151 is the actual position of the left shoulder joint for the k-th frame ( For example, the probability is 70%). Similarly, the second-stage processing unit 153 calculates, using deep learning, the probability that the estimated position of the left shoulder joint calculated by the second-stage processing unit 153 is the actual position of the left shoulder joint for the kth frame ( For example, the probability is 80%). The third-stage processing unit 155 calculates the probability that the estimated position of the left shoulder joint calculated by the third-stage processing unit 155 is the actual position of the left shoulder joint for the k-th frame using deep learning (for example, the probability 90%). The reliability determination unit 17 multiplies those probabilities to calculate the reliability of the estimated position of the left shoulder joint (reliability = 70% × 80% × 90%).

  An operation of the image recognition apparatus 1 according to the embodiment will be described. 4A and 4B are flowcharts for explaining this. In the embodiment, since the algorithm for calculating the estimated position of the joint is the same for any of the predetermined joints described above, the calculation of the estimated position of the left shoulder joint will be described as an example.

  With reference to FIG. 1 and FIG. 4A, the reliability determination unit 17 has not calculated the reliability at the time when the operation of the image recognition apparatus 1 starts. Therefore, the stage number determination unit 19 selects the all stage selection mode shown in FIG.

  The person detection unit 11 performs image processing for person detection in real time on a frame of a moving image (an example of a time-series image) sent from the imaging unit 3 (step S1). When the person detection unit 11 cannot detect the frame in which the person is captured in step S1 (No in step S2), the person detection unit 11 repeats the process of step S1.

  For example, when there is no frame in which a person is captured from the first frame to the (k-1) th frame, the person detection unit 11 performs the process of step S1 and the step S2 in No from the first frame to the k-1th frame. Repeat the judgment. It is assumed that the person 21 is copied in the kth frame shown in FIG. FIG. 5 is a schematic diagram illustrating an example of the kth frame. In step S1, the person detection unit 11 detects the person 21 copied in the kth frame (Yes in step S2). Thereby, the image recognition apparatus 1 starts a process of calculating the estimated position of the left shoulder joint of the person 21.

  The input image generation unit 13 cuts out a rectangular region in which the person 21 is copied from the kth frame, and generates an input image Im-1 shown in FIG. 6 (step S3). FIG. 6 is a schematic diagram illustrating an example of the input image Im-1. A generic name of the input image is referred to as an input image Im.

  The input image generation unit 13 inputs the input image Im−1 to the first stage processing unit 151, the second stage processing unit 153, and the third stage processing unit 155.

  The first stage processing unit 151 performs image recognition processing using deep learning on the input image Im-1, and calculates an estimated position P1-1 of the left shoulder joint (step S4). FIG. 7 is a schematic diagram illustrating an example of the relationship between the estimated position P1-1 of the left shoulder joint and the input image Im-1. Since the first-stage processing unit 151 roughly calculates the estimated position P1-1 of the left shoulder joint, the deviation between the estimated position P1-1 of the left shoulder joint and the actual position P0 of the left shoulder joint is relatively large. The generic name of the estimated position of the left shoulder joint calculated by the first stage processing unit 151 is referred to as an estimated position P1 of the left shoulder joint.

  The cascade processing unit 15 stores information indicating the estimated position P1-1 of the left shoulder joint calculated by the first stage processing unit 151. The first stage processing unit 151 outputs the information (step S5).

  The second stage processing unit 153 sets the rectangular area to the input image Im-1. FIG. 8 is a schematic diagram showing an example of the input image Im-1 in which the rectangular area 23 is set. The rectangular area 23 includes an estimated position P1-1 of the left shoulder joint shown in FIG. The second-stage processing unit 153 cuts out a portion surrounded by the rectangular area 23 from the input image Im-1, and generates a cut-out image shown in FIG. 9 (step S6). This is referred to as a second-stage cutout image Im2-1. FIG. 9 is a schematic diagram illustrating an example of the two-stage cut-out image Im2-1. The area of the second-stage cutout image Im2-1 is smaller than the area of the input image Im-1. A part of the person 21 including the left shoulder is shown in the second-stage cutout image Im2-1. A generic name of the second-stage cutout image is referred to as a second-stage cutout image Im2.

  The second-stage processing unit 153 performs image recognition processing using deep learning on the second-stage cutout image Im2-1 to calculate the estimated position P2-1 of the left shoulder joint (step S7). FIG. 10 is a schematic diagram illustrating an example of the relationship between the estimated position P2-1 of the left shoulder joint and the second-stage cutout image Im2-1. Since the second stage processing unit 153 calculates the estimated position P2-1 of the left shoulder joint using the estimated position P1-1 of the left shoulder joint calculated by the first stage processing unit 151 (FIG. 7), the estimation of the left shoulder joint is performed. The position P2-1 is close to the actual position P0 of the left shoulder joint. A generic name of the estimated position of the left shoulder joint calculated by the second-stage processing unit 153 is referred to as an estimated position P2 of the left shoulder joint.

  The cascade processing unit 15 stores information indicating the estimated position P2-1 of the left shoulder joint calculated by the second stage processing unit 153. The second stage processing unit 153 outputs the information (step S8).

  The third stage processing unit 155 sets the rectangular area to the input image Im-1. FIG. 11 is a schematic diagram illustrating an example of the input image Im-1 in which the rectangular area 25 is set. The rectangular area 25 includes the estimated position P2-1 of the left shoulder joint shown in FIG. The area of the rectangular area 25 is smaller than the area of the rectangular area 23 (FIG. 8). The third-stage processing unit 155 cuts out a portion surrounded by the rectangular area 25 from the input image Im-1, and generates a cut-out image shown in FIG. 12 (step S9). This is referred to as a third-stage cutout image Im3-1. FIG. 12 is a schematic diagram illustrating an example of the third-stage cutout image Im3-1. The area of the third-stage cutout image Im3-1 is smaller than the area of the second-stage cutout image Im2-1. A part of the person 21 including the left shoulder is shown in the third-stage cutout image Im3-1. A generic name of the third-stage cutout image is referred to as a third-stage cutout image Im3.

  The third-stage processing unit 155 performs image recognition processing using deep learning on the third-stage cut-out image Im3-1, and calculates the estimated position P3-1 of the left shoulder joint (step S10). FIG. 13 is a schematic diagram illustrating an example of the relationship between the estimated position P3-1 of the left shoulder joint and the third-stage cutout image Im3-1. Since the third-stage processing unit 155 calculates the estimated position P3-1 of the left shoulder joint using the estimated position P2-1 of the left shoulder joint calculated by the second-stage processing unit 153 (FIG. 10), the left shoulder joint is estimated. The position P3-1 is closer to the actual position P0 of the left shoulder joint. In FIG. 13, the estimated position P3-1 of the left shoulder joint and the actual position P0 of the left shoulder joint are substantially the same. A generic name of the estimated position of the left shoulder joint calculated by the third stage processing unit 155 is referred to as an estimated position P3 of the left shoulder joint.

  The cascade processing unit 15 stores information indicating the estimated position P3-1 of the left shoulder joint calculated by the third-stage processing unit 155. The third stage processing unit 155 outputs the information (step S11). The estimated position P3-1 of the left shoulder joint is information indicating the estimated position of the left shoulder joint in the kth frame shown in FIG. The reliability determination unit 17 stores this information. At the present time, the estimated position of the left shoulder joint at the k-th frame is the current estimated position of the left shoulder joint.

  The reliability determination unit 17 determines whether or not the reliability can be calculated (step S12). For the calculation of the reliability, the current estimated position of the left shoulder joint and the past estimated position of the left shoulder joint are used. Here, the estimated position of the left shoulder joint in the kth frame is the current estimated position of the left shoulder joint, and the estimated position of the left shoulder joint in the k−1th frame is the past estimated position of the left shoulder joint. As described above, since the person 21 is not copied in the frame before the kth frame, the estimated position of the left shoulder joint in the (k−1) th frame is not calculated. Therefore, the reliability determination unit 17 determines that the reliability cannot be calculated (No in step S12). Thereby, the person detection part 11 processes step S1. Here, the process of step S1 is performed for the (k + 1) th frame.

  It is assumed that the person 21 is copied in the (k + 1) th frame. The person detection unit 11 detects the person 21 captured in the (k + 1) th frame (Yes in step S2). The input image generation unit 13 performs the process of step S3. The cascade processing unit 15 performs the processes of Step S3 to Step S11. The information output in step S11 is information indicating the estimated position of the left shoulder joint in the (k + 1) th frame. The reliability determination unit 17 stores this information. At the present time, the estimated position of the left shoulder joint in the (k + 1) th frame is the current estimated position of the left shoulder joint, and the estimated position of the left shoulder joint in the kth frame is the past estimated position of the left shoulder joint.

  The reliability determination unit 17 determines whether or not the reliability can be calculated (step S12). Since the reliability determination unit 17 stores information indicating the current estimated position of the left shoulder joint and information indicating the past estimated position of the left shoulder joint, it is determined that the reliability can be calculated (in step S12). Yes).

  The reliability determination unit 17 determines the current estimated position of the left shoulder joint (here, the estimated position of the left shoulder joint in the (k + 1) th frame) and the past estimated position of the left shoulder joint (here, the left shoulder joint in the kth frame). And the reliability of the current estimated position of the left shoulder joint is calculated. The reliability determination unit 17 determines whether or not the reliability is high (step S13).

  When the reliability of the current estimated position of the left shoulder joint is not high (No in step S13), the stage number determination unit 19 selects the all stage selection mode shown in FIG. Therefore, the image recognition apparatus 1 performs the processing from step S1 to step S13 for the next frame (here, k + 2nd frame).

  When the reliability of the current estimated position of the left shoulder joint is high (Yes in step S13), the stage number determination unit 19 selects the final stage selection mode shown in FIG. Accordingly, the cascade processing unit 15 executes the final stage selection mode for the next frame. For example, the reliability determination unit 17 determines that the reliability of the current estimated position of the left shoulder joint is not high for each of the k + 1th to k + 8th frames (No in step S13), and the cascade processing unit 15 After executing the all-stage selection mode for each of the frames, it is determined that the reliability of the current estimated position of the left shoulder joint is high for the k + 9th frame (Yes in step S13). The stage number determination unit 19 selects the last stage selection mode shown in FIG. 3 for the k + 10th frame.

  Referring to FIGS. 1 and 4B, person detection unit 11 performs image processing for person detection in real time on the k + 10th frame shown in FIG. 14 (step S14). FIG. 14 is a schematic diagram illustrating an example of the (k + 10) th frame. A person 21 is shown in this frame. In step S14, the person detection unit 11 detects the person 21 captured in the k + 10th frame (Yes in step S15). When the person detection unit 11 cannot detect the person 21 from the k + 10th frame in step S14 (No in step S15), the person 21 is outside the imaging range of the imaging unit 3, and the stage number determination unit 19 Switch from stage selection mode to full stage selection mode. As a result, the image recognition apparatus 1 performs the processing from step S1 to step S13 on the next frame (here, the k + 11th frame).

  When the person detection unit 11 detects the person 21 from the k + 10th frame (Yes in step S15), the input image generation unit 13 cuts out a rectangular area in which the person 21 is copied from the k + 10th frame, and performs the input shown in FIG. An image Im-2 is generated (step S16). FIG. 15 is a schematic diagram illustrating an example of the input image Im-2.

  The input image generation unit 13 inputs the input image Im-2 to the third stage processing unit 155. In order for the third-stage processing unit 155 to calculate the estimated position P3 of the left shoulder joint, it is necessary to create a third-stage cutout image Im3 as a premise. The third-stage cutout image Im3 is, for example, like the third-stage cutout image Im3-1 shown in FIGS. 12 and 13, in order for the third-stage processing unit 155 to calculate the estimated position P3 of the left shoulder joint. This is a cut-out image to be used.

  In order to create the third-stage cutout image Im3, the estimated position P2 of the left shoulder joint calculated by the second-stage processing unit 153 is required (FIG. 2). However, in the final stage selection mode shown in FIG. 3, since the second stage processing unit 153 is omitted, the estimated position P2 of the left shoulder joint at the k + 10th frame is not calculated.

  Therefore, for the calculation of the estimated position P2 of the left shoulder joint in the k + 10th frame, the estimated position P2 of the left shoulder joint calculated by the second-stage processing unit 153 using the second-stage cut-out image Im2 in the latest several frames is used. Use. Here, the k + 9th frame to the k + 6th frame will be described as examples of the most recent frames. FIG. 16 shows the estimated position P2- of the left shoulder joint calculated by the second-stage processing unit 153 using the second-stage cut-out images Im2-2 to Im2-5 cut out from the k + 9th frame to the k + 6th frame, respectively. It is a schematic diagram which shows 2-P2-5.

  The estimation unit 12 calculates the estimated position P2-6 of the left shoulder joint at the k + 10th frame using the estimated positions P2-2 to P2-5 of the left shoulder joint shown in FIG. 16 (step S17). This calculation method includes extrapolation, object tracking, deep learning, and the like, and any of these may be used.

  After step S17, the third-stage processing unit 155 sets the rectangular area to the input image Im-2. FIG. 17 is a schematic diagram illustrating an example of the input image Im-2 in which the rectangular area 27 is set. The rectangular area 27 includes the estimated position P2-6 of the left shoulder joint calculated in step S17. The third-stage processing unit 155 cuts out a portion surrounded by the rectangular area 27 from the input image Im-2 shown in FIG. 17, and generates a third-stage cut-out image Im3-2 shown in FIG. 18 (step S18). . FIG. 18 is a schematic diagram illustrating an example of the third-stage cutout image Im3-2. A part of the person 21 including the left shoulder is shown in the third-stage cutout image Im3-2.

  The third-stage processing unit 155 performs image recognition processing using deep learning on the third-stage cut-out image Im3-2, and calculates the estimated position P3-2 of the left shoulder joint (step S19). FIG. 19 is a schematic diagram illustrating an example of the relationship between the estimated position P3-2 of the left shoulder joint calculated in step S19 and the third-stage cutout image Im3-2.

  The cascade processing unit 15 stores information indicating the estimated position P3-2 of the left shoulder joint calculated by the third-stage processing unit 155. The third stage processing unit 155 outputs the information (step S20). The estimated position P3-2 of the left shoulder joint is information indicating the estimated position of the left shoulder joint in the k + 10th frame shown in FIG. The reliability determination unit 17 stores this information. At this time, the estimated position of the left shoulder joint in the (k + 10) th frame is the current estimated position of the left shoulder joint.

  The reliability determination unit 17 determines the current estimated position of the left shoulder joint (here, the estimated position of the left shoulder joint at the k + 10th frame) and the past estimated position of the left shoulder joint (here, the left shoulder joint at the k + 9th frame). And the reliability of the current estimated position of the left shoulder joint is calculated. The reliability determination unit 17 determines whether or not the reliability is high (step S21).

  When the reliability of the current estimated position of the left shoulder joint is high (Yes in step S21), the stage number determination unit 19 selects the final stage selection mode shown in FIG. As a result, the image recognition apparatus 1 performs the processing of step S14 to step S21 for the next frame (here, k + 11th frame).

  When the reliability of the current estimated position of the left shoulder joint is not high (No in step S21), the stage number determination unit 19 selects the all stage selection mode shown in FIG. As a result, the image recognition apparatus 1 performs the processing from step S1 to step S13 on the next frame (here, the k + 11th frame).

  The main effects of the embodiment will be described. The stage number determination unit 19 selects the entire stage selection mode shown in FIG. 2 or the final stage selection mode shown in FIG. When the final stage selection mode is selected, only the third stage processing unit 155 performs image recognition processing using deep learning. For this reason, according to the image recognition apparatus 1 according to the embodiment, not all stages of the cascade processing unit 15 always perform image recognition processing using deep learning, but a smaller number of stages, Image recognition processing using deep learning can be performed. Therefore, according to the image recognition apparatus 1 according to the embodiment, the image recognition process is required when performing image recognition on a moving image (an example of a time-series image) by combining the cascade processing unit 15 and deep learning. Time can be shortened.

  A modification of the image recognition device 1 according to the embodiment will be described. In the embodiment, one threshold value is used for the determination of the reliability, but the first modification example uses two threshold values for the determination of the reliability. The two threshold values are a first threshold value and a second threshold value, and the first threshold value is smaller than the second threshold value.

  In the first modification, the reliability determination unit 17 illustrated in FIG. 1 determines that the distance between the current estimated position of the left shoulder joint and the past estimated position of the left shoulder joint is smaller than the first threshold value. It is determined that the reliability of the current estimated position is high. The reliability determination unit 17 determines that the reliability of the current estimated position of the left shoulder joint is medium when the distance is greater than or equal to the first threshold and less than the second threshold. The reliability determination unit 17 determines that the reliability of the current estimated position of the left shoulder joint is low when the distance is greater than the second threshold value.

  When it is determined that the reliability is low, the stage number determination unit 19 instructs the cascade processing unit 15 to execute the all-stage selection mode (FIG. 2). The point that the all-stage selection mode of the first modification is different from the all-stage selection mode of the embodiment is that, when the reliability determination unit 17 determines that the reliability is high in step S13 shown in FIG. 4A, the reliability is high. When determining as medium, it may be determined that the reliability is low.

  When it is determined that the reliability is high, the stage number determination unit 19 instructs the cascade processing unit 15 to execute the final stage selection mode (FIG. 3). The last stage selection mode of the first modified example is different from the last stage selection mode of the embodiment in that when the reliability determination unit 17 determines that the reliability is high in step S21 shown in FIG. 4B, the reliability is high. When determining as medium, it may be determined that the reliability is low.

  When the reliability is determined to be medium, the stage number determination unit 19 instructs the cascade processing unit 15 to execute the two-stage selection mode. The second-stage selection mode is a mode in which the first-stage processing unit 151 is omitted and the joint position is calculated using the second-stage processing unit and the third-stage processing unit 155. FIG. 20 is a schematic diagram for explaining the two-stage selection mode. Since the first stage processing unit 151 is omitted in the second stage selection mode, the input image Im is input to the second stage processing unit 153 and the third stage processing unit 155.

  FIG. 21 is a flowchart for explaining the operation in the two-stage selection mode. Steps S14 to S16 are the same as steps S14 to S16 shown in FIG. 4B.

  The input image generation unit 13 inputs the input image Im-2 illustrated in FIG. 15 to the second-stage processing unit 153 and the third-stage processing unit 155. In order for the second stage processing unit 153 to calculate the estimated position P2 of the left shoulder joint, it is necessary to create the second stage cutout image Im2. The second-stage cutout image Im2 is for the second-stage processing unit 153 to calculate the estimated position P2 of the left shoulder joint, for example, as in the second-stage cutout image Im2-1 shown in FIGS. This is a cut-out image to be used. In order to create the second-stage cutout image Im2, the left shoulder joint estimated position P1 calculated by the first-stage processing unit 151 is required. However, since the first-stage processing unit 151 is omitted in the two-stage selection mode, the estimated position P1 of the left shoulder joint at the k + 10th frame is not calculated.

  Therefore, the estimation unit 12 uses the estimated position P1 of the left shoulder joint calculated by the first-stage processing unit 151 using the input image Im in the latest few frames (for example, k + 9th frame to k + 6th frame) An estimated position P1 of the left shoulder joint is calculated (step S22). This is calculated by the same method (extrapolation, object tracking, deep learning, etc.) as the calculation of the estimated position P2-6 of the left shoulder joint shown in FIG.

  After step S22, the second-stage processing unit 153 sets the rectangular area to the input image Im-2. FIG. 22 is a schematic diagram illustrating an example of the input image Im-2 in which the rectangular area 29 is set. The rectangular area 29 includes the estimated position P1 of the left shoulder joint calculated in step S22.

  The second-stage processing unit 153 cuts out a portion surrounded by the rectangular area 29 from the input image Im-2 shown in FIG. 22 and generates a second-stage cut-out image Im2-6 (step S23). FIG. 23 is a schematic diagram illustrating an example of the second-stage cutout image Im2-6. A part of the person 21 including the left shoulder is shown in the second-stage cutout image Im2-6.

  The second-stage processing unit 153 performs image recognition processing using deep learning on the second-stage cutout image Im2-6 illustrated in FIG. 23, and calculates an estimated position P2-7 of the left shoulder joint (step S24). ). FIG. 24 is a schematic diagram illustrating an example of a relationship between the estimated position P2-7 of the left shoulder joint calculated in step S24 and the second-stage cutout image Im2-6.

  The cascade processing unit 15 stores information indicating the estimated position P2-7 of the left shoulder joint calculated in step S24. The second stage processing unit 153 outputs the information (step S25).

  Steps S9 to S11 are the same as steps S9 to S11 shown in FIG. 4A. The reliability determination unit 17 illustrated in FIG. 1 determines the reliability using the current estimated position of the left shoulder joint (information output in step S11) and the past estimated position of the left shoulder joint (step S26). When the reliability determination unit 17 determines that the reliability is high, the stage number determination unit 19 selects the final stage selection mode for the next frame. When the reliability determination unit 17 determines that the reliability is medium, the stage number determination unit 19 selects the 2-stage selection mode for the next frame. When the reliability determination unit 17 determines that the reliability is low, the stage number determination unit 19 selects the all-stage selection mode for the next frame.

  A second modification will be described. As shown in FIGS. 2 and 3, the image recognition apparatus 1 according to the embodiment uses information indicating the estimated joint position calculated by the third-stage processing unit 155 as an output of the cascade processing unit 15, and uses this as an output of the joint. The current estimated position. In the second modification, any one of the first mode, the second mode, and the third mode can be selected. The first mode is a mode in which information indicating the estimated joint position calculated by the first-stage processing unit 151 is output from the cascade processing unit 15 and is used as the current estimated position of the joint. The second mode is a mode in which information indicating the estimated joint position calculated by the second-stage processing unit 153 is output from the cascade processing unit 15 and is used as the current estimated position of the joint. The third mode is a mode in which information indicating the estimated position of the joint calculated by the third-stage processing unit 155 is used as the output of the cascade processing unit 15 and this is the current estimated position of the joint.

  FIG. 25 is an explanatory diagram illustrating the first mode executed in the second modification. FIG. 26 is an explanatory diagram illustrating a second mode executed in the second modification. FIG. 27 is an explanatory diagram illustrating a third mode executed in the second modification. In the second modification, the cascade processing unit 15 illustrated in FIG. 1 includes a switch S1 and a switch S2. These switches are software switches. The switch S1 includes a connection between the output unit of the first stage processing unit 151 and the input unit of the second stage processing unit 153, and a connection between the output unit of the first stage processing unit 151 and the output unit of the cascade processing unit 15. Switch. The switch S2 includes a connection between the output unit of the second stage processing unit 153 and the input unit of the third stage processing unit 155, and a connection between the output unit of the second stage processing unit 153 and the output unit of the cascade processing unit 15. Switch.

  Referring to FIG. 25, in the first mode, stage number determination unit 19 controls switch S1 to connect the output unit of first stage processing unit 151 and the output unit of cascade type processing unit 15. The switch S2 connects the output unit of the second stage processing unit 153 and the input unit of the third stage processing unit 155, but the output unit of the second stage processing unit 153 and the output unit of the cascade processing unit 15 May be connected. The first mode can be said to be a mode in which information indicating the joint position calculated by the first stage processing unit 151 illustrated in FIG. 2 is output from the cascade processing unit 15.

  Referring to FIG. 26, in the second mode, stage number determination unit 19 controls switch S1 to connect the output unit of first stage processing unit 151 and the input unit of second stage processing unit 153, In addition, the switch S2 is controlled to connect the output unit of the second stage processing unit 153 and the output unit of the cascade processing unit 15. The second mode can be said to be a mode in which information indicating the joint position calculated by the second-stage processing unit 153 illustrated in FIG. 2 is output from the cascade processing unit 15.

  Referring to FIG. 27, in the case of the third mode, stage number determination unit 19 controls switch S1 to connect the output unit of first stage processing unit 151 and the input unit of second stage processing unit 153, In addition, the switch S2 is controlled to connect the output unit of the second stage processing unit 153 and the input unit of the third stage processing unit 155. The third mode can be said to be a mode in which information indicating the estimated joint position calculated by the third-stage processing unit 155 shown in FIG. 2 is output from the cascade-type processing unit 15 (that is, in the all-stage selection mode). is there).

  In the first mode shown in FIG. 25, the first-stage processing unit 151 calculates the estimated joint position, and the second-stage processing unit 153 and the third-stage processing unit 155 are omitted. For this reason, the reliability of the current estimated position of the joint is low, but the calculation speed of the current estimated position of the joint is fast.

  In the third mode shown in FIG. 27, the first-stage processing unit 151, the second-stage processing unit 153, and the third-stage processing unit 155 calculate joint estimated positions. For this reason, the reliability of the current estimated position of the joint is high, but the calculation speed of the current estimated position of the joint is slow.

  In the second mode shown in FIG. 26, the first stage processing unit 151 and the second stage processing unit 153 calculate the joint position, and the third stage processing unit 155 is omitted. For this reason, the reliability of the current estimated position of the joint is higher than that in the first mode, but is lower than that in the third mode. The calculation speed of the current estimated position of the joint is slower than the first mode, but faster than the third mode.

  When the resolution of the moving image (FIG. 1) input to the image recognition device 1 is high, it is difficult to calculate the current estimated position of the joint in real time in the third mode shown in FIG. Therefore, in order to be able to calculate the current estimated position of the joint in real time, the stage number determination unit 19 selects any one of the first mode to the third mode according to the resolution of the moving image. There are a first threshold value and a second threshold value as resolution threshold values. Assume that the first threshold value is smaller than the second threshold value.

  The stage number determination unit 19 selects the third mode shown in FIG. 27 when the resolution of the moving image input to the image recognition apparatus 1 is smaller than the first threshold value. The stage number determination unit 19 selects the second mode shown in FIG. 26 when the resolution of the moving image input to the image recognition apparatus 1 is equal to or higher than the first threshold value and smaller than the second threshold value. The stage number determination unit 19 selects the first mode shown in FIG. 25 when the resolution of the moving image input to the image recognition apparatus 1 is larger than the second threshold value.

  Note that the user may be able to select any one of the first mode, the second mode, and the third mode. When the user inputs an input for selecting the first mode to the image recognition apparatus 1, the stage number determination unit 19 selects the first mode. When the user inputs an input for selecting the second mode to the image recognition apparatus 1, the stage number determination unit 19 selects the second mode. When the user inputs an input for selecting the third mode to the image recognition apparatus 1, the stage number determination unit 19 selects the third mode.

DESCRIPTION OF SYMBOLS 1 Image recognition apparatus 12 Estimation part 15 Cascade type | mold processing part (an example of a processing part)
17 Reliability determination unit (an example of a determination unit)
19 stage number determination unit (an example of a determination unit)
21 Person 23, 25, 27 Rectangular area Im Input image Im2 Second stage cut-out image Im3 Third stage cut-out image P0 Left shoulder joint actual position P1 Left shoulder joint estimated position P2 calculated by the first stage processing unit Second stage Estimated position of the left shoulder joint calculated by the processing unit P3 Estimated position of the left shoulder joint calculated by the third stage processing unit

Claims (6)

A processing unit having a cascade structure, and performing image recognition processing using deep learning at each stage of the cascade structure;
A determination unit that determines one or more stages for performing the image recognition processing among the stages of the cascade structure;
The image recognition apparatus in which the one or more stages determined by the determination unit perform the image recognition processing on a time-series image. The processing unit outputs a result of the image recognition processing performed on the time-series image in the last stage among the one or more stages,
The image recognition device according to claim 1, wherein the determination unit determines the one or more stages based on the result. The processing unit outputs a result of the image recognition processing performed on the time-series image in the last stage among the one or more stages,
The image recognition device further includes a determination unit that determines the reliability of the result based on the result,
The image recognition apparatus according to claim 1, wherein the determination unit determines the one or more stages based on the reliability. The processing unit calculates an estimated position of the object by the image recognition processing,
The determination unit sets, as a first image, an image that has been subjected to the image recognition processing by the processing unit among the plurality of images that constitute the time-series image, and performs the processing before the first image. The image subjected to the image recognition processing in the unit is set as a second image, a distance between the estimated position of the object in the first image and the estimated position of the object in the second image is calculated, The image recognition apparatus according to claim 3, wherein the reliability is determined according to a distance. If the first stage is not the first stage among the one or more stages, the result output by the stage positioned immediately before is estimated based on the past results output by the stage positioned immediately before the first stage. And an estimator for
The image recognition apparatus according to claim 1, wherein the first stage performs the image recognition process using a result estimated by the estimation unit. An image recognition method using a processing unit that has a cascade structure and performs image recognition processing using deep learning at each stage of the cascade structure,
A first step of determining one or more stages for performing the image recognition processing among the stages of the cascade structure;
The image recognition method, wherein the one or more stages determined in the first step include a second step of performing the image recognition process on a time-series image.

Images