JP2019121045A - Posture estimation system, behavior estimation system, and posture estimation program - Google Patents

Abstract

To provide a posture estimation system capable of accurately estimating a posture of a person in a photographed image while suppressing learning data used for machine learning, regardless of the posture of the person.SOLUTION: A posture estimation system, which estimates the posture of an object person based on a photographed image, includes: a human region detection unit that detects a region including an image of the object person in the photographed image as a human region; a head region detection unit that detects a region including a head image of the object person in the photographed image as a head region; an image rotation unit that unifies a direction of the head region with respect to a predetermined point in the human region in the photographed image in which a plurality of postures of the object person are respectively photographed by rotating each of the photographed images in a predetermined direction; and a posture estimation unit that estimates a joint point of the an object person as a posture by machine learning based on the photographed image after rotation.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a posture estimation system, an action estimation system, and a posture estimation program.

  Japan's life expectancy has become remarkable due to the improvement of living standards, the improvement of sanitation environment, and the improvement of medical standards following the postwar high economic growth. For this reason, coupled with the decline in the birth rate, it has become an aging society with a high aging rate. In such an aging society, an increase in the number of carers requiring care is expected due to illness, injury, and aging.

  Those in need of care etc. are relatively likely to fall during a walk or fall from a bed and get injured in a facility such as a hospital or nursing home. Therefore, development of a system for detecting the condition of the care recipient has been advanced in order to allow the care worker or the like to arrive immediately when the care recipient becomes such a condition.

  The following technology is disclosed in Patent Document 1 below. That is, a photographed image including an image of a person in a standing posture taken with a fisheye lens camera is rotated so that the image of the person is on the head side and the leg side is on the bottom, and the direction of the person is unified. Do. Then, the presence of a person is detected by an object detection method of a neural network selected according to the position of the person in the image after rotation. Thus, the presence of a person in a standing position can be detected with high accuracy from the image of the fisheye lens camera whose distortion changes according to the position in the image.

International Publication No. 2013/001941

  However, the technique disclosed in Patent Document 1 can detect the presence of a person in a standing posture with high accuracy in a captured image, but can not detect the presence of a person in a posture other than a standing posture with high accuracy. is there. Further, even if it is possible to detect the presence of the person in the standing posture, there is a problem that it is not possible to detect the posture of the person with high accuracy regardless of the posture of the person in the photographed image.

  The present invention has been made to solve such problems. That is, it is possible to provide a posture estimation system, an action estimation system, and a posture estimation program capable of accurately estimating the posture of a person while suppressing learning data used for machine learning regardless of the person's posture in a photographed image. With the goal.

  The above problems of the present invention are solved by the following means.

  (1) A posture estimation system for estimating a posture of a subject based on a photographed image, the human area detection unit detecting a region including an image of the subject in the photographed image as a human area, and the photographed image A head area detection unit for detecting an area including an image of the head of the subject as the head area, and a position in the human area in the photographed image in which a plurality of postures of the subject are respectively captured The joint point of the target person is machine-learned based on an image rotating unit that unifies the direction of the head region with respect to a fixed point in the predetermined direction by rotating the captured image, and the captured image after rotation. A posture estimation unit that estimates a posture as the posture.

  (2) The human region is a region within the contour of the image of the subject person or a region within a rectangle circumscribed by the contour of the image of the subject person, and the head region is an image of the head of the subject person The posture estimation system according to (1) above, which is a region within the contour of or a region within a rectangle circumscribed by the contour of the image of the head.

  (3) The posture estimation system according to (1) or (2), wherein the predetermined direction is a vertically upward direction in the photographed image.

  (4) The posture estimation according to any one of (1) to (3), wherein the direction of the head region is a direction from the center of gravity of the human region which is the predetermined point toward the center of gravity of the head region. system.

  (5) The posture estimation system according to any one of (1) to (4), wherein the posture estimation unit estimates the posture of the subject by Deep Pose.

  (6) A distance calculation unit that calculates the distance from the center of the captured image to the human region, and the approximation of the human region in the captured image after rotation by the image rotation unit when the distance is less than a predetermined threshold An approximate ellipse calculation unit that calculates an ellipse, and an image re-rotation unit that rotates the captured image after rotation again such that the direction of the major axis of the approximate ellipse is in the horizontal direction in the captured image The posture estimation unit according to any one of (1) to (5), wherein the posture estimation unit estimates the posture of the subject based on the photographed image after rotation by the image re-rotation unit. system.

  (7) The posture estimation system according to (6), wherein the posture estimation unit selects a dictionary used in the machine learning based on the distance calculated by the distance calculation unit.

  (8) The image processing apparatus further includes a head size calculation unit that calculates the size of the head region in the captured image as a head size, and the posture estimation unit is configured to use a dictionary used for the machine learning. The posture estimation system according to (6) above, which is selected based on the size.

  (9) The human body / head size ratio calculating unit further calculates a ratio of the area of the human region to the area of the head region as a human body / head size ratio, and the posture estimation unit is the machine learning The posture estimation system according to any one of the above (1) to (6), which selects a dictionary to be used in accordance with the human body / head size ratio.

  (10) The human area is an area within a rectangle circumscribed by the contour of the image of the subject, and the aspect ratio is calculated to calculate the aspect ratio which is the ratio of the length of the vertical side of the rectangle to the length of the horizontal side The posture estimation system according to (2), further including a part, wherein the posture estimation unit selects a dictionary used in the machine learning based on the aspect ratio.

  (11) A distance calculation unit that calculates a distance from the center of the captured image to the human region, a head size calculation unit that calculates the size of the head region in the captured image as a head size, A human body / head size ratio calculating unit that calculates the ratio of the area of the head area to the area of a human area as a human body / head size ratio, and a rectangle in which the human area is circumscribed by the contour of the image of the subject The aspect ratio calculating unit further calculates an aspect ratio that is a ratio of the length of the vertical side of the human area to the length of the horizontal side of the human area, and the posture estimation unit further includes: Any one of the above (1) to (5), wherein a dictionary used for learning is selected based on at least one of the calculated distance, the head size, the human body / head size ratio, and the aspect ratio. Pose estimation system described in crab Temu.

  (12) The posture estimation system according to any one of the above (1) to (11), and the rotation before rotation by reversely rotating the posture estimated as the joint point corresponding to the photographed image after rotation. A pre-rotation posture calculation unit that calculates the posture corresponding to a photographed image; and an action estimation unit that estimates the action of the subject based on changes among the plurality of postures calculated by the pre-rotation posture calculation unit And an action estimation system having:

  (13) A procedure (a) of detecting a region including an image of a subject in a photographed image as a human region, and a procedure of detecting a region including an image of a head of the subject in the photographed image as a head region (B) In the photographed image in which a plurality of postures of the target person are imaged, the direction of the head area with respect to the predetermined point in the human area is rotated in the predetermined direction by rotating the photographed image A posture estimation program for causing a computer to execute a procedure (c) for unifying and a procedure (d) for estimating a joint position of the subject by machine learning based on the captured image after rotation.

  The direction of the head of a person in the photographed image is unified into a predetermined direction by rotating the photographed image, and the joint point of the person is estimated as a posture by machine learning based on the photographed image after rotation. This makes it possible to estimate the posture of the person with high accuracy with a small amount of learning data regardless of the posture of the person in the photographed image.

FIG. 1 is a diagram showing a schematic configuration of a posture estimation system. It is a block diagram which shows the structure of an attitude | position estimation apparatus. It is a block diagram which shows the function of the control part of an attitude | position estimation apparatus. It is an explanatory view for explaining rotation of a photography picture in order to unify head direction upward. It is explanatory drawing which shows the picked-up image before rotation, and the picked-up image after rotation. It is an explanatory view for explaining recalculation of a person rectangle in a photography picture after rotation. It is an explanatory view for explaining recalculation of a person rectangle in a photography picture after rotation using a person silhouette. It is explanatory drawing which shows the picked-up image before rotation again, and the picked-up image after rotation again. It is a figure which shows the list of the dictionary for every divided model used for machine learning. It is explanatory drawing which shows the picked-up image in the case where object size is small, and when head size is large and small. It is explanatory drawing which shows the picked-up image in the case where an object distance is large, and a posture of a subject is a standing position and a sitting position. It is an explanatory view showing a case where a subject distance is large, and a human rectangle aspect ratio of a subject of a standing posture is small and large. It is an explanatory view showing a photography picture which is an input picture of machine learning, and a presumed posture outputted. It is a flow chart which shows operation of an action presumption system. It is a flow chart which shows operation of an action presumption system.

  Hereinafter, with reference to the drawings, a posture estimation system, an action estimation system, and a posture estimation program according to an embodiment of the present invention will be described. In the drawings, the same elements will be denoted by the same reference symbols, without redundant description. Also, the dimensional proportions of the drawings are exaggerated for the convenience of the description, and may differ from the actual proportions.

  FIG. 1 is a diagram showing a schematic configuration of a posture estimation system.

  The posture estimation system 10 includes a posture estimation device 100, an imaging device 200, and a communication network 300. Posture estimation apparatus 100 is communicably connected to imaging apparatus 200 by communication network 300. Posture estimation system 10 may be configured only by posture estimation apparatus 100. Posture estimation system 10 constitutes an action estimation system.

  The posture estimation apparatus 100 estimates the posture of a person who is the target person 400 included in the captured image received from the imaging device 200 by calculating joint points of the target person 400 by machine learning. Posture estimation apparatus 100 may use, for example, deep learning as machine learning. Deep learning includes, for example, Deep Pose, CNN (Convolution Neural Network), and Res Net. Note that machine learning other than deep learning such as SVM (Support Vector Machine) and Random Forest may be used.

  The imaging device 200 is configured of, for example, a near-infrared camera, and installed at a predetermined position, thereby photographing an imaging region viewed from the predetermined position as a viewpoint. That is, the imaging device 200 irradiates near infrared rays toward the imaging area with an LED (Light Emitting Device), and the reflected light of the near infrared rays reflected by an object in the imaging area is received by a CMOS (Complementary Metal Oxide Semiconductor) sensor By doing this, the shooting area can be shot. The photographed image may be a monochrome image in which the reflectance of near-infrared light is each pixel. The predetermined position may be, for example, the ceiling of the room of the subject 400. The imaging area may be, for example, a three-dimensional area including the entire floor of the living room. The imaging device 200 may capture the imaging area as a moving image composed of a plurality of captured images at a frame rate of 15 fps to 30 fps, for example.

  The communication network 300 may use a network interface according to a wired communication standard such as Ethernet (registered trademark). The communication network 300 may use a network interface according to a wireless communication standard such as Bluetooth (registered trademark) or IEEE 802.11.

  FIG. 2 is a block diagram showing the configuration of the posture estimation apparatus. Posture estimation apparatus 100 includes control unit 110, storage unit 120, display unit 130, input unit 140, and communication unit 150. These components are connected to one another via a bus 160. Posture estimation apparatus 100 may be configured by a computer.

  The control unit 110 is configured by a CPU (Central Processing Unit), and performs control and arithmetic processing of each unit of the posture estimation device 100 according to a program. The operation of the control unit 110 will be described later.

  The storage unit 120 may be configured by a random access memory (RAM), a read only memory (ROM), and a flash memory. The RAM temporarily stores programs and data as a work area of the control unit 110. The ROM stores various programs and various data in advance. The flash memory stores various programs including the operation system and various data.

  The display unit 130 is, for example, a liquid crystal display, and displays various information.

  Input unit 140 is formed of, for example, a touch panel or various keys. The input unit 140 is used for various operations and inputs.

  The communication unit 150 is an interface for communicating with an external device. For communication, a network interface according to standards such as Ethernet (registered trademark), SATA, PCI Express, USB, and IEEE 1394 can be used. In addition, for communication, a wireless communication interface such as Bluetooth (registered trademark), IEEE 802.11, 4G, etc. may be used. The communication unit 150 receives a photographed image from the photographing device 200. Also, the communication unit 150 can transmit the estimation result of the posture and behavior of a person to a portable terminal (not shown) which is an external device.

  The operation of the control unit 110 will be described.

  FIG. 3 is a block diagram showing the function of the control unit of the posture estimation device. The control unit 110 includes a human area detection unit 111, a head area detection unit 112, an image rotation unit 113, a dictionary selection unit 114, a posture estimation unit 115, an image reverse rotation unit 116, and an action estimation unit 117. The image rotation unit 113 constitutes a distance calculation unit, an approximate ellipse calculation unit, and an image rerotation unit. The dictionary selection unit 114 configures a head size calculation unit, a human body / head size ratio calculation unit, and an aspect ratio calculation unit. The image reverse rotation unit 116 constitutes a pre-rotation posture calculation unit.

  The human area detection unit 111 detects, as a human area, an area including an image of the target person 400 in an image captured by the imaging device 200. The human region is a region within the contour of the image of the target person 400 in the captured image, or a region within a rectangle circumscribed by the contour of the image of the target person 400. A region within the outline of the image of the target person 400 in the captured image may be detected as a silhouette of the image of the target person 400 (hereinafter, referred to as a “person silhouette”). The area within the contour of the image of the target person 400 in the captured image may be detected as a collection of pixels that constitute the image of the target person 400 in the captured image. An area in a rectangle circumscribing the contour of the image of the subject 400 may be detected as a rectangle circumscribed by the contour of the image of the subject 400 (hereinafter, referred to as a “human rectangle”). Hereinafter, in order to simplify the description, the human region is described as being detected as a human silhouette or a human rectangle.

  The human silhouette can be detected, for example, by extracting a range of pixels having a relatively large difference based on a time difference that differentiates the images before and after the photographing time. The human silhouette may be detected by background subtraction that subtracts the background image from the captured image. A human rectangle can be detected, for example, by calculating a rectangle circumscribed by a human silhouette.

  The head area detection unit 112 detects an area including an image of the head of the subject 400 as a head area. The head area is an area within the outline of the image of the head of the subject 400 in the captured image, or an area within a rectangle circumscribed by the outline of the image of the head of the subject 400. A region within the contour of the image of the head of the target person 400 in the captured image may be detected as a silhouette of the image of the head of the target person 400 in the captured image (hereinafter referred to as “head silhouette”). The area within the rectangle circumscribed by the outline of the image of the head of the target person 400 may be detected as a collection of pixels constituting the image of the head of the target person 400 in the captured image. An area within a rectangle circumscribed by the contour of the image of the head of the subject 400 is detected as a rectangle circumscribing the contour of the image of the head of the subject 400 in the captured image (hereinafter referred to as “head rectangle”) obtain. Hereinafter, in order to simplify the description, the head region is described as being detected as a head silhouette or a head rectangle.

  The head silhouette can be detected, for example, by extracting a relatively circular area with a relatively high reflectance to near infrared light. The human rectangle can be detected, for example, by calculating a rectangle circumscribed by the head silhouette.

  The image rotation unit 113 calculates the head direction from the human rectangle or human silhouette detected by the human region detection unit 111 and the head rectangle or head silhouette detected by the head region detection unit 112. The head direction is the direction of the head region with respect to a predetermined point in the human region in the captured image. Hereinafter, in order to simplify the description, it is assumed that the head direction is calculated from the human rectangle and the head rectangle. The predetermined point may be, for example, the center of gravity of a human rectangle.

  The image rotation unit 113 unifies the calculated head direction into a predetermined direction by rotating the captured image. Here, regardless of the posture of the target person 400 photographed in the photographed image, the head direction is unified to a predetermined direction by rotating the photographed image. That is, even if the posture of the subject person 400 photographed in the photographed image is any of standing, middle and lower back, sitting, lying and other postures, the head direction is set to a predetermined direction by rotating the photographed image. Unify. The head direction may be, for example, the center of gravity of the head rectangle with respect to the center of gravity of the human rectangle. The predetermined direction may be a vertically upward direction (a direction parallel to the vertical side of the rectangular captured image and toward the upper side of the captured image) in the captured image.

  By rotating the photographed image and unifying the head direction into a predetermined direction, it is possible to reduce the variation of the posture of the object person 400 in the photographed image which is the target of posture estimation by machine learning. As a result, even if the posture of the target person 400 captured in the photographed image is different, it is possible to suppress a decrease in learning data used for machine learning, so that posture estimation accuracy by increasing variations of the posture of the target person 400 Can be prevented.

  FIG. 4 is an explanatory view for explaining rotation of a photographed image for unifying the head direction to the upper direction.

  In the example of FIG. 4, the captured image is rotated so that the head direction is the vertically upward direction in the captured image. The angular difference between the head direction and the vertically upward direction before rotation is +100 degrees (+ indicates a counterclockwise direction). Therefore, the head direction can be made vertically upward by rotating the captured image by -100 degrees as the required rotation angle.

  FIG. 5 is an explanatory view showing a photographed image before rotation and a photographed image after rotation. In addition, in FIG. 5, in order to show the attitude | position of the subject person 400 clearly, the picked-up image is represented by the illustration by a simple diagram (following, in FIG. 6, FIG. 8, FIG. 10-FIG. 13). The upper part of FIG. 5 is a photographed image before rotation, and the lower part is a photographed image after rotation.

  In the photographed image before rotation, postures including different postures of the target person 400 are respectively imaged, and the head directions are different. In the photographed image after rotation, the head direction of the target person 400 is unified in the vertically upward direction.

  FIG. 6 is an explanatory diagram for describing recalculation of a human rectangle in a captured image after rotation. The upper part of FIG. 6 is a photographed image before rotation, and the lower part is a photographed image after rotation. In each photographed image on the upper side, a human rectangle calculated before rotation is indicated by a gray line. In each lower-stage photographed image, a human rectangle calculated before rotation is indicated by a gray line, and a human rectangle recalculated after rotation is indicated by a black line.

  As shown in the human rectangle shown by the gray line in the upper part of FIG. 6 and the human rectangle shown by the black line in the lower part, the human rectangle has a vertical side parallel to the vertical upper direction in the photographed image and a horizontal side May be calculated to be parallel to the horizontal direction (the direction parallel to the horizontal side of the rectangular captured image) in the captured image. In the photographed image after rotation, since the head direction is the vertically upward direction in the photographed image, the size of the human rectangle is calculated by recalculating the human rectangle in the photographed image after rotation, and the image of the target person 400 in the photographed image Can be closer to the size of As a result, since the calculation accuracy of the human rectangle can be improved, it is possible to select a more appropriate dictionary in dictionary selection using the size of the human rectangle by the dictionary selection unit 114 described later. Here, the dictionary is data for setting a learned model of machine learning, and is data defining, for example, weighting in a neural network of deep learning.

  FIG. 7 is an explanatory diagram for describing recalculation of a human rectangle in a photographed image after rotation using a human silhouette. The upper part of FIG. 7 is a human silhouette in the photographed image before rotation, and the lower part is a human silhouette in the photographed image after rotation. In each upper person silhouette, human rectangles calculated before rotation are shown by gray lines. In each lower person silhouette, a human rectangle calculated before rotation is indicated by a gray line, and a human rectangle recalculated after rotation is indicated by a black line.

  As shown in FIG. 7, the human rectangle can be calculated as a rectangle circumscribing the human silhouette based on the human silhouette. Therefore, a human silhouette may be calculated before rotation, and the calculated human silhouette may be rotated by a necessary rotation angle, and a human rectangle may be recalculated for the human silhouette after rotation.

  The image rotation unit 113 calculates the distance from the center of the captured image to the human region (hereinafter, referred to as “target distance”). The target distance can be calculated as the distance between the coordinates of the center of gravity of the human rectangle or human silhouette that is a human region and the coordinates of the center of the captured image. When the target distance is equal to or less than a predetermined threshold, the image rotation unit 113 calculates an approximate ellipse of a human silhouette in the captured image after rotation. The predetermined threshold may be set experimentally from the viewpoint of posture estimation accuracy. The image rotation unit 113 rotates the captured image after rotation again such that the direction of the major axis of the calculated approximate ellipse is in the horizontal direction. That is, after rotating the captured image to unify the head direction vertically upward, the image rotation unit 113 rotates the captured image after rotation to the major axis of the approximate ellipse if the target distance is equal to or less than a predetermined threshold. Rotate again so that the direction is horizontal.

  FIG. 8 is an explanatory view showing a photographed image before the second rotation and a photographed image after the second rotation. The upper part of FIG. 8 is a photographed image before rotation again, and the lower part is a photographed image after rotation again.

  In the photographed image before the second rotation, the subjects 400 near the camera in the standing posture are respectively imaged, and the direction of the major axis of the approximate ellipse of the human silhouette is different. In the photographed image after the second rotation, the direction of the major axis of the approximate ellipse is unified in the horizontal direction.

  The dictionary selection unit 114 selects a dictionary to be used for machine learning for posture estimation of the target person 400 based on a captured image after rotation (after rotation again if rotated again, and so forth). Specifically, the dictionary selection unit 114 selects a dictionary corresponding to each model from the dictionaries stored in the storage unit 120 for each of the five divided models.

  FIG. 9 is a diagram showing a list of dictionaries for each partitioned model used for machine learning.

  The dictionary 1 is a dictionary corresponding to the direct / standing position model. The direct and standing position models are models that are classified when the target person 400 is in a standing position directly under the camera of the imaging device 200. The photographed image is divided into the directly below / standing position model when the target distance is small and the size of the head region (hereinafter referred to as “head size”) is large. This is the case where the target person 400 is directly below the camera and the head size is large, since the head of the target person 400 is at a short distance directly below the camera, the target person 400 stands directly below the camera. It is because the possibility of being in the position posture is relatively high. The dictionary 1 is selected when the photographed image is divided into the direct / standing position model.

  The dictionary 2 is a dictionary corresponding to the immediate down, sitting down position model. The direct-seat and sitting-down position models are models that are classified when the target person 400 is in a sitting or lying position directly under the camera of the imaging device 200. The photographed image is divided into the direct sitting and sitting position model when the target distance is small and the head size is small. This is a case where the target person 400 is directly below the camera and the head size is small, and the head of the target person 400 is far below the camera, so the target person 400 is seated directly below the camera. Or because it is relatively likely to be in a recumbent posture. The dictionary 2 is selected when the photographed image is divided into the direct sitting and sitting position model.

  FIG. 10 is an explanatory view showing photographed images in the case where the target distance is small and the head size is large and small. The left figure is a photographed image when the target distance is small and the head size is large. The right figure is a photographed image when the target distance is small and the head size is small. In each photographed image, a head rectangle is indicated by a solid line.

  As shown in the left diagram of FIG. 10, when the target distance is small and the head size is large, the dictionary 1 is selected. In the example of the figure, the subject person 400 in the standing posture is photographed directly below the camera. As shown in the left diagram, when the target distance is small and the head size is small, the dictionary 2 is selected. In the example of the figure, the target person 400 in the posture of lying down directly under the camera is photographed.

  The dictionary 3 is a dictionary corresponding to the distant / slant standing model. The distant / diagonal standing model is a model that is classified when the target person 400 is in a posture of standing upright with respect to the camera at a distance from the camera. The photographed image has a large object distance and a ratio of a human body size, which is the size of a human region, to a head size, a large human body / head size ratio, and a side (horizontal side) of a human rectangle. When the human rectangle aspect ratio, which is the ratio of the vertical sides (vertical sides), is large, the model is divided into the distant / diagonal standing model. This is because the target person 400 is in the standing posture since the human body / head size ratio is large, and the target person 400 is obliquely oriented because the rectangular aspect ratio is large. On the other hand, there is a relatively high possibility of being in an upright standing posture. The dictionary 3 is selected when the photographed image is divided into the distant and diagonal standing position model.

  The dictionary 4 is a dictionary corresponding to the distant / front standing model. The distant / front standing model is a model that is classified when the target person 400 is in a standing position facing the camera (or back, the same applies hereinafter) at a distance from the camera. The photographed image is classified into the distant / frontal standing model when the object distance is large, the human body / head size ratio is large, and the human rectangle aspect ratio is small. This is because the target person 400 is in the standing posture since the human body / head size ratio is large, and the target person 400 is facing front because the rectangular aspect ratio is small. On the other hand, it is relatively likely to be in a standing position facing the front. The dictionary 4 is selected when the photographed image is divided into the distant / front standing model.

  The dictionary 5 is a dictionary corresponding to the distant / seated sitting position model. The distant / seated recumbent position model is a model that is classified when the subject 400 is in a sitting or recumbent position away from the camera. The photographed image is classified into the distant / seated recumbent position model when the object distance is large and the human body / head size ratio is small. This is because the subject 400 is in the sitting or lying position because the human body / head size ratio is small, so the subject 400 is relatively likely to be in the sitting or lying position at a distance from the camera. It is. The dictionary 5 is selected when the photographed image is classified into the distant / seated recumbent position model.

  FIG. 11 is an explanatory view showing photographed images in the case where the posture of the subject is in the standing position and in the case of the sitting position in the case where the target distance is large. FIG. 12 is an explanatory view showing the case where the human rectangle aspect ratio of the target person in the standing posture is small and large in the case where the target distance is large.

  The upper part of FIG. 11 is a photographed image when the target distance is large and the human body / head size ratio is large. The following figure is a photographed image when the object distance is large and the human body / head size ratio is small. The relationship between a human rectangle and a head rectangle is indicated by a solid line on the right of each captured image.

  The upper part of FIG. 12 is a photographed image when the object distance is large, the human body / head size ratio is large, and the human rectangle aspect ratio is small. The figure below shows the photographed image when the object distance is large, the human body / head size ratio is large, and the human rectangle aspect ratio is large. A human rectangle is indicated by a solid line on the right side of each captured image.

  When the target distance is large and the human body / head size ratio is large as in the upper diagram of FIG. 11, the image corresponds to a photographed image obtained by photographing a target person 400 in a standing position from a distance. Also, in the figure, the rectangular aspect ratio is smaller. Therefore, the dictionary 4 is selected. As shown in the figure below, when the target distance is large and the human body / head size ratio is small, it corresponds to a photographed image obtained by photographing the target person 400 in a sitting posture from a distance. Therefore, the dictionary 5 is selected.

  As shown in the upper diagram of FIG. 12, when the target distance is large, the human body / head size ratio is large, and the human rectangle aspect ratio is small, the captured image is a captured image of the target person 400 in a standing position from a distance. Applicable Therefore, the dictionary 4 is selected. When the target distance is large, the human body / head size ratio is large, and the human rectangle aspect ratio is large as shown in the following figure, the image corresponds to a photographed image obtained by photographing the target person 400 in a standing posture from a distance from a distance. Therefore, dictionary 3 is selected.

  The dictionary selection unit 114 can calculate the head size as the number of pixels of the head silhouette.

  The dictionary selection unit 114 can calculate the human body / head size ratio by dividing the number of pixels in a human rectangle or a human silhouette by the number of pixels in a head rectangle or a head silhouette. Hereinafter, in order to simplify the description, the human body / head size ratio is described as being calculated by dividing the number of pixels in the human rectangle by the number of pixels in the head rectangle.

  The dictionary selection unit 114 can calculate the rectangular aspect ratio by dividing the number of pixels corresponding to the length of the vertical side of the human rectangle by the number of pixels corresponding to the length of the horizontal side.

  The posture estimation unit 115 estimates a posture as a joint point of the target person 400 by machine learning based on the captured image after rotation by the image rotation unit 113. As machine learning, Deep Pose is preferably used.

  FIG. 13 is an explanatory view showing a photographed image which is an input image of machine learning and an estimated posture to be output. The upper part of FIG. 13 is a captured image after rotation which is an input image, and the lower part is a posture estimated as a joint point. In addition, the joint point which is an estimated posture is superimposed and shown by the picked-up image of the upper stage.

  As shown in FIG. 13, in the photographed image input to the posture estimation unit 115, the head direction is unified in the vertical upward direction. For this reason, for example, the photographed image in which the posture of standing position, lying position and falling position is photographed is rotated by the image rotating unit 113, and the head direction is unified vertically upward, so the same model is obtained. There is a relatively high probability of being divided into (for example, a distant / frontal standing model). For this reason, since it is possible to estimate joint points using a common dictionary for a plurality of postures of the target person 400, it is possible to suppress the data amount of learning data used for pre-time machine learning.

  The image reverse rotation unit 116 reversely rotates the posture estimated as the joint point to calculate the posture corresponding to the photographed image before the rotation. That is, the image reverse rotation unit 116 reversely rotates the posture estimated as the joint point in the captured image after rotation by the same rotation amount as the rotation direction in which the captured image is rotated by the image rotation unit 113. The amount of rotation can be acquired from the image rotation unit 113. As described above, for example, the photographed image in which the posture of standing position, lying position, and falling position is photographed is rotated by the image rotating unit 113, so that the head direction is unified in the vertically upward direction. The head direction of the posture estimated as a joint point by the estimation unit 115 is also vertically upward. Therefore, the image reverse rotation unit 116 reversely rotates the posture estimated as the joint point to calculate the posture corresponding to the captured image before rotation, and estimates the action of the subject 400 by the action estimation unit 117 described later. The estimation accuracy of the action of the target person 400 is improved by using the

  A plurality of postures based on joint points of the subject 400 after reverse rotation by the image reverse rotation unit 116 are input to the action estimation unit 117. The action estimation unit 117 estimates the action of the target person 400 based on the change between the postures due to the plurality of joint points. The action estimation unit 117 may, for example, estimate the action of the target person 400 based on the difference in posture due to the joint point. The difference in posture due to the joint point may be a difference in posture due to the joint point estimated for each adjacent frame of the video composed of a plurality of photographed images. For example, the motion of falling can be estimated by the fact that, after the difference in posture due to the joint point exceeds a predetermined threshold, the difference in posture is almost eliminated.

  14 and 15 are flowcharts showing the operation of the behavior estimation system. The flowchart may be executed by the control unit 110 of the posture estimation device 100 according to a program.

  The control unit 110 detects a human area as a human rectangle or a human silhouette from the captured image received from the imaging device 200 (S101). When the human area is not detected (S102: NO), control unit 110 continues detection of the human area.

  When the control unit 110 detects a human region (S102: YES), the head region detection unit 112 detects a head region as a head rectangle or a head silhouette (S103).

  The control unit 110 rotates the photographed image such that the head direction is the vertically upward direction in the photographed image (S104).

  When the target distance is equal to or less than the predetermined threshold (S105: YES), the control unit 110 rotates the captured image again so that the major axis of the approximate ellipse of the human silhouette that is the human region is horizontal in the captured image (S105). S106). When the head size is equal to or larger than the predetermined threshold (S107: YES), the control unit 110 selects the dictionary 1 corresponding to the direct / standing position model (S108). The threshold may be set experimentally from the viewpoint of posture estimation accuracy. When the head size is not equal to or larger than the predetermined threshold (S107: NO), the control unit 110 selects the dictionary 2 corresponding to the direct / sitting recumbent position model (S109).

  If the target distance is not equal to or less than the predetermined threshold (S105: NO), the control unit 110 determines whether the human body / head size ratio is equal to or more than the predetermined threshold (S110). The threshold may be set experimentally from the viewpoint of posture estimation accuracy. If the human body / head size ratio is equal to or greater than the predetermined threshold (S110: YES), the control unit 110 determines whether the aspect ratio is equal to or greater than the predetermined threshold (S111).

  When the aspect ratio is equal to or more than the predetermined threshold (S111: YES), the control unit 110 selects the dictionary 3 corresponding to the distant / diagonal standing model (S112). When the aspect ratio is not equal to or more than the predetermined threshold (S111: NO), the control unit 110 selects the dictionary 4 corresponding to the distant / front standing model (S113).

  When the human body / head size ratio is not equal to or larger than the predetermined threshold (S110: NO), the control unit 110 selects the dictionary 5 corresponding to the distant / sitting prone position model (S114).

  The control unit 110 estimates the posture of the subject 400 by calculating joint points of the captured image by machine learning using the selected dictionary (S115).

  The control unit 110 reversely rotates the posture based on the joint point to calculate the posture of the target person 400 corresponding to the photographed image before rotation as a joint point (S116).

  The control unit 110 estimates the action of the target person 400 based on the posture by the joint point after the reverse rotation (S117).

  The present embodiment has the following effects.

  The direction of the subject's head in the captured image is unified into a predetermined direction by rotating the captured image, and the joint point of the target person is estimated as a posture by machine learning based on the captured image after rotation. Thus, the posture of the subject can be estimated with high accuracy with a small amount of learning data regardless of the posture of the subject in the photographed image.

  Furthermore, the human area is detected as an area in the outline of the human image or an area in a rectangle circumscribed by the outline of the human image, and the head area is an area in the outline of the subject's head or the head The image is detected as an area in a rectangle circumscribed by the outline of the image of the part. Then, the direction of the head region with respect to a predetermined point in the human region in the photographed image is taken as the direction of the head. Thereby, the posture of the subject can be estimated more accurately.

  Furthermore, a predetermined direction for unifying the direction of the head is taken as the vertical upper direction in the photographed image. As a result, since the learning data can be easily increased, the posture of the subject can be easily and accurately estimated.

  Furthermore, the direction of the head region is a direction from the center of gravity of the human region, which is a predetermined point, to the center of gravity of the head region. Thereby, the direction of the head can be appropriately unified by rotation of the photographed image.

  Furthermore, estimation of the posture of the subject by joint points is performed by Deep Pose. Thus, highly reliable attitude estimation can be realized.

  Furthermore, when the distance from the center of the captured image to the human region is less than a predetermined threshold value, the approximate ellipse of the human region is calculated, and the direction of the major axis of the approximate ellipse is horizontal in the captured image. Is rotated again, and the posture is estimated based on the photographed image after the rotation again. As a result, by photographing from a distance, even if the direction of the head is not clear, it is possible to estimate the posture of the subject with high accuracy.

  Furthermore, a dictionary used for machine learning is selected based on the distance from the center of the photographed image to the human region. As a result, it is possible to improve the estimation accuracy of the posture of the object person by changing the dictionary between the photographed image photographed from a short distance and the photographed image photographed from a long distance.

  Furthermore, a dictionary used for machine learning is selected based on the head size. In this way, it is possible to improve the estimation accuracy of the posture of the object person by changing the dictionary between the one with a high possibility of standing and the other with a high possibility of standing in particular for a photographed image taken from a short distance.

  Furthermore, a dictionary used for machine learning is selected based on the human body / head size ratio. As a result, it is possible to improve the estimation accuracy of the posture of the object person by changing the dictionary between the one with a high possibility of standing and the other with respect to a photographed image photographed from a particularly long distance.

  Furthermore, a dictionary used for machine learning is selected based on the aspect ratio of the human rectangle. In this way, it is possible to use a dictionary with a high probability that the target person was photographed from an oblique direction, and that the target person was particularly photographed from the front or back side. By changing it, the estimation accuracy of the posture of the subject can be improved.

  Furthermore, a dictionary used for machine learning is selected based on at least one of the distance from the center of the captured image to the human region, the head size, the human body / head size ratio, and the aspect ratio. As a result, more accurate posture estimation can be realized with less learning data by selecting a more appropriate dictionary.

  Furthermore, the posture corresponding to the photographed image before rotation is calculated by reversely rotating the posture estimated as the joint point corresponding to the photographed image after rotation, and based on the change among the calculated postures. Estimate the subject's behavior. This makes it possible to estimate the behavior of the subject with high accuracy with a small amount of learning data.

  The present invention is not limited to the embodiments described above.

  For example, by rotating the photographed image, the head direction may be unified to a predetermined direction other than the vertical upper direction of the photographed image. For example, the predetermined direction may be the horizontal direction of the captured image.

  Moreover, although the captured image is divided into five models and the dictionary used for machine learning is selected, the number of models may be less than five. That is, for example, it may be divided into two models based only on the magnitude of the target distance, and a dictionary corresponding to each model may be selected.

  In addition, part or all of the processing executed by the program in the embodiment may be replaced with hardware such as a circuit.

10 attitude estimation system,
100 attitude estimation devices,
200 shooting device,
300 communication network.

Claims (13)

A posture estimation system for estimating a posture of a subject based on a captured image, comprising:
A human area detection unit configured to detect an area including an image of the subject in the captured image as a human area;
A head area detection unit which detects an area including an image of the head of the subject in the photographed image as a head area;
An image rotation for unifying the direction of the head region with respect to a predetermined point in the human region in the photographed image in which a plurality of postures of the target person are respectively captured, by rotating the photographed image. Department,
A posture estimation unit configured to estimate the joint point of the subject as a posture by machine learning based on the captured image after rotation;
Attitude estimation system having.   The human region is a region within the contour of the image of the subject person or a region within a rectangle circumscribed by the contour of the image of the subject person, and the head region is within the contour of the image of the head of the subject person The posture estimation system according to claim 1, wherein the region is a region within a rectangle circumscribed by the region of or the contour of the image of the head.   The posture estimation system according to claim 1, wherein the predetermined direction is a vertically upward direction in the photographed image.   The posture estimation system according to any one of claims 1 to 3, wherein a direction of the head region is a direction from a center of gravity of the human region which is the predetermined point to a center of gravity of the head region.   The posture estimation system according to any one of claims 1 to 4, wherein the posture estimation unit estimates the posture of the subject by Deep Pose. A distance calculation unit that calculates a distance from the center of the captured image to the human region;
An approximate ellipse calculation unit that calculates an approximate ellipse of the human region in the photographed image after rotation by the image rotation unit when the distance is equal to or less than a predetermined threshold;
The image re-rotating unit further rotates the photographed image after rotation so that the direction of the major axis of the approximate ellipse is horizontal in the photographed image.
The posture estimation system according to any one of claims 1 to 5, wherein the posture estimation unit estimates the posture of the subject based on the photographed image after rotation by the image re-rotation unit.   The posture estimation system according to claim 6, wherein the posture estimation unit selects a dictionary used in the machine learning based on the distance calculated by the distance calculation unit. The image processing apparatus further includes a head size calculation unit that calculates the size of the head region in the captured image as a head size,
The posture estimation system according to claim 6, wherein the posture estimation unit selects a dictionary used for the machine learning based on the head size. The human body / head size ratio calculator further includes a ratio of the area of the human area to the area of the head area as a human body / head size ratio,
The posture estimation system according to any one of claims 1 to 6, wherein the posture estimation unit selects a dictionary used in the machine learning based on the human body / head size ratio. The human region is a region within a rectangle circumscribed by the contour of the image of the subject,
The image processing apparatus further includes an aspect ratio calculation unit that calculates an aspect ratio that is a ratio of the length of the vertical side of the rectangle to the length of the horizontal side of the rectangle,
The posture estimation system according to claim 2, wherein the posture estimation unit selects a dictionary used in the machine learning based on the aspect ratio. A distance calculation unit that calculates a distance from the center of the captured image to the human region;
A head size calculation unit that calculates the size of the head region in the captured image as a head size;
A human body / head size ratio calculating unit that calculates a ratio of an area of the head area to an area of the human area as a human body / head size ratio;
Aspect ratio calculation for calculating an aspect ratio which is a ratio of the length of the vertical side of the human region to the length of the horizontal side when the human region is a region within a rectangle circumscribed by the contour of the image of the subject person Have a part, and
The posture estimation unit selects a dictionary used in the machine learning based on at least one of the calculated distance, the head size, the human body / head size ratio, and the aspect ratio. The posture estimation system according to any one of 1 to 5. The posture estimation system according to any one of claims 1 to 11.
A pre-rotation posture calculation unit that calculates the posture corresponding to the captured image before rotation by reversely rotating the posture estimated as the joint point corresponding to the captured image after rotation;
An action estimation unit that estimates the action of the target person based on changes between the plurality of postures calculated by the pre-rotation posture calculation unit;
An action estimation system having Detecting a region including an image of a subject in a captured image as a human region (a);
Detecting a region including an image of the head of the subject in the captured image as a head region (b);
Procedure for unifying the direction of the head region with respect to a predetermined point in the human region in the photographed image in which a plurality of postures of the subject are respectively imaged, by rotating the photographed image in a predetermined direction (c )When,
A procedure (d) of estimating a joint position of the subject by machine learning based on the captured image after rotation;
Pose estimation program to make a computer execute.