JP7353438B2 - Video image judgment method - Google Patents

Description

The present invention relates to a video image determination method for determining whether a video image of a walking subject satisfies gait analysis conditions.

There is a technology that compares a moving image of the user in a moving image shot in advance with a captured moving image of the user and evaluates the degree of difference in the content of the action (for example, Patent Document 1 ).

Japanese Patent Application Publication No. 2010-258613

In the case of Patent Document 1, since the conditions under which the user's video image was taken are not determined, for example, the operator may also judge data that is not worthy of comparative evaluation due to poor shooting conditions of the user's video image itself. It was decided that a comparison would be made with the video images.

The present invention has been made in view of the above problems, and relates to a technique for acquiring a moving image and determining whether the acquired moving image satisfies walking analysis conditions.

The present invention satisfies a gait analysis condition for determining whether gait analysis is possible, such that a moving image of a walking subject is photographed, and includes at least one of a predetermined walking condition or a predetermined photographing condition. A moving image determining method for determining whether the moving image The present invention relates to a moving image determining method in which the image is composed of a plurality of frames, and the walking moving image determining step performs the determination after all frames are acquired in the moving image acquiring step or for each acquired frame.

According to the method provided by the present invention, by determining whether the acquired moving image of a walking subject satisfies the gait analysis conditions, only the moving images that satisfy the gait analysis conditions are targeted for gait analysis. It is also possible to reduce the amount of processing required.

3 is a flowchart showing a moving image determination method. It is a flowchart showing details of a determination process. It is a flowchart showing details of a determination process. FIG. 3 is a diagram showing skeleton points and centroids to be extracted. FIG. 3 is an explanatory diagram for estimating the position of a subject's face. (a) is a screen image diagram in which a graph showing a gait analysis result includes a total score based on the determination result of a moving image, and (b) is a screen image diagram showing determination results for each determination condition of a moving image. 1 is a conceptual diagram of a mobile terminal 100. FIG. (a) is a screen image diagram of the display device of the mobile terminal 100, and (b) is a screen image diagram (another example) of the display device of the mobile terminal 100. It is a flowchart in which a walking path guide process and a subject mask process are added to the video acquisition process. (a) is a screen image of the walking route guide display (when there is no subject in the shooting range), (b) is a screen image of the walking route guide display (when there is a subject in the shooting range), (c) is the walking route guide display (d) is a screen image of the walking path guide display (when the position of the subject and the imaging device is not appropriate). (a) is a screen image diagram of the subject mask display (when the subject is not within the predetermined range), and (b) is a screen image diagram of the subject mask display (when the subject is within the predetermined range).

Embodiments of the present invention will be described below.
First, an overview of this embodiment will be explained.
The moving image determination method (hereinafter sometimes referred to as the present method) of the present embodiment is a method for determining whether a moving image of a walking subject satisfies gait analysis conditions. This method analyzes a moving image taken of a walking subject and determines whether the moving image satisfies the gait analysis conditions when analyzing the gait manner of the subject. Since it is possible to analyze only the moving images that satisfy the walking condition of the subject, it is not necessary to analyze the moving images that do not satisfy the walking analysis conditions, and it is possible to reduce the processing load.
This method includes a video acquisition step and a walking video determination step. The moving image acquisition step is a step of acquiring moving image data in which a walking subject is photographed. The walking video determination step determines whether the acquired video data satisfies gait analysis conditions for determining that gait analysis is possible, which consists of at least one of predetermined walking conditions or predetermined shooting conditions. This is the process of The moving image determined by this method is composed of a plurality of frames, and in the walking moving image determining step, the determination is performed after all frames are acquired in the moving image acquiring step, or for each acquired frame.

This method will be explained in detail below.
The walking subjects may be of any age, from children such as infants, toddlers, and children to adults. However, since the analysis condition is that they are walking, infants and elderly people who cannot walk independently are not applicable.
The photographed moving image may be one taken at any location, and is not limited to a laboratory where gait measurement is performed, but also includes a moving image taken in daily life. The photograph may be taken with any photographic device, and in addition to digital cameras and video cameras, photographic devices and personal computers equipped with mobile phones, tablet terminals, PDAs, and mobile personal computers, which are generally referred to as mobile terminals, may be used. The target is a moving image taken with a photographing device. It can be taken by any photographer. The moving image data is data composed of multiple frames.
Gait analysis conditions are defined as at least one of predetermined gait conditions or predetermined imaging conditions necessary for analyzing the gait behavior of a walking subject photographed in the acquired video image data. This is a condition for determining that it is possible. The predetermined walking conditions include whether the subject is walking a predetermined number of steps or more, the subject's walking speed, whether the subject is walking within a predetermined range (does the left/right sway exceed a predetermined range), and whether the subject's face is facing forward? These include whether the subject is facing the subject, and whether the subject's hands are within a predetermined range. The predetermined photographing conditions include whether the photograph is taken so that skeletal data can be obtained, whether the whole body is photographed, whether the distance between the photographing device and the subject is within a predetermined range, and whether the photograph is taken over a predetermined number of steps.
Determining whether the gait analysis conditions for determining whether gait analysis is possible is not limited to the fact that a given gait analysis is not possible (or impossible) unless the gait analysis conditions are met; Although analysis can be performed (predetermined gait analysis processing can be executed), it also includes determining whether a predetermined gait analysis result (gait analysis result with a predetermined accuracy) cannot be obtained. The details will be described later, but specifically, among the predetermined walking conditions or predetermined shooting conditions determined by this method, there are some conditions in which the predetermined gait analysis cannot be performed with video data that does not satisfy the conditions, and Even if the video data does not satisfy certain conditions, it is possible to perform a predetermined gait analysis, for example by sacrificing accuracy. In other words, there is a high possibility that a prescribed gait analysis result (a gait analysis result with a predetermined accuracy) cannot be obtained for video data that has not been determined to be capable of gait analysis. Contains things you can do. Video data that does not meet the conditions necessary for performing a prescribed gait analysis cannot be used for the prescribed gait analysis, but it is possible to perform a prescribed gait analysis even if the prescribed gait conditions or photography conditions are not met. For video data that can be processed, a predetermined gait analysis is performed, and the gait analysis results include information on which predetermined walking conditions or predetermined shooting conditions the video data satisfies or does not satisfy. good. Furthermore, when determining walking analysis conditions, if there are a plurality of predetermined walking conditions or predetermined photographing conditions, priority may be given in the order in which the conditions are determined. By prioritizing, it is possible to reduce the processing load.
Obtaining moving images refers to obtaining data shot by a shooting device one after another, obtaining data shot in the past via a network or a predetermined medium, and moving images stored on a predetermined server. This includes things obtained by accessing image data.
In the walking video determination step, the data captured by the imaging device may be sequentially obtained in the video acquisition step, and the determination may be performed for each acquired frame, or the determination may be performed after all frames of previously captured data have been acquired. Good too.

The processing flow of this method will be explained using FIG. 1. The flowchart shown in FIG. 1 is also a flowchart of the moving image determination application software processed by this method.
The process (step S100) is a video acquisition process of acquiring a video image of a walking subject.

The process (step S110) determines whether the acquired moving image satisfies a gait analysis condition for determining that gait analysis is possible, which is composed of at least one of a predetermined walking condition or a predetermined shooting condition. This is a walking video determination process. Details of the walking video determination process are shown in Figures 2-1 and 2-2. In the walking video determination step, determination is performed on a frame-by-frame basis. In this embodiment, when the number of frames constituting a moving image is n, the determination is made in units of one frame, and the steps S111 to S130 shown in FIGS. 2-1 and 2-2 are performed for each frame. Although the steps shown in FIGS. 2-1 and 2-2 are performed for n times, the process may be repeated for n times before proceeding to the next step.

In this embodiment, a skeletal point learning model obtained by machine learning is used to identify a subject by extracting the skeletal points of the subject from an acquired video image, and perform a gait analysis of the subject. In this embodiment, we use known skeletal point extraction techniques such as OpenPose (URL=https://github.com/CMU-Perceptual-Computing-Lab/openpose) and VisionPose (URL=https://www.next-system Extract using .com/visionpose) etc.
In the step (step S111), three-dimensional skeleton points are extracted from the acquired moving image. When a moving image includes multiple subjects, three-dimensional skeleton points are extracted for each subject. FIG. 3 shows three-dimensional skeleton points extracted from a moving image.
In this embodiment, the 16 skeleton points and centroids shown in FIG. 3 are extracted. Since the center of gravity is used in gait analysis, it is performed together with the extraction of skeletal points, but it does not have to be performed at the same time. The center of gravity may be calculated from the extracted skeleton points. Although 16 skeleton points are extracted in this embodiment, the number of skeleton points to be extracted is not limited to this.
Note that the process (step S111) corresponds to a skeleton point extraction process.

In the process (step S112), it is determined whether the skeletal points and the center of gravity extracted in step S111 satisfy the gait analysis conditions for one subject. The gait analysis condition in step S112 is one of the predetermined imaging conditions, and in this embodiment, there is a frame from which a predetermined number (16 in this embodiment) of skeletal points and centers of gravity can be extracted for one subject. be. If it is determined that the gait analysis conditions are satisfied, the process advances to step S113, and if it is determined that the gait analysis conditions are not satisfied, this process is ended. If a predetermined number of skeletal points and center of gravity are not extracted for a single subject, it is considered that the video image does not have the clarity or number of pixels suitable for gait analysis, and gait analysis cannot be performed, or even if it is performed. Since accurate analysis cannot be performed, subsequent processing is not performed.
Note that the step (step S112) corresponds to a skeleton point determination step.

In the process (step S113), it is determined whether the subject's whole body is being photographed. In this embodiment, it is determined that the subject's whole body is being imaged when the skeletal points have a predetermined positional relationship. It is determined whether the skeleton points extracted in step S112 are in a predetermined positional relationship, and if it is determined that they are in a predetermined positional relationship, the process proceeds to step S114, and the skeleton points are in a predetermined positional relationship. If it is not determined that this is the case, this process ends. When skeletal points are in a predetermined positional relationship, for example, a skeletal point indicating the top of the head is located at the left and right acromion (acromion) at the largest value of the X coordinate shown in Figure 3 among the extracted skeletal points. ) is located in a predetermined positional relationship between the skeletal points, such as the skeletal point indicating the ankle having a larger value of the X coordinate than the skeletal point indicating the ankle. If the extracted skeletal points do not have a predetermined positional relationship, the moving image cannot be subjected to gait analysis, or even if it is performed, the analysis cannot be performed accurately, so the subsequent processing is not performed. Only frames in which the extracted skeleton points have a predetermined positional relationship are subjected to the subsequent processing.
Note that the step (step S113) corresponds to a whole body determination step.

The step (step S114) is a step of determining whether there is a frame in which the subject is photographed within a predetermined range with respect to the photographing device. The closer the distance between the imaging device and the subject, the better the accuracy of skeletal point extraction, so if the video image is taken at a position where the imaging device and the subject are within a predetermined range, it may not be suitable for gait analysis. This judgment is made because the quality is high. This judgment is an essential judgment condition.
In this step, the human body dimensions of the subject are acquired, and the length in the video image of the body part corresponding to the acquired human body dimensions (e.g., the number of pixels in the video image) and the depth direction between the shooting position of the video image and the position of the subject are calculated. The ratio with the distance (distance in the Y-axis direction in Figure 3) is calculated, and the actual length of the depth dimension is estimated using this ratio and the actual length of the human body dimension obtained from the subject, and the estimation result is Determine whether the analysis conditions are met. Human body dimensions refer to the dimensions of each part of the human body, and specifically include height, eye height, acromion height, front arm length, shoulder width, center of gravity height, knee joint height, etc. The human body dimensions may be selected from one or more of those exemplified above. When a plurality of human body dimensions are selected, the above-mentioned ratio may be calculated for each human body dimension, and then the average value of the estimated values of the actual lengths of the above-mentioned depth dimensions may be calculated as the estimation result. Further, the human body size may be selected depending on the body of the subject reflected in the moving image. For example, if the subject's feet are not sufficiently reflected in the video image, it is preferable to select front arm length and shoulder width that allow the human body dimensions (the number of pixels in the video image) to be determined without using the foot position. Instead of using human body dimensions, the distance in the depth direction (the distance in the Y-axis direction in Figure 3) between the shooting position of the moving image and the subject's position is optically estimated by utilizing light reflection as described later. However, it may be determined whether the estimation result satisfies the gait analysis conditions. The gait analysis condition in step S114 is one of the predetermined shooting conditions, and is that the frame of the moving image is shot at a distance of 1 to 7 meters between the shooting position of the moving image and the position of the subject. If it is determined that the gait analysis conditions are satisfied, the process proceeds to step S116, and if it is determined that the gait analysis conditions are not satisfied, the process proceeds to step S115.
Note that the process (step S114) corresponds to a skeletal point determination process, a human body size process, and a distance estimation process.

The process (step S115) is a process performed when it is determined in step S114 that the walking analysis conditions are not satisfied. In this step, as the determination result in step S114, a result indicating that the frame is not a moving image shot at a distance of 1 to 7 meters between the shooting position of the moving image and the subject's position is registered, and the process proceeds to step S116.

The step (step S116) is a step of determining whether there is a moving image in which the subject walks four or more steps. For gait analysis, it is desirable to use video images of four or more consecutive steps in order to perform measurements in one gait cycle on each side.For example, left heel touches the ground → right heel touches the ground → left heel touches the ground → right heel This judgment is made because a video image including walking while touching the ground is required. This judgment is an essential judgment condition.
This process is performed for the video frame determined to satisfy the gait analysis conditions in step S114. Therefore, the process is not performed on video frames for which it is not determined in step S114 that the gait analysis conditions are satisfied and for which the determination result to that effect is registered in step S115. In this step, the positions where the left and right heels touch the ground are estimated from the skeletal points and centers of gravity of the left and right ankles extracted in step S111. Specifically, when the center of gravity is at its lowest position (when the value in the X direction in Figure 3 is the smallest), it can be estimated that the heel is in contact with the ground. It can be assumed that the left heel touched the ground when the heel was positioned at the position (when the value in the Y direction in Fig. 3 is the largest), and that the right heel touched the ground when the center of gravity was located furthest forward relative to the skeletal point of the right ankle. It can be estimated. In this way, it is determined whether there is a moving image in which the heel alternately touches the ground twice. The gait analysis condition in step S116 is one of the predetermined gait conditions, and is whether there is a video in which the heel alternately touches the ground twice (or there is a video in which the heel is walked for four or more steps). If it is determined that the gait analysis conditions are satisfied, the process proceeds to step S118, and if it is determined that the gait analysis conditions are not satisfied, the process proceeds to step S117.
Note that the step (step S116) corresponds to a skeleton point determination step and a step count determination step.

The process (step S117) is a process performed when it is determined in step S116 that the walking analysis conditions are not satisfied. In this step, as the determination result in step S116, a result indicating that there is no video in which the user walks four steps or more is registered, and the process advances to step S118.

The process (step S118) is a process performed when it is determined in step S114 and step S116 that the walking analysis conditions are satisfied. In this step, because it is determined that the gait analysis conditions are satisfied in step S114 and step S116, in other words, the frame of the video is taken at a distance of 1 to 7 m between the video shooting position and the subject's position. If the user has walked four or more steps, one point is added to the total score, and the process advances to step S119.
Here, the total score will be explained. The overall score is used to quantify the extent to which the judged video image satisfies the gait analysis conditions and how suitable it is for gait analysis, based on the results of each judgment made in the gait video judgment process. It is used.

The step (step S119) is a step of determining whether there is a moving image in which the subject walks in a straight line. In general gait analysis, a video image of a subject naturally walking in a straight line toward a destination is analyzed, so a video image in which the subject walks meandering from side to side is not suitable for gait analysis. Therefore, it is determined whether the subject is walking within a certain range in the left and right directions. This judgment is an essential judgment condition.
In this step, the center of gravity of the subject is determined, and it is determined whether the movement range of the center of gravity in the left-right direction (Z-axis direction in FIG. 3) with respect to the imaging device is within a predetermined range. In this embodiment, the center of gravity extracted in step S111 is used as the center of gravity of the subject. The gait analysis condition in step S119 is one of the predetermined gait conditions, and is that the subject's center of gravity moves within 1 meter in the left-right direction with respect to the imaging device. If it is determined that the gait analysis conditions are satisfied, the process proceeds to step S121, and if it is determined that the gait analysis conditions are not satisfied, the process proceeds to step S120.
Note that the step (step S119) corresponds to a skeleton point determination step and a center of gravity determination step.

The process (step S120) is a process performed when it is determined in step S119 that the walking analysis conditions are not satisfied. In this process, as a determination result in step S119, a result indicating that the subject's center of gravity is not a moving image in which the movement range in the left and right direction with respect to the imaging device is within 1 meter (he is not walking in a straight line) is registered, and in step S122 Proceed to.

The process (step S121) is a process performed when it is determined in step S119 that the walking analysis conditions are satisfied. In this step, since it is determined in step S119 that the gait analysis conditions are satisfied, one point is added to the total score, and the process proceeds to step S122.

The step (step S122) is a step of determining whether there is a moving image in which the subject is not running. In gait analysis, running data is not analyzed, so video images of the subject running are not suitable for gait analysis. Therefore, it is determined whether there is a video in which the subject is not running. This judgment is an essential judgment condition.
The determination in this step is performed by determining a gait index that suggests that the subject is running.
We will explain the gait index that suggests that the subject is running.
Gait indicators that suggest that the subject is running include walking speed, time in which the right foot and left foot are in contact with the ground, and changes in foot position.
In the case of walking speed, the time required to move a predetermined distance (for example, 5 m) is measured based on the subject's center of gravity position, and the speed is calculated. As the subject's center of gravity position used at this time, the center of gravity position extracted in step S111 may be used.
In the case of the time when the right foot and the left foot are in contact with the ground, the determination is made based on whether or not there is a time when neither the right heel nor the left heel is in contact with the ground. If there is a time when neither the right heel nor the left heel is in contact with the ground, it is determined that the user is running. Whether or not the right heel and the left heel are in contact with the ground is estimated using the same method as the step count determination step in step S116.
In the case of a change in foot position, if the vertical movement range (in the X-axis direction in FIG. 3) of the skeleton points of the left and right ankles extracted in step S111 is greater than or equal to a predetermined range, it is determined that the user is running.
The walking analysis condition in step S122 is one of predetermined walking conditions, and is that there is a frame in which it is determined that the user is not running based on any of the walking indicators described above. If it is determined that the gait analysis conditions are satisfied, the process proceeds to step S124, and if it is determined that the gait analysis conditions are not satisfied, the process proceeds to step S123.
Note that the process (step S122) corresponds to a skeletal point determination process and a walking determination process.

The process (step S123) is a process performed when it is determined in step S122 that the walking analysis conditions are not satisfied. In this step, as the determination result in step S122, a result indicating that there is no video image in which it has been determined that the subject is not running is registered, and the process proceeds to step S125.

The process (step S124) is a process performed when it is determined in step S122 that the walking analysis conditions are satisfied. In this step, since it is determined in step S122 that the gait analysis conditions are satisfied, one point is added to the total score, and the process proceeds to step S125.

The process (step S125) is a process of determining whether the orientation of the subject's face in the moving image is within a predetermined range in the vertical and horizontal directions. If the orientation of the subject's face exceeds the predetermined range in the vertical, horizontal, or horizontal directions, there is a high possibility that the subject is walking looking away, and the video is not suitable for gait analysis. Therefore, it is determined whether the subject is walking with his or her face facing within a certain range in the vertical and horizontal directions. This determination is a desirable determination condition.
In this step, determination is made using the vertex (head), glabella, neck, left and right acromion, and center of gravity position extracted in step S111. The determination method in this step will be explained using FIG. 4.
A head coordinate system and a body coordinate system are created from the crown of the head, between the eyebrows, the neck, the left and right acromions, and the center of gravity. The head coordinate system consists of the X-axis (vertical axis) passing through the top of the head and neck, the Y-axis (front-back axis) passing between the eyebrows and perpendicular to the X-axis, and the Z-axis (lateral-lateral axis) perpendicular to the X-axis and Y-axis. It consists of The body coordinate system consists of an X-axis (vertical axis) passing through the center of gravity and the midpoint of the left and right acromions, a Z-axis (lateral axis) parallel to a straight line passing through the left and right acromions, and a Y-axis perpendicular to the X-axis and Z-axis. (Anteroposterior direction axis) Although the heights of the Z-axis and Y-axis (X-axis intercept) of the body coordinate system are arbitrary, they can be set to the height of the center of gravity of a triangle whose three vertices are the left and right acromions and the center of gravity. When the head coordinate system is within a predetermined angle (for example, ±30 degrees) in the vertical and horizontal directions with respect to the body coordinate system, it is determined that the orientation of the subject's face is within a predetermined range in the vertical and horizontal directions. More specifically, if the angle formed by the X-axis of the head coordinate system and the X-axis of the body coordinate system is 30 degrees or less, it is determined that the head tilt is small. Furthermore, if the angle formed by the Y-axis of the head coordinate system and the Y-axis of the body coordinate system is 30 degrees or less, it is determined that the left and right direction of the face (looking away) is small. The Z-axis is used instead of the Y-axis, and if the angle between the Z-axis of the head coordinate system and the Z-axis of the body coordinate system is less than 30 degrees, it is determined that the left and right direction of the face (looking away) is small. Good too. The walking analysis condition in step S125 is one of the predetermined walking conditions, and is that the movement range of the subject's face in the vertical and horizontal directions is within ±30 degrees. In the above example, the permissible angle in the vertical direction of the head and the permissible angle in the horizontal direction of the head are both ±30 degrees, but the present invention is not limited to this. Walking with your head tilted in the vertical direction, that is, looking down while looking down at your feet, is easier to maintain a normal gait than walking while looking away from the left and right, so the permissible vertical angle of the head is may be set larger than the permissible angle in the horizontal direction of the head.
If the orientation of the subject's face exceeds ±30 degrees for only a short period of time (number of frames) in the video, it is preferable to determine that the video as a whole is suitable for gait analysis. Therefore, in the condition determination in step S125, it is preferable to set in advance a "threshold ratio" that allows the face direction to fluctuate vertically or horizontally. If the time or number of frames in which at least one of the vertical or horizontal directions of the subject's face exceeds ±30 degrees is equal to or greater than this threshold percentage of the entire walking time or number of frames, the relevant It is preferable to determine that the walking is inappropriate, that is, that the walking analysis conditions are not satisfied. This threshold ratio is preferably 10% or more and 30% or less. If it is determined that the predetermined condition is satisfied, the process proceeds to step S127, and if it is determined that the gait analysis condition is not satisfied, the process proceeds to step S126.
Note that the step (step S125) corresponds to a skeleton point determination step and a face determination step.

The process (step S126) is a process performed when it is determined in step S125 that the walking analysis conditions are not satisfied. In this step, a result indicating that the subject's face direction is not within a predetermined range in the vertical and horizontal directions is registered as the determination result in step S125, and the process proceeds to step S128.

The process (step S127) is a process performed when it is determined in step S125 that the walking analysis conditions are satisfied. In this step, since it is determined in step S125 that the gait analysis conditions are satisfied, one point is added to the total score, and the process proceeds to step S128.

The process (step S128) is a process of determining whether the positions of both hands of the subject in the moving image are below a predetermined position. In a natural walking state, both wrists are at a lower position than the prescribed position (elbows), so if the subject's hands are higher than the prescribed position, there is a high possibility that the subject is holding something in his hands, making the video suitable for gait analysis. isn't it. Therefore, it is determined whether both hands of the subject are below a predetermined position. This determination is a desirable determination condition.
In this step, both hands of the subject are determined, and it is determined whether the positions of both hands are below a predetermined position. In this embodiment, the predetermined positions are the positions of both elbows. In this embodiment, the positions of the skeletal points of both wrists of the subject extracted in step S111 are compared with the positions of the skeletal points of both elbows, and the positions of the skeletal points of both wrists are lower than the positions of the skeletal points of both elbows. It is determined whether the value of the X axis in FIG. 3 is small. The gait analysis condition in step S128 is one of the predetermined gait conditions, and is that the position of both hands of the subject is lower than the position of both elbows. If it is determined that the gait analysis conditions are satisfied, the process proceeds to step S130, and if it is determined that the gait analysis conditions are not satisfied, the process proceeds to step S129.
Note that the step (step S128) corresponds to a skeleton point determination step and a both hands determination step.

The process (step S129) is a process performed when it is determined in step S128 that the walking analysis conditions are not satisfied. In this process, as the determination result in step S128, a result indicating that the position of both hands of the subject is not below a predetermined position is registered in the moving image, and this process ends.

The process (step S130) is a process performed when it is determined in step S128 that the walking analysis conditions are satisfied. In this process, since it is determined in step S128 that the gait analysis conditions are satisfied, one point is added to the total score, and the process ends.

Next, the content of evaluation using the total score will be explained.
As described above, a predetermined point is added to the total score according to the determination results in steps S114, S116, S119, S122, S125, and S128. Based on this total score, it is possible to evaluate whether the moving image is suitable for gait analysis. Furthermore, it is also possible to evaluate the accuracy of the walking analysis results using the video image based on the total score. In this embodiment, regardless of the gait analysis conditions, if the gait analysis conditions are met, one point is added as a predetermined point, but the points to be added may vary depending on the gait analysis conditions. For example, the points to be added may be different depending on an essential judgment condition and a desirable judgment condition. The total score can be output regardless of whether it is in paper or electronic media. Further, a total score may be included in the result of performing gait analysis processing using the determined moving image. By doing this, it is possible to know what kind of moving image was used to obtain the result of the gait analysis process, which can be used as a reference for the accuracy of the result of the gait analysis process.

FIGS. 5(a) and 5(b) show output examples of determination results and total scores. In this embodiment, a case is shown in which the result of determining whether a moving image used to analyze the gait development of an infant is suitable for gait analysis is output to a display device included in a mobile terminal.
The horizontal axis in FIG. 5(a) indicates the date when the moving image was taken, and the vertical axis indicates the number of days of development. The number of days of development here refers to the number of days of gait development of the subject (infant) calculated by performing a prescribed gait analysis using a video image that has been determined to be suitable for gait analysis. be. The accuracy of the calculated number of days for gait development is affected by the judgment results of the video images used for gait analysis and the overall score. In FIG. 5A, by changing the color of the plot according to the total score, it is possible to understand what kind of determination result the moving image used for gait analysis has. For example, in descending order of overall score, green (white in Figure 5(a)) → yellow (shaded with small dots in Figure 5(a)) → orange (shaded with large dots in Figure 5(a)). Plot. The color of the plot is green when the total score is a perfect score, yellow when the total score is -1 to -3 points, and orange when it is less than that. The color of the plot and the corresponding score are preferably set in consideration of the content of the evaluation.
FIG. 5B shows the determination results for each determination condition for a moving image on a predetermined shooting date. In FIG. 5(b), for example, by specifying the shooting date (click or touch), the corresponding judgment result may be displayed, or by specifying the plot corresponding to the shooting date (click or touch). , the corresponding determination result may be displayed. Depending on the determination result for each determination condition, the color of the box displayed alongside the determination conditions (for example, the number of steps in FIG. 5(b)) is changed. In FIG. 5(b), since the number of steps determined in step S116 satisfies the walking analysis condition of 4 steps, a green color (white in FIG. 5(b)) indicating a good determination result is displayed, and step S119 Regarding the straight-line walking determined in , since it was diagonal walking, orange color (shading with large dots in FIG. 5(b)) indicating a bad determination result is displayed. In this way, it is possible to determine whether the overall score is high or low based on what conditions are determined, and this can be reflected in subsequent photographing.

In this way, by determining a moving image under various predetermined conditions, it is possible to evaluate whether the moving image is suitable for walking analysis. Furthermore, since moving images that cannot be subjected to gait analysis can be excluded from analysis before gait analysis is performed, processing efficiency can be improved. Further, it becomes possible to visually understand the determination result. Furthermore, since the determination result of the video image is reflected in the gait analysis result using the video image, it becomes easier to understand what kind of determination result the gait analysis result using the video image has.

<Information processing terminal>
Next, an information processing terminal (portable terminal) 100 including a photographing device that executes the above-described moving image determination method will be described. A conceptual diagram of the information processing terminal (portable terminal) 100 is shown in FIG.
If the information processing terminal (mobile terminal) 100 is equipped with a photographing device (camera) 110, it can be a mobile phone, tablet terminal, PDA, mobile personal computer, wearable device, personal computer, etc., which are generally referred to as mobile terminals. Either is fine. The mobile terminal 100 includes a photographing device 110 as a photographing means, a determining section 120 as a determining means, a storage section 130, and an attitude sensor 140. Although not shown in the drawings, it also includes input devices such as a keyboard (including a touch keyboard), a pointing device, an arithmetic processing unit, a display device, and the like.
The photographing device 110 is a means for photographing a moving image, and corresponds to a camera included in the mobile terminal 100.
The determination unit 120 has means for executing the above-described moving image determination method, and is included in the arithmetic processing device.
The storage unit 130 is a means for storing captured moving images. It is also a means for storing judgment results and evaluation results (total score) by the judgment unit 120, and also a means for storing a computer program for executing the above-described moving image judgment method.
The attitude sensor 140 is a sensor that measures the attitude of the mobile terminal 100, and is an acceleration sensor, an angular velocity sensor, etc. that the mobile terminal 100 has.

Since the mobile terminal 100 includes the photographing device 110, the photographing device 110 corresponds to the video acquisition step of the video image determination method described above.
Since the mobile terminal 100 is provided with the attitude sensor 140, the attitude of the photographing device 110 can be determined. Step S110 in FIG. 1 includes a process of determining the posture of the photographing device, and in this process, posture data is acquired from the posture sensor 140, and the arithmetic processing unit determines the posture of the photographing device 110 at the time of photographing based on the acquired posture data. If the posture is within a predetermined range, no advice will be displayed on the display device included in the mobile terminal 100, or "normal" will be displayed, and if the posture is outside of that, the message "The imaging device 110 is Advice such as "It's tilted" may be displayed. In this embodiment, the predetermined range is ±25 degrees in the left-right and front-back directions. With this configuration, it is possible to capture a moving image suitable for gait analysis, and it is possible to improve the accuracy of gait analysis using the captured (obtained) moving image.

In the mobile terminal 100, the determination unit 120 sequentially determines the video images photographed (acquired) by the photographing device 110 using the video determination method described above. At this time, the arithmetic processing device displays advice on the display device based on the determination results of step 114, step S116, step S119, step S122, step S125, and step S128 shown in FIGS. 2-1 and 2-2. It's okay. FIGS. 7A and 7B show screen images of the display device of the mobile terminal 100. For example, if it is not determined in step S128 that the position of both hands of the subject is lower than the position of both elbows, the arithmetic processing unit indicates to the display means that "both hands are not lowered" as shown in FIG. 7(a). and advice will be displayed. Similarly, if it is determined in step S114 that the distance between the photographing device 110 and the position of the subject is 1 to 7 meters and the subject is not photographed, the arithmetic processing unit causes the display means to display the image as shown in FIG. 7(b). display the following advice: ``The camera distance is not 1 to 7 meters.'' With this configuration, it is possible to capture a moving image suitable for gait analysis, and it is possible to improve the accuracy of gait analysis using the captured (obtained) moving image.

<Computer program>
A computer program (hereinafter referred to as this program) for causing the mobile terminal 100 to execute acquisition processing and determination processing is installed in the mobile terminal 100. In this embodiment, the computer program is the above-mentioned moving image determination application software.
The acquisition process is a process of acquiring a moving image of a walking subject. The source of the video image does not matter, such as from a predetermined medium, via a network, or from a photographing device. The determination process is a process of determining whether the walking condition or the photographing condition satisfies a predetermined condition. This program executes the processes shown in the flowcharts of FIGS. 1, 2-1, and 2-1.

<Modified example>
In this embodiment, a skeletal point learning model obtained by machine learning is used to identify a subject by extracting the skeletal points of the subject from the acquired video image, and the gait analysis of the subject is performed. Not limited to. For example, the subject may be identified through other image analysis, and the subject's movements may be extracted and gait analysis may be performed. In this case, the same effect can be obtained by setting conditions suitable for the image analysis as the gait analysis conditions to be determined.

In this embodiment, when performing the determination process in step S114, the human body size data of the subject is acquired, and the distance between the imaging device and the position of the subject is estimated based on the data and the position of the center of gravity of the subject. Not limited to. By setting a scale and a reference point at the shooting position, and acquiring moving images of the patient walking so that they are photographed together with the scale and reference point, and also extracting the scale and reference point, the distance between the imaging device and the subject's position can be calculated. It becomes possible to estimate. For example, it is effective when the shooting location can be fixed, such as determining the moving image of a user of a predetermined facility (nursery school, elderly care facility). In addition, remote sensing technology measures the distance from the subject to the imaging device by irradiating the subject with near-infrared light, visible light, and ultraviolet light, and capturing the reflected light with an optical sensor installed in the imaging device. may also be used. The optical sensor may be provided in the photographing device itself or may be installed near the photographing device. In this way, the distance in the depth direction (distance in the Y-axis direction in FIG. 3) between the shooting position of the moving image and the position of the subject may be optically estimated, and it may be determined whether the estimation result satisfies the gait analysis conditions. .

In this embodiment, it is determined whether or not the walking analysis condition, which is composed of at least one of a predetermined walking condition or a predetermined photographing condition, is satisfied. In addition to this, the walking analysis conditions may include whether the posture of the photographing device is within a predetermined range.
For example, when photographing a moving image using a photographing device included in the mobile terminal 100, posture information of the mobile terminal 100 at the time of photographing is added to the moving image and stored, and the posture information is acquired in the determination step. It may be determined that the gait analysis condition is satisfied if the posture information is within a predetermined range, and that the gait analysis condition is not satisfied if the posture information is not within the predetermined range. For example, after the process of step S130 shown in FIG. 2-2, as an attitude acquisition step and a determination step to obtain attitude information of the photographing device, it is determined whether the acquired attitude information is within ±25 degrees in the left-right, front-back, and front-back directions. All you need to do is provide a process. The gait analysis condition in this process is one of the predetermined photographing conditions, specifically whether the posture information of the photographing device is within ±25 degrees in the left-right and front-back directions. If it is determined that the gait analysis conditions are satisfied, it is preferable to add one point to the total score. With this configuration, it is possible to further evaluate whether the moving image is suitable for walking analysis.

In this embodiment, the mobile terminal 100 determines the attitude of the photographing device 110, and displays advice such as "normal" or "the photographing device 110 is tilted" on the display device based on the determination result. In addition to this, the determination unit 120 acquires posture information at the time of shooting the moving image, and if the posture information is within a predetermined range, the gait analysis condition is satisfied, and if the posture information is not within the predetermined range, the gait analysis condition is satisfied. It may be determined that the moving image does not satisfy the following conditions. In this way, based on the posture information from the posture sensor 140 of the mobile terminal 100, in addition to displaying advice regarding the photographing posture of the photographing device 110 during photographing, it is also possible to evaluate the photographed moving image.

The video images acquired in the video image acquisition process are processed through a walking path guide process that guides the photographer as to the range in which the subject should walk and the orientation of the imaging device, and a subject masking process that processes the captured video images so that the subject cannot be identified. It is also possible to use moving image data that has been subjected to Based on FIG. 8, the walking path guide process and the subject mask process will be described.

The process (step S101) is a process of determining whether or not the camera is in a shooting preparation state. The "photographing preparation state" is a state in which the photographing device 110 is ready for photographing and a subject is displayed on the photographing screen. If it is not in the shooting preparation state, the process moves to step S103, and if it is in the shooting preparation state, the process moves to step S102.

The process (step S102) is a process of displaying a walking route guide and a horizontal guide.
The "walking route guide" is one of the displays used by the photographer who shoots the walking video to check whether he or she has captured a moving image that allows accurate walking analysis. The optimum position for the subject to start walking is displayed on the photographic screen, as well as the optimum position for the walking direction. FIG. 9 shows a display example of the walking path guide. In FIG. 9A, there is no subject in the photographing range, so no subject is displayed on the photographing screen. The walking route guide 500 displayed on the photographic screen in FIG. 9(a) indicates the optimal position for the subject to start walking. The display position of the walking route guide 500 is set in consideration of the distance from the photographing device 110 and the distance for the number of steps (4 or more steps) required for walking analysis. Specifically, the walking path guide 500 is displayed at a position about 6 to 8 meters, preferably about 7 meters from the photographing device 110. Although the walking route guide 500 in FIG. 9(a) is a diagram imitating the shape of a human, the display mode is not limited to this, and for example, footprint marks may be displayed at the optimal position as the walking start position. . The walking path guide 510 in FIG. 9(a) shows the optimal position for the subject's walking direction and walking range. The display position of the walking path guide 510 is set to be perpendicular to the photographing device 110 based on the position (orientation) of the photographing device 110. Further, if the subject meanders in the left-right direction when walking towards the imaging device 110, it is not possible to capture a moving image that allows accurate walking analysis. Therefore, the walking path guide 510 is set in consideration of the direction of the subject and the imaging device 110, the distance between the subject and the imaging device 110, and the walking range (walking width) in the left and right direction. Specifically, at a distance of about 6 to 8 meters, preferably about 7 meters from the imaging device 110, the subject walks vertically toward the imaging device 110, and the walking range is about 2 meters, preferably about 1 meter. The walking path guide 510 is displayed as follows. Although the walking route guide 510 in FIG. 9A is a dotted line, the display mode is not limited to this, and for example, a diagram simulating blocks or a diagram simulating a rope may be displayed.
Similar to the walking path guide, the "horizontal guide" is one of the displays used by the photographer who shoots the walking video to check whether he or she is shooting a moving image that allows accurate walking analysis. The horizontal guide is information regarding the attitude of the imaging device 110. The attitude of the photographing device 110 is a symbol that detects the attitude of the mobile terminal 100 with respect to the direction of gravity using an attitude sensor (for example, a gyro sensor) provided in the mobile terminal 100, and indicates the current photographing direction based on the detection result. A posture display marker “●” is displayed on the horizontal guide 520. FIG. 9 shows a display example of the horizontal guide 520. When the attitude display marker "●" overlaps the center of the cross, the horizontal guide 520 indicates that the attitude of the photographing device 110 is horizontal to the ground plane of the photographer, and the posture display marker "●" overlaps with the center of the cross. If the position is deviated, it indicates that the posture of the photographing device 110 is not horizontal to the ground plane of the photographer. In FIG. 9A, there is no subject in the photographing range, so no subject is displayed on the photographing screen. In FIG. 9B, there is a subject in the photographing range, and the photographing device 110 is moved or the subject is prompted to move so that the display position of the walking path guide 500 overlaps the subject and is displayed on the photographing screen. At this time, if the height of the subject's face, preferably the height of the eyes, and the height of the lens of the photographing device 110 are approximately the same, it is possible to photograph a moving image that allows accurate gait analysis. In FIG. 9(c), the height of the lens of the photographing device 110 and the height of the subject's face are approximately the same, and the posture of the photographing device 110 is horizontal to the ground plane of the photographer, so the horizontal guide 520 is It shows that the attitude of the device 110 is horizontal. In FIG. 9D, the subject is at the display position of the walking path guide 500, but the height of the lens of the photographing device 110 is higher than the subject's face, so the photographing device 110 is positioned at an angle that looks down on the subject. It is provided. Therefore, the horizontal guide 520 in FIG. 9(d) is displayed with the attitude display marker "●" shifted from the center of the cross. By checking the horizontal guide 520, the photographer can understand whether the posture of the photographing device 110 is tilted, and can adjust the photographing device 110 to the correct posture while checking the horizontal guide 520. Note that the horizontal guide 520 is not limited to the display mode shown in FIG. 9, but may be any other display mode such as a numerical value or a grid line displayed on the photographing screen. Note that in FIG. 9, the walking path guide 500, the walking path guide 510, and the horizontal guide 520 are displayed on the photographing screen, but they may be guided by audio, or the display and audio may be used together.

The step (step S103) is a step of determining whether or not the photographing device 110 is photographing. "During shooting" here indicates, for example, that the shooting button is being pressed. If it is determined that shooting is not in progress, this process ends. If it is determined that shooting is in progress, the process moves to step S104.

The process (step S104) is a process of determining whether or not a subject, which is a subject, exists within a predetermined photographing range. As described above, in this embodiment, the subject's skeletal points are extracted and processed using a machine learning model as described above. Therefore, the predetermined photographing range may be determined by using a machine learning model to determine the distance between the subject and the photographing device 110 that is suitable for extracting the subject's face, preferably the eyes (for example, whether it is within 8 meters or not). If the machine learning model is able to extract the subject's face, preferably the eyes, it may be determined that the face is within a predetermined range. If it is determined that it is within the predetermined range, the process moves to step S105, and if it is determined that it is not within the predetermined range, the process moves to step S106.

In the step (step S105), since it is determined in step S104 that the subject is present within the predetermined photographing range, mosaic processing is performed around the subject's eyes so that the subject cannot be identified. For the areas near the eyes to be mosaiced, a machine learning model is used to extract the subject's skeletal points, and the positions of the eyes are estimated and determined from the extracted skeletal points. By applying mosaic processing near the subject's eyes (corresponding to 550 in Figure 10(b)) as shown in Figure 10(b), it becomes difficult to identify individual subjects from the video images used for gait analysis. It becomes possible to perform gait analysis with care (sending moving images for gait analysis). In addition, in Fig. 10(b), an image resembling sunglasses was superimposed near the subject's eyes to make it difficult to identify the individual subject, but other images or so-called mosaic (blurring) may also be added. . In order to make it difficult to identify individual subjects, it is preferable to apply mosaic processing to the entire face, but since the skeletal points of the face are used when performing gait analysis, it is possible to extract the skeletal points necessary for gait analysis. Mosaic processing will be applied to areas that make it difficult to identify individual subjects.

In the step (step S106), since it is determined in step S104 that the subject does not exist within the predetermined photographing range, processing is performed so that the subject cannot be identified by performing mosaic processing on the entire photographing range. FIG. 10(a) shows an example in which mosaic processing (corresponding to 560 in FIG. 10(a)) is applied to the entire imaging range. Step S106 is a process that is transferred when it is determined that the subject does not exist within the predetermined range. Mosaic processing makes it difficult to identify individual subjects. Note that a predetermined image, for example, a raindrop image, may be superimposed over the entire shooting range, or a so-called mosaic (blurring) process may be performed.

The step (step S107) is a step of determining whether or not the photographing by the photographing device 110 has ended. The "end of shooting" is determined, for example, based on whether or not the pressing of the shooting button is released. If it is determined that the shooting has ended, this process ends. If it is not determined that the photographing has ended, the process moves to step S104.

Screen images when the walking route guide process and the subject mask process are performed will be described using FIGS. 9 and 10.
FIG. 9A is an image diagram of the photographing screen when the photographing device 110 is in a photographing preparation state. Walking path guide 500, walking path guide 510, and horizontal guide 520 are displayed regardless of whether a subject is present at the imaging position. If the walking path guide 500, walking path guide 510, and horizontal guide 520 are displayed when the photographing preparation state is in progress, the photographer can easily adjust the position of the subject and the photographing device 110.
FIG. 9B is an image diagram of the photographing screen when the subject is present at the photographing position when the photographing device 110 is in the photographing preparation state. As shown in FIG. 9B, the photographer can adjust the position of the subject and the posture of the imaging device 110 based on the displayed walking path guide 500, walking path guide 510, and horizontal guide 520. .
FIG. 9C is an image of a photographing screen during photographing after adjusting the position of the subject and the posture of the photographing device 110. 530 indicating that the image is being photographed is displayed when the image is being photographed. At this time, the walking path guide 500 may be displayed, but as shown in FIG. 9(c), the walking path guide 500 may not be displayed, but the walking path guide 510 and the horizontal guide 520 may be displayed. . Since the walking path guide 510 and the horizontal guide 520 are displayed even during photographing, the photographer can take photographs while adjusting the walking position of the subject and the posture of the photographing device 110.
9(d) shows that in the state of FIG. 9(b), the horizontal guide 520 indicates that the posture of the imaging device 110 is not horizontal because the imaging device 110 is located at a position higher than the subject's eye level. . Further, an advice may be displayed saying "The camera is tilted." When such a display is displayed, the photographer may adjust the posture of the photographing device 110 so that it is horizontal.
FIG. 9 is a screen image when the walking path guide process is performed. FIG. 10 shows a screen image when a subject masking process is performed in addition to the walking path guiding process.
FIG. 10A is an image diagram of the imaging screen when imaging is in progress and the subject does not exist within the predetermined range. As shown in FIG. 10(a), since the subject does not exist within the predetermined range, mask processing is applied to the entire photographic screen.
FIG. 10(b) is a photographic screen image when photographing is in progress and the subject is within a predetermined range. As shown in FIG. 10(b), since the subject exists within a predetermined range, the subject's eyes are masked.

As shown in FIGS. 9 and 10, in this embodiment, mask processing is performed during the shooting state. This is because it is easier for the photographer to adjust the position of the subject and the posture of the imaging device 110 if mask processing is not performed in the imaging preparation state. Since there may be cases where it is desired to confirm what kind of mask processing will be applied to the photographed image before photographing, mask processing may be performed even in the photographing preparation state. Further, as shown in FIG. 9C, although the walking path guide 500 is not displayed in the photographing state, it may be displayed. The display mode may be appropriately determined so that the photographer can easily adjust the position of the subject and the posture of the imaging device 110.

Although the flowchart in FIG. 8 describes a case in which mask processing is sequentially performed on captured moving images when imaging is performed with the imaging device 110, masking processing may also be performed on existing moving images. In this case, when performing gait analysis using an existing video image, the processes of steps S104 to S107 may be executed and the video image subjected to mask processing may be stored. By doing this, even when gait analysis is performed using data other than video images taken for gait analysis, it is difficult to identify the subject with the data that is memorized and saved, so gait analysis can be performed with confidence. (Sending video images for gait analysis) becomes possible.

In this embodiment, the subject's information is only the captured moving image (only information that can be acquired and estimated from the moving image), but in addition to the captured moving image, subject information, especially items related to walking, is included. It may also be possible to input it. The items related to walking may be determined based on the items to be analyzed using the walking analysis results. For example, when analyzing the walking development level of infants and young children, age in days (age in months, age, or date of birth), gender, human body dimensions such as height and foot size, weight, and the date on which they stood up (in days) , presence/absence and type of shoes, presence/absence and type of diapers, etc. may be input. Furthermore, when analyzing the health level of an elderly person, it is sufficient to input information such as age (date of birth), gender, human body dimensions such as height, weight, medical history of the feet, lifestyle, blood pressure, etc.

The walking video determining step described above may include, as a predetermined photographing condition, a light amount determining step of determining the amount of light at the time of photographing. This is because the amount of light at the time of shooting affects the accuracy of walking analysis of the captured moving image. As for the judgment conditions, if the illuminance is within a predetermined range, one point is added to the total score, and if the illuminance is not within the predetermined range, that fact may be registered as a judgment result, and the process may proceed to the subsequent steps. If the amount of light is too large (too bright) or too low (too dark), it will affect the accuracy of walking analysis, so it is preferable to set an upper limit and a lower limit. Furthermore, since the types of light are different when the shooting location is outside and indoors, a plurality of predetermined ranges may be set, and the predetermined range used for determination may be determined depending on the shooting location. The predetermined range may be set in "lumens", which represents the amount of light, or "lux", which represents the brightness of the illuminated surface. Further, the amount of light may be measured using a light meter or a light meter application provided in the photographing device 110, or may be determined using a value measured using a light meter provided separately from the photographing device 110. .

The walking video determining step described above may include, as a predetermined photographing condition, a photographing device shake determining step of determining shaking of the photographer and camera shake during photographing. This is because if the photographer uses the camera 110 to take a picture while shaking, the captured video image will be blurred, which will affect the accuracy of walking analysis of the blurred video image. be. The judgment conditions are that if the value measured by the gyro sensor (angular velocity sensor) provided in the photographing device 110 is within a predetermined range (below a predetermined value), one point is added to the overall score, and it must be within the predetermined range. It is only necessary to register that fact as a determination result and proceed to the subsequent steps.

The above embodiment includes the following technical ideas.
<1>
Determining whether a video image of a walking subject satisfies a gait analysis condition for determining that gait analysis is possible, which is comprised of at least one of a predetermined walking condition or a predetermined imaging condition. A moving image determination method, comprising:
a video acquisition step of acquiring the video image;
A moving image determining method, comprising: a walking moving image determining step of determining whether the acquired moving image satisfies the walking analysis conditions.
Furthermore, the moving image is composed of a plurality of frames, and the walking video determining step is characterized in that the walking video determining step performs the determination after all frames are acquired in the video acquiring step or for each acquired frame. Judgment method.
<2>
The walking video determination step includes a skeletal point extraction step and a skeletal point determination step executed after the skeletal point extraction step,
The skeletal point extraction step extracts the skeletal points of the subject from the video image using a skeletal point learning model obtained by machine learning,
The skeleton point determination step includes:
The moving image determination method according to <1>, wherein the skeletal points extracted in the skeletal point extraction step are used to determine whether the gait analysis conditions are satisfied.
<3>
further comprising a distance estimating step of estimating a distance between the shooting position of the video image and the position of the subject,
The moving image determination method according to <1> or <2>, wherein the predetermined shooting conditions include that there is a frame in which the estimated distance is within a predetermined range.
<4>
further comprising a step count determination step of determining the number of steps taken by the subject;
The video image determination method according to any one of <1> to <3>, wherein the predetermined walking condition includes that the determined number of steps is greater than or equal to a predetermined number of steps.
<5>
further comprising a center of gravity determination step of determining the center of gravity of the subject;
The moving image determination method according to any one of <1> to <4>, wherein the predetermined walking condition includes that the determined center of gravity is within a predetermined range in the left-right direction.
<6>
further comprising a gait determination step of determining a gait index suggesting that the subject is running,
The video image determination method according to any one of <1> to <5>, wherein the predetermined walking condition includes the presence of a frame in which it is determined that the person is not running.
<7>
further comprising a face determination step of determining the subject's face,
The moving image determination method according to any one of <1> to <6>, wherein the predetermined walking condition includes that the determined direction of the face is within a predetermined range in the left-right direction.
<8>
further comprising a two-handed determination step of determining both hands of the subject,
The video image determination method according to any one of <1> to <7>, wherein the predetermined walking condition includes that the determined both hands are at a height equal to or lower than a predetermined position.
<9>
The moving image determination method according to any one of <1> to <8>, wherein when the moving image includes a plurality of subjects, the skeleton point extraction is performed for each subject.
<10>
The moving image according to any one of <2> to <9>, wherein the skeletal point extraction step extracts 16 or more skeletal points of the subject from the moving image using the skeletal point learning model. Judgment method.
<11>
The method for determining a moving image according to any one of <2> to <10>, wherein the step of determining a walking video is terminated for a frame in which a skeletal point of the subject cannot be extracted.
<12>
The walking video determining step is performed when a predetermined number of skeletal points cannot be extracted for one subject in the skeletal point extraction step and/or when the center of gravity of the subject cannot be calculated based on the extracted skeletal points. The moving image determination method according to any one of <2> to <11>, characterized in that:
<13>
The video according to any one of <2> to <12>, wherein the predetermined walking condition includes extracting a predetermined number of skeletal points for one subject and calculating the center of gravity of the subject in the skeletal point extraction step. Image judgment method.
<14>
further comprising a whole body determination step of determining whether the whole body of the subject is photographed;
The whole body determination step determines whether the skeletal points of the subject extracted in the skeletal point extraction step are in a predetermined positional relationship;
The moving image determination method according to any one of <2> to <13>, wherein the predetermined imaging conditions include that skeletal points of the subject are in a predetermined positional relationship.
<15>
The predetermined imaging conditions include that the skeletal points of the subject extracted in the skeletal point extraction step are in a predetermined positional relationship;
The video determination according to <14>, wherein the walking video determination step includes terminating the step if the skeletal points of the subject extracted in the skeletal point extraction step do not have a predetermined positional relationship. Method.
<16>
<3> The predetermined photographing conditions include the presence of a moving image in which the distance between the photographing position of the moving image and the position of the subject is estimated to be 1 to 7 m in the distance estimation step. The moving image determination method according to any one of <15>.
<17>
The walking video determining step stores the estimation result in association with a video in which it is estimated in the distance estimation step that the distance between the shooting position of the video and the position of the subject is not within a predetermined range. The moving image determination method according to any one of <3> to <16>, characterized in that:
<18>
In the step count determining step, if there is a moving image in which the left and right heels estimated from the skeletal points of the left and right ankles extracted in the skeletal point extraction step and the center of gravity alternately touch the ground twice, it is determined that the number of steps has exceeded the predetermined number of steps. The moving image determination method according to any one of <4> to <17>.
<19>
The walking video determining step is characterized by storing the determination result in association with a video in which the determined number of steps is not equal to or greater than the predetermined number of steps in the step count determining step. >The moving image determination method according to any one of the above.
<20>
In the walking video determination step, the distance estimation step estimates that the distance between the shooting position of the video image and the subject's position is within a predetermined range, and the step count determination step determines that the determined number of steps is within the predetermined range. The moving image determination method according to any one of <4> to <19>, characterized in that when it is determined that the number of steps is greater than or equal to the number of steps, a point is added to a score stored in association with the estimated and determined moving image.
<21>
The predetermined walking condition includes determining in the center of gravity determination step that the center of gravity of the subject extracted in the skeletal point extraction step has a horizontal movement range of 1 meter or less with respect to the imaging device. The moving image determination method according to any one of > to <20>.
<22>
In the walking video determining step, if the center of gravity determining step does not determine that the subject's center of gravity has a horizontal movement range of 1 meter or less with respect to the imaging device, the determination result is stored in association with the video image. The moving image determination method according to any one of <5> to <21>, characterized in that:
<23>
The walking video determining step adds points to a score stored in association with a moving image in which it has been determined in the center of gravity determining step that the subject's center of gravity has a movement range of 1 meter or less in the left and right direction with respect to the imaging device. The moving image determination method according to any one of <5> to <22>.
<24>
The gait index determined in the gait determination step is at least one of gait speed, ground contact time of the right foot and left foot, or change in foot position,
In the case of the walking speed, the time required to move a predetermined distance is measured based on the center of gravity of the subject calculated using the skeletal points extracted in the skeletal point extraction step, and if the calculated speed is above a predetermined range, , determines that it is running,
In the case of the ground contact time of the right foot and the left foot, if there is a time when neither the right heel nor the left heel is in contact with the ground based on the positions where the left and right heels touch the ground extracted in the step count determination step, it is determined that the running is running,
In the case of a change in the foot position, if the vertical movement range of the left and right ankle skeletal points extracted in the skeletal point extraction step exceeds a predetermined range, it is determined that the foot is running. 23>The moving image determination method according to any one of the above.
<25>
Any one of <6> to <24>, wherein the walking video determining step stores the determination result in association with the video data for which running has been determined in the walking determining step. The video image determination method described in .
<26>
Any one of <6> to <25>, wherein the walking video determining step adds points to the scores stored in association with the moving image data for which running was not determined in the walking determining step. The moving image determination method according to (1).
<27>
The predetermined walking condition includes any one of <7> to <26> including that the face determination step determines that the vertical and horizontal movement range of the face is within ±30 degrees. The video image determination method described in .
<28>
The walking video determining step determines, through the face determining step, the time or number of frames in which the movement range of the face in at least one of the vertical or horizontal directions exceeds ±30 degrees, and the total time or number of frames walking. The moving image according to <27>, wherein it is determined that the predetermined walking condition is not satisfied when the threshold ratio is equal to or higher than the threshold ratio, and the threshold ratio is 10% or more and 30% or less. Judgment method.
<29>
The walking video determining step stores the determination result in association with a video in which the vertical and horizontal movement range of the face orientation is not determined to be within ±30 degrees in the face determining step. The moving image determination method according to any one of <7> to <28>, characterized in that:
<30>
The walking video determining step adds points to the scores stored in association with the video images in which the vertical and horizontal movement range of the face direction is determined to be within ±30 degrees in the face determining step. The moving image determination method according to any one of <7> to <29>, characterized in that:
<31>
The predetermined walking condition is determined by comparing the positions of the skeletal points of both wrists and the skeletal points of both elbows of the subject extracted in the skeletal point extraction step in the two-hand determination step, and determining whether the positions of both hands of the subject are The moving image determining method according to any one of <8> to <30>, including determining that the moving image is lower than the elbow position.
<32>
The walking video determining step is characterized in that the determination result is stored in association with a video image in which the positions of both hands of the subject were not determined to be lower than the positions of both elbows in the both hands determining step. The moving image determination method according to any one of > to <31>.
<33>
The walking video determining step is characterized in that points are added to a score stored in association with a video image in which the positions of both hands of the subject are determined to be lower than the positions of both elbows in the both hands determining step. The moving image determination method according to any one of > to <32>.
<34>
further comprising a light amount determining step of determining the amount of light at the time of photographing,
The moving image determination method according to any one of <1> to <33>, wherein the predetermined photographing conditions include that the amount of light is within a predetermined range.
<35>
According to <34>, the walking video determining step stores the determination result in association with a video image in which the light amount is not determined to be within the predetermined range in the light amount determining step. Video image judgment method.
<36>
<34> or <35> The walking video determining step adds points to a score stored in association with a moving image in which the light amount is determined to be within the predetermined range in the light amount determining step. ＞The moving image determination method described in ＞.
<37>
further comprising a photographing device shake determination step of determining the amount of light at the time of photographing;
The moving image determination method according to any one of <1> to <36>, wherein the predetermined photographing condition includes that shaking of the photographing device is within a predetermined range.
<38>
The walking video determining step is characterized in that the determination result is stored in association with a video in which the shaking of the photographing device was not determined to be within the predetermined range in the photographing device shake determining step. 37>.
<39>
The walking video determining step is characterized in that points are added to a score stored in association with a moving image in which the shaking of the photographing device is determined to be within the predetermined range in the photographing device shake determining step. 37> or the moving image determination method described in <38>.
<40>
In the walking video determining step, the video is determined based on the points added based on the determination results of the center of gravity determining step, the walking determining step, the face determining step, the both hands determining step, the light amount determining step, and the photographing device sway determining step. <23>, <26>, <30>, characterized by evaluating whether the image is a moving image suitable for gait analysis, or evaluating the accuracy of the gait analysis result using the moving image; The moving image determination method according to any one of <33>, <36>, and <39>.
<41>
a human body dimension acquisition step of acquiring dimension data regarding a part or all of the human body of the subject;
Estimating the distance between the shooting position of the moving image and the position of the subject based on the length of the part and the depth distance to the subject in the moving image of the subject, and the human body size data. The moving image determination method according to any one of <3> to <40>, further comprising a distance estimation step.
<42>
Determining whether a video image of a walking subject satisfies a gait analysis condition for determining that gait analysis is possible, which is comprised of at least one of a predetermined walking condition or a predetermined imaging condition. A moving image determination program,
a video acquisition process for acquiring the video image;
gait video determination processing for determining whether the acquired video image satisfies the gait analysis condition;
The moving image is composed of multiple frames,
A video image determination program characterized in that the walking video determination process performs determination after all frames are acquired by the video acquisition process or for each acquired frame.
<43>
Determining whether a video image of a walking subject satisfies a gait analysis condition for determining that gait analysis is possible, which is comprised of at least one of a predetermined walking condition or a predetermined imaging condition. A moving image determination device,
a video acquisition means for acquiring the video image;
gait video determining means for determining whether the acquired video image satisfies the gait analysis condition;
The moving image is composed of multiple frames,
The moving image determining device is characterized in that the walking moving image determining means performs the determination after all frames are acquired by the moving image acquiring means or for each acquired frame.
<44>
The moving image determination device is a mobile terminal, further comprising a photographing means, a posture sensor, and a display means,
The walking video determining means determines whether the posture of the photographing means at the time of photographing is within ±25 degrees in the left-right and front-back directions based on the detection result of the posture sensor, and the determination result is within ±25 degrees in the left-right and front-back directions. If it is within the range, the display means is controlled to display information indicating that the display is normal, or information accompanying the determination result is not displayed on the display means; The moving image determining device according to <43>, wherein if the error is not within ±25 degrees, the display means displays information to that effect.
<45>
The moving image determination device further includes a display means,
When the walking video determining means makes a determination for each frame acquired by the video acquiring means, the walking video determining means controls to display advice regarding photography to the photographer on the display means in accordance with each determination result <43> Or the moving image determination device according to <44>.
<46>
The moving image determining device according to any one of <43> to <45>, wherein the moving image determining device stores the moving image determining program described in <42>.
<47>
The video according to any one of <1> to <41>, further comprising a walking path guiding step of guiding a photographer about the range in which the subject should walk and the posture of a photographing device for photographing the moving image. Image judgment method.
<48>
The moving image determination method according to any one of <1> to <41> and <47>, further comprising a subject masking step of performing mask processing on the moving image so that the subject cannot be identified.

100 Information processing terminal (mobile terminal)
110 Photographing device 120 Judgment unit 130 Storage unit 140 Posture sensor 500 Walking path guide 510 Walking path guide 520 Horizontal guide 530 Display during shooting 550 Mosaic aspect near eyes 560 Overall mosaic aspect

Claims (11)

Whether a video image of a walking subject satisfies the gait analysis conditions for determining that the video image is a video image that can be analyzed for gait analysis, which is composed of at least one of a predetermined walking condition or a predetermined shooting condition. A video image determination method for determining,
a video acquisition step of acquiring the video image;
a walking video determination step of determining whether the acquired video image satisfies the walking analysis conditions;
a distance estimation step of estimating the distance between the shooting position of the video image and the position of the subject;
a step count determination step of determining the number of steps taken by the subject ;
The moving image is composed of multiple frames,
The walking video determining step performs a determination after all frames are acquired in the video acquiring step or for each acquired frame,
The predetermined shooting condition includes that there is a frame in which the estimated distance is within a predetermined range,
A moving image determining method , wherein the predetermined walking condition includes that the determined number of steps is greater than or equal to a predetermined number of steps . Whether a video image of a walking subject satisfies the gait analysis conditions for determining that the video image is a video image that can be analyzed for gait analysis, which is composed of at least one of a predetermined walking condition or a predetermined shooting condition. A video image determination method for determining,
a video acquisition step of acquiring the video image;
a walking video determination step of determining whether the acquired video image satisfies the walking analysis conditions;
a distance estimation step of estimating the distance between the shooting position of the video image and the position of the subject;
a center of gravity determination step of determining the center of gravity of the subject ;
The moving image is composed of multiple frames,
The walking video determining step performs a determination after all frames are acquired in the video acquiring step or for each acquired frame,
The predetermined shooting condition includes that there is a frame in which the estimated distance is within a predetermined range,
The moving image determination method is characterized in that the predetermined walking condition includes that the determined center of gravity is within a predetermined range in the left-right direction . Whether a video image of a walking subject satisfies the gait analysis conditions for determining that the video image is a video image that can be analyzed for gait analysis, which is composed of at least one of a predetermined walking condition or a predetermined shooting condition. A video image determination method for determining,
a video acquisition step of acquiring the video image;
a walking video determination step of determining whether the acquired video image satisfies the walking analysis conditions;
a distance estimation step of estimating the distance between the shooting position of the video image and the position of the subject;
a gait determination step of determining a gait index suggesting that the subject is running ;
The moving image is composed of multiple frames,
The walking video determining step performs a determination after all frames are acquired in the video acquiring step or for each acquired frame,
The predetermined shooting condition includes that there is a frame in which the estimated distance is within a predetermined range,
A moving image determining method , wherein the predetermined walking condition includes that there is a frame in which it is determined that the person is not running . Whether a video image of a walking subject satisfies the gait analysis conditions for determining that the video image is a video image that can be analyzed for gait analysis, which is composed of at least one of a predetermined walking condition or a predetermined shooting condition. A video image determination method for determining,
a video acquisition step of acquiring the video image;
a walking video determination step of determining whether the acquired video image satisfies the walking analysis conditions;
a distance estimation step of estimating the distance between the shooting position of the video image and the position of the subject;
a face determination step of determining the face of the subject ;
The moving image is composed of multiple frames,
The walking video determining step performs a determination after all frames are acquired in the video acquiring step or for each acquired frame,
The predetermined shooting condition includes that there is a frame in which the estimated distance is within a predetermined range,
The moving image determination method is characterized in that the predetermined walking condition includes that the determined direction of the face is within a predetermined range in the left-right direction . Whether a video image of a walking subject satisfies the gait analysis conditions for determining that the video image is a video image that can be analyzed for gait analysis, which is composed of at least one of a predetermined walking condition or a predetermined shooting condition. A video image determination method for determining,
a video acquisition step of acquiring the video image;
a walking video determination step of determining whether the acquired video image satisfies the walking analysis conditions;
a distance estimation step of estimating the distance between the shooting position of the video image and the position of the subject;
a two-handed determination step of determining both hands of the subject ;
The moving image is composed of multiple frames,
The walking video determining step performs a determination after all frames are acquired in the video acquiring step or for each acquired frame,
The predetermined shooting condition includes that there is a frame in which the estimated distance is within a predetermined range,
The moving image determination method is characterized in that the predetermined walking condition includes that the determined both hands are at a height equal to or lower than a predetermined position . The walking video determination step includes:
storing a score based on the determination result in association with the moving image;
For frames that are determined not to satisfy the gait analysis conditions, either end the gait video determination process or do not add a score based on the determination result to the score for the frame. The moving image determination method according to any one of claims 1 to 5 . The walking video determination step includes a skeletal point extraction step and a skeletal point determination step executed after the skeletal point extraction step,
The skeletal point extraction step extracts the skeletal points of the subject from the video image using a skeletal point learning model obtained by machine learning,
The skeleton point determination step includes:
Determining whether the gait analysis condition is satisfied using the skeletal points extracted by the skeletal point extraction step,
The walking video determination step includes:
6. The moving image determining method according to claim 1, wherein if it is determined in the skeleton point determining step that the walking analysis condition is not satisfied, the walking video determining step is terminated. The walking video determination step includes a skeletal point extraction step, a skeletal point determination step executed after the skeletal point extraction step, and a whole body determination step,
The skeletal point extraction step extracts the skeletal points of the subject from the video image using a skeletal point learning model obtained by machine learning,
The whole body determination step determines whether the skeletal points of the subject extracted in the skeletal point extraction step are in a predetermined positional relationship,
The moving image determining method according to any one of claims 1 to 5, wherein the walking video determining step determines whether the walking analysis condition is satisfied based on the determination result of the whole body determining step. Whether a video image of a walking subject satisfies the gait analysis conditions for determining that the video image is a video image that can be analyzed for gait analysis, which is composed of at least one of a predetermined walking condition or a predetermined shooting condition. A moving image determination program that determines
a video acquisition process for acquiring the video image;
walking video determination processing for determining whether the acquired video image satisfies the walking analysis conditions;
distance estimation processing for estimating the distance between the shooting position of the video image and the position of the subject;
Step count determination processing for determining the number of steps of the subject ,
The moving image is composed of multiple frames,
The walking video determination process performs a determination after all frames are acquired by the video acquisition process or for each acquired frame,
The predetermined shooting condition includes that there is a frame in which the estimated distance is within a predetermined range,
The moving image determination program is characterized in that the predetermined walking condition includes that the determined number of steps is greater than or equal to a predetermined number of steps . Whether a video image of a walking subject satisfies the gait analysis conditions for determining that the video image is a video image that can be analyzed for gait analysis, which is composed of at least one of a predetermined walking condition or a predetermined shooting condition. A moving image determination device that determines
a video acquisition means for acquiring the video image;
walking video determining means for determining whether the acquired video image satisfies the walking analysis conditions;
distance estimating means for estimating the distance between the shooting position of the moving image and the position of the subject;
Step count determining means for determining the number of steps of the subject ,
The moving image is composed of multiple frames,
The walking video determining means performs the determination after all frames have been acquired by the video acquiring means or for each acquired frame,
The predetermined shooting condition includes that there is a frame in which the estimated distance is within a predetermined range,
The moving image determination device is characterized in that the predetermined walking condition includes that the determined number of steps is equal to or greater than a predetermined number of steps . The moving image determination device further includes a display means,
10. The walking video determining means, when performing the determination for each frame acquired by the video acquiring means, controls the display means to display advice regarding photography to the photographer in accordance with each determination result. The moving image determination device described in .

Images