JP2022115705A - Motion analysis data acquisition method and motion analysis data acquisition system - Google Patents

Abstract

To provide a motion analysis data acquisition method and a motion analysis data acquisition system capable of acquiring data that appropriately reflects the motion of a subject so that direct and detailed analysis of the motion of the subject can be performed for evaluation of human body functions, etc.SOLUTION: A motion analysis data acquisition method comprises imaging a subject during the following test with a camera of a mobile terminal, and generating data indicating the coordinates of a plurality of feature points on the subject's body on the basis of the imaging result. The subject is made to stand up from sitting on a chair 41 at a position P1, and is moved along markers 42 to 44 to a position P2 separated to the right side from the position P1. Consequently, for the imaged right half of the subject, data indicating the coordinates of the feature points and how the feature points are connected is acquired during the motion of standing up from the chair at the position P1 and during the motion from the position P1 to the position P2.SELECTED DRAWING: Figure 5

Description

The present invention relates to a motion analysis data acquisition method and a motion analysis data acquisition system for acquiring data for analyzing motions of people when they move.

Conventionally, for the purpose of medical evaluation of a person's bodily functions, there is a case where a motion such as walking is performed and verified. The “Timed Up and Go (TUG) Test” in Non-Patent Document 1 is one example. The subject was asked to stand up from a chair, walk 3m, then turn around, walk back to the original chair, and sit on the chair again. Non-Patent Document 1 discusses the correlation between the time taken for this series of movements and the subject's balance, walking speed, and functional ability.

Podsiadlo D and Richardson S. The Timed "Up & Go": A Basic Functional Mobility Test for Frail Older Adults. Go": A Test of Basic Functional Mobility for Frail Elderly Persons", J Am Geriatr Soc. 1991;39:142-148.

In the TUG test of Non-Patent Document 1, only the time required for a series of actions performed by the subject is used as an evaluation index. On the other hand, the inventors thought that if a series of motions of a subject could be analyzed more directly and in detail, important information could be obtained for medical evaluation of a person's bodily functions. Therefore, in order to enable direct and detailed analysis of the subject's motion, we aimed to acquire data for motion analysis that appropriately reflected the series of motions of the subject.

An object of the present invention is to acquire motion analysis data that can acquire data that appropriately reflects the motion of a subject so that direct and detailed analysis of the motion of the subject can be performed for evaluation of human body functions. The object is to provide a method and a motion analysis data acquisition system.

A motion analysis data acquisition method of the present invention comprises imaging means for converting an image of a subject into data and outputting the data; and a feature point deriving means for generating data indicating the coordinates of the feature points of the subject, and a method of obtaining data for analyzing a motion of a person during movement, comprising: a subject motion step of standing up from a sitting state and walking or running to a second position separated from the first position in a direction parallel to the horizontal direction and intersecting the optical axis of the imaging means; an imaging step of causing the imaging means to continuously image at least one of the left half of the body and the right half of the subject during movement in the subject movement step; first analysis data representing the coordinates of the plurality of feature points at each timing of the subject during the motion of standing up from the object and during the motion of moving from the first position to the second position, based on the data; and a feature point generating step of causing the feature point deriving means to generate

According to the motion analysis data acquisition method of the present invention, the subject is moved from the first position to the second position along the direction intersecting the optical axis of the imaging means. Then, at least one of the right half of the body and the left half of the subject's body is continuously imaged by the imaging means. For this reason, subject images can be obtained in which the subject's walking or running appearance is continuously captured from the side of the direction of travel. Then, based on the image of the subject, data indicating the coordinates of a plurality of feature points on the human body are obtained.
For this reason, such data should appropriately reflect the status of the subject's movements as seen from the side, such as the degree of the subject's forward leaning posture, the speed at which the leg is swung, the speed at which the hand is waved, and the degree to which the knee is raised. It is possible to The strength and weakness of a person's bodily functions often appear in these elements. For example, subjects in a frail physical condition tend to lean forward and have weak hand movements. In this way, the state of motion seen from the side of the subject can be an important factor in determining the person's physical function. In the present invention, information indicating such important determination factors can be numerically acquired as the coordinates of feature points on the body. Therefore, by analyzing numerical values, it is possible to directly and in detail evaluate physical functions. As described above, according to the present invention, it is possible to acquire data that enable direct and detailed analysis of the motion of a subject for the purpose of evaluating a person's bodily functions.

Further, in the present invention, in the subject operation step, the direction is changed at the second position toward the imaging means, and the subject moves toward the imaging means from the second position to a third position closer to the imaging means. in the imaging step, causing the imaging means to continuously image the front of the subject during the motion in the subject motion step; and in the feature point generating step, executed in the imaging step. coordinates of the plurality of feature points at each timing of the subject during movement from the second position to the third position, based on the data output from the imaging means regarding the imaging to be performed; It is preferable to cause the feature point deriving means to generate the second analysis data. According to this, it is possible to obtain data for evaluating the physical function based on the movement of the subject as seen from the front, such as the lateral movement of the trunk due to walking.

Further, in the present invention, in the subject movement step, visible markers are installed for the subject to recognize the moving route, and the markers indicate the second position and the third position, respectively. and/or a path marker extending along a line segment indicating an assumed straight path through which the subject passes. According to this, the subject can recognize the moving route accurately by visually recognizing the marker. Therefore, it is possible to obtain data for accurately evaluating the physical function.

Further, in the present invention, it is preferable that the route marker extends along two line segments that extend parallel to the assumed straight route so as to sandwich the assumed straight route. For example, if an assumed straight path is indicated by only one line segment, the subject may move with both feet aligned on that one line segment, which may make it difficult for the subject to move. On the other hand, by forming two line segments sandwiching the assumed straight path, it is possible to secure a width for the subject to put their feet on, so that the subject can feel that it is easy to move.

Also, in the present invention, it is preferable to include a deriving step of deriving the coordinates of the marker based on the data output from the imaging means regarding the imaging performed in the imaging step. According to this, the subject's position can be grasped based on the derived coordinates of the marker.

Further, in the present invention, it is preferable that the deriving step uses a neural network that has been trained to detect an object with respect to the marker. According to this method, since a neural network is used, the accuracy of detecting the coordinates of the marker is increased.

Further, in the present invention, the feature point derivation means may, for each human body in the subject,
data indicating the coordinates of the plurality of feature points are individually generated, and the coordinates of the marker derived in the derivation step and the first analysis data and the second analysis data generated in the feature point generation step an evaluation step of evaluating the positional relationship between the plurality of feature points and the assumed straight path for each of the person's body, based on the results of the evaluation; and a determination step of determining which one of them is the subject. According to this, it is possible to determine which person is the subject even when there are a plurality of persons as subjects. Therefore,
Data on a subject can be appropriately acquired even if a plurality of persons are captured during imaging.

A motion analysis data acquisition system according to another aspect of the present invention includes imaging means for converting an image of a subject into data and outputting the data; a feature point deriving means for generating data indicating the coordinates of a plurality of feature points on a person's body included in a subject; an object placed at a first position on which a subject sits; a visible marker for allowing the subject to recognize a path to travel from the first position to a second position spaced apart in a direction intersecting the causing the imaging means to image at least one of the left half of the body and the right half of the subject when the subject moves from the first position to the second position, and based on the data output from the imaging means at this time; and the feature point deriving means generates data for analysis indicating the coordinates of the plurality of feature points at each timing of the subject during the motion of standing up from the object and during the motion of moving from the first position to the second position. to generate

According to the motion analysis data acquisition system of the present invention, the subject recognizes the movement path by the marker, and moves from the first position to the second position along the direction intersecting the optical axis of the imaging means. Then, at least one of the right half of the body and the left half of the subject's body is continuously imaged by the imaging means. For this reason, subject images can be obtained in which the subject's walking or running figure is continuously captured from the side with respect to the direction of travel. Then, based on the image of the subject, data indicating the coordinates of a plurality of feature points on the human body is obtained. For this reason, such data should appropriately reflect the state of the subject's movements as seen from the side, such as the degree of the subject's forward leaning posture, the speed at which the leg is swung, the speed at which the hand is waved, and the degree to which the knee is raised. It is possible to The strength and weakness of a person's bodily functions often appear in these elements. For example, subjects in a frail physical condition tend to lean forward and have weak hand movements. in this way,
The state of movement seen from the side of the subject can be an important decision factor for the evaluation of a person's physical function. In the present invention, information indicating such important determination factors can be numerically obtained as coordinates of feature points on the body. Therefore, by analyzing numerical values, it is possible to directly and in detail evaluate physical function. As described above, according to the present invention, it is possible to acquire data that enable direct and detailed analysis of the motion of a subject for the purpose of evaluating a person's bodily functions.

Further, in the present invention, a mobile terminal and first and second computers are provided, and data converted from a subject image via a wired connection from the mobile terminal functioning as the imaging means is transferred to the first computer. It is preferable that data indicating the coordinates of the plurality of feature points is output to the computer and output from the first computer functioning as the feature point deriving means to the second computer via a wired connection. . Since the output data reflects the movement of the subject, it contains information that is personal to the subject. On the other hand, data output between the mobile terminal and the first and second computers is performed via a wired connection. Therefore, it is possible to reduce the risk of data related to personal information being leaked to the outside.

Further, in the present invention, the first computer includes a data recording unit for recording data from the portable terminal, and after generating data indicating the coordinates of the plurality of characteristic points, It is preferable to delete the recorded data from the portable terminal. According to this, used data is deleted. Therefore, it is possible to further reduce the risk of data related to personal information leaking to the outside.

1 is a block diagram showing part of the configuration of an analysis data acquisition system according to one embodiment of the present invention; FIG. FIG. 2 is a diagram showing the content of data acquired by the data acquisition device of FIG. 1, showing a plurality of feature points representing the general shape of a subject's body and line segments representing how these feature points are connected. FIG. 2 is a diagram showing the contents of data acquired by the data acquisition device of FIG. 1, showing the feature points of FIG. It is a figure which shows a line segment. FIG. 2 is a plan view of a course used in a test conducted by a subject in this embodiment; 5 is a diagram showing a situation in which a subject travels the course of FIG. 4; FIG. FIG. 5 is a flow diagram of an analysis data acquisition method according to an embodiment of the present invention implemented using the system of FIG. 1 and the course of FIG. 4; 2 is a diagram showing a screen displayed on the display of the mobile terminal in FIG. 1; FIG. 2 is a diagram showing a screen displayed on the display of the PC in FIG. 1; FIG. FIG. 7 is a flowchart of motion analysis data acquisition processing executed by the data acquisition device of FIG. 1 in step S7 of FIG. 6; FIG. 10 is a diagram for explaining the processing executed by the data acquisition device of FIG. 1 in the position evaluation step and subject determination step of FIG. 9;

An analysis data acquisition system 1 according to one embodiment of the present invention will be described with reference to FIGS. 1 to 10. FIG. The analysis data acquisition system 1 acquires analysis data used for analyzing the motion of the subject based on the results of continuously imaging the subject performing a predetermined motion. The details are described below.

As shown in FIG. 1, the analysis data acquisition system 1 includes a mobile terminal 10 (imaging means in the present invention), a data acquisition device 20 (first computer and feature point derivation means in the present invention), and a PC 30 (second computer in the present invention). computers). Between the mobile terminal 10 and the data acquisition device 20, and between the data acquisition device 20 and the PC 30,
It is connected by a wired connection such as a USB (Universal Serial Bus) method. Thereby, data communication is performed between the mobile terminal 10 and the data acquisition device 20 and between the data acquisition device 20 and the PC 30 . mobile terminal 10,
Both the data acquisition device 20 and the PC 30 are constructed by hardware such as a computer and software such as program data stored in a memory device (hereinafter referred to as memory). These software can be recorded on various recording media and distributed. The computer has a CPU (Central Processing Uni
t), ROM (Read-Only Memory) and RAM (Random Ac
cess Memory), a hard disk drive, and hardware such as various interfaces such as an input/output interface. In each of the portable terminal 10, the data acquisition device 20 and the PC 30, hardware executes various types of information processing such as arithmetic processing and input/output processing according to software. As a result, the following functions are realized in these devices.

The mobile terminal 10 is provided with a touch panel display. A photographic image, a control image for user input, and the like are displayed on the screen of the touch panel display. Various user input processes are executed when the user touches the screen of the touch panel display with a finger or the like.

The mobile terminal 10 is equipped with a camera. A camera has an optical system, a photoelectric conversion element, and various electronic circuits, and converts a subject image into data as described below and outputs the data. The optical system
It includes lenses, slits, etc., and an object image is formed on the photoelectric conversion element through these.
A photoelectric conversion element converts a subject image into an analog electrical signal. The electronic circuit converts the analog electrical signal generated by the photoelectric conversion element into a digital electrical signal and outputs the digital electrical signal. The mobile terminal 10 generates a moving image data file representing a moving image of a subject based on the digital electrical signal output from the camera. A moving image in the present embodiment refers to an image that represents the movement of a subject by including a group of images in which a plurality of images showing subject images at respective points in time are consecutively arranged in time. The camera of the mobile terminal 10 is used to capture the motion of the subject, as will be described later. When imaging the motion of the subject, the portable terminal 10 is used without being connected to the data acquisition device 20 . After imaging the motion of the subject, the mobile terminal 10 and the data acquisition device 20 are connected, and the moving image data file is transmitted from the mobile terminal 10 to the data acquisition device 20 .

A web browser application (hereinafter referred to as “browser”) is installed in the mobile terminal 10 . When the browser is activated on the mobile terminal 10,
Specified IP (Internet Protocol) address or URL (Uni
Data communication processing is executed according to the form Resource Locator. For example, the image displayed on the touch panel display by the function of the browser (Fig. 7
3), when the IP address corresponding to the data acquisition device 20 is input, data communication is started between the mobile terminal 10 and the data acquisition device 20 . TCP/IP (Transmission Control Protocol) is used for this data communication.
ol/Internet Protocol) is used. Data including moving image data files are transmitted and received between the mobile terminal 10 and the data acquisition device 20 according to the function of the browser.

Based on the video data file transmitted from the mobile terminal 10, the data acquisition device 20
Generate motion analysis data for subjects. A detailed method of data generation will be described later.
The motion analysis data includes data indicating the subject's body at each time point. This data, as shown in FIG. 2, represents the coordinates of a plurality of feature points indicating the position of each part included in the subject's body and how these feature points are connected. For example, feature point F1 is a point corresponding to the subject's nose. A feature point F2 is a point corresponding to the subject's right shoulder. A feature point F3 is a point corresponding to the subject's left knee. A feature point F4 is a point corresponding to the subject's right heel. A feature point F5 is a point corresponding to the subject's neck. Also, in FIG. 2, these feature points are connected to each other by line segments. For example, line segment L1 is a line segment connecting feature point F1 and feature point F5.
A line segment L2 is a line segment connecting the feature points F2 and F5. These line segments show how the feature points are connected to each other, and represent the general shape of the subject's body including the head, neck, torso, arms, and legs as a whole. The motion analysis data includes feature points as shown in FIG. 2 and data indicating how the feature points are connected in a temporally continuous manner. As a result, the position of each part of the body at each point in the subject's movement is indicated by the motion analysis data, and how the position of each part of the body changes over time is also displayed as the motion analysis data. indicated by . For example, FIG. 3 shows coordinates of feature points represented by motion analysis data generated based on a moving image data file of a walking subject and how the feature points are connected. As shown in FIG. 3, temporally consecutive times T1, T2, T3, T4, T5,
The outline of the body of the walking subject at each of T6 and T7 is represented by characteristic points and line segments connecting them. Motion analysis data may be, for example, data representing coordinate values of feature points and vector values indicating how to connect them, or data representing images of circles corresponding to feature points and line segments connecting them. may be Also, both of these data may be included in the motion analysis data.

The PC 30 is a personal computer that receives motion analysis data from the data acquisition device 20 . As the PC 30, a desktop personal computer, a notebook personal computer, a tablet PC, or the like is used. The PC 30 is provided with a display and connected or mounted with input devices for user input such as a keyboard and a mouse. The display screen displays photographic images, control images for user input, and the like. A browser is installed in the PC 30 . When the browser is activated on the PC 30, data communication processing is executed according to the designated IP address or URL. For example, in the image displayed on the display by the function of the browser, the I corresponding to the data acquisition device 20
When the P address is input, data communication is started between the PC 30 and the data acquisition device 20 . This data communication uses TCP/IP (Transmission Control
ol Protocol/Internet Protocol) is used. Data including motion analysis data is transmitted and received between the PC 30 and the data acquisition device 20 according to the function of the browser.

Next, the motion of the subject from whom analysis data is acquired will be described. The movement of the subject is carried out by a test using an L-shaped course 40 (moving route in the present invention) shown in FIG. The course 40 is set on an indoor floor, and is the first course along the X direction in FIG.
It consists of a straight path 40a (assumed straight path in the present invention) and a second straight path 40b along the Y direction in FIG. 4 (assumed straight path in the present invention). The X and Y directions are orthogonal directions along the floor. The first linear paths 40a are located at positions P1 separated from each other in the X direction.
(corresponding to the first position in the present invention) and the position P2 (corresponding to the second position in the present invention) are connected. The second straight path 40b connects a position P2 and a position P3 (corresponding to the third position in the present invention) separated from each other in the Y direction. A chair 41 on which the subject sits at the start of the movement is installed at position P1. A mobile terminal 10 is installed at a position separated from the position P3 by a predetermined distance in the Y direction. The mobile terminal 10 is arranged such that the optical axis LX of the camera passes through positions P2 and P3 along the Y direction in plan view. The mobile terminal 10 is arranged so that the chair 41 and the entire course 40 are included within its imaging range. An image of a scene photographed by the camera of the mobile terminal 10 is shown in FIG. again,
The position of the mobile terminal 10 is fixed using a camera stand or the like.

Markers 42 to 48 are formed on the floor of the room in which the course 40 is set for the subject to recognize the course 40 . The markers 42-46 (route markers in the present invention) have linear shapes. These may be formed, for example, by sticking a colored elongated adhesive tape on the floor surface in a line segment, or may be formed by drawing on the floor surface with paint or the like. Among them, the marker 42 is positioned from the position P1 along the first linear path 40a to the position P
extends up to 2. The markers 43 and 44 extend parallel to the first linear path 40a so as to sandwich the first linear path 40a between them in the Y direction. The distance between the markers 42 and 43 in the Y direction is equal to the distance between the markers 42 and 44 in the Y direction. Marker 45 connects with the endpoint of marker 42 at position P2, from which second linear path 4
It extends along 0b to position P3. Markers 46 and 47 are connected to the endpoints of markers 43 and 44 and extend therefrom parallel to second straight path 40b. Markers 46 and 47 sandwich the first linear path 40a between each other with respect to the X direction. The distance between the markers 45 and 46 in the X direction is equal to the distance between the markers 45 and 47 in the X direction. With these, the markers 42 to 46 are aligned with the first straight path 40a and the second straight path 40
The entire course 40 consisting of b is indicated. The markers 48 and 49 (position markers in the present invention) have the shape of circles filled with a predetermined color. These may be formed, for example, by attaching a round adhesive tape filled with a predetermined color to the floor surface, or may be formed by drawing on the floor surface with paint or the like. Marker 48 is placed at position P2 and marker 49 is placed at position P3. As a result, the marker 48 points to the position P2 and the marker 49 points to the position P3.

The test using course 40 is as follows. As shown in FIG. 5, the subject first
The operation is started from the state of being seated on the chair 41 at the position P1. Next, the subject stands up from the chair 41 and walks along the marker 42 in the area between the markers 43 and 44 to the position P2.
At position P2, it turns to position P3 and walks the area between markers 46 and 47 along marker 45 to position P3. When the subject reaches position P3, the motion ends. The state of the subject from the start of the motion to the end of the motion is imaged by the mobile terminal 10 .

Next, the overall flow of the analysis data acquisition method according to one embodiment of the present invention using the analysis data acquisition system 1 will be described with reference to FIG. First, as shown in FIG. 4, a course 40, a chair 41 and a mobile terminal 10 are installed (step S1). Next, with the subject sitting on the chair 41 and the ID recording board placed in front of the mobile terminal 10, imaging by the mobile terminal 10 is started (step S2). The subject's ID number and imaging date set in advance are recorded on the ID recording board (see FIG. 7). Next, course 4
0, and the portable terminal 10 is caused to take a moving image of the subject's movement from start to finish (step S3). Note that step S3 corresponds to the subject operation step and imaging step in the present invention. As a result, the right half of the subject's body (
3) is imaged, and the front (see FIG. 3) of the subject walking along the second straight path 40b, which is the movement path from position P2 to position P3, is imaged. A moving image data file indicating the imaging result is stored in a memory or the like in the mobile terminal 10 . Next, the mobile terminal 10
and the data acquisition device 20 are connected (step S4).

Next, a browser is activated on the mobile terminal 10, and the mobile terminal 10 and the data acquisition device 2 are connected.
0, data communication is started (step S5). 7 on the touch panel display of the mobile terminal 10 based on the data transmitted from the data acquisition device 20 to the mobile terminal 10.
is displayed. The screen IM1 is an image for transferring the moving image data file stored in the memory in the mobile terminal 10 to the data acquisition device 20 in step S3. A control image C for specifying an IP address or URL is displayed on the screen IM1.
1. Control image C2 for inputting the ID number, control image C3 for selecting the moving image data file to be transferred, and control image C4 for starting transfer (hereinafter referred to as “image C1”, “image C2 ”, “Image C3” and “Image C4”.) are included. When the user touches the position of these control images on the touch panel display with a finger or the like, the control image corresponding to the touch position is selected, and the process of inputting characters and numbers and selecting items from the list are performed. Selecting processing, etc. are executed. The image C1 in FIG. 7 shows the result of inputting the IP address corresponding to the data acquisition device 20 . The ID number input through the image C2 corresponds to the ID number recorded on the ID recording board in step S2. When a moving image data file is selected from the file list through image C3, image V corresponding to the beginning of the selected moving image data file is displayed below image C3. In step S2, since the ID recording board is imaged immediately after the imaging is started, the ID recorded on the board through the image V in FIG.
The number is visible. By selecting a moving image data file through the image C3, the user can check the ID number displayed as the image V, enter the ID number through the image C2, or confirm the contents of the entered ID number. After inputting the ID number through the image C2 and selecting the moving image data file through the image C3, by selecting the image C4, transfer of the moving image file data and the ID number from the portable terminal 10 to the data acquisition device 20 is started. (step S6). In the data acquisition device 20, the moving image file data transferred together with the ID number is associated with the ID number and stored in a memory or the like.

Next, in the data acquisition device 20, a motion analysis data acquisition process is executed based on the moving image data file transferred from the portable terminal 10 (step S7). The details of this processing will be described later. Note that step S7 corresponds to the feature point generating step in the present invention. When the motion analysis data acquisition process is completed, the data acquisition device 20 deletes the moving image data file used to acquire the motion analysis data from the memory, hard disk drive, or the like (data recording unit in the present invention).

Next, in the PC 30 connected to the data acquisition device 20, download processing of the motion analysis data generated by the data acquisition device 20 is executed (step S8). A browser is activated on the PC 30 and an IP address or the like corresponding to the data acquisition device 20 is specified. As a result, the screen IM2 shown in FIG. The screen IM2 is an image for transferring the motion analysis data generated by the data acquisition device 20 in step S7 to the PC30.

The screen IM2 includes an information display T showing file information. Also, the screen IM
2, based on the data transmitted from the data acquisition device 20, a control image C11 for starting download and a control image C12 for deleting the file.
(hereinafter referred to as “image C11” and “image C12”). The information display T and the images C11 and C12 display a plurality of character strings and icon images in tabular form. The information display T includes a character string indicating the ID number, file registration date and time, and status. These strings are displayed in a horizontal and vertical tabular format, which gives the I
The relationship between the file registration dates and statuses corresponding to the D numbers and ID numbers is shown. There are four statuses: "Acquired", "Converting", "Completed", and "Error". “Obtained” indicates that the motion analysis data has been transferred to the PC 30 . “Converting” indicates that the data acquisition device 20 is in the process of converting the moving image file data into motion analysis data. "Completed" indicates that the data acquisition device 20 has completed the conversion process from the moving image file data to the motion analysis data. "Error" indicates that an error has occurred for some reason in the conversion processing from the moving image file data to the motion analysis data in the data acquisition device 20, and the conversion processing has not been completed.

The image C11 includes an icon image for starting download. The icon image is displayed at each position corresponding to the ID number in the information display T in the vertical direction. In addition, an ID for which conversion processing from video file data to motion analysis data has not been completed
At the position corresponding to the number, the character string "under preparation" is displayed instead of the icon image.
When an icon image is selected using an input device such as a mouse, motion analysis data associated with the ID number corresponding to the icon image is transferred from the data acquisition device 20 to the PC 30.
downloaded to.

Image C12 includes an icon image for deleting a file. The icon images are displayed at respective positions corresponding to the ID numbers in the information display T in the vertical direction. When an icon image is selected using an input device, an I
The motion analysis data associated with the D number is deleted from the memory or the like in the data acquisition device 20 (step S9). Note that, in the present embodiment, the data acquisition device 20 waits for a predetermined length (for example, 10 minutes) from the start of the conversion process to the motion analysis data before the start of the conversion process to the motion analysis data. It is configured to delete video data files that have passed time from memory or the like. By configuring the data acquisition device 20 in this way, it is possible to prevent the moving image data file from remaining in the device when the conversion process ends abnormally or is stopped.

The processing of the data acquisition device 20 for acquiring the motion analysis data from the moving image data file in step S7 of FIG. 6 will be described below with reference to FIG. First, the data acquisition device 20 extracts the markers 48 and 4 based on specific frame images in the moving image data file.
9 is executed to detect the position in the image (step S11). A frame image corresponds to a still image of the subject at each timing. The reason why the positions of the markers 48 and 49 in the image are detected is that it is possible to grasp the accurate position of the subject with respect to the course 40 reflected in the imaging indicated by the moving image data file based on the positions (see step S15). .
Note that step S11 corresponds to the derivation step in the present invention.

In this embodiment, "SSD" object detection (Liu, W., Angue
lov, D., Erhan, D., Szegedy, C., Reed, S., Fu, CY, & Berg, AC (2016, Oct.
ober). SSD: Single shot multibox detector.
vision (pp. 21-37)) is adopted. The outline of image processing by “SSD” object detection is as follows. "SSD" object detection provides information indicating which locations in the raw image to be processed contain the object to be detected. This information is acquired as the coordinates of the four corners of a rectangular bounding box (corresponding to frames B1 and B2 in FIG. 5) indicating the area where the object to be detected exists. To obtain this information, default boxes, which are rectangular frames of different shapes and sizes, are set for each position in the original image to be processed. Then, as a neural network, when the original image is input, for each default box, the difference between the default box and the bounding box and the value indicating the classification of the subject included in the default box are output. A trained network is used. The differences between the default box and the bounding box are obtained, for example, as the X and Y position differences between the centers of the boxes and the width and height differences between the boxes. The value indicating the classification of the subject included in the default box is obtained as a value indicating the likelihood that the subject included in the default box corresponds to that type for each preset type. In this embodiment, since the detection targets are limited to the markers 48 and 49, a neural network that has been specially trained to output the likelihood of corresponding to the markers 48 and 49 with high accuracy is used. ing. And, in this embodiment, the marker 48
and 49, the default box with the highest likelihood is extracted as a candidate for object detection. From the coordinates indicating the extracted default box and their difference,
As the coordinates of the boxes B1 and B2 showing the markers 48 and 49, the predicted coordinates of the bounding box are obtained.

Next, the data acquisition device 20 performs a derivation process of the feature points of the body of each person included in the subject and how they are connected with respect to each frame image (step S12). In this embodiment, as this process, "OpenPose" (Zhe Cao and Tomas Simon
d Shih-En Wei and Yaser Sheikh. Realtime Multi-Person 2D Pose Estimation using
Part Affinity Fields. In CVPR, 2017.) has been adopted. "Open Pose"
The neural network used in is configured to mainly execute the following first to third processes. The first process is a process of generating a feature map image, which is an image obtained by roughly extracting the features of the subject included in the original image. The second process is a process of generating an image corresponding to a vector map indicating directions connecting positions corresponding to body parts in a region corresponding to a person in the input image. For example, as shown in FIG. 2, in the region between the feature point F1 indicating the subject's neck and the feature point F5 indicating the right shoulder, the feature point F1
and the feature point F5 are distributed. The third process is a two-dimensional calculation of the probability that each part of the body (for example, the feature point F1 indicating the subject's neck, the feature point F5 indicating the right shoulder, etc.) exists in the region corresponding to the person in the input image. This process generates an image corresponding to a reliability map showing a typical distribution. This neural network has the first to third
By repeating the second and third processes a plurality of times using the image generated by the process 1 as an input, the finally obtained output is the coordinates of the feature points of the person indicated by the original image and the distance between the feature points. It is learned using deep learning so that it becomes data indicating how to connect (that is, data representing the general shape of the body as shown in Fig. 2). When a plurality of persons are included in the image, how the feature points are connected is derived for each person while distinguishing the persons. in short,
Data indicating feature points and how the feature points are connected is generated individually for each person. The neural network also derives a numerical value indicating the likelihood of each feature point for each feature point. Note that the neural network may be configured to derive feature points and connection methods not only for people but also for human-like objects.

Next, the data acquisition device 20 extracts subject candidates from among persons indicated by the coordinates of the feature points derived in step S12 and the manner of connection between the feature points (step S13).
). If a plurality of persons are included in the image, the data derived in step S12 are data indicating the coordinates of the feature points and how the feature points are connected to each other for the plurality of persons. Therefore, in step S13, out of the numerical values indicating the plausibility of the feature points extracted in step S12, the numerical value at which the plausibility has a predetermined height (for example, when the numerical value is derived in the range of 0 to 1 A person having a predetermined number (for example, 5) or more of feature points of 0.4) or more in is extracted as a subject candidate.

Next, the data acquisition device 20 evaluates the positional relationship between the person and the course 40 for each subject candidate extracted in step S13 (step S14). Note that step S14 corresponds to the evaluation step in the present invention. Specifically, based on the coordinates of feature points in a predetermined frame (for example, a frame in which the subject is walking),
Execute the following process. For each person, the average value calculated for the X coordinate value of the feature points and the maximum value of the Y coordinate values of the feature points (the Y coordinate value of the lowest feature point in the image) are taken as the representative position of the person. . The data acquisition device 20 derives the linear distance between this representative position and the course 40 . For deriving the straight-line distance, the marker 4 obtained in step S11
8 and 49 image positions (coordinates) are used. That is, based on the positions of the markers 48 and 49, the position in the image of each line segment along the markers 42 and 43 shown in FIG. 10 is obtained. By calculating the linear distance between the representative position and the line segment based on this, the linear distance between the representative position and the course 40 is calculated.

Then, the data acquisition device 20 determines that the person with the shortest linear distance derived in step S14 is the subject (step S15). Note that step S15 corresponds to the determination step in the present invention. For example, FIG. 10 shows representative position R1 (X1, Y1) for person M1 and representative position R2 (X2, Y2) for person M2. In this case, the data acquisition device 20 determines that the straight line distance D1 between R1 regarding the person M1 and the course 40 is smaller than the straight line distance D2 between the R2 regarding the person M2 and the course 40.
1 is determined to be a subject. The data indicating the coordinates of the feature points of the subject determined in step S15 and how the feature points are connected to each other correspond to the motion analysis data acquired in step S7 of FIG. The data for motion analysis includes the coordinates of the feature points and the connection between the feature points during the motion of standing up from the chair at position P1 and during the motion from position P1 to position 2 for the right half of the body image of the subject. data (first analysis data in the present invention) and data indicating the coordinates of feature points during movement from position P2 to position 3 and how the feature points are connected to each other in the front image of the subject (this data). data for second analysis in the invention).

According to the present embodiment described above, in the test using the course 40 shown in FIG. walk along the direction to Then, the right half of the subject's body is continuously imaged as a moving image by the camera of the mobile terminal 10 . As a result, a moving image of the subject walking can be obtained continuously from the side of the direction of travel. Then, based on the moving image, the data acquisition device 20 acquires motion analysis data indicating the coordinates of a plurality of feature points on the human body and how the feature points are connected to each other. For this reason, such data should include the conditions of the subject's movements as seen from the side, such as the degree of the subject's forward leaning posture, the speed of swinging the leg, the speed of waving the hand, and the degree to which the knee is raised. It is possible to reflect The strength and weakness of a person's bodily functions often appear in these elements. For example, frail subjects tend to lean forward and have weak hand movements. In this way, the state of motion seen from the side of the subject can be an important factor in determining the person's physical function. This embodiment can numerically acquire information indicating such important determination factors as the coordinates of feature points on the body. Therefore, by analyzing numerical values, it is possible to directly and in detail evaluate physical functions. As described above, according to the present embodiment, it is possible to acquire motion analysis data that enables direct and detailed analysis of the motion of a subject for evaluation of a person's bodily functions.

Further, in the present embodiment, in the test using the course 40 shown in FIG.
2, the direction is changed toward the mobile terminal 10, and from there to the position P3, the mobile terminal 1
Walk towards 0. Then, the camera of the mobile terminal 10 continuously captures the front of the subject as a moving image. For this reason, we can obtain a moving image that continuously captures the subject walking from the front. Then, based on the moving image, the data acquisition device 20 acquires motion analysis data indicating the coordinates of a plurality of feature points on the human body and how the feature points are connected to each other. According to this, it is possible to acquire motion analysis data for evaluating physical function, etc., based on the state of the motion seen from the front of the subject, such as left-to-right sway of the trunk due to walking, for example.

Further, in the present embodiment, the marker 4 that visually indicates the course 40 for the subject
2 to 49 are installed. According to this, the subject can recognize the moving route accurately by visually recognizing these markers. Therefore, it is possible to acquire motion analysis data for accurately evaluating body functions. Of these markers, markers 43 and 44 are the first
They are formed so as to sandwich the markers 42 along the straight path 40a. Similarly,
Markers 46 and 47 are formed such that they sandwich marker 45 along second linear path 40b. For example, if the movement path is indicated only by the markers 42 and 45, the subject may walk with both feet aligned with the markers 42 and 45 (as if moving on a balance beam), and may find it difficult to walk. have a nature. On the other hand, marker 42
By forming the markers 43 and 44 and 46 and 47 so as to sandwich the markers 43 and 44 and 45, the subject can secure a width for placing the foot between them, so that the subject can feel that walking is easy.

Further, in this embodiment, the mobile terminal 10 and the data acquisition device 20, and the data acquisition device 20 and the PC 30 are connected by a wired connection such as a USB system. The moving image data files and moving image analysis data exchanged between these devices reflect the subject's walking appearance, so they contain information that is personal to the subject. On the other hand, as described above, data is exchanged via a wired connection.
Therefore, it is possible to reduce the risk of data related to personal information leaking to the outside. Furthermore, in this embodiment, when the motion analysis data acquisition process is completed, the data acquisition device 20 deletes the moving image data file used for acquiring the motion analysis data from the memory or the like. In addition, the data acquisition device 20 automatically deletes all moving image data files from the memory or the like when the power of the data acquisition device 20 is once turned off and then turned on again. Therefore, it is possible to further reduce the risk of data related to personal information leaking to the outside.

<Other Modifications>
The above is a description of preferred embodiments of the present invention, but the present invention is not limited to the above-described embodiments, and various modifications can be made within the scope described in Means for Solving the Problems. It is possible.

For example, in the above-described embodiment, the path along which the subject walks is the first straight path 40a and the second straight path 40a.
It is set as a straight path 40b. However, it does not necessarily have to be a straight path. It may be set as a meandering path or a curving path. A first straight path 40 a is set along the X direction orthogonal to the optical axis of the camera of the mobile terminal 10 . by this,
A moving image of the right half of the subject walking along the first straight path 40a can be acquired. However, the first straight path 40a does not have to be orthogonal to the optical axis of the camera as long as a moving image of the subject's right or left half can be obtained. For example, the first straight path 40a may be set to intersect the optical axis of the camera at an angle other than 90°. Also, the subject may run rather than walk. Further, the subject may be allowed to walk not only through the route of position P1→position P2→position P3, but also through the route of position P3→position P2→position P1 after reaching position P3. In this case, not only the right half of the body and the front, but also the left half of the body and the back of the subject can be acquired.

Further, in the above-described embodiment, detection of the positions of the markers 48 and 49 is performed using a neural network. However, other methods may be used for detection. For example, it may be detected based on color information of pixels in the image.

Further, in the above-described embodiment, in step S15, among the candidates for the subject extracted in step S13, the person with the shortest linear distance between the representative position derived in step S14 and the course 40 is determined to be the subject. ing. However, the determination of the subject may be made based on the distance between the representative position and the specific position on the course 40 instead of the linear distance between the representative position and the course 40 . The specific position is a position having a predetermined positional relationship with the course 40, and may be, for example, the same position as the midpoint of the marker 42 in the X direction and the same position as the midpoint of the marker 45 in the Y direction.

1 analysis data acquisition system 10 portable terminal 20 data acquisition device 30 PC

Claims (10)

an imaging means for converting an image of a subject into data and outputting the data; and data indicating coordinates of a plurality of characteristic points on a human body included in the subject indicated by the data, based on the data output from the imaging means. A method for obtaining data for analyzing the movement of a person using a feature point derivation means to generate, comprising:
The subject is caused to stand up from sitting on the object in a first position, and a second position spaced apart from the first position in a direction parallel to the horizontal and intersecting the optical axis of the imaging means.
a subject motion step of walking or running to a position;
an imaging step of causing the imaging means to continuously image at least one of the left half and right half of the subject during movement in the subject movement step;
Based on the data output from the image capturing means regarding the image capturing performed in the image capturing step, the subject's motion of standing up from the object and moving from the first position to the second position. and a feature point generating step for causing the feature point deriving means to generate first analysis data indicating the coordinates of the plurality of feature points at each timing. In the subject movement step, the subject is caused to change direction toward the imaging means at the second position and walk or run toward the imaging means from the second position to a third position closer to the imaging means;
In the imaging step, causing the imaging means to continuously image the front of the subject during the motion in the subject motion step,
In the feature point generation step, each timing of the subject during movement from the second position to the third position based on the data output from the imaging means regarding the imaging performed in the imaging step. indicating the coordinates of the plurality of feature points in the second
2. The motion analysis data acquisition method according to claim 1, wherein the analysis data is generated by the feature point deriving means. In the subject movement step, a visible marker is installed for the subject to recognize the moving route,
The markers include at least one of position markers respectively indicating the second position and the third position, and path markers extending along a line segment indicating an assumed straight path through which the subject passes. 3. The motion analysis data acquisition method according to claim 2, wherein: 4. The motion analysis data acquisition method according to claim 3, wherein the path marker extends along two line segments extending parallel to the assumed straight path so as to sandwich the assumed straight path. 5. The motion analysis data according to claim 3, further comprising a deriving step of deriving the coordinates of the marker based on data output from the imaging means regarding the imaging performed in the imaging step. Acquisition method. 6. The motion analysis data acquisition method according to claim 5, wherein the derivation step uses a neural network that has been trained to detect objects with respect to the markers. wherein the feature point deriving means individually generates data indicating the coordinates of the plurality of feature points for each human body in the subject;
The plurality of feature points for each of the body of the person based on the coordinates of the marker derived in the derivation step and the first analysis data and the second analysis data generated in the feature point generation step. and an evaluation step of evaluating the positional relationship between the assumed straight path and
7. The motion analysis data according to claim 5, further comprising a determination step of determining which part of the person's body is the subject based on the evaluation result of the evaluation step. Acquisition method. imaging means for converting a subject image into data and outputting the data;
feature point deriving means for generating, based on the data output from the imaging means, data indicating the coordinates of a plurality of feature points on a human body included in the subject indicated by the data;
an object on which the subject sits, placed in a first position;
a visible marker for making the subject recognize a route from the first position to a second position spaced apart from the first position in a direction intersecting the optical axis of the imaging means,
As the subject stands up from sitting on the object, the subject moves from the first position to the second position.
At the time of moving to the position, at least one of the left half of the body and the right half of the subject is imaged by the imaging means, and based on the data output from the imaging means at this time,
Analysis data indicating the coordinates of the plurality of feature points at each timing of the subject during the motion of standing up from the object and during the motion of moving from the first position to the second position to the feature point deriving means. A motion analysis data acquisition system characterized by generating. comprising a mobile terminal and first and second computers;
data converted from the subject image is output to the first computer via a wired connection from the portable terminal functioning as the imaging means;
8. Data indicating the coordinates of the plurality of feature points is output from the first computer functioning as the feature point deriving means to the second computer via a wired connection. The motion analysis data acquisition system described in . The first computer has a data recording section for recording data from the mobile terminal, and from the mobile terminal recorded in the data recording section after generating the data indicating the coordinates of the plurality of characteristic points. 10. The motion analysis data acquisition system according to claim 8 or 9, wherein the data of is deleted.

Images