JP7366299B1 - Information processing method, computer program and information processing device - Google Patents

Abstract

The present invention provides an information processing method, a computer program, and an information processing device that can support implementation of a functional reach test.
[Solution] In the information processing method according to the present embodiment, an information processing device acquires a video image of a subject performing a predetermined action related to a functional reach test, and performs the predetermined action from the video image. A start frame to start and an end frame to end the predetermined action are specified, and based on the start frame and the end frame, a distance traveled by the predetermined part of the subject due to the predetermined action is calculated.
[Selection diagram] Figure 1

Description

The present invention relates to an information processing method, a computer program, and an information processing apparatus for processing information related to a functional reach test.

In Patent Document 1, at least two measurement items are selected from a plurality of measurement items related to the body, input of a first measurement result that is the measurement result of the measurement items is received, and a weighting coefficient is determined according to the combination of measurement items. A determination system has been proposed in which the first measurement result is weighted using the weighted method, and it is determined whether or not a mobile product for the elderly can be used based on the weighted second measurement result.

JP2020-194348A

Patent Document 1 lists reach distance in a functional reach test as one of a plurality of measurement items. The functional reach test is a test used, for example, in nursing care settings to measure the balance ability of a cared person. This is done by measuring the distance traveled. The functional reach test is performed manually, for example, by a measurer using a ruler or measuring tape to measure the distance traveled, so it is time-consuming to carry out.

The present invention has been made in view of such circumstances, and its purpose is to provide an information processing method, a computer program, and an information processing device that can support the implementation of functional reach tests. .

In an information processing method according to an embodiment, an information processing device acquires a moving image of a subject performing a predetermined movement related to a functional reach test, and the subject's hands are at the height of the shoulders. or the angle formed by the subject's hand and body is within a range of 90 degrees ± a predetermined angle, the position of the subject's hand is monotonous to one side in the horizontal direction. detecting from the moving image a motion in which the movement is monotonically displaced to the other direction, identifying a frame corresponding to the start point of displacement in one direction in the movement as a start frame for starting the predetermined movement; identifying an end frame at which the predetermined action ends from the image, and calculating a distance traveled by the predetermined part of the subject due to the predetermined action based on the start frame and the end frame;
Further, in the information processing method according to one embodiment, the information processing device acquires a moving image of a subject performing a predetermined action related to a functional reach test, and selects a start frame for starting the predetermined action from the moving image. and specifying an end frame at which the predetermined action ends, and based on the start frame and the end frame, calculate the distance traveled by the predetermined part of the subject due to the predetermined action, and calculate the distance traveled by the predetermined part of the subject in the end frame. The distance is corrected according to the difference in the position of the shoulders or the difference in the position of the left and right hands.

According to one embodiment, it can be expected to support implementation of functional reach tests.

FIG. 1 is a schematic diagram for explaining an overview of an information processing system according to the present embodiment. It is a schematic diagram for explaining the outline of a functional reach test. FIG. 1 is a block diagram showing an example of a configuration of a server device according to the present embodiment. FIG. 2 is a block diagram showing the configuration of a terminal device according to the present embodiment. It is a flowchart which shows the procedure of the process performed by the server apparatus based on this Embodiment. FIG. 2 is a schematic diagram showing an example of skeleton extraction. FIG. 3 is a schematic diagram for explaining calculation of reach distance. FIG. 3 is a schematic diagram for explaining correction of reach distance. It is a schematic diagram for explaining a movement strategy. It is a schematic diagram for explaining a movement strategy. It is a flowchart which shows the procedure of the process performed by the terminal device based on this Embodiment. FIG. 2 is a schematic diagram showing an example of displaying measurement results by a terminal device.

A specific example of an information processing system according to an embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is indicated by the scope of the claims, and is intended to include all changes within the meaning and scope equivalent to the scope of the claims.

<System overview>
FIG. 1 is a schematic diagram for explaining an overview of an information processing system according to this embodiment. The information processing system according to the present embodiment is a system that supports a functional reach test by a subject 101, such as an elderly person or a cared person. In the information processing system according to the present embodiment, a user 102, such as a family member or a caregiver of a subject 101, uses a terminal device 3 such as a smartphone or a tablet terminal device to help the subject 101 perform a functional reach test. Photograph the person performing the relevant action. The user 102 transmits the data of the moving image obtained by photographing to the server device 1, and requests measurement of the reach distance in the functional reach test.

The server device 1 receives the moving image data transmitted from the user's terminal device 3, and measures the reach distance of the functional reach test for the target person 101 captured in the moving image based on the received moving image. Perform the processing to do. Further, in this embodiment, the server device 1 performs a process of identifying the movement strategy regarding the functional reach test of the subject 101. The server device 1 transmits information regarding the reach distance and movement strategy of the functional reach test obtained as a result of these processes to the requesting terminal device 3 as a measurement result. Based on the information of these measurement results received from the server device 1, the terminal device 3 displays and notifies the user 102 of information regarding the reach distance and movement strategy regarding the functional reach test of the subject.

FIG. 2 is a schematic diagram for explaining an overview of the functional reach test. In the functional reach test, the measurement subject 101 first stands upright and takes a posture with his arms raised substantially horizontally to the ground (substantially perpendicular to his body) (see the left side of FIG. 2). The subject 101 stretches his arms forward as much as possible while keeping his arms raised (see the right side of FIG. 2), and then returns to the original posture. When the subject 101 performs this series of movements, the measurement start position is the position of the hand when the subject 101 first stands upright and raises his arm, and the measurement ends when the arm reaches the farthest point when he extends his arm forward. The reach distance of the functional reach test is the distance traveled in the horizontal direction from the start position to the end position. This reach distance can be used as an index of the balance ability of the subject 101. In addition, if the subject 101 can return to the original position without moving the position of the feet after extending the arm, the reach distance is determined as a success, and if the subject 101 cannot return to the original position, the reach distance is determined as a failure. No need to measure. The user 102 photographs the above series of actions by the subject 101 using the terminal device 3 and transmits the data of the moving image obtained by the photographing to the server device 1 .

Further, in the present embodiment, the server device 1 identifies what kind of strategy the subject 101 adopted when performing the above-described functional reach test based on the moving image. In this embodiment, the server device 1 specifies whether the strategy taken by the subject 101 is, for example, a hip joint strategy, an ankle joint strategy, or a heel raise strategy. However, the above three strategies are just examples and are not limited to these. The server device 1 may classify and identify the target person's 101 strategies into two strategies, a hip joint strategy and an ankle joint strategy, or may classify and identify them into four or more strategies. Although a detailed explanation will be omitted, the hip joint strategy is a strategy in which the above movement is performed by bending only the hip joint, the ankle joint strategy is a strategy in which the movement is performed by bending the ankle and hip joint, and the heel raise strategy is a strategy in which the above movement is performed by bending only the hip joint. This is a strategy of raising the body and performing the movements.

The server device 1 according to the present embodiment measures the reach distance of the target person 101 based on the moving image received from the terminal device 3 and specifies the movement strategy. The server device 1 transmits information regarding the measured reach distance and the identified movement strategy to the terminal device 3, and the terminal device 3 that receives this displays information regarding the reach distance and movement strategy of the functional reach test of the subject 101. . At this time, the server device 1 transmits information such as advice to the target person 101 based on the measurement result of the reach distance and the identification result of the movement strategy to the terminal device 3, and the terminal device 3 transmits information such as advice to the target person 101 based on the result of measuring the reach distance and the movement strategy. Information such as advice may also be displayed.

<Device configuration>
FIG. 3 is a block diagram showing an example of the configuration of the server device 1 according to the present embodiment. The server device 1 according to the present embodiment includes a processing section 11, a storage section 12, a communication section (transceiver) 13, and the like. Note that although the present embodiment will be described assuming that the processing is performed by one server device, the processing may be performed in a distributed manner by a plurality of server devices.

The processing unit 11 includes an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or a quantum processor, a ROM (Read Only Memory), a RAM (Random Access Memory), etc. It is configured using a storage device. The processing unit 11 reads out and executes the server program 12a stored in the storage unit 12 to perform processes such as measuring reach distance in a functional reach test based on a moving image and specifying a movement strategy. Performs various processing.

The storage unit 12 is configured using a large-capacity storage device such as a hard disk, for example. The storage unit 12 stores various programs executed by the processing unit 11 and various data necessary for processing by the processing unit 11. In this embodiment, the storage unit 12 stores a server program 12a executed by the processing unit 11. The storage unit 12 is also provided with a subject information storage unit 12b that stores information regarding the subject 101 who is the subject of the functional reach test.

In this embodiment, the server program (computer program, program product) 12a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disk, and the server device 1 reads the server program 12a from the recording medium 99 and stores it. It is stored in section 12. However, the server program 12a may be written into the storage unit 12, for example, during the manufacturing stage of the server device 1. Further, for example, the server program 12a may be distributed by another remote server device, and the server device 1 may obtain it through communication. For example, the server program 12 a may be recorded on the recording medium 99 by a writing device and written into the storage unit 12 of the server device 1 . The server program 12a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 99.

The subject information storage unit 12b stores information such as the height and age of the subject 101 in association with, for example, the ID or name of the subject 101. In addition, the subject information storage unit 12b associates the results of the functional reach test conducted by the subject 101 (reach distance, movement strategy, etc.) with information such as the date on which the functional reach test was conducted, and Store as history.

The communication unit 13 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN (Local Area Network), the Internet, and the like. In this embodiment, the communication unit 13 communicates with one or more terminal devices 3 via the network N. The communication unit 13 transmits data provided from the processing unit 11 to other devices, and also provides data received from other devices to the processing unit 11.

Note that the storage unit 12 may be an external storage device connected to the server device 1. Further, the server device 1 may be a multicomputer including a plurality of computers, or may be a virtual machine virtually constructed by software. Further, the server device 1 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that accepts operation input, a display unit that displays an image, etc. .

Furthermore, the server device 1 according to the present embodiment includes a moving image acquisition section 11a, a skeleton extraction section 11b, a frame identification section 11c, and a processing section 11 that reads and executes a server program 12a stored in the storage section 12. , the distance calculation section 11d, the strategy identification section 11e, the measurement result notification section 11f, etc. are implemented in the processing section 11 as software-like functional sections. In this figure, as the functional units of the processing unit 11, functional units related to the functional reach test are illustrated, and functional units related to other processes are omitted.

The moving image acquisition unit 11a performs a process of acquiring a moving image previously shot by the user 102 with the terminal device 3 by communicating with the terminal device 3 used by the user 102 through the communication unit 13. The moving image acquisition unit 11a temporarily stores the moving image data acquired from the terminal device 3 in the storage unit 12. Note that in this embodiment, the user 102 photographs the movement of the subject 101 with the terminal device 3, the video data is stored in the terminal device 3 after the photographing is completed, and the user then records the video data. It is assumed that moving images are sent and received from the terminal device 3 to the server device 1 in such a manner that the selected image is selected and uploaded to the server device 1. However, the terminal device 3 may transmit the moving image to the server device 1 in real time while photographing the movements of the subject, and the server device 1 may receive the moving image in real time and store it in the storage unit 12.

The skeleton extraction unit 11b performs processing for extracting the skeleton of the subject 101 captured in the video image acquired by the video image acquisition unit 11a. In the present embodiment, the skeleton extraction unit 11b extracts a skeleton from an image using, for example, a learning model that has been subjected to machine learning in advance, so-called AI (Artificial Intelligence). For example, OpenPose can be used as an AI for extracting a skeleton from an image, but the AI is not limited to this. The skeleton extraction unit 11b can extract the skeleton of the subject 101 performing the functional reach test using various existing methods for extracting a human skeleton from an image. Note that since the technique for extracting a skeleton from an image is an existing technique, detailed description thereof will be omitted in this embodiment.

The frame identifying unit 11c uses the skeleton information extracted by the skeleton extracting unit 11b to select a frame corresponding to the start of measurement of the functional reach test (see the left side of FIG. 2) among a plurality of frames constituting the video image. A process is performed to identify the start frame and the end frame corresponding to the end of measurement of the functional reach test (see the right side of FIG. 2). In this embodiment, a moving image is a plurality of frames (still images) connected in chronological order. For example, the frame identifying unit 11c identifies one or more frames in which the subject 101 is in the starting posture of the functional reach test from among a plurality of frames constituting a moving image, and selects these as starting frame candidates. do. The starting posture of the functional reach test is determined based on skeletal information, for example, whether the subject 101's hand height is approximately the same as the shoulder height (the difference between the two heights is within a predetermined range), or The determination is made based on the fact that the angle between the hand and the body of 101 is approximately a right angle (the angle is within the range of 90 degrees ± a predetermined angle).

The frame identifying unit 11c also determines that the position of the hand of the subject 101 is monotonically displaced horizontally in either the left or right direction, and then monotonously displaced in the opposite direction, based on the frame that is a candidate for the start frame. Detect motion from video images. When this series of motions is detected, the frame specifying unit 11c selects one frame in which the displacement of the hand position has started in this series of motions from among the plurality of candidates for the start frame, and selects the selected frame. This frame is the starting frame. The frame identifying unit 11c also selects one frame in which the position of the hand of the subject 101 is farthest from the position in the start frame from among the multiple frames included in this series of actions, and ends the selected frame. Frame.

The distance calculation unit 11d performs a process of calculating the reach distance of the subject 101 in the functional reach test based on the start frame and end frame specified by the frame identification unit 11c. The distance calculation unit 11d calculates the distance in the horizontal direction between the position of the toe and the position of the hand of the subject 101 (hereinafter referred to as the first distance X1) based on the skeletal information of the subject 101 in the start frame. Similarly, the distance calculation unit 11d calculates the distance in the horizontal direction between the position of the toe and the position of the hand of the subject 101 (hereinafter referred to as second distance X2) based on the skeletal information of the subject 101 in the end frame. do. The distance calculation unit 11d can calculate the difference between the calculated first distance X1 and second distance X2 as the reach distance of the functional reach test.

Note that the first distance, second distance, and reach distance calculated by the distance calculation unit 11d using the above method are numerical values calculated based on the number of pixels of frames (images) that constitute a moving image. Therefore, the distance calculation unit 11d can calculate the actual value of the reach distance by multiplying the number of pixels of the calculated reach distance by the length per pixel. Here, the distance calculation unit 11d can calculate the length per pixel based on, for example, the height of the subject 101 stored in the subject information storage unit 12b. The distance calculation unit 11d calculates the number of pixels corresponding to the height of the subject photographed in the start frame, for example, and divides the height of the subject 101 stored in the subject information storage unit 12b by the number of pixels, so that 1 The length per pixel can be calculated. Note that the method of calculating the length per pixel by the distance calculation unit 11d does not need to be based on the height of the subject 101, and various existing methods may be adopted. For example, an index object or the like having a predetermined length can be photographed together with the subject 101, and the distance calculation unit 11d can calculate the length per pixel based on the length of the index object in the moving image.

Further, in the present embodiment, the distance calculation unit 11d can correct the reach distance calculated by the above method based on the movement of the subject 101 in the functional reach test. For example, when the subject 101 rotates his shoulders and extends his hand in a functional reach test, the reach distance becomes longer than when he extends his hand without rotating his shoulders. Therefore, the distance calculation unit 11d identifies the positions of the left and right shoulders or the positions of the left and right hands of the subject 101 in the end frame, and calculates the difference between the positions of the left and right shoulders or the difference between the positions of the left and right hands in the horizontal direction. . The distance calculation unit 11d corrects the reach distance by, for example, subtracting half the distance of the difference from the above reach distance based on the calculated difference. The distance calculation unit 11d can use the corrected value as the final reach distance, and use it as the measurement result of the functional reach test.

The strategy specifying unit 11e performs a process of specifying the strategy taken by the subject 101 in the movement of the functional reach test, based on the posture of the subject 101 in the end frame specified by the frame specifying unit 11c. In this embodiment, the strategy specifying unit 11e specifies the strategy taken by the subject 101 from among three strategies: a hip joint strategy, an ankle joint strategy, and a heel raise strategy. For example, when the heels of the subject 101 are off the ground, the strategy identifying unit 11e identifies the strategy of the subject 101 as a heel-up strategy based on the skeletal information of the subject 101 in the end frame. The strategy identifying unit 11e also calculates the ankle angle of the subject 101 in the end frame, and if the ankle angle is 90 degrees or less, it is a hip joint strategy, and if the ankle angle exceeds 90 degrees, it is an ankle joint strategy. .

The measurement result notification unit 11f performs a process of notifying the terminal device 3 of the measurement results of the functional reach test. In the present embodiment, the measurement result notification unit 11f transmits information including the reach distance calculated by the distance calculation unit 11d and the movement strategy specified by the strategy identification unit 11e to the terminal device 3 as a measurement result.

FIG. 4 is a block diagram showing the configuration of the terminal device 3 according to this embodiment. The terminal device 3 according to the present embodiment includes a processing section 31, a storage section 32, a communication section (transceiver) 33, a display section 34, an operation section 35, a camera 36, etc. . The terminal device 3 is a device used by a user 102 who cares for the subject 101, for example, and may be configured using an information processing device such as a smartphone, a tablet terminal device, or a personal computer. Note that in the information processing system according to the present embodiment, the terminal device 3 is configured to be equipped with the camera 36 and capable of photographing the subject 101, but the present invention is not limited to this. The information processing system may have a configuration in which the user 102 takes a picture with a photographing device such as a video camera that is different from the terminal device 3, and the terminal device 3 acquires a moving image from the photographing device and sends it to the server device 1. good.

The processing unit 31 is configured using an arithmetic processing unit such as a CPU or an MPU, a ROM, and the like. The processing unit 31 reads and executes the program 32a stored in the storage unit 32 to perform various processes such as processing related to capturing a moving image and processing for displaying measurement results of a functional reach test.

The storage unit 32 is configured using, for example, a nonvolatile memory element such as a flash memory, or a storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data necessary for processing by the processing unit 31. In this embodiment, the storage unit 32 stores a program 32a executed by the processing unit 31. In this embodiment, the program 32a is distributed by a remote server device or the like, and the terminal device 3 acquires it through communication and stores it in the storage unit 32. However, the program 32a may be written into the storage unit 32, for example, during the manufacturing stage of the terminal device 3. For example, the program 32a may be stored in the storage unit 32 by reading the program 32a recorded on a recording medium 98 such as a memory card or an optical disk by the terminal device 3. For example, the program 32a may be recorded on the recording medium 98 and read by a writing device and written into the storage unit 32 of the terminal device 3. The program 32a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 98.

The communication unit 33 communicates with various devices via a network N including a mobile phone communication network, wireless LAN, the Internet, and the like. In this embodiment, the communication unit 33 communicates with the server device 1 via the network N. The communication unit 33 transmits data provided from the processing unit 31 to other devices, and also provides data received from other devices to the processing unit 31.

The display section 34 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing section 31. The operation unit 35 accepts user operations and notifies the processing unit 31 of the accepted operations. For example, the operation unit 35 receives user operations using input devices such as mechanical buttons or a touch panel provided on the surface of the display unit 34 . Further, for example, the operation unit 35 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be detachable from the terminal device 3.

The camera 36 is configured using an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera 36 is provided in the housing of the terminal device 3 or the like, and provides the processing unit 31 with an image taken by an image sensor.

Further, in the terminal device 3 according to the present embodiment, the processing unit 31 reads out and executes the program 32a stored in the storage unit 32, so that the shooting processing unit 31a, the moving image transmission processing unit 31b, the display processing unit 31c, etc. is implemented in the processing section 31 as a software-like functional section. Note that the program 32a may be a program dedicated to the information processing system according to this embodiment, or may be a general-purpose program such as an Internet browser or a web browser.

The photographing processing unit 31a photographs a moving image by controlling the operation of the camera 36. The photographing processing unit 31a receives or automatically determines settings such as the resolution and frame rate of a moving image to be photographed from the user 102, and performs photographing according to these settings. The photographing processing section 31a stores the data of the moving image obtained by photographing with the camera 36 in the storage section 32.

The moving image transmission processing unit 31b performs a process of transmitting data of a moving image photographed of the subject 101 performing a functional reach test movement to the server device 1, and requesting measurement of reach distance. The moving image transmission processing unit 31b displays a screen for uploading moving image data to the server device 1, for example, and selects a moving image to be uploaded from among one or more moving images stored in the storage unit 32. A selection operation is accepted from the user 102. At this time, the moving image transmission processing unit 31b receives input of information such as the name, height, age, etc. of the subject 101. The moving image transmission processing unit 31b associates the selected moving image with the input information regarding the target person 101 and transmits it to the server device 1.

The display processing unit 31c performs processing to display various characters, images, etc. on the display unit 34. In this embodiment, the display processing unit 31c displays the measurement results of the functional reach test obtained from the server device 1. For example, the display processing unit 31c displays information such as the reach distance obtained as a measurement result of the functional reach test and the movement strategy of the subject 101 regarding the functional reach test. At this time, the display processing unit 31c may display the above-mentioned start frame and end frame, which are the basis for calculating the reach distance, together with the measurement results. Furthermore, the display processing section 31c displays a screen on the display section 34 for accepting the selection of a moving image to be uploaded by the moving image transmission processing section 31b. The display processing unit 31c may perform various screen displays other than these.

<Functional reach test support processing>
FIG. 5 is a flowchart showing the procedure of processing performed by the server device 1 according to the present embodiment. The moving image acquisition unit 11a of the processing unit 11 of the server device 1 according to the present embodiment performs a functional reach test by the subject 101 by communicating with the terminal device 3 of the user 102 through the communication unit 13. Data of a moving image of the action is acquired (step S1). The skeleton extracting unit 11b of the processing unit 11 performs a process of extracting the skeleton of the subject 101 captured in the moving image based on the moving image acquired in step S1 (step S2).

FIG. 6 is a schematic diagram showing an example of skeleton extraction. The upper part of FIG. 6 shows an image immediately before the subject 101 captured in the video starts performing the functional reach test. The lower part of FIG. 6 simply shows the results of skeleton extraction performed on the image of the subject 101 in the upper part. The skeletal extraction unit 11b according to the present embodiment extracts skeletal information of the subject 101 from each frame constituting a moving image using a technique called OpenPose that extracts skeletal information using a learning model that has undergone deep learning, for example. Extract. However, the skeleton extraction unit 11b may perform skeleton extraction using a technique different from OpenPose. The skeletal information of the subject 101 obtained by the skeletal extraction process of the skeletal extraction unit 11b is information representing, for example, the head, waist, joints of limbs, etc. of the human body as a plurality of points. Note that skeletal information is often represented by about 10 to 20 points, but the skeletal information in the lower row of Figure 6 is represented by five points: finger, shoulder, hip (or crotch), ankle, and toe of subject 101. The head is shown by a circular symbol.

After performing the skeletal information extraction process in step S2, the frame specifying unit 11c of the processing unit 11 uses the skeletal information extracted from each frame making up the moving image to select a A process of specifying a start frame corresponding to the start point of the functional reach test operation is performed (step S3). Here, the frame identification unit 11c determines for each frame whether or not the posture of the subject 101 is the starting posture of the functional reach test, that is, the posture with the arms extended substantially horizontally, based on the skeletal information extracted from each frame. The frame in which the subject 101 in the starting posture is photographed is determined to be the starting frame candidate.

Note that the frame specifying unit 11c can determine whether the posture of the subject 101 is the starting posture based on either of the following two conditions.
(1) The height of the hand (fingers) of the subject 101 is approximately the same as the height of the shoulder, for example, the difference between the height of the hand and the height of the shoulder is within a predetermined range.
(2) The angle between the hand (arm) and the body of the subject 101 is approximately a right angle, for example, the angle between the hand and the body is within the range of 90 degrees ± a predetermined angle.

The frame identifying unit 11c also tracks the position of a predetermined part of the subject 101, for example, a hand, in each frame of the moving image based on the frame that is a candidate for the start frame. The frame identifying unit 11c detects an action in which the position of the hand is monotonically displaced in one horizontal direction by a distance of a predetermined number of pixels or more, and then monotonously displaced in the opposite direction by a distance of a predetermined number of pixels or more. Detect from. When this series of motions, that is, the motion of the functional reach test, is detected, the frame specifying unit 11c selects one frame from the candidates in which the displacement of the hand position has started in this series of motions, and selects it. The frame can be set as the starting frame.

Next, the frame specifying unit 11c specifies the frame in which the subject 101 extends the hand farthest in the functional reach test movement as the end frame (step S4). Here, the frame specifying unit 11c selects one frame in which the position of the hand of the subject 101 is farthest from the position of the hand of the start frame in the series of motions detected when specifying the start frame. Then, the selected frame can be set as the end frame.

Next, the distance calculation unit 11d of the processing unit 11 calculates the reach distance of the subject 101 in the functional reach test based on the start frame specified in step S3 and the end frame specified in step S4 (step S5). . FIG. 7 is a schematic diagram for explaining calculation of reach distance. The left side of FIG. 7 shows the skeletal information of the subject 101 in the start frame, and the right side of FIG. 7 shows the skeletal information of the subject 101 in the end frame. The distance calculation unit 11d calculates the horizontal distance between the finger and toe of the subject 101 as a first distance X1 based on the skeleton information of the start frame. Similarly, the distance calculation unit 11d calculates the horizontal distance between the finger and toe of the subject 101 as the second distance X2 based on the skeletal information of the end frame. The distance calculation unit 11d calculates the reach distance of the target person 101 by performing calculation based on the following equation (1).

(Reach distance) = (Second distance X2) - (First distance X1) ... (1)

In this embodiment, the distance calculation unit 11d calculates the reach distance using the position of the toe of the subject 101 as a reference. As a result, it is expected that the reach distance can be calculated while reducing the effects of camera shake, etc. that occur when the user 102 shoots a moving image, for example. Note that the distance calculation unit 11d may calculate the reach distance based on a position different from the position of the toe of the subject 101. However, if a position that may change during a functional reach test, such as the position of the heel of subject 101, is used as a reference, the accuracy of the calculated reach distance may decrease. It is preferable to use as a reference a position that is unlikely to change in the reach test operation, or whose position will change by a small amount even if it changes. The reference position may be the position of an object other than the body of the subject 101, for example, an object existing in the background. Alternatively, for example, if the required accuracy is obtained, a position that may change during the movement of the functional reach test, such as the position of the heel of the subject 101, may be used as a reference.

Next, the distance calculation unit 11d corrects the reach distance calculated in step S5 (step S6). For example, in a functional reach test, if the subject 101 rotates his shoulders and extends only one arm forward, the reach distance will be longer than if he extends both arms without rotating his shoulders. In this embodiment, the distance calculation unit 11d corrects the reach distance in order to reduce errors caused by the presence or absence of shoulder rotation. FIG. 8 is a schematic diagram for explaining reach distance correction. The distance calculation unit 11d calculates the distance between the left shoulder and the right shoulder of the subject 101 in the horizontal direction as an error distance Y1 based on the skeleton information of the end frame. Similarly, the distance calculation unit 11d calculates distances in the horizontal direction between the left hand tip and the right hand tip of the subject 101 as an error distance Y2 based on the skeleton information of the end frame. Note that the distance calculation unit 11d only needs to calculate at least one of the error distances Y1 and Y2. The distance calculation unit 11d corrects the reach distance of the target person 101 by performing calculation based on the following equation (2). Note that in equation (2), the error distance Y is either one of the above-mentioned error distances Y1 and Y2, but it may be, for example, the average value, maximum value, or minimum value of the error distances Y1 and Y2.

(Reach distance after correction) = (Reach distance before correction) - (Error distance Y) ÷ 2 (2)

Note that these first distance X1, second distance X2, reach distance, error distance Y, etc. are calculated as the number of pixels in a frame of a moving image. The distance calculation unit 11d calculates the actual reach distance by multiplying the reach distance calculated and corrected using the above equations (1) and (2) by the length per pixel. Note that in this embodiment, the distance calculation unit 11d calculates the length per pixel based on the height of the subject 101 stored in the subject information storage unit 12b.

Next, the strategy identifying unit 11e of the processing unit 11 identifies the movement strategy of the subject 101 in the functional reach test based on the skeleton information of the end frame (step S7). 9 and 10 are schematic diagrams for explaining the operation strategy. An example of a hip joint strategy is shown in the upper part of FIG. The hip joint strategy is a strategy that primarily involves bending the hip joint to perform functional reach test movements. The strategy specifying unit 11e can specify that the strategy of the subject 101 is the hip joint strategy when the angle α of the ankle of the subject 101 is 90 degrees or less. Note that the angle α of the ankle of the subject 101 can be, for example, the angle formed by a line segment connecting the tip of the foot and the heel and a line segment connecting the heel and the crotch (hip). Further, the lower part of FIG. 9 shows an example of an ankle joint strategy. Ankle joint strategy is a strategy for performing functional reach test movements by bending the ankle and hip joint. The strategy specifying unit 11e can specify that the strategy of the subject 101 is the ankle joint strategy when the angle α of the ankle of the subject 101 exceeds 90 degrees. Also shown in FIG. 10 is an example of a heel raise strategy. The heel raise strategy is a strategy in which the subject 101 raises his heel off the ground and performs a functional reach test motion. The strategy identifying unit 11e determines that the strategy of the target person 101 is a heel-up strategy when the heel of the target person 101 is off the ground or the position of the heel is higher than the position of the toe in the skeletal information of the end frame. It can be specified that

In the present embodiment, the strategy identifying unit 11e first determines whether the target person's 101's strategy is a heel-raising strategy or another strategy, depending on the heel position of the target person's 101's skeletal information in the end frame. to decide. If it is determined that the strategy of the target person 101 is not a heel raising strategy, the strategy identifying unit 11e calculates the angle α of the target person's ankle, and based on whether this angle α exceeds 90 degrees, Determine whether it is a strategy or a hip strategy. However, this order of determination is just an example, and is not limited to this.

Next, the measurement result notifying unit 11f of the processing unit 11 transmits the measurement results including information on the reach distance calculated and corrected in steps S5 and S6 and the movement strategy specified in step S7 to the video acquired in step S1. The image is transmitted to the terminal device 3 that is the source of the image (step S8). Furthermore, the processing unit 11 stores these measurement results in the subject information storage unit 12b in association with, for example, the identification information of the subject 101 (step S9), and ends the process.

FIG. 11 is a flowchart showing the procedure of processing performed by the terminal device 3 according to the present embodiment. The photographing processing section 31a of the processing section 31 of the terminal device 3 according to the present embodiment performs a video photographing process based on the operation of the user 102, for example (step S21). The photographing processing section 31a stores the moving image obtained by photographing in the storage section 32 (step S22). When the user 102 photographs the subject 101 performing the functional reach test, a moving image for measuring the reach distance of the subject 101 is stored in the storage unit 32 of the terminal device 3 .

Next, the display processing section 31c of the processing section 31 displays a video upload screen for measuring the reach distance on the display section 34, for example, in response to a menu operation by the user 102, etc. (Step S23). Although not shown, a list of moving images stored in the storage unit 32 is displayed on the upload screen, and the moving image transmission processing unit 31b of the processing unit 31 selects the moving images to be uploaded from among the moving images displayed in the list on this screen. A selection of a moving image is received from the user 102 (step S24). The moving image transmission processing unit 31b transmits the moving image whose selection was accepted in step S24 to the server device 1 (step S25).

The display processing unit 31c determines whether or not the reach distance measurement result transmitted from the server device 1 as a response to the video transmitted in step S25 has been received (step S26). If the measurement results have not been received (S26: NO), the display processing unit 31c waits until the measurement results are received from the server device 1. If the measurement result is received (S26: YES), the display processing unit 31c displays the received measurement result on the display unit 34 (step S27), and ends the process.

FIG. 12 is a schematic diagram showing an example of a display of measurement results by the terminal device 3. As shown in FIG. Based on the measurement results received from the server device 1, the terminal device 3 according to the present embodiment displays the illustrated measurement result notification screen on the display unit 34 to notify the user 102 of the measurement results of the functional reach test. . In the illustrated measurement result notification screen, the terminal device 3 displays a title string of "Functional Reach Test" at the top of the screen, and below this title string two images corresponding to the start frame and end frame. Display them side by side.

In order to display these two images, for example, the server device 1 extracts and measures images corresponding to the start frame specified in step S3 and the end frame specified in step S4 of the flowchart shown in FIG. 5 from the moving image. The terminal device 3 is notified along with the result. Alternatively, the server device 1 notifies the terminal device 3 of the frames of the video image that the specified start frame and end frame are, and the terminal device 3 converts the video image into the start frame and end frame based on this notification. A corresponding image may be extracted and displayed.

Furthermore, on the measurement result notification screen, the terminal device 3 displays a start position marker such as a vertical line indicating the position of the tip of the hand raised by the subject 101, superimposed on the above-described start frame image. In addition, the terminal device 3 displays, on the image of the end frame described above, the same start position marker as the one displayed superimposed on the start frame, and an end point such as a vertical line indicating the position of the hand extended by the subject 101 in the end frame. Display overlapping the position marker. In order to display these markers, for example, the server device 1 transmits information on the position of the hand obtained in skeleton extraction to the terminal device 3 together with the measurement results.

On the measurement result notification screen, the terminal device 3 displays text information such as the measurement result below two images corresponding to the start frame and the end frame. In this example, the terminal device 3 displays information such as the name, age, and height of the target person 101, and the reach distance and movement strategy based on the measurement results of the server device 1 as character string information. Note that the information such as the name, age, and height of the target person 101 displayed by the target person 101 may be, for example, information input by the user 102 when uploading a video image, information stored in the target person information storage unit 12b of the server device 1, Alternatively, it may be appropriately acquired from information stored in the terminal device 3.

<Summary>
In the information processing system according to the present embodiment configured as above, the server device 1 acquires a moving image of the subject 101 performing a predetermined movement related to a functional reach test from the terminal device 3, and from the acquired moving image. A start frame for starting a predetermined action and an end frame for ending the predetermined action are specified, and a reach in which a predetermined part of the subject's 101's hand or the like is moved by the predetermined action of the functional reach test based on the specified start frame and end frame. Calculate distance. As a result, in the information processing system according to the present embodiment, it is not necessary for the user 102 to measure the movement distance of the target person's 101's hand using a measuring instrument such as a measuring tape or a ruler for the functional reach test. 102 can obtain the reach distance by photographing the movements of the subject 101, so it can be expected to support implementation of functional reach tests.

Further, in the information processing system according to the present embodiment, the server device 1 extracts skeletal information of the target person 101 from the moving image using an appropriate method such as OpenPose, and calculates the reach distance based on the extracted skeletal information. As a result, the information processing system can be expected to accurately measure the reach distance of the target person 101 based on the two-dimensional moving image.

Further, in the information processing system according to the present embodiment, the server device 1 detects an action in which the position of the hand of the subject 101 monotonically displaces to one side in the horizontal direction in the moving image, and then monotonously displaces to the other side, The frame corresponding to the start point of displacement in one direction in this series of operations is specified as the start frame. As a result, the information processing system can be expected to accurately detect a predetermined motion related to a functional reach test from a moving image and to accurately identify a start frame.

Further, in the information processing system according to the present embodiment, the server device 1 detects that the height of the hand of the subject 101 is within a predetermined range with respect to the height of the shoulder (the height of the hand and the height of the shoulder are approximately the same). Or, a frame in which the angle formed by the hand and body of the subject 101 is within the range of 90 degrees ± a predetermined angle (the angle formed by the hand and body is approximately a right angle) is selected as a candidate, and the above is performed using the candidate frame as a reference. Detect a series of actions. This allows the information processing system to detect functional reach test movements based on the frame that captures the initial posture of the functional reach test movement, which is expected to measure reach distance with greater accuracy. can.

Further, in the information processing system according to the present embodiment, the server device 1 specifies the frame in which the amount of displacement of the hand position of the subject 101 from the start frame in the above series of operations is the maximum as the end frame. Thereby, the information processing system can be expected to identify the position where the subject 101 reaches forward the most as the end frame, and accurately measure the reach distance based on the start frame and the end frame.

Further, in the information processing system according to the present embodiment, the server device 1 calculates the distance that the position of the hand of the subject 101 has moved in the horizontal direction with respect to the position of the toe of the subject 101 as a reference. The information processing system according to the present embodiment uses the position of the toe of the subject 101 as a reference, so that even if, for example, camera shake occurs when the user 102 takes a picture, this It is expected that the reach distance can be measured accurately by reducing the influence of blur.

Further, in the information processing system according to the present embodiment, the server device 1 corrects the calculated reach distance according to the difference between the positions of the left and right shoulders or the difference between the positions of the left and right hands of the subject 101 in the end frame. As a result, the information processing system can suppress differences in reach distance depending on whether or not the subject 101 rotates his/her shoulders when performing a movement in a functional reach test, for example, and measure the reach distance with higher accuracy. We can expect that.

Further, in the information processing system according to the present embodiment, the server device 1 specifies the movement strategy of the subject 101 related to the movement of the functional reach test according to the angle of the ankle of the subject 101 in the end frame. The server device 1 can specify either a hip joint strategy or an ankle joint strategy depending on the angle of the ankle of the subject 101, for example. Furthermore, the server device 1 can specify any one of the hip joint strategy, ankle joint strategy, or heel raising strategy, depending on, for example, the angle of the ankle and the position of the heel of the subject 101 in the end frame. Thereby, the information processing system can provide the user 102 with information regarding the movement strategy of the target person 101 in addition to the reach distance of the target person 101, and can be expected to support implementation of the functional reach test.

Further, in the information processing system according to the present embodiment, the server device 1 notifies the predetermined terminal device 3 of information regarding the calculated reach distance and the identified movement strategy. Thereby, the information processing system can notify information such as the reach distance calculated by the server device 1 and the specified movement strategy to the terminal device 3 used by the user 102, and the user 102 uses his own terminal device 3. Since this information can be easily checked using the following methods, it is expected that it will support the implementation of functional reach tests.

Note that in this embodiment, the user 102 transmits a moving image from the terminal device 3 to the server device 1, and the server device 1 that receives the video performs processing such as calculating reach distance and specifying a movement strategy, and displays the processing results. Although the configuration is such that the information is transmitted to the terminal device 3, the present invention is not limited to this. A configuration may be adopted in which the terminal device 3 performs processing such as calculating reach distance and specifying a movement strategy based on a moving image, that is, a configuration in which the terminal device 3 performs the processing alone without using the server device 1.

The embodiments disclosed herein are illustrative in all respects and should not be considered restrictive. The scope of the present invention is indicated by the claims rather than the above-mentioned meaning, and is intended to include meanings equivalent to the claims and all changes within the scope.

Items described in each embodiment can be combined with each other. Moreover, the independent claims and dependent claims recited in the claims may be combined with each other in any and all combinations, regardless of the form in which they are cited. Further, although the scope of claims uses a format in which claims refer to two or more other claims (multi-claim format), the invention is not limited to this format. It may be written using a format that describes multiple claims (multi-multi-claims) that cite at least one multiple claim.

1 Server device (information processing device, computer)
3 terminal device 11 processing unit 11a moving image acquisition unit 11b skeleton extraction unit 11c frame identification unit 11d distance calculation unit 11e strategy identification unit 11f measurement result notification unit 12 storage unit 12a server program (computer program)
12b Subject information storage section 13 Communication section 31 Processing section 31a Photographing processing section 31b Moving image transmission processing section 31c Display processing section 32 Storage section 32a Program 33 Communication section 34 Display section 35 Operation section 36 Camera 98, 99 Recording medium 101 Subject 102 User N Network

Claims (12)

The information processing device
Obtain a video image of the subject performing the specified movements related to the functional reach test,
Based on a frame in which the height of the subject's hand is within a predetermined range relative to the shoulder height, or the angle formed by the subject's hand and body is within a range of 90 degrees ± a predetermined angle, Detecting from the video image an action in which the position of the subject's hand monotonically displaces in one direction in the horizontal direction and then monotonously displaces in the other direction;
identifying a frame corresponding to the start point of displacement in the one direction in the operation as a start frame for starting the predetermined operation;
identifying an end frame in which the predetermined operation ends from the video image;
calculating a distance traveled by a predetermined part of the subject due to the predetermined action based on the start frame and the end frame;
Information processing method. The information processing device
Obtain a video image of the subject performing the specified movements related to the functional reach test,
Identifying a start frame for starting the predetermined action and an end frame for ending the predetermined action from the video image;
Based on the start frame and the end frame, calculate the distance traveled by the predetermined part of the subject due to the predetermined action;
correcting the distance according to a difference in the position of the left and right shoulders of the subject in the end frame or a difference in the position of the left and right hands of the subject;
Information processing method. Extracting skeletal information of the subject from the video image,
calculating the distance based on the extracted skeleton information;
The information processing method according to claim 1 or claim 2 . identifying a frame in which the amount of displacement of the hand position from the start frame in the movement is the maximum as the end frame;
The information processing method according to claim 1 . Calculating the distance that the subject's hand position has moved in the horizontal direction with respect to the subject's toe position as a reference;
The information processing method according to claim 1 or claim 2 . identifying a movement strategy of the subject related to the predetermined movement according to the angle of the ankle of the subject in the end frame;
The information processing method according to claim 1 or claim 2 . Depending on the angle of the subject's ankle and the position of the subject's heel in the end frame, the subject's movement strategy related to the predetermined movement is a hip joint strategy, an ankle joint strategy, or a heel raise strategy. identify,
The information processing method according to claim 6 . Notifying information regarding the calculated distance and the identified movement strategy to a predetermined terminal device;
The information processing method according to claim 6 . to the computer,
Obtain a video image of the subject performing the specified movements related to the functional reach test,
Based on a frame in which the height of the subject's hand is within a predetermined range relative to the shoulder height, or the angle formed by the subject's hand and body is within a range of 90 degrees ± a predetermined angle, Detecting from the video image an action in which the position of the subject's hand monotonically displaces in one direction in the horizontal direction and then monotonously displaces in the other direction;
identifying a frame corresponding to the start point of displacement in the one direction in the operation as a start frame for starting the predetermined operation;
identifying an end frame in which the predetermined operation ends from the video image;
A computer program that causes a computer program to execute a process of calculating a distance traveled by a predetermined part of the subject due to the predetermined action based on the start frame and the end frame. to the computer,
Obtain a video image of the subject performing the specified movements related to the functional reach test,
Identifying a start frame for starting the predetermined action and an end frame for ending the predetermined action from the video image;
Based on the start frame and the end frame, calculate the distance traveled by the predetermined part of the subject due to the predetermined action;
The distance is corrected according to the difference in the positions of the left and right shoulders or the difference in the positions of the left and right hands of the subject in the end frame.
A computer program that executes a process. Equipped with a processing section,
The processing unit includes:
Obtain a video image of the subject performing the specified movements related to the functional reach test,
Based on a frame in which the height of the subject's hand is within a predetermined range relative to the shoulder height, or the angle formed by the subject's hand and body is within a range of 90 degrees ± a predetermined angle, Detecting from the video image an action in which the position of the subject's hand monotonically displaces in one direction in the horizontal direction and then monotonously displaces in the other direction;
identifying a frame corresponding to the start point of displacement in the one direction in the operation as a start frame for starting the predetermined operation;
identifying an end frame in which the predetermined operation ends from the video image;
calculating a distance traveled by a predetermined part of the subject due to the predetermined action based on the start frame and the end frame;
Information processing device. Equipped with a processing section,
The processing unit includes:
Obtain a video image of the subject performing the specified movements related to the functional reach test,
Identifying a start frame for starting the predetermined action and an end frame for ending the predetermined action from the video image;
Based on the start frame and the end frame, calculate the distance traveled by the predetermined part of the subject due to the predetermined action;
correcting the distance according to a difference in the position of the left and right shoulders of the subject in the end frame or a difference in the position of the left and right hands of the subject;
Information processing device.

Images