JP7454326B1 - Condition evaluation method, condition evaluation system and condition evaluation program - Google Patents

Abstract

[Problem] Evaluate the condition of a subject from a video of the subject standing and sitting. A state evaluation method according to an embodiment is a state evaluation method executed by a state evaluation system, and includes an acquisition step of obtaining a standing and sitting video of a subject standing and sitting from the side of the subject. and an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video, and an evaluation step of evaluating the condition of the subject based on the plurality of feature points. [Selection diagram] Figure 8

Description

The present invention relates to a condition evaluation method, a condition evaluation system, and a condition evaluation program.

A known method for evaluating the health status of elderly people is to repeatedly sit and stand. In this method, the health condition of an elderly person is generally evaluated by measuring the time it takes to stand up and sit down a predetermined number of times.

On the other hand, in recent years, technology has been developed that uses AI to analyze videos of a subject's movements and evaluate the quality of the movements and the condition of the subject. For example, Cited Document 1 discloses a technique for evaluating the quality of a subject's standing body forward bending based on a video of the subject's standing body bending.

JP2023-72656A

There is a desire to develop technology that can evaluate the health status of elderly people based on videos of them standing and sitting.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to evaluate the condition of a subject based on videos of the subject standing and sitting.

A condition evaluation method according to one embodiment is a condition evaluation method executed by a condition evaluation system, and includes an acquisition step of acquiring a standing and sitting video of a subject standing and sitting from the side of the subject; The method includes an extraction step of extracting a plurality of feature points of the subject from a sitting video, and an evaluation step of evaluating the condition of the subject based on the plurality of feature points.

According to one embodiment, the condition of the subject can be evaluated from a video of the subject standing and sitting.

1 is a diagram showing an example of the configuration of a condition evaluation system 1000. FIG. 1 is a diagram showing an example of a hardware configuration of a condition evaluation device 1. FIG. 2 is a diagram showing an example of a hardware configuration of a user terminal 2. FIG. 1 is a diagram showing an example of a functional configuration of a condition evaluation device 1. FIG. 2 is a diagram showing an example of a functional configuration of a user terminal 2. FIG. 3 is a flowchart illustrating an example of processing executed by the condition evaluation system 1000. It is a figure which shows the standing-and-sitting animation 121 typically. It is a figure which shows an example of the feature point p extracted from the standing-and-sitting video 121. 1 is a flowchart showing an example of a method for evaluating stability in a standing position. It is a figure explaining the inclination a of a body at the time of a standing position. 3 is a flowchart illustrating an example of a method for evaluating standing posture. It is a figure explaining the extension e of the hip joint in a standing position. It is a flowchart which shows an example of the evaluation method of the compression fracture risk when sitting. It is a figure showing an example of auxiliary tool A. It is a figure which shows an example of evaluation result display screen sc.

Hereinafter, each embodiment of the present invention will be described with reference to the accompanying drawings. Note that in the descriptions of the specifications and drawings related to each embodiment, the same reference numerals are given to the constituent elements having substantially the same functional configuration to omit redundant explanation.

<System configuration>
First, an overview of the condition evaluation system 1000 according to this embodiment will be explained. The condition evaluation system 1000 is an information processing system that evaluates the condition of a subject based on a video of the subject standing and sitting taken from the side of the subject.

The subject is a person whose condition is to be evaluated, for example, an elderly person.

Standing up and sitting down is a series of movements in which a subject sitting on a chair stands up from the chair and then sits down on the chair again.

The lateral side is the right side or left side of the subject sitting on the chair.

The condition of the subject includes the stability of the subject's standing position, the subject's muscular strength, the subject's standing posture, and the risk of compression fracture when the subject is seated. A compression fracture is a fracture of the spine caused by force applied from above and below. These conditions are effective as indicators for evaluating the subject's ADL (Activities of Daily Living).

FIG. 1 is a diagram showing an example of the configuration of a condition evaluation system 1000. As shown in FIG. 1, the condition evaluation system 1000 includes a condition evaluation device 1 and a user terminal 2, which are communicably connected to each other via a network N. The network N is, for example, a wired LAN (Local Area Network), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination thereof. In the example of FIG. 1, the condition evaluation system 1000 includes one condition evaluation device 1 and one user terminal 2, but may include a plurality of each.

The condition evaluation device 1 is an information processing device that evaluates the condition of a subject based on videos of the subject standing and sitting. The state evaluation device 1 is, for example, a PC (Personal Computer), a smartphone, a tablet terminal, a server device, or a microcomputer, but is not limited thereto. The condition evaluation device 1 will be described in detail later.

The user terminal 2 is an information processing device used by a user of the condition evaluation system 1000. In the following, it is assumed that a user shoots a standing video and obtains an evaluation result based on the standing video, but the user who obtains the evaluation result and the photographer who shoots the standing video are different. It can be a person. The user terminal 2 is, for example, a PC, a smartphone, or a tablet terminal, but is not limited thereto.

<Hardware configuration of condition evaluation device 1>
Next, the hardware configuration of the condition evaluation device 1 will be explained. FIG. 2 is a diagram showing an example of the hardware configuration of the condition evaluation device 1. As shown in FIG. 2, the state evaluation device 1 includes a processor 101, a memory 102, a storage 103, a communication I/F 104, an input device 105, and an output device 106, which are interconnected via a bus B1. and a drive device 107.

The processor 101 controls each configuration of the condition evaluation device 1 by expanding various programs including the OS (Operating System) and the condition evaluation program stored in the storage 103 into the memory 102 and executing them. Achieve function 1. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or a DSP (Digital Signal Processor), but is not limited thereto. .

The memory 102 is, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), or a combination thereof. The ROM is, for example, a PROM (Programmable ROM), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), or a combination thereof. The RAM is, for example, DRAM (Dynamic RAM) or SRAM (Static RAM), but is not limited thereto.

The storage 103 stores various programs and data including an OS and a status evaluation program. The storage 103 is, for example, a flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an SCM (Storage Class Memories), but is not limited thereto.

The communication I/F 104 is an interface for connecting the state evaluation device 1 to an external device via the network N and controlling communication. The communication I/F 104 is, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), or Ethernet (registered trademark), but is not limited thereto.

The input device 105 is a device for inputting information to the condition evaluation device 1. The input device 105 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a photographing device (camera), various sensors, or an operation button, but is not limited thereto.

The output device 106 is a device for outputting information from the condition evaluation device 1. The output device 106 is, for example, a display, a projector, a printer, a speaker, or a vibrator, but is not limited thereto.

The drive device 107 is a device that reads and writes data on the recording medium 108. The drive device 107 is, for example, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or an SD card reader, but is not limited thereto. The recording medium 108 is, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), an FD (Floppy Disc), an MO (Magneto-Optical disk), a BD (Blu-ray (registered trademark) Disc), or a USB (registered trademark). ) memory or SD card, but is not limited to this.

In this embodiment, the condition evaluation program may be written into the memory 102 or the storage 103 at the manufacturing stage of the condition evaluation device 1, or may be provided to the condition evaluation device 1 via the network N. The information may be provided to the condition evaluation device 1 via a non-transitory computer-readable recording medium such as the recording medium 108 .

<Hardware configuration of user terminal 2>
Next, the hardware configuration of the user terminal 2 will be explained. FIG. 3 is a diagram showing an example of the hardware configuration of the user terminal 2. As shown in FIG. As shown in FIG. 3, the user terminal 2 includes a processor 201, a memory 202, a storage 203, a communication I/F 204, an input device 205, and an output device 206, which are interconnected via a bus B2. , is provided.

The processor 201 controls each configuration of the user terminal 2 and realizes the functions of the user terminal 2 by expanding various programs including the OS and the status evaluation program stored in the storage 203 into the memory 202 and executing them. . The processor 201 is, for example, a CPU, MPU, GPU, ASIC, or DSP, but is not limited thereto.

Memory 202 is, for example, ROM, RAM, or a combination thereof. The ROM is, for example, a PROM, an EPROM, an EEPROM, or a combination thereof. RAM is, for example, DRAM or SRAM, but is not limited thereto.

The storage 203 stores various programs and data including an OS and a status evaluation program. The storage 203 is, for example, a flash memory, an HDD, an SSD, or an SCM, but is not limited thereto.

The communication I/F 204 is an interface for connecting the user terminal 2 to an external device via the network N and controlling communication. The communication I/F 204 is, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), or Ethernet (registered trademark), but is not limited thereto.

The input device 205 is a device for inputting information to the user terminal 2. The input device 205 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a photographing device (camera), various sensors, or an operation button, but is not limited thereto. The user terminal 2 includes a photographing device 205C as the input device 205.

The output device 206 is a device for outputting information from the user terminal 2. The output device 206 is, for example, a display, a projector, a printer, a speaker, or a vibrator, but is not limited thereto. The user terminal 2 includes a display device 206D as the output device 206.

The drive device 207 is a device that reads and writes data on the recording medium 208. The drive device 207 is, for example, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or an SD card reader, but is not limited thereto. The recording medium 208 is, for example, a CD, DVD, FD, MO, BD, USB (registered trademark) memory, or SD card, but is not limited thereto.

In addition, in this embodiment, the condition evaluation program may be written into the memory 202 or the storage 203 at the manufacturing stage of the user terminal 2, may be provided to the user terminal 2 via the network N, or may be provided to the user terminal 2 via the recording medium. The information may be provided to the user terminal 2 via a non-transitory computer-readable recording medium such as 208.

<Functional configuration of condition evaluation device 1>
Next, the functional configuration of the condition evaluation device 1 will be explained. FIG. 4 is a diagram showing an example of the functional configuration of the condition evaluation device 1. As shown in FIG. 4, the condition evaluation device 1 includes a communication section 11, a storage section 12, and a control section 13.

The communication unit 11 is realized by the communication I/F 104. The communication unit 11 transmits and receives information to and from the user terminal 2 via the network N. The communication unit 11 receives the standing/sitting video 121 from the user terminal 2 and transmits the evaluation result 123 to the user terminal 2.

The storage unit 12 is realized by a memory 102 and a storage 103. The storage unit 12 stores a standing and sitting video 121, an extracted model 122, and an evaluation result 123.

The standing/sitting video 121 is video data of a standing/sitting video of the subject photographed by a user (photographer). The standing and sitting video 121 is stored in association with subject information, user information, terminal information, photographing information, and the like.

The subject information is information regarding the subject whose standing and sitting video 121 was shot. The target person information includes, for example, the target person's name, age, gender, and date of birth, but is not limited thereto.

The user information is information regarding the user (photographer) who shot the standing and sitting video 121. The user information includes, for example, the user's name, age, gender, date of birth, affiliation, and position, but is not limited thereto.

The terminal information is information regarding the user terminal 2 that transmitted the standing/sitting video 121. The terminal information includes, for example, the model of the user terminal 2, the user, and the manufacturing date, but is not limited thereto.

The shooting information is information regarding shooting of the standing and sitting video 121. The photographing information includes, for example, the photographing date and time, the brightness at the time of photographing, the model of the photographing device 205C, the tilt of the photographing device 205C, and the zoom magnification, but is not limited thereto.

The extraction model 122 is a trained machine learning model called a skeleton estimation model, skeleton extraction model, skeleton detection model, posture estimation model, etc., which has been trained to be able to extract human feature points p from a video image. It is. As the extraction model 122, any machine learning model that can extract the feature points p of a person can be used. Feature points p include, but are not limited to, for example, toes, heels, knees, hips (buttocks), abdomen, chest, neck, shoulders, elbows, wrists, fingers, head, eyes, ears, mouth, and nose. .

The evaluation result 123 is information indicating the evaluation result of the subject's condition based on the standing and sitting video 121. The evaluation result 123 will be described in detail later.

The control unit 13 is realized by the processor 101 reading a program from the memory 102, executing it, and cooperating with other hardware configurations. The control unit 13 controls the entire operation of the condition evaluation device 1 . The control unit 13 includes an acquisition unit 131, an extraction unit 132, and an evaluation unit 133.

The acquisition unit 131 acquires the standing and sitting video 121 from the user terminal 2 and stores it in the storage unit 12 .

The extraction unit 132 uses the extraction model 122 to extract a plurality of feature points p of the subject from the standing and sitting video 121.

The evaluation unit 133 evaluates the condition of the subject based on the feature points p of the subject extracted by the extraction unit 132 and stores the evaluation result 123 in the storage unit 12.

<Functional configuration of condition evaluation device 1>
Next, the functional configuration of the condition evaluation device 1 will be explained. FIG. 4 is a diagram showing an example of the functional configuration of the condition evaluation device 1. As shown in FIG. 4, the condition evaluation device 1 includes a communication section 11, a storage section 12, and a control section 13.

The communication unit 11 is realized by the communication I/F 104. The communication unit 11 transmits and receives information to and from the user terminal 2 via the network N. The communication unit 11 receives the standing/sitting video 121 from the user terminal 2 and transmits the evaluation result 123 to the user terminal 2.

The storage unit 12 is realized by a memory 102 and a storage 103. The storage unit 12 stores a standing and sitting video 121, an extraction model 122, feature point information 123, and an evaluation result 124.

The standing/sitting video 121 is video data of a standing/sitting video of the subject acquired from the user terminal 2. The standing and sitting video 121 is stored in association with subject information, user information, terminal information, photographing information, and the like.

The subject information is information regarding the subject whose standing and sitting video 121 was shot. The target person information includes, for example, the target person's name, age, gender, and date of birth, but is not limited thereto.

The user information is information regarding the user (photographer) who shot the standing and sitting video 121. The user information includes, for example, the user's name, age, gender, date of birth, affiliation, and position, but is not limited thereto.

The terminal information is information regarding the user terminal 2 that transmitted the standing/sitting video 121. The terminal information includes, for example, the model of the user terminal 2, the user, and the manufacturing date, but is not limited thereto.

The shooting information is information regarding shooting of the standing and sitting video 121. The photographing information includes, for example, the photographing date and time, the brightness at the time of photographing, the model of the photographing device 205C, the tilt of the photographing device 205C, and the zoom magnification, but is not limited thereto.

The extraction model 122 is a trained machine learning model called a skeleton estimation model, skeleton extraction model, skeleton detection model, posture estimation model, etc., which has been trained to be able to extract human feature points from video images. be. When the extraction model 122 receives an image, it outputs feature point information of a person appearing in the image. The feature point information is information regarding the feature point p of a person, and includes information indicating the type and coordinates of the feature point p. When an image showing a person is input, feature point information of one or more feature points is output. As the extraction model 122, any machine learning model that can extract human feature points can be used. Feature points p include, but are not limited to, for example, toes, heels, knees, hips (buttocks), abdomen, chest, neck, shoulders, elbows, wrists, fingers, head, eyes, ears, mouth, and nose. .

The feature point information 123 is feature point information of the target person's feature point p extracted from the standing and sitting video 121. The feature point information 123 is stored in association with the standing and sitting video 121 from which the feature point p is extracted. The storage unit 12 stores time-series data of feature point information 123 for one standing and sitting video 121.

The evaluation result 124 is information indicating the evaluation result of the subject's condition based on the standing and sitting video 121. The evaluation result 124 will be described in detail later.

The control unit 13 is realized by the processor 101 reading a program from the memory 102, executing it, and cooperating with other hardware configurations. The control unit 13 controls the entire operation of the condition evaluation device 1 . The control unit 13 includes an acquisition unit 131, an extraction unit 132, and an evaluation unit 133.

The acquisition unit 131 acquires the standing and sitting video 121 from the user terminal 2 and stores it in the storage unit 12 .

The extraction unit 132 uses the extraction model 122 to extract a plurality of feature points p of the subject from the standing and sitting video 121. Specifically, the extraction unit 132 inputs the images forming the standing and sitting video 121 to the extraction model 122, acquires the feature point information 123 output by the extraction model 122, and stores it in the storage unit 12. The extraction unit 132 may extract the feature points p from all the images making up the standing and sitting video 121, or may extract the feature points p from images for each predetermined frame. In any case, the storage unit 12 stores time-series data of the feature point information 123.

The evaluation unit 133 evaluates the condition of the subject based on the feature points p of the subject extracted by the extraction unit 132 and stores the evaluation result 123 in the storage unit 12.

Note that the functional configuration of the condition evaluation device 1 is not limited to the above example. For example, the condition evaluation device 1 may include a part of the above functional configuration, and the user terminal 2 may include the rest. Moreover, the condition evaluation device 1 may be provided with functional configurations other than those described above. Further, each functional configuration of the condition evaluation device 1 may be realized by software as described above, or by hardware such as an IC chip, SoC (System on Chip), LSI (Large Scale Integration), or microcomputer. may be done.

<Functional configuration of user terminal 2>
Next, the functional configuration of the user terminal 2 will be explained. FIG. 5 is a diagram showing an example of the functional configuration of the user terminal 2. As shown in FIG. As shown in FIG. 5, the user terminal 2 includes a communication section 21, a storage section 22, and a control section 23.

The communication unit 21 is realized by the communication I/F 204. The communication unit 21 transmits and receives information to and from the condition evaluation device 1 via the network N. The communication unit 21 transmits the standing/sitting video 221 to the condition evaluation device 1 and receives the evaluation result 222 from the condition evaluation device 1 .

The storage unit 22 is realized by a memory 202 and a storage 203. The storage unit 22 stores a standing/sitting video 221 and an evaluation result 222.

The standing/sitting video 221 is video data of a standing/sitting video of the subject transmitted from the user terminal 2 to the condition evaluation device 1 . The standing/sitting video 221 may be a video captured by the user using the imaging device 205C of the user terminal 2, or may be a video captured via the network N by a imaging device other than the user terminal 2. . The standing and sitting video 221 is stored in association with subject information, user information, terminal information, photographing information, and the like.

The subject information is information regarding the subject whose standing and sitting video 221 was shot. The target person information includes, for example, the target person's name, age, gender, and date of birth, but is not limited thereto.

The user information is information regarding the user (photographer) who shot the standing and sitting video 221. The user information includes, for example, the user's name, age, gender, date of birth, affiliation, and position, but is not limited thereto.

The terminal information is information regarding the user terminal 2 that transmitted the standing/sitting video 221. The terminal information includes, for example, the model of the user terminal 2, the user, and the manufacturing date, but is not limited thereto.

The shooting information is information regarding shooting of the standing and sitting video 221. The photographing information includes, for example, the photographing date and time, the brightness at the time of photographing, the model of the photographing device 205C, the tilt of the photographing device 205C, and the zoom magnification, but is not limited thereto.

The evaluation result 222 is information indicating the evaluation result of the subject's condition based on the standing/sitting video 221 obtained from the condition evaluation device 1 . The evaluation result 222 will be described in detail later.

The control unit 23 is realized by the processor 201 reading a program from the memory 202, executing it, and cooperating with other hardware configurations. The control unit 23 controls the entire operation of the user terminal 2. The control unit 23 includes an acquisition unit 231, a shooting control unit 232, and a display control unit 233.

The acquisition unit 231 acquires the evaluation result 222 from the condition evaluation device 1 and stores it in the storage unit 22.

The photographing control unit 232 controls the photographing of the standing and sitting video 221 by the photographing device 205C.

The display control unit 233 controls the screen displayed on the display device 206D.

Note that the functional configuration of the user terminal 2 is not limited to the above example. For example, the user terminal 2 may include a part of the above functional configuration, and the state evaluation device 1 may include the rest. Further, the user terminal 2 may have a functional configuration other than the above. Further, each functional configuration of the user terminal 2 may be realized by software as described above, or may be realized by hardware such as an IC chip, SoC, LSI, or microcomputer.

<Processing executed by condition evaluation system 1000>
Next, the processing executed by the condition evaluation system 1000 will be explained. FIG. 6 is a flowchart illustrating an example of processing executed by the condition evaluation system 1000. In the following, a case where the user uses the user terminal 2 to shoot a standing video 221 will be described as an example.

(Step S101)
First, a user shoots a standing and sitting video 221 of a subject (step S101). Specifically, in response to a user's operation, the photographing control unit 232 controls the photographing device 205C to photograph the subject standing and sitting, and uses the obtained standing and sitting video 221 as subject information, user information, and terminal information. , photographing information, etc., and are stored in the storage unit 22. Here, it is assumed that the subject repeatedly stands and sits as many times as possible, with an upper limit of 50 times. Note that the upper limit number of times can be set arbitrarily. Further, the number of times of standing and sitting may be determined in advance.

(Step S102)
Next, the user terminal 2 transmits the standing/sitting video 221 stored in the storage unit 22 to the condition evaluation device 1 in association with subject information, user information, terminal information, photographing information, etc. (step S102).

(Step S103)
The acquisition unit 131 of the condition evaluation device 1 acquires the standing and sitting video 221 transmitted by the user terminal 2, and stores it in the storage unit 12 as the standing and sitting video 121 in association with subject information, user information, terminal information, shooting information, etc. Save to.

FIG. 7 is a diagram schematically showing the standing/sitting video 121. FIG. 7 shows images im1 to im5 that constitute a portion of the sit-to-stand video 121 that corresponds to one sit-to-stand (a portion in which the subject stands up from a sitting position and sits down again). Image im1 is a state where the subject is sitting, image im2 is a state where the subject is about to stand up, image im3 is a state where the subject is standing up, image im4 is a state where the subject is about to sit down, and image im5 is a state where the subject is sitting down again. This is a photograph of the situation.

(Step S104)
When the standing and sitting video 121 is stored in the storage unit 12, the extraction unit 132 uses the extraction model 122 to extract a plurality of feature points p of the subject from the standing and sitting video 121, and stores the feature point information 123. 22 (step S104).

FIG. 8 is a diagram showing an example of feature points p extracted from the standing and sitting video 121. In FIG. 8, feature points p1 to p8 are displayed superimposed on images im1 to im5 of FIG. 7. The feature points p1 to p8 are feature points p of the heel, knee, waist, wrist, elbow, shoulder, neck, and head, respectively. As shown in FIG. 8, feature points p1 to p8 are extracted from images im1 to im5, respectively. Hereinafter, the feature points p1 to p8 will be referred to as a heel p1, a knee p2, a waist p3, a wrist p4, an elbow p5, a shoulder p6, a neck p7, and a head p8, respectively. Note that, as described above, the type and coordinates of the extracted feature point p are not limited to the example shown in FIG. 8 .

(Step S105)
The evaluation unit 133 evaluates the stability of the subject's standing position based on the feature point information 123 stored in the storage unit 12 (step S105). More specifically, the evaluation unit 133 calculates the evaluation value v1 of the stability of the standing position based on the body inclination a during the standing position of the subject in multiple standing and sitting positions.

FIG. 9 is a flowchart illustrating an example of a method for evaluating standing stability by the evaluation unit 133. The process in FIG. 9 corresponds to the internal process in step S105.

(Step S201)
First, the evaluation unit 133 detects the standing position of the subject from the standing/sitting video 121 (step S201). Hereinafter, three detection methods will be described as methods for detecting standing position. The evaluation unit 133 may use any evaluation method.

(1) First detection method When a person stands up or sits down, the part of the body above the knees is at the highest position when standing (see FIG. 8). For this reason, for example, if the subject stands and sits x times, the height of the feature point p above the knees (vertical coordinates of the feature point) will be the maximum value of x times at approximately the same height. (peak) appears. Therefore, the evaluation unit 133 detects each time point at which the height of the feature point p above the knee becomes the maximum as the time of standing. The feature point p above the knee is, for example, the abdomen, chest, neck, shoulder, elbow, wrist, finger, head, eyes, ear, mouth, or nose, but is not limited thereto.

(2) Second detection method When the user stands up and sits down, the angle of the knees is greatest in the standing position (see FIG. 8). Therefore, for example, if a subject stands and sits x times, the angle of the knees (the angle between the straight line connecting heel p1 and knee p2 and the straight line connecting knee p2 and hip p3) will be approximately the same. x maximum values (peaks) of magnitude appear. Therefore, the evaluation unit 133 detects each time point at which the knee angle reaches its maximum value as a standing position.

(3) Third detection method When the user stands up and sits down, the body inclination a becomes the smallest in the standing position (see FIG. 8). The body inclination a here refers to the inclination of the body axis with respect to the vertical axis. For this reason, for example, when the subject stands and sits x times, the minimum value (bottom) of the x times of approximately the same magnitude appears in the body inclination a. Therefore, the evaluation unit 133 detects each time point when the body inclination a becomes the minimum value as the time of standing.

(Step S202)
Next, the evaluation unit 133 calculates the body inclination a in the standing position detected in step S201 (step S202). For example, if the subject stands up and sits down x times, body inclinations a1 to ax during standing up x times are calculated.

FIG. 10 is a diagram illustrating the body inclination a in a standing position using the image im3 of FIG. As shown in FIG. 10, the inclination a of the body when standing is, for example, the angle θ1 between the vertical line L1 and the line L2 connecting the heel p1 and the neck p7. The line L2 may be a line connecting the heel p1 and the shoulder p8.

(Step S203)
Subsequently, the evaluation unit 133 calculates the evaluation value v1 of the stability of the standing position of the subject based on the x times of body inclinations a1 to ax during standing that were calculated in step S202 (step S203). . Generally, it is considered that the smaller the inclination a of the body when standing (the angle θ1 is closer to 0), the more stable the standing position is. Therefore, for example, the evaluation unit 133 evaluates the stability of the standing position higher as the representative value ar of the body inclinations a1 to ax during standing is smaller (the absolute value of the angle θ1 is smaller). If the evaluation value v1 is higher as the stability of the standing position is higher, the evaluation unit 133 determines that the smaller the representative value ar of the body inclinations a1 to ax during standing is (the smaller the absolute value of the angle θ1), the more stable the standing position is. The evaluation value v1 of gender is calculated to be high. The representative value ar is, for example, the minimum value, average value, median value, or average value of the top X% of the slopes a1 to ax, but is not limited thereto. Note that the evaluation value v1 may be a multilevel rank obtained by comparing the representative value ar and a preset threshold value.

Through the above processing, the stability of the subject's standing position is evaluated from the standing/sitting video 121, and the evaluation value v1 is calculated. The evaluation unit 133 stores the evaluation value v1 in the storage unit 12 as the evaluation result 124.

(Step S106)
Now, return to FIG. 6. The evaluation unit 133 evaluates the subject's muscle strength based on the feature point information 123 stored in the storage unit 12 (step S106). More specifically, the evaluation unit 133 calculates the evaluation value v2 of the subject's muscle strength based on the number of times or time the subject has stood up and sat down. Since the number of times of standing and sitting is the same as the number of times of standing, it may be measured by any of the above-mentioned methods of detecting standing. Two methods for evaluating muscle strength will be described below. The evaluation unit 133 may use any evaluation method.

(1) First evaluation method When a subject repeatedly stands and sits as many times as possible with a maximum of X times, it is generally considered that the more the subject stands and sits, the higher the subject's muscle strength is . Therefore, the evaluation unit 133 evaluates the subject's muscle strength higher as the number of times the subject stands and sits increases. If the evaluation value v2 is larger as the subject's muscle strength is higher, the evaluation unit 133 calculates the evaluation value v2 to be larger as the number of times x that the subject stands up and sits increases. Note that the evaluation value v2 may be the number of times x, or may be a multilevel rank obtained by comparing the number of times x with a preset threshold.
(2) Second evaluation method When a subject repeatedly stands and sits a predetermined number of times . Therefore, the evaluation unit 133 evaluates the subject's muscle strength higher as the time T required for the subject to stand and sit a predetermined number of times X is shorter. If the evaluation value v2 is larger as the subject's muscle strength is higher, the evaluation unit 133 calculates the evaluation value v2 to be larger as the required time T required for the subject to stand up and sit down a predetermined number of times X is shorter. Note that the evaluation value v2 may be the time T, or may be a multilevel rank obtained by comparing the required time T and a preset threshold value.

Here, the required time T is the time required from the sitting position before starting the first sitting/standing position to the sitting position after the end of the Xth sitting/standing position. The evaluation unit 133 detects the sitting position before the first standing/sitting start and the sitting position after the X-th standing/sitting, and calculates the time between the two as the required time T.

Hereinafter, three detection methods will be described as methods for detecting sitting positions. The evaluation unit 133 may use any detection method.

(1) First detection method When a person stands or sits, the body part above the knees is at the lowest position when sitting (see FIG. 8). For this reason, for example, if a subject stands and sits X times, the height of the feature point p above the knee (vertical coordinates of the feature point p) is the minimum of the X times at approximately the same height. The value (bottom) appears. Therefore, the evaluation unit 133 detects each time point at which the height of the feature point p above the knee becomes the minimum as a time when the person is in a sitting position. The feature point p above the knee is, for example, the abdomen, chest, neck, shoulder, elbow, wrist, finger, head, eyes, ear, mouth, or nose, but is not limited thereto.

(2) Second detection method When standing and sitting, the angle of the knees is the smallest when sitting (see FIG. 8). For this reason, for example, if a subject stands and sits X times, the angle of the knees (the angle between the straight line connecting heel p1 and knee p2 and the straight line connecting knee p2 and hip p3) will be approximately the same. The minimum value (bottom) of the X times of magnitude appears. Therefore, the evaluation unit 133 detects each time point when the knee angle becomes the minimum value as a time when the person is in a sitting position.

(3) Third detection method When a person stands up or sits down, the body inclination t is greatest in the standing position (see FIG. 8). The body inclination t here refers to the inclination of the body axis with respect to the vertical axis. For this reason, for example, when a subject stands up and sits down X times, a maximum value (peak) of approximately the same magnitude for the X times appears in the body inclination t. Therefore, the evaluation unit 133 detects each time point when the body inclination t reaches the maximum value as the time of sitting.

Through the above processing, the subject's muscle strength is evaluated from the standing/sitting video 121, and an evaluation value v2 is calculated. The evaluation unit 133 stores the evaluation value v2 in the storage unit 12 as the evaluation result 124.

(Step S107)
The evaluation unit 133 evaluates the standing posture of the subject based on the feature point information 123 stored in the storage unit 12 (step S107). More specifically, the evaluation unit 133 calculates the evaluation value v3 of the standing posture based on the elongation e of the hip joint of the subject in the standing position during a plurality of standing and sitting positions.

FIG. 11 is a flowchart illustrating an example of a method for evaluating standing posture by the evaluation unit 133. The process in FIG. 11 corresponds to the internal process in step S107.

(Step S301)
First, the evaluation unit 133 detects the standing position of the subject from the standing/sitting video 121 (step S301). The method for detecting the standing position is as described above.

(Step S302)
Next, the evaluation unit 133 calculates the elongation e of the hip joint in the standing position detected in step S301 (step S302). For example, if the subject stands up and sits down x times, the extensions e1 to ex of the hip joint during standing up to x times are calculated.

FIG. 12 is a diagram illustrating the extension e of the hip joint in a standing position using the image im3 of FIG. 8. As shown in FIG. 12, the extension e of the hip joint during standing is, for example, the angle θ2 formed by a line L3 connecting the neck p7 and the waist p3 and a line L4 connecting the waist p3 and the knee p2. The line L3 may be a line connecting the shoulder p6 and the waist p3. The line L4 may be a line connecting the waist p3 and the heel p1.

(Step S303)
Subsequently, the evaluation unit 133 calculates an evaluation value v3 of the subject's standing posture based on the hip joint extensions e1 to ex during the x times of standing calculated in step S302 (step S303). Generally, it is considered that the smaller the extension e of the hip joint during standing (the angle θ2 is closer to 0), the better the standing posture. Therefore, for example, the evaluation unit 133 evaluates the standing posture more highly as the representative value tr of the hip joint extensions e1 to tx during standing is smaller (the absolute value of angle θ2 is smaller). If the evaluation value v3 is higher as the standing posture is better, the evaluation unit 133 determines that the smaller the representative value tr of the hip joint extension e1 to ex during standing is (the smaller the absolute value of the angle θ2), the better the standing posture is. Calculate a high evaluation value v3. The representative value er is, for example, the minimum value, average value, median value, or average value of the top X% of the elongations e1 to ex, but is not limited thereto. Note that the evaluation value v3 may be a multilevel rank obtained by comparing the representative value er with a preset threshold value.

Through the above processing, the standing posture of the subject is evaluated from the standing/sitting video 121, and an evaluation value v3 is calculated. The evaluation unit 133 stores the evaluation value v3 in the storage unit 12 as the evaluation result 124.

(Step S108)
Now, return to FIG. 6. The evaluation unit 133 evaluates the compression fracture risk of the subject while sitting, based on the feature point information 123 stored in the storage unit 12 (step S108). More specifically, the evaluation unit 133 calculates an evaluation value v4 of the risk of compression fracture when the subject is seated, based on the time tu required for standing up and the time td required for sitting. The standing up time tu is the time required for the subject to stand up. The required sitting time td is the time required for the subject to sit.

FIG. 13 is a flowchart illustrating an example of a method for evaluating the risk of compression fracture during sitting by the evaluation unit 133. The process in FIG. 13 corresponds to the internal process in step S108.

(Step S401)
First, the evaluation unit 133 detects the standing and sitting positions of the subject from the standing and sitting video 121 (step S401). The method for detecting standing and sitting positions is as described above.

(Step S402)
Next, the evaluation unit 133 calculates the required time tu for standing and the required time td for sitting (see FIG. 8) for each of standing and sitting based on the timings of the standing and sitting positions detected in step S401 (step S402). For example, if the subject stands up and sits down x times, the required standing times tu1 to tux and the required sitting times td1 to tdx for x times are calculated.

(Step S403)
Subsequently, the evaluation unit 133 calculates the sitting/standing time ratio r for each sitting/standing position. The sitting/standing time ratio r is the ratio of the required sitting time td to the required standing time tu (r=td/tu).

(Step S404)
Thereafter, the evaluation unit 133 calculates an evaluation value v4 of the compression fracture risk when the subject is seated, based on the sitting/standing time ratios r1 to rx for x times calculated in step S403 (step S404). In general, it is considered that the risk of compression fracture during sitting is higher as it is faster to sit down than to stand up, that is, the smaller the ratio of the required sitting time td to the required standing time tu (the smaller the sitting/standing time ratio r). Therefore, for example, the evaluation unit 133 evaluates the compression fracture risk while sitting higher as the representative value rr of the sitting/standing time ratios r1 to rx is smaller. If the evaluation value v4 is lower as the risk of compression fracture when sitting is higher, the evaluation unit 133 calculates the evaluation value v4 of the risk of compression fracture when sitting to be lower as the representative value rr of the sitting/standing time ratios r1 to rx is smaller. . The representative value rr is, for example, the minimum value, average value, median value, or average value of the top X% of the sitting/standing time ratios r1 to rx in ascending order, but is not limited thereto. Note that the evaluation value v4 may be a multilevel rank obtained by comparing the representative value rr and a preset threshold value.

Through the above processing, the risk of compression fracture when the subject is seated is evaluated from the standing/sitting video 121, and an evaluation value v4 is calculated. The evaluation unit 133 stores the evaluation value v4 in the storage unit 12 as the evaluation result 124.

(Step S109)
The evaluation unit 133 comprehensively evaluates the condition of the subject based on the evaluation results of the stability of the subject's standing position, the muscle strength of the subject, the standing posture of the subject, and the risk of compression fracture when the subject is seated. (Step S109). Specifically, the evaluation unit 133 calculates a comprehensive evaluation value v5 of the condition of the subject based on the evaluation values v1 to v4. The evaluation value v5 is, for example, the average value or the total value of the evaluation values v1 to v4, but is not limited thereto. The evaluation unit 133 stores the evaluation value v5 in the storage unit 12 as the evaluation result 124.

(Step S110)
The evaluation unit 133 determines whether the subject is using the auxiliary tool A (step S110). Auxiliary tool A is any tool that assists the subject in standing up or sitting down. Examples of the auxiliary tools A include, but are not limited to, handrails, tables, and armrests of chairs.

FIG. 14 is a diagram showing an example of the auxiliary tool A. In the example of FIG. 14, the aid A is a handrail, and the subject is using the aid A to sit and stand.

Whether or not the subject is using the assistive device A may be registered by the user, and information to that effect may be sent from the user terminal 2 to the condition evaluation device 1 in association with the standing and sitting images 121, or the evaluation The determination may be made by the unit 133 analyzing the standing/sitting image 221. The determination method can be arbitrarily selected depending on the type of auxiliary tool A.

If the subject is using auxiliary tool A (step S110: YES), the process advances to step S111. If the subject is not using assistive device A (step S110: NO), the process proceeds to step S112.

(Step S111)
If the subject is using assistive device A, the evaluation unit 133 modifies the evaluation result 124 stored in the storage unit 12 so that the evaluation is lowered (step S111). That is, when the subject is using the assistive device A, the evaluation unit 133 evaluates the condition of the subject lower than when the subject is not using the assistive device A. Specifically, the evaluation unit 133 modifies the evaluation values v1 to v5 so that the respective evaluations decrease. The evaluation unit 133 may multiply the evaluation values v1 to v5 by a predetermined multiplier, or may add or subtract a predetermined value. The amount of modification of the evaluation value v may differ depending on the type of evaluation. The evaluation value 133 stores the corrected evaluation values v1 to v5 in the storage unit 12 as the evaluation result 124.

(Step S112)
After that, the evaluation unit 133 generates a message m to the target person based on the evaluation values v1 to v5 (step S112). The message m is generated, for example, by combining messages selected according to the evaluation values v1 to v5 from among those prepared in advance for each of the evaluation values v1 to v5. The evaluation unit 133 stores the message m in the storage unit 12 as the evaluation result 124.

(Step S113)
Now, return to FIG. 6. In response to a request from the user terminal 2, the condition evaluation device 1 transmits the evaluation result 124 including the evaluation values v1 to v5 and the message m to the user terminal 2 in association with the standing and sitting video 121 (step S113).

(Step S114)
The acquisition unit 231 of the user terminal 2 acquires the evaluation result 124 transmitted by the state evaluation device 1, and stores it in the storage unit 22 as the evaluation result 222 in association with the target video 221 (step S114).

(Step S115)
The display control unit 233 displays the evaluation result 222 of the standing and sitting video 221 on the display device 206D in response to the user's operation (step S115).

FIG. 15 is a diagram showing an example of the evaluation result display screen sc. The evaluation result display screen sc is a screen that displays the evaluation results 222. The evaluation result display screen sc in FIG. 15 displays a standing/sitting video 221, its evaluation values v1 to v5, and a message m. In the example shown in FIG. 15, each evaluation value v is evaluated with a score in five stages, and the evaluation value v1 is 5 points, the evaluation value v2 is 2 points, the evaluation value v3 is 4 points, the evaluation value v4 is 2 points, and the evaluation value v4 is 2 points. The value v5 is 13 points. This means that the subject whose standing-and-sitting video 221 was photographed has a high stability in the standing position, low muscle strength, good posture in the standing position, and a high risk of compression fracture when sitting. In addition, as message m, the message ``I am able to stand up straight. Let's do our best in strength training!'' is generated.

The subject and the user can easily understand the condition of the subject by referring to the evaluation result display screen sc.

<Summary>
As explained above, according to the present embodiment, the condition evaluation method executed by the condition evaluation system 1000 includes the acquisition step of acquiring the standing and sitting video 121 of the subject standing and sitting, which is shot from the side of the subject. , a state evaluation method including an extraction step of extracting a plurality of feature points p of the subject from the sitting/standing video 121, and an evaluation step of evaluating the state of the subject based on the plurality of feature points p can be realized.

Thereby, the condition evaluation system 1000 can evaluate the condition of the subject based on the standing and sitting video 121 of the subject. By using this condition evaluation system 1000, the user can easily grasp various conditions of the subject simply by photographing the standing and sitting video 221 of the subject.

<Additional notes>
This embodiment includes the following disclosure.

(Additional note 1)
A condition evaluation method performed by a condition evaluation system, comprising:
an acquisition step of acquiring a standing and sitting video of a subject standing and sitting taken from the side of the subject;
an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video;
an evaluation step of evaluating the condition of the subject based on the plurality of feature points;
condition assessment methods including;

(Additional note 2)
The condition evaluation method according to supplementary note 1, wherein the evaluation step evaluates the stability of the standing position of the subject.

(Additional note 3)
The condition evaluation method according to supplementary note 2, wherein the evaluation step evaluates the stability based on the inclination of the subject's body when standing.

(Additional note 4)
The condition evaluation method according to appendix 3, wherein the inclination in the standing position is an angle formed by a vertical line and a line connecting the subject's heel and neck.

(Appendix 5)
The condition evaluation method according to supplementary note 1, wherein the evaluation step evaluates the muscle strength of the subject.

(Appendix 6)
The condition evaluation method according to supplementary note 5, wherein the evaluation step evaluates the muscle strength based on the number of times or the required time of the subject to stand and sit.

(Appendix 7)
The condition evaluation method according to supplementary note 1, wherein the evaluation step evaluates the standing posture of the subject.

(Appendix 8)
The condition evaluation method according to supplementary note 7, wherein the evaluation step evaluates the posture based on the extension of the hip joint of the subject while standing.

(Appendix 9)
The condition evaluation method according to appendix 8, wherein the extension of the hip joint is the angle formed by a line connecting the neck and waist of the subject and a line connecting the waist and knees.

(Appendix 10)
The condition evaluation method according to supplementary note 1, wherein the evaluation step evaluates the risk of compression fracture of the subject while sitting.

(Appendix 11)
The condition evaluation method according to appendix 10, wherein the evaluation step evaluates the compression fracture risk based on the time required for standing up and the time required for sitting of the subject.

(Appendix 12)
The condition evaluation method according to supplementary note 1, in which, in the evaluation step, the condition of the subject is evaluated lower when the subject is using an auxiliary device, compared to when the subject is not using the auxiliary device.

(Appendix 13)
an acquisition unit that acquires a standing and sitting video of a subject standing and sitting taken from a side of the subject;
an extraction unit that extracts a plurality of feature points of the subject from the standing and sitting video;
an evaluation unit that evaluates the condition of the subject based on the plurality of feature points;
A condition evaluation system equipped with.

(Appendix 14)
to the computer,
an acquisition step of acquiring a standing and sitting video of a subject standing and sitting taken from the side of the subject;
an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video;
an evaluation step of evaluating the condition of the subject based on the plurality of feature points;
A condition evaluation program that executes a condition evaluation method including.

The embodiments disclosed herein are illustrative in all respects and should not be considered restrictive. The scope of the present invention is indicated by the claims rather than the above-mentioned meaning, and is intended to include meanings equivalent to the claims and all changes within the scope. Furthermore, the present invention is not limited to the embodiments described above, and can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. The embodiments are also included in the technical scope of the present invention.

1: Status evaluation device 2: User terminal 121: Standing/sitting video 122: Extraction model 123: Feature point information 124: Evaluation result 131: Acquisition unit 132: Extraction unit 133: Evaluation unit

Claims (19)

A condition evaluation method performed by a condition evaluation system, comprising:
an acquisition step of acquiring a standing and sitting video of a subject standing and sitting taken from the side of the subject;
an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video;
an evaluation step of evaluating the condition of the subject based on the plurality of feature points;
including;
The evaluation step is a state evaluation method for evaluating the stability of the standing position of the subject. A condition evaluation method performed by a condition evaluation system, comprising:
an acquisition step of acquiring a standing and sitting video of a subject standing and sitting taken from the side of the subject;
an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video;
an evaluation step of evaluating the condition of the subject based on the plurality of feature points;
including;
The evaluation step is a state evaluation method of evaluating the standing posture of the subject. A condition evaluation method performed by a condition evaluation system, comprising:
an acquisition step of acquiring a standing and sitting video of a subject standing and sitting taken from the side of the subject;
an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video;
an evaluation step of evaluating the condition of the subject based on the plurality of feature points;
including;
The evaluation step is a condition evaluation method for evaluating the risk of compression fracture when the subject is seated. A condition evaluation method performed by a condition evaluation system, comprising:
an acquisition step of acquiring a standing and sitting video of a subject standing and sitting taken from the side of the subject;
an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video;
an evaluation step of evaluating the condition of the subject based on the plurality of feature points;
including;
In the evaluation step, when the subject is using an auxiliary device, the condition of the subject is evaluated lower than when the subject is not using the auxiliary device. The condition evaluation method according to claim 1 , wherein the evaluation step evaluates the stability based on the inclination of the subject's body when standing. 6. The condition evaluation method according to claim 5, wherein the inclination during standing is an angle between a vertical line and a line connecting the subject's heel and neck. 5. The condition evaluation method according to claim 1, wherein the evaluation step evaluates the muscle strength of the subject. 8. The condition evaluation method according to claim 7, wherein the evaluation step evaluates the muscle strength based on the number of times or the time required for the subject to stand up and sit down. 3. The condition evaluation method according to claim 2, wherein the evaluation step evaluates the posture based on extension of the hip joint of the subject while standing. 10. The condition evaluation method according to claim 9, wherein the extension of the hip joint is an angle formed by a line connecting the neck and waist of the subject and a line connecting the waist and knees. 4. The condition evaluation method according to claim 3, wherein the evaluation step evaluates the compression fracture risk based on the time required for standing up and the time required for sitting of the subject. an acquisition unit that acquires a standing and sitting video of a subject standing and sitting taken from a side of the subject;
an extraction unit that extracts a plurality of feature points of the subject from the standing and sitting video;
an evaluation unit that evaluates the condition of the subject based on the plurality of feature points;
Equipped with
The evaluation unit is a condition evaluation system that evaluates the stability of the standing position of the subject. an acquisition unit that acquires a standing and sitting video of a subject standing and sitting taken from a side of the subject;
an extraction unit that extracts a plurality of feature points of the subject from the standing and sitting video;
an evaluation unit that evaluates the condition of the subject based on the plurality of feature points;
Equipped with
The evaluation unit is a state evaluation system that evaluates the standing posture of the subject. an acquisition unit that acquires a standing and sitting video of a subject standing and sitting taken from a side of the subject;
an extraction unit that extracts a plurality of feature points of the subject from the standing and sitting video;
an evaluation unit that evaluates the condition of the subject based on the plurality of feature points;
Equipped with
The evaluation unit is a condition evaluation system that evaluates the risk of compression fracture when the subject is seated. an acquisition unit that acquires a standing and sitting video of a subject standing and sitting taken from a side of the subject;
an extraction unit that extracts a plurality of feature points of the subject from the standing and sitting video;
an evaluation unit that evaluates the condition of the subject based on the plurality of feature points;
Equipped with
The evaluation unit is a condition evaluation system that evaluates the condition of the subject lower when the subject is using an auxiliary device than when the subject is not using the auxiliary device. to the computer,
an acquisition step of acquiring a standing and sitting video of a subject standing and sitting taken from the side of the subject;
an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video;
an evaluation step of evaluating the condition of the subject based on the plurality of feature points;
including;
The evaluation step is a condition evaluation program that executes a condition evaluation method for evaluating the stability of the standing position of the subject. to the computer,
an acquisition step of acquiring a standing and sitting video of a subject standing and sitting taken from the side of the subject;
an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video;
an evaluation step of evaluating the condition of the subject based on the plurality of feature points;
including;
The evaluation step is a state evaluation program that executes a state evaluation method for evaluating the standing posture of the subject. to the computer,
an acquisition step of acquiring a standing and sitting video of a subject standing and sitting taken from the side of the subject;
an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video;
an evaluation step of evaluating the condition of the subject based on the plurality of feature points;
including;
The evaluation step is a condition evaluation program that executes a condition evaluation method for evaluating the risk of compression fracture when the subject is seated. to the computer,
an acquisition step of acquiring a standing and sitting video of a subject standing and sitting taken from the side of the subject;
an extraction step of extracting a plurality of feature points of the subject from the standing and sitting video;
an evaluation step of evaluating the condition of the subject based on the plurality of feature points;
including;
The evaluation step is a condition evaluation program that executes a condition evaluation method that evaluates the condition of the subject lower when the subject is using an auxiliary device than when the subject is not using the auxiliary device.