JP2021043730A - Stationary determining system, and computer program - Google Patents

Abstract

To provide a system for determining a user movement and a computer program therefor which can stably determine a user movement.SOLUTION: The present invention is directed to a system for determining whether or not a user body is stationary. The system has a photographing unit, a site specifying unit, an image region setting unit, and a stationary determining unit. The photographing unit acquires photographed data including a plurality of frames by photographing the user movement. The site specifying unit specifies a predetermined site of the user body with respect to the acquired photographed data. The image region setting unit sets an image region corresponding to the specified predetermined site. The stationary determining unit determines whether or not the predetermined site is stationary based on a difference between frames of the image region.SELECTED DRAWING: Figure 7

Description

The present invention relates to a static determination system and a computer program.

Patent Document 1 discloses a method of acquiring an image of a user performing an exercise motion, generating a human motion screening score based on the image, and generating a personalized exercise program based on the score. ..

Japanese Unexamined Patent Publication No. 2016-73789

In order to evaluate the movement of such a user, it has been necessary to stably determine the stillness of the body.

The present invention has been made in view of such circumstances, and provides a static determination system and a computer program that enable stable static determination of a user's operation.

According to the present invention, it is a system for determining whether or not the user's body is stationary, and includes a photographing unit, a site specifying unit, an image area setting unit, and a stationary determination unit. The user's motion is photographed to acquire imaging data including a plurality of frames, the site specifying unit identifies a predetermined portion in the user's body with respect to the acquired imaging data, and the image area setting unit , A system that sets an image area associated with the specified predetermined part, and the stationary determination unit determines whether or not the predetermined part is stationary based on the difference between the frames of the image area. Is provided.

With such a configuration, it is possible to stably determine the stationary state of a specific part of the user's body.

It is a block diagram of the determination system 1 of the 1st Embodiment of this invention. It is a figure which shows the appearance of the mobile terminal 10. It is a block diagram which shows the hardware composition of a mobile terminal 10. It is a functional block diagram of the determination system 1. It is a flowchart which shows the process flow of the determination method. It is explanatory drawing which shows the identification of the part | region by the part identification part 32. It is a figure which shows an example of the area setting by the image area setting unit 33. It is a figure which shows another example of the area setting by the image area setting unit 33. It is a figure explaining the stationary determination by the stationary determination unit 34. It is a block diagram of the determination system 1 of the 2nd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The various features shown in the embodiments shown below can be combined with each other. In addition, the invention is independently established for each feature.

<1. First Embodiment>
(1-1. Overall configuration of judgment system 1)
As shown in FIG. 1, the determination system 1 of the first embodiment of the present invention includes a mobile terminal 10. As the mobile terminal 10, a smartphone, a tablet terminal, or the like on which a plurality of application programs can be installed can be used.

In the present embodiment, the mobile terminal 10 includes a camera 20 and is arranged so as to photograph the user U from the front side. The configuration shown in FIG. 1 is an example, and the mobile terminal 10 may be arranged at a position other than the front side of the user U.

(1-2. Configuration of mobile terminal 10)
As shown in FIG. 2, the housing 12 of the mobile terminal 10 is provided with a touch panel display 14 which is an image display means for displaying an image on the screen on the front surface 12A thereof. The touch panel display 14 includes, for example, an LCD (Liquid Crystal Display) and a touch sensor. The LCD displays various images, and the touch sensor accepts various input operations performed using an indicator such as a finger, a stylus, or a pen.

Further, the front surface 12A of the mobile terminal 10 is provided with a microphone 16 for inputting sound, a speaker 18 for outputting sound, and a camera 20 for capturing a subject. The camera 20 is provided not only on the front surface 12A of the housing 12 but also on the back surface of the housing 12. Further, the side surface 12B of the housing 12 is provided with operation buttons 22 such as a power button for starting or stopping the mobile terminal 10 and a sound volume adjustment button output by the speaker 18.

Further, the housing 12 of the mobile terminal 10 is provided with a slot into which a memory card is inserted, a USB (Universal Serial Bus) terminal, and the like.

FIG. 3 is a functional block diagram showing an electrical configuration of the mobile terminal 10.

In addition to the above configuration, the mobile terminal 10 includes a control unit 24, a storage unit 26, and a communication unit 28.

The control unit 24 is, for example, a CPU (Central Processing Unit), a microprocessor, a DSP (Digital Signal Processor), or the like, and controls the entire operation of the mobile terminal 10.

The storage unit 26 includes, for example, a RAM (Random Access Memory), a DRAM (Dynamic Random Access Memory), and the like, and is used as a work area or the like when the control unit 24 executes processing based on various programs.

The storage unit 26 further includes a non-volatile memory such as a flash memory, and stores various data such as images and a program or the like used for processing of the control unit 24. The program stored in the storage unit 26 includes, for example, an OS (Operating System) for realizing the basic functions of the mobile terminal 10, a driver for controlling various hardware, e-mail, web browsing, and various other functions. It is an application program to realize it. Further, the storage unit 26 stores in advance a program for executing the determination method according to the present embodiment, which will be described in detail later.

The communication unit 28 is, for example, a NIC (Network Interface Controller) and has a function of connecting to a communication network such as a mobile phone network. The communication unit 28 has a function of connecting to a wireless LAN (Local Area Network) instead of or together with the NIC, a function of connecting to a wireless WAN (Wide Area Network), for example, a short-range Bluetooth (registered trademark) or the like. It may have a function that enables wireless communication, infrared communication, and the like.

The control unit 24, the storage unit 26, the communication unit 28, the touch panel display 14, the microphone 16, the speaker 18, the camera 20, and the operation buttons 22 are electrically connected to each other via the system bus 30. Therefore, the control unit 24 accesses the storage unit 26, displays an image on the touch panel display 14, grasps the operation state of the touch panel display 14 and the operation buttons 22 by the user, inputs sound to the microphone 16, and sounds from the speaker 18. Output, control of the camera 20, access to various communication networks and other information processing devices via the communication unit 28, and the like.

(1-3. Functional block diagram)
FIG. 4 is a functional block diagram relating to the determination function according to the present embodiment.

As shown in FIG. 4, the control unit 24 includes a photographing unit 31, a site specifying unit 32, an image area setting unit 33, and a rest determination unit 34.

The shooting unit 31 shoots the operation of the user U and acquires shooting data including a plurality of frames. The part identification unit 32 identifies a predetermined part of the user U to be determined to be stationary with respect to the acquired shooting data.

The image area setting unit 33 sets an image area associated with the specified predetermined portion. The stillness determination unit 34 determines whether or not the specified predetermined portion is stationary based on the difference between the frames of the image area.

Each functional configuration included in the control unit 24 described above is realized by a computer program stored in the storage unit 26. Further, the computer program may be stored in a non-temporary recording medium that can be read by a computer, and may be read from the recording medium as needed.

(1-4. Flow of judgment processing)
With reference to FIG. 5, the flow of the static determination process using the determination system 1 will be described in detail.

In step S110, the photographing unit 31 photographs the operation of the user U. Here, when the shooting is performed at N frames / second for T seconds, the number of frames M included in the shooting data is calculated by N × T. N and T are positive real numbers and M is a natural number. Each frame contains a still image. N is, for example, 60 and T is, for example, 5. In this case, the shooting data includes 300 still images (M = 60 × 5 = 300).

In step S120, the site identification unit 32 identifies the site of the user U, which is the target of the static determination, with respect to the acquired shooting data. As an example, this identification can be performed based on a posture model by two-dimensional posture estimation for each still image constituting the shooting data.

In the two-dimensional posture estimation, the posture of the user U can be estimated two-dimensionally by estimating the coordinates of a plurality of feature points of the user U in the still image by, for example, an algorithm using deep learning. Further, by estimating the two-dimensional posture for a plurality of still images, it is possible to measure the movement of the skeleton of the user U.

FIG. 6 shows an example of arranging a plurality of feature points. FIG. 6 illustrates an image when the user U is standing on one leg with his right foot. Here, the feature points 0 to 14 are the head (0), the neck (1), the right shoulder (2), the right elbow (3), the right hand (4), the left shoulder (5), the left elbow (6), respectively. It corresponds to the left hand (7), hip (8), right hip joint (9), right knee (10), right ankle (11), left hip joint (12), left knee (13), and left ankle (14).

In this way, when the coordinates of the feature points 0 to 14 of the user U are acquired in each still image by the two-dimensional posture estimation, the user's posture can be estimated based on these coordinates. The number and arrangement of feature points are not limited to those of the present embodiment, and can be changed as appropriate.

In this way, the coordinates of a plurality of feature points obtained by the two-dimensional posture estimation for each still image constituting each shooting data can be acquired as measurement data. Here, when the shooting data is composed of M frames, the measurement data corresponding to the shooting data is also composed of M frames.

The site specifying unit 32 identifies a predetermined site in the body of the user U based on the acquired measurement data (that is, the coordinates of a plurality of feature points). Here, the site specifying unit 32 may change the site to be specified according to the operation of the user U. In this case, the part specifying unit 32 does not need to acquire the measurement data for all the feature points, and may acquire the measurement data only for the part to be specified for each operation performed by the user U.

In step S130, the image area setting unit 33 sets the image area D associated with the specified predetermined portion. When setting the image area D, feature points corresponding to each operation of the user U are stored in the storage unit 26. As an example, in the motion of standing on one leg with the right foot shown in FIG. 7, the feature points 1 to 7 which are the parts of the upper body are associated as predetermined parts. Therefore, the image area setting unit 33 refers to the storage unit 26 and sets the image including the feature points 1 to 7 as the image area D. Further, for example, when the portion to be specified is the upper body, the image area setting unit 33 may set the image above the feature point 8 (waist) as the image area D.

Further, in another example shown in FIG. 8, a case where the user U extends his / her left arm is illustrated. In such a case, in order to estimate the position of the left arm from the coordinate positions of the feature points 5 to 7, an image including the feature points 5 to 7 may be set as the image area D.

In step S140, the rest determination unit 34 determines whether or not the specified predetermined portion is stationary based on the difference between the frames of the image area. Specifically, the static determination unit 34 calculates the difference in gradation value for each pixel included in the image area for three consecutive frames in the acquired shooting data, and the specified predetermined portion is stationary. Determine if it is.

Hereinafter, a specific description will be given with reference to FIG. FIG. 9 shows three consecutive images G1 to G3 selected from the acquired shooting data. It is assumed that the shooting data is composed of 300 still images (M = 300), and is composed of a start point frame F1 to an end point frame F300. The images G1 to G3 correspond to frames Fn to Fn + 2 (1 ≦ n ≦ 298), respectively.

The images G1 to G3 include image areas D1 to D3 set by the image area setting unit 33, respectively. Here, the image areas D1 to D3 have gradation values from 0 to 255 for each of a plurality of pixels. That is, it can be expressed by the following (Equation 1) to (Equation 3).

Here, D1 (i, j) means each pixel included in the image area D1. The operator "∋" means a possible value. The static determination unit 34 calculates the absolute value of the difference for each image region in which the frames are continuous as the difference images F12 and F23.

Next, the static determination unit 34 calculates the logical product of the difference images F12 and F23 to calculate the logical product image Fa.

Here, the operator "∩" means an intersection. Next, the static determination unit 34 performs binarization processing by comparing with an arbitrary threshold value T, and calculates the binarized difference image Ft.

By the above processing, in the image areas D1 to D3, the gradation value is 255 for the portion (pixel) determined to be stationary, and the gradation value is 0 for the pixel determined not to be stationary. The threshold value T may be appropriately set according to the accuracy obtained in the static determination.

Next, the total sum F of the pixel values in the difference image Ft is calculated, and the determination value Fv of the static determination is calculated.

As described above, a plurality of (298 in this embodiment) determination values Fv can be obtained for the frames Fn to Fn + 2 (1 ≦ n ≦ 298). The static determination is performed by performing a predetermined statistical process on the determination value Fv and comparing it with a predetermined threshold value. The statistical processing here is a process of calculating a statistically obtained value such as a simple sum, a weighted sum, or a value based on the simple sum, a weighted sum, or a value based on the simple sum, a weighted average, or the like (simple average, weighted average).

As described above, the determination system 1 according to the present embodiment includes a photographing unit 31, a site specifying unit 32, an image area setting unit 33, and a rest determination unit 34. The shooting unit 31 captures the user's movements and acquires shooting data including a plurality of frames. The site identification unit 32 identifies a predetermined site in the user's body with respect to the acquired imaging data.

The image area setting unit 33 sets an image area associated with the specified predetermined portion. The stillness determination unit 34 determines whether or not a predetermined portion is stationary based on the difference between the frames of the image area.

With such a configuration, it is possible to suppress the influence of blurring or the like at the time of detection on a specific part of the user's body, and it is possible to stably perform a stationary determination.

<2. Second Embodiment>
The determination system 1 according to the second embodiment will be described with reference to FIG. The determination system 1 according to the second embodiment is different from the first embodiment in that it includes a master unit 10a and a slave unit 10b as the mobile terminal 10. Hereinafter, the differences from the first embodiment will be mainly described.

As shown in FIG. 10, in the present embodiment, the slave units 10b are arranged on the right side, the left side, and the rear side of the user U. The slave unit 10b is composed of the mobile terminal 10 like the master unit 10a. The master unit 10a and the slave unit 10b are configured to be able to communicate with each other through the communication unit 28. The slave unit 10b functions as a shooting unit 31, shoots the operation of the user U from a plurality of directions, and acquires a plurality of shooting data.

The shooting data acquired by the slave unit 10b is transmitted to the master unit 10a. The site specifying unit 32 of the master unit 10a identifies a predetermined site for each of the acquired plurality of captured data.

The stationary determination unit 34 sets weighting based on the shooting direction for the user U with respect to the shooting data acquired from a plurality of directions. The stillness determination unit 34 determines whether or not a predetermined portion of the user is stationary based on the difference between the frames of the image area set by the image area setting unit 33 and the weighting.

With such a configuration, it is possible to analyze whether or not the user's body is stationary based on a plurality of shooting data, and it is possible to perform a more stable stationary determination.

Here, preferably, the stationary determination unit 34 may set the weighting based on the shooting direction according to the operation of the user U who shoots. The weighting based on the imaging direction is set in association with the specified predetermined portion. That is, in the operation of the user U, it is conceivable to set a large weighting for the shooting data from the direction in which the movement is easily detected for each predetermined part.

With such a configuration, it is possible to change the weighting based on the shooting direction according to the movement of the user, and it is possible to perform the stillness determination more stably.

<3. Other embodiments>
Although the embodiments have been described above, the scope of application of the technical scope of the present application is not limited to the above embodiments.

For example, the rest determination unit 34 may be specified to select three frames selected based on the determination value of the rest determination from the continuous L frames included in the acquired shooting data as the determination target frames.

In this case, the static determination unit 34 calculates, for example, the static determination determination value for three consecutive frames (L = 3) in P ways in which the starting frame is different. Here, P is a natural number. The stationary determination unit 34 may select a frame in which the order in which the determination values according to P are arranged in ascending order of the movement of the user U is within the Q position is selected as the determination target frame. In addition, L can be a value other than 3.

For example, if P = 90 and Q = 10, 10 determination values Fv with smaller values are selected from 90 determination values Fv. Further, if P = 90, Q = 1, and the determination value Fv of three consecutive frames starting from the frame F121 (that is, frames F121 to F123) is the minimum, the frames F121 to F123 become the determination target frames. .. Although the determination value Fv may be calculated for three consecutive frames starting from the frame F0, the reliability of the data immediately after the start of imaging is low. That is, the reliability of the determination may be improved by excluding the first plurality of frames from the measurement data and specifying the determination target frame.

Further, in selecting the evaluation target frame when shooting from a plurality of directions as in the second embodiment, the judgment target frame for static determination is calculated from the shooting data from the shooting direction having the largest weighting. May be elected. In this case, the determination value Fv of the frame in the other shooting direction corresponding to the selected determination target frame is calculated, and the static determination is performed in consideration of the weighting based on the photographing direction for the determination value Fv in each direction.

Further, in the above embodiment, the determination system 1 is configured by using the mobile terminal 10 such as a smartphone or a tablet, but at least one of the devices constituting the determination system 1 is capable of communicating and has a shooting function. It may be any device (eg, Web camera) that it has.

Further, the present invention can also be realized as a program for operating a mobile terminal for realizing the above-mentioned determination system.

Furthermore, the present invention can also be realized as a non-temporary recording medium that can be read by a computer and stores the above-mentioned program.

<4. Features of the embodiment>
The features of the embodiments of the present invention are summarized below.

The determination system 1 includes a photographing unit 31, a site specifying unit 32, an image area setting unit 33, and a rest determination unit 34. The shooting unit 31 captures the user's movements and acquires shooting data including a plurality of frames. The site identification unit 32 identifies a predetermined site in the user's body with respect to the acquired imaging data. The image area setting unit 33 sets an image area associated with the specified predetermined portion. The stillness determination unit 34 determines whether or not a predetermined portion is stationary based on the difference between the frames of the image area.

With such a configuration, it is possible to suppress the influence of blurring or the like at the time of detection on a specific part of the user's body, and it is possible to stably perform a stationary determination.

The part specifying unit 32 included in the determination system 1 may specify a predetermined part in the body of the user U based on the posture model estimated from the shooting data.

With such a configuration, it is possible to stably identify a part of the user's body.

The site specifying unit 32 may specify a predetermined site based on the operation of the user U.

With such a configuration, a part to be specified can be set for each operation, and the part can be specified more stably.

The plurality of frames of the shooting data are composed of M frames, and the stationary determination unit 34 determines a predetermined number of continuous L frames in P ways having different starting points from the shooting data based on the difference between the frames. The determination value Fv for the degree of rest of the part is calculated, and the determination value Fv according to P is arranged in ascending order of the movement of the predetermined part, and the order is within the Q position. It may be determined whether or not the part is stationary.

With such a configuration, the target of the stationary determination can be narrowed down, and the stationary determination can be performed more stably.

The photographing unit 31 photographs the user's movements from a plurality of directions to acquire a plurality of photographing data, and the site specifying unit 32 specifies the predetermined portion with respect to the acquired plurality of photographed data. May be good.

With such a configuration, a plurality of shooting data can be analyzed, and static determination can be performed more stably.

The stillness determination unit 34 sets weighting for the user U based on the shooting direction for the plurality of shooting data, and the predetermined portion is stationary based on the difference between the frames of the image area and the weighting of the shooting data. It may be determined whether or not it is.

With such a configuration, a plurality of data are weighted in the shooting direction, and the stillness determination can be performed more stably.

The stationary determination unit 34 may set the weighting based on the shooting direction according to the operation of the user U.

With such a configuration, the weighting in the shooting direction is changed according to the operation, and the stationary determination can be performed more stably.

Although various embodiments according to the present invention have been described, they are presented as examples and are not intended to limit the scope of the invention. The novel embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and the equivalent scope thereof.

1: Judgment system 10: Mobile terminal 10a: Master unit 10b: Slave unit 11: Feature point 12: Housing 12A: Front surface 12B: Side surface 13: Feature point 14: Touch panel display 16: Microphone 18: Speaker 20: Camera 22: Operation Button 24: Control unit 26: Storage unit 28: Communication unit 30: System bus 31: Imaging unit 32: Part identification unit 33: Image area setting unit 34: Stillness determination unit U: User

Claims (8)

It is a system that determines whether the user's body is stationary.
It is equipped with a shooting unit, a part identification unit, an image area setting unit, and a stillness determination unit.
The shooting unit captures the movement of the user and acquires shooting data including a plurality of frames.
The site identification unit identifies a predetermined site in the user's body with respect to the acquired shooting data.
The image area setting unit sets an image area associated with the specified predetermined portion, and sets the image area.
The rest determination unit is a system that determines whether or not the predetermined portion is stationary based on the difference between the frames of the image region. The system according to claim 1.
The site identification unit is a system that identifies a predetermined site in the user's body based on a posture model estimated from the imaging data. The system according to claim 1 or 2.
The site specifying unit is a system that identifies a predetermined site based on the movement of the user. The system according to any one of claims 1 to 3.
The plurality of frames of the shooting data are composed of M frames.
The stationary determination unit
For continuous L frames in P ways with different starting frames from the shooting data, a judgment value for the degree of rest of the predetermined portion is calculated based on the difference between the frames.
It is determined whether or not the specified predetermined part is stationary based on the L frame in which the order of the determination values according to P is arranged in ascending order of the movement of the predetermined part within the Q position. ,system. The system according to any one of claims 1 to 4.
The shooting unit captures the user's movements from a plurality of directions and acquires a plurality of shooting data.
The site identification unit is a system that identifies the predetermined site with respect to the acquired plurality of imaging data. The system according to claim 5.
The stationary determination unit
For the plurality of the shooting data, weighting based on the shooting direction for the user is set.
A system that determines whether or not the predetermined portion is stationary based on the difference between the frames of the image region and the weighting of the shooting data. The system according to claim 6.
The static determination unit is a system that sets weighting based on the shooting direction according to the operation of the user. A computer program that causes a computer to execute a method for determining whether or not a user's body is stationary, which includes a shooting step, a site identification step, an image area setting step, and a rest determination step.
In the shooting step, the movement of the user is shot to acquire shooting data including a plurality of frames, and the shooting data is acquired.
In the site identification step, a predetermined site in the user's body is identified with respect to the acquired imaging data.
In the image area setting step, an image area associated with the specified predetermined portion is set.
In the rest determination step, a computer program that determines whether or not the predetermined portion is stationary based on the difference between the frames of the image area.

Images