WO2021192905A1

WO2021192905A1 - Guide method

Info

Publication number: WO2021192905A1
Application number: PCT/JP2021/008566
Authority: WO
Inventors: 宏紀寺島; 克幸永井
Original assignee: Ｎｅｃソリューションイノベータ株式会社
Priority date: 2020-03-25
Filing date: 2021-03-04
Publication date: 2021-09-30
Also published as: JPWO2021192905A1; CN115299036A; JP7323234B2

Abstract

An imaging device 400 that captures the way a person walks has: a detection unit 421 that detects the orientation of the imaging device; and a display unit 422 that displays, on a screen display part, guidelines indicating the position at which the person is walking. The display unit 422 displays different guidelines depending on the orientation of the imaging device detected by the detection unit 421.

Description

Guide method

The present invention relates to a guide method, a photographing device, and a recording medium.

It is known to analyze the user's gait based on the data.

For example, Patent Document 1 is a document that describes walking analysis of a person. Patent Document 1 is created by a data acquisition unit that acquires two types of image data from a depth sensor, a skeleton information creation unit that creates skeleton information based on the image data acquired by the data acquisition unit, and a skeleton information creation unit. A gait analyzer having a correction processing unit that corrects skeleton information and an analysis processing unit that analyzes the user's gait using the corrected skeleton information is described.

International Publication No. 2017/170832

In the technique described in Patent Document 1, it is essential to use a depth sensor (3D sensor), but it is desired to perform gait analysis based on image data acquired by a camera without using a 3D sensor such as a depth sensor. There is a need.

In order to improve the accuracy of gait analysis based on image data, it is desirable to make the shooting conditions at the time of image data shooting as uniform as possible. However, the image data to be analyzed is acquired at different timings. Therefore, it was difficult to match the shooting conditions.

In this way, there was a problem that it was difficult to match the shooting conditions when acquiring image data.

Therefore, an object of the present invention is to provide a guide method, a photographing device, and a recording medium that solve the problem that it is difficult to arrange the photographing conditions when acquiring image data.

A guide method, which is a form of the present disclosure, in order to achieve such an object
A shooting device that shoots a person walking
Detects the orientation of the shooting device and
A guide line indicating the position where the person walks is displayed on the screen display.
When the guide line is displayed on the screen display unit, the guide line is displayed differently depending on the direction of the detected imaging device.

In addition, the imaging device which is another form of the present disclosure is
It is a shooting device that shoots a person walking.
A detector that detects the orientation of the imaging device and
A display unit that displays a guide line indicating the position where a person walks on the screen display unit,
Have,
The display unit has a configuration in which different guide lines are displayed according to the orientation of the photographing device detected by the detection unit.

In addition, the recording medium which is another form of the present disclosure is
For a shooting device that shoots a person walking
A detector that detects the orientation of the imaging device and
A display unit that displays a guide line indicating the position where a person walks on the screen display unit,
Realized,
The display unit is a computer-readable recording medium that records a program that displays the guide lines that differ depending on the orientation of the photographing device detected by the detection unit.

According to each configuration as described above, it is possible to arrange the shooting conditions when acquiring the image data.

It is a figure which shows the structural example of the walking posture measurement system in 1st Embodiment of this disclosure. It is a figure which shows an example at the time of photographing the walking posture in the front-back direction. It is a figure which shows an example at the time of photographing the walking posture in the left-right direction. It is a block diagram which shows the configuration example of the smartphone shown in FIG. It is a block diagram which shows the structural example of the photographing auxiliary part shown in FIG. It is a figure which shows the example of the shooting assistance display at the time of holding horizontally. It is a figure which shows the example of the shooting assistance display at the time of holding vertically. It is a figure which shows the operation example of the angle adjustment information output part shown in FIG. It is a block diagram which shows the structural example of the walking posture measuring apparatus shown in FIG. It is a figure which shows an example of the skeleton information shown in FIG. It is a figure which shows an example of the measurement result information shown in FIG. It is a figure for demonstrating the processing of the measured value calculation part. It is a figure for demonstrating the processing of the measured value calculation part. It is a figure for demonstrating the processing of the measured value calculation part. It is a figure for demonstrating the processing of the measured value calculation part. It is a figure for demonstrating the processing of the measured value calculation part. It is a figure for demonstrating the processing of the measured value calculation part. It is a figure for demonstrating the processing of the measured value calculation part. It is a figure for demonstrating the processing of the measured value calculation part. It is a flowchart which shows the operation example of the shooting assistance part in a smartphone. It is a flowchart which shows the operation example of the walking posture measuring apparatus. It is a flowchart which shows an example of the process of calculating the measured value information. It is a figure for demonstrating the tracking example of the walking posture measuring apparatus in the 2nd Embodiment of this disclosure. It is a block diagram which shows the structural example of the walking posture measuring apparatus in 2nd Embodiment of this disclosure. It is a figure which shows the structural example of the tracking part shown in FIG. It is a figure which shows an example of the figure generated by the inclusion figure generation part shown in FIG. It is a figure which shows an example of the figure generated by the inclusion figure generation part shown in FIG. It is a flowchart which shows the operation example of the tracking part. It is a figure which shows an example of the hardware composition which the photographing apparatus has in 3rd Embodiment of this disclosure. It is a block diagram which shows the structural example of the photographing apparatus. It is a block diagram which shows the structural example of the information processing apparatus in 4th Embodiment of this disclosure. It is a block diagram which shows the structural example of the information processing apparatus in 5th Embodiment of this disclosure. It is a block diagram which shows the structural example of the tracking apparatus in 6th Embodiment of this disclosure.

[First Embodiment]
The first embodiment of the present disclosure will be described with reference to FIGS. 1 to 22. FIG. 1 is a diagram showing a configuration example of the walking posture measurement system 100. FIG. 2 is a diagram showing an example of photographing a walking posture in the front-back direction. FIG. 3 is a diagram showing an example of photographing a walking posture in the left-right direction. FIG. 4 is a block diagram showing a configuration example of the smartphone 200. FIG. 5 is a block diagram showing a configuration example of the photographing auxiliary unit 212. FIG. 6 is a diagram showing an example of a shooting assistance display when the camera is held horizontally. FIG. 7 is a diagram showing an example of a shooting auxiliary display when the camera is held vertically. FIG. 8 is a diagram showing an operation example of the angle adjustment information output unit 2124. FIG. 9 is a block diagram showing a configuration example of the walking posture measuring device 300. FIG. 10 is a diagram showing an example of skeleton information 334. FIG. 11 is a diagram showing an example of measurement result information 336. FIG. 12 is a diagram for explaining the processing of the actually measured value calculation unit 343. 13 to 19 are diagrams for explaining the processing of the actually measured value calculation unit 343. FIG. 20 is a flowchart showing an operation example of the photographing assisting unit 212 in the smartphone 200. FIG. 21 is a flowchart showing an operation example of the walking posture measuring device 300. FIG. 22 is a flowchart showing an example of processing for calculating the measured value information 335.

In the first embodiment of the present disclosure, the walking posture measuring system 100 that measures the walking posture of a person based on a moving image acquired by using a photographing device such as a smartphone 200 will be described. In the walking posture measurement system 100, a moving image showing a person walking in the front-back direction of an image (that is, a plurality of image data), a moving image showing a person walking in the left-right direction of the image, and a moving image showing a person walking in the left-right direction of the image, using a smartphone 200, To shoot. Then, the walking posture measuring system 100 measures the walking posture such as stride length, walking speed, straightness according to blurring of the head during walking, etc., based on a plurality of image data which are moving images acquired by shooting. Further, as will be described later, the walking posture measurement system 100 includes a mechanism for aligning shooting conditions as much as possible when acquiring image data using the smartphone 200, a mechanism for measuring an actually measured value from the image data, and the like. I'm out.

FIG. 1 shows a configuration example of the walking posture measurement system 100. Referring to FIG. 1, the walking posture measuring system 100 includes, for example, a smartphone 200 and a walking posture measuring device 300. As shown in FIG. 1, the smartphone 200 and the walking posture measuring device 300 are connected so as to be able to communicate with each other by, for example, wirelessly or by wire.

The smartphone 200 functions as a photographing device for photographing a person walking. The smartphone 200 may be a smartphone having general functions such as a camera function, a touch panel 201 for displaying a screen, and various sensors such as a GPS sensor and an acceleration sensor.

In the case of the present embodiment, as shown in FIG. 2, the photographer takes a picture of a person walking from the back of the screen to the front, etc., while holding the smartphone 200 vertically (holding it vertically). In other words, the photographer takes a picture of a person walking in the front-back direction while the short side of the rectangular-shaped smartphone 200 is held vertically so as to be horizontal to the ground. Further, as shown in FIG. 3, the photographer takes a picture of a person walking in the left-right direction such as from the left to the right of the screen while the smartphone 200 is held sideways (sideways). In other words, the photographer takes a picture of a person walking in the front-back direction while the smartphone 200 is held sideways with the long side horizontal to the ground. In this way, in the walking posture measurement system 100, the photographer shoots a type of moving image (a plurality of image data) according to the orientation of the smartphone 200. The appearance of a person walking in the front-back direction and the appearance of a person walking in the left-right direction may be divided into two shots using, for example, one smartphone 200, or two smartphones 200 may be used. You may shoot at the same time.

FIG. 4 shows a characteristic configuration example in the present embodiment of the smartphone 200. Referring to FIG. 4, the smartphone 200 has a measurement motion photographing unit 210 in addition to a general configuration as a smartphone such as an acceleration sensor and a gyro sensor for detecting the orientation (vertical orientation, horizontal orientation) of the smartphone 200. doing. Further, as shown in FIG. 4, the measurement motion photographing unit 210 includes an image data photographing unit 211 and an imaging assisting unit 212.

For example, the smartphone 200 has an arithmetic unit such as a CPU (Central Processing Unit) and a storage device. The smartphone 200 realizes the above-mentioned processing unit by, for example, executing a program stored in the storage device by the arithmetic unit.

The measurement motion photographing unit 210 photographs a person walking, which is the measurement motion in the present embodiment, in response to the photographer's operation on the smartphone 200. In addition, the measurement motion photographing unit 210 includes a function for assisting shooting so that shooting can be performed under the same conditions as possible when performing shooting. For example, the measurement motion photographing unit 210 has a camera operation function for photographing a person walking, and also captures the walking direction, angle, size reflected on the screen, etc. when photographing the person walking. It is a shooting application that has a guide function to meet the conditions as much as possible. As described above, the measurement motion photographing unit 210 includes an image data photographing unit 211 and an imaging assisting unit 212.

The image data shooting unit 211 acquires a moving image (a plurality of image data) by shooting a person using the camera of the smartphone 200. Further, the image data shooting unit 211 associates the moving image (image data) shot by the image data shooting unit 211 with information indicating the acquisition date and time of the image data, information acquired by the shooting assistance unit 212, which will be described later, and the like. Can be done.

The shooting assistance unit 212 assists the image data shooting unit 211 to arrange the shooting conditions as much as possible when acquiring the image data. FIG. 5 shows an example of the configuration included in the photographing assisting unit 212. Referring to FIG. 5, the photographing auxiliary unit 212 includes, for example, a guide line display unit 2121, an angle information display unit 2122, a height information input unit 2123, and an angle adjustment information output unit 2124.

The guide line display unit 2121 displays the guide line 2011 on the touch panel 201, which serves as a guideline for the position where the person walks, such as the position of the person's feet when shooting the person. By walking a person along the guide line 2011 displayed on the touch panel 201 as much as possible, it is possible to align the walking direction, angle, size displayed on the screen, etc. when the person walks. .. In addition, a mark or the like may be placed in the real world so that the position coincides with the position indicated by the guide line 2011, and the person may walk using the mark.

In the case of the present embodiment, the guide line display unit 2121 displays different guide lines 2011 on the touch panel 201 depending on whether the smartphone 200 is held vertically or horizontally. For example, when it is determined that the smartphone 200 is held horizontally based on the information acquired from the acceleration sensor or the like, the guide line display unit 2121 is for photographing a person walking in the left-right direction of the screen as shown in FIG. The guide line 2011 is displayed on the touch panel 201. In other words, the guide line display unit 2121 displays a guide line 2011 for guiding a person walking in the left-right direction on the touch panel 201. With reference to FIG. 6, it can be seen that the shooting area, which is the area shot by the camera of the smartphone 200, is displayed on the touch panel 201, and the guide line 2011 is displayed. Further, when it is determined that the smartphone 200 is held vertically based on the information acquired from the acceleration sensor or the like, the guide line display unit 2121 photographs a person walking in the front-back direction of the screen as shown in FIG. The guide line 2011 for this purpose is displayed on the touch panel 201. In other words, the guide line display unit 2121 displays a guide line 2011 for guiding a person walking in the front-back direction on the touch panel 201.

The position where the guide line display unit 2121 displays the guide line 2011 is, for example, predetermined. For example, when the smartphone 200 is held horizontally, the guide line display unit 2121 displays the guide line 2011 below the center of the photographing area (for example, about the middle of the lower half area) as shown in FIG. Further, for example, when the smartphone 200 is held vertically, the guide line display unit 2121 displays the guide line 2011 in the center of the photographing area as shown in FIG. 7. The position where the guide line display unit 2121 displays the guide line 2011 may be other than those illustrated above.

The angle information display unit 2122 acquires information indicating the angle of the smartphone 200, such as an inclination in the left-right direction and an inclination in the front-back direction, from an acceleration sensor or a gyro sensor of the smartphone 200. Then, the angle information display unit 2122 displays the acquired information on the touch panel 201 as angle information 2012 indicating the angle of the smartphone 200. When shooting a person walking, it is desirable that the smartphone 200 is not tilted as much as possible. By displaying the angle information 2012 on the touch panel 201 by the angle information display unit 2122, it is possible to correct the angle of the smartphone 200 when the photographer takes a picture of a person walking, and it is desirable that the smartphone 200 is not tilted. It is possible to take a picture of a person walking in a state.

In the case of the present embodiment, the angle information display unit 2122 displays information indicating the inclination in the left-right direction and information indicating the inclination in the front-back direction on the touch panel 201. In other words, the angle information display unit 2122 displays information indicating the inclination of the smartphone 200 in the horizontal direction and the vertical direction. For example, as shown in FIGS. 6 and 7, the angle information display unit 2122 displays the angle information 2012 at a predetermined position on the touch panel 201. The display of the angle information 2012 by the angle information display unit 2122 is the same (display according to the inclination) although the display position changes depending on whether the smartphone 200 is held vertically or horizontally. It doesn't matter.

The height information input unit 2123 receives input of height information indicating the height h from the ground of the smartphone 200 from a person, and displays the height display unit 2013 displaying the input height h on the touch panel 201 on the touch panel 201. do. The height h received by the height information input unit 2123 can be utilized when the walking posture measuring device 300 calculates the actually measured value W.

As shown in FIGS. 6 and 7, the height information input unit 2123 displays the height display unit 2013 at a predetermined position on the touch panel 201. Then, the height information input unit 2123 receives, for example, an input of information indicating the height h from the person by touching the height display unit 2013 by a person who operates the smartphone 200. After that, the height information input unit 2123 displays the information indicating the received height h on the height display unit 2013. The display of the height display unit 2013 by the height information input unit 2123 may be the same as the display content although the display position changes depending on whether the smartphone 200 is held vertically or horizontally.

The angle adjustment information output unit 2124 outputs information for adjusting the inclination of the smartphone 200. For example, the angle adjustment information output unit 2124 outputs information according to the inclination of the smartphone 200, which differs depending on the inclination of the smartphone 200. Specifically, for example, the angle adjustment information output unit 2124 outputs a sound adjusted according to the inclination of the smartphone 200 as information for adjusting the inclination of the smartphone 200.

FIG. 8 shows an example of processing by the angle adjustment information output unit 2124. Referring to FIG. 8, for example, the angle adjustment information output unit 2124 outputs a sound whose length is adjusted according to the inclination of the smartphone 200 in the left-right direction, and outputs the sound whose length is adjusted according to the inclination of the smartphone 200 in the front-back direction. Outputs a sound with adjusted pitch. In this way, the angle adjustment information output unit 2124 makes different adjustments depending on how the smartphone 200 is tilted. For example, referring to FIG. 8, the angle adjustment information output unit 2124 adjusts the length of one sound so that the more the smartphone 200 is tilted to the left, the shorter the length of one sound is. Further, the angle adjustment information output unit 2124 adjusts the length of the sound so that the longer the smartphone 200 is tilted to the right, the longer the length of one sound is. Further, the angle adjustment information output unit 2124 adjusts the pitch so that the pitch is lowered as the smartphone 200 is tilted toward the front (for example, the photographer side). Further, the angle adjustment information output unit 2124 adjusts the pitch so that the pitch increases as the smartphone 200 tilts toward the back (for example, the side opposite to the photographer).

According to the above configuration, when the smartphone 200 is tilted in either direction, the angle adjustment information output unit 2124 adjusts at least one of the length and pitch of the sound according to the tilt. Outputs two types of sounds. On the other hand, when the smartphone 200 is not tilted, the length and pitch of the sound are not adjusted, so that the angle adjustment information output unit 2124 outputs one type of sound. By configuring the angle adjustment information output unit 2124 to output the adjusted sound in this way, for example, even if it is difficult for the photographer to see the touch panel 201, the angle of the smartphone 200 can be easily adjusted. Can be done.

Note that the information for adjusting the tilt of the smartphone 200, which is output by the angle adjustment information output unit 2124, is not necessarily limited to the case of sound. For example, the angle adjustment information output unit 2124 may illuminate a light or vibrate the smartphone 200 instead of the sound. For example, the angle adjustment information output unit 2124 may be configured to perform processing such as illuminating a light or vibrating the smartphone 200 when approaching an angle having no inclination, which is a correct angle. The angle adjustment information output unit 2124 outputs a sound according to the inclination in the left-right direction and illuminates the light according to the inclination in the front-back direction, and outputs a sound and illuminates the light according to the inclination in the left-right direction. The above-mentioned processes may be combined in various combinations such as.

The above is a configuration example of the smartphone 200, which is characteristic of this embodiment.

The walking posture measuring device 300 is a server device that measures the walking posture such as stride length, walking speed, straightness, etc., based on a moving image (that is, a plurality of image data) taken by the smartphone 200. FIG. 9 shows a configuration example of the walking posture measuring device 300. Referring to FIG. 9, the walking posture measuring device 300 has, for example, a screen display unit 310, a communication I / F unit 320, a storage unit 330, and an arithmetic processing unit 340 as main components. There is.

The screen display unit 310 includes a screen display device such as a touch panel or a liquid crystal display. The screen display unit 310 displays the position of the skeleton indicated by the skeleton information 334 in the image data included in the image information 333, the skeleton information 334, the measurement result information 336, and the image information 333 in response to an instruction from the arithmetic processing unit 340. It is possible to display superimposed images, etc.

The communication I / F unit 320 includes a data communication circuit. The communication I / F unit 320 performs data communication with an external device, a smartphone 200, or the like connected via a communication line.

The storage unit 330 is a storage device such as a hard disk or a memory. The storage unit 330 stores processing information and a program 337 required for various processes in the arithmetic processing unit 340. The program 337 realizes various processing units by being read and executed by the arithmetic processing unit 340. The program 337 is read in advance from an external device or a recording medium via a data input / output function such as the communication I / F unit 320, and is stored in the storage unit 330. The main information stored in the storage unit 330 includes, for example, the trained model 331, the camera setting information 332, the image information 333, the skeleton information 334, the measured value information 335, and the measurement result information 336.

The trained model 331 is a trained model used by the skeleton recognition unit 342 when performing skeleton recognition. The trained model 331 is generated in advance by performing machine learning using teacher data such as image data containing skeleton coordinates in an external device or the like, and is externally generated via a communication I / F unit 320 or the like. It is acquired from a device or the like and stored in the storage unit 330.

The trained model 331 may be updated by a re-learning process using additional teacher data.

The camera setting information 332 includes information indicating the parameters of the camera possessed by the smartphone 200, which is used when the smartphone 200 captures the walking of a person. The camera setting information 332 includes, for example, information indicating a vertical viewing angle θ and a horizontal viewing angle ψ of the camera.

The camera setting information 332 is acquired in advance from the smartphone 200 or the like via, for example, the communication I / F unit 320 or the like, and is stored in the storage unit 330. When the camera setting information 332 is acquired from the smartphone 200, the camera setting information 332 may be acquired from the smartphone 200 together with the image data and stored in the storage unit 330.

The image information 333 includes image data (video) acquired by the camera of the smartphone 200. In the image information 333, for example, image data, information indicating the date and time when the smartphone 200 acquired the image data, and information indicating the height input by the height information input unit 2123 are provided for each unit of moving image. Etc. are associated with each other. Further, in the image information 333, a moving image of a person walking in the left-right direction and a moving image of a person walking in the front-back direction are associated with each other. As will be described later, the measuring unit 344 measures the walking posture corresponding to the moving image of the person walking in the left-right direction and the moving image of the person walking in the front-back direction.

The skeleton information 334 includes information indicating the coordinates of each part of the person recognized by the skeleton recognition unit 342. FIG. 10 shows an example of skeleton information 334. With reference to FIG. 10, in the skeleton information 334, for example, the time and the position information of each part are associated with each person to be recognized. The time indicates the elapsed time from the start of movie shooting, the time when the movie was shot, and the like. Further, the position information of each part includes information indicating the coordinates of each part in the image data such as the position of the pelvis.

The parts included in the position information of each part correspond to the trained model 331. For example, in FIG. 10, the pelvis, the center of the spine, ..., Are illustrated. The position information of each part can include, for example, about 30 parts such as right shoulder, ..., left elbow, ..., right knee, ..., etc. (other than the examples may be used). The part included in the position information of each part may be other than those illustrated in FIG. 10 and the like.

The actual measurement value information 335 includes information indicating the actual measurement value W calculated by the actual measurement value calculation unit 343. For example, in the actually measured value information 335, the actually measured value W is associated with the reference line, the identification information of the image data, and the like. The measured value information 335 may include information indicating the stride length and the like. A detailed description of the processing of the measured value calculation unit 343 will be described later.

The measurement result information 336 shows the result of the walking posture measurement measured by the measuring unit 344. FIG. 11 shows an example of the measurement result information 336. With reference to FIG. 11, in the measurement result information 336, for example, time, walking speed, stride length, straightness, and so on are associated with each person to be measured. Here, the time indicates the elapsed time from the start of moving image shooting, the time when the moving image was taken, and the like. The walking speed indicates the speed at which a person walks. The stride length indicates the length of the toe gap (or heel gap) between the right foot and the left foot when the person walks. In addition, the straightness indicates the degree of shaking and blurring of the head and body when a person walks. The measurement result information 336 may include information indicating a pitch or the like indicating the time required for one step.

Here, among various information included in the measurement result information 336, the walking speed and the stride are the information measured by the measured value calculation unit 343 and the measurement unit 344 based on the moving image (image data) of the person walking in the left-right direction. .. Further, among various information included in the measurement result information 336, the straightness is information measured by the measuring unit 344 based on a moving image (image data) in which a person walks in the front-back direction. For example, as described above, the measurement result information 336 is measured by the measuring unit 344 based on the information measured by the measuring unit 344 based on the moving image of the person walking in the left-right direction and the measuring unit 344 based on the moving image of the person walking in the front-back direction. Information and is included.

The arithmetic processing unit 340 has a microprocessor such as an MPU and its peripheral circuits. The arithmetic processing unit 340 reads the program 337 from the storage unit 330 and executes it, thereby realizing various processing units in cooperation with the hardware and the program 337. The main processing units realized by the arithmetic processing unit 340 include, for example, an image acquisition unit 341, a skeleton recognition unit 342, an actual measurement value calculation unit 343, a measurement unit 344, and an output unit 345.

The image acquisition unit 341 acquires a moving image (a plurality of image data) acquired by the smartphone 200 from the smartphone 200 via the communication I / F unit 320. Then, the image acquisition unit 341 stores the acquired image data in the storage unit 330 as image information 333 in association with, for example, information indicating the acquisition date and time and height of the image data.

In the case of the present embodiment, the image acquisition unit 341 associates or supports a moving image (image data) of a person walking in the left-right direction and a moving image (image data) of the person walking in the front-back direction from the smartphone 200. Get it so that it can be attached. Then, the image acquisition unit 341 associates the acquired two types of moving images and stores them in the storage unit 330 as image information 333.

The skeleton recognition unit 342 uses the trained model 331 to recognize the skeleton of the person whose walking posture is to be measured in the image data. For example, the skeleton recognition unit 342 recognizes each part such as the upper part of the spine, the right shoulder, the left shoulder, the right elbow, the left elbow, the right wrist, the left wrist, the right hand, the left hand, and so on. Further, the skeleton recognition unit 342 calculates the coordinates in the screen data of each recognized part. Then, the skeleton recognition unit 342 stores the recognition / calculation result in the storage unit 330 as skeleton information 334 for each person by associating it with the identification information for identifying the person.

The portion recognized by the skeleton recognition unit 342 corresponds to the trained model 331 (teacher data used when learning the trained model 331). Therefore, the skeleton recognition unit 342 may recognize a part other than the above-exemplified parts according to the trained model 331.

The measured value calculation unit 343 has a height h of the camera of the smartphone 200 from the floor surface, a vertical viewing angle θ of the camera of the smartphone 200, a horizontal viewing angle ψ, and a reference line screen having an arbitrary height on the screen. Based on the ratio α from half, the measured value W from one end of the screen to the other of the reference line is calculated. In addition, the measured value calculation unit 343 can calculate the stride length of a person or the like by using the calculated measured value W. Then, the actual measurement value calculation unit 343 stores the calculated actual measurement value W and the like as the actual measurement value information 335 in the storage unit 330. The actual measurement value calculation unit 343 can perform the process of calculating the actual measurement value W for each frame (that is, for each image data) in the moving image.

In general, only the plane position on the screen can be acquired from the image data such as the camera of the smartphone 200 whose depth value cannot be acquired, and the actual length in the real world cannot be immediately known. The measured value calculation unit 343 makes it possible to calculate the measured value W, which is the actual length (that is, the length in the real world), by using the various values described above.

FIG. 12 shows a mathematical formula used by the measured value calculation unit 343 to calculate the measured value W. As shown in FIG. 12, the actual measurement value calculation unit 343 calculates the actual measurement value W by solving the equation shown in Equation 1. That is, the measured value calculation unit 343 determines the height h of the camera of the smartphone 200 from the floor surface, the parameters of the camera of the smartphone 200, and the ratio α from half of the screen of the reference line for calculating the measured value W. The measured value W is calculated based on.

Here, the actually measured value W indicates, for example, the actual length of the reference line in the image data from the screen edge to the edge. Further, h is the height of the camera (smartphone 200) from the floor surface. The value of height h is input by the photographer when acquiring image data using the height information input unit 2123. Further, θ is a vertical viewing angle, and ψ is a horizontal viewing angle. α is the ratio of the reference line of any height on the screen from half the screen.

Hereinafter, Equation 1 will be described in more detail with reference to FIGS. 13 to 18.

For example, as shown in FIG. 13, it is assumed that a person on a reference line at an arbitrary height on the screen is photographed by using a camera of a smartphone 200 located at a height h. In this case, as shown in FIG. 14, the position of the smartphone 200 located at the height h is set as the origin O. Further, the point of contact with the floor when the smartphone 200 is lowered vertically toward the floor is defined as the point G, and the position of the person is defined as the point P. Further, the angle formed by the line connecting the origin O and the point P and the ground (or a line horizontal to the ground) is defined as an angle φ.

In the above case, assuming that the distance d from the point G to the point P is d, the number is 3 rather than 2 as shown in FIG.

Further, as shown in FIG. 16, the angle φ is the ratio of the positions of the arbitrary points P on the screen, and the vertical viewing angle θ is the equation 4.

Further, when the scene shown in FIG. 14 is viewed from directly above, it is as shown in FIG. In FIG. 17, one end of a line having an arbitrary height on the screen passing through the point P is designated as the point Q. In this case, as shown in FIG. 18, if the width between the point P and the point Q is the width w, the number is 6 rather than 5. Further, since the number 7 is more than the number 3 and the number 4, substituting the number 7 into the number 6 gives the number 8.

From the above, since the measured value W, which is the full length of the horizontal edge of the screen at the distance d from the smartphone 200, is 2w, the above-mentioned number 8 is doubled to the above-mentioned number 1. From the above, it can be seen that the measured value W can be calculated by calculating Equation 1.

Note that the reference line at an arbitrary height on the screen, which is the calculation source of the ratio α, can be specified based on, for example, the position of a person's foot in the image data. That is, the actually measured value calculation unit 343 can perform a process of calculating the ratio α after specifying the reference line based on the position of the foot of the person in the image data. For example, the measured value calculation unit 343 can use a line having a slope of 0 passing through the average value of the Y coordinates of the left and right feet of the person, the Y coordinate of any of the feet, and the like as a reference line (other than the example). It doesn't matter). The above processing by the actually measured value calculation unit 343 may be performed, for example, for each frame (that is, for each image data). Further, the height of the reference line on the screen may be predetermined according to the position of the guide line 2011 displayed by the guide line display unit 2121 or the like.

In addition, the measured value calculation unit 343 can calculate the stride length and the like based on the calculated measured value W. For example, as shown in FIG. 19, the image data in which the legs are open before and after walking, which is included in the moving image of a person walking in the left-right direction, the calculated measured value W, the resolution of the image data, and the like. Based on the above, the measured value calculation unit 343 can calculate the stride length. Specifically, for example, assuming that the resolution of the image data is 1920 × 1080, the measured value k per pixel on the reference line is k = W / 1920. Therefore, for example, when the difference between the x-coordinate values of the left and right feet (for example, the toe portion recognized by the skeleton recognition unit 342) in the image data shown in FIG. 19 is 100 pixels, the actual measurement value calculation unit 343 , The actual stride can be calculated by calculating 100 × k. In this way, the measured value calculation unit 343 can calculate the stride length based on the calculated measured value W, the resolution, and the number of pixels.

For example, as described above, the actual measurement value calculation unit 343 can calculate the actual measurement value W of the length from the screen edge to the end of the reference line, and can calculate the stride length of the person based on the actual measurement value W. ..

The measurement unit 344 measures the walking posture of the person by using the calculation result by the actual measurement value calculation unit 343 and the recognition result by the skeleton recognition unit 342. Then, the measurement unit 344 stores the measurement result and the like as the measurement result information 336 in the storage unit 330.

As described above, the measurement unit 344 can perform measurement based on a moving image of a person walking in the left-right direction and measurement based on a moving image of a person walking in the front-back direction. For example, the measuring unit 344 can calculate the walking speed, pitch, and the like based on a moving image (image data) of a person walking in the left-right direction. For example, the measuring unit 344 calculates the moving distance between frames of the portion recognized by the skeleton recognition unit 342, and calculates the walking speed based on the calculated moving distance and the time of each frame (image data). Can be done. When calculating the walking speed, the measuring unit 344 may use the calculation result by the measured value calculation unit 343 such as the stride length. Further, the measuring unit 344 can calculate the straightness and the like based on the moving image (image data) in which the person walks in the front-back direction. For example, the measuring unit 344 can calculate the straightness based on the fluctuation of the coordinates of the head recognized by the skeleton recognition unit 342.

Note that the measurement unit 344 may be configured to perform measurements other than those illustrated above.

The output unit 345 can output the skeleton information 334, the measured value information 335, the measurement result information 336, the moving image included in the image information 333, and the skeleton information 334 superimposed on the moving image. For example, the output by the output unit 345 is performed by displaying each of the above information on the screen display unit 310 or transmitting the information to an external device connected via the communication I / F unit 320.

The above is a configuration example of the walking posture measuring device 300.

Subsequently, an operation example of the walking posture measurement system 100 will be described with reference to FIGS. 20 to 22. First, an operation example of the angle adjustment information output unit 2124 will be described with reference to FIG.

FIG. 20 shows an operation example of the angle adjustment information output unit 2124. Referring to FIG. 20, when the smartphone 200 is tilted toward the front and back (step S101, Yes), the angle adjustment information output unit 2124 corrects the pitch according to the tilt (step S102). That is, the angle adjustment information output unit 2124 corrects the pitch more greatly as the smartphone 200 is tilted toward the front or the back. Then, the angle adjustment information output unit 2124 outputs the corrected sound (step S103).

Further, when the smartphone 200 is tilted in the left-right direction (step S104, Yes), the angle adjustment information output unit 2124 corrects the sound length according to the tilt (step S102). That is, the angle adjustment information output unit 2124 corrects the length of the sound more as the smartphone 200 is tilted to the left or right. Then, the angle adjustment information output unit 2124 outputs the corrected sound (step S103).

The sound output by the angle adjustment information output unit 2124 ends when the end condition is satisfied (step S107, Yes). The conditions for termination include, for example, that the shooting of the moving image has started, the shooting of the moving image has ended, a predetermined time has passed since the tilt of the smartphone 200 disappeared and the sounds matched, and the output was stopped by a person. And so on. The conditions for termination may be other than those illustrated above.

The above is an operation example of the angle adjustment information output unit 2124. Subsequently, an example of the overall operation of the walking posture measuring device 300 will be described with reference to FIG. FIG. 21 shows an overall operation example of the walking posture measuring device 300. Referring to FIG. 21, the image acquisition unit 341 acquires a moving image (a plurality of image data) acquired by the smartphone 200 from the smartphone 200 via the communication I / F unit 320 (step S201). In the case of the present embodiment, the image acquisition unit 341 associates or supports a moving image (image data) of a person walking in the left-right direction and a moving image (image data) of the person walking in the front-back direction from the smartphone 200. Get it so that it can be attached.

The skeleton recognition unit 342 recognizes the skeleton of the person whose walking posture is to be measured in the image data by using the trained model 331 (step S202).

The measuring unit 344 acquires the measured value W or the like indicated by the measured value information 335 (step S203). The actual measurement value W indicated by the actual measurement value information 335 may be calculated in advance by the actual measurement value calculation unit 343, for example, in parallel with the skeleton recognition process by the skeleton recognition unit 342, or by the skeleton recognition unit. After the recognition process by 342, it may be calculated by the actual measurement value calculation unit 343.

The measurement unit 344 measures the walking posture of the person by using the calculation result by the actual measurement value calculation unit 343 (step S204). For example, the measurement unit 344 performs measurement based on a moving image of a person walking in the left-right direction and measurement based on a moving image of a person walking in the front-back direction.

The output unit 345 outputs the skeleton information 334, the measured value information 335, the measurement result information 336, the moving image included in the image information 333, and the skeleton information 334 superimposed on the moving image (step S205).

The above is an example of the overall operation of the walking posture measuring device 300. Subsequently, with reference to FIG. 22, an example of the process in which the actually measured value calculation unit 343 calculates the actually measured value W will be described.

With reference to FIG. 22, the measured value calculation unit 343 acquires information indicating the height input by the height information input unit 2123 included in the image information 333. Further, the measured value calculation unit 343 acquires information indicating the vertical viewing angle θ and the horizontal viewing angle ψ of the camera with reference to the camera setting information 332 (step S301).

The measured value calculation unit 343 determines the reference line for calculating the measured value W and calculates the ratio α (step S302). For example, the measured value calculation unit 343 specifies the reference line based on the position of the foot of a person in the image data. Further, the measured value calculation unit 343 calculates the ratio α from the half of the screen of the reference line based on the specified reference line.

The measured value calculation unit 343 calculates the measured value W based on the height h, the vertical viewing angle θ, the horizontal viewing angle ψ, and the ratio α (step S303). For example, the actually measured value calculation unit 343 calculates the actually measured value W by solving the equation shown by the above-mentioned equation 1. Then, the actual measurement value calculation unit 343 stores the calculated actual measurement value W as the actual measurement value information 335 in the storage unit 330.

Further, the measured value calculation unit 343 takes the difference between the x-coordinate values of the left and right feet (for example, the toes) in the image data. Then, the stride is calculated based on the difference between the x-coordinate values, the actually measured value W, and the resolution (step S304).

The above is a processing example of the measured value calculation unit 343.

As described above, the walking posture measuring device 300 has a skeleton recognition unit 342 and a measuring unit 344. According to such a configuration, the measuring unit 344 can measure the walking posture based on the result of the skeleton recognition by the skeleton recognition unit 342. As a result, the measuring unit 344 can measure the walking posture based on the image data acquired by using the camera or the like of the smartphone 200 without using the depth sensor or the like.

Further, the walking posture measuring device 300 is configured to acquire a moving image showing a person walking in the front-back direction of the image and a moving image showing a person walking in the left-right direction of the image. According to such a configuration, the measuring unit 344 performs measurement based on a moving image showing a person walking in the front-back direction of the image, and also performs measurement based on a moving image showing a person walking in the left-right direction of the image. Can be done. As a result, the measuring unit 344 can measure various walking postures, which is difficult from one moving image, based on the image data acquired by using the camera or the like of the smartphone 200 without using the depth sensor or the like.

Further, the walking posture measuring device 300 has an actually measured value calculation unit 343. According to such a configuration, the measured value calculation unit 343 can calculate the measured value. As a result, it becomes possible to calculate the stride length and the like based on the moving image taken by the camera of the smartphone 200. As a result, the walking posture measuring device 300 can accurately measure the walking posture based on the moving image taken by the camera of the smartphone 200.

In addition, the smartphone 200 has a shooting assistance unit 212. According to such a configuration, the smartphone 200 includes a video showing a person walking in the front-back direction of the image and a video showing a person walking in the left-right direction of the image with the assistance of the shooting assistance unit 212. Can be obtained. As a result, the smartphone 200 can acquire the above two types of moving images while keeping the shooting conditions as close as possible. Thereby, when the measurement using two kinds of moving images is performed, the accuracy of the measurement by the measuring unit 344 can be improved. Further, according to the above configuration, it is possible to arrange the shooting conditions when acquiring the moving image at different timings. That is, it is possible to arrange the shooting conditions when shooting the image data in the same direction acquired at various timings. As a result, it is possible to compare image data in the same direction with high accuracy, and it is possible to improve the accuracy of analysis.

Further, the photographing auxiliary unit 212 has an angle adjustment information output unit 2124. According to such a configuration, the angle adjustment information output unit 2124 can output information such as a sound for adjusting the inclination of the smartphone 200. As a result, the angle of the smartphone 200 can be easily adjusted even when it is difficult to see the screen of the smartphone 200. As a result, it becomes easy to arrange the shooting conditions, and the accuracy of the measurement by the measuring unit 344 can be easily improved.

In addition, in this embodiment, the case where the smartphone 200 is used as the photographing device is illustrated. However, the moving image may be acquired using a photographing device other than the smartphone 200. That is, the photographing device included in the walking posture measuring system 100 is not limited to the smartphone 200.

Further, in the present embodiment, a case where the function as the walking posture measuring device 300 is realized by one information processing device is illustrated. However, the function of the walking posture measuring device 300 may be realized by, for example, a plurality of information processing devices connected via a network. In other words, the function of the walking posture measuring device 300 is not limited to the case where it is realized by one information processing device, and may be realized, for example, on the cloud.

Further, the photographing assisting unit 212 may have all of the plurality of functions illustrated in FIG. 5, or may have some (at least one) of the functions illustrated in FIG. For example, the photographing assisting unit 212 may have only the function as the angle adjustment information output unit 2124 without displaying the angle information 2012 by the angle information display unit 2122.

Further, the auxiliary function by the photographing assisting unit 212 described in the present embodiment may be applied to a system other than the walking posture measurement system 100. The auxiliary function by the shooting auxiliary unit 212 may be applied to various scenes in which it is necessary to arrange the shooting conditions as much as possible when acquiring the image data. Similarly, the calculation process of the actual measurement value W by the actual measurement value calculation unit 343 may be applied to a system other than the walking posture measurement system 100. The calculation process of the measured value W by the measured value calculation unit 343 can be applied to various situations in which the measured value is calculated based on the image data.

[Second Embodiment]
Next, a second embodiment of the present invention will be described with reference to FIGS. 23 to 28. FIG. 23 is a diagram for explaining a tracking example of the walking posture measuring device 300. FIG. 24 is a block diagram showing a configuration example of the walking posture measuring device 300 according to the second embodiment. FIG. 25 is a diagram showing a configuration example of the tracking unit 346. 26 and 27 are diagrams showing an example of a figure generated by the inclusion figure generation unit 3461. FIG. 28 is a flowchart showing an operation example of the tracking unit 346.

In the second embodiment of the present disclosure, a modified example of the walking posture measuring device 300 described in the first embodiment will be described. For example, as shown in FIG. 23, when a plurality of people exist in the image data, if the same person is not properly tracked between the frames, the walking speed using the moving distance between the frames can be calculated accurately. You will not be able to do it. Therefore, in the case of the present embodiment, in addition to the configuration of the walking posture measuring device 300 described in the first embodiment, the tracking unit 346 is provided. The tracking unit 346 is configured to perform tracking based on the result of recognition by the skeletal recognition unit 342.

FIG. 24 shows a configuration example of the walking posture measuring device 300 according to the second embodiment. Referring to FIG. 24, the walking posture measuring device 300 has a tracking unit 346 in addition to the configuration described in the first embodiment. Hereinafter, a configuration characteristic of the present embodiment will be described.

The tracking unit 346 tracks the person based on the recognition result by the skeleton recognition unit 342. For example, the tracking unit 346 tracks a person by assigning a recognition number to the recognized person. That is, the tracking unit 346 tracks the person by assigning the same recognition number to the person determined to be the same between the image data one frame before and the image data of the current frame. FIG. 25 shows a more detailed configuration example of the tracking unit 346. Referring to FIG. 25, the tracking unit 346 includes, for example, an inclusion figure generation unit 3461, an average skeleton coordinate calculation unit 3462, and a comparison tracking unit 3464.

The inclusion figure generation unit 3461 generates a figure including the coordinates of all the parts recognized by the skeleton recognition unit 342 (coordinates included in the skeleton information 334) for each person. For example, the inclusion figure generation unit 3461 generates one of the smallest, convex hull, rectangle, and circle that includes all the coordinates. In addition, the inclusion figure generation unit 3461 calculates the area of the generated figure.

For example, the inclusion figure generation unit 3461 performs the generation of the inclusion figure and the calculation process of the area for the image data of each frame. When a plurality of persons are included in the image data of one frame, the inclusion figure generation unit 3461 performs the generation of the inclusion figure and the calculation process of the area for each of the plurality of persons included in the image data.

FIG. 26 shows an example of a figure including the coordinates of a person walking in the front-back direction of the screen. As shown in FIG. 26, the inclusion figure generation unit 3461 can generate any of the smallest convex hulls, rectangles, and circles that include all the coordinates. Further, FIG. 27 shows an example of a figure including the coordinates of a person walking in the left-right direction of the screen. In the case shown in FIG. 27 as in the case shown in FIG. 26, the inclusion figure generation unit 3461 can generate any one of the smallest convex hull, rectangle, and circle that includes all the coordinates.

It should be noted that which of the convex hull, the rectangle, and the circle the inclusion figure generation unit 3461 generates is predetermined, for example. For example, the inclusion figure generation unit 3461 is defined to generate a circle as a figure containing coordinates.

The average skeleton coordinate calculation unit 3462 calculates the average value of the coordinates (coordinates included in the skeleton information 334) of all the parts recognized by the skeleton recognition unit 342 for each person. As a result, the average skeleton coordinate calculation unit 3462 calculates the average skeleton coordinates based on the coordinates of each part of the person recognized by the skeleton recognition unit 342.

Similar to the case of the inclusion figure generation unit 3461, the average skeleton coordinate calculation unit 3462 performs the calculation process of the average skeleton coordinates for the image data of each frame. When a plurality of persons are included in the image data of one frame, the average skeleton coordinate calculation unit 3462 performs the calculation process of the average skeleton coordinates for each of the plurality of persons included in the image data.

The comparison tracking unit 3464 tracks a person based on the area of the included figure calculated by the inclusion figure generation unit 3461 and the average skeleton coordinates calculated by the average skeleton coordinate calculation unit 3462. For example, the comparison tracking unit 3464 compares the area corresponding to the person to be tracked calculated in the current frame and the area calculated one frame before. Further, the comparison tracking unit 3464 performs a comparison between the average skeleton coordinates corresponding to the person to be tracked and the average skeleton coordinates calculated one frame before, according to the result of the comparison.

Specifically, for example, when one of the areas corresponding to each person in the previous frame has a difference from the area corresponding to the person to be tracked in the current frame within the first permissible value. The comparison tracking unit 3464 determines that the person one frame before and the person in the current frame whose area difference belongs to the first allowable value are the same person. As a result, the comparison tracking unit 3464 sets, for example, the recognition number corresponding to the person one frame before determined to be the same as the recognition number corresponding to the person in the current frame.

Further, when a plurality of areas in which the difference from the area to be tracked is within the first permissible value are calculated one frame before, the comparison tracking unit 3464 has a difference between the average skeleton coordinates of the person to be tracked and the area. A comparison is made with the average skeletal coordinates of the person one frame before, which was within the first permissible value. Then, the comparison tracking unit 3464 determines that the person one frame before and the person in the current frame whose difference in the average skeleton coordinates belongs to the second allowable value are the same person. As a result, the comparison tracking unit 3464 sets, for example, the recognition number corresponding to the person one frame before determined to be the same person as the recognition number corresponding to the person in the current frame. When there are a plurality of persons whose average skeleton coordinate difference is within the second allowable value, the comparison tracking unit 3464 determines that the person whose average skeleton coordinate difference is closest to the second allowable value is the same person. Can be done. When there are a plurality of persons whose average skeleton coordinate difference belongs to the second allowable value or less, the comparison tracking unit 3464 may be configured to perform a process other than the above example, such as determining an error.

In addition, when the area where the difference from the tracked area is within the first allowable value is not calculated at all one frame before, and when the difference between the average skeleton coordinates is within the second allowable value one frame before. If the person is not present, the comparison tracking unit 3464 determines that the person to be tracked is a newly recognized person. In this case, the comparison tracking unit 3464 assigns a new recognition number to the newly recognized person.

The comparison tracking unit 3464 can use, as the first allowable value, an estimated area value estimated based on the rate of increase / decrease in the area corresponding to the person determined to be the same person in the past several frames. .. The first permissible value may be a predetermined value.

Further, the comparison tracking unit 3464 can use, as the second allowable value, an estimated coordinate value estimated from the moving speed of the average skeleton coordinates corresponding to the person determined to be the same person in the past several frames. The second permissible value may be a predetermined value like the first permissible value.

The above is a configuration example of the tracking unit 346. Subsequently, an operation example of the tracking unit 346 will be described with reference to FIG. 28.

With reference to FIG. 28, the inclusion figure generation unit 3461 generates a figure including the coordinates of all the parts recognized by the skeleton recognition unit 342 (coordinates included in the skeleton information 334) for each person. In addition, the inclusion figure generation unit 3461 calculates the area of the generated figure (step S401).

The average skeleton coordinate calculation unit 3462 calculates the average skeleton coordinates by calculating the average value of the coordinates (coordinates included in the skeleton information 334) of all the parts recognized by the skeleton recognition unit 342 for each person (step S402). ).

The comparison tracking unit 3464 compares the area corresponding to the person to be tracked calculated in the current frame and the area calculated one frame before (step S403).

When one of the areas of each person in the previous frame has a difference from the area corresponding to the person to be tracked in the current frame within the first permissible value (step S403, 1), the comparative tracking unit. 3463 determines that the person one frame before and the person in the current frame whose area difference belongs to the first permissible value are the same person (step S404). As a result, the comparison tracking unit 3464 sets the recognition number corresponding to the person one frame before as the recognition number of the current frame, for example.

Further, when a plurality of areas in which the difference from the area to be tracked is within the first permissible value are calculated one frame before (step S403, a plurality), the comparative tracking unit 3464 has the average skeletal coordinates of the person to be tracked. And the average skeleton coordinates of the person one frame before the difference in area was within the first permissible value (step S405). Then, when there is a person whose difference from the average skeleton coordinates of the tracking target is within the second allowable value one frame before (step S405, Yes), the comparative tracking unit 3464 is the person one frame before and the tracking target. It is determined that the person is the same person (step S404). When there are a plurality of persons whose difference from the average skeleton coordinates of the tracking target is within the second allowable value one frame before, the comparison tracking unit 3464 uses, for example, the same person whose difference is closest to the second allowable value. It can be judged that it is a person. Further, when there is no person whose difference from the average skeleton coordinates of the tracking target is within the second allowable value one frame before (step S405, No), the comparative tracking unit 3464 newly recognizes the person to be tracked. It is determined that the person is a person (step S406). In this case, the comparison tracking unit 3464 assigns a new recognition number to the newly recognized person.

Further, when the area where the difference from the area to be tracked is within the first permissible value is not calculated at all one frame before (steps S403, 0), the comparative tracking unit 3464 newly sets the person to be tracked. (Step S406). In this case, the comparison tracking unit 3464 assigns a new recognition number to the newly recognized person.

The above is an operation example of the tracking unit 346.

As described above, the walking posture measuring device 300 in the present embodiment has a tracking unit 346. With such a configuration, the tracking unit 346 can track the same person based on the recognition result by the skeleton recognition unit 342. As a result, it is possible to suppress the erroneous calculation of the walking speed, and it is possible to improve the accuracy of the walking posture measurement.

In the present embodiment, a case where the area of the included figure generated by the included figure generation unit 3461 is calculated has been illustrated. However, the inclusion figure generation unit 3461 may be configured to calculate a value other than the area based on the generated inclusion figure. For example, the inclusion figure generation unit 3461 may calculate the height, diameter, and the like of the generated inclusion figure instead of the area. In this case, the comparison tracking unit 3464 will compare values such as the height calculated based on the figure generated by the inclusion figure generation unit 3461 instead of the area.

In addition, the inclusion figure generation unit 3461 is supposed to generate a figure included in all coordinates. However, the included figure generation unit 3461 may be configured to generate a figure including a part of the coordinates recognized by the skeleton recognition unit 342, for example, generating a figure included in the coordinates corresponding to the upper body of the person. I do not care. Further, the average skeleton coordinate calculation unit 3462 may also calculate the average coordinates based on a part of the coordinates recognized by the skeleton recognition unit 342, such as the average coordinates of the coordinates corresponding to the upper body of the person.

Further, the comparison tracking unit 3464 may be configured to perform only one of the comparison of the values based on the inclusion figure generated by the inclusion figure generation unit 3461 and the comparison of the average skeleton coordinates.

Further, the tracking using the recognition result by the skeleton recognition unit 342 described in the present embodiment may be applied to the case of tracking a person other than the scene of measuring the walking posture. That is, the function as the tracking unit 346 may be applied to a device other than the walking posture measuring device 300 that requires tracking a person. As described above, the method of tracking a person using skeletal information described in the present embodiment is not limited to the case of measuring the walking posture, and can be used in various situations.

In addition, in the case of this embodiment as well, various modified examples may be adopted as in the first embodiment.

[Third Embodiment]
Next, a third embodiment of the present invention will be described with reference to FIGS. 29 and 30. 29 and 30 show a configuration example of the photographing apparatus 400.

The photographing device 400 photographs a person walking. FIG. 29 shows an example of the hardware configuration of the photographing device 400. Referring to FIG. 29, the photographing apparatus 400 has the following hardware configuration as an example in addition to a camera for performing photographing.
-CPU (Central Processing Unit) 401 (arithmetic unit)
-ROM (Read Only Memory) 402 (storage device)
-RAM (Random Access Memory) 403 (storage device)
-Program group 404 loaded into RAM 403
A storage device 405 that stores a program group 404.
-Drive device 406 that reads and writes the recording medium 410 external to the information processing device.
-Communication interface 407 that connects to the communication network 411 outside the information processing device
-I / O interface 408 that inputs and outputs data
-Bus 409 connecting each component

Further, the photographing device 400 can realize the functions as the detection unit 421 and the display unit 422 shown in FIG. 30 by the CPU 401 acquiring the program group 404 and executing the program group 404. The program group 404 is stored in, for example, a storage device 405 or a ROM 402 in advance, and the CPU 401 loads the program group 404 into a RAM 403 or the like and executes the program group 404 as needed. Further, the program group 404 may be supplied to the CPU 401 via the communication network 411, or may be stored in the recording medium 410 in advance, and the drive device 406 may read the program and supply the program to the CPU 401.

Note that FIG. 29 shows an example of the hardware configuration of the photographing device 400. The hardware configuration of the photographing device 400 is not limited to the above-mentioned case. For example, the photographing device 400 may be composed of a part of the above-described configuration, such as not having the drive device 406.

The detection unit 421 detects the orientation of the photographing device 400. For example, the detection unit 421 detects whether the photographing device 400 is in the vertical orientation or in the horizontal orientation.

The display unit 422 displays a guide line indicating the position where the person walks on the screen display unit. For example, the display unit 422 displays different guide lines according to the orientation of the photographing device 400 detected by the detection unit 421.

As described above, the photographing device 400 has a detection unit 421 and a display unit 422. With such a configuration, the display unit 422 can display different guide lines according to the orientation of the photographing device 400 detected by the detection unit 421. As a result, when acquiring image data according to the orientation of the photographing device 400, it is possible to provide appropriate assistance according to each. As a result, when acquiring a plurality of image data, it is possible to make the shooting conditions as uniform as possible.

The above-mentioned photographing device 400 can be realized by incorporating a predetermined program into the photographing device 400. Specifically, in the program according to another embodiment of the present invention, the photographing device 400 has a detection unit 421 that detects the direction of the photographing device and a display unit that displays a guide line indicating a position where a person walks on the screen display unit. The display unit 422 is a program that realizes 422 and displays the guide lines that differ depending on the orientation of the photographing device detected by the detection unit 421.

Further, in the guide method executed by the above-described photographing device 400, the photographing device 400 that photographs the state of walking of a person detects the direction of the photographing device and displays a guide line indicating the position where the person walks on the screen display unit. When displaying and displaying the guide line on the screen display unit, a different guide line is displayed according to the detected orientation of the photographing device.

Even the invention of the program or the guide method having the above-mentioned configuration can achieve the above-mentioned object of the present invention because it has the same action and effect as the above-mentioned photographing apparatus 400.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described with reference to FIG. FIG. 31 shows
A configuration example of the information processing device 500 is shown.

The information processing device 500 has, for example, a configuration similar to the hardware configuration of the photographing device 400 described with reference to FIG. 29. Further, the information processing device 500 can realize the function as the calculation unit 521 shown in FIG. 31 by the CPU acquiring the program group included in the information processing device 500 and executing the program group by the CPU.

The calculation unit 521 calculates the actual length at a predetermined position in the image data based on the parameters of the photographing device for acquiring the image data and the information indicating the height of the photographing device when acquiring the image data. do.

As described above, the information processing device 500 has a calculation unit 521. With such a configuration, the calculation unit 521 can calculate the actual length at a predetermined position in the image data based on various information. As a result, it becomes possible to perform analysis using the actual length based on the image data acquired by a photographing device such as a smartphone.

The information processing device 500 described above can be realized by incorporating a predetermined program into the information processing device 500. Specifically, the program according to another embodiment of the present invention provides the information processing device 500 with information indicating the parameters of the photographing device for acquiring image data and the height of the photographing device when acquiring the image data. Based on this, it is a program for realizing the calculation unit 521 that calculates the actual length at a predetermined position in the image data.

Further, in the calculation method executed by the information processing device 500 described above, the information processing device 500 obtains the parameters of the photographing device for acquiring the image data and the information indicating the height of the photographing device when acquiring the image data. It is a method of acquiring and calculating the actual length at a predetermined position in the image data based on the acquired information.

Even the invention of the program or the calculation method having the above-mentioned configuration can achieve the above-mentioned object of the present invention because it has the same action and effect as the above-mentioned information processing apparatus 500.

[Fifth Embodiment]
Next, a fifth embodiment of the present invention will be described with reference to FIG. FIG. 32 shows a configuration example of the information processing device 600.

The information processing device 600 has, for example, a configuration similar to the hardware configuration of the photographing device 400 described with reference to FIG. 29. Further, the information processing device 600 can realize the functions as the detection unit 621 and the output unit 622 shown in FIG. 32 by the CPU acquiring the program group possessed by the information processing device 600 and executing the program group by the CPU. You can.

The detection unit 621 detects the tilt of the information processing device.

The output unit 622 outputs information according to the inclination of the information processing device detected by the detection unit 621, which differs depending on the inclination of the information processing device.

As described above, the information processing device 600 has a detection unit 621 and an output unit 622. According to such a configuration, the output unit 622 can output information according to the inclination of the information processing device detected by the detection unit 621. As a result, the operator who operates the information processing apparatus 600 can correct the inclination according to the output information.

The information processing device 600 described above can be realized by incorporating a predetermined program into the information processing device 600. Specifically, in the program according to another embodiment of the present invention, the information processing device 600 has a detection unit 621 that detects the tilt of the information processing device 600 and a detection unit 621 that differs depending on how the information processing device 600 is tilted. This is a program for realizing an output unit 622 that outputs information according to the inclination of the information processing apparatus 600 detected by the information processing apparatus 600.

Further, in the calculation method executed by the information processing device 600 described above, the information processing device 600 detects the tilt of the information processing device 600, and the detected tilt of the information processing device differs depending on how the information processing device is tilted. It is a method of outputting information according to.

Even the invention of the program or the output method having the above-mentioned configuration can achieve the above-mentioned object of the present invention because it has the same operation and effect as the above-mentioned information processing apparatus 600.

[Sixth Embodiment]
Next, a sixth embodiment of the present invention will be described with reference to FIG. 33. FIG. 33 shows a configuration example of the tracking device 700.

The tracking device 700 has, for example, a configuration similar to the hardware configuration of the photographing device 400 described with reference to FIG. 29. Further, the tracking device 700 can realize the functions as the acquisition unit 721 and the tracking unit 722 shown in FIG. 33 by the CPU acquiring the program group included in the tracking device 700 and executing the program group.

The acquisition unit 721 acquires information indicating a plurality of parts of the recognized person by recognizing the skeleton of the person in the image data.

The tracking unit 722 tracks the same person among a plurality of image data based on the information acquired by the acquisition unit 721.

As described above, the tracking device 700 has an acquisition unit 721 and a tracking unit 722. With such a configuration, the tracking unit 722 can perform tracking based on the information indicating the portion acquired by the acquisition unit 721. This makes it possible to realize easy tracking.

The tracking device 700 described above can be realized by incorporating a predetermined program into the tracking device 700. Specifically, in the program according to another embodiment of the present invention, the tracking device 700 has an acquisition unit 721 that acquires information indicating a plurality of recognized parts of the person by recognizing the skeleton of the person in the image data. This is a program for realizing a tracking unit 722 that tracks the same person among a plurality of image data based on the information acquired by the acquisition unit 721.

Further, in the tracking method executed by the tracking device 700 described above, the tracking device 700 acquires information indicating a plurality of parts of the recognized person by recognizing the skeleton of the person in the image data, and the acquired information is used. Based on this, it is a method of tracking the same person among a plurality of image data.

Even the invention of the program or the tracking method having the above-mentioned configuration can achieve the above-mentioned object of the present invention because it has the same action and effect as the above-mentioned tracking device 700.

<Additional notes>
Part or all of the above embodiments may also be described as in the appendix below. Hereinafter, the outline of the guide method and the like in the present invention will be described. However, the present invention is not limited to the following configurations.

(Appendix 1)
A shooting device that shoots a person walking
Detects the orientation of the shooting device and
A guide line indicating the position where the person walks is displayed on the screen display.
A guide method for displaying the guide lines that differ depending on the direction of the detected photographing device when the guide lines are displayed on the screen display unit.
(Appendix 2)
The guide method described in Appendix 1
A guide method for displaying different guide lines depending on whether the photographing device is in portrait orientation or landscape orientation.
(Appendix 3)
The guide method described in Appendix 1 or Appendix 2,
A guide method for displaying the guide line for guiding a person walking in the front-back direction of the screen when the photographing device is oriented vertically.
(Appendix 4)
The guide method according to any one of Supplementary note 1 to Supplementary note 3.
A guide method that displays a guide line to guide a person walking in the left-right direction of the screen when the shooting device is in landscape orientation.
(Appendix 5)
The guide method according to any one of Supplementary note 1 to Supplementary note 4.
A guide method that detects the tilt of the imaging device and displays information indicating the detected tilt on the screen display.
(Appendix 6)
The guide method according to any one of Supplementary note 1 to Supplementary note 5.
A guide method that detects the tilt of the imaging device and outputs information according to the detected tilt.
(Appendix 7)
The guide method described in Appendix 6
A guide method that detects the tilt in the left-right direction and the tilt in the front-back direction, and outputs different information when the tilt in the left-right direction is detected and when the tilt in the front-back direction is detected.
(Appendix 8)
The guide method according to Appendix 6 or Appendix 7.
A guide method that outputs the sound adjusted by different methods when the tilt in the left-right direction is detected and when the tilt in the front-back direction is detected.
(Appendix 9)
It is a shooting device that shoots a person walking.
A detector that detects the orientation of the imaging device and
A display unit that displays a guide line indicating the position where a person walks on the screen display unit,
Have,
The display unit is an imaging device that displays different guide lines according to the orientation of the imaging device detected by the detection unit.
(Appendix 10)
For a shooting device that shoots a person walking
A detector that detects the orientation of the imaging device and
A display unit that displays a guide line indicating the position where a person walks on the screen display unit,
Realized,
The display unit is a program that displays the guide lines that differ depending on the orientation of the photographing device detected by the detection unit.
(Appendix 11)
Information processing device
The parameters of the photographing device for acquiring the image data and the information indicating the height of the photographing device when acquiring the image data are acquired, and based on the acquired information, the actual length at a predetermined position in the image data is obtained. Calculation method to calculate.
(Appendix 12)
The calculation method described in Appendix 11.
A calculation method in which a reference line for calculating a length is determined based on a predetermined standard, and the actual length of the determined reference line from end to end of image data is calculated.
(Appendix 13)
The calculation method described in Appendix 12,
A calculation method for determining the reference line based on the position of a person's foot in image data.
(Appendix 14)
The calculation method according to Appendix 12 or Appendix 13.
The ratio of the position of the reference line on the screen indicated by the image data from half the screen is calculated, and the actual length of the reference line is calculated based on the calculated ratio, the parameter, and the height. Calculation method.
(Appendix 15)
The calculation method according to any one of Supplementary note 11 to Supplementary note 14.
The calculation method includes information indicating the vertical viewing angle and the horizontal viewing angle of the photographing apparatus.
(Appendix 16)
The calculation method according to any one of Supplementary note 11 to Supplementary note 15.
The actual length based on Equation 1 using the vertical viewing angle θ of the photographing device, the horizontal viewing angle ψ, the ratio α of the position of the reference line determined on the screen from half the screen, and the height h. Calculation method for calculating W.
(Appendix 17)
The calculation method according to any one of Supplementary note 11 to Supplementary note 16.
A calculation method for calculating the stride length of a person based on the calculated length and the resolution of image data.
(Appendix 18)
The calculation method described in Appendix 17,
Get the number of pixels between the left and right feet of a person in the image data
A calculation method that calculates the stride length of a person based on the calculated length, the resolution of the image data, and the number of acquired pixels.
(Appendix 19)
It has a calculation unit that calculates the actual length at a predetermined position in the image data based on the parameters of the photographing device for acquiring the image data and the information indicating the height of the photographing device when acquiring the image data. Information processing device.
(Appendix 20)
For information processing equipment
Realized a calculation unit that calculates the actual length at a predetermined position in the image data based on the parameters of the imaging device that acquires the image data and the information that indicates the height of the imaging device when acquiring the image data. Program to do.
(Appendix 21)
Information processing device
Detects the tilt of the information processing device and
An output method that outputs information according to the tilt of the information processing device, which differs depending on the tilt of the information processing device.
(Appendix 22)
The output method described in Appendix 21.
Detects the tilt of the information processing device in the left-right direction and the tilt in the front-back direction,
An output method that outputs different information when the tilt in the left-right direction is detected and when the tilt in the front-back direction is detected.
(Appendix 23)
The output method according to Appendix 21 or Appendix 22.
An output method that outputs sound adjusted by different methods when the tilt in the left-right direction is detected and when the tilt in the front-back direction is detected.
(Appendix 24)
The output method described in Appendix 23.
When the tilt in the left-right direction is detected, the sound with either the pitch or the length of the sound adjusted is output, and when the tilt in the front-back direction is detected, the tilt in the left-right direction of the pitch and the length of the sound is output. An output method that outputs a sound adjusted in a different way than when it was detected.
(Appendix 25)
The output method according to any one of Supplementary note 21 to Supplementary note 24.
Outputs two types of sound when the information processing device is tilted,
An output method that outputs one type of sound when the information processing device is not tilted.
(Appendix 26)
The output method according to any one of Supplementary note 1 to Supplementary note 25.
An output method that displays information indicating the tilt of the information processing device on the screen display.
(Appendix 27)
A detector that detects the tilt of the information processing device and
An output unit that outputs information according to the inclination of the information processing device detected by the detection unit, which differs depending on the inclination of the information processing device.
Information processing device with.
(Appendix 28)
The information processing device according to Appendix 27.
The detection unit detects the tilt of the information processing device in the left-right direction and the tilt in the front-back direction.
The output unit is an information processing device that outputs different information depending on whether the detection unit detects a tilt in the left-right direction or a tilt in the front-back direction.
(Appendix 29)
For information processing equipment
A detector that detects the tilt of the information processing device and
An output unit that outputs information according to the inclination of the information processing device detected by the detection unit, which differs depending on the inclination of the information processing device.
A program to realize.
(Appendix 30)
The program described in Appendix 29.
The detection unit detects the tilt of the information processing device in the left-right direction and the tilt in the front-back direction.
The output unit is a program that outputs different information depending on whether the detection unit detects a tilt in the left-right direction or a tilt in the front-back direction.
(Appendix 31)
Information processing device
By recognizing the skeleton of a person in the image data, information indicating multiple parts of the recognized person is acquired, and information is obtained.
A tracking method that tracks the same person among multiple image data based on the acquired information.
(Appendix 32)
The tracking method described in Appendix 31.
Generate an inclusion figure that includes at least a part of the recognized part,
A tracking method for tracking the same person based on a value corresponding to the generated inclusion figure.
(Appendix 33)
The tracking method described in Appendix 32.
A tracking method for tracking the same person based on the difference in values between image data according to the included figure.
(Appendix 34)
The tracking method described in Appendix 33.
The number at which the difference between the value corresponding to the included figure of the tracking target and the value corresponding to the included figure corresponding to the person included in the image data different from the image data to which the tracking target belongs is within a predetermined value. If there is one, a tracking method that determines that a person whose difference is within a predetermined value is the same person as the person to be tracked.
(Appendix 35)
The tracking method described in Appendix 34.
The predetermined value is a tracking method determined according to the degree of change in the value according to the included figure among a plurality of image data.
(Appendix 36)
The tracking method according to any one of Appendix 31 to Appendix 35.
Calculate the average value of the coordinates of at least a part of the coordinates of the recognized part,
A tracking method that tracks the same person based on the calculated results.
(Appendix 37)
The tracking method described in Appendix 36.
A tracking method for tracking the same person based on the difference in the average value between image data.
(Appendix 38)
The tracking method according to Appendix 36 or Appendix 37.
A tracking method for tracking the same person based on the difference between the average value of the tracking target and the average value corresponding to a person included in image data different from the image data to which the tracking target belongs.
(Appendix 39)
An acquisition unit that acquires information indicating multiple parts of the recognized person by recognizing the skeleton of the person in the image data, and an acquisition unit.
A tracking unit that tracks the same person among a plurality of image data based on the information acquired by the acquisition unit.
Tracking device with.
(Appendix 40)
In the tracking device,
An acquisition unit that acquires information indicating multiple parts of the recognized person by recognizing the skeleton of the person in the image data, and an acquisition unit.
A tracking unit that tracks the same person among a plurality of image data based on the information acquired by the acquisition unit.
A program to realize.

Note that the programs described in each of the above embodiments and appendices may be stored in a storage device or recorded in a computer-readable recording medium. For example, the recording medium is a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

Although the invention of the present application has been described above with reference to each of the above embodiments, the invention of the present application is not limited to the above-described embodiment. Various changes that can be understood by those skilled in the art can be made to the structure and details of the present invention within the scope of the present invention.

The present invention enjoys the benefit of priority claim based on the patent application of Japanese Patent Application No. 2020-053974 filed in Japan on March 25, 2020, and is described in the patent application. All contents are included in this specification.

100 Walking posture measurement system 200 Smartphone 210 Measurement operation Imaging unit 211 Image data imaging unit 212 Imaging auxiliary unit 2121 Guide line display unit 2122 Angle information display unit 2123 Height information input unit 2124 Angle adjustment information output unit 201 Touch panel 2011 Guide line 2012 Angle information 2013 Height display unit 300 Walking posture measuring device 310 Screen display unit 320 Communication I / F unit 330 Storage unit 331 Learned model 332 Camera setting information 333 Image information 334 Skeleton information 335 Measured value information 336 Measurement result information 337 Program 340 Arithmetic processing unit 341 Image acquisition unit 342 Skeleton recognition unit 343 Measured value calculation unit 344 Measurement unit 345 Output unit 346 Tracking unit 3461 Inclusion figure generation unit 3462 Average skeleton coordinate calculation unit 3464 Comparison tracking unit 400 Imaging device 401 CPU
402 ROM
403 RAM
404 Program group 405 Storage device 406 Drive device 407 Communication interface 408 Input / output interface 409 Bus 410 Recording medium 411 Communication network 421 Detection unit 422 Display unit 500 Information processing device 521 Calculation unit 600 Information processing device 621 Detection unit 622 Output unit 700 Tracking device 721 Acquisition unit 722 Tracking unit

Claims

A shooting device that shoots a person walking
Detects the orientation of the shooting device and
A guide line indicating the position where the person walks is displayed on the screen display.
A guide method for displaying the guide lines that differ depending on the direction of the detected photographing device when the guide lines are displayed on the screen display unit.
The guide method according to claim 1.
A guide method for displaying different guide lines depending on whether the photographing device is in portrait orientation or landscape orientation.
The guide method according to claim 1 or 2.
A guide method for displaying the guide line for guiding a person walking in the front-back direction of the screen when the photographing device is oriented vertically.
The guide method according to any one of claims 1 to 3.
A guide method that displays a guide line to guide a person walking in the left-right direction of the screen when the shooting device is in landscape orientation.
The guide method according to any one of claims 1 to 4.
A guide method that detects the tilt of the imaging device and displays information indicating the detected tilt on the screen display.
The guide method according to any one of claims 1 to 5.
A guide method that detects the tilt of the imaging device and outputs information according to the detected tilt.
The guide method according to claim 6.
A guide method that detects the tilt in the left-right direction and the tilt in the front-back direction, and outputs different information when the tilt in the left-right direction is detected and when the tilt in the front-back direction is detected.
The guide method according to claim 6 or 7.
A guide method that outputs the sound adjusted by different methods when the tilt in the left-right direction is detected and when the tilt in the front-back direction is detected.
It is a shooting device that shoots a person walking.
A detector that detects the orientation of the imaging device and
A display unit that displays a guide line indicating the position where a person walks on the screen display unit,
Have,
The display unit is an imaging device that displays different guide lines according to the orientation of the imaging device detected by the detection unit.
For a shooting device that shoots a person walking
A detector that detects the orientation of the imaging device and
A display unit that displays a guide line indicating the position where a person walks on the screen display unit,
Realized,
The display unit is a computer-readable recording medium on which a program for displaying the guide lines, which differs depending on the orientation of the photographing device detected by the detection unit, is recorded.