JP7188422B2 - Image processing device, analysis system, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an analysis system, an image processing method, and a program.

BACKGROUND ART In recent years, various techniques for analyzing the motion of a subject have been studied for the purpose of improving sports skills and the like.
For example, Patent Literature 1 discloses a technique in the field of swing motion analysis technology that acquires information about the motion of a subject using a sensor, generates an index image according to the sensor information, and displays it to the user.

JP 2015-002913 A

However, in the technique described in Patent Document 1, although it is possible to generate an index image based on information obtained from a sensor worn by a subject, the generated index image alone does not Movement is difficult to represent adequately.
As described above, in the conventional technology, it is difficult to present the measurement result of the movement of the subject in an easy-to-understand manner.

The present invention has been made in view of such a situation, and an object of the present invention is to present the measurement result of the movement of the subject in an easy-to-understand manner.

In order to achieve the above object, an image processing apparatus according to one aspect of the present invention includes a plurality of operation modes prepared in advance, including a first mode in which a photographing apparatus is caused to sequentially photograph predetermined actions of a subject. and a mode that is started based on the detection of an external operation while the sequential shooting is being performed in the first mode, and the sequential shooting is performed when the start timing is detected based on the measurement result of the predetermined action acquired by the sensor. and a second mode for starting moving image recording of the predetermined operation is set, the photographing device is controlled to photograph at a photographing speed based on the set operation mode. and image acquisition means for acquiring an image of the predetermined action;
measurement result acquisition means for acquiring a measurement result of the movement of the subject measured by the sensor at a sampling rate based on the set operation mode;
Based on the measurement result obtained by the measurement result obtaining means, as an index indicating the movement of the subject, a three-dimensional object whose plane direction corresponds to the front-rear direction, the left-right direction, and the vertical direction of the subject index generation means for generating a box image;
display control means for causing a display means to display both the image of the predetermined action acquired by the image acquisition means and the index generated by the index generation means;
characterized by comprising

Advantageous Effects of Invention According to the present invention, it is possible to present a measurement result of a subject's movement in a more comprehensible manner.

1 is a system configuration diagram showing the configuration of an analysis system according to one embodiment of the present invention; FIG. It is a schematic diagram which shows the example of the type of usage of an analysis system. 4 is a block diagram showing the hardware configuration of the sensor unit; FIG. It is a block diagram which shows the hardware constitutions of an imaging device. It is a block diagram which shows the hardware constitutions of a processing apparatus. 3 is a functional block diagram showing a functional configuration for executing information detection processing among the functional configurations of the sensor unit; FIG. It is a schematic diagram which shows the mounting state of a sensor unit. FIG. 4 is a schematic diagram showing an example of information detected by a sensor unit; 2 is a functional block diagram showing a functional configuration for executing imaging processing among the functional configurations of the imaging device; FIG. FIG. 4 is a functional block diagram showing a functional configuration for executing analysis result display processing among the functional configurations of the processing device; FIG. 4 is a schematic diagram showing a display example in real-time mode; FIG. 4 is a schematic diagram showing a display example in one-shot mode; FIG. 4 is a schematic diagram showing a display example of a simplified display image of characteristic points in a golf swing. 4 is a flowchart for explaining the flow of information detection processing executed by a sensor unit; 4 is a flowchart for explaining the flow of imaging processing executed by the imaging device; 7 is a flowchart for explaining the flow of analysis result display processing executed by the processing device; FIG. 3 is a schematic diagram schematically showing the timing of processing between devices in one-shot mode; FIG. 4 is a schematic diagram showing a display form example of a BOX animation;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[System configuration]
FIG. 1 is a system configuration diagram showing the configuration of an analysis system S according to one embodiment of the present invention. Also, FIG. 2 is a schematic diagram showing an example of a usage pattern of the analysis system S. As shown in FIG.
As shown in FIGS. 1 and 2, the analysis system S includes a sensor unit 1, an imaging device 2, and a processing device 3. FIG. Further, the processing device 3, the sensor unit 1, and the imaging device 2 are configured to be communicable via Bluetooth low energy/Bluetooth LE (trademark) (hereinafter referred to as “BLE”). The device 3 is configured to be communicable with Wi-Fi (Wireless Fidelity).
The sensor unit 1 is attached to an object to be measured, senses the movement of the object to be measured, and transmits sensor information to the processing device 3 . In this embodiment, the sensor unit 1 is attached to the waist or the like of a person performing a golf swing motion (hereafter referred to as "person to be measured P") to sense the motion.

The photographing device 2 photographs the object to be measured, and transmits data of a moving image representing the movement of the object to be measured to the processing device 3 . In this embodiment, the photographing device 2 photographs a moving image of the person to be measured P performing a golf swing motion.

The processing device 3 analyzes the sensor information acquired from the sensor unit 1 attached to the object to be measured, and converts the analysis result (in this embodiment, the change in posture and the state of movement of the object to be measured) into the measurement for which the sensor information was acquired. It is displayed together with a moving image showing the movement of the object. In the present embodiment, the processing device 3 animates the analysis result of the measurement target using a box-shaped object (an index indicating the three-dimensional movement of the measurement target) that schematically shows the three-dimensional movement of the measurement target. indicate. In addition, the processing device 3 displays an image that schematically represents the movement of the measurement target based on the sensor information until the analysis result of the measurement target is displayed.

[Hardware configuration]
FIG. 3 is a block diagram showing the hardware configuration of the sensor unit 1. As shown in FIG.
The sensor unit 1 is configured as a device equipped with various sensors that detect the movement of the object to be measured.
As shown in FIG. 3, the sensor unit 1 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, a RAM (Random Access Memory) 113, a bus 114, an input/output interface 115, a sensor A unit 116 , an input unit 117 , an output unit 118 , a storage unit 119 and a communication unit 120 are provided. It should be noted that the sensor unit 1 may have a configuration in which a removable medium such as a semiconductor memory can be attached.

The CPU 111 executes various processes according to programs recorded in the ROM 112 or programs loaded from the storage unit 119 to the RAM 113 .

The RAM 113 also stores data necessary for the CPU 111 to execute various processes.

The CPU 111 , ROM 112 and RAM 113 are interconnected via a bus 114 . An input/output interface 115 is also connected to this bus 114 . A sensor unit 116 , an input unit 117 , an output unit 118 , a storage unit 119 and a communication unit 120 are connected to the input/output interface 115 .

The sensor unit 116 includes a 3-axis acceleration sensor that measures acceleration in 3-axis directions, a 3-axis angular velocity sensor that measures angular velocities in 3-axis directions, and a 3-axis geomagnetic sensor that measures geomagnetism in 3-axis directions. . The sensor unit 116 measures acceleration, angular velocity, and geomagnetism in three-axis directions using a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis geomagnetic sensor at preset sampling intervals (for example, 0.001 seconds). The acceleration and angular velocity data measured by the sensor unit 116 are stored in the storage unit 119 or transmitted to the processing device 3 in association with the measurement time data.

The input unit 117 is composed of various buttons and the like, and inputs various kinds of information according to user's instruction operation.
The output unit 118 is composed of a lamp, a speaker, a vibration motor, or the like, and outputs light, sound, or vibration signals.
The storage unit 119 is composed of a semiconductor memory such as a DRAM (Dynamic Random Access Memory) and stores various data.
The communication unit 120 controls communication with other devices through direct wireless communication between terminals. In this embodiment, the communication unit 120 communicates with the processing device 3 via BLE (registered trademark), and communicates with the imaging device 2 via Wi-Fi.

FIG. 4 is a block diagram showing the hardware configuration of the imaging device 2. As shown in FIG.
The photographing device 2 has an image photographing function, and is configured by, for example, a digital camera. In this embodiment, the photographing device 2 is configured as a device in which the photographing function is unitized separately from the display.
As shown in FIG. 4, the photographing apparatus 2 includes a CPU 211, a ROM 212, a RAM 213, a bus 214, an input/output interface 215, an imaging unit 216, an input unit 217, an output unit 218, and a storage unit 219. , a communication unit 220 and a drive 221 .
Among these, the configurations other than the imaging unit 216, the communication unit 220 and the drive 221 are the same as the corresponding portions in FIG.

The imaging unit 216 includes an optical lens unit and an image sensor (not shown).

The optical lens section is composed of a lens for condensing light, such as a focus lens and a zoom lens, in order to photograph an object.
A focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. A zoom lens is a lens that can freely change the focal length within a certain range.
The optical lens unit is also provided with peripheral circuits for adjusting setting parameters such as focus, exposure, white balance, etc., as required.

The image sensor is composed of a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens section. Therefore, the photoelectric conversion element photoelectrically converts (pictures) a subject image, accumulates an image signal for a certain period of time, and sequentially supplies the accumulated image signal to the AFE as an analog signal.
The AFE executes various signal processing such as A/D (Analog/Digital) conversion processing on this analog image signal. A digital signal is generated by various signal processing and output as an output signal of the imaging unit 216 .
Such an output signal of the imaging unit 216 is hereinafter referred to as "captured image data". The captured image data is appropriately supplied to the CPU 211 and the like.

The communication unit 220 controls communication with other devices through direct wireless communication between terminals. In this embodiment, the communication unit 220 communicates with the processing device 3 via BLE (registered trademark), and communicates with the imaging device 2 and the processing device 3 via Wi-Fi.
A removable medium 231 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is mounted in the drive 221 as appropriate. A program read from the removable medium 231 by the drive 221 is installed in the storage unit 219 as necessary. The removable medium 231 can also store various data such as image data stored in the storage unit 219 in the same manner as the storage unit 219 .

FIG. 5 is a block diagram showing the hardware configuration of the processing device 3. As shown in FIG.
The processing device 3 is an information processing device having an information display function, and is configured as a smart phone, for example.

As shown in FIG. 5, the processing device 3 includes a CPU 311, a ROM 312, a RAM 313, a bus 314, an input/output interface 315, an imaging unit 316, a sensor unit 317, an input unit 318, and an output unit 319. , a storage unit 320 , a communication unit 321 , and a drive 322 . A removable medium 331 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is installed in the drive 322 as appropriate.
Among these, the configurations other than the input section 318, the output section 319 and the communication section 321 are the same as the corresponding portions in FIGS.

The input unit 318 is composed of various buttons, a touch panel, and the like, and inputs various kinds of information in accordance with the user's instruction operation.
The output unit 319 includes a display, a speaker, and the like, and outputs images and sounds.
A communication unit 321 controls communication with another device (not shown) via a network including the Internet. Also, the communication unit 321 controls communication with other devices through direct wireless communication between terminals. In this embodiment, the communication unit 321 communicates with the sensor unit 1 and the imaging device 2 via BLE (registered trademark), or communicates with the imaging device 2 via Wi-Fi.

[Functional configuration]
FIG. 6 is a functional block diagram showing a functional configuration for executing information detection processing among the functional configurations of the sensor unit 1. As shown in FIG.
The information detection process is a series of processes in which the sensor unit 1 attached to the measurement object senses the movement of the measurement object when the analysis system S analyzes the movement of the measurement object.
When the information detection process is executed, as shown in FIG. 6, the CPU 111 performs a first communication control unit 151, a second communication control unit 152, a calibration execution unit 153, a mode setting unit 154, and synchronization control. A unit 155, a detection processing unit 156, and a sensor information transmission control unit 157 function.
A sensor information storage unit 171 is set in one area of the storage unit 119 .

The first communication control unit 151 controls BLE communication of the sensor unit 1, and performs pairing processing with other devices and data transmission/reception processing.
The second communication control unit 152 controls Wi-Fi communication of the sensor unit 1, and executes data transmission/reception processing with other devices.

The calibration execution unit 153 acquires sensor information in a reference state in response to an instruction from the processing device 3, and performs calibration using the acquired result as a reference value. In this embodiment, the calibration execution unit 153 is based on a state in which the sensor unit 1 is attached to the waist of the person to be measured P and is addressed to a position for making a golf swing. The acquired sensor information is used as a reference value.

FIG. 7A is a schematic diagram showing an attached state of the sensor unit 1, and FIG. 7B is a schematic diagram showing an example of information detected by the sensor unit 1. FIG.
When calibration is performed, as shown in FIG. 7A, for example, the sensor unit 1 is attached to the waist of the person P to be measured by a belt or the like, and the sensor unit 1 is attached to the ball at a position for the person P to swing golf. Stand still for a predetermined time (for example, 2 seconds) in a facing address posture. At this time, as shown in FIG. 7B, the sensor unit 1 uses the direction of gravity detected by the three-axis acceleration sensor and the orientation detected by the Inclination angle in the left-right direction and rotation angle around the vertical axis) are acquired, and the acquired sensor information is used as a reference value at the time of address. Then, after that, when the person to be measured P swings, each sensor information detected by the sensor unit 1 is set for each sensor information with respect to the reference value at address for a predetermined time (for example, two seconds) or more. If it is within the threshold range, the address state is detected.
In addition to the information shown in FIG. 7B, the sensor unit 1 is equipped with a 3-axis acceleration sensor, so it is possible to detect movement (parallel movement, etc.) of the sensor unit 1, and it is equipped with a 3-axis geomagnetic sensor. Therefore, the direction in which the sensor unit 1 is facing can also be detected.

The mode setting unit 154 sets the operation mode to one of a plurality of prepared modes in accordance with an instruction from the processing device 3 . In this embodiment, the analysis system S has a real-time mode that displays an animation generated from the sensor information acquired by the sensor unit 1 together with a live view image of the person P to be measured, and a golf swing of the person P to be measured. A one-shot mode is prepared in which an animation generated from the sensor information acquired by the sensor unit 1 in the swing is synthesized and displayed on the moving image of the swing.

After the one-shot mode is set, the synchronization control section 155 synchronizes the reference times of the sensor unit 1 and the photographing device 2 via Wi-Fi communication. The sensor unit 1 and the imaging device 2 can synchronize the reference time using, for example, SNTP (Simple Network Time Protocol).

The detection processing unit 156 sequentially acquires various types of sensor information when the sensor unit 1 is set to the real-time mode and the one-shot mode, and associates the acquired sensor information with the acquired time. is attached and stored in the sensor information storage unit 171 . Note that the sensor information stored in the sensor information storage unit 171 may be discarded sequentially when a predetermined period of time has passed since the time when the information was acquired. However, in the one-shot mode, the sensor information acquired from the address of the golf swing to the end of the swing is discarded at least after the transmission to the processing device 3 is completed.
Moreover, the detection processing unit 156 detects the timing of a predetermined characteristic point in the golf swing based on the sensor information acquired in the one-shot mode. For example, the detection processing unit 156 analyzes the waveform of sensor information acquired in the one-shot mode, and detects the timing of each point of address, top, downswing, impact, and follow in a golf swing.

When the detection processing unit 156 detects the address timing and the follow timing in the one-shot mode, the detection processing unit 156 outputs a signal indicating that the address timing has been detected (hereinafter referred to as an "address detection signal"). ) and a signal indicating that the follow timing has been detected (hereinafter referred to as a “follow detection signal”) are sequentially transmitted to the processing device 3 by BLE.

Furthermore, the detection processing unit 156 detects the range from the start to the end of the golf swing (hereinafter referred to as "one shot" as appropriate). In this embodiment, the detection processing unit 156 detects the timing from the address to the follow in the golf swing of the person P to be measured as the one-shot range. Then, the detection processing unit 156 transmits a signal indicating the detected one-shot range (hereinafter referred to as "one-shot range detection signal") to the processing device 3 by BLE.

The sensor information transmission control unit 157 controls transmission of the sensor information acquired by the detection processing unit 156 to the processing device 3 . In this embodiment, the detection processing unit 156 can acquire sensor information at, for example, about 1000 samples/second. Then, the sensor information transmission control unit 157 converts the sensor information acquired by the detection processing unit 156 into a low sampling rate (for example, about 30 samples/second) in the real-time mode and transmits it to the processing device 3. In shot mode, the data is converted to a high sampling rate (for example, about 240 samples/second) and transmitted to the processing device 3 . In the one-shot mode, sensor information within the one-shot range is transmitted to the processing device 3 . In this embodiment, the sensor information transmission control unit 157 transmits the sensor information of predetermined characteristic points (address, top, downswing, impact, follow) in the golf swing of the person to be measured P to the range of one shot. It is transmitted to the processing device 3 prior to the sensor information. By transmitting the sensor information of the predetermined characteristic point to the processing device 3 in advance in this manner, the simple display image can be displayed in the processing device 3 at an earlier timing.

Further, when the sensor information transmission control unit 157 transmits the sensor information to the processing device 3 in the one-shot mode, the address, top, downswing, impact, and follow timings detected by the detection processing unit 156 are It also transmits to the processing device 3 .
In this embodiment, the sensor information acquired by the detection processing unit 156 is converted into a waveform that suppresses the influence of noise by filtering the output signals of various sensors, and the sensor information transmission control unit 157 The sensor information acquired from the waveform represented by the sensor information of the processing result is transmitted to the processing device 3 .
This makes it possible to analyze the movement of the person to be measured P by referring to information that is more reliable than sensor information that varies greatly due to the influence of noise.

Next, the functional configuration of the photographing device 2 will be described.
FIG. 8 is a functional block diagram showing a functional configuration for executing imaging processing among the functional configurations of the imaging device 2. As shown in FIG.
The photographing process is a series of processes for photographing a moving image representing the movement of the measurement target in synchronization with acquisition of sensor information by the sensor unit 1 when the analysis system S analyzes the movement of the measurement target.
When the shooting process is executed, as shown in FIG. 8, the CPU 211 includes a first communication control unit 251, a second communication control unit 252, a shooting control unit 253, a synchronization control unit 254, and a moving image transmission control unit. 255 works.
A moving image data storage unit 271 is set in one area of the storage unit 219 .

The first communication control unit 251 controls BLE communication of the imaging device 2, and performs pairing processing with other devices and data transmission/reception processing.
The second communication control unit 252 controls Wi-Fi communication of the photographing device 2, and executes data transmission/reception processing with other devices. In this embodiment, when Wi-Fi communication is performed, the imaging device 2 serves as an access point (AP) and the other device serves as a station (ST).

When the processing device 3 instructs to shift to the real-time mode, the imaging control unit 253 controls the imaging unit 216 to perform a live view image at a preset imaging speed (for example, 30 fps) as the imaging speed of the live view image. Take a view image. Further, when the processing device 3 instructs the shooting control unit 253 to shoot a moving image for recording, the shooting control unit 253 controls the imaging unit 216 to set a shooting speed preset as the shooting speed of the moving image for recording (for example, 240 fps) for recording. Note that the captured moving image for recording is stored in the moving image data storage unit 271 .

After the one-shot mode is set, the synchronization control section 254 synchronizes the reference times of the sensor unit 1 and the photographing device 2 via Wi-Fi communication.
The moving image transmission control unit 255 transmits to the processing device 3 a live view image captured under the control of the imaging control unit 253 or a moving image for recording. Specifically, in the real-time mode, the moving image transmission control unit 255 sequentially transmits the captured live view images to the processing device 3 when the processing device 3 issues an instruction to capture a live view image. Further, in the one-shot mode, when the processing device 3 issues an instruction to shoot a moving image for recording, the moving image transmission control unit 255 controls the range from the start to the end of the moving image for recording notified from the processing device 3 ( That is, the one-shot range) is cut out and transmitted to the processing device 3 .

Next, the functional configuration of the processing device 3 will be described.
FIG. 9 is a functional block diagram showing a functional configuration for executing the analysis result display process among the functional configurations of the processing device 3. As shown in FIG.
The analysis result display process analyzes the sensor information detected by the sensor unit 1 at the time corresponding to the moving image captured by the imaging device 2, and creates an animation that imitates the three-dimensional movement of the measurement subject P's body. is generated, combined with the moving image captured by the imaging device 2, and displayed.
When the analysis result display process is executed, as shown in FIG. A shot shooting management unit 355, a sensor information acquisition unit 356, an image acquisition unit 357, an analysis processing unit 358, an image generation unit 359, a simple display generation unit 360, and a writing processing unit 361 function.
In one area of the storage unit 320, a moving image data storage unit 371, a sensor information storage unit 372, and a composite image storage unit 373 are set.

The first communication control unit 351 controls BLE communication of the processing device 3, and executes pairing processing with other devices and data transmission/reception processing. In this embodiment, when communication is performed by BLE, the processing device 3 serves as a master and the other devices serve as slaves.
The second communication control unit 352 controls Wi-Fi communication and executes data transmission/reception processing with other devices.

The calibration management section 353 instructs the sensor unit 1 to perform calibration. In this embodiment, the calibration in the sensor unit 1 is performed once after the sensor unit 1 is activated and before shifting to the real-time mode or the one-shot mode.

The mode control unit 354 instructs the sensor unit 1 and the imaging device 2 to shift to the real-time mode by BLE when an operation for shifting to the real-time mode is performed. Further, the mode control section 354 instructs the sensor unit 1 to shift to the one-shot mode when an operation for shifting to the one-shot mode is performed.

The one-shot shooting management unit 355 manages shooting of moving images in the range of one-shot in the real-time mode. Specifically, when the one-shot shooting management unit 355 receives an address detection signal from the sensor unit 1, it instructs the shooting device 2 to shoot a moving image for recording. Further, when the one-shot shooting management unit 355 receives a follow detection signal from the sensor unit 1, it instructs the shooting device 2 to finish shooting a moving image for recording. Further, when the one-shot shooting management unit 355 receives a one-shot range detection signal from the sensor unit 1, the one-shot shooting management unit 355 notifies the shooting device 2 of the one-shot range to be extracted from the captured moving image. In addition, in the one-shot mode, the one-shot shooting management unit 355 instructs the shooting device 2 to transmit the moving image after the acquisition of the sensor information from the sensor unit 1 is completed.

In the real-time mode, the sensor information acquisition unit 356 sequentially acquires sensor information at a cycle (low sampling rate) corresponding to the live view image capturing speed (eg, 30 fps) from the sensor unit 1 by BLE. Further, in the one-shot mode, the sensor information acquisition unit 356 acquires sensor information in the one-shot range with a period (high sampling rate) corresponding to the shooting speed (for example, 240 fps) of the moving image for recording by BLE. Get from Further, in the present embodiment, the sensor information acquisition unit 356, in the one-shot mode, prior to acquiring the sensor information of the one-shot range, detects a predetermined characteristic point in the golf swing of the measurement target person P. (address, top, downswing, impact, follow) sensor information is acquired from the sensor unit 1 by BLE.

In the real-time mode, the image acquisition unit 357 sequentially acquires live view images transmitted from the imaging device 2 via Wi-Fi. In addition, in the one-shot mode, the image acquisition unit 357 captures a moving image for recording by the imaging device 2, cuts out the range of the one-shot, and then collectively acquires the cut-out moving image via Wi-Fi. .

The analysis processing unit 358 analyzes various sensor information acquired from the sensor unit 1, and generates an animation represented by a box-shaped object that imitates the three-dimensional movement of the body of the person to be measured P (hereinafter referred to as “box animation” as appropriate). ) is generated. In the present embodiment, in the real-time mode, the analysis processing unit 358 generates data representing the temporal change of the BOX animation from the sensor information of the period corresponding to the imaging speed of the live view image. Also, in the one-shot mode, the analysis processing unit 358 generates data representing the temporal change of the BOX animation from the sensor information of the cycle corresponding to the shooting speed of the moving image for recording.

The image generation unit 359 displays the BOX animation generated by the analysis processing unit 358 together with the live view image acquired in the real-time mode.
FIG. 10 is a schematic diagram showing a display example in the real-time mode.
As shown in FIG. 10, in the real-time mode, the image generation unit 359 displays, for example, the person P to be measured in the center, and performs a BOX animation An at a predetermined position set in advance around the image of the person P to be measured. indicate. Note that the BOX animation An may be displayed around the image of the person P to be measured and at a position close to the position of the body of the person P to be measured where the sensor unit 1 is worn. Further, the image generator 359 displays the numerical value of the sensor information corresponding to the BOX animation An at a predetermined position (here, the lower left) within the image. Specifically, in the example shown in FIG. 10, the tilt angle in the front-rear direction of the body is "forward tilt: 21 degrees", the rotation angle about the vertical axis is "rotation: 15 degrees", and the tilt angle in the left-right direction is "horizontal. : 7 degrees” is displayed.

In addition, the image generation unit 359 synthesizes the BOX animation generated by the analysis processing unit 358 with the moving image for recording acquired in the one-shot mode, and displays it. Hereinafter, the image displayed by the image generation unit 359 in this way will be referred to as an “analysis result moving image” as appropriate.
FIG. 11 is a schematic diagram showing a display example in the one-shot mode.
As shown in FIG. 11 , in the one-shot mode, the image generating unit 359 displays the analysis result moving image by, for example, superimposing the BOX animation An on the moving image of the person to be measured P by transmitting the BOX animation An. In the present embodiment, the BOX animation An also displays straight lines indicating the axes of the sensor unit 1 (vertical direction and horizontal direction here). The axis of the sensor unit 1 moves with the motion of the sensor unit 1 . Further, in the present embodiment, in the display example in the one-shot mode, an axis fixed on the display screen (that is, an axis representing absolute coordinates) is also displayed.
Note that the temporal change of the BOX animation is displayed in association with the timing of the captured moving image of the person P to be measured.

By displaying the sensor information and the moving image of the person to be measured P in the one-shot mode in this way, it is possible to easily show the tendency of the movement of the person to be measured P by the BOX animation, and it is difficult to appear in the BOX animation. It becomes possible to clearly indicate the state of the body by the sensor information. In addition, since the axes of the sensor unit 1 and the axes fixed on the display screen are displayed together, it is possible to display the parallel movement of the person P to be measured, and the movement of the body of the person P to be measured can be more easily visualized. It becomes easy to understand.

In the one-shot mode, the simplified display generation unit 360 generates predetermined characteristic points (address, top, downswing, impact, Follow-up) data is referred to, and numerical values of each sensor information are displayed in sequence on swing images corresponding to each characteristic point prepared in advance. A series of images displayed by the simple display generator 360 is hereinafter referred to as a "simple display image" as appropriate.

Here, in the one-shot mode, when the sensor unit 1 acquires sensor information and the imaging device 2 shoots a moving image, the processing device 3 first causes the sensor information acquisition unit 356 to acquire sensor information from the sensor unit 1. Get information. Next, the image acquisition unit 357 acquires moving image data for recording from the imaging device 2 . At this time, it takes a certain amount of time for the image acquisition unit 357 to acquire moving image data for recording from the photographing device 2 because the amount of data is relatively large. Therefore, in the present embodiment, after the image acquiring unit 357 starts acquiring moving image data for recording from the imaging device 2, the simple display generating unit 360 displays the simple display image. After the image acquisition unit 357 completes acquiring data of the moving image for recording from the photographing device 2, the simple display generation unit 360 continues until the image generation unit 359 synthesizes the BOX animation with the moving image for recording and displays it. During this period, the display of the simple display image is continued.

FIG. 12 is a schematic diagram showing a display example of a simple display image of characteristic points in a golf swing.
As shown in FIG. 12, the simplified display generation unit 360 generates a swing image of the measurement target person P with respect to images prepared in advance, which show each characteristic point of a golf swing: address, top, downswing, impact, and follow. , numerical values of sensor information of corresponding characteristic points are also displayed.
When the simple display generation unit 360 displays the simple display images, the simple display images of each characteristic point are sequentially displayed for a predetermined period of time. , it is possible to cyclically display each simple display image.
By displaying the simplified display image, it is possible to present information related to the movement of the measurement target person P, which has a certain significance, in advance.
In addition, in FIG. 12, the inclination angles of the body of the person to be measured P in the front-rear direction and the left-right direction at the time of address may be set to 0 for display. It may be configured to display the amount of change in the angle.

The write processing unit 361 writes the analysis result moving image displayed by the image generation unit 359 to the storage unit 320 or the removable medium 331 as data in a predetermined storage format (MPEG or the like). Note that the write processing unit 361 writes out the analysis result moving image when an operation to instruct saving of the analysis result moving image is performed.

[motion]
Next, the operation of the analysis system S will be explained.
FIG. 13 is a flowchart for explaining the flow of information detection processing executed by the sensor unit 1. As shown in FIG.
The information detection process is started when the sensor unit 1 is activated.

In step S1, the first communication control unit 151 connects with the processing device 3 as a slave by BLE.
In step S2, the calibration execution unit 153 acquires sensor information in a reference state in response to an instruction from the processing device 3, and performs calibration using the acquired result as a reference value.

In step S3, the calibration executing unit 153 notifies the processing device 3 by BLE that the calibration has been completed.
In step S<b>4 , the mode setting unit 154 sets the real-time mode according to the instruction from the processing device 3 .
In step S<b>5 , the detection processing unit 156 sequentially acquires various types of sensor information, and starts processing for storing the acquired sensor information in the sensor information storage unit 171 in association with the acquired time.

In step S<b>6 , the sensor information transmission control unit 157 converts the sensor information acquired by the detection processing unit 156 into a low sampling rate (for example, about 30 samples/second) and sequentially transmits the sensor information to the processing device 3 .
In step S<b>7 , the mode setting unit 154 sets the one-shot mode according to the instruction from the processing device 3 .

In step S8, the second communication control unit 152 connects to the photographing device 2 as a station via Wi-Fi.
In step S9, the synchronization control section 155 synchronizes the reference times of the sensor unit 1 and the photographing device 2 via Wi-Fi communication.

In step S10, the detection processing unit 156 analyzes the waveform of the acquired sensor information, detects the timing of the address point in the golf swing, and transmits an address detection signal to the processing device 3 by BLE.

In step S11, the detection processing unit 156 analyzes the waveform of the acquired sensor information, detects the timing of the follow point in the golf swing, and transmits a follow detection signal to the processing device 3 by BLE.

In step S<b>12 , the detection processing unit 156 detects the timing from the address to the follow in the golf swing of the person to be measured P as the one-shot range, and transmits a one-shot range detection signal to the processing device 3 .
In step S13, the detection processing unit 156 analyzes the waveform of the acquired sensor information, and sequentially detects the timing of each point of the top, downswing, and impact points in the golf swing.

In step S14, the sensor information transmission control unit 157 transmits the sensor information of predetermined characteristic points (address, top, downswing, impact, follow) in the golf swing of the person to be measured P to the processing device 3 by BLE. Send. By transmitting the sensor information of the predetermined characteristic point to the processing device 3 in advance in this manner, the simple display image can be displayed in the processing device 3 at an earlier timing. Note that the processes of steps S13 and S14 may be executed before step S12.

In step S15, the sensor information transmission control unit 157 converts the sensor information in the one-shot range into a high sampling rate (for example, about 240 samples/second) and transmits it to the processing device 3 by BLE. When transmitting the sensor information of the one-shot range to the processing device 3, the sensor information of the characteristic points already transmitted in step S14 may be excluded from the transmission. In this case, redundant transmission of the same sensor information can be suppressed, so sensor information can be transmitted more efficiently.
After step S15, the information detection process ends.

Next, the photographing process executed by the photographing device 2 will be described.
FIG. 14 is a flowchart for explaining the flow of imaging processing executed by the imaging device 2. As shown in FIG.
The shooting process is started when an operation is performed via the input unit 217 to instruct the start of the shooting process.

In step S21, the first communication control unit 251 connects with the processing device 3 as a slave by BLE.
In step S22, the second communication control unit 252 connects to the processing device 3 as an access point via Wi-Fi.

In step S23, in response to an instruction from the processing device 3 to shift to the real-time mode, the imaging control unit 253 controls the imaging unit 216 to set an imaging speed ( For example, processing for shooting a live view image at 30 fps is started.
In step S<b>24 , the moving image transmission control unit 255 sequentially transmits the captured live view images to the processing device 3 .

In step S25, the second communication control section 252 connects with the sensor unit 1 as an access point via Wi-Fi.
In step S26, the synchronization control section 254 synchronizes the reference times of the sensor unit 1 and the photographing device 2 via Wi-Fi communication.

In step S27, in response to an instruction to shoot a moving image for recording from the processing device 3, the shooting control unit 253 controls the imaging unit 216 to shoot at a shooting speed set in advance as the moving image shooting speed for recording. Start the process of shooting a moving image for recording at a speed (for example, 240 fps).
In step S<b>28 , the imaging control unit 253 controls the imaging unit 216 in response to an instruction from the processing device 3 to end shooting of the moving image for recording, and ends shooting of the moving image for recording.

In step S<b>29 , the moving image transmission control unit 256 cuts out the one-shot range of the moving image for recording in response to the notification of the one-shot range from the processing device 3 .
In step S<b>30 , the moving image transmission control unit 256 starts the process of transmitting the moving image from which the one-shot range is cut out to the processing device 3 in response to the instruction to transmit the moving image from the processing device 3 .
In step S31, the moving image transmission control unit 256 completes transmission of the moving image started in step S30.
After step S31, the photographing process ends.

Next, analysis result display processing executed by the processing device 3 will be described.
FIG. 15 is a flowchart for explaining the flow of analysis result display processing executed by the processing device 3 .
The analysis result display process is started when an operation is performed via the input unit 318 to instruct the start of the analysis result display process.

In step S41, the first communication control unit 351 connects with the sensor unit 1 and the imaging device 2 as a master by BLE.
In step S42, the second communication control unit 352 connects with the photographing device 2 as a station via Wi-Fi.
In step S43, the calibration management section 353 instructs the sensor unit 1 to perform calibration.

In step S<b>44 , the calibration management unit 353 receives notification from the sensor unit 1 that the execution of calibration has been completed by BLE.
In step S45, the mode control unit 354 receives input of an operation (instruction for remote imaging) for shifting to the real-time mode.
In step S46, the mode control unit 354 instructs the sensor unit 1 and the photographing device 2 to shift to the real-time mode using BLE.

In step S47, the sensor information acquisition unit 356 acquires sensor information of a period (low sampling rate) corresponding to the live view image shooting speed (for example, 30 fps) from the sensor unit 1 by BLE. In addition, the image acquisition unit 357 sequentially acquires live view images transmitted from the imaging device 2 via Wi-Fi.

In step S48, the analysis processing unit 358 sequentially generates data representing temporal changes in the BOX animation from the sensor information of the period corresponding to the imaging speed of the live view image acquired from the sensor unit 1, and the image generating unit 359 sequentially displays the acquired live view image together with the BOX animation generated by the analysis processing unit 358 .

In step S49, the mode control unit 354 accepts input of an operation for shifting to the one-shot mode.
In step S50, the mode control unit 354 instructs the sensor unit 1 to shift to the one-shot mode using BLE. At this time, the mode control unit 354 also transmits the SSID (Service Set Identifier) of the imaging device 2 to the sensor unit 1 .

In step S<b>51 , in response to receiving the address detection signal from the sensor unit 1 , the one-shot shooting management unit 355 instructs the shooting device 2 to shoot a moving image for recording.
In step S52, in response to receiving the follow detection signal from the sensor unit 1, the one-shot shooting management unit 355 instructs the shooting device 2 to finish shooting the moving image for recording.

In step S<b>53 , in response to receiving the one-shot range detection signal from the sensor unit 1 , the one-shot shooting management unit 355 notifies the shooting device 2 of the one-shot range to be extracted from the captured moving image.

In step S54, the sensor information acquisition unit 356 receives the sensor information of predetermined characteristic points (address, top, downswing, impact, follow) in the golf swing of the person to be measured P from the sensor unit 1 via BLE. do.

In step S55, data on predetermined characteristic points (address, top, downswing, impact, follow) in the golf swing of the person to be measured P is referred to, and corresponding to each characteristic point prepared in advance. Numerical values of each sensor information are displayed in sequence on the image of the swing. As a result, a simple display image is displayed.

In step S<b>56 , the sensor information acquisition unit 356 acquires sensor information with a period (high sampling rate) corresponding to the recording speed of moving images for recording (for example, 240 fps) from the sensor unit 1 by BLE.

In step S<b>57 , the one-shot shooting management unit 355 receives the moving image from the shooting device 2 .

In step S<b>58 , the one-shot shooting management unit 355 determines whether or not the moving image has been received from the shooting device 2 .
If the moving image has not been completely received from the photographing device 2, NO is determined in step S58, and the process proceeds to step S57.
On the other hand, if the moving image has been completely received from the photographing device 2, YES is determined in step S58, and the process proceeds to step S59.

In step S<b>59 , the analysis processing unit 358 generates data representing temporal changes in the BOX animation from the sensor information acquired from the sensor unit 1 and having a period corresponding to the recording speed of the moving image for recording.

In step S60, the image generation unit 359 synthesizes the BOX animation generated by the analysis processing unit 358 with the acquired moving image for recording and displays it. As a result, the analysis result moving image is displayed.

In step S61, the write processing unit 361 determines whether or not an operation to instruct saving of the analysis result moving image has been performed.
If an operation to instruct saving of the analysis result moving image has not been performed, a determination of NO is made in step S61, and the analysis result display process ends.
On the other hand, if an operation to instruct saving of the analysis result moving image has been performed, a determination of YES is made in step S61, and the process proceeds to step S62.

In step S62, the write processing unit 361 writes the analysis result moving image displayed by the image generation unit 359 to the storage unit 320 or the removable medium 331 as data in a predetermined storage format (MPEG or the like).
After step S62, the analysis result display process ends.

Through such processing, in the analysis system S, the sensor information obtained by measuring the movement of the body of the person to be measured P and the moving image of the movement of the person to be measured P are acquired in synchronization with each other in time. be able to. Then, based on the sensor information, a BOX animation that imitates the three-dimensional movement of the body of the person to be measured P is generated and combined with the moving image captured by the imaging device 2 to display the analysis result moving image. can be done.
In the analysis result video, the video of the measurement subject P himself, the BOX animation that clearly shows the three-dimensional movement of the measurement subject P, and the sensor information obtained by measuring the body movement of the measurement subject P. Displayed with synchronized time.
Therefore, the BOX animation makes it possible to intuitively and easily grasp information that is difficult to grasp only from a moving image of the person P to be measured. It is possible to clearly grasp the characteristics of the movement. In addition, since temporal changes in the BOX animation are associated with sensor information, it is possible to obtain accurate analysis results by referring to the BOX animation.

Further, in the one-shot mode, the analysis system S displays predetermined information regarding the movement of the person to be measured P prior to displaying the analysis result moving image, thereby suppressing unnecessary waiting time. is doing.
FIG. 16 is a schematic diagram schematically showing the timing of processing between devices in the one-shot mode.
As shown in FIG. 16 , in the one-shot mode, when the processing device 3 acquires the sensor information and the moving image for displaying the analysis result moving image, first, a predetermined characteristic characteristic of the golf swing of the measurement target person P is obtained. Acquire point sensor information (step S54 in FIG. 15), and while displaying information about the movement of the person to be measured P as a simple display image (step S55 in FIG. 15), acquire sensor information for the one-shot range. (Step S56), then, data of the moving image is received (Step S57 in FIG. 15). Then, until the reception of the moving image data is completed (step S58 in FIG. 5: YES), the BOX animation is generated (step S59 in FIG. 5), and the analysis result moving image can be displayed (step S60 in FIG. 5). ) to continue displaying the simple display image.
Therefore, rather than simply waiting for the time for the processing device 3 to acquire moving image data, information regarding the movement of the measurement target person P, which has a certain significance, can be displayed in advance. can be grasped.
In addition, by presenting the information about the movement of the measurement target person P in advance, the time period during which the presented information is viewed by the user is added to the minimum time required to generate the analysis result moving image. can be created. Therefore, it is possible to secure a longer time for generating the analysis result moving image, and it is also possible to present a more effective analysis result moving image.

Also, in the analysis system S, in the one-shot mode, the address of the subject P is automatically detected, and a moving image for recording can be taken.
Therefore, by repeating the action of addressing and swinging, the person P to be measured can successively capture moving images for recording with a simple action.

[Modification 1]
In the above-described embodiment, the display form of the BOX animation can be modified in various ways as long as it imitates the three-dimensional movement of the measurement subject P's body.
FIG. 17 is a schematic diagram showing a display form example of the BOX animation An.
In the example shown in FIG. 17, a plurality of viewpoints such as a front view, a top view, a left side view, etc. can be selected as the viewpoint of the BOX animation An. In the example shown in FIG. An is displayed in a small size at the top of the screen.
In addition, the display color of the BOX animation An can be selected so that the three-dimensional movement of the measurement subject P's body is easier to understand.
In the example shown in FIG. 17, the display colors of the BOX animation An for each viewpoint are set to different colors.
Note that when the BOX animation is superimposed on the moving image of the person to be measured P, the background color may be detected and a predetermined color that is difficult to assimilate with the background color (for example, a complementary color, etc.) may be selected. .

[Modification 2]
In the above-described embodiment, as a display example in the real-time mode, the case where the person P to be measured is displayed in the center and the BOX animation An is displayed at a predetermined position set in advance around the image of the person P to be measured has been described. . On the other hand, even in the real-time mode, the BOX animation An may be transparently superimposed on the moving image of the person to be measured P as in the display example of the one-shot mode shown in FIG. 11 .
Conversely, in the one-shot mode as well, as in the display example of the real-time mode shown in FIG. BOX animation An may be displayed in .

[Modification 3]
In the above-described embodiment, one sensor unit 1 is attached to the waist of the person to be measured P or the like to obtain sensor information, but the present invention is not limited to this. That is, a plurality of sensor units 1 may be attached to a plurality of body parts such as the shoulders and waist of the person to be measured P, and a BOX animation representing the movement of each of these body parts may be displayed.
In this case, it is possible to display the difference in the movement of each body part of the person to be measured P (for example, the twist of the shoulder and the waist), and it is possible to present the measurement results more effectively.
Also, the body part to which the sensor unit 1 is attached can be shoulders, neck, arms, thighs, head, etc., in addition to the waist.

[Modification 4]
In the above-described embodiment, in the display example in the one-shot mode, a continuous moving image of the movement of the person to be measured P is displayed. On the other hand, it is possible to display an intermittent moving image of the movement of the person P to be measured.
In this case, the processing load for generating the analysis result moving image is reduced, and the analysis result moving image can be presented earlier.

[Modification 5]
In the above-described embodiment, when the BOX animation is superimposed on the moving image of the person P to be measured, various different values can be set for the degree of transparency depending on the purpose. For example, when displaying a human-shaped object as a guide for the movement that the measurement subject P should perform, if the body of the measurement subject P and the human-shaped object overlap significantly, the transparency of the BOX animation is increased ( That is, it is displayed lightly), and when the overlap between the body of the person to be measured P and the humanoid object is small, it is possible to reduce the transparency of the BOX animation (that is, display it darker).
As a result, the BOX animation can be displayed in a more effective form when displaying the motion guide of the person P to be measured.

[Modification 6]
In the above-described embodiment, by photographing the measurement subject P wearing the sensor unit 1 with the photographing device 2, the mounting position of the sensor unit 1 can be automatically determined.
In this case, when the BOX animation is displayed at a position close to the position of the body where the sensor unit 1 is attached to the person to be measured P, the display position of the BOX animation is automatically determined. becomes possible.

The analysis system S configured as described above includes a sensor unit 1 , an imaging device 2 , and a processing device 3 . The sensor unit 1 includes a detection processing unit 156 that is worn by a subject and measures the movement of the subject using a sensor. The photographing device 2 includes a photographing control unit 253 that photographs an image of the subject. The processing device 3 includes an image acquisition unit 357 , a sensor information acquisition unit 356 , an analysis processing unit 358 and an image generation unit 359 .
The image acquisition unit 357 acquires an image of the target person.
The sensor information acquisition unit 356 acquires the measurement result of the subject's movement measured by the sensor.
The analysis processing unit 358 generates an index indicating the subject's movement based on the measurement result acquired by the sensor information acquisition unit 356 .
The image generation unit 359 causes the output unit 319 to display both the image acquired by the image acquisition unit 357 and the index generated by the analysis processing unit 358 .
As a result, the moving image of the subject himself/herself, the index representing the movement of the subject in an easy-to-understand manner, and the measurement result obtained by measuring the movement of the subject's body are displayed together.
As a result, it is possible to intuitively and easily grasp information that is difficult to grasp only from a video of the subject's movement, and by displaying the index together with the measurement results, the characteristics of the movement can be clearly and accurately identified. It is possible to grasp the
Therefore, according to the present invention, it is possible to present the measurement result of the movement of the subject in an easy-to-understand manner.

The analysis processing unit 358 generates an index indicating the three-dimensional movement of the subject based on the measurement results.
This makes it possible to grasp the movement of the subject more intuitively and easily.

The analysis processing unit 358 generates a three-dimensional box image as an index indicating the subject's movement based on the measurement result.
This makes it possible to grasp the movement of the subject more intuitively and easily.

The sensor information acquisition unit 356 acquires the measurement result measured by the sensor attached to the subject.
This makes it possible to obtain more accurate measurement results.

The image acquisition unit 357 acquires an image captured by a camera that captures an image of the subject.
This makes it possible to capture a more appropriate image of the subject.

The image generator 359 displays the target person's image with the index superimposed thereon.
This makes it possible to easily grasp the movement of the subject and the change in the index.

The image generation unit 359 superimposes the index on the target person's image and displays it at a position corresponding to the position where the sensor measures the movement of the target person's body.
As a result, it is possible to more easily grasp the movement of the part of the subject's body.

The image generator 359 displays indicators around the image of the subject.
This makes it possible to display the index indicating the three-dimensional movement of the subject in a state in which the subject's image can be easily recognized.

The image generation unit 359 displays the index around the image of the subject in association with the position where the sensor on the body of the subject measures the movement.
As a result, it is possible to more easily grasp the movement of the target person's body part in a state where the image of the target person can be easily recognized.

The sensor information acquisition unit 356 acquires measurement results obtained by measuring movements of multiple parts of the subject's body with sensors.
The analysis processing unit 358 generates indices for multiple parts.
The image generation unit 359 causes the output unit 319 to display the image acquired by the image acquisition unit 357 and the indices of a plurality of regions generated by the analysis processing unit 358 in association with each other.
As a result, it is possible to display the difference in the movement of each part of the subject's body (for example, the twist of the shoulder and waist), and to present the measurement results more effectively.

The image generation unit 359 causes the output unit 319 to display intermittent images in time series among the images acquired by the image acquisition unit 357 in association with the index generated by the analysis processing unit 358 .
As a result, the processing load for generating the analysis result is reduced, and the measurement result can be presented earlier.

The image generator 359 transparently displays the index.
As a result, it is possible to display the index indicating the three-dimensional movement of the subject while ensuring the visibility of the moving image of the subject.

The image generator 359 displays an image that serves as a guide for the movement of the subject, and changes the transmittance of the index according to the degree of overlap between the movement of the subject and the image that serves as the guide.
This makes it possible to display the index in a more effective form when displaying a guide for the movement of the subject.

The image generation unit 359 displays an axis fixedly displayed with respect to the movement of the index.
As a result, it is possible to display the parallel movement of the subject and the like, making it easier to grasp the movement of the subject's body.

It should be noted that the present invention is not limited to the above-described embodiments, and includes modifications, improvements, and the like within the scope of achieving the object of the present invention.

In the above-described embodiment, when the processing device 3 displays the simple display images, the total time required to acquire the moving image from the imaging device 2 and generate the analysis result moving image is estimated, and the estimated time is expressed by the number of simple display images. The simple display images may be displayed in order for each divided time. As a result, it is possible to display the analysis result in a more comprehensible form while matching the display content of the simple display image and the display start timing of the analysis result moving image.

Further, in the above-described embodiment, the analysis system S is composed of the sensor unit 1, the imaging device 2, and the processing device 3, but the present invention is not limited to this. For example, the analysis system S may be configured as a device in which the sensor unit 1 and the processing device 3 are integrated by a smartphone or the like having the functions of the sensor unit 1 and the processing device 3 .

In the above-described embodiment, when displaying the temporal change of the BOX animation, based on the sensor information acquired by the sensor unit 1, the characteristics of the movement, such as the change in acceleration, are displayed in different colors or as indicators. Depending on the form, it may be identified and displayed. For example, when the body is moving with acceleration greater than the reference, it is possible to color the body of the person to be measured P red.

In the above-described embodiment, when the BOX animation is displayed at a predetermined position around the image of the person to be measured P, as in the display example in the real-time mode, the sensor unit 1 is worn by the person to be measured P. It is also possible to identify and indicate the position of the body with a lead line or a marker. Also, a BOX animation may be superimposed and displayed on the position of the body of the person P to be measured where the sensor unit 1 is attached.

In the above-described embodiment, it is possible to determine whether or not the person P to be measured has taken a swing by setting detailed conditions based on sensor information. For example, if the angle of rotation about the vertical axis is a first angle or more and rotates in the take-back direction, it is determined to be the top, and if it rotates about the vertical axis from the top at the first angular velocity or more, it is determined to be a downswing, When the rotation angle about the vertical axis is the second angle or more and the ball rotates in the follow direction, it can be determined that the follow is performed, and when all these conditions are satisfied, it can be determined that the swing has been performed. Further conditions may be set for the times when these determination conditions are satisfied. For example, conditions can be set such that each condition must be sequentially satisfied within two seconds.

In the above-described embodiment, when detecting the timing of a predetermined characteristic point in a golf swing based on sensor information, time-series data is analyzed in chronological order to determine the characteristic point, or It is possible to determine other characteristic points by retroactively analyzing the time series from the timing that can be clearly discriminated as a characteristic point.
This makes it possible to more reliably detect the timing of a predetermined characteristic point in a golf swing.

In the above-described embodiment, when the simple display generation unit 360 displays a simple display image, a cancel button for canceling processing related to the generation of BOX animation data by the analysis processing unit 358 and the display of the analysis result moving image by the image generation unit 359 is pressed. It may be displayed.
As a result, by checking the simple display image, it is possible to suppress unnecessary processing from being performed when it is determined that generation of the analysis result moving image is unnecessary.

In the above-described embodiment, when the sensor information of the one-shot range acquired in the one-shot mode is transmitted from the sensor unit 1 to the processing device 3, in step S14 of FIG. The entire sensor information of the one-shot range may be transmitted to the processing device 3 in step S15 of FIG. 13 without transmitting the sensor information of the characteristic points in advance. In this case, the time required for transmitting sensor information can be shortened as a whole.

In the above-described embodiment, the detection processing unit 156 detects the timing of a predetermined characteristic point in the golf swing based on the sensor information acquired in the one-shot mode. For example, the detection processing unit 156 may analyze the waveform of the sensor information acquired in the one-shot mode, and detect the timing at which the waveform starts to change from a state where the amount of change in the waveform is small as the address timing. After detecting the range of the downswing and detecting the timing of the downswing, the point of time a predetermined time before the timing of the downswing may be reset as the point of time of the address to determine the range of the one-shot.

In the above-described embodiment, the case where the sensor unit 1 is attached to the subject P who swings golf and the analysis system S is used to analyze the golf swing has been described, but the present invention is not limited to this. In other words, the analysis system S according to the present invention can be used for various sports such as baseball, tennis, track and field, etc., in which the subject player can be photographed at a fixed position of the angle of view. For example, the analysis system S according to the present invention can be used for a batter swinging at a baseball bat, a pitcher pitching on the mound, a tennis server, a sprinter photographed by a camera running side by side, and the like.

Further, in the above-described embodiment, the processing device 3 to which the present invention is applied has been described as an example of a smartphone, but it is not particularly limited to this.
For example, the present invention can be applied to general electronic devices having an image processing function. Specifically, for example, the present invention is applicable to notebook personal computers, printers, television receivers, video cameras, portable navigation devices, mobile phones, portable game machines, and the like.

The series of processes described above can be executed by hardware or by software.
In other words, the functional configurations of FIGS. 6, 8 and 9 are merely examples and are not particularly limited. That is, it is sufficient if the analysis system S has a function capable of executing the above-described series of processes as a whole, and what kind of functional blocks are used to realize this function are particularly shown in the examples of FIGS. is not limited to
Also, one functional block may be composed of hardware alone, software alone, or a combination thereof.
The functional configuration in this embodiment is realized by a processor that executes arithmetic processing, and processors that can be used in this embodiment are composed of various single processing units such as single processors, multiprocessors, and multicore processors. In addition, it includes a combination of these various processing devices and a processing circuit such as ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array).

When a series of processes is to be executed by software, a program constituting the software is installed in a computer or the like from a network or a recording medium.
The computer may be a computer built into dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a general-purpose personal computer.

Recording media containing such programs are not only configured by removable media 231 and 331 shown in FIGS. It consists of a recording medium, etc., which is provided to the user in a The removable media 231 and 331 are configured by, for example, magnetic disks (including floppy disks), optical disks, or magneto-optical disks. Optical discs are composed of, for example, CD-ROMs (Compact Disk-Read Only Memory), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Discs, and the like. The magneto-optical disk is composed of an MD (Mini-Disk) or the like. Recording media provided to the user in a state of being pre-installed in the device main body are, for example, ROMs 112, 212, and 312 in FIGS. , 320 and the like.

In this specification, the steps of writing a program recorded on a recording medium are not necessarily processed chronologically according to the order, but may be executed in parallel or individually. It also includes the processing to be executed.
Further, in this specification, the term "system" means an overall device composed of a plurality of devices, a plurality of means, or the like.

Although several embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the gist of the present invention. These embodiments and their modifications are included in the scope and gist of the invention described in this specification and the like, and are included in the scope of the invention described in the claims and equivalents thereof.

The invention described in the scope of claims at the time of filing of the present application will be additionally described below.
[Appendix 1]
an image acquisition means for acquiring an image of a subject;
measurement result acquisition means for acquiring a measurement result of measuring the movement of the subject by a sensor;
index generation means for generating an index indicating the movement of the subject based on the measurement result acquired by the measurement result acquisition means;
display control means for displaying both the image acquired by the image acquisition means and the index generated by the index generation means on a display means;
An image processing device comprising:
[Appendix 2]
The image processing apparatus according to Supplementary Note 1, wherein the index generating means generates an index indicating the three-dimensional movement of the subject based on the measurement result.
[Appendix 3]
3. The image processing apparatus according to appendix 1 or 2, wherein the index generating means generates a three-dimensional box image as an index indicating the movement of the subject based on the measurement result.
[Appendix 4]
4. The image processing apparatus according to any one of appendices 1 to 3, wherein the measurement result acquisition means acquires the measurement result measured by a sensor attached to the subject.
[Appendix 5]
5. The image processing apparatus according to any one of appendices 1 to 4, wherein the image obtaining means obtains the image photographed by a photographing device for photographing the subject.
[Appendix 6]
6. The image processing apparatus according to any one of appendices 1 to 5, wherein the display control means displays the index superimposed on the image of the subject.
[Appendix 7]
The display control means according to appendix 6, wherein, in the image of the subject, the indicator is superimposed and displayed at a position corresponding to a position on the subject's body where the sensor measures the movement. Image processing device.
[Appendix 8]
6. The image processing apparatus according to any one of appendices 1 to 5, wherein the display control means displays the index around the image of the subject.
[Appendix 9]
The image according to Supplementary Note 8, wherein the display control means displays the indicator in association with the position of the subject's body where the sensor measures the movement around the image of the subject. processing equipment.
[Appendix 10]
The measurement result acquisition means acquires measurement results obtained by measuring movements of a plurality of parts of the subject's body with a sensor,
The index generation means generates the indices for the plurality of parts,
Supplementary Note 1, wherein the display control means causes the display means to display the image acquired by the image acquisition means and the indices of the plurality of parts generated by the index generation means in association with each other. 10. The image processing device according to any one of 9.
[Appendix 11]
The display control means causes the display means to display intermittent images in time series among the images acquired by the image acquisition means in association with the index generated by the index generation means. 11. The image processing apparatus according to any one of appendices 1 to 10.
[Appendix 12]
12. The image processing apparatus according to any one of appendices 1 to 11, wherein the display control means transparently displays the index.
[Appendix 13]
The display control means displays an image that serves as a guide for the movement of the subject, and changes the transmittance of the index according to the degree of overlap between the movement of the subject and the image that serves as the guide. 13. The image processing apparatus according to any one of additional remarks 1 to 12.
[Appendix 14]
14. The image processing apparatus according to any one of appendices 1 to 13, wherein the display control means displays an axis fixedly displayed with respect to the movement of the index.
[Appendix 15]
a photographing device comprising photographing means for photographing an image of a subject;
a detection device that is worn by the subject and includes a measuring means that measures the movement of the subject using a sensor;
an image acquiring means for acquiring an image of the subject photographed from the photographing device;
a measurement result acquisition means for acquiring a measurement result of measuring the movement of the subject by a sensor from the detection device;
index generation means for generating an index indicating the movement of the subject based on the measurement result acquired by the measurement result acquisition means;
an image processing apparatus comprising display control means for causing a display means to display both the image acquired by the image acquisition means and the index generated by the index generation means;
An analysis system comprising:
[Appendix 16]
An image processing method executed by an image processing device,
an image acquisition step of acquiring a captured image of the subject;
a measurement result acquisition step of acquiring a measurement result obtained by measuring the movement of the subject with a sensor;
an index generation step of generating an index indicating the movement of the subject based on the measurement result obtained in the measurement result obtaining step;
a display control step of displaying both the image obtained in the image obtaining step and the index generated in the index generating step on a display;
An image processing method comprising:
[Appendix 17]
In the computer that controls the image processing device,
an image acquisition function for acquiring an image of a target person;
a measurement result acquisition function for acquiring measurement results obtained by measuring the movement of the subject with a sensor;
an index generation function that generates an index indicating the movement of the subject based on the measurement results obtained by the measurement result obtaining function;
a display control function for displaying both the image acquired by the image acquisition function and the index generated by the index generation function on display means;
A program characterized by realizing

S Analysis system 1... Sensor unit 2... Imaging device 3... Processing device 111, 211, 311... CPU, 112, 212, 312... ROM, 113, 213, 313 RAM 114, 214, 314 bus 115, 215, 315 input/output interface 116, 317 sensor section 117, 217, 318 input section 118, 218 , 319... output section, 119, 219, 320... storage section, 120, 220, 321... communication section, 121, 322... drive, 216, 316... imaging section, 231, 331 Removable media 151, 251, 351 First communication control unit 152, 252, 352 Second communication control unit 153 Calibration execution unit 154 Mode setting unit , 155, 254... synchronization control unit, 156... detection processing unit, 171, 372... sensor information storage unit, 253... shooting control unit, 255... moving image transmission control unit, 271, 371 ... moving image data storage section, 353 ... calibration management section, 354 ... mode control section, 355 ... one-shot photography management section, 356 ... sensor information acquisition section, 357 ... image acquisition Section 358 Analysis processing section 359 Image generation section 360 Simple display generation section 361 Writing processing section 373 Composite image storage section

Claims (16)

Among a plurality of operation modes prepared in advance, a first mode in which the photographing device sequentially photographs a predetermined action by the subject, and an external operation during the sequential photographing in the first mode. a second mode that starts based on detection, in which when the start timing is detected based on the measurement result of the predetermined action acquired by a sensor, the moving image recording of the predetermined action that is being sequentially shot is started; is set, image acquisition means for acquiring an image of the predetermined operation by controlling the photographing device to photograph at a photographing speed based on the set operation mode;
measurement result acquisition means for acquiring a measurement result of the movement of the subject measured by the sensor at a sampling rate based on the set operation mode;
Based on the measurement result obtained by the measurement result obtaining means, as an index indicating the movement of the subject, a three-dimensional object whose plane direction corresponds to the front-rear direction, the left-right direction, and the vertical direction of the subject index generation means for generating a box image;
display control means for causing a display means to display both the image of the predetermined action acquired by the image acquisition means and the index generated by the index generation means;
An image processing device comprising: 2. The image processing apparatus according to claim 1, wherein said index generating means generates an index indicating a three-dimensional movement of said subject based on said measurement result. 3. The image processing apparatus according to claim 1, wherein the measurement result acquisition means acquires the measurement result measured by a sensor attached to the subject. 4. The image processing apparatus according to any one of claims 1 to 3, wherein said image acquiring means acquires an image of said predetermined action taken by an image taking device for taking an image of said subject. 5. The image processing apparatus according to any one of claims 1 to 4, wherein the display control means displays the index superimposed on the image of the subject. 6. The display control means according to claim 5, wherein, in the image of the subject, the index is superimposed and displayed at a position corresponding to a position where the sensor measures the movement of the body of the subject. image processing device. 5. The image processing apparatus according to any one of claims 1 to 4, wherein the display control means displays the index around the image of the subject. 8. The display control means according to claim 7, wherein the display control means displays the index in association with the position of the subject's body where the sensor measures the movement around the image of the subject. Image processing device. The measurement result acquisition means acquires measurement results obtained by measuring movements of a plurality of parts of the subject's body with a sensor,
The index generation means generates the indices for the plurality of parts,
The display control means causes the display means to display the image of the predetermined motion acquired by the image acquisition means and the indicators of the plurality of parts generated by the indicator generation means in association with each other. The image processing apparatus according to any one of claims 1 to 8, wherein: The display control means associates intermittent images in a time series among the images of the predetermined motion acquired by the image acquisition means with the indicators generated by the indicator generation means, and displays them on the display means. 10. The image processing apparatus according to any one of claims 1 to 9, wherein the image processing apparatus is configured to display. 11. The image processing apparatus according to claim 1, wherein said display control means transparently displays said index. The display control means displays an image that serves as a guide for the movement of the subject, and changes the transmittance of the index according to the degree of overlap between the movement of the subject and the image that serves as the guide. The image processing apparatus according to any one of claims 1 to 11, wherein: 13. The image processing apparatus according to any one of claims 1 to 12, wherein said display control means displays an axis fixedly displayed with respect to movement of said index. a photographing device comprising photographing means for photographing an image of a subject;
a detection device that is worn by the subject and includes a measuring means that measures the movement of the subject using a sensor;
Among a plurality of operation modes prepared in advance, a first mode in which the photographing device is caused to sequentially photograph a predetermined action of the subject, and an external operation during the sequential photographing in the first mode. A second mode in which the moving image recording of the predetermined motion being sequentially captured is started when the start timing is detected based on the measurement result of the predetermined motion acquired by the sensor. and an image acquiring means for acquiring an image of the predetermined operation by controlling the photographing device to photograph at a photographing speed based on the set operation mode when either of the above is set;
measurement result acquisition means for acquiring, from the detection device, a measurement result of the movement of the subject measured by the sensor at a sampling rate based on the set operation mode;
Based on the measurement result obtained by the measurement result obtaining means, as an index indicating the movement of the subject, a three-dimensional object whose plane direction corresponds to the front-rear direction, the left-right direction, and the vertical direction of the subject index generation means for generating a box image;
an image processing apparatus comprising display control means for causing a display means to display both the image of the predetermined action acquired by the image acquisition means and the index generated by the index generation means;
An analysis system comprising: An image processing method executed by an image processing device,
Among a plurality of operation modes prepared in advance, a first mode in which the photographing device sequentially photographs a predetermined action by the subject, and an external operation during the sequential photographing in the first mode. a second mode that starts based on detection, in which when the start timing is detected based on the measurement result of the predetermined action acquired by a sensor, the moving image recording of the predetermined action that is being sequentially shot is started; is set, an image acquisition step of controlling the photographing device to photograph at a photographing speed based on the set operation mode, and acquiring an image of the predetermined operation;
a measurement result acquisition step of acquiring a measurement result of the movement of the subject measured by the sensor at a sampling rate based on the set operation mode;
Based on the measurement result obtained in the measurement result obtaining step, as an index indicating the movement of the subject, a three-dimensional object whose plane directions correspond to the front-rear direction, the left-right direction, and the vertical direction of the subject an index generation step for generating a box image;
a display control step of causing a display unit to display both the image of the predetermined action obtained in the image obtaining step and the index generated in the index generating step;
An image processing method comprising: In the computer that controls the image processing device,
Among a plurality of operation modes prepared in advance, a first mode in which the photographing device sequentially photographs a predetermined action by the subject, and an external operation during the sequential photographing in the first mode. a second mode that starts based on detection, in which when the start timing is detected based on the measurement result of the predetermined action acquired by a sensor, the moving image recording of the predetermined action that is being sequentially shot is started; is set, an image acquisition function for acquiring an image of the predetermined operation by controlling the photographing device to photograph at a photographing speed based on the set operation mode;
A measurement result acquisition function for acquiring measurement results obtained by measuring the movement of the subject with a sensor;
Based on the measurement result acquired by the measurement result acquisition function, as an index indicating the movement of the subject, a three-dimensional object whose plane direction corresponds to the front-back direction, the left-right direction, and the vertical direction of the subject Index generation function that generates a box image,
a display control function for displaying both the image of the predetermined action acquired by the image acquisition function and the index generated by the index generation function on a display unit;
A program characterized by realizing

Images