JP5664714B2 - Image processing apparatus and program - Google Patents

Description

The present invention relates to an image processing apparatus and a program.

  By analyzing a moving image obtained by photographing the locus of the moving object from a plurality of directions, it is possible to easily grasp the locus of the moving object from multiple angles. In particular, it is possible to easily check whether the swing form is good or bad by analyzing a moving image obtained by photographing a swing operator during a golf swing from a plurality of directions.

According to Patent Document 1, a technique is disclosed in which a moving image taken during a swing photographed from a plurality of directions can be analyzed in real time and a form can be automatically checked.
Further, according to Patent Document 2, a technique is disclosed that can analyze a moving image taken during a swing taken from a plurality of directions by a simple operation.
Further, according to Patent Document 3, a technique is disclosed in which a swing form can be checked by associating a moving image taken at a time taken from a plurality of directions with impact analysis data.

Japanese Patent No. 2794018 JP 2006-263169 A JP 2003-088604 A

  Here, when a moving image shot from a plurality of directions is converted into a moving image for single-screen display and reproduced, it is desirable that the reproduced moving images are synchronized with each other.

An object of the present invention to allow simultaneous reproduction in synchronization moving image of multiple.

The invention according to claim 1 is an image processing apparatus that controls the reproduction of a plurality of moving images , and is a reference that serves as a synchronization reference for each moving image when simultaneously reproducing a plurality of moving images taken at different frame rates. Setting means for arbitrarily setting a frame for each moving image, and reference frames of the moving images set by the setting means are reproduced at the same reproduction timing when the moving images are simultaneously reproduced. determining means for determining individual reproduction start frame in each of said moving image as, when simultaneously reproducing the respective moving image individually to start playback from that each playback start frame of each moving image, at least Reproduction control means for controlling each reference frame of each moving image to be reproduced at the same reproduction timing.

According to the present invention, it is possible to simultaneously reproduce synchronously moving image of multiple.

1 is a diagram illustrating an overall configuration of an image processing system. It is a figure which shows the internal structure of an image processing apparatus. It is a flowchart which shows a synchronous process. It is a flowchart which shows a synchronous process. It is a flowchart which shows a synchronous process. It is a flowchart which shows a synchronous process. It is a flowchart which shows a synchronous process. It is a conceptual diagram of reference position information. It is a figure which shows an example of a reference | standard position extraction panel. It is a figure which shows an example of a reference | standard position extraction panel. It is a figure which shows an example of a reference | standard position extraction panel. It is a figure which shows an example of a reference | standard position extraction panel. It is a figure which shows an example of a reference | standard position extraction panel. It is a conceptual diagram of reference position information. It is a conceptual diagram of reference position information. It is a figure which shows an example of a synchronous panel. It is a conceptual diagram of reference position information. It is a flowchart which shows a moving image production | generation process. It is a figure which shows an example of the moving image for 1 screen display which can be written in a portable storage medium.

FIG. 1 shows an overall configuration of an image processing system 100 according to an embodiment of the present invention.
The image processing system 100 is a system for displaying or analyzing a trajectory of a moving object, and in this embodiment, a case where a trajectory of a golf swing is displayed or analyzed as an example of a hit ball game will be described.

  The image processing system 100 includes high-speed cameras 1 to 3, a HUB 4, an image processing device 10, a printer 5, a media writing unit 6, and the like. These are connected by a LAN (Local Area Network) cable 7.

The high-speed cameras 1 to 3 are high-function digital cameras that capture about 300 to 1200 still images per second and output moving images. The number of frames captured per second (frame rate) is represented by fps (frame per second).
In this embodiment, the high-speed cameras 1 and 2 can capture 300 still images per second, and the frame rate is 300 [fps]. The high-speed camera 3 can shoot 1200 still images per second, and the frame rate is 1200 [fps].

  The high-speed camera 1 is a camera that captures the front of the swing operator, and outputs a moving image of the front to the image processing apparatus 10. A still image 1 </ b> A in FIG. 1 is an example of a one-frame still image taken by the high-speed camera 1.

  The high-speed camera 2 is a camera that captures the side and rear of the swing operator, and outputs a side and rear moving image (hereinafter simply referred to as “backward moving image”) to the image processing apparatus 10. Here, “rear” refers to a direction opposite to the direction of flight when the direction of flight of the golf ball is the front. A still image 2 </ b> A in FIG. 1 is an example of a one-frame still image taken by the high speed camera 2. In the present embodiment, a golf ball is shown as an example of a hit object.

The high-speed camera 3 is a camera that captures the moment of impact between the golf club head and the golf ball, and is a camera that captures the golf ball from the front direction of the swing operator. The frame rate is higher than the other high-speed cameras 1 and 2 in order to capture the moment of impact that moves at the highest speed. A still image 3 </ b> A in FIG. 1 is an example of a one-frame still image taken by the high speed camera 3.
The high speed cameras 1 to 3 output the captured moving image to the image processing apparatus 10 via the LAN cable 7.

  The image processing apparatus 10 inputs moving images taken by the high speed cameras 1 to 3 and displays or analyzes the input moving images.

FIG. 2 shows an internal configuration of the image processing apparatus 10.
The image processing apparatus 10 includes a control unit 11, an input unit 12, a display unit 13, a storage unit 14, a connection unit 15, and the like. Each unit is connected by a bus 16. In the present embodiment, the image processing apparatus 10 is configured by an information processing apparatus such as a personal computer or a desktop computer.

  The control unit 11 includes a CPU, a RAM, and the like (not shown), and centrally controls each unit of the image processing apparatus 10 in cooperation with the storage unit 14.

  The input unit 12 includes a keyboard having a cursor key and various function keys, a mouse, and the like, and outputs an operation signal to the control unit 11.

  The display unit 13 includes an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, and the like, and performs screen display according to a display signal from the control unit 11.

  The storage unit 14 is configured by an HDD (Hard Disk Drive) or the like, and stores various programs and various data.

  The connection unit 15 is configured by a bus interface such as a USB (Universal Serial Bus) and inputs the moving image output from the high-speed cameras 1 to 3, while the moving image or the still image is input to the printer 5 or the media writing unit 6. Output an image.

Returning to FIG. 1, the HUB 4 is a line concentrator interposed between the high-speed cameras 1 to 3 and the image processing apparatus 10 and connects them.
The printer 5 performs print processing on a print medium (for example, paper) in accordance with a print instruction from the image processing apparatus 10. The printer 5 is configured by a printing device such as a page printer.
The media writing unit 6 is a device that writes a digital moving image on a medium such as a DVD (portable storage medium) in accordance with a writing instruction from the image processing apparatus 10. The media writing unit 6 may be built in the image processing apparatus 10.

The synchronization processing of the image processing apparatus 10 will be described with reference to FIGS. 3A to 3E.
As a premise, it is assumed that the swing operator and the ball at the time of golf swing are photographed at the same time by the high-speed cameras 1 to 3, and the photographed moving images in the respective directions are stored by the image processing device 10.

The control unit 11 reads one of the moving images in the front, rear, and ball directions taken by the high-speed cameras 1 to 3 (step S1).
The moving images in each direction read here are moving images that have been photographed at the same time by artificially matching the timings of photographing start with each other, but are not accurately synchronized in units of frames. For example, the control unit 11 may select a moving image in each direction such as “FRONT.mov (front moving image)”, “BACK.mov (rear moving image)”, “BL.mov (ball moving image)”, and the like. A name is given and stored in the storage unit 14.

The control unit 11 converts the moving image into a still image group based on the frame rate of the moving image (step S2).
In the present embodiment, a moving image in which the frame rate of the moving image of the front and the moving image of the rear is 300 [fps] and the frame rate of the moving image of the ball is 1200 [fps] is converted into a still image group. Will be described.

The control unit 11 converts the moving image into a still image group, and then stores the still image group in the storage unit 14 (step S3).
When storing the still image group, the control unit 11 assigns a serial number to each still image in the still image group and stores it. For example, the control unit 11 applies “FR0000001.jpg ...” to each still image in the front still image group, “BK0000001.jpg ...” to each still image in the rear still image group, and each still image in the ball still image group. “BL0000001.jpg…” is stored as a sequential number.

The control unit 11 stores the image information of the still image group in the storage unit 14 (step S4).
The image information is specifically frame rate information, and the control unit 11 stores the frame rate information of the still image group with a name. For example, the control unit 11 sets “FRfps” as the frame rate information of the front still image group, “BKfps” as the frame rate information of the rear still image group, and “BLfps” as the frame rate information of the ball still image group. And remember it. In this embodiment, FRfps is 300 [fps], BKfps is 300 [fps], and BLfps is 1200 [fps].

The control unit 11 determines whether or not the processing from steps S1 to S4 has been completed for the moving images in the three directions of the front, back, and ball (step S5).
If not completed (step S5; N), the control unit 11 proceeds to step S1 and performs the processing described above for the moving image in the other direction that has not been processed.
If it has been completed (step S5; Y), the control unit 11 reads the stored front still image group (step S6).

The control unit 11 initializes the reference position information stored in advance in the storage unit 14 (step S7).
FIG. 4 shows a conceptual diagram of the reference position information.
The reference position information is information on image numbers at four predetermined timings. Specifically, the four timings are (1) “takeback start”, (2) “downward start”, (3) “impact moment”, and (4) “follow-through end”.
In step S7, the control unit 11 initializes the image number at each timing, for example, “0000001”.

The control unit 11 displays the reference position extraction panel on the display unit 13 (step S8).
FIG. 5 shows the reference position extraction panel P1 displayed on the display unit 13.
The reference position extraction panel P1 includes four frames F1 to F4 and a gauge G1. On the upper part of the frames F1 to F4, characters “takeback start”, “downward start”, “impact moment”, and “follow-through end” are displayed.
In the frames F1 to F4, all the same still images initialized in step S7 are displayed. In the subsequent processing, the still image corresponding to each timing is updated and determined.

The control unit 11 determines whether any of the four frames F1 to F4 has been selected via the input unit 12 (step S9).
If any frame is not selected (step S9; N), the control unit 11 waits until it is selected.
When any frame is selected (step S9; Y), the control unit 11 sets the still image in the selected frame to be editable (step S10).

The control unit 11 determines whether there is an editing instruction (step S11).
When there is no editing instruction (step S11; N), the control unit 11 stands by until an instruction is given.
When there is an editing instruction (step S11; Y), the control unit 11 determines whether or not there is a thinning operation of the gauge G1 (step S12).
The thinning-out operation of the gauge G1 means that the slider G is operated instead of operating the “G” of the gauge G1. When the “G” of the gauge G1 is operated, the still images in the selected frame are switched and displayed one by one, and when the slider is operated, they are switched and displayed for each predetermined thinning amount.

When the thinning operation of the gauge G1 is not performed (step S12; N), the control unit 11 proceeds to step S14.
When the thinning operation of the gauge G1 is performed (step S12; Y), the control unit 11 reads a predetermined thinning amount (step S13).

  The control unit 11 calculates the position of the still image in the frame F1 in accordance with the “▲” of the gauge G1 or the operation instruction by the slider (step S14).

The control unit 11 determines whether or not the calculated position of the still image is within a range where the still image exists (step S15).
When it is not within the range where the still image exists (step S15: N), the control unit 11 proceeds to step S11.
When it is within the range where the still image exists (step S15; Y), the control unit 11 switches and displays the still image in the frame F1 (step S16).

FIG. 6 shows the reference position extraction panel P2 when the frame F1 is selected.
When the frame F1 is selected and the “▲” of the gauge G1 is operated or the slider is operated, the position of the still image is switched. The still image at the switched position is displayed in the frame F1.

The control unit 11 updates the reference position information initialized in step S7 (step S17).
The control unit 11 determines whether another frame has been selected (step S18).
The other frames are frames F2 to F4.
When another frame is selected (step S18; Y), the control unit 11 proceeds to step S10 and performs the same process as the process performed on the frame F1 until step S18. In other words, still images are switched at other timings.

7 to 9 show the reference position extraction panels P3 to P5 when the frames F2 to F4 are selected.
As in the case where the frame F1 is selected, when the frames F2 to F4 are selected and the “G” of the gauge G1 is operated or the slider is operated, the position of the still image is switched. The still image at the switched position is displayed in frames F2 to F4.

  When no other frame is selected (step S18; N), the control unit 11 determines reference position information of the front still image (step S19).

FIG. 10 is a conceptual diagram of the reference position information after determining the reference position information of the front still image.
As shown in FIG. 10, the image numbers of the timings “start takeback”, “start down”, “impact moment”, and “followthrough end” in the front still image are determined by the processing of steps S10 to S19. The

The control unit 11 calculates provisional reference position information of the rear still image based on the determined reference position information of the front still image (step S20), and calculates the calculated reference position information of the rear still image. Update (step S21).
The temporary reference position information of the rear still image to be updated is the same image number as the reference position information of the front still image determined in step S19.

The control unit 11 calculates temporary reference position information of the still image of the ball (step S22), and updates the calculated temporary reference position information of the still image of the ball (step S23).
The provisional reference position information of the still image of the ball is calculated and updated based on a predetermined calculation formula. The predetermined calculation formulas are shown in the following formulas (1) to (5).

[Image number corresponding to the position of 70% of the number of still images of the ball] = X (1)
[Reference position of “takeback start”] = X − (“impact instant” position of front still image− “takeback start” position) × BLfps / FRfps (2)
[Reference position of “down swing start”] = X − (“impact instant” position of front still image− “start down swing” position) × BLfps / FRfps (3)
[Reference position of “impact moment”] = X (4)
[Reference position of “follow-through end”] = X + (“follow-through end” position of front still image− “impact instant” position) × BLfps / FRfps (5)

FIG. 11 is a conceptual diagram of the reference position information after determining or updating the reference position information of the still image in each direction.
As shown in FIG. 11, the provisional reference position information of the rear still image is the same as the reference position information of the front still image. The provisional reference position information of the still image of the ball is calculated based on the above formulas (1) to (5).

The control unit 11 reads the reference position information (step S24).
Based on the reference position information, the control unit 11 reads a front still image corresponding to each timing of “takeback start”, “downward start”, “impact moment”, and “follow-through end”, and displays it on the display unit 13. Displayed (step S25).

The control unit 11 determines whether or not all front still images corresponding to the respective timings have been displayed (step S26).
When not displaying (step S26; N), the control part 11 transfers to step S25.
When displayed (step S26; Y), the control unit 11 performs the rear corresponding to the timings of “takeback start”, “downward start”, “impact moment”, and “follow-through end” based on the reference position information. Are read and displayed on the display unit 13 (step S27).

The control unit 11 determines whether all the rear still images corresponding to the respective timings have been displayed (step S28).
When not displaying (step S28; N), the control part 11 transfers to step S27.
When displayed (step S28; Y), the control unit 11 reads a still image of the ball based on the reference position information and displays it on the display unit 13 (step S29).

The control unit 11 displays the synchronization gauge on the display unit 13 (step S30).
FIG. 12 shows a synchronization panel P6 including still images in each direction and synchronization gauges G2 and G3.
On the synchronization panel P6, still images based on the reference position information are displayed in the frames F1 to F14. A front still image is displayed in the frames F1 to F4, a rear still image is displayed in the frames F5 to F8, and a ball still image is displayed in the frames F9 to F14.

On the synchronization panel P6, the synchronization gauges G2 and G3 are displayed at the rear corresponding to the “impact moment” and below the frames F7 and F12 displaying the still image of the ball.
As will be described later, when the synchronization gauges G2 and G3 are operated via the input unit 12, the “provisional” reference position information of the rear still image and the “provisional” reference position information of the ball still image become “ Will be decided.

The control unit 11 determines whether one of the frames F7 and F12 for displaying the rear or the still image of the ball corresponding to the “impact moment” is selected (step S31).
If any frame is not selected (step S31; N), the control unit 11 waits until it is selected.
When any frame is selected (step S31; Y), the control unit 11 sets the still image in the selected frame to be editable (step S32).
In the following description, it is assumed that the frame F7 is selected.

The control unit 11 determines whether there is an editing instruction (step S33).
When there is no editing instruction (step S33; N), the control unit 11 stands by until an instruction is issued.
When there is an editing instruction (step S33; Y), the control unit 11 determines whether or not there is a thinning operation of the gauge G2 (step S34).
Since the thinning-out operation of the gauge G2 is the same as that of the gauge G1, description thereof is omitted here. Further, the operation of “▲” of the gauge G2 and the operation of the slider are the same as those of the gauge G1, and therefore the description thereof is omitted here.

When the thinning operation of the gauge G2 is not performed (step S34; N), the control unit 11 proceeds to step S14.
When the thinning operation is performed on the gauge G2 (step S34; Y), the control unit 11 reads a predetermined thinning amount (step S35).

  The control unit 11 calculates the position of the rear still image in the frame F7 in accordance with the “▲” of the gauge G2 or the operation instruction by the slider (step S36).

The control unit 11 determines whether or not the calculated position of the rear still image is within the range where the still image exists (step S37).
If the still image is not within the existing range (step S37: N), the control unit 11 proceeds to step S33.
When it is within the range where the still image exists (step S37; Y), the control unit 11 switches and displays the rear still image in the frame F7 (step S38).

The control unit 11 updates the reference position information shown in FIG. 11 (step S39).
The control unit 11 determines whether or not the still image to be operated is a rear still image (step S40). That is, it is determined whether or not the frame selected in step S32 is the frame F7 and the operation of the gauge G2 has been performed.

  When the still image to be operated is a rear still image (step S40; Y), the control unit 11 changes the timing in accordance with the still image behind the “impact moment” displayed by switching in step S38. The corresponding rear still image is updated and displayed in the frames F5, F6, and F8 (step S41).

  When the still image to be operated is not a rear still image (step S40; N), that is, when the operation of the gauge G3 is performed, the control unit 11 performs the front and rear operations around the still image of the ball at the “impact moment”. The still image of the ball is updated and displayed in the frames F9 to F11, F13, and F14 (step S42).

The control unit 11 determines whether there is a selection instruction for another frame (here, the frame F12) (step S43).
When there is a selection instruction (step S43; Y), the control unit 11 proceeds to step S33.
When there is no selection instruction (step S43; N), the control unit 11 determines reference position information of still images in all three directions (step S44), and ends the synchronization process.

FIG. 13 shows a conceptual diagram of the reference position information after determining the reference position information of all three directions of still images.
As shown in FIG. 13, the provisional reference position information of the rear still image and the provisional reference position information of the ball still image are finely adjusted by the processing in steps S33 to S44, and the reference of all three directions of the still image is obtained. Location information is determined.

The moving image generation process will be described with reference to FIG.
The control unit 11 reads the reference position information stored in the storage unit 14 (step S51). Specifically, the control unit 11 reads the start image number and the end image number in each direction.

As shown in FIG. 13, the start image number in each direction is the image number of the front still image (hereinafter referred to as “front ST”) corresponding to “takeback start”, and the image number of the rear still image (hereinafter referred to as “ Back ST "), the image number of the still image of the ball (hereinafter," ball ST ").
The end image number is the image number of the front still image (hereinafter referred to as “front ED”) corresponding to “follow-through end”. In the present embodiment, the front ST is “001558”, the rear ST is “0001092”, and the ball ST is “0005386”. The front ED is “0002023”.

The control unit 11 reads still images in three directions based on the image number (steps S52 to S54).
The three-direction still images are front, rear, and ball still images corresponding to “takeback start”.

The control unit 11 lays out and synthesizes the read still images in the three directions into one still image (step S55).
The control unit 11 stores the synthesized still image in the storage unit 14 (step S56).

The control unit 11 increments the front ST and the rear ST by +1 (steps S57 and S58), and increments the ball ST by + Δd (step S59).
Δd is determined by BLfps / FRfps. BLfps and FRfps are the frame rate information of the ball still image group and the front still image group stored in step S4, as described above. In the present embodiment, Δd = BLfps / FRfps = 1200/300 = “4”.

The control unit 11 determines whether or not front side ST> front side ED (step S60).
When front ST> front ED is not satisfied (step S60; N), the controller 11 proceeds to step S52.
When front ST> front ED (step S60; Y), the control unit 11 converts a plurality of synthesized still images into moving images (step S61), and ends the moving image generation process.
The moving image generated by conversion in step S61 is a moving image for one-screen display that can be written to a portable storage medium. After the moving image conversion process, the control unit 11 may write the converted moving image for one-screen display on a portable storage medium such as a DVD by the media writing unit 6.

FIG. 15 shows an example of a one-screen display moving image D1 that can be written to a portable storage medium.
In the moving image D1 for one-screen display, a front moving image, a rear moving image, and a ball moving image are simultaneously displayed on one screen. When the moving image D1 is reproduced, the front moving image, the rear moving image, and the ball moving image are reproduced in synchronization with each other.

  As described above, according to the present embodiment, it is possible to convert a moving image obtained by capturing moving object trajectories from a plurality of directions into a synchronized moving image and a moving image writable in a portable storage medium. . In addition, moving images having different frame rates can be combined and converted into a moving image for one-screen display. By browsing the converted moving image, the trajectory of the moving object can be grasped from multiple angles.

  Also, a synchronized moving image and portable storage medium for a moving image in which a trajectory of a swing operator during a golf swing operation as an example of a hitting ball game and a trajectory of a golf ball as an example of a hitting object are taken from a plurality of directions. Can be converted into a writable moving image.

  In addition, in a moving image taken in any one direction from among a plurality of moving images taken from a plurality of directions, it is possible to synchronize moving images taken from a plurality of directions based on four predetermined timings.

  Further, the input unit can further finely adjust a plurality of synchronized moving images to achieve highly accurate synchronization.

In addition, the media writing unit can write a synchronized moving image on a portable storage medium such as a DVD.
The description in the above embodiment shows an example of the present invention, and the present invention is not limited to this.
For example, the moving image handled in the present invention is not limited to a movie of shooting a golf swing, but can be applied to a moving image of a moving object, and is not limited to golf swings and golf balls. The present invention can be widely applied to a person who takes a picture of a swing operator and a hitting object.
For example, as a batting ball game other than golf, the movement trajectory of the baseball bat swing and the movement trajectory of the ball, the movement trajectory of the tennis racket swing and the movement trajectory of the ball, or the movement of the table tennis racket swing are taken. The present invention can be applied to photographing a locus and a movement locus of a ball, or photographing a movement locus of an ice hockey racket swing and a movement locus of a pad (ball).

DESCRIPTION OF SYMBOLS 100 Image processing system 10 Image processing apparatus 11 Control part 12 Input part 13 Display part 14 Storage part 15 Connection part 1-3 High speed camera 4 HUB
5 Printer 6 Media writing unit

Claims (6)

An image processing apparatus for controlling reproduction of a plurality of moving images,
At the same time when reproducing a plurality of moving images captured at different frame rates, the reference frame as a synchronization reference of the moving image, and setting means for setting arbitrarily for respective each video image,
Determination for individually determining a playback start frame for each moving image so that a reference frame of each moving image set by the setting means is played at the same playback timing when the moving images are played back simultaneously Means,
Wherein when simultaneously reproducing the moving image, the causes of the respective reproduction start frame of each moving image individually reproduction started, at least the control so that the reference frame of each of the moving image is reproduced at the same reproduction timing Playback control means to
An image processing apparatus comprising: The setting means designates a predetermined first reference frame from a plurality of frames included in the first moving image, and a second one to be synchronized with the first reference frame among the plurality of frames included in the second moving image. By arbitrarily selecting a reference frame, a reference frame serving as a synchronization reference for each moving image is set.
The image processing apparatus according to claim 1. The reproduction control means controls the moving images to be reproduced and displayed simultaneously on the same display screen.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The playback start frame of each moving image determined by the determining means is a playback start frame when starting playback of each moving image simultaneously.
The image processing apparatus according to any one of claims 1 to 3. The plurality of moving images are moving images taken by a plurality of photographing means for the same subject.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A program for controlling a computer of an image processing apparatus for controlling reproduction of a plurality of moving images,
The computer,
Setting means for simultaneously when reproducing a plurality of moving images captured at different frame rates, the reference frame as a synchronization reference of the moving image is set arbitrarily for respective each video image,
Determination for individually determining a playback start frame for each moving image so that a reference frame of each moving image set by the setting means is played at the same playback timing when the moving images are played back simultaneously means,
Wherein when simultaneously reproducing the moving image, the causes of the respective reproduction start frame of each moving image individually reproduction started, at least the control so that the reference frame of each of the moving image is reproduced at the same reproduction timing Playback control means,
A computer-readable program designed to function as a computer.

Images