JP6075356B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for comparing operations in images.

Conventionally, an operation such as a golf swing is imaged and compared with a moving image as an example.
For example, in the image recording / reproducing apparatus described in Patent Document 1, in two moving images to be compared, a specific reproduction point of each moving image is designated, and the designated reproduction points are aligned to reproduce two moving images. .

JP-A-10-304299

  However, in the conventional techniques including the technique described in Patent Document 1, since the user visually detects the difference between the operation of one moving image and the operation of the other moving image as a model, It was difficult to clearly understand the differences.

  The present invention has been made in view of such a situation, and an object of the present invention is to make it easier to grasp a difference in motion of a subject in an image.

A first image group in which a series of motions of the first subject is imaged and a second image group in which a series of motions of the second subject corresponding to the series of motions of the first subject are captured are acquired. Image acquisition means;
Setting means for setting a reference line serving as a reference for operation on subjects in the first image group and the second image group acquired by the image acquisition means;
The operation speed in the series of operations of the first subject in the first image group acquired by the image acquisition means is compared with the operation speed in the series of operations of the second subject in the second image group. An operation comparison means;
Reproduction control means for controlling the reproduction speed of the first image group or the second image group based on the comparison result of the operation comparison means;
Equipped with a,
The operation comparison unit includes a first reference line in a series of operations of the first subject in the first image group set by the setting unit, and the second image group set by the setting unit. Is compared with the second reference line in the series of operations of the second subject in the above, the operation speed in the series of operations of the first subject and the operation speed in the series of operations of the second subject. The comparison result is obtained .

  According to the present invention, it is possible to more easily grasp the difference in motion of a subject in a plurality of images.

1 is a block diagram illustrating a hardware configuration of an image processing apparatus according to an embodiment of the present invention. It is a functional block diagram which shows the functional structure for performing an operation | movement comparison process among the functional structures of the image processing apparatus 1 of FIG. It is a figure which shows the specific example which sets an operation | movement reference axis in an example of the image of a frame, and detects operation | movement. It is a figure which shows the specific example which sets an operation | movement reference axis and detects operation | movement in the other example of the image of a flame | frame. It is a figure which shows the specific example which sets the operation | movement reference axis of a golf club. 3 is a flowchart showing an example of a flow of operation comparison processing executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG. 2. It is a figure which shows the example of a display of the comparison result of operation | movement. It is a flowchart which shows an example of the flow of the axis | shaft production | generation process which the image processing apparatus 1 of FIG. 1 which has the function structure of FIG. 2 performs. It is a figure which shows the example of the comparison result of the operation | movement in a modification. 3 is a flowchart showing a flow of operation speed correction display processing executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG. 2. It is a schematic diagram which shows the state which displays the data of a moving image in association with motion information by motion speed correction display processing. 3 is a flowchart showing a flow of comparison result display processing executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG. 2.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
(Constitution)
FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus according to an embodiment of the present invention.
The image processing apparatus 1 is configured as a digital camera, for example.

  The image processing apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an imaging unit 16, and an input unit. 17, an output unit 18, a storage unit 19, a communication unit 20, and a drive 21.

  The CPU 11 executes various processes according to a program loaded in the RAM 13 from the storage unit 19 such as a program recorded in the ROM 12 or a program for operation comparison processing.

  The RAM 13 appropriately stores data necessary for the CPU 11 to execute various processes.

  The CPU 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output interface 15 is also connected to the bus 14. An imaging unit 16, an input unit 17, an output unit 18, a storage unit 19, a communication unit 20, and a drive 21 are connected to the input / output interface 15.

  Although not shown, the imaging unit 16 includes an optical lens unit and an image sensor.

The optical lens unit is configured by a lens that collects light, for example, a focus lens or a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.

The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. Through various signal processing, a digital signal is generated and output as an output signal of the imaging unit 16.
Hereinafter, the output signal of the imaging unit 16 is referred to as “captured image data”. Data of the captured image is appropriately supplied to the CPU 11, RAM 13, and the like.

The input unit 17 includes various buttons and the like, and inputs various types of information according to user instruction operations.
The output unit 18 includes a display, a speaker, and the like, and outputs images and sounds.
The storage unit 19 is composed of a hard disk or a DRAM (Dynamic Random Access Memory) or the like, and stores various image data.
The communication unit 20 controls communication performed with other devices (not shown) via a network including the Internet.

  A removable medium 31 composed of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 21. The program read from the removable medium 31 by the drive 21 is installed in the storage unit 19 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 19 in the same manner as the storage unit 19.

FIG. 2 is a functional block diagram showing a functional configuration for executing the operation comparison process among the functional configurations of the image processing apparatus 1.
The action comparison process is a series of processes for comparing the action of the imaged subject with the action to be compared and displaying the difference.
In this embodiment, an example in which a golf swing of a player imaged by the image processing apparatus 1 is compared with a golf swing of a professional golfer acquired in advance will be described. However, it is also possible to compare various operations such as comparing a player's current golf swing taken with the image processing device 1 with a past golf swing.

  When the operation comparison process is executed, as shown in FIG. 2, in the CPU 11, the imaging control unit 41, the image acquisition unit 42, the operation detection unit 43, the operation comparison unit 44, and the comparison result display control unit. 45 functions.

  However, FIG. 2 is an exemplification, and at least a part of the functions of the CPU 11 may be provided with hardware that performs image processing such as GA (Graphics Accelerator), and may be delegated to this.

When the operation comparison process is executed, an acquired image storage unit 51 provided as one area of the storage unit 19 and a comparison information storage unit 52 are used.
In the acquired image storage unit 51, data of a captured image output from the imaging unit 16 and acquired by the image acquisition unit 42 is stored.
The comparison information storage unit 52 stores comparison moving image data (described later) output from the operation comparison unit 44. In the present embodiment, the motion reference data (to be described later) is set by the motion detection unit 43 and the motion reference axis data is attached to the moving image data of the professional golfer to be compared. Assume that it is stored in the unit 52.

The imaging control unit 41 sets various imaging conditions and controls the imaging operation in the imaging unit 16.
In the present embodiment, in order to capture a series of motions of the subject, the imaging control unit 41 causes the imaging unit 16 to continuously capture the subject as a single image and output it as moving image data.

  The image acquisition unit 42 acquires moving image data output from the imaging unit 16. Thereafter, the image acquisition unit 42 stores the moving image data in the acquired image storage unit 51.

  The motion detection unit 43 refers to the moving image data stored in the acquired image storage unit 51 in accordance with the operation comparison processing start instruction input via the input unit 17, and configures each moving image data. The motion of the subject is detected in the frame image. Specifically, the motion detection unit 43 performs contour extraction by applying an edge detection filter to each frame image constituting the moving image data, and noise (unnecessary contour line) of the extracted contour line is extracted. Perform removal processing. That is, the motion detection unit 43 has a contour extraction function for extracting the contour of the subject. In addition, the motion detection unit 43 specifies a closed region representing the body in the image on which the noise removal process has been performed. In the present embodiment, a model (hereinafter referred to as “body model”) that defines the contours and joint positions of the body is stored in advance, and a closed region that represents the body by referring to the contour of the body model. Is detected. Furthermore, the motion detection unit 43 extracts end points and bending points in the longitudinal direction of the closed region representing the body. Then, the motion detection unit 43 sets each motion reference axis serving as a body axis during motion with reference to the joint position of the body model based on the extracted end points and bending points. That is, the motion detection unit 43 has an axis setting function for setting the motion reference axis. The motion detection unit 43 detects the motion of the player by tracking the motion reference axis between a plurality of frames.

Since the end point and the bending point in the contour line can be detected relatively easily, the operation reference axis can be easily set by using these.
When detecting a player's movement for moving images from the front at the time of addressing, the arm that leads the swing (the left arm for a right-handed player) operates without being hidden in the main process of the swing. Therefore, in this embodiment, the upper arm portion of the arm that leads the swing is specified as the motion reference axis that is most closely watched. Hereinafter, the motion reference axis set for the upper arm portion of the left arm is appropriately referred to as a “main motion reference axis”, and the other motion reference axes are appropriately referred to as “auxiliary motion reference axes”.

  The motion detection unit 43 outputs, to the motion comparison unit 44, data indicating each set motion reference axis for each frame image constituting the moving image data in association with each frame image.

  The motion comparison unit 44 refers to the motion reference axis set by the motion detection unit 43, and compares the motion of the subject in the data of two different moving images. Specifically, the motion comparison unit 44 compares the angle of the main motion reference axis in each frame in a series of motions in the player's moving image and the moving image of the professional golfer to be compared, and Identify the corresponding action. For example, the motion comparison unit 44 sets the rotation angle in the counterclockwise direction with the upward direction of the frame being 0 degree, and specifies a predetermined motion in the swing based on the rotation angle of the main motion reference axis. Then, the motion comparison unit 44 matches the motion of the main motion reference axis in a series of motions while collating the motion of the main motion reference axis in the data of the player's motion image and the data of the moving image of the professional golfer to be compared. An image of a frame having substantially the same rotation angle is specified as a corresponding operation image.

  Further, the motion comparison unit 44 compares the corresponding motion images in the moving image of the player and the moving image of the professional golfer to be compared, and from the position and rotation angle of the motion reference axis set in the image, Detect differences in subject movement. Specifically, the difference between the position and rotation angle of the motion reference axis of each frame in the moving image of the professional golfer and the position and rotation angle of the motion reference axis in the player's moving image are calculated. Data indicating the difference (hereinafter referred to as “comparison result data” as appropriate) is added. As the comparison result data, for example, a vector and an angle indicating how much the moving direction and the rotation angle of the motion reference axis of interest differ from the movement direction and the rotation angle of the corresponding motion reference axis in the comparison target moving image are added. The

Then, the motion comparison unit 44 generates video data of the player with the comparison result data (hereinafter referred to as “comparative video data” as appropriate) and stores the comparison video data in the comparison information storage unit 52. To do.
Thereby, the moving image data for comparison becomes moving image data including comparison result data indicating a difference from the moving image of the professional golfer.

  When the operation comparison by the operation comparison unit 44 is completed, the comparison result display control unit 45 reads and decodes the moving image data for comparison from the comparison information storage unit 52, and displays the operation reference axis superimposed on the moving image of the player for comparison. Display video data.

(Motion detection method)
Next, the operation detection method in the operation detection unit 43 will be specifically described.
FIG. 3 is a diagram illustrating a specific example in which an operation is detected by setting an operation reference axis in an example of a frame image. The example of the image of the frame shown in FIG. 3 shows the state of the subject when the golf club is located near the tipeg at the time of addressing or impact.

  The motion detection unit 43 first applies an edge detection process to the frame image shown in FIG. 3A using an edge detection filter. As a result, as shown in FIG. 3B, the contour line of the image in the entire frame including the contour line of the subject is detected.

  Next, the motion detection unit 43 removes the floating contour line and the contour line deviating from the center of the frame as noise from the frame image after edge detection shown in FIG. Is identified as the outline of the subject (player). Thereby, as shown in FIG.3 (c), the outline of the to-be-photographed object (player) imaged in the frame center part is extracted. It should be noted that since the contour line of the player as the subject can model a typical shape, a model of the body contour line can be prepared in advance. By collating with a model of a body contour line prepared in advance, it becomes possible to specify the contour line of the subject with higher accuracy.

  Next, the motion detection unit 43 extracts the end points and the bending points in the longitudinal direction of the closed region representing the contour line of the subject from the frame image shown in FIG. 3C, and refers to the joint position of the body model. Then, the motion reference axis of each part of the body is set. That is, the motion detection unit 43 traces the contour line from the end point in the longitudinal direction to the inside in the closed region representing the contour line of the subject, and extracts the portion where the contour line is bent as the bending point. In the case of the contour line of the player's body, the bending points appear in pairs in the left-right direction of the frame (for example, the inner side and the outer side of one ankle, the outer side and the inner side of the knee, the left and right parts of the waist). Therefore, the motion detection unit 43 sets each motion reference axis by connecting the center portion of the pair of bending points as the joint position of the body model with the end point in the longitudinal direction as a starting point. Note that the motion reference axis is set to a body part that is a measure of motion in addition to an axis that connects end points and joints. For example, a motion reference axis that connects the left and right flexion points of the waist is set to indicate hip rotation, and the center and neck of the motion reference axis that connects the left and right flexion points of the upper body are shown to indicate the inclination of the upper body. Is set as a reference axis.

  Specifically, the motion detection unit 43 detects the bending point located on the ankle from the end point located on the toe of the subject to the middle point of the bending point located on the ankle with respect to the image of the frame shown in FIG. Positioned at the top of the head from the midpoint to the midpoint of the flexion point located at the knee, from the left part of the hip to the right part, and from the midpoint of the flexion point located at the knee to the intersection with the axis connecting the left and right parts of the hip The motion reference axis is set from the end point to the middle point of the bending point located at the neck. Thereby, as shown in FIG.3 (d), the operation | movement reference axis of a head and a lower half body is set.

  Next, the motion detection unit 43 moves from the tip of the subject (bending point holding the golf club) to the middle point of the bending point located on the wrist, from the middle point of the bending point located on the wrist to the bending point located on the elbow. The motion reference axes are set from the middle point of the bending point located on the elbow to the middle point of the bending point located on the shoulder up to the middle point. Thereby, as shown in FIG.3 (e), the motion reference axis from a shoulder to an arm tip is set.

  Next, the motion detection unit 43 extends from the left part of the waist to the right part (the positions of the left and right waist parts) from the left shoulder to the right shoulder (the position of the left and right shoulder joints defined based on the bending points of the left and right shoulder parts). The motion reference axis is set up to the waist reference position defined based on the bending point. Thereby, as shown in FIG. 3F, the horizontal motion reference axis of the upper body is set.

Furthermore, the motion detection unit 43 sets the motion reference axis from the midpoint of the bending point located at the subject's neck to the midpoint of the axis connecting the left and right portions of the waist. Thereby, as shown in FIG. 3G, an operation reference axis representing the rotation axis of the upper body is set.
FIG. 4 is a diagram illustrating a specific example in which an operation is detected by setting an operation reference axis in another example of a frame image. The example of the frame image shown in FIG. 4 shows the state of the subject when the golf club is at a position higher than the player's head, such as before the top after the backswing.

Also in the case illustrated in FIG. 4, the motion detection unit 43 extracts the end points and the bending points in the long direction of the closed region representing the contour line of the subject, similarly to the case illustrated in FIG. 3, and determines the joint position of the body model. Referring to, a motion reference axis for each part of the body is set. In the state illustrated in FIG. 4, the right arm is hidden, and the contour line cannot be specified. Therefore, the motion detection unit 43 does not set the motion reference axis related to the right arm. On the other hand, when the right arm appears again and the contour can be specified, the motion detection unit 43 sets the motion reference axis again.
Here, since the golf club held by the player is a straight rod-like member, it can be easily distinguished from the player's body, and a unique motion reference axis is set for the golf club.

However, since the golf club may be hidden behind the user, in this case, the motion detection unit 43 recognizes the straight bar-shaped member as a golf club.
FIG. 5 is a diagram showing a specific example of setting a motion reference axis of a golf club. The example of the image of the frame shown in FIG. 5 shows a state in which the golf club is hidden behind the player's head, such as when the player is in the top state after the backswing.
In the case shown in FIG. 5, the contour lines of the divided golf clubs appear on the left and right sides of the player's head, but the motion detection unit 43 connects the divided golf clubs on the left and right sides. An operation reference axis is set as the contour line of the golf club.

(Operation)
Next, the operation will be described.
(Operation comparison process)
First, an operation comparison process (main flow) executed by the image processing apparatus 1 will be described.
FIG. 6 is a flowchart showing an example of the flow of operation comparison processing executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG.
The operation comparison process is started in response to an instruction input of execution of the operation comparison process by the user via the input unit 17.

In FIG. 6, when the motion comparison process is started, the motion detection unit 43 acquires and decodes moving image data from the acquired image storage unit 51 in step S <b> 1, and stores decoded frame data in the buffer of the RAM 13. Store in the area.
In step S <b> 2, the motion detection unit 43 detects the motion of the subject in the decoded frame image by executing an axis generation process described later.

  In step S3, the motion comparison unit 44 refers to the motion reference axis set by the motion detection unit 43, compares the moving image of the player with the motion of the subject in the moving image data to be compared, and compares the result. Comparison video data with data attached is generated. The comparison moving image data is stored in the comparison information storage unit 52.

  In step S4, the comparison result display control unit 45 reads the comparison moving image data stored in the comparison information storage unit 52, and compares the moving image of the player with the operation reference axis in the moving image data to be compared. Display while identifying. At this time, the comparison result display control unit 45 identifies and displays a portion of each motion reference axis of the moving image of the player that is different from the motion reference axis in the moving image to be compared with a different color.

FIG. 7 is a diagram illustrating a display example of the operation comparison result. 7A shows one frame in the moving image of the professional golfer to be compared, and FIG. 7B shows one frame in the moving image of the player.
In the example shown in FIG. 7, the movement of the player's right thigh is open relative to the movement of the right golf thigh of the professional golfer to be compared (a state in which the outward transition is large). Therefore, in FIG. 7, the motion reference axis set to the right thigh in the moving image of the player is displayed with a special color (for example, red) that calls attention, and is distinguished from the color (for example, green) of the other motion reference axis. Is displayed. As a result, the difference between the player's action and the professional golfer's action can be clearly identified and displayed.

In step S5, the comparison result display control unit 45 determines whether or not the comparison moving image data is the last frame.
If the comparison moving image data is not the final frame, NO is determined in step S5, and the process proceeds to step S1.
On the other hand, if the comparison moving image data is the last frame, it is determined YES in step S5, and the operation comparison process ends.

(Axis generation processing)
Next, the axis generation process executed in step S2 of the operation comparison process will be described.
FIG. 8 is a flowchart showing an example of the flow of axis generation processing executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG.
When the axis generation process is started, the motion detection unit 43 reads one frame of decoded frame data stored in the buffer area of the RAM 13 in step S11.

In step S <b> 12, the motion detection unit 43 applies an edge detection filter to the frame image. At this time, noises such as background contour lines are removed from the contour lines extracted by the edge detection filter.
In step S13, the motion detection unit 43 refers to the body model in the contour extracted by the edge detection filter, detects a closed region representing the body, and operates based on the end points and the bending points in the longitudinal direction. Set the reference axis.

In step S <b> 14, the motion detection unit 43 sets an axis connecting the left and right hips of the motion reference axis, and an axis (horizontal axis) connecting the left shoulder and the right shoulder.
After the process of step S14, the process returns to step S3 of the operation comparison process.

As described above, in the image processing apparatus 1 according to the present embodiment, the image acquisition unit 42 is a professional golfer in which the image of the subject in which the motion of the subject is captured and the motion of the professional golfer to be compared with the motion of the player is captured. The image group of is acquired. The motion detection unit 43 having an axis setting function sets a motion reference axis in the moving image of the player and the moving image of the professional golfer. Then, the motion comparison unit 44 compares the moving image of the player with the moving image of the professional golfer to be compared based on the motion reference axis. Further, the comparison result display control unit 45 performs control for displaying the difference between the moving image of the player and the moving image of the professional golfer based on the comparison result of the operation comparing unit 44.
Therefore, it is possible to accurately compare the moving image of the player and the moving image of the professional golfer based on the difference in motion of the motion reference axis, and to clearly detect the difference in motion.

Accordingly, since the difference between the moving image to be compared and the moving image of the player can be displayed in a form that can be recognized more easily, it is possible to more easily grasp the difference in the operation to be compared.
The motion detection unit 43 having an axis setting function sets a motion reference axis in at least one of the player and the professional golfer, and the motion comparison unit 44 is based on the motion reference axis set by the motion detection unit 43. Compare differences in player and professional golfer behavior.
Therefore, it is possible to clearly detect the motion of different subjects based on the difference in motion of the motion reference axis.
In addition, the motion detection unit 43 extracts the contour line of the subject by the contour line extraction function, and sets the motion reference axis based on the end point and the bending point of the contour line extracted by the axis setting function.
Therefore, it is possible to set an appropriate motion reference axis from the image of the subject and compare the motion of the subject.
Further, when the movements of the player and the professional golfer are different, the comparison result display control unit 45 identifies and displays at least one of the set motion reference axes.
Therefore, it is possible to more easily grasp the difference in operation between the player and the professional golfer.
Further, the comparison result display control unit 45 displays the motion reference axis superimposed on the moving image of the player.
Therefore, it becomes possible to grasp the movement of the player more easily.
In the present embodiment, the motion detection unit 43 refers to the body model, extracts the contour line representing the body of the subject, and sets each joint position, but without referring to the body model, It is also possible to analyze the detected contour line and set a contour line representing the body of the subject and each joint position. In this case, the motion reference axis may be set with the end point of the closed region of the contour line representing the body as the starting point, and only the motion reference axis that can be more reliably set based on the contour line may be set. Further, the joint position may be set at a position that is easy to define, such as an intermediate point of the bending point in the contour line or an intersection of the contour line and the motion reference axis.

(Modification)
In the first embodiment, the case where the difference in motion reference axis between the moving image of the player and the moving image to be compared is identified and displayed in different colors has been described as an example.
On the other hand, in addition to the color or instead of the color, the direction and size of the motion difference of the motion reference axis between the moving image of the player and the moving image to be compared is superimposed with an index such as an arrow. It is also possible to display.
FIG. 9 is a diagram illustrating an example of a comparison result of operations in the present modification.

In FIG. 9, it is indicated by arrows that the right leg moves inward and the arm moves in the backswing direction in the player movement relative to the professional golfer movement. This arrow is displayed in a size corresponding to the difference between the movement of the professional golfer to be compared and the movement of the player.
As shown in FIG. 9, the comparison result display control unit 45 indicates the comparison result data associated with the motion reference axis as an arrow (an index indicating the direction and size of the difference in motion). On the other hand, it is possible to make a display form that can more easily grasp in which direction the movement of the player is different.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.
In the first embodiment, a case has been described in which the moving image for comparison is reproduced by reproducing the moving image data for comparison so that the movement of the body motion reference axis in the moving image of the player is compared with the movement of the professional golfer to be compared.

On the other hand, in the present embodiment, the comparative moving image data and the moving image data of the professional golfer to be compared are reproduced side by side, and even if there is a difference in swing speed between the player's operation and the professional golfer's operation, The moving image data is displayed in a state where the operation information is associated.
Even in this case, the edge detection filter is applied to the data of each frame in the moving image data, and the moving image data for comparison is generated by setting the operation reference axis, as in the first embodiment.

That is, the hardware configuration and the functional configuration of the image processing apparatus 1 according to the present embodiment are the same as those in FIGS. 1 and 2 in the first embodiment. A flow of processing for correcting and displaying a speed difference of a certain operation will be described.
FIG. 10 is a flowchart showing the flow of the operation speed correction display process executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG. FIG. 11 is a schematic diagram showing a state in which moving image data is displayed in association with operation information by the operation speed correction display process. Note that the angle shown in the frame in FIG. 11 represents the rotation angle of the main motion reference axis.

The operation speed correction display process is started in response to an instruction input from the user via the input unit 17 for execution of the operation speed correction display process.
In FIG. 10, when the operation speed correction display process is started, in step S101, the comparison result display control unit 45 reads the comparison moving image data and the data of the moving image of the professional golfer to be compared from the comparison information storage unit 52. Decode the first frame.
In step S102, the comparison result display control unit 45 detects the rotation angle of the main motion reference axis in each decoded frame.

In step S103, the comparison result display control unit 45 determines whether or not the rotation angle of the main motion reference axis is different in the decoded frames of the comparison moving image data and the moving image data of the professional golfer to be compared. Do.
If the rotation angle of the main motion reference axis is not different in the decoded frame of the moving image data for comparison and the moving image data of the professional golfer to be compared, NO is determined in step S103, and the process proceeds to step S105. To do. In the first frame of FIG. 11, since the rotation angles of the respective frames coincide with each other, NO is determined in step S103.

On the other hand, if the rotation angle of the main motion reference axis is different in the decoded frames of the comparative moving image data and the moving image data of the professional golfer to be compared, it is determined YES in step S103, and the processing is performed. The process proceeds to step S104.
In step S104, the comparison result display control unit 45 sequentially acquires frames of moving image data with a delayed back swing (that is, moving image data with a larger rotation angle of the main motion reference axis), Decoding is performed up to a frame that is the same as the rotation angle of the main motion reference axis in the other moving image data.

In step S105, the comparison result display control unit 45 displays the current frame of the moving image data for comparison and the moving image data of the professional golfer to be compared.
In step S106, the comparison result display control unit 45 decodes the next frame in each of the moving image data for comparison and the moving image data of the professional golfer to be compared.

In step S107, the comparison result display control unit 45 detects the rotation angle of the main motion reference axis in each decoded frame.
In step S108, the comparison result display control unit 45 determines whether or not the rotation angle of the main motion reference axis is different in the decoded frames of the comparison moving image data and the moving image data of the professional golfer to be compared. Do.

  If the rotation angle of the main motion reference axis is not different in the decoded frames of the moving image data for comparison and the moving image data of the professional golfer to be compared, NO is determined in step S108, and the process proceeds to step S112. To do. In the example shown in FIG. 11, since the rotation angle of the main motion reference axis in each frame is different in the frame of frame number 6, NO is determined in step S108.

On the other hand, when the rotation angle of the main motion reference axis is different in the decoded frame of the comparison moving image data and the moving image data of the professional golfer to be compared, it is determined YES in step S108, and the processing is performed. The process proceeds to step S109.
In step S109, the comparison result display control unit 45 has a larger rotational angular velocity of the main motion reference axis in the moving image data of the player than in the moving image data of the professional golfer to be compared (that is, in the previous frame). It is determined whether or not there is a large difference between the rotation angle of the main motion reference axis and the rotation angle of the main motion reference axis of the current frame.

If the player moving image data has a larger rotational angular velocity of the main motion reference axis than the professional golfer moving image data to be compared, YES is determined in step S109, and the process proceeds to step S110. .
On the other hand, if the player's moving image data does not have a larger rotational angular velocity of the main motion reference axis than the professional golfer's moving image data to be compared, NO is determined in step S109, and the processing is performed. The process proceeds to step S111. In the example shown in FIG. 11, the data of the professional golfer's moving image has a larger rotational angular velocity of the main motion reference axis, and therefore NO is determined in step S109.

In step S110, the comparison result display control unit 45 decodes the next frame of the moving image data of the professional golfer to be compared.
In step S111, the comparison result display control unit 45 decodes the next frame of the moving image data of the player.
Through the processing in steps S110 and S111, as shown in FIG. 11, frames in the moving image data having the slower swing speed are thinned out.

In step S112, the comparison result display control unit 45 detects the rotation angle of the main motion reference axis in each decoded frame.
After such step S112, the process proceeds to step S108.
In step S113, the comparison result display control unit 45 displays the current frame of the comparison moving image data and the moving image data of the professional golfer to be compared.
In step S114, the comparison result display control unit 45 determines whether any of the comparison moving image data and the moving image data of the professional golfer to be compared is the last frame.

If either the comparison moving image data or the moving image data of the professional golfer to be compared is not the last frame, NO is determined in step S114, and the process proceeds to step S106.
If either the comparison moving image data or the moving image data of the professional golfer to be compared is the last frame, YES is determined in step S114, and the operation speed correction display process is ended.

As described above, in the image processing apparatus 1 according to the present embodiment, the comparison result display control unit 45 reproduces the moving image data for comparison and the moving image data of the professional golfer to be compared, and the difference in the swing speed of the subject. If there is, the frame in the moving image data having the slower swing speed is thinned to correct the speed difference.
For this reason, in the moving image data for comparison and the moving image data of the professional golfer to be compared, the information on the motion of the subject to be reproduced is in a corresponding state.

Accordingly, since the difference between the moving image to be compared and the moving image of the player can be displayed in a form that can be recognized more easily, it is possible to more easily grasp the difference in the operation to be compared.
In the present embodiment, when displaying the moving image data for comparison and the moving image data of the professional golfer, the motion reference axis set in each frame is displayed superimposed on the subject as shown in the first embodiment. It's also good.

(Third embodiment)
Next, a third embodiment of the present invention will be described.
In the second embodiment, the comparative moving image data and the moving image data of the professional golfer to be compared are reproduced side by side, and even if there is a difference in swing speed between the player's motion and the professional golfer's motion, the slower motion For the moving image data, frames were thinned out.

On the other hand, in this embodiment, when there is a difference in swing speed between the player's action and the professional golfer's action, the moving picture data with the slower action is reproduced as usual, and the moving picture with the faster action is reproduced. In the data, the same frame is repeatedly displayed and the frame is interpolated to display moving image data in a state in which the player's motion and the professional golfer's motion are associated with each other.
Even in this case, the edge detection filter is applied to the data of each frame in the moving image data, and the moving image data for comparison is generated by setting the operation reference axis, as in the first embodiment.

That is, the hardware configuration and the functional configuration of the image processing apparatus 1 according to the present embodiment are the same as those in FIGS. 1 and 2 in the first embodiment. A flow of processing for identifying and displaying a comparison result of an operation will be described.
FIG. 12 is a flowchart showing the flow of comparison result display processing executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG.

The comparison result display process is started in response to an instruction input of execution of the comparison result display process by the user via the input unit 17.
In FIG. 12, when the comparison result display process is started, in step S201, the comparison result display control unit 45 acquires the moving image data for comparison and the moving image data of the professional golfer to be compared from the comparison information storage unit 52. The decoded frame data is stored in the buffer area of the RAM 13.

In step S202, the comparison result display control unit 45 determines that the rotation angle of the main motion reference axis in the decoded frame of the comparison moving image data and the moving image data of the professional golfer to be compared is Determine whether they are different.
If the rotation angle of the main motion reference axis is not different in the decoded frames of the moving image data for comparison and the moving image data of the professional golfer to be compared, NO is determined in step S202, and the process proceeds to step S204. To do.

On the other hand, if the rotation angle of the main motion reference axis is different in the decoded frame of the comparison moving image data and the moving image data of the professional golfer to be compared, it is determined as YES in step S202, and the processing is performed. The process proceeds to step S203.
In step S <b> 203, the comparison result display control unit 45 decodes the next frame of the moving image data whose operation is slower, and stores the decoded frame data in the buffer area of the RAM 13.
After such step S203, the process proceeds to step S201.

In step S204, the comparison result display control unit 45 determines whether or not the frame number of the point at which the difference in operation speed is detected. The frame number of the point for detecting the difference in operation speed is set in advance for each fixed number of frames in the data of the moving image of the professional golfer to be compared.
If it is not the frame number of the point at which the difference in operating speed is detected, NO is determined in step S204, and the process proceeds to step S201.

On the other hand, if the frame number of the point at which the difference in operating speed is detected, YES is determined in step S204, and the process proceeds to step S205.
In step S205, the comparison result display control unit 45 determines whether or not there is a difference in the number of decoded frames in the comparison moving image data and the moving image data of the professional golfer to be compared.

If there is no difference in the number of decoded frames in the moving image data for comparison and the moving image data of the professional golfer to be compared, NO is determined in step S205, and the process proceeds to step S206.
On the other hand, if there is a difference in the number of decoded frames in the moving image data for comparison and the moving image data of the professional golfer to be compared, YES is determined in step S205, and the process proceeds to step S207.

In step S206, the comparison result display control unit 45 acquires frame data stored in the buffer area of the RAM 13, and displays a moving image without performing identification display.
In step S207, the comparison result display control unit 45 acquires frame data stored in the buffer area of the RAM 13, and there is a difference in swing speed between the moving image data for comparison and the moving image data of the professional golfer to be compared. A moving image is displayed. At this time, the comparison result display control unit 45 displays the same frame image multiple times as to the moving image data with the faster swing speed, and operates the moving image data frame with the slower swing speed. Displays the image of the corresponding frame.

As a result, the moving image data with the faster swing speed is interpolated by repeatedly displaying the same frame, and the moving image data with the slower swing speed is displayed so as to catch up. In this state, it is identified and displayed that there is a difference in swing speed, such as displaying a frame surrounding the frame. Therefore, it is possible to display an operation in a frame image having a difference in swing speed in a form that can be easily recognized.
After such step S207, the process proceeds to step S201.

  As described above, in the image processing apparatus 1 according to the present embodiment, the comparison result display control unit 45 reproduces the moving image data for comparison and the moving image data of the professional golfer to be compared, and the difference in the swing speed of the subject. If there is, the moving image data with the slower operation is reproduced as usual, and the same frame is repeatedly displayed (interpolated and displayed) in the moving image data with the earlier operation. When the same frame is repeatedly displayed, it is identified and displayed that there is a difference in swing speed, such as displaying a frame surrounding the frame.

  Therefore, moving image data is displayed in a state in which the player motion and the professional golfer motion information are associated with each other, and the motion in the frame image having a difference in swing speed is displayed in a form that is easier to recognize.

Therefore, since the difference between the moving image to be compared and the moving image of the player can be displayed in a form that can be recognized more easily, it is possible to more easily grasp the difference in motion of the subject in the plurality of images.
In the present embodiment, when displaying the moving image data for comparison and the moving image data of the professional golfer, the motion reference axis set in each frame is displayed superimposed on the subject as shown in the first embodiment. It's also good.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

  In the above-described embodiment, the case where the present invention is applied has been described as an example for moving image data in which a golf swing is copied. However, the present invention can be applied to comparison of various operations. . For example, the present invention can be applied in the case of comparing operations such as a baseball bat swing, a tennis racket swing, or a dance choreography.

In the above-described embodiment, the image processing apparatus 1 to which the present invention is applied has been described using a digital camera as an example, but is not particularly limited thereto.
For example, the present invention can be applied to general electronic devices having an image processing function. Specifically, for example, the present invention can be applied to a notebook personal computer, a printer, a television receiver, a video camera, a portable navigation device, a mobile phone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or can be executed by software.
In other words, the functional configuration of FIG. 2 is merely an example and is not particularly limited. That is, it is sufficient that the image processing apparatus 1 has a function capable of executing the above-described series of processing as a whole, and what functional blocks are used to realize this function is not particularly limited to the example of FIG. .
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in this. The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 12 in FIG. 1 in which a program is recorded, the hard disk included in the storage unit 19 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the order, but is not necessarily performed in chronological order, either in parallel or individually. The process to be executed is also included.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and the equivalents thereof.

The invention described in the scope of claims at the beginning of the filing of the present application will be appended.
[Appendix 1]
Image acquisition means for acquiring a first image group in which the motion of the subject is imaged and a second image group in which the motion of the subject to be compared with the motion of the subject is imaged;
Axis setting means for setting axes serving as the reference for the movement of the subject in the first image group and the second image group acquired by the image acquisition means,
An operation comparison means for comparing the movement of the subject in the first image group and the movement of the subject in the second image group based on the axes set by the axis setting means;
Display control means for performing control for displaying a difference between the motion of the subject in the first image group and the motion of the subject in the second image group based on the comparison result of the motion comparison means;
An image processing apparatus comprising:
[Appendix 2]
The axis setting means sets the axis to at least one portion of the subject in the first image group and the subject in the second image group,
The operation comparison unit compares the difference in operation between the subject in the first image group and the subject in the second image group based on the axis set by the axis setting unit. An image processing apparatus according to 1.
[Appendix 3]
Contour line extracting means for extracting a contour line of the subject;
The image processing apparatus according to appendix 1 or 2, wherein the axis setting unit sets the axis based on an end point and a bending point of the contour line extracted by the contour line extracting unit.
[Appendix 4]
The display control means includes
Supplementary note 1 wherein when the subject in the first image group and the subject in the second image group are different in movement, at least one of the set axes is identified and displayed. 4. The image processing device according to any one of items 1 to 3.
[Appendix 5]
The display control means includes
When the subject in the first image group and the subject in the second image group are different from each other, an index indicating the direction and size of the difference in motion is provided on at least one of the set axes. The image processing apparatus according to any one of appendices 1 to 3, wherein the image processing apparatus displays the image.
[Appendix 6]
The display control means includes
When the speed of the movement of the subject in the first image group is different from the speed of the movement of the subject in the second image group, the movement of the subject in the first image group and the second image group Any one of appendices 1 to 5, wherein the first image group and the second image group are displayed by correcting the difference in speed by thinning out the image group having the slower speed. The image processing apparatus described in one.
[Appendix 7]
The display control means includes
When the speed of the movement of the subject in the first image group is different from the speed of the movement of the subject in the second image group, the movement of the subject in the first image group and the second image group Any one of appendices 1 to 5, wherein the first image group and the second image group are displayed by correcting the difference in speed by interpolating an image having a higher speed. The image processing apparatus described in one.
[Appendix 8]
The image processing apparatus according to any one of appendices 1 to 7, wherein the display control unit displays the axis so as to overlap the image of the subject.
[Appendix 9]
An image processing method in an image treatment apparatus,
An image acquisition step of acquiring a first image group in which the motion of the subject is captured and a second image group in which the motion of the subject to be compared with the motion of the subject is captured;
An axis setting step for setting respective axes serving as a reference for the movement of the subject in the first image group and the second image group acquired by the image acquisition step;
An operation comparison step of comparing the movement of the subject in the first image group and the movement of the subject in the second image group based on the axes set by the axis setting step, respectively.
A display control step for performing control for displaying a difference between the motion of the subject in the first image group and the motion of the subject in the second image group based on the comparison result of the motion comparison step;
An image processing method comprising:
[Appendix 10]
Computer
Image acquisition means for acquiring a first image group in which the motion of the subject is imaged and a second image group in which the motion of the subject to be compared with the motion of the subject is imaged;
Axis setting means for setting axes serving as the reference for the movement of the subject in the first image group and the second image group acquired by the image acquisition means,
Operation comparison means for comparing the motion of the subject in the first image group and the motion of the subject in the second image group based on the axes set by the axis setting means, respectively.
Display control means for performing control for displaying a difference between the motion of the subject in the first image group and the motion of the subject in the second image group based on the comparison result of the motion comparison means;
A program characterized by functioning as

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Bus, 15 ... Input / output interface, 16 ... Imaging part, 17 ... Input unit, 18 ... output unit, 19 ... storage unit, 20 ... communication unit, 21 ... drive, 31 ... removable media, 41 ... imaging control unit, 42 ... Image acquisition unit 43... Motion detection unit 44... Comparison result display control unit 51... Acquisition image storage unit 52.

Claims (13)

A first image group in which a series of motions of the first subject is imaged and a second image group in which a series of motions of the second subject corresponding to the series of motions of the first subject are captured are acquired. Image acquisition means;
Setting means for setting a reference line serving as a reference for operation on subjects in the first image group and the second image group acquired by the image acquisition means;
The operation speed in the series of operations of the first subject in the first image group acquired by the image acquisition means is compared with the operation speed in the series of operations of the second subject in the second image group. An operation comparison means;
Reproduction control means for controlling the reproduction speed of the first image group or the second image group based on the comparison result of the operation comparison means;
Equipped with a,
The operation comparison unit includes a first reference line in a series of operations of the first subject in the first image group set by the setting unit, and the second image group set by the setting unit. Is compared with the second reference line in the series of operations of the second subject in the above, the operation speed in the series of operations of the first subject and the operation speed in the series of operations of the second subject. An image processing apparatus characterized by obtaining a comparison result . The reproduction control unit is configured to reproduce the first image so that a difference in operation speed between the first subject and the second subject is reduced based on the comparison result of the operation comparison unit. The image processing apparatus according to claim 1, wherein a playback speed of the group or the second image group is controlled. The reproduction control unit is configured to reproduce the first image group or the first image group so that the operation speeds of the first subject and the second subject are substantially matched based on the comparison result of the operation comparison unit. The image processing apparatus according to claim 1, wherein the reproduction speed of the second image group is controlled. The motion comparison means compares the motion speeds of the first subject and the second subject based on a change in posture in a series of motions of the first subject and the second subject,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Contour line extracting means for extracting the contour line of the subject is further provided. The reference line setting means sets the reference line based on the end point and the bending point of the contour line extracted by the contour line extracting means. The image processing apparatus according to claim 1 , wherein the image processing apparatus is characterized. The reproduction control means includes
When the movement of the first subject in the first image group is different from the movement of the second subject in the second image group, the set reference line is identified and displayed. The image processing apparatus according to claim 1 . The reproduction control means includes
When the movement of the first subject in the first image group is different from the movement of the second subject in the second image group, an index indicating the direction and magnitude of the movement is displayed on the set reference line. The image processing apparatus according to claim 1 , wherein the image processing apparatus displays the image. The reproduction control means includes
When the movement of the first subject in the first image group is different from the movement of the second subject in the second image group, the color of the set reference line is changed and displayed. The image processing apparatus according to any one of claims 1 to 7 . The reproduction control means includes
When the operating speed of the subject in the first image group is different from the operating speed of the subject in the second image group, the operating speed of the subject out of the first image group and the second image group is by thinning out the image of the slower image group, any one of claims 1 to 8, characterized in that to correct the difference of the operating speed of displaying the first image group and the second image group The image processing apparatus according to item 1. The reproduction control means includes
When the operating speed of the subject in the first image group is different from the operating speed of the subject in the second image group, the operating speed of the subject out of the first image group and the second image group is by interpolating the earlier picture number, any of claims 1 to 9, characterized in that to correct the difference of the operating speed of displaying the first image group and the second image group 1 The image processing apparatus according to item. The image processing apparatus according to claim 3 , wherein the reproduction control unit displays the reference line so as to overlap the image of the subject. An image processing method in an image treatment apparatus,
A first image group in which a series of motions of the first subject is imaged and a second image group in which a series of motions of the second subject corresponding to the series of motions of the first subject are captured are acquired. An image acquisition step;
In the first image group acquired by the setting step of setting a reference line as a reference of the operation for the subject in the first image group and the second image group acquired by the image acquisition means, and in the first image group acquired by the image acquisition step An operation comparison step for comparing an operation speed in a series of operations of the first subject and an operation speed in a series of operations of the second subject in the second image group;
A reproduction control step for controlling the reproduction speed of the first image group or the second image group based on the comparison result of the operation comparison step,
The operation comparison step includes a first reference line in a series of operations of the first subject in the first image group set in the setting step, and the second image group set in the setting step. Is compared with the second reference line in the series of operations of the second subject in the above, the operation speed in the series of operations of the first subject and the operation speed in the series of operations of the second subject. An image processing method characterized by obtaining a comparison result . Computer
A first image group in which a series of motions of the first subject is imaged and a second image group in which a series of motions of the second subject corresponding to the series of motions of the first subject are captured are acquired. Image acquisition means,
Setting means for setting a reference line serving as a reference for operation on subjects in the first image group and the second image group acquired by the image acquisition means;
The operation speed in the series of operations of the first subject in the first image group acquired by the image acquisition means is compared with the operation speed in the series of operations of the second subject in the second image group. Operation comparison means,
Based on the comparison result of the operation comparison means, it functions as a reproduction control means for controlling the reproduction speed of the first image group or the second image group,
The operation comparison unit includes a first reference line in a series of operations of the first subject in the first image group set by the setting unit, and the second image group set by the setting unit. Is compared with the second reference line in the series of operations of the second subject in the above, the operation speed in the series of operations of the first subject and the operation speed in the series of operations of the second subject. program characterized Rukoto to function so as to obtain a comparison result.