JP4762050B2 - Display processing apparatus, display processing method, and computer program - Google Patents

Description

  The present invention relates to a display processing apparatus, a display processing method, and a computer program that automatically perform direction processing that represents the next motion of a moving object when performing moving image display processing of the moving object having motion data. About.

  Conventionally, there are various video contents (videos). For example, in order to provide guidance on movements in dance and sports, etc. There is a video. Some of these instructional videos have an arrow inserted in the image to indicate the next movement in advance (for example, FIGS. 7 to 13 of Patent Document 1).

  In addition, the above-described moving image content is created using a so-called motion capture technique (see Patent Document 2). Motion capture is the coordinate value of the place where the marker is attached in 3D coordinates by having the person perform dance and sports movements with multiple markers to acquire data at the required place of the human body And motion data (motion information) representing an angle. By using the motion data acquired in this way, it is possible to create a moving image content in which an image representing a human body operates, and the moving image content based on the motion data can easily change the viewpoint (change the display direction). it can.

Further, Japanese Patent Application Laid-Open No. 2003-228561 discloses that a video obtained by synthesizing a live-action video and a computer graphic character image based on three-dimensional information is disclosed.
Japanese Patent Laid-Open No. 10-319957 Japanese Patent Laid-Open No. 10-222668 JP 2000-23037 A

  As in the instructional video, inserting an arrow indicating movement is very laborious because the video editor needs to manually add an arrow image while watching the movement of each part of the subject. There is a problem. In addition, for moving image content created using motion data based on motion capture technology, manual editing is required in the same manner as described above when an arrow indicating motion is inserted. In particular, when displaying a moving image from a plurality of viewpoints, an arrow image corresponding to each viewpoint is required, which causes a problem that the amount of editing work is enormous.

  Since Patent Document 3 merely synthesizes a live-action image and a character image based on three-dimensional information, until an arrow image indicating movement is synthesized in accordance with the movement of a model that is a subject. I can't.

  The present invention has been made in view of such circumstances, and by using the motion data (motion information) obtained by the motion capture technology, the direction of movement next to the moving object without performing manual editing. An object of the present invention is to provide a display processing device, a display processing method, and a computer program that can automatically add direction information indicating the above.

In order to solve the above problems, a display processing device according to the present invention is a display processing device that performs display processing of a moving image including motion information of a plurality of points related to a moving object for each frame of the moving image indicating the moving object. Among the stored movement information, a calculation means for calculating a difference between the movement information at a plurality of points corresponding to frames that are temporally forward and backward, and a difference between the calculated movement information at the plurality of points. in based on the largest difference, and generating means for generating direction information representing the direction of movement of the moving body, and adding means for adding a frame the generated direction information, relative to a point corresponding to the largest difference, direction information And a means for performing a display process of the frame to which is added.

The display processing apparatus according to the present invention further includes means for detecting whether or not the difference between the motion information at the plurality of points calculated by the calculation means is below a reference value, and the difference between the motion information at the plurality of points. When it is detected that the value falls below a reference value, the generation means does not generate direction information.

Furthermore, the display processing apparatus according to the present invention is characterized in that the generation means generates an image representing a moving direction of the moving object as direction information.
Furthermore, the display processing apparatus according to the present invention further includes a comparison unit that compares the largest difference calculated by the calculation unit with a comparison value, and the generation unit generates based on a comparison result of the comparison unit. It is characterized in that the form of the image to be changed is changed.
The display processing apparatus according to the present invention is characterized in that the generation means generates a character representing a direction in which an action moves as direction information.

The display processing apparatus according to the present invention is characterized in that the adding means adds direction information to a frame at a fixed time interval.
The display processing apparatus according to the present invention is to pair the frames included frame to a previous time slot according to the calculation of the difference corresponding to the direction information, the difference of the operation information of the plurality of points, below the reference value comprising a specifying means for specifying either said adding means is characterized by adding directional information to the frame in accordance with differences less than the reference value the specifying unit has identified.
Furthermore, the display processing apparatus according to the present invention includes a playback means for performing a music playback process in accordance with the video display process, wherein the music is attached to the video, and the additional means performs the playback process with the playback means Direction information is added to a frame to be displayed according to the time signature of the music to be performed.

The display processing apparatus according to the present invention includes a voice table in which a difference between motion information and a voice is associated, a voice identification unit that identifies a voice corresponding to the largest difference calculated by the calculation unit from the voice table, And an audio output means for performing an output process of the specified audio.

In addition, the display processing apparatus according to the present invention includes a receiving unit that receives a search instruction for a suspended state of a moving object, and when the receiving unit receives the search instruction, the display processing apparatus moves back and forth in time in the stored operation information. means for calculating a difference between the operation information of the plurality of points corresponding to the frame, the difference of the operation information of the calculated plurality of points, and a frame specifying means for specifying a frame that falls below the reference value, the frame And a means for performing display processing in a state where the adding means adds direction information to the frame specified by the specifying means.

The display processing method according to the present invention is the display processing method in which the display processing device that stores motion information of a plurality of points related to a moving object included in each frame of the moving image indicating the moving object performs the display processing of the moving image. The apparatus calculates a difference between the motion information of a plurality of points corresponding to frames that are temporally forward and backward in the stored motion information, and based on the largest difference among the calculated motion information of the points. Generating direction information representing the moving direction of the moving object, adding the generated direction information to the frame with respect to the point corresponding to the largest difference, and performing display processing of the frame to which the direction information is added. And
The display processing method according to the present invention is characterized in that the display processing device calculates a difference in advance, stores the calculated difference, and generates direction information based on the stored difference.

A computer program according to the present invention is a computer program for causing a computer storing motion information of a plurality of points related to a moving object included in each frame of a moving image indicating a moving object to perform a moving image display process. Among the calculated motion information, the calculation means for calculating the difference between the motion information of a plurality of points corresponding to the temporally preceding and following frames, and the difference among the motion information of the plurality of points calculated by the calculation means Generating means for generating direction information representing the moving direction of the moving object based on the large difference, and adding means for adding the direction information generated by the generating means to the frame with respect to the point corresponding to the largest difference ; It is made to function as a means to perform the display process of the frame which added direction information.

  In the present invention, the direction information is generated based on the difference of motion information (motion data) obtained by the motion capture technology and added to the frame to be displayed, so the direction information indicating the next motion is automatically Can be displayed. As a result, it is possible to produce a teaching video or the like that can grasp the next moving direction of a moving object at a lower cost and more easily than before without performing manual editing. Also, since the present invention uses motion information obtained by the motion capture technology, the present invention can be applied to all types of motion images as long as the motion image can be obtained by the motion capture technology. In addition to the human body, the moving body also includes a moving ball and objects that can be moved and combined. The motion information obtained by the motion capture technique generally includes at least one of an angle (rotation angle) with respect to each coordinate axis and a coordinate value with respect to each coordinate axis in the three-dimensional coordinate system.

  In addition, the present invention calculates the difference between the motion information in advance in addition to generating the motion information difference and the direction information based on the difference in real time when performing the moving image display process. The direction information may be generated based on the difference stored in the display processing device and stored when the moving image display processing is performed. In the latter case, the processing load on the display processing device is reduced in the moving image display processing, which is preferable when the processor capacity of the display processing device is low or when the content of the moving image display processing by network distribution is distributed. . Furthermore, when the viewpoint direction of the moving image to be displayed is determined, the difference is calculated in advance, and direction information corresponding to the viewpoint direction is generated based on the calculated difference, and the display processing device displays the generated direction information. It is also possible to store and store the moving direction display process by adding the direction information stored when performing the operation display process to the frame. In this case, the processing load can be further reduced.

  In the present invention, since the direction information is not generated when the calculated difference is below the reference value, the direction information is not displayed when the motion of the moving object is at a level below the reference value. For this reason, the subtle movement of the moving object is discarded and is not an object to which the direction information is added, so that it is possible to prevent the direction information from being displayed excessively.

  In the present invention, an image representing the moving direction is generated, so that the user can grasp at a glance the moving direction of the moving object next from the image added to the display screen by the moving image display process. Note that an image of an arrow indicating a direction is preferable as the image, but a tapered image such as a wedge shape capable of expressing the direction is also applicable.

  In the present invention, since the form of the image to be generated is changed based on the comparison result, by viewing the form of the displayed image, the user can predict the amount of movement of the moving object and is currently displayed. It becomes possible to grasp the next movement of the moving object in detail. Note that examples of changing the form include increasing the image as the moving amount of the moving object increases, making the image longer, and increasing the color of the image. In particular, the image corresponding to the direction information is an arrow image. In this case, it is also possible to apply a larger heel portion as the moving amount of the moving body increases.

  In the present invention, since the character representing the moving direction is generated, the character representing the next moving direction of the moving object is shown on the display screen, and the user can surely grasp the next movement. Specifically, in the case of video content for dance instruction, when the model is a scene where the right arm is raised in the next frame, the text “Raise the right arm!” Appears on the display screen. become. Of course, it is possible to display both the character and the image so that the next moving direction can be easily understood.

  In the present invention, since the direction information is added to the frames at regular intervals, the direction information is displayed at the same interval, and the addition of the direction information can be performed by a routine process. A smooth process flow can be created for the addition of direction information.

  In the present invention, the direction information is added to the frame related to the difference below the reference value, so that the direction information is obtained when the motion of the moving object is small or when the moving object is temporarily stopped (in the pause state). Is displayed, the direction information is presented to the user in a timely manner before the next operation starts, and the user watching the moving image can easily grasp the next movement of the moving object.

  In the present invention, since the direction information is added to the frame according to the time signature of the music accompanying the moving image, the direction information is presented to the user in accordance with the rhythm of the music. Therefore, when the model moves according to the music, such as the video content for teaching dance, the direction information indicating the next movement is also displayed along with the music, so the user can adjust the music to the rhythm of the music. You can grasp the next movement and learn the appropriate movement according to the music smoothly.

  In the present invention, since the voice corresponding to the difference is specified from the voice table and the output process is performed, the user can grasp the next movement of the moving object not only visually but also by auditory sense. Can be predicted more reliably. As a result, the instruction content in the moving image content for instructing operations such as dance and sports can be appealed to the user more strongly and the instruction ability can be enhanced.

  In the present invention, when an instruction to search for a paused state (pause state) of a moving object is received, a frame related to a difference below a reference value is specified, and direction information is added to the frame to perform image display processing. As a user, the user can quickly display the pose state in a series of actions, and can learn the next action from the pose state as a starting point, providing an effective function when you want to learn movement for each pose. it can.

In the present invention, using the motion information obtained by the motion capture technology, the difference between the motion information is obtained, the direction information is generated based on the difference, and added to the frame that is the target of the video display process. Therefore, the direction information indicating the next operation can be automatically displayed, and the image editing work indicating the operation direction which has been necessary in the past can be eliminated.
Further, in the present invention, it is also possible to obtain and store a difference in motion information in advance and generate direction information based on the stored difference and add it to the frame during the moving image display process. In this case, the processing load on the apparatus can be reduced.

  In the present invention, when the calculated difference is less than the reference value, direction information is not generated, so direction information does not appear for subtle movements of moving objects, and direction information is prevented from being displayed excessively. it can.

In the present invention, since an image showing the moving direction is generated, the user can grasp at a glance the moving direction of the moving object.
Furthermore, in the present invention, since the generated image form is changed based on the comparison result, the user can predict the amount of movement of the next moving object according to the form of the displayed image.
Furthermore, in the present invention, since the character indicating the moving direction is generated, the user can surely grasp the next movement by reading the character on the display screen.

  In the present invention, since direction information is added to frames at regular intervals, the direction information can be displayed to the user at the same timing, and the user side can grasp the movement periodically, Direction information addition processing can be performed efficiently and routinely.

  In the present invention, since direction information is added to a frame related to a difference that is less than the reference value, when the motion of the moving object is small or when the moving object is temporarily stopped (in a pause state) Direction information can be displayed, and the direction information can be presented to the user in a timely manner before the next operation starts.

  Furthermore, in the present invention, since the direction information is added to the frame according to the time signature of the music accompanying the moving image, the direction information can be presented to the user in accordance with the rhythm of the music, and the user can select the natural information suitable for the music. The direction information can be confirmed with a simple rhythm.

  In the present invention, since the voice corresponding to the difference is specified from the voice table and the output process is performed, the next movement of the moving object is surely transmitted to the user by both visual and auditory senses such as direction information and voice. Can do.

  In the present invention, when an instruction to search for a paused state (pause state) of a moving object is received, a frame related to a difference below a reference value is specified, and direction information is added to the frame to perform image display processing. Therefore, the pose state can be quickly seen in a series of operations, and when learning dance and sports movements, the next operation can be efficiently learned from the pose as a starting point.

  FIG. 1 shows a display processing apparatus 1 according to an embodiment of the present invention. The display processing apparatus 1 according to the present embodiment employs a general-purpose personal computer. The display apparatus 10 displays various images, images, and the like on the computer main body 1a, and the keyboard 11 and the mouse 12 receive operation instructions from the user. Is connected. The display processing apparatus 1 stores in advance moving image data including motion data (corresponding to operation information) obtained by the motion capture technology for each frame, and the time is varied by the motion data when reproducing the moving image. It is characterized by grasping the difference in motion of a moving object and generating an arrow image (corresponding to direction information) indicating the motion direction of the next moving object so that it can be automatically displayed in the video. Hereinafter, the display processing apparatus 1 will be described in detail.

  The display processing device 1 includes a RAM 3, a ROM 4, a display interface 5, an operation unit interface 6, an audio output processing unit 7, and a hard disk device 9 in a control unit 2 (processor) that performs various control processes inside the computer main body 1 a. Are connected by an internal bus 1b. The RAM 3 temporarily stores data, folders, and the like according to the processing of the control unit 2, and the ROM 4 stores in advance a program that defines basic processing contents performed by the control unit 2.

  The display interface 5 is connected to a display device 10 that is an external device in the display processing device 1 of the present embodiment, and sequentially displays video that is reproduced by performing display processing of moving image data to the display device 10. 10 is displayed on the screen 10a. The operation unit interface 6 is connected to a keyboard 11 and a mouse 12, and receives instructions from the user for the menu screens 18, 19 and the like shown in FIGS. Process to tell to. Further, the audio output processing unit 7 functions as an audio output unit and a reproduction unit, and connects the speaker 8 and performs processing such as reproduction and amplification of music data and audio data based on a control instruction of the control unit 2. Then, the process of outputting the reproduced sound from the speaker 8 is performed.

  The hard disk device 9 stores various programs, data, and the like. In the present embodiment, as a program, a system program 14 that defines a base process for operating the computer main body 1a, and a display that defines a display process according to the present invention. Processing program (corresponding to the computer program of the present invention) 15, moving image data D to be displayed based on motion capture technology, audio table T storing audio data for instructing the next operation by voice, various processing judgments Threshold data 16 including reference and discrimination values, menu image data 17 and the like are stored.

  The moving image data D of the present embodiment stored in the hard disk device 9 is moving image content for dance instruction, and motion data representing the coordinates of the marker in the three-dimensional space is attached to the dancer by using a motion capture technique. The contents are included in units of frames constituting a moving image (video). Such moving image data D can be displayed and reproduced by designating a viewpoint at the time of reproduction, and viewing a motion of a human body image (corresponding to a moving object) according to a dancer from a desired direction.

  As shown in FIG. 2, a specific image of the moving body data D includes a human body image 20 representing a dancer created by three-dimensional computer graphics technology based on motion data, which is composed of an X axis, a Y axis, and a Z axis. A UVW coordinate system is provided for determining the imaging direction of the camera 22 (corresponding to the viewpoint direction at the time of reproduction) and the position, which are located in the XYZ coordinate system and are different from the XYZ coordinate system. By defining the relative relationship, it is possible to display an image from the direction and position desired by the user.

  In addition, as shown in FIG. 3B, the human body image 20 (see FIG. 3A) created by the three-dimensional computer graphics technique includes a rod-shaped link member called a bone B corresponding to the bone of the human body. It is created by covering the connected body with a skin corresponding to the skin of the human body. Further, points P1 to P17 attached to the various points of the bone B shown in FIG. 3B correspond to marker positions attached to the original dancer, and motion data exists for each of these points P1 to P17. Yes. Note that the positions and the numbers of the points P1 to P17 shown in FIG. 3B are examples, and can be appropriately changed according to the positions and the numbers of markers attached to the dancers.

  FIG. 4 is a diagram showing an image of the human body image 20 created in frames for each time constituting the moving image data D. The moving image data D is composed of frames f1, f2, f3... At times t1, t2, t3..., And by sequentially reproducing images corresponding to the frames f1, f2, f3, etc. A video (moving image) in which the human body image 20 representing the dancer included in the frames f1, f2, f3, etc. moves can be reproduced and displayed. Note that the moving image data D of the present embodiment has 60 frames per second (60 frames / second), but this value is merely an example, and the range of the number of frames when acquiring motion data If so, it can be appropriately increased or decreased according to the required moving image quality.

  FIG. 5 schematically shows the contents of the motion data M1 and M2 corresponding to the human body image 20 included in the first frame f1 and the second frame f2 at times t1 and t2. The motion data M1 and M2 have rotation angles and coordinate values for the X, Y, and Z axes in the XYZ coordinate system shown in FIG. 2 for each point P1 to P17 shown in FIG. Based on the motion data for each frame, a human body image 20 (when the viewpoint is a viewpoint from the front of the human body image 20) included in each frame shown in FIG. 4 is created. In addition, as for the rotation angle, the arrow directions attached to the X, Y, and Z axes in FIG. 2 are the plus (+) and minus (−) directions, respectively.

  FIG. 6 shows the contents of the voice table T stored in the hard disk device 9. The audio table T associates a difference between motion data obtained by processing defined by the display processing program 15 to be described later and audio data. Specifically, each point P1 between frames that are temporally changed. Dedicated audio data is associated with the difference status of each element (rotation angle, coordinates) included in the motion data for .about.P17.

For example, when the difference in the rotation angle of the X axis with respect to the point P1 that is the head apex of the human body image 20 is positive, the audio data A with the content “shake the head forward” is associated, and the point P1 When the difference in the rotation angle of the X axis with respect to is negative, the voice data B having the content “shake your head back” is associated. Further, when the difference in the rotation angle on the Y axis with respect to the point P6 that is the tip of the left arm of the human body image 20 is positive, the audio data C with the content “raise the left arm” is associated with the Y axis on the point P6. When the difference in rotation angle is negative, the voice data D having the content “lower the right arm” is associated. As described above, the voice table T associates the voice data having the above-described contents when the difference between the rotation angles of the respective axes of the points P1 to P17 is plus and minus.

  Note that the rotation angle of each axis is 0 degrees when the Y axis is viewed from the YZ plane parallel to the X axis and in the positive to negative direction of the X axis in FIG. The counterclockwise direction is a positive rotation angle, and the clockwise rotation direction is a negative rotation angle. In the Y-axis, when viewing the XZ plane parallel to the Y-axis and in the positive to negative direction of the Y-axis, the positive side of the X-axis is 0 degrees, the counterclockwise rotation direction is the positive direction, and the clockwise rotation The direction is negative. Further, with respect to the Z axis, when viewing the XY plane parallel to the Z axis and in the positive to negative direction of the Z axis, the negative side of the X axis is 0 degrees, the counterclockwise rotation direction is the positive direction, and the clockwise rotation The direction is negative.

  The audio table T also associates audio data according to the difference state of the coordinate values in the motion data. For example, when the coordinate difference with respect to the X axis with respect to the point P6 that is the tip of the left arm of the human body is positive, The audio data E with the content “stretch left arm to the side” is associated.

  On the other hand, the menu screens 18 and 19 shown in FIGS. 7A and 7B are examples of contents corresponding to the menu image data 17 stored in the hard disk device 9. The menu screen 18 in FIG. 7A corresponds to the menu contents that are displayed first when the display processing program 15 is activated, and is referred to as a first button 18 a called “viewpoint direction setting” and “video playback”. A second button 18b, a third button 18c called "pause search" (corresponding to a receiving means for receiving a search instruction for a pause state), and a fourth button 18d called "end" are provided. When the first button 18a is selected by operating the keyboard 11 or the mouse 12, the display processing program 15 defines that the menu screen 19 for viewpoint selection shown in FIG. 7B is displayed. Yes.

  The menu screen 19 in FIG. 7B includes arrow-shaped direction buttons 19a to 19e for selecting a viewpoint direction, and a determination button 19f. When one of the direction buttons 19a to 19e for specifying any one of the five viewpoints in the front-rear, left-right, and upper directions is specified and selected with respect to the human body, the determination button 19f is selected. The display processing program 15 defines that the viewpoint is determined by the direction and the display returns to the menu screen 18 of FIG. Note that the method of specifying the viewpoint direction based on the menu screen 19 in FIG. 7B is an example, and other specification methods can of course be applied. For example, any one of 360 degrees can be applied by operating the mouse 12. You may make it designate the angle of.

  When the second button 18b in FIG. 7A is selected, the display processing program 15 starts the reproduction process of the moving image data D. If a plurality of keys (for example, the ALT key and the F4 key) on the keyboard 11 are operated simultaneously after the start of the playback process, the display processing device 1 has received a playback stop instruction. Furthermore, when the third button 18c is selected, a search process for a paused state, which will be described later, is started. Furthermore, when the fourth button 18d is selected, all processing of the display processing program 15 is completed.

  Next, processing contents defined by the display processing program 15 will be described. The display processing program 15 defines the content of control processing performed by the control unit 2. When the display processing program 15 is activated, the display processing program 15 first reads the menu image data 17 on the menu screen 18 in FIG. 7A and outputs it from the display interface 5. Thus, it is defined that the control unit 2 performs processing for displaying the menu screen 18 on the screen 10 a of the display device 10. When the menu screen 18 is displayed and the operation unit interface 6 informs the control unit 2 that the first button 18a for “viewpoint direction setting” has been selected, the menu screen 19 in FIG. 7B is displayed. Thus, it is defined that the control unit 2 determines the viewpoint direction related to the video reproduction in accordance with the operation of selecting and selecting the viewpoint direction.

  Furthermore, the display processing program 15 controls the difference between the motion data included in each of the frames to be displayed and the subsequent frames when performing the moving image display processing of the moving image data D based on the selection of the second button 18b. It is defined that the unit 2 calculates as the calculation means. For example, when the display process of the first frame f1 at time t1 shown in FIG. 4 is performed, each of the points P1 to P17 included in the motion data included in each of the first frame 1 and the second frame f2 at time t2 is performed. The control unit 2 calculates a difference between elements (rotation angle, coordinates).

  Furthermore, the display processing program 15 controls whether or not the calculated difference result falls below the reference value for rotation angle and the reference value for coordinates included in the threshold value data 16 stored in the hard disk device 9. It defines that the part 2 detects as a detection means. The reference value for the rotation angle in this embodiment is a value “25 degrees”, and the reference value for coordinates is a value “20 scales” on the coordinate axis. The control unit 2 detects whether or not the result for each point with respect to the rotation angle is less than “25 degrees” among the difference results calculated for the points P1 to P17, and for each point with respect to the coordinates. It is detected whether or not the result falls below “20 scales”. Each reference value (25 degrees, 20 scales) described above is merely an example, and other values can of course be applied.

  In the above-described detection processing, when the results for the rotation angles and coordinates at all points are below the reference value, the movement of the human body image 20 between frames that are temporally back and forth is a body shake that does not correspond to the motion, Alternatively, since the human body image 20 can be determined to be in a pause state (pause state) in a series of movements, the display processing program 15 defines that an arrow image representing the movement direction is not generated.

  Further, in the detection process described above, if the result for the rotation angle and coordinates does not fall below the reference value at any one of the points, the display processing program 15 defines that an arrow image is generated. The generation of the arrow image is performed mainly in six parts of the human body image 20 (head, left arm, right arm, body part, left leg, right leg). Specifically, in FIG. Points P1 and P2 corresponding to the head, points P5 and P6 corresponding to the left arm, points P8 and P9 corresponding to the right arm, points P3, P4, P7, P10, P11 corresponding to the body part, points corresponding to the left leg An arrow image with respect to each point of P12 to P14 and points P15 to P17 corresponding to the right leg having the largest amount of motion (the point having the largest absolute difference value of rotation angle and the absolute value of coordinate difference). Is generated.

  FIG. 8 shows the movement of the right arm between frames that move forward and backward in time, and the solid bone B related to the points P8 and P9 shows the state of the right arm in the temporally front frame, and points P8 ′ and 9 The wavy line related to 'shows the state of the right arm in the temporally rear frame. FIG. 8 is a diagram based on the viewpoint direction shown in FIG. 2 (a direction parallel to the Y-axis direction and going from positive to negative in the Y-axis direction). In the case of the movement shown in FIG. 8, the point P9 corresponding to the tip of the right arm has a larger amount of motion than the point 8 corresponding to the elbow portion of the right arm, so that the point P9 is an object for generating an arrow image.

  In the case of FIG. 8, the coordinate values for the point P9 in the temporally previous frame are (X, Y, Z) = (20, 60, 0), and the rotation angle for each coordinate is (X, Y, Z) = ( 100 degrees, 180 degrees, 30 degrees), and the coordinate values of the point P9 'in the back-front frame in time are (X, Y, Z) = (50, 60, 140), and the rotation angle with respect to each coordinate is If (X, Y, Z) = (100 degrees, 140 degrees, 30 degrees), the difference results for the coordinate values are (30, 0, 140), and the difference results for the rotation angle are (0 degrees, −40 degrees). , 0 degrees).

  Since the difference result of the coordinate values obtained above becomes vector data from the point P9 in the temporally preceding frame to the point P9 'in the temporally backward frame, the vector data and the rotation angle The display processing program 15 defines that the control unit 2 generates an arrow image 25 corresponding to the difference result. The generated arrow image 25 has the same size, the rear end 25a of the arrow image is matched with the target point P9 in the difference result, and the tip 25b of the arrow image 25 is based on the difference result with respect to the rotation angle. The direction of rotation is set to the direction represented by the vector data (difference result).

  Further, as shown in FIGS. 9A and 9B, the generated arrow image 25 is a difference calculation target so that it is positioned between the human body image 20 and the camera 22 that determines the viewpoint direction and the viewpoint position. The display processing program 15 stipulates that the control unit 2 performs processing to be added as an adding means to the front frame that has become. In FIG. 9B, when the Y coordinate value of the human body image 20 is y1, the Y coordinate value for the arrangement position of the arrow image 25 is y2, and the Y coordinate value of the camera 22 is y3, y1 <y2 <y3. The relationship is established. Note that the human body image 20 included in the image of the frame is also created by the control unit 2 according to the processing content defined based on the three-dimensional computer graphics technology of the display processing program 15. By arranging the arrow image 25 between the human body image 20 and the camera 22 in this way, the arrow image 25 is hidden (embedded) by the human body image 20 in the display in the viewpoint direction from the camera 22. The arrow image 25 can be shown to the user satisfactorily.

  Further, the display processing program 15 stipulates that display processing be performed on the frame including the human body image 20 and the arrow image 25 created as described above, and the control unit 2 displays the frame on the display interface 5. The reproduction image subjected to the display process is displayed on the screen 10 a of the display device 10.

  The display processing program 15 also defines audio data output processing in addition to processing related to the arrow image 25. Specifically, when the difference calculation of the motion data described above calculates the absolute difference value for any point for each part (head, left arm, right arm, body part, left leg, right leg) of the human body image 20. The display processing program 15 stipulates that the control unit 2 specifies the voice data corresponding to the absolute difference value from the voice table T of FIG. The specified audio data is sent to the audio output processing unit 7 and output from the speaker 8 in synchronization with the process of displaying the frame including the arrow image 25 on the display device 10 under the control of the control unit 2.

  The control unit 2 linearly performs the processing related to the display of the arrow image 25 and the output of the sound as described above in conjunction with the moving image display processing, and the display processing program 15 receives a moving image reproduction stop instruction from the user. In this case, it is specified that the above-described processing is continued until the display processing of the last frame is performed.

  The first flowchart in FIG. 10 is an arrangement of processing procedures (contents of the display processing method) related to display and audio output related to the arrow image 25 defined by the display processing program 15 described above. Hereinafter, according to the first flowchart, a series of processing contents in the display processing device 1 after the display processing program 15 is activated will be described.

  First, the display processing apparatus 1 determines whether or not an instruction for video reproduction from the user has been received from the menu image 18 of FIG. 7A displayed in accordance with the activation of the display processing program 15 (S1). When the instruction is not received (S1: NO), the instruction is waited. When the instruction is received (S1: YES), the difference between the motion data included in each of the first frame f1 and the second frame f2 in FIG. Is calculated (S2). Then, the display processing device 1 determines whether or not the differences for all the calculated points P1 to P17 are less than the reference value of the threshold data 16 (S3).

  When the differences with respect to all the points P1 to P17 are less than the reference value of the threshold data 16 (S3: YES), the display processing device 1 performs the display process of the first frame f1 without generating the arrow image 25 ( S6). In this case (S3: YES), the screen content displayed is the one representing only the human body image 20 as in the prior art (see FIG. 4).

  On the other hand, when the difference with respect to all the points P1 to P17 does not fall below the reference value of the threshold value data 16 (S3: NO), the display processing device 1 determines the target points P1 to P17 based on the calculated difference value. An arrow image 25 is generated for either one (S4). In addition to the generation of the arrow image 25, the display processing device 1 also specifies the audio data corresponding to the difference value from the audio table T. Then, the display processing apparatus 1 adds the generated arrow image 25 to the first frame f1 including the created human body image 20 in the positional relationship shown in FIGS. 9A and 9B (S5). The display process of the first frame f1 is performed, and the human body image 20 and the arrow image 25 are displayed on the display device 10 (S6). At this time, the display processing device 1 performs a reproduction process of the specified audio data, and outputs a sound (for example, a sound saying “raised left arm”) from the speaker 8 in accordance with the display of the first frame f1. To do.

  Then, the display processing device 1 determines whether or not the display processing has been completed up to the final frame, or whether or not a reproduction stop instruction has been received from the user (S7). When the display process has not been completed to the end and no reproduction stop instruction has been received (S7: NO), the next frame (second frame f2 and third frame f3 in FIG. 4) that moves back and forth in time is displayed. The difference between the motion data is calculated (S2), and thereafter, the processing steps from S2 to S7 are repeated until the display process is completed until the last frame or the reproduction stop instruction is received. In addition, when the display process is completed up to the final frame, or when a reproduction stop instruction is received from the user (S7: YES), the display processing apparatus 1 ends the series of processes.

  FIG. 11 shows the contents of each frame image of a moving image sequentially displayed by the processing of the display processing apparatus 1 described above. Each frame image shown in FIG. 11 is appropriately attached with an arrow image 25 indicating the next operation direction of the human body image 20 as compared with the frame image shown in FIG. Since the next movement direction can be grasped in advance, it is easy to learn movements such as dance or sports.

  In addition, in the present embodiment, for example, voices such as “raise your right arm” at time t1, “raise your left arm” at time t2, and “lower your right arm” and “lower left arm” at time t3 are displayed. Accordingly, the user can grasp the contents of the next operation by hearing as well as by vision, and can sufficiently prepare for and prepare for the next operation. In FIG. 11, since the human body image 20 is in a paused state in the fourth and fifth frames f4 and f5 from time t4 to t5, in the first flowchart in FIG. S3: YES). Similarly, since the difference (motion data difference) due to the motion of the human body image 20 between the times t8 and t9 is small and below the reference value, the arrow image 25 is not added even in the eighth frame f8 at the time t8.

  In addition to the video playback process described above, the display processing program 15 stored in the hard disk device 9 shown in FIG. 1 is in a paused state (pause state or small motion) of the human body image 20 included in the moving image data D. It also defines a process for searching (status). The search process in the paused state is started by selecting the third button 18c in FIG. 7A, and the control unit 2 first calculates the difference in motion data between frames that are included in the moving image data D and that are temporally related. The display processing program 15 prescribes to do this. Further, the control unit 2 determines whether or not the calculated difference is less than the above-described reference value included in the threshold data 16.

  When the control unit 2 identifies a difference that is lower than the reference value, the controller 2 generates an arrow image 25 based on the difference in the time zone after the difference, as well as the previous and subsequent frames related to the calculation of the difference. A front frame is specified, an arrow image and a selectable button (see FIG. 13) are added to the frame, and frame display processing is performed in that state. Note that the display process at this time is a display process of a still image, not a moving image.

  The second flowchart of FIG. 12 is an arrangement of the search processing procedure for pause defined by the display processing program 15 described above. The organized contents will be described in sequence. First, the display processing device 1 determines whether or not a pause search instruction from the user has been received from the menu image 18 of FIG. 7A (S10). When the instruction is not accepted (S10: NO), the instruction is waited. When the instruction is accepted (S10: YES), it is included in each of the first first frame f1 and the second frame f2 included in the moving image data D. The difference of motion data to be calculated is calculated (S11), and it is determined whether or not the differences for all the calculated points P1 to P17 are below the reference value (S12).

  If the differences for all the points P1 to P17 are not less than the reference value (S12: NO), the motion data difference between the next and subsequent frames (second frame f2, third frame f3) is calculated (S11). Henceforth, the process of S11-S12 is repeated until the difference with respect to all the calculated points P1-P17 is less than a reference value.

  Moreover, when the difference with respect to all the points P1-P17 is less than a reference value (S12: YES), the display processing apparatus 1 produces | generates the arrow image 25 based on the difference before the difference which falls below the reference value (S13). Then, an arrow image or the like is added to the frame related to the difference below the reference value (S14), and a frame display process is performed as a still image (S15).

  FIG. 13 shows an example of the still image 30 displayed on the screen 10 a of the display device 10 through the pause search process described above, and an arrow image 25 is added to the human body image 20. Further, the still image 30 is provided with a reproduction start button 30a, a search button 30b for receiving a search instruction for the next paused state, and an end button 30c for ending the process. When the reproduction start button 30a is selected, a moving image display process (contents shown in the first flowchart of FIG. 10) is started for moving image data D after the frame corresponding to the still image 30 being displayed. When the search button 30b is selected, a pause search process (contents shown in the second flowchart of FIG. 12) is started for the moving image data D after the frame corresponding to the still image 30 being displayed. .

  By performing the pause search processing as described above, the user can quickly find the pause state that is the starting point for learning the operation from the moving image data D, and can learn the operation efficiently.

  In addition, this invention is not limited to the content mentioned above, A various modification example is applicable. For example, the frame to which the generated arrow image 25 is added may be sequentially added to a frame at a certain time interval (for example, every 5 milliseconds). In this case, the frame to be added to the arrow image 25 can be specified at time intervals, the processing related to specifying the addition target can be simplified, and the arrow image 25 is also displayed at a constant interval. As a user, it becomes easy to grasp the timing when the arrow image 25 is displayed on the screen 10a.

  Moreover, you may add the produced | generated arrow image 25 to the frame when the human body image 20 is a pause state or a motion is small. In this case, the difference of the motion data is calculated for the frames included in the time zone before the preceding frame among the temporally related frames related to the calculation of the difference used to generate the arrow image 25, The control unit 2 determines whether or not the calculated difference is lower than the reference value, similarly to the processing step of S12 in the second flowchart of FIG. When the control unit 2 specifies a difference that is lower than the reference value, an arrow image is added to each frame before and after that used to calculate the specified difference. By performing such processing, for example, an arrow image similar to the arrow image 25 attached to the sixth frame f6 is also applied to the fourth frame f4 and the fifth frame f5 in which the human body image 20 illustrated in FIG. 25 will be added. Therefore, when the human body image 20 is in a paused state or has little movement, the fourth and fifth frames f4 and f5 in the previous frame (in the case of FIG. 11, the sixth frame f6 in the case of FIG. 11) (the sixth frame f6). ), An arrow image 25 indicating the next movement direction appears, so that the user can know the direction of movement in advance, and the movement can be easily learned.

  Further, as shown in FIG. 14A, when the moving image data D to be reproduced is accompanied by the music data 23 reproduced together with the video, the time signature of the music data 23 that is reproduced along with the moving image display processing. The generated arrow image 25 is added to the display processing target frame corresponding to the rhythm R1, R2, R3, R4 in FIG. 14B, and the arrow image 25 is displayed in accordance with the rhythm of the music data 23. May be. In this case, since the arrow image 25 is displayed in accordance with the music rhythm, the user can check the arrow image 25 with a good rhythm, and in particular, the arrow image display suitable for learning the movement of the dance that operates with the music can be performed.

  The form of the arrow image 25 to be generated may be changed according to the difference value of the motion data. In this case, the comparison value (a plurality of comparisons are possible) are included in the threshold data 16, and the control unit 2 compares the difference result with respect to the coordinate value as a scalar quantity and compares the comparison value with the comparison value. The arrow image is generated by changing.

  Specifically, when the value “60” is set as the first comparison value and the value “100” is set as the second comparison value, and the calculated difference result (scalar amount) is less than the first comparison value, the control unit 2 As shown in FIG. 15A, a white arrow image 40 is generated. Thereafter, when the calculated difference result (scalar amount) is not less than the first comparison value and less than the second comparison value, a gray arrow image 41 (shown by hatching in the figure) is generated as shown in FIG. When the calculated difference result (scalar amount) is equal to or greater than the second comparison value, a black arrow image 42 is generated as shown in FIG. By changing the form (color) of the arrow image generated in this way, the user can grasp the next operation amount according to the form of the arrow image, so that it becomes easier to cope with the next operation.

  In addition, as a variation of the form of the arrow image, as shown in FIGS. 15D to 15F, when it is less than the first comparison value, the short arrow image 45 in FIG. 15D, the first comparison value or more, If it is less than the second comparison value, it may be an arrow image 46 having an intermediate length shown in FIG. 15 (e), and if it is more than the second comparison value, it may be a long arrow image 47 shown in FIG. 15 (f). It is also possible to change the size of the arrow image itself and change the color density of the arrow image. It should be noted that when dividing the form of the arrow image in more detail, it is preferable to use a comparison value of 3 or more. Conversely, when simplifying the process, the comparison process is performed using only one comparison value. Also good.

  Further, the number of frames used for arrow image generation (difference calculation) is not limited to two, and an arrow image may be generated based on a difference between three or more frames that change in time. When three or more frames are used in this way, the trajectory of the target point can also be grasped, and the generated arrow image may be a shape corresponding to the trajectory instead of a straight line.

  For example, as shown in FIG. 16, when the point P9 at the tip of the right arm that is the target of the generation of the arrow image moves along an arc-shaped locus in a total of four frames that are temporally continuous, a plurality of points P9 between the frames of the point P9. The control unit 2 generates an arc-shaped arrow image 50 obtained by connecting the vectors of the differences (difference results corresponding to the coordinate values). In this case, since the trajectory of the operation can be grasped by the form of the arrow image 50, the user can learn up to the details of the movement.

  Furthermore, in order to reduce the processing load of the control unit 2 during the moving image display process, the motion data difference between frames that are temporally changed in advance is calculated and associated with the frame related to the calculation in FIG. The difference result may be stored in the hard disk device 9, and the arrow image 25 may be generated based on the difference result stored during the moving image display process and added to the frame including the created human body image 20.

  The third flowchart of FIG. 17 shows a processing procedure (contents of the display processing method of the modified example) defined by the display processing program 15 of the modified example in the case where the difference calculated in advance is stored. According to the third flowchart, first, the display processing apparatus 1 according to the modified example determines whether or not a video reproduction instruction from the user has been received from the menu image 18 of FIG. 7A (S20). Is not received (S20: NO), the instruction is waited. If the instruction is received (S20: YES), an arrow image is generated based on the stored difference (S21), and is generated in the display processing target frame. The arrow image thus added is added (S22), and the frame is displayed (S23).

  Then, the display processing device 1 determines whether the display process has been completed up to the final frame or whether a playback stop instruction has been received from the user (S24). If no instruction has been received (S24: NO), the process returns to the step of generating an arrow image based on the difference associated with the next frame to be displayed (S21). In addition, when the display process is completed up to the final frame, or when a reproduction stop instruction is received from the user (S24: YES), the display processing apparatus 1 ends the process. As described above, the process of the third flowchart in FIG. 17 is suitable for the control unit 2 having a low processing capability because the number of processes during the moving image display process is smaller than that in the first flowchart of FIG. If the viewpoint direction, the viewpoint position, and the like are determined in advance, an arrow image is also generated in advance in the processing step of S21 in the third flowchart, and the generated arrow image is stored in the hard disk device 9 in association with each frame. In addition, at the time of moving image display processing, the processing load can be further reduced by simply adding the stored arrow image to the display processing target frame.

  In addition to the arrow image, the image generated as the direction information indicating the next operation direction may be a tapered image, a triangular image, or the like as the image indicating the direction. Further, as shown in a display image 55 in FIG. 18, a character 56 representing the calculated difference result may be generated and added to a frame including the human body image 20 and displayed. By displaying the character 56 in this way, an instruction can be given to the user more directly. In addition to the generation of the character 56, a character table for storing a character representing an action in association with various kinds of difference contents as shown in the voice table T of FIG. It is also possible to perform processing for specifying a corresponding character from the character table based on the difference contents and adding it to the frame. In addition, when a character is added to a frame, the above-described arrow image may be added to clearly indicate the next operation direction to the user using both the arrow image and the character.

  Furthermore, when the specification of the display processing device 1 is simplified, the processing related to the voice table T illustrated in FIG. 6 may be omitted, and the display processing with only the arrow image may be performed. Further, the reference value that prevents the arrow image 25 from being generated may be freely settable on the user side. In this case, the user who desires to display the arrow image 25 even for subtle movements sets the reference value small (for example, “10 degrees” as the reference value for the rotation angle, In the case of a user who wants to grasp a rough movement, the reference value is set to a large value (for example, “30 degrees” as the reference value for the rotation angle, “30 scales” as a reference value for coordinates). Note that the process of detecting whether or not the motion data difference is below the reference value may be omitted, and the arrow image 25 may be always displayed when there is any movement.

  In addition, when it is desired to reduce the processing burden as much as possible, only one of the rotation angle and each coordinate value for each coordinate is used in the difference calculation process of the motion data associated with the moving object (human body). Thus, the arrow image 25 can be generated.

  Furthermore, in the present invention, the target of the moving body is not limited to the human body, and any moving object associated with motion data can be applied. Therefore, for example, the present invention can be applied to an assembly manual moving video when moving and assembling an object as in the display image 56 shown in FIG. 19A. In this case, the object moves to the first object 31. The assembling direction of the second object 20 ′ (corresponding to a moving object) to be assembled can be represented by an arrow image 25, and the assembling method can be confirmed in detail by freely changing the viewpoint direction. Become. Furthermore, as in the display image 57 shown in FIG. 19B, the moving direction of the ball 20 ″ as a moving body can be represented by the arrow image 25.

  Further, as the display processing apparatus 1 of the present invention, a portable personal computer, a PD, a mobile phone and the like having a display display can be applied in addition to using the computer main body 1a of the personal computer. Further, in addition to completing the processing only within the apparatus, the server apparatus 110 performs the above-described moving image display process as in the network system 100 illustrated in FIG. 20, and the processed moving image is transmitted to the network NW and the relay base stations 100 </ b> A to 100 </ b> A. It is also possible to sequentially display each image shown in FIG. 11 that is distributed to a terminal device such as the cellular phone 120 through 100N and subjected to display processing by the terminal device.

It is a block diagram which shows the main structures of the display processing apparatus which concerns on embodiment of this invention. It is the schematic explaining the human body in a three-dimensional space, the positional relationship of a viewpoint direction, etc. (A) Schematic diagram showing human body image created by three-dimensional computer graphic technology, (b) is a schematic diagram showing points and bones corresponding to markers by motion capture technology. It is the schematic which shows the state of the human body image contained in each flame | frame which comprises moving image data. It is a figure showing the content of the motion data contained in each frame. It is a chart which shows an audio | voice table. (A) is the schematic which shows the menu screen for selecting the processing content, (b) is the schematic which shows the menu screen for the setting of a viewpoint direction. It is the schematic which shows the produced | generated arrow image. (A) and (b) are the figures for demonstrating the arrangement position with respect to the human body and camera of the produced | generated arrow image. It is a 1st flowchart which shows the processing content of a display processing method. It is the schematic which shows a series of flame | frames containing the produced | generated arrow image. It is a 2nd flowchart showing the search processing content of the pause state. It is the schematic which shows the image displayed based on the search. (A) is the schematic of the moving image data accompanying music data, (b) is a time chart which shows the display timing of the arrow image matched with the time signature of the music. (A)-(c) is the schematic which shows the arrow image of the modification which changed the color according to the difference, (d)-(e) is the arrow image of another modification which changed the length according to the difference. FIG. It is an arrow image of an arc-shaped modification. It is a 3rd flow chart which shows a processing procedure at the time of calculating and memorizing a difference beforehand. It is the schematic which shows the display image which added the character showing a motion direction. (A) is the schematic which shows the display image in the case of assembling an object, (b) is the schematic which shows the display image of the moving ball | bowl. It is the schematic which shows the network system to which this invention is applied.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Display processing apparatus 2 Control part 5 Display interface 6 Operation part interface 7 Audio | voice output process part 8 Speaker 9 Hard disk apparatus 10 Display apparatus 11 Keyboard 12 Mouse 15 Display processing program 16 Threshold data 17 Menu image data 20 Human body image 22 Camera 25 Arrow Image D Movie data M1, M2 Motion data T Audio table P1-P17 points

Claims (13)

In a display processing device that performs display processing of a moving image including motion information of a plurality of points related to the moving object for each frame of the moving image indicating the moving object ,
Means for storing the operation information;
Calculating means for calculating the difference between the motion information of a plurality of points corresponding to the frames that move back and forth in time among the stored motion information;
Generating means for generating direction information representing the moving direction of the moving object based on the largest difference among the calculated differences in the movement information of the plurality of points ;
An adding means for adding the generated direction information to the frame with respect to the point corresponding to the largest difference ;
A display processing apparatus comprising: means for performing display processing of a frame to which direction information is added. Means for detecting whether or not the difference between the motion information of the plurality of points calculated by the calculation means is below a reference value;
The display processing apparatus according to claim 1, wherein when the difference between the motion information of the plurality of points is detected to be less than a reference value, the generation unit does not generate direction information.   The display processing apparatus according to claim 1, wherein the generation unit generates an image representing a moving direction of the moving object as direction information. Comparing means for comparing the largest difference calculated by the calculating means with a comparison value;
The display processing apparatus according to claim 3, wherein the generation unit changes a form of an image to be generated based on a comparison result of the comparison unit.   The display processing apparatus according to claim 1, wherein the generation unit generates a character representing a direction in which an action moves as direction information.   The display processing apparatus according to claim 1, wherein the adding unit adds direction information to a frame at regular time intervals. And against the frame included difference frame the previous time slot according to the calculation of in accordance with the direction information comprises a particular means the difference between the operation information of the plurality of points, to identify falls below the reference value,
The display processing apparatus according to claim 1, wherein the adding unit adds direction information to a frame related to a difference that falls below a reference value specified by the specifying unit. The video comes with music,
Provided with playback means for playing back music along with the video display processing,
The display processing apparatus according to claim 1, wherein the adding unit adds direction information to a frame to be displayed according to the time signature of the music to be played back by the playback unit. A voice table in which the difference between the motion information and the voice is associated;
A voice identification unit that identifies a voice corresponding to the largest difference calculated by the calculation unit from the voice table;
The display processing apparatus according to claim 1, further comprising: an audio output unit that performs an output process of the specified audio. Accepting means for accepting a search instruction for a suspended state of a moving object;
Means for calculating a difference between motion information at a plurality of points corresponding to frames that are temporally forward and backward in the stored motion information when the accepting means accepts a search instruction;
Difference of the calculated motion information of the plurality of points, and a frame specifying means for specifying a frame that falls below the reference value,
The display processing apparatus according to any one of claims 1 to 9, further comprising: a display unit configured to perform display processing in a state where the adding unit adds direction information to the frame specified by the frame specifying unit. In a display processing method in which a display processing device that stores motion information of a plurality of points related to a moving object included in each frame of a moving image indicating a moving object performs display processing of the moving image,
The display processing device includes:
In the stored motion information, calculate the difference between the motion information of a plurality of points corresponding to frames that are temporally mixed,
Based on the largest difference among the calculated motion information differences of a plurality of points , generate direction information representing the moving direction of the moving object,
The generated direction information is added to the frame for the point corresponding to the largest difference ,
A display processing method characterized by performing display processing of a frame to which direction information is added. The display processing device includes:
Calculate the difference in advance, store the calculated difference,
The display processing method according to claim 11, wherein the direction information is generated based on the stored difference. In a computer program for causing a computer that stores motion information of a plurality of points related to a moving object included in each frame of a moving image indicating a moving object to perform a moving image display process,
Computer
Calculating means for calculating the difference between the motion information of a plurality of points corresponding to the frames that move back and forth in time among the stored motion information;
Generating means for generating direction information indicating the moving direction of the moving object based on the largest difference among the differences in the motion information of the plurality of points calculated by the calculating means;
Adding means for adding the direction information generated by the generating means to the frame at a point corresponding to the largest difference ;
A computer program that functions as means for performing display processing of a frame to which direction information is added.

Images