JP7165003B2 - Skin animation correction system, skin animation correction program, and skin animation correction method - Google Patents

Description

The present invention relates to a skin moving image correction system, a skin moving image correction program, and a skin moving image correction method for correcting a skin moving image obtained by photographing a person's face.

2. Description of the Related Art Conventionally, correction processing is performed on an image of a person's face to improve its impression. In recent years, in particular, a technique has been known that can perform such correction processing on a moving image.

For example, in the apparatus described in Patent Document 1, by performing correction based on a representative color that is a color that represents the target area to be corrected, even if the representative color changes for each frame that constitutes a moving image, , natural correction results can be obtained.

JP 2016-171445 A

By the way, it is known that the impression of aging on a person's face is affected not only by static features such as skin color, age spots, and wrinkles, but also by dynamic features such as movement of the skin when facial expressions change. ing. In addition, it is known that the skin motility characteristics of each part of the face change according to the age of the subject. be.

Accordingly, an object of the present invention is to provide a moving image correction system capable of improving the impression of aging by correcting movements of the face and skin.

In order to solve the above problems, the present invention is a skin moving image correction system that corrects a moving image of a person's face based on motion information related to movement of feature points on the person's face with respect to facial expressions. hand,
The motion information includes target motion information as a correction target,
a moving image acquiring means for acquiring a moving image of a person's face;
feature point specifying means for specifying one or more of the feature points on the face in the moving image;
and correction means for generating a corrected moving image in which movement of the face is corrected by changing the movement of the feature points in the moving image based on the desired movement information.

In this way, by correcting the motion of the feature points using the desired motion information as the information (correction target) of the motion of the desired feature points, it is possible to generate a corrected moving image that can improve the impression of the motion of the face, for example. become.

In a preferred embodiment of the present invention, the motion information includes pre-correction motion information regarding motion of the feature points in the moving image,
measuring means for measuring the motion information before correction from the moving image;
storage means for storing a plurality of the desired exercise information,
The correcting means is characterized in that, based on the pre-correction motion information measured by the measuring means, the desired motion information to be used for generating the corrected moving image is determined from among a plurality of the desired motion information. .
With such a configuration, it is possible to determine what kind of correction is to be performed according to the movement of the subject's face in the acquired moving image, so that more appropriate correction can be performed. become.

In a preferred embodiment of the present invention, the apparatus further comprises unit facial expression setting means for setting a portion from one facial expression to another facial expression as a unit facial expression to be corrected,
The correcting means is characterized in that the corrected moving image is generated by correcting the movement of the face for each of the unit facial expressions.
In this way, in a video in which facial expressions change continuously, by setting each facial expression change as a unit facial expression and performing correction for each unit facial expression, the movement of each unit facial expression can be corrected individually. Accordingly, motion correction can be performed more appropriately.

In a preferred embodiment of the present invention, the motion information includes at least one of motion velocity and acceleration of the feature point.
With such a configuration, for example, from the timing at which the motion speed of each point reaches its maximum and the timing at which the acceleration reaches its maximum, it is possible to specify the timing of the start or end of the motion and the timing of the center of the motion. can be corrected.

A preferred form of the present invention is characterized in that the feature points include cheek points.
By adopting such a configuration, it is possible to correct the movement of the cheeks, which has a particularly large effect on the impression of a person, so that it is possible to correct moving images more effectively.

In a preferred embodiment of the present invention, the correction means sequentially corrects the acquired moving images to generate the corrected moving images in parallel with acquisition of the moving images by the moving image acquisition means,
In parallel with the generation of the corrected moving image by the correcting means, the apparatus further comprises display means for sequentially displaying the generated corrected moving image and transmitting the processing result.
With such a configuration, for example, it is possible to check the corrected moving image almost simultaneously while shooting the moving image. In addition, it is possible to transmit the corrected moving image to the communication partner almost in real time in parallel with the shooting, and use the corrected moving image in the video call.

In a preferred embodiment of the present invention, the feature points include a reference point on the face that serves as a reference for movement of the face when expressing an expression, and a reference point that follows movement of other parts of the face when expressing an expression. a tracking point, which is a point on the skin that moves with
The feature point identifying means includes tracking point identifying means for identifying the following point and reference point identifying means for identifying the reference point,
The correcting means is characterized in that the motion of the tracking point is changed to generate the corrected moving image.
By adopting such a configuration, it is possible to correct the followability of the skin movement accompanying the movement of the face, and for example, an effect of improving the impression of aging can be expected.

A preferred embodiment of the present invention is characterized in that the movement information includes information on the difference between the movement timing of the reference point and the movement timing of the following point.
In this way, by using information about the difference between the timing of movement of the reference point and the timing of movement of the tracking point as movement information, it is possible to correct the followability of skin movement for each part of the face.

In a preferred embodiment of the present invention, the correcting means changes the motion of the tracking point so as to reduce the difference between the timing of the motion of the reference point and the timing of the motion of the tracking point, thereby producing the corrected moving image. is characterized by generating
With such a configuration, it is possible to generate a corrected moving image in which the motion of the tracking point is adjusted in accordance with the motion of the reference point.

In a preferred embodiment of the present invention, the correction means shortens the difference between the timing of the motion of the reference point and the timing of the motion of the tracking point by advancing the timing of the motion of the tracking point. is characterized by generating
By adopting such a configuration, it is possible to correct the delay in the motion of the follow-up point with respect to the motion of the reference point, and improve the followability of the skin motion, so an effect of improving the impression of aging can be expected.

In a preferred embodiment of the present invention, a plurality of tracking points are defined,
The correction means adjusts the movement of each tracking point such that the smaller the distance between each tracking point and the reference point, the smaller the difference between the timing of the movement of the reference point and the timing of the movement of the tracking point. It is characterized by correcting and generating the corrected moving image.
By adopting such a configuration, it is possible to correct the movement of the tracking point step by step according to the distance from the reference point, thereby generating a smoother and more natural corrected moving image.

The present invention is a skin moving image correction program for correcting a moving image of a person's face based on motion information relating to movement of feature points on the person's face with respect to facial expressions,
The motion information includes target motion information as a correction target,
a moving image acquiring means for acquiring a moving image of a person's face, and
feature point specifying means for specifying one or more of the feature points on the face in the moving image;
and correction means for generating a corrected moving image in which the movement of the face is corrected by changing the movement of the feature points in the moving image based on the desired movement information.

The present invention is a skin moving image correction method for correcting a moving image of a person's face based on motion information related to movement of feature points on the person's face with respect to facial expression, comprising:
The motion information includes target motion information as a correction target,
a moving image acquiring step of acquiring a moving image of a person's face;
a feature point identifying step of identifying one or more of the feature points on the face in the moving image;
and a correction step of changing the motion of the feature points in the moving image based on the target motion information to generate a corrected moving image in which the movement of the face is corrected.

According to the present invention, it is possible to provide a moving image correction system capable of improving the impression of aging by correcting facial movements.

It is a figure which shows an example of the feature point in Embodiment 1 and Embodiment 2 of this invention. FIG. 10 is a conceptual diagram showing the tendency of each age group of delay in movement of a follow-up point on a person's face with respect to facial expressions. 1 is a diagram showing the configuration of a skin moving image correction system according to Embodiments 1 and 2 of the present invention; FIG. 1 is a functional block diagram of a skin moving image correction system according to Embodiments 1 and 2 of the present invention; FIG. It is a figure which shows an example of the desired motion information in Embodiment 1 and Embodiment 2 of this invention. 4 is a flow chart showing an example of processing from acquiring a moving image to generating a corrected moving image in Embodiment 1 of the present invention. 5 is a flowchart showing an example of processing related to generation of a corrected moving image according to Embodiments 1 and 2 of the present invention; 10 is a flow chart showing an example of processing from acquiring a moving image to generating a corrected moving image in Embodiment 2 of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The embodiments shown below are examples of the present invention, and the present invention is not limited to the following embodiments, and various configurations can be adopted.

For example, although the configuration, operation, etc. of the moving image correction system for skin will be described below, a method, a server device, a computer program, a recording medium, etc. having the same configuration can also achieve the same effect. Also, the program may be stored in a recording medium. Using this recording medium, the program can be installed in a computer, for example. Here, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM.

In the present invention, a corrected moving image is generated by correcting skin movement in a moving image of a person's face. Here, in an embodiment described later, a moving image obtained by photographing a state in which facial expressions change continuously is acquired and corrected. In such a case, the moving image to be acquired includes a plurality of facial expression expressions (facial expression changes). In order to appropriately correct each of them, in the present invention, the process of expressing each facial expression is defined as a "unit facial expression". That is, in the present invention, a unit facial expression refers to the start of the facial movement in facial expression, such as the process from a straight face to a smiling expression, or the process from the start to the end of mouth movement accompanying utterance. It refers to a group of movements from the time of starting to the time of finishing.

In the present invention, a feature point refers to a point on the face that moves along with expression of an expression, and in the embodiments described later, the feature point includes a reference point and a follow-up point. In the present invention, motion characteristics of feature points are corrected to generate a corrected moving image.

Also, motion information indicates information about the motion of the feature point. In the embodiments described later, the target motion information about the motion of the preferred feature point and the pre-correction motion information about the motion of the tracking point in the acquired moving image are included. As the motion information, various information related to the motion of the feature points, such as the motion velocity and acceleration of the feature points, and the timing difference of the motion from the reference point, can be arbitrarily used.

As the motion information, information indicating motion characteristics of the skin surface accompanying changes in facial expression can be used. Specific examples of movement characteristics include followability of facial skin movement when facial expressions change (hereinafter simply referred to as followability), and elasticity of facial skin when facial expressions change (hereinafter simply referred to as stretchability). . In addition, the deformability of facial skin (hereinafter simply referred to as "deformability") in response to changes in facial expression can also be used as the motion characteristic.

The followability of skin movement is the degree of delay in the movement of facial skin that changes following changes in facial expression. When facial expression changes occur, the skin on the face changes with a delay, and the smaller the degree of delay, the better the followability.

Followability can be evaluated quantitatively by observing any two feature points on the face when the facial expression changes and measuring the degree of deviation in the timing of movement of these two feature points. More specifically, the followability can be quantitatively measured as the difference in the time at which the movement speed of at least two feature points set at arbitrary positions on the face reaches its maximum in changes in facial expression.

In addition, the stretchability of the skin refers to the ease with which the skin stretches and contracts when the facial expression changes. For example, when there is a facial expression change that stretches the skin of the face, the higher the ratio of the increase in the distance in the stretching direction in an arbitrary area to the increase in the distance in the entire stretching direction, the higher the stretchability is. can be evaluated.

Elasticity can be quantified by calculating the distance between feature points in facial expression changes. For example, first, for all the feature points set on the face, the sum of the distances between adjacent feature points increased due to changes in facial expression is calculated. At the same time, for some feature points existing in a specific region of the face, the sum of the distances between adjacent feature points increased due to changes in facial expression is calculated. By dividing the latter numerical value by the former numerical value, the stretchability can be quantitatively measured.

In addition, by quantifying the deformation method (distortion method) of an area set at an arbitrary position on the face based on the feature points, which changes when the facial expression changes, and using it as motion information, it is possible to determine the shape of the facial skin when the facial expression changes. It can also be in the form of correcting deformability. A specific method for measuring deformability is not particularly limited. For example, information on the shape of an arbitrary area set on the skin of the face before and after the change in facial expression is acquired using the optical flow method or motion capture method using the captured video, and distortion analysis and deformation of the shape are obtained. A method of analysis can be exemplified.

In the present invention, the values obtained by quantifying the followability, stretchability, deformability, etc. as described above are used as motion information, and correction is performed based on this information, thereby correcting the movement of the face and improving the impression. It is possible to generate a corrected moving image. In addition, below, the case where followability is corrected will be described in detail with a specific example.

<Embodiment 1>
In this embodiment, an example of using the skin moving image correction system according to the present invention for a video call will be described. That is, in the present embodiment, acquisition of a moving image and generation of a corrected moving image are executed in parallel, and the corrected moving image is displayed to the other party of the call almost in real time.

In the present invention, the reference point is a point on the face that moves along with expression of facial expressions, and in this embodiment, the point on the chin is exemplified. In this way, it is preferable to use a point that moves almost at the same time as the facial expression changes as the reference point. In addition, a plurality of reference points may be set so that different reference points are used for each unit facial expression. may be used as a reference point in the unit facial expression.

The follow-up point is a point on the skin that moves as if it is pulled by the movements of other parts of the face, especially a reference point, etc., when an expression is expressed. exemplify a plurality of points of

FIG. 1 is a diagram showing an example of reference points and tracking points (feature points) in this embodiment. Thus, in this example, the point on the chin is defined as the reference point R, and the three points on the cheeks are defined as the follow-up points A to C in order of proximity to the reference point R. Although FIG. 1 shows the left side of the face, in the present embodiment, the following points A to C are set symmetrically on the right side of the cheek and treated in the same way.

FIG. 2 is a conceptual diagram showing tendencies of age group-specific delays occurring between movement of a reference point and movement of a follow-up point on the face when expression is expressed. The horizontal axis indicates the distance from the reference point R, and the vertical axis indicates the difference between the time when the movement speed of the reference point R reaches its maximum and the time when the movement speed of each following point reaches its maximum. That is, FIG. 2 is a diagram schematically showing the tendency of the magnitude of the delay in the movement of each following point with respect to the movement of the reference point R according to the distance from the reference point R for each age group.

In this way, the motion of the follow-up point in expression of facial expressions lags behind the motion of the reference point, and the lag tends to increase with age. In addition, it is known that the impression of aging on the person to be photographed changes when he or she sees a moving image showing the process of facial expression changes, compared to when he or she sees still images showing various expressions of the person. .

From the above, it can be inferred that the delay in the motion of the follow-up point with respect to the motion of the reference point in expressing facial expressions gives the viewer an aged impression. In this embodiment, a configuration for generating a corrected moving image that can improve the impression of aging by improving followability will be described.

FIG. 3 is a diagram showing the configuration of the skin moving image correction system according to the first embodiment. As described above, the skin moving image correction system according to the present embodiment is configured such that the server device 1 and the display devices 2a and 2b can communicate with each other via the network. To simplify the explanation, one server device 1 and two display devices 2 are shown here. may be provided with three or more. Further, the display device 2 may be configured to store the skin moving image correction program and function as each unit included in the server device 1 .

FIG. 4 is a functional block diagram of the skin moving image correction system according to the first embodiment. As shown here, the server device 1 includes moving image acquisition means 11, tracking point identification means 12, reference point identification means 13, measurement means 14, unit facial expression setting means 15, correction means 16, display Means 17 and storage means 18 are provided.

A moving image acquiring means 11 acquires a moving image including a process of expression expression in a person's face, and a tracking point specifying means 12 and a reference point specifying means 13 each acquire a following point for the human face included in the moving image. and identify a reference point.

In this embodiment, the parts of the face that serve as the tracking points and the reference points are determined in advance, and the tracking points and the reference points are specified by detecting the points from the moving image. The configuration may be such that the tracking point and the reference point are specified based on the movement of each part of the face. For example, a method is conceivable in which the movement of the face associated with the expression of an expression is detected, the point that moves the fastest at that time is set as the reference point, and the point that moves after that is set as the follow point. Also, the method of specifying the reference point and the follow-up point may be different, such as specifying the reference point for each unit expression according to the movement of the face, and always using the point of the cheek as the follow-up point. Any other method may be used to identify the tracking point and the reference point.

The measuring unit 14 measures pre-correction motion information regarding the motion of the tracking point included in the moving image acquired by the moving image acquisition unit 11 . In this embodiment, the motion speed is measured in chronological order for the tracking points specified by the tracking point specifying means 12 . Further, in this embodiment, the motion speed of the reference point specified by the reference point specifying means 13 is similarly measured in time series, and based on this, the difference between the motion timing of the reference point and the motion timing of the following point is calculated. Measure.

Here, in the present embodiment, the forehead point is specified as a reference point that serves as a reference for facial expression movement so as not to be affected by the movement of the entire head, and the relative motion speed with respect to the forehead point is measured. . As a reference point, a point that moves with the movement of the head and is not affected by the movement of the facial expression (does not move with the expression of the facial expression) is preferable. It should be noted that specifying the reference point is not essential, and the absolute motion speed of the reference point and the follow-up point may simply be measured.

A unit facial expression setting means 15 sets a unit facial expression in the moving image acquired by the moving image acquiring means 11 . In this embodiment, facial movements are analyzed, and the facial movements from the start to the end of each facial expression are set as a unit facial expression.

The correction means 16 generates a corrected moving image by correcting the motion of the tracking point in the moving image based on the desired motion information stored in the storage means 18, which will be described later. In this embodiment, correction is performed for each unit facial expression set by the unit facial expression setting means 15 .

The display unit 17 performs display processing on the corrected moving image generated by the correction unit, and transmits the processing result. In this embodiment, the corrected moving image generated from the moving image shot on the display device 2a is processed for display by the display means 17, and the processing result is transmitted to the display device 2b on the other party's side.

The storage unit 18 stores target motion information that serves as a guideline for the motion of the tracking point for correcting the moving image. FIG. 5 is a diagram showing an example of desired motion information. As shown here, in this embodiment, information indicating motion characteristics for each age is stored for each tracking point. In this case, the storage means 18 stores a measure of the magnitude of the delay in the motion timing of each follow-up point with respect to the motion timing of the reference point for each age, and uses it as the desired motion information.

In other words, if the facial movement is to be corrected to the twenties, the information indicating the motion characteristics of each tracking point in the twenties can be used as the target motion information. can be used as the desired motion information. In the case of this example, for example, the estimated delay at the follow-up point C for people in their twenties is stored as 30 ms, and the estimated delay at the follow-up point C for people in their fifties is stored as 100 ms.

In this example, a numerical value indicating the size of the delay at each tracking point is stored for each age and used as desired motion information. It is also possible to store a plurality of functions representing the magnitude of .

In addition, in this embodiment, the difference between the reference point and the tracking point at the timing when the movement speed becomes maximum in each unit facial expression (the delay of the tracking point with respect to the reference point) is defined as the "magnitude of delay". You may express the delay in any way. For example, the magnitude of the delay may be expressed by the difference in the timing (beginning of movement or end of movement) at which the acceleration of each of the reference point and the following point is maximized in each unit facial expression.

In the case of correcting various motion characteristics other than followability, it is also possible to similarly store reference values for each age group and use them as desired motion information. Also in this case, motion information may be stored as a function indicating motion characteristics.

The display devices 2a and 2b are provided with input means 21 and output means 22, respectively. As the display device 2, a general computer such as a PC (Personal Computer) or a smart phone can be used. The display devices 2a and 2b store a dedicated application for functioning as the display device 2 in the skin moving image correction system according to this embodiment and a web browser application for using web pages.

As the input means 21, a photographing device such as a general camera capable of photographing at least moving images can be adopted, and as the output means 22, a general display device capable of displaying at least corrected moving images, etc., can be adopted. be able to. In the present embodiment, a sound collecting device such as a microphone capable of acquiring sound is further provided as the input means 21, and a video call is performed by transmitting the acquired sound together with the corrected moving image to the display device 2 of the other party.

FIG. 6 is a flow chart showing the flow of processing from acquiring a moving image to generating a corrected moving image in this embodiment. When shooting and transmission of a moving image are started via the input means 21, the process of FIG. 6 is started, and the moving image acquisition means 11 acquires a moving image in step S11. Here, the captured moving images are sequentially transmitted to the server device 1 in parallel with the capturing of the moving images by the input means 21, and the moving image acquisition means 11 sequentially acquires the transmitted moving images.

Next, in step S12, the unit facial expression setting means 15 sets a unit facial expression for the moving image acquired in step S11. In this embodiment, the movement of the face is analyzed, and one unit facial expression is set from the time when the facial expression starts changing to the time when the facial expression ends. The unit facial expression may be set by any method.

Specifically, for example, facial movement models for expressing various expressions are stored in the storage means 18 or an external database, etc., and the acquired moving image is analyzed in light of the facial movement models. Therefore, a method of setting a unit facial expression can be considered. In addition, it is also possible to specify the tracking points and reference points and measure the motion information before step S12, and set the unit facial expression based on the measurement results.

When the setting of the unit facial expression is completed, the process proceeds to step S13, and the correction means 16 generates a corrected moving image for each unit facial expression. FIG. 7 is a flowchart showing an example of processing related to generation of a corrected moving image in this embodiment.

First, in step S21, the tracking point specifying means 12 and the reference point specifying means 13 specify the reference point and the following point on the human face included in the moving image. As described above, in this embodiment, the chin point is set as a reference point, and the three cheek points are set as tracking points A to C in advance. Although the point is specified, the configuration may be such that the tracking point is specified based on the movement of the face in the moving image.

After the reference point and the tracking point have been specified, the measurement means 14 measures pre-correction motion information in steps S22 to S24. In this embodiment, first, in step S22, the movement speed of each point is measured from the amount of movement of the reference point and each follow-up point between frames constituting the moving image.

In the following step S23, based on the motion velocities of the reference point and each following point measured in step S22, the timing of movement of the reference point and each following point in the target unit facial expression is specified. In this embodiment, the timing at which the motion speed of the reference point and each tracking point reaches its maximum is specified.

Then, in step S24, the magnitude of the motion delay of each tracking point with respect to the reference point is calculated from the timing of the motion of each tracking point and the reference point specified in step S23. In the present embodiment, the magnitude of the motion delay of each tracking point calculated here is used as pre-correction motion information to generate a corrected moving image.

In step S25, the correcting means 16 adjusts the timing of movement of each tracking point. Here, in the present embodiment, the magnitude of the motion delay (pre-correction motion information) of each tracking point calculated in step S24 indicates the motion characteristics for each age for each tracking point stored in the storage means 18. In comparison with the information, it is specified in which age the pre-correction motion information is close to the motion characteristics.

Then, the correcting means 16 determines the timing of the post-correction motion at each tracking point using the information indicating the motion characteristics for each age in the age one step younger than the age specified from the pre-correction motion information as the desired motion information. do. However, the selection of the desired motion information is not limited to this. For example, the desired motion information may be selected so as to accept designation of the correction strength via the display device 2 and perform correction at a degree based on the designation. .

Here, in the present embodiment, the target motion information is defined such that the closer the tracking point is to the reference point, the smaller the motion timing delay, and based on this, the corrected moving image is generated. In this way, by adjusting the timing of the motion of each tracking point according to the distance of each tracking point to the reference point, a more natural corrected moving image can be generated.

When the adjustment of the motion timing of the tracking points in step S25 is completed, in step S26, based on this, the correction means 16 determines the time-series motion speed of each tracking point. In this embodiment, the timing at which the maximum speed is reached is adjusted in step S25, and the motion speed in time series is determined by linear interpolation in step S26. In addition, it may be changed so as to be adjusted by acceleration.

In step S27, the image of each frame constituting the moving image is adjusted based on the motion speed of each tracking point to generate a corrected moving image. A general image editing technique may be used to adjust the image of each frame. Frame images can be adjusted. At this time, it is preferable to adjust each frame in consideration of the continuity with the preceding and succeeding frames.

When the generation of the corrected moving image for the unit facial expression is completed in this way, returning to the flow chart of FIG. Here, in the present embodiment, when the corrected moving image generated from the moving image shot by the display device 2a is processed for display by the display means 17, the processing result is transmitted to the display device 2b on the other party's side, It is displayed to the other party via the output means 22 of the display device 2b. Note that the display device 2a may similarly display the corrected moving image.

In this way, sequentially input moving images are acquired, correction is performed for each unit facial expression to generate a corrected moving image, and displayed on the other party's side of the call until an instruction to end shooting is received via the display device 2 in step S15. Displayed continuously in device 2 . By continuously performing the sequential processing in this way, it is possible to display the corrected moving image almost in real time.

The flow of processing shown in FIGS. 6 and 7 is an example, and the order and details of processing may be changed arbitrarily. Also, in this example, after acquiring the moving image in step S11, in step S12, the unit facial expression is set and the corrected moving image is generated for each unit facial expression. may For example, a configuration may be adopted in which a unit facial expression is not included in a portion where the facial expression does not change, and the corrected moving image generation and display processing in steps S13 and S14 are skipped.

<Embodiment 2>
Another embodiment of the present invention will be described below. In addition, the same code|symbol is attached|subjected about the structure similar to Embodiment 1, and description is abbreviate|omitted.

In this embodiment, a configuration will be described in which moving images are acquired via the display device 2 and corrected moving images are stored in the storage means 18 . In this embodiment, it is not necessary to generate a corrected moving image at the same time as shooting, and a configuration is shown in which a corrected moving image is generated by performing processing after shooting a moving image.

FIG. 8 is a flow chart showing the flow of processing from acquiring a moving image to generating a corrected moving image in this embodiment. First, in step S<b>31 , the moving image acquiring means 11 acquires moving images via the display device 2 . Acquisition of the moving image here does not need to be performed at the same time as the shooting.

Next, in step S32, the unit facial expression setting means 15 sets a unit facial expression for the acquired moving image. Here, the unit facial expression may be set in the same manner as in the first embodiment, or an input regarding the start point and the end point of the unit facial expression may be received via the display device 2 and set based on this. Alternatively, after setting the unit facial expression in the same manner as in the first embodiment, an input for adjusting the range of the unit facial expression may be received.

In step S33, a corrected moving image is generated as in the first embodiment. That is, for each unit facial expression, a corrected moving image is generated based on the motion information. Here, for example, input such as "correct the movement so that the user looks one step younger" or "correct the movement so that the movement becomes the one in their twenties" is received, and the desired exercise information is selected based on the input. good too. Conversely, it is also possible to accept an input such as "correction to look one step older".

When the generation of the corrected moving image for each unit facial expression is completed for the entire moving image, in step S34, these corrected moving images are synthesized to generate the entire corrected moving image. Here, the motion of the tracking point may be adjusted so that the corrected moving images for each unit facial expression are connected naturally.

Then, in step S35, the generated corrected moving image is stored in the storage means 18, and the process ends. The display means 17 displays the corrected moving image stored in the storage means 18 according to the display request of the display device 2 and transmits the processing result. Further, the corrected moving image may be downloaded to the display device 2 so that it can be confirmed.

As described above, according to the skin moving image correction system according to the present invention, it is possible to generate a corrected moving image with an improved impression by correcting the movement of the skin when expressing an expression. Further, by determining the target motion information according to the pre-correction motion information as in the present embodiment, it is expected that the motion of the face can be naturally corrected.

In the present embodiment, a configuration is shown in which the followability is corrected by specifying a reference point and a follow-up point as feature points and generating a corrected moving image using the difference in motion timing as motion information. , the present invention is not limited to this. In addition, in order to correct the stretchability and deformability described above, information such as distances between feature points and changes in areas defined by a plurality of feature points may be used as motion information. Moreover, it is good also as a structure which correct|amends with respect to a compound exercise|movement characteristic by combining these.

1 Server Device 11 Moving Image Acquisition Means 12 Tracking Point Identification Means 13 Reference Point Identification Means 14 Measurement Means 15 Unit Facial Expression Setting Means 16 Correction Means 17 Display Means 18 Storage Means 2 Display Device 21 Input Means 22 Output Means R Reference Points

Claims (11)

A skin moving image correction system for correcting a moving image of a person's face based on motion information related to movement of feature points on the person's face with respect to facial expressions,
The motion information includes target motion information as a correction target,
a moving image acquiring means for acquiring a moving image of a person's face;
feature point specifying means for specifying one or more of the feature points on the face in the moving image;
a correcting means for changing the motion of the feature points in the moving image based on the target motion information to generate a corrected moving image in which the movement of the face is corrected ;
The feature points are a reference point on the face that serves as a reference for movement when expressing an expression, and a point on the skin that moves following the movement of other parts of the face when expressing an expression. including a point and
The feature point identifying means includes tracking point identifying means for identifying the following point and reference point identifying means for identifying the reference point,
the motion information includes information about a difference between the motion timing of the reference point and the motion timing of the tracking point;
The skin moving image correction system, wherein the correcting means generates the corrected moving image by changing the motion of the tracking point. the motion information includes pre-correction motion information relating to motion of the feature points in the moving image;
measuring means for measuring the motion information before correction from the moving image;
storage means for storing a plurality of the desired exercise information,
The correcting means is characterized in that, based on the pre-correction motion information measured by the measuring means, the desired motion information to be used for generating the corrected moving image is determined from among a plurality of the desired motion information. , The skin animation correction system according to claim 1. further comprising unit facial expression setting means for setting a portion from one facial expression to another facial expression in the moving image as a unit facial expression to be corrected;
3. The skin moving image correction system according to claim 1, wherein said correction means generates said corrected moving image by correcting facial movements for each said unit expression. 4. The skin moving image correction system according to claim 1, wherein said motion information includes at least one of motion velocity and acceleration of said feature point. 5. The skin moving image correction system according to claim 1, wherein said feature points include cheek points. The correction means sequentially corrects the acquired moving images in parallel with acquisition of the moving images by the moving image acquisition means to generate the corrected moving images,
6. The apparatus according to any one of claims 1 to 5, further comprising display means for sequentially performing display processing of the generated corrected moving images and transmitting processing results in parallel with generation of the corrected moving images by the correcting means. The skin moving image correction system described in . The correcting means generates the corrected moving image by changing the motion of the tracking point so that the difference between the timing of the motion of the reference point and the timing of the motion of the tracking point is reduced. The skin moving image correction system according to claim 1 , wherein The correcting means generates the corrected moving image in which the difference between the timing of the movement of the reference point and the timing of the movement of the tracking point is reduced by advancing the timing of the movement of the tracking point. 8. The skin moving image correction system according to claim 7 . A plurality of said tracking points are defined,
The correction means adjusts the movement of each tracking point such that the smaller the distance between each tracking point and the reference point, the smaller the difference between the timing of the movement of the reference point and the timing of the movement of the tracking point. 9. The skin moving image correction system according to any one of claims 1, 7 and 8 , wherein correction is performed to generate the corrected moving image. A skin moving image correction program for correcting a moving image of a person's face based on motion information related to movement of feature points on the person's face with respect to facial expressions,
The motion information includes target motion information as a correction target,
a moving image acquiring means for acquiring a moving image of a person's face, and
feature point specifying means for specifying one or more of the feature points on the face in the moving image;
functioning as correction means for generating a corrected moving image in which the motion of the face is corrected by changing the motion of the feature points in the moving image based on the target motion information ;
The feature points are a reference point on the face that serves as a reference for facial movement when expressing an expression, and a point on the skin that moves following the movement of other parts of the face when expressing an expression. including a tracking point and
The feature point identifying means includes tracking point identifying means for identifying the following point and reference point identifying means for identifying the reference point,
the motion information includes information about a difference between the motion timing of the reference point and the motion timing of the tracking point;
The skin moving image correction program, wherein the correcting means generates the corrected moving image by changing the movement of the following point. A skin moving image correction method for correcting a moving image of a person's face based on motion information relating to movement of feature points on the person's face with respect to facial expressions, comprising:
The motion information includes target motion information as a correction target,
a moving image acquiring step of acquiring a moving image of a person's face;
a feature point identifying step of identifying one or more of the feature points on the face in the moving image;
a correction step of generating a corrected moving image in which the movement of the face is corrected by changing the movement of the feature points in the moving image based on the target movement information ;
The feature points are a reference point on the face that serves as a reference for facial movement when expressing an expression, and a point on the skin that moves following the movement of other parts of the face when expressing an expression. including a tracking point and
The feature point identification step includes a tracking point identification step of identifying the tracking point and a reference point identification step of identifying the reference point,
the motion information includes information about a difference between the motion timing of the reference point and the motion timing of the tracking point;
A skin moving image correcting method, wherein the correcting step is a step of generating the corrected moving image by changing the motion of the tracking point.

Images