JP2020043516A - Skin moving image correction system, skin moving image correction program, and skin moving image correction method - Google Patents

Abstract

To provide a moving image correction system capable of improving an aging impression by correcting movement of a face or skin in a moving image.SOLUTION: The skin moving image correction system for correcting a photographed moving image of a human face on the basis of movement information on movements of feature points on the human face for an expression of a facial expression, the movement information including target movement information for a correction target, includes: moving image acquisition means for acquiring a photographed moving image of a human face; feature point specifying means for specifying one or more feature points on the face in the moving image; and correction means for generating a corrected moving image in which the movement of the face has been corrected by changing the movements of the feature points in the moving image on the basis of the target movement information.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a skin moving image correction system, a skin moving image correction program, and a skin moving image correction method for correcting a skin moving image obtained by capturing a human face.

  2. Description of the Related Art Conventionally, a correction process is performed on an image of a human face to improve an impression. In particular, in recent years, a technique capable of performing such a correction process on a moving image has been known.

  For example, in the device described in Patent Literature 1, by performing correction based on a representative color that is a color representing a target area to be corrected, even if the representative color fluctuates for each frame constituting a moving image. Thus, a natural correction result can be obtained.

JP-A-2006-171445

  By the way, it is known that the aging impression on a human face is affected not only by static characteristics such as skin color, spots and wrinkles, but also by dynamic characteristics such as movement of the skin when the facial expression changes. ing. In addition, it is known that the skin movement characteristics of each part of the face change according to the age of the subject, and from these facts, it is inferred that the change in the skin movement characteristics due to aging gives an aging impression. You.

  Accordingly, an object of the present invention is to provide a moving image correction system that can improve the aging impression by correcting the movement of the face and skin.

In order to solve the above problems, the present invention is a skin moving image correction system that corrects a moving image obtained by photographing a human face, based on motion information on the motion of a feature point on the human face with respect to expression of an expression. hand,
The exercise information includes target exercise information to be a correction target,
Moving image acquisition means for acquiring a moving image of a human face,
Feature point specifying means for specifying one or more feature points on the face in the moving image;
And correcting means for changing a motion of the feature point in the moving image based on the target motion information to generate a corrected moving image in which a face motion is corrected.

  As described above, by correcting the motion of the feature point using the target motion information as the preferable motion information of the feature point (correction target), it is possible to generate a corrected moving image that can improve the impression of the face motion, for example. become.

In a preferred embodiment of the present invention, the motion information includes pre-correction motion information regarding the motion of the feature point in the moving image,
From the moving image, measuring means for measuring the pre-correction motion information,
Storage means for storing a plurality of the target exercise information,
The correction means determines the target motion information used for generating the corrected moving image from a plurality of the target motion information based on the pre-correction motion information measured by the measurement means. .
With such a configuration, it is possible to determine what correction should be performed according to the movement of the face of the subject in the acquired moving image, so that the correction can be performed more appropriately. become.

In a preferred embodiment of the present invention, the moving image further comprises a unit expression setting unit that sets a portion from a certain expression to another expression as a unit expression to be corrected,
The correction means may correct the face movement for each unit expression to generate the corrected moving image.
As described above, in a moving image in which facial expressions change continuously, each facial expression change is set as a unit facial expression, and by correcting each facial expression, the movement of each facial expression can be individually corrected. Accordingly, the motion can be corrected more appropriately.

In a preferred aspect of the present invention, the motion information includes at least one of a motion speed and an acceleration of the feature point.
With such a configuration, for example, the timing of the start or end of the movement or the timing of the center of the movement can be specified from the timing at which the movement speed of each point is maximized or the timing at which the acceleration is maximized. Can be corrected.

In a preferred aspect of the present invention, the feature points include a cheek point.
By adopting such a configuration, it is possible to correct the movement of the cheek, which greatly affects the impression of a person, and therefore, it is possible to more effectively correct the moving image.

In a preferred embodiment of the present invention, the correction unit, in parallel with the acquisition of the moving image by the moving image acquisition unit, to sequentially correct the acquired moving image to generate the corrected moving image,
In parallel with the generation of the corrected moving image by the correction unit, a display unit for sequentially displaying the generated corrected moving image and transmitting a processing result is further provided.
With such a configuration, for example, a corrected moving image can be checked almost simultaneously with capturing a moving image. In addition, the corrected moving image can be transmitted to the communication partner almost in real time in parallel with the shooting, so that the corrected moving image can be used for a video call.

In a preferred embodiment of the present invention, the feature points follow a reference point on the face, which is a reference for the movement of the face when expressing an expression, and follow the movement of other parts on the face when expressing the expression. A tracking point, which is a point on the skin that moves
As the characteristic point identifying means, a tracking point identifying means for identifying the tracking point, and a reference point identifying means for identifying the reference point,
The correction means may generate the corrected moving image by changing the movement of the follow-up point.
With such a configuration, it is possible to correct the followability of the movement of the skin caused by the movement of the face, and for example, an effect of improving the aging impression can be expected.

In a preferred aspect of the present invention, the movement information includes information on a difference between the movement timing of the reference point and the movement timing of the following point.
In this way, by using information on the difference between the timing of the movement of the reference point and the timing of the movement of the follow-up point as the movement information, it is possible to correct the followability of the movement of the skin for each part of the face.

In a preferred aspect of the present invention, the correction means changes the motion of the tracking point so that a difference between the timing of the motion of the reference point and the timing of the motion of the tracking point is reduced, and Is generated.
With such a configuration, it is possible to generate a corrected moving image in which the movement of the following point is adjusted in accordance with the movement of the reference point.

In a preferred embodiment of the present invention, the correction means reduces the difference between the timing of the movement of the reference point and the timing of the movement of the tracking point by advancing the timing of the movement of the tracking point. Is generated.
With such a configuration, it is possible to correct the delay of the movement of the follow-up point with respect to the movement of the reference point, and it is possible to improve the followability of the skin movement, so that an effect of improving the aging impression can be expected.

In a preferred embodiment of the present invention, a plurality of the following points are defined,
The correction means reduces the difference between the timing of the movement of the reference point and the timing of the movement of the tracking point as the distance between each of the tracking points and the reference point is smaller. The correction is performed to generate the corrected moving image.
With such a configuration, the movement of the following point can be corrected stepwise according to the distance from the reference point, and a smoother and more natural corrected moving image can be generated.

The present invention is a skin moving image correction program for correcting a moving image obtained by photographing a human face, based on motion information on motion of a feature point on a human face for expression of an expression,
The exercise information includes target exercise information to be a correction target,
A computer, a moving image acquisition unit that acquires a moving image of a human face,
Feature point specifying means for specifying one or more feature points on the face in the moving image;
Based on the target motion information, the motion of the feature point in the moving image is changed to function as a correction unit that generates a corrected moving image in which the motion of the face is corrected.

The present invention is a skin moving image correction method for correcting a moving image obtained by photographing a human face, based on motion information on motion of a feature point on the human face with respect to expression of expression,
The exercise information includes target exercise information to be a correction target,
A moving image acquiring step of acquiring a moving image of a human face,
A feature point specifying step of specifying one or a plurality of the feature points on the face in the moving image;
A correcting step of changing a motion of the feature point in the moving image based on the target motion information to generate a corrected moving image in which a motion of the face is corrected.

  ADVANTAGE OF THE INVENTION According to this invention, the moving image correction system which can improve an aging impression can be provided by performing correction about a face movement.

It is a figure showing an example of the feature point in Embodiment 1 and Embodiment 2 of the present invention. It is a conceptual diagram which shows the tendency for the delay which arises in the movement of the follow-up point on the human face with respect to expression of expression for every age group. It is a figure showing composition of a skin moving picture correction system in Embodiment 1 and Embodiment 2 of the present invention. It is a functional block diagram of the skin moving image correction system in Embodiment 1 and Embodiment 2 of the present invention. It is a figure showing an example of target exercise information in Embodiment 1 and Embodiment 2 of the present invention. 5 is a flowchart illustrating an example of processing from acquisition of a moving image to generation of a corrected moving image according to the first embodiment of the present invention. 9 is a flowchart illustrating an example of processing related to generation of a corrected moving image according to the first and second embodiments of the present invention. 9 is a flowchart illustrating an example of processing from acquisition of a moving image to generation of a corrected moving image according to the second embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The embodiment described below is an example of the present invention, and the present invention is not limited to the following embodiment, and various configurations can be adopted.

  For example, a configuration, an operation, and the like of the skin moving image correction system will be described below. However, a method, a server device, a computer program, a recording medium, and the like having the same configuration can also provide the same operation and effect. Further, the program may be stored in a recording medium. By using this recording medium, for example, the program can be installed in a computer. Here, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM.

  According to the present invention, a corrected moving image is generated by correcting the movement of the skin in a moving image obtained by capturing a human face. Here, in an embodiment to be described later, correction is performed by acquiring a moving image that captures a continuously changing expression. In such a case, the moving image to be acquired includes a plurality of expression expressions (expression changes). In order to appropriately correct each of them, in the present invention, each expression expression process is defined as "unit expression". That is, in the present invention, the unit facial expression is the start of the facial movement in the facial expression, such as the process from the state of a true face to the expression of a smile to a smiling face, or the process from the start to the end of the movement of the mouth accompanying vocalization. It refers to a group of movements from the point in time to the point in time when it ends.

  In the present invention, a feature point refers to a point on a face that moves with the expression of an expression, and in a later-described embodiment, includes a reference point and a follow-up point as the feature points. In the present invention, the motion characteristics of the feature points are corrected, and a corrected moving image is generated.

  In addition, the motion information indicates information regarding the motion of the feature point. In an embodiment described later, target motion information regarding the motion of the preferable feature point and pre-correction motion information regarding the motion of the follow-up point in the acquired moving image are included. As the motion information, various information on the motion of the feature point, such as the motion speed and acceleration of the feature point, the difference in the motion timing with respect to the reference point, and the like can be arbitrarily used.

  As the motion information, information indicating the motion characteristics of the skin surface due to the change in facial expression can be used. Specific examples of the movement characteristics include the followability of the skin movement of the face in a change in facial expression (hereinafter, also simply referred to as followability) and the elasticity of the skin of the face in the change of facial expression (hereinafter, also simply referred to as elasticity). . In addition, as the motion characteristic, the deformability of the skin of the face due to a change in the expression (hereinafter, also simply referred to as deformability) can be adopted.

  The ability to follow the skin movement refers to the degree of delay in the movement of the skin of the face that changes following the change in facial expression. When the facial expression changes, the skin of the face changes with a delay from the movement, but the smaller the degree of the delay, the better the tracking ability.

  The followability can be quantitatively evaluated by observing any two feature points of the face when the facial expression changes, and measuring the degree of deviation of the movement timing of the two feature points. More specifically, the following ability can be quantitatively measured as a difference in time when the movement speed of at least two feature points set at arbitrary positions of the face in the facial expression change becomes maximum.

  The skin elasticity refers to the ease with which the skin expands and contracts when the facial expression changes. For example, when there is a facial expression change in which the skin of the face extends, the higher the ratio of the increase in the distance in the extension direction in a given area to the increase in the distance in the entire extension direction, the higher the “excellent elasticity”. Can be evaluated.

  The elasticity can be quantified by calculating the distance between feature points in a facial expression change. For example, first, for all the feature points set on the face, the sum of the distances between the feature points adjacent to each other, which has been increased due to the change in the expression, is calculated. At the same time, with respect to some feature points existing in a specific area of the face, the sum of distances between feature points adjacent to each other, which is increased due to a change in expression, is calculated. The elasticity can be quantitatively measured by dividing the latter value by the former value.

  In addition, by quantifying a deformation method (distortion method) of a region set at an arbitrary position of the face based on a feature point, which changes in a facial expression change, and using the information as motion information, the facial skin in the facial expression change is quantified. A form in which the deformability is corrected may be adopted. The specific method for measuring the deformability is not particularly limited. For example, information on the shape of an arbitrary region set on the skin of the face before and after the facial expression change is obtained by using an optical flow method or a motion capture method using a shot moving image, and distortion analysis and deformation of the shape are performed. A method for performing the analysis can be exemplified.

  According to the present invention, the value obtained by quantifying the followability, elasticity, deformability, and the like as described above is used as motion information, and correction is performed based on the information, thereby correcting the face motion and improving the impression. It is possible to generate a corrected moving image. In the following, a case where the followability is corrected will be described in detail with a specific example.

<First embodiment>
In the present embodiment, an example in which the skin moving image correction system according to the present invention is used for a video call will be described. That is, in the present embodiment, acquisition of a moving image and generation of a corrected moving image are performed in parallel, and the corrected moving image is displayed to the other party in near real time.

  In the present invention, the reference point is a point on the face that moves with the expression of an expression, and in the present embodiment, a point on the jaw is exemplified. As described above, it is preferable that a point that moves almost simultaneously with the change of the expression be set as the reference point. In addition, a plurality of reference points may be set to use different reference points for each unit facial expression, or a specific reference point may not be set in advance, and a face motion may be detected from a moving image to achieve the fastest movement. May be used as a reference point in the unit expression.

  Further, the follow-up point is a point on the skin that moves as it is pulled by the movement of another part on the face, especially the reference point, etc., when the facial expression is expressed. Are exemplified.

  FIG. 1 is a diagram illustrating an example of a reference point and a follow-up point (feature point) according to the present embodiment. As described above, in this example, the points on the jaw are set as the reference points R, and the three points on the cheeks are set as the follow-up points A to C in order of proximity to the reference points R. Although the left face is shown in FIG. 1, in the present embodiment, the following points A to C are set symmetrically on the right cheek, and they are handled similarly.

  FIG. 2 is a conceptual diagram showing the tendency of the delay occurring for each age group between the movement of the reference point and the movement of the follow-up point on the face when expressing an expression. The abscissa indicates the distance from the reference point R, and the ordinate indicates the difference between the time when the movement speed of the reference point R becomes maximum and the time when the movement speed of each of the following points becomes maximum in the expression. That is, FIG. 2 is a diagram schematically showing the tendency of the magnitude of the delay of the movement of each following point with respect to the movement of the reference point R according to the distance from the reference point R for each age group.

  As described above, the movement of the follow-up point in the expression of the facial expression is delayed with respect to the movement of the reference point, and the delay tends to increase as the age increases. In addition, it is known that when a moving image showing a process of a facial expression change is viewed, an aging impression with respect to the subject changes as compared with a case where a still image showing various facial expressions of the subject is viewed. .

  From the above, it can be inferred that the aging impression is given to the viewer due to the delay of the movement of the follow-up point with respect to the movement of the reference point in the expression of the facial expression of the person. In the present embodiment, a configuration for generating a corrected moving image that can improve the aging impression by improving the followability will be described.

  FIG. 3 is a diagram illustrating a configuration of the skin moving image correction system according to the first embodiment. As described above, the skin moving image correction system according to the present embodiment is configured so that the server device 1 and the display devices 2a and 2b can communicate with each other via the network. Although one server device 1 and two display devices 2 are shown here for simplicity of explanation, the server device 1 may be realized by cooperating a plurality of computer devices. May be provided three or more. Further, a configuration may be adopted in which a skin moving image correction program is stored in the display device 2 so as to function as each unit included in the server device 1.

  FIG. 4 is a functional block diagram of the skin moving image correction system according to the first embodiment. As shown here, the server device 1 includes a moving image acquisition unit 11, a following point identification unit 12, a reference point identification unit 13, a measurement unit 14, a unit expression setting unit 15, a correction unit 16, a display unit Means 17 and storage means 18.

  The moving image obtaining means 11 obtains a moving image including a process of expressing a facial expression on a human face, and the tracking point specifying means 12 and the reference point specifying means 13 respectively determine a tracking point for the human face included in the moving image. And a reference point.

  In the present embodiment, a face portion to be a tracking point and a reference point is determined in advance, and the tracking point and the reference point are specified by detecting the point from the moving image. A configuration in which the following point and the reference point are specified based on the movement of each part of the face may be adopted. For example, a method of detecting the movement of the face accompanying the expression of the facial expression, specifying the point that moves fastest at that time as the reference point, and specifying the point that moves so as to follow up as the following point, or the like can be considered. The reference point and the follow-up point may be specified differently, for example, the reference point is specified for each unit expression by the movement of the face, and the follow-up point is always a cheek point. In addition, the tracking point and the reference point may be specified by an arbitrary method.

  The measuring unit 14 measures pre-correction motion information on the motion of the following point included in the moving image acquired by the moving image acquiring unit 11. In the present embodiment, the movement speed is measured in a time series for the tracking point specified by the tracking point specifying unit 12. In the present embodiment, the movement speed of the reference point identified by the reference point identification unit 13 is similarly measured in time series, and the difference between the timing of the movement of the reference point and the timing of the movement of the following point is determined based on the measurement. Measure.

  Here, in the present embodiment, the point of the forehead is specified as a reference point that is a reference of the movement of the facial expression, and the relative movement speed with respect to the point of the forehead is measured so as not to be affected by the movement of the entire head. . The reference point is preferably a point that moves with the movement of the head and is not affected by the movement of the facial expression (does not move with the expression of the facial expression). The specification of the reference point is not essential, and the absolute movement speeds of the reference point and the following point may be simply measured.

  The unit facial expression setting unit 15 sets a unit facial expression in the moving image acquired by the moving image acquiring unit 11. In the present embodiment, the motion of the face is analyzed, and the motion of the face from the start to the end of the facial expression in each expression is set as a unit expression.

  The correction unit 16 generates a corrected moving image in which the motion of the following point in the moving image is corrected based on the target motion information stored in the storage unit 18 described below. In the present embodiment, the correction is performed for each unit expression set by the unit expression setting unit 15.

  The display unit 17 performs display processing of the corrected moving image generated by the correction unit, and transmits a processing result. In the present embodiment, the corrected moving image generated from the moving image captured by the display device 2a is subjected to display processing by the display unit 17, and the processing result is transmitted to the display device 2b on the other side of the call.

  The storage unit 18 stores target motion information, which is a measure of the motion of a follow-up point for correcting a moving image. FIG. 5 is a diagram illustrating an example of the target exercise information. As shown here, in the present embodiment, information indicating the movement characteristics of each follow-up point for each age is stored. Here, the storage means 18 stores, by age, a measure of the magnitude of the delay in the timing of the movement of each follow-up point with respect to the timing of the movement of the reference point, and uses this as target movement information.

  That is, when the movement of the face is to be corrected in the twenties, information indicating the motion characteristics of each of the following points in the twenties can be used as the target movement information. The information indicating the motion characteristics of the target can be used as the target motion information. In the case of this example, for example, the estimated delay at the tracking point C in the 20's is stored as 30 ms, and the estimated delay at the tracking point C in the 50's is 100 ms.

  In addition, here, a configuration is shown in which a standard of the magnitude of the delay at each tracking point is stored as a numerical value for each age, and this is used as target motion information. It is good also as composition which stores a plurality of functions showing the size of each for every age.

  Further, in the present embodiment, the difference between the reference point and the follow-up point (the delay of the follow-up point with respect to the reference point) of the timing at which the movement speed is maximized in each unit expression is defined as “the magnitude of the delay”. The delay may be expressed in any manner. For example, the magnitude of the delay may be expressed by the difference between the timing (the start or end of movement) at which the acceleration of each of the reference point and the following point in each unit expression is maximized.

  In the case where various motion characteristics other than the followability are corrected, similarly, a value serving as a guide for each age can be stored and used as target motion information. Also in this case, the motion information may be stored as a function indicating the motion characteristics.

  The display devices 2a and 2b include an input unit 21 and an output unit 22, respectively. As the display device 2, a general computer such as a PC (Personal Computer) or a smartphone can be used. The display devices 2a and 2b store a dedicated application that functions as the display device 2 in the skin moving image correction system according to the present embodiment and a web browser application that uses a web page.

  As the input unit 21, a photographing device such as a general camera capable of capturing at least a moving image can be employed, and as the output unit 22, a general display device capable of displaying at least a corrected moving image is employed. be able to. In the present embodiment, a sound collecting device such as a microphone capable of acquiring sound is further provided as the input means 21, and a video call is performed by transmitting the acquired sound to the display device 2 of the other party together with the corrected moving image.

  FIG. 6 is a flowchart illustrating a flow of processing from acquisition of a moving image to generation of a corrected moving image in the present embodiment. When shooting and transmission of a moving image are started via the input unit 21, the processing in FIG. 6 is started, and the moving image obtaining unit 11 obtains a moving image in step S11. Here, in parallel with the shooting of the moving image by the input unit 21, the shot moving images are sequentially transmitted to the server device 1, and the transmitted moving images are sequentially obtained by the moving image obtaining unit 11.

  Next, in step S12, the unit facial expression setting unit 15 sets a unit facial expression for the moving image acquired in step S11. In this embodiment, the movement of the face is analyzed, and the time from when the change of the expression starts to when the change of the expression ends is set as one unit expression. The setting of the unit expression may be performed by any method.

  More specifically, for example, models of facial movements at the time of expressing various facial expressions are stored in the storage unit 18 or an external database or the like, and the obtained moving images are analyzed with reference to the facial movement models. Thus, a method of setting a unit expression can be considered. Alternatively, before step S12, the tracking point or the reference point may be specified and the motion information may be measured, and the unit expression may be set based on the measurement result.

  When the setting of the unit expression is completed, the process proceeds to step S13, and the correcting unit 16 generates a corrected moving image for each unit expression. FIG. 7 is a flowchart illustrating an example of processing related to generation of a corrected moving image according to the present embodiment.

  First, in step S21, the tracking point specifying means 12 and the reference point specifying means 13 specify a reference point and a tracking point on a human face included in the moving image. As described above, in the present embodiment, three points on the cheek are set as tracking points A to C in advance with the points on the jaw as the reference points, and these points are detected from the moving image to thereby determine the reference points and the tracking points. Although the points are specified, a configuration may be employed in which the following points are specified based on the movement of the face in the moving image.

  When the specification of the reference point and the following point is completed, in steps S22 to S24, the measuring means 14 measures the pre-correction motion information. In the present embodiment, first, in step S22, the movement speed of each point is measured from the movement amount of the reference point between the frames constituting the moving image and each of the following points.

  In the following step S23, the timing of movement of the reference point and each of the following points in the target unit expression is specified based on the movement speed of the reference point and each of the following points measured in step S22. In the present embodiment, the timing at which the movement speed of the reference point and each of the following points becomes maximum is specified.

  Then, in step S24, the magnitude of the delay in the movement of each tracking point with respect to the reference point is calculated from the movement timing of the reference point and each tracking point specified in step S23. In the present embodiment, the calculated magnitude of the movement delay of each following point is used as pre-correction motion information for generating a corrected moving image.

  In step S25, the correction means 16 adjusts the timing of the movement of each following point. Here, in the present embodiment, the magnitude of the motion delay (the motion information before correction) of each tracking point calculated in step S <b> 24 indicates the motion characteristic of each tracking point stored in the storage unit 18 for each age. In comparison with the information, the movement information before correction is specified which movement characteristic is closer to the movement.

  Then, the correction means 16 determines the corrected motion timing at each of the following points by using, as target motion information, information indicating the motion characteristics of each age in the age one stage younger than the age specified from the pre-correction motion information. I do. However, the selection of the target exercise information is not limited to this. For example, the target exercise information may be selected such that the designation of the correction strength is received via the display device 2 and the correction is performed to a degree based on the correction strength. .

  Here, in the present embodiment, the target motion information is defined such that the delay of the motion timing is smaller as the follow-up point is closer to the reference point, and a corrected moving image is generated based on the target motion information. As described above, by adjusting the timing of the movement of each tracking point according to the distance of each tracking point to the reference point, a more natural corrected moving image can be generated.

  When the adjustment of the timing of the movement of the following point in step S25 is completed, in step S26, the correction means 16 determines the time-series movement speed of each following point based on the adjustment. In the present embodiment, the timing at which the maximum speed is reached is adjusted in step S25, and the time-series motion speed is determined by linear interpolation in step S26. In addition, you may change so that it may adjust by acceleration.

  In step S27, the image of each frame constituting the moving image is adjusted based on the moving speed of each following point, and a corrected moving image is generated. For adjusting the image of each frame, a general image editing technique may be used.For example, a mesh-like area is set in the image, and each area is stretched or shrunk in accordance with the movement of each following point to thereby adjust each area. You can adjust the image of the frame. At this time, it is preferable to adjust each frame in consideration of continuity with the preceding and following frames.

  When the generation of the corrected moving image for the unit expression is completed in this way, the process returns to the flowchart of FIG. 6, and the display unit 17 displays the corrected moving image and transmits the processing result to the display device 2 in step S14. Here, in the present embodiment, when the corrected moving image generated from the moving image captured by the display device 2a is subjected to display processing by the display unit 17, the processing result is transmitted to the display device 2b of the other party, It is displayed to the other party via the output means 22 of the display device 2b. The corrected moving image may be displayed on the display device 2a in the same manner.

  In this manner, until a shooting end instruction is received via the display device 2 in step S15, sequentially input moving images are obtained, corrected for each unit expression, and a corrected moving image is generated. It is displayed continuously in the device 2. As described above, by sequentially performing the processing, the corrected moving image can be displayed almost in real time.

  Note that the processing flow shown in FIGS. 6 and 7 is an example, and the order and details of the processing contents may be arbitrarily changed. Also, in this example, after the moving image is acquired in step S11, the unit expression is set in step S12, and the flow of generating the corrected moving image for each unit expression has been described. However, the corrected moving image is not necessarily generated for the entire range of the moving image. You may. For example, it is possible to adopt a configuration in which the unit expression is not included in a portion where the expression does not change, and the generation and display processing of the corrected moving image in steps S13 and S14 are skipped.

<Embodiment 2>
Hereinafter, another embodiment of the present invention will be described. Note that the same components as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  In the present embodiment, a configuration in which a moving image is obtained via the display device 2 and the corrected moving image is stored in the storage unit 18 will be described. In the present embodiment, it is not necessary to generate a corrected moving image at the same time as shooting, and a configuration is shown in which processing is performed after shooting a moving image to generate a corrected moving image.

  FIG. 8 is a flowchart illustrating a flow of processing from acquisition of a moving image to generation of a corrected moving image in the present embodiment. First, in step S31, the moving image acquisition unit 11 acquires a moving image via the display device 2. The acquisition of the moving image here does not need to be performed at the same time as the photographing. For example, the moving image data stored in the storage device included in the display device 2 may be acquired.

  Next, in step S32, the unit facial expression setting unit 15 sets a unit facial expression for the obtained moving image. Here, the unit expression may be set in the same manner as in the first embodiment. Alternatively, an input regarding the start point and the end point of the unit expression may be received via the display device 2 and set based on the input. Alternatively, after setting a unit expression in the same manner as in the first embodiment, an input for adjusting the range of the unit expression may be received.

  In step S33, a corrected moving image is generated as in the first embodiment. That is, a corrected moving image is generated for each unit expression based on the exercise information. Here, for example, an input such as "correct the motion so as to look one step younger" or "correct the motion so as to be in the twenties" is received, and the target motion information is selected based on the input. Is also good. Conversely, it is also possible to accept an input such as "correct so that it looks one step older".

  When the generation of the corrected moving image for each unit expression for the entire moving image is completed, in step S34, the corrected moving images are combined to generate the entire corrected moving image. Here, the movement of the following point may be adjusted so that the connection between the corrected moving images for each unit expression becomes natural.

  Then, in step S35, the generated corrected moving image is stored in the storage unit 18, and the process ends. The display unit 17 performs display processing of the corrected moving image stored in the storage unit 18 according to a display request of the display device 2, and transmits a processing result. Further, the corrected moving image may be downloaded to the display device 2 and can be checked.

  As described above, according to the skin moving image correction system of the present invention, it is possible to generate a corrected moving image with an improved impression by correcting the movement of the skin when expressing a facial expression. Further, by determining the target motion information according to the pre-correction motion information as in the present embodiment, an effect that natural correction can be performed on the face motion can be expected.

  In the present embodiment, a configuration in which the reference point and the following point are specified as the feature points, and the difference in the timing of the movement is used as the movement information to generate the corrected moving image, thereby correcting the following property has been described. However, the present invention is not limited to this. In addition, in order to correct the above-described elasticity and deformability, a configuration may be used in which information such as a distance between feature points or a change in the area of a region defined by a plurality of feature points is used as motion information. In addition, a configuration in which these are combined to perform correction on a composite motion characteristic may be adopted.

DESCRIPTION OF SYMBOLS 1 Server apparatus 11 Moving image acquisition means 12 Tracking point specification means 13 Reference point specification means 14 Measurement means 15 Unit expression setting means 16 Correction means 17 Display means 18 Storage means 2 Display device 21 Input means 22 Output means R Reference point

Claims (13)

A skin moving image correction system that corrects a moving image obtained by photographing a human face, based on motion information on motion of a feature point on the human face with respect to expression of facial expression,
The exercise information includes target exercise information to be a correction target,
Moving image acquisition means for acquiring a moving image of a human face,
Feature point specifying means for specifying one or more feature points on the face in the moving image;
Correction means for changing the movement of the feature point in the moving image based on the target movement information to generate a corrected moving image in which the movement of the face is corrected. The motion information includes motion information before correction related to the motion of the feature point in the moving image,
From the moving image, measuring means for measuring the pre-correction motion information,
Storage means for storing a plurality of the target exercise information,
The correction means determines the target motion information used for generating the corrected moving image from a plurality of the target motion information based on the pre-correction motion information measured by the measurement means. The skin moving image correction system according to claim 1. The moving image further includes a unit expression setting unit that sets a portion from a certain expression to another expression as a unit expression to be corrected,
The skin moving image correction system according to claim 1, wherein the correction unit generates the corrected moving image by correcting a face movement for each unit expression.   The skin moving image correction system according to claim 1, wherein the movement information includes at least one of a movement speed and an acceleration of the feature point.   The skin moving image correction system according to claim 1, wherein the feature point includes a cheek point. The correction unit, in parallel with the acquisition of the moving image by the moving image acquisition unit, sequentially correcting the acquired moving image to generate the corrected moving image,
The display device according to claim 1, further comprising a display unit configured to sequentially display the generated corrected moving image and transmit a processing result in parallel with the generation of the corrected moving image by the correction unit. The skin video correction system described in 1. The feature point is a reference point on the face that serves as a reference for the movement of the face when expressing an expression, and a point on the skin that moves following the movement of other parts on the face when expressing the expression. A tracking point, and
As the characteristic point identifying means, a tracking point identifying means for identifying the tracking point, and a reference point identifying means for identifying the reference point,
The skin moving image correction system according to claim 1, wherein the correction unit changes the movement of the follow-up point to generate the corrected moving image.   The skin moving image correction system according to claim 7, wherein the movement information includes information on a difference between the movement timing of the reference point and the movement timing of the following point.   The correction means generates the corrected moving image by changing the movement of the tracking point so that a difference between the timing of the movement of the reference point and the timing of the movement of the tracking point is reduced. The skin moving image correction system according to claim 7 or 8, wherein   The correction means generates the corrected moving image in which the difference between the timing of the movement of the reference point and the timing of the movement of the tracking point is reduced by advancing the timing of the movement of the tracking point. The skin moving image correction system according to claim 9, wherein A plurality of the following points are defined,
The correction means reduces the difference between the timing of the movement of the reference point and the timing of the movement of the tracking point as the distance between each of the tracking points and the reference point is smaller. The skin moving image correction system according to any one of claims 7 to 10, wherein the correction is performed to generate the corrected moving image. A skin moving image correction program for performing correction of a moving image obtained by photographing a human face, based on motion information about a motion of a feature point on the human face with respect to expression of facial expression,
The exercise information includes target exercise information to be a correction target,
A computer, a moving image acquisition unit that acquires a moving image of a human face,
Feature point specifying means for specifying one or more feature points on the face in the moving image;
A skin moving image correction program that functions as a correction unit that changes a motion of the feature point in the moving image based on the target motion information to generate a corrected moving image in which a face motion is corrected. A skin moving image correction method for correcting a moving image obtained by photographing a human face, based on motion information on motion of a feature point on the human face for expression of an expression,
The exercise information includes target exercise information to be a correction target,
A moving image acquiring step of acquiring a moving image of a human face,
A feature point specifying step of specifying one or a plurality of the feature points on the face in the moving image;
Correcting the movement of the feature point in the moving image based on the target movement information to generate a corrected moving image in which the motion of the face is corrected.