JP7009997B2 - Video generation system and video display system - Google Patents

Description

The present application relates to a moving image generation system and a moving image display system. The present application also relates to a data structure used in a moving image generation system and a robot system including a moving image generation system.

An "emboded conversational agent" using a computer has been developed. The anthropomorphic dialogue agent displays a human or a human-like character on the screen of a display device (display) and interacts with the user in response to the user's voice or keyboard input. A moving image is displayed on the screen of the display, and the facial expression (especially the movement of the mouth) and the posture of the subject image change according to the content of the sound emitted by the subject such as a human being. The video is generated by the video generation system.

The main video generation systems that display realistic humans on the screen of a display are classified into two types. In the first moving image generation system, a lip sync animation is synthesized from a two-dimensional (2D) photograph or moving image. For the generation of such a composite moving image, a method using morphing of a still image (Non-Patent Document 1) or a method of preparing a large number of images of the mouth and displaying them in an optimum order (Non-Patent Document 2) can be used. In the second moving image generation system, a human three-dimensional (3D) model and its utterance animation are created and drawn. Studies on modeling and animation of human faces necessary for this drawing have been conducted (Non-Patent Document 3). For realistic drawing, it is necessary to combine various techniques such as hair modeling, animation, rendering, and skin rendering.

U.S. Pat. No. 6,636,220 Japanese Unexamined Patent Publication No. 2006-323774

T. Ezzat and T. Poggio, "Visual speech synthesis by morphing visemes", International Journal of Computer Vision, 38 (1): 45-57, 2000. L. Wang, X. Qian, W. Han, and F. K. Soong, "Synthesizing photo-real talking head via trajectory-guided sample selection", In INTER-SPEECH, Vol. 10, pp. 446-449, 2010. Z. Deng and J. Noh, "Computer facial animation: A survey", In Data-driven 3D facial animation, pp. 1-28. Springer, 2008. A. Schodl, R. Szeliski, DH Salesin, and I. Essa, "Video Textures", In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '00, pp. 489-498, New York, NY, USA, 2000. ACM Press / Addison-Wesley Publishing Co.

Among the above-mentioned conventional techniques, according to the first method, since the mouth only moves according to the utterance content, it is far from a realistic change in human facial expression. In order to generate a more natural animation, it needs to be combined with a 3D model. When the second method is used, it is difficult to draw a realistic dialogue scene by combining various techniques such as hair modeling, animation, and rendering.

The embodiments of the present disclosure provide a moving image generation system and a moving image display system capable of displaying a realistic change in a human facial expression or posture.

An exemplary moving image generation system of the present disclosure includes a recording device in which a moving image clip, which is a sequence of a plurality of frames including a subject image, is recorded, and each frame included in a frame group selected from the plurality of frames and others. The computer is provided with a computer that reconstructs the plurality of frames based on the similarity between the frames and generates the data of the composite moving image, and the similarity is the feature amount of the subject image in each frame. Based on the above, the computer has a similarity with the Nth frame as the N + 1th frame, which is the next frame of the Nth frame (N is a positive integer) of the synthetic moving image. Based on this, one frame is determined from a plurality of candidate frames selected from the frame group.

According to the embodiment of the present disclosure, it is possible to generate a moving image in which the frame sequence of the moving image clip obtained by shooting is reconstructed based on the similarity of the subject images.

FIG. 1 is a diagram showing an example of a basic configuration of a moving image generation system according to the present disclosure. FIG. 2 is a diagram simulating a configuration example of a table that defines a relationship between a certain frame and "a plurality of candidate frames" associated with the frame. FIG. 3 is a diagram showing a configuration example of a non-limiting exemplary embodiment of the moving image generation system according to the present disclosure. FIG. 4 is a diagram showing a configuration example of a moving image clip in which a standby section and an utterance section are alternately repeated. FIG. 5 is a diagram showing a plurality of feature points of the subject image automatically extracted by the photographing apparatus. FIG. 6 is a diagram schematically showing the feature points of the subject image in the i-th frame i and the j-th frame j in the moving image clip. FIG. 7 is a diagram schematically showing an example of a matrix that defines the distances D _i and _j between the frames i and the frame j obtained by the pre-calculation. FIG. 8 is a diagram schematically showing an example of a matrix that defines the transition costs C _i and _j between the frames i and the frame j obtained by the pre-calculation. FIG. 9 is a diagram showing an example of inter-frame transition when cross-dissolve processing is used for frame interpolation. FIG. 10 is a diagram schematically showing an example in which a predetermined frame is excluded from the candidate frames. FIG. 11 is a diagram showing a modified example of the moving image display system. FIG. 12 is a diagram schematically showing an exemplary embodiment of the robot system according to the present disclosure. FIG. 13 is a flowchart showing an example of the processing procedure to be performed in advance. FIG. 14 is a flowchart showing an example of a moving image generation and moving image display processing procedure. FIG. 15 is a diagram showing an example of the internal configuration of the computer 20.

<Findings of the Inventor of the present Application> As a technique capable of generating a realistic moving image of a human being, the technique of "Video Textures" is disclosed in Non-Patent Document 4 and Patent Document 1. "Video Textures" is a technology that can acquire a moving image clip for a finite time, which normally has a start and an end, by shooting, and continuously play back a moving image synthesized from the moving image clip. With this technique, instead of simply connecting the end and start points of a video clip and looping infinitely, the probability of transition from one frame to the next in the displayed video is based on the similarity between the frames. Do it. Therefore, it is possible to generate a frame sequence without simple repetition.

In "Video Textures", a plurality of frames similar to each other are selected in advance as candidate frames from a large number of frames constituting the moving image clip acquired by shooting. When rearranging frames and synthesizing moving images, the "transition" from the current frame to the next frame is performed according to a probability distribution according to the degree of similarity. More specifically, one frame is selected from a plurality of candidate frames similar to the current frame according to the probability distribution according to the degree of similarity. This probability distribution is defined by a function whose variable is the similarity between frames, and transitions to frames with high similarity occur frequently.

In "Video Textures", the similarity between frames is given by the distance (frame-to-frame distance) between the two frames of interest. This "distance" is defined by adding the difference (absolute value or square) of the pixel values in the two frames to all the pixels constituting the frame. The smaller the "distance" defined in this way, the higher the similarity between frames.

According to the study by the inventors of the present application, when trying to realize a video generation system using the technology of "Video Textures", there is a problem that the amount of calculation of the similarity between frames is enormous, and between frames with high similarity. It turns out that there is a problem that the transition of is repeated unnaturally.

The inventor of the present application synthesizes a natural moving image that can be used for dialogue with a smaller amount of calculation by using the similarity defined based on the feature amount of the subject image in the frame instead of the similarity of the entire frame. I found what I could do. We also found that by limiting (not adopting) a part of the range of candidate frames selected based on the degree of similarity according to the past display situation, it is possible to suppress unnatural repetition of the sequence in the synthetic video. In addition, Patent Document 2 discloses that the positions of a plurality of feature points arranged on facial parts are detected. The apparatus of Patent Document 2 associates feature positions between captured facial images, and analyzes how each feature position moves and how the image pattern changes with vocalization.

In the present disclosure, the degree of similarity is defined based on the "feature amount of the subject image" in the frame. A typical example of a subject is a human face, which may include part or all of the human body. Further, the subject may be an animal or a robot that can be anthropomorphized. In a preferred embodiment, the subject image includes images of faces, especially eyes and mouth, that define facial expressions such as humans. Such subject images are identified by various features that can be used in the field of pattern recognition. In certain embodiments, this feature quantity can be defined based on facial expressions and orientations, i.e., a plurality of points (feature points) extracted to represent the appearance of the face. For example, the position coordinates themselves in the frame of the feature points of the subject image can be the feature quantity. In this case, in the two frames, the sum of the differences in the position coordinates of the corresponding feature points corresponds to the distance for defining the degree of similarity. The position of the feature point may be three-dimensional coordinates in the space where the subject is located. The pixel value at each feature point of the subject image, for example, chromaticity or lightness may be included in the feature amount.

As described above, in the present disclosure, since the similarity between frames is defined based on the feature amount of the subject image included in the frame, not the entire frame, the similarity between frames can be obtained with a smaller amount of calculation. ..

Further, according to another aspect of the present disclosure, some frames are excluded from the plurality of candidate frames selected based on the similarity according to the order of the frames constituting the composite moving image. By excluding some frames, it is possible to suppress the repeated use of the same frames with high similarity in a synthetic video within a short period of time. For example, when there is a frame ( _KL is an integer of 1 or more) contained in a predetermined number of _TL frames ( _TL is an integer of 2 or more) before the Nth frame by a predetermined number of _KL times or more. Claims can be selected as "some frames" above and excluded from candidate frames.

<Explanation of the principle of video generation related to this disclosure>
Hereinafter, prior to the description of the embodiment, the basic principle of the moving image generation system according to the present disclosure will be described.

First, refer to FIG. FIG. 1 shows an example of the basic configuration of the moving image generation system according to the present disclosure. The moving image generation system 100 shown in the figure includes a recording device 10 and a computer 20.

The recording device 10 records a moving image clip acquired by shooting in advance. A moving image clip is a sequence of multiple frames containing an image of a subject, such as a front-facing person.

The computer 20 reconstructs the frames based on the similarity between the frames included in the frame group of the moving image clip to generate the data of the synthesized moving image. As described above, this "similarity" is defined based on the feature amount of the subject image in each frame.

The composite moving image generated by the moving image generation system 100 is a sequence of a plurality of frames whose display order is rearranged. That is, the composite moving image is composed of frames in which the order of the frames in the moving image clip acquired by shooting is rearranged in a new order at least in part. When synthesizing a moving image, the computer 20 selects the N + 1st frame, which is the next frame of the Nth frame (N is a positive integer) in the synthesized moving image, from "a plurality of candidate frames". "Multiple candidate frames" are selected from the frame group of the moving image clip based on the similarity.

From the plurality of candidate frames selected in this way, the computer 10 determines one frame (for example, the frame 12 including the subject image 14) as the N + 1th frame in the synthesized moving image. There can be various ways to determine one frame from multiple candidate frames. If you try to select the frame with the highest similarity, a specific frame will always be selected from the candidate frames. In a preferred embodiment, frames are stochastically selected based on a probability distribution function with similarity as a parameter.

In the composite moving image generated in this way, the change in the feature amount of the subject image becomes small between consecutive frames. For example, the moving images are synthesized so that the positions of the face, eyes and mouth in the frame and their sizes change naturally. Moreover, since the destination frame transitioning from the same frame is not fixed to one and changes stochastically, the monotonousness of the facial expression or movement of the subject is reduced even if the moving image is displayed for a long time. obtain.

"Multiple candidate frames" are determined in advance for each frame. Depending on the content of each frame (appearance of the subject image), the configuration of the "plurality of candidate frames" associated with that frame may differ. In a preferred embodiment, the relationship between a frame and the "plurality of candidate frames" associated with that frame can be represented, for example, by a table.

FIG. 2 is a diagram simulating an example of the configuration of such a table. This table has a configuration capable of storing a value corresponding to the similarity (for example, "distance" between frames) for 12 frames selected from the moving image clip. In reality, a table can be prepared for a huge number of frames exceeding tens of thousands. The twelve frames are typically twelve consecutive frames acquired by photography. However, the order of the frames does not necessarily have to match the order acquired at the time of shooting.

Here, i and j are integers of 1 to 12. In the i-by-j column of the table, the numerical value corresponding to the similarity between the frame i and the frame j is stored. It is assumed that the numerical value in this example indicates how much the coordinates of the feature points on the face of the person who is the subject are shifted for each frame. In the table of FIG. 2, for the sake of simplicity, an example of a numerical value indicating the degree of similarity is described only in the row of i = 3 in the table. The smaller this value, the higher the similarity of the subject images. That is, the smaller this value is, the smaller the change in the position of the feature point on the face of the person who is the subject is small even if the frame changes.

The frame j having the highest similarity to the frame i of i = 3 is a frame of j = 4 excluding the frame of j = 3, which is the same frame. For example, frames j with j = 4, 5, and 11 are selected as the three candidate frames having a relatively high degree of similarity. For such candidate frames, if the similarity between the frames i of i = 1 to 12 and the frames j of j = 1 to 12 is obtained in advance, the similarities are obtained in advance for each of the frames i of i = 1 to 12. It is possible to select.

The operation of determining one frame from a plurality of candidate frames is executed by a computer when generating a composite moving image. As mentioned above, in the preferred embodiment, the operation of this determination is stochastical. For example, when the frame i of i = 3 is the Nth frame of the composite video, one frame is selected from a plurality of candidate frames, that is, the frames j of j = 4, 5, and 11 for the N + 1th frame. obtain. At this time, if the frame with the highest degree of similarity of the subject image (the smallest change in the position of the feature point) is always selected, the frame i with i = 3 always transitions to the frame j with j = 4. As a result, the variety of changes in the subject image in the composite video is lost. In order to secure such diversity, it is preferable that a frame is stochastically selected from the candidate frames composed of the frames j of j = 4, 5, and 11. However, if the candidate frame is composed of frames with relatively high similarity, frames may be randomly selected from such candidate frames regardless of the similarity, or some other rule. The frame may be selected according to.

A typical example of a subject image in a moving image is a human being, but it may be an anthropomorphic animal or a robot. "Reconstruction" is not only a mode in which a plurality of frames actually shot are used as they are to generate a composite video, but also a frame other than the frames acquired by shooting is additionally used to generate a composite video. Aspects are also included. A typical example of a mode in which a plurality of frames actually shot are used as they are to generate a composite moving image is to generate a composite moving image by rearranging the display order of the frames acquired at the time of shooting. An example of "a frame other than a frame acquired by shooting" may be a frame generated by morphing, for example, from a frame acquired at the time of shooting.

<Embodiment>
Hereinafter, embodiments according to the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art. The inventors of the present application provide the accompanying drawings and the following description for those skilled in the art to fully understand the present disclosure. These are not intended to limit the subject matter described in the claims.

1. 1. Basic configuration First, refer to FIG. FIG. 3 shows a configuration example of a non-limiting exemplary embodiment of the moving image generation system according to the present disclosure.

The illustrated moving image generation system 100 includes a recording device 10 in which a moving image clip is recorded, and a computer 20 that reconstructs a plurality of frames constituting the moving image clip to generate synthetic moving image data.

The moving image generation system 100 displays a synthetic moving image on the screen of a display 50 such as a liquid crystal display or an organic EL display. The display 50 may be a projector that projects a moving image on the screen. The display 50 may be connected to the moving image generation system 100 by wire or wirelessly at all times or as needed. One or both of the recording device 10 and the computer 20 in the moving image generation system 100 may be provided at a position away from the display 50. The display 50 may be a part of the moving image generation system 100 or may not be included in the moving image generation system 100. The display 50 in this embodiment has a built-in speaker (not shown) that outputs sound.

The computer 20 may have a known configuration and may typically be a commercially available general purpose computer with a built-in central processing unit (CPU) and memory. The memory contains a program including a plurality of instructions that define the order of processing by the computer 20. During operation, the computer 20 operates according to the instructions of this program.

As described above, the computer 20 determines one frame from a plurality of candidate frames as the N + 1th frame, which is the next frame of the Nth frame (N is a positive integer) in the synthetic moving image. In a preferred embodiment, this "decision" can be continuously performed in a mode in which a composite moving image including a standby subject image is displayed on the display 50. When the display frame transitions from one frame (Nth frame) of the composite video to the next frame (N + 1st frame), the transition occurs between two frames whose similarity satisfies a predetermined relationship. Occurs. As a result, the subject image in the composite moving image can show a natural movement from the user's point of view.

FIG. 3 shows a moving image display system 200 including a moving image generation system 100. The moving image display system 200 includes an interface device 60 such as a microphone or a keyboard, and a dialogue engine 70, in addition to the moving image generation system 100. The dialogue engine 70 obtains some input from the user via the interface device 60, and causes the computer 20 to perform a dialogue in response to the input, for example, an appropriate reply to the user. For example, when the user utters "hello", the user's voice is converted into a digital signal by the interface device 60 and input to the dialogue engine 70. The dialogue engine 70 discriminates the content of the words uttered by the user by voice recognition, and makes a response according to the discriminant result. Specifically, the computer 20 reads an appropriate portion of the moving image clip from the recording device 10 and reproduces it in response to the response instruction by the dialogue engine 70.

The dialogue engine 70 may be realized by a program installed in the computer 20. The dialogue engine 70 may be located away from the display 50. The interface device 60 is typically placed near the display 50. Alternatively, the interface device 60 may be integrated with the display 50. In some embodiments, the moving image display system 200 includes a housing that houses a recording device 10, a computer 20, a display 50, and an interface device 60.

The moving image clip recorded on the recording device 10 includes a moving image portion for reply, as will be described later. As a result, when, for example, a voice of "hello" is emitted from the speaker of the display 50, a moving image of a human being saying "hello" in accordance with the voice is displayed on the screen of the display 50. That is, a moving image showing a change in the appearance of the face corresponding to the voices of "ko", "n", "ni", "chi", and "wa" is displayed in accordance with the voice. Such a moving image can be most easily generated by playing a series of frames that say "hello", which were previously acquired by shooting, according to the sequence at the time of shooting.

Unlike the above-mentioned prior art, the moving image generation system according to the present embodiment does not generate an utterance animation from an input character string or voice. The moving image generation system according to the present embodiment is suitably used for a dialogue system that selects an appropriate response from predetermined dialogues, such as a simple visitor response and reservation reception system.

During the period when there is no input from the user, the subject image in the standby state is projected on the display 50. Such a period is called a "waiting period". Further, the operating state of the moving image generation system 100 in the "standby period" is called a "standby mode". On the other hand, the period during which the voice and the moving image for responding to the user are played is called the "utterance period". Further, the operating state of the moving image generation system 100 in the "utterance period" is called "utterance mode".

According to the embodiment of the present disclosure, the subject image displayed on the display 50 during the standby period is not a still image but a moving image synthesized by the computer 50. As described with reference to FIG. 1, this composite moving image is a new sheathkens of frames generated by the computer 50 reconstructing the frame group of the moving image clip. Therefore, the order of the frames constituting the moving image displayed in the waiting period is determined by the probabilistic inter-frame transition as described above. According to the moving image generation system of the present disclosure, the data of the synthetic moving image during the waiting period and the data of the reproduced moving image during the utterance period are alternately sent to the display 50. The length of the waiting period varies depending on the timing of input by the user, but the length of the utterance period is defined by the length of the moving image portion selected according to the content of the utterance. The "moving image clip" in the present embodiment includes a moving image portion of the subject image that emits a predetermined line and a moving image portion of the subject image that is in a state before and after issuing such a line.

As described above, the computer 20 in the present embodiment is programmed to generate data of the synthetic moving image and select a partial clip that is a part of the moving image clip according to the input of the user. At the direction of the computer 20, the display 50 displays the generated synthetic moving image or the selected partial clip.

2. 2. Acquisition of Video Clips The recording system used to acquire video clips includes, for example, one computer and one video recording device. The moving image shooting device used in this embodiment is Kinect (registered trademark) manufactured by Microsoft (registered trademark). A moving image shooting device such as a normal camera that does not acquire depth information may be used.

Have the person (subject) who will be the model of the dialogue agent read out the lines prepared in advance and perform them, and record the situation with the camera. Make an appropriate record of when you started reading the lines. Recording can be performed by two people, a model actor and a person (operator) who operates a computer. Before recording, the operator prepares a line to be performed by the actor, and registers the text and voice in the system.

In the present embodiment, immediately after the start of recording, the actor waits without doing anything in order to record the standby state (dialogue section). When the operator selects a line, the actor speaks the selected line (speech section). For example, when the line "hello" is selected, the actor says "hello". The operator inputs to the system when the actor has finished playing the line. After that, recording in the standby state is continued for a while, and when the operator selects a line again, recording of the next line starts.

By repeating the above procedure, as illustrated in FIG. 4, a moving image clip in which the waiting section and the utterance section are alternately repeated is obtained. As a result, the number of candidate frames for the transition position before and after the utterance state increases, so that it becomes easier to realize a more natural frame transition when generating a moving image. The same line may be recorded multiple times. When a plurality of videos of utterance sections that utter the same line are acquired, it is possible to avoid repeatedly playing the exact same video even when the same line is repeatedly uttered in response to user input at the time of video generation. Also, as the number of uttered videos in which the same line is uttered increases, the number of frames that can be selected when transitioning from various frames in the standby state to the first frame in each uttered state increases, thus realizing a natural inter-frame transition. It will be easier to do.

It is not always necessary to make all recordings in succession. If you record continuously, you can minimize small changes in appearance such as hairstyle and clothes. The information stored by the recording system is voice, camera RGB data, face recognition result, posture recognition result, and timing of start and end of utterance input by the operator.

As shown in FIG. 5, according to the present embodiment, the photographing device 80 can automatically extract a plurality of feature points of a subject image and acquire face orientation and posture information. In FIG. 5, some of the positions f _{i, k} (1 ≦ k ≦ F) of the facial feature points of the subject image in the i-th frame i in the moving image clip are described. In addition, some of the positions s _{i, k} (1 ≦ k ≦ S) of the posture feature points of the subject image are described. Here, "F" and "S" are the maximum value of the number of facial feature points and the maximum value of the number of posture feature points used for processing, respectively.

When the orientation, facial expression, and posture of the subject's face change, the positions f _{i, k} , s _{i, k} of these feature points change. Even for the same corresponding feature points, their positions can vary from frame to frame. In the present embodiment, when a moving image clip is acquired by shooting a subject, a predetermined number of feature point positions are acquired for each frame.

In the example shown in FIG. 5, the position of the human head as a subject is indicated by the intersection of the alternate long and short dash line 15X extending in the horizontal direction (X-axis direction) and the alternate long and short dash line 15Y extending in the vertical direction (Y-axis direction). ing. Further, the intersection of each of the one broken line 16X extending in the horizontal direction and the two broken lines 16Y extending in the vertical direction indicates the position indicating the posture of the human being as the subject. According to the photographing device 80, it is possible to display the difference in posture from the time when recording is started on the display of the recording system. If the difference is too large during shooting, the operator can interrupt the recording or use it to determine the posture when resuming when there is a break during shooting.

3. 3. Pre-calculation Next, pre-calculation is performed to determine a candidate frame for the frame transition to be performed when the composite video is generated.

In the present embodiment, as a preliminary calculation, the transition cost between frames is calculated for any frame pair in the recording clip, and the cost matrix is obtained. The transition cost is a numerical value of the unnaturalness when transitioning from one frame in the recording clip to another frame. This calculation can follow the method disclosed in Non-Patent Document 4. The entire contents of Non-Patent Document 4 are incorporated herein by reference. However, the similarity between frames in the present embodiment is performed by using the feature points of face recognition, not the comparison for each pixel.

Using the obtained cost matrix, a plurality of frames that realize low-cost transitions are extracted for each of the transition from the standby state to the standby state and the transition from the standby state to the utterance state. A plurality of frames that are transition destinations with low cost are candidate frames. It also performs a process of cutting out the sound recorded in parallel with the shooting of the candidate frame, and a process of converting the RGB data of each frame for reproduction.

3.1. Similarity between Frames FIG. 6 schematically shows the feature points of the subject image in the i-th frame i and the j-th frame j in the moving image clip. Here, 1 ≦ i <j. In frame i of FIG. 6, arrows indicating the positions f _{i, 15} and s _{i, 2} of the two feature points are shown schematically. It is assumed that the positions f _{i and 15} are the 15th facial feature points, and the s _{i and 2} are the second posture features. Further, in the j-th frame of FIG. 6, arrows indicating the positions f _{j, 15} and s _{j, 2} of the two feature points corresponding to the two feature points in the frame i are described. When a transition occurs from frame i to frame j, a distance (difference) between f _{i, 15} in frame i and position f _{j, 15} in frame j may occur for the fifteenth facial feature point. When such a distance (difference) is large, the movement of the feature point can be recognized by the user when the transition from the frame i to the frame j occurs at the time of moving image generation.

In the present embodiment, the distance (difference) between the feature points of the subject image is used as a value indicating the degree of similarity between the frames. The smaller this distance, the higher the similarity. In the present embodiment, first, the distance between the feature points of the subject image is obtained from the face detection by Kinect (registered trademark) and the posture information. Then, the total of the distances of the plurality of feature points is defined as the "inter-frame distance". Hereinafter, an example of this “inter-frame distance” will be described.

According to the face recognition specifications of Kinect (registered trademark), the number F of facial feature points is 121. Of the 20 posture feature points defined by Kinect (registered trademark), those that are always visible when the upper body is photographed (head, right shoulder, left shoulder, center of shoulder) can be used. In this case, the number S of posture feature points is 4. The distance between the i-th frame and the j-th frame is expressed by the following equation when the weighting parameters α and β are introduced.

In this embodiment, the average distance is standardized, and the face and posture distances are used in a ratio of 7 to 3 as shown in the following equations 2 and 3.

The distances D _i and _j between the frames i and the frame j obtained by such calculation can be expressed as a matrix as shown in FIG. 7.

3.2. Transition cost In the embodiment of the present disclosure, interpolation is performed by cross-dissolve processing at the time of transition between non-adjacent frames. The number of frames to be cross-dissolved is fixed, and it is d. (In this embodiment, d = 3, which corresponds to 0.25 seconds, is used.). The smaller the distance between the edges in the image, the more natural interpolation is possible. We think that the distance of this edge can be approximated by the distance of the feature points. That is, the cost of transitioning from frame i to frame j can be defined by using the following equation 4.

The transition costs C _i and _j between the frames i and the frame j obtained by such a calculation can be expressed as a matrix as shown in FIG.

The transition cost from the i-th frame to the i + 1th frame is zero. Further, considering not only the static difference between the frames but also the difference in the movement expressed by the plurality of frames, it is preferable to take the sum of the distances for the plurality of adjacent frames.

When specifying the cost of transitioning from one frame to another by using a plurality of frames (framesets), it is preferable to consider the order of the frames in the frameset. The reason is that if the similarity determined by the frame set is simply calculated as the sum of the similarity of each frame, the moving image may become unnatural. For example, consider an example in which a synthetic moving image in the direction in which the mouth opens is generated from N frames (frame sets). By simply using the sum of the similarity of each frame, there is a possibility that a frame set for N frames in the direction in which the mouth closes is selected. The videos played in that order are very unnatural. Therefore, in calculating the transition cost, the order of the frames in the frame set may be added as a parameter.

For example, when using a frame set to calculate the transition cost when transitioning from frame i to frame j, the transition cost includes a first set of consecutive frames including frame i, and frame j. A second set of contiguous frames can be obtained as a linear combination of similarities between corresponding frames of the same order.

3.3. Transitions between standby states Non-Patent Document 1 describes, as a method of selecting a transition to be actually used from the above-mentioned transition cost matrix, a method of selecting a transition of the minimum cost for each frame and a method of selecting a transition of a cost below a certain threshold. Is disclosed. In the embodiment of the present disclosure, the former is used, but the latter may be used in order to select the minimum number of more accurate transitions.

In the moving image clip in the embodiment of the present disclosure, as described above, the standby state section and the utterance state section are alternately included. Therefore, if the technique of Video Textures is simply applied, only transitions within the same section may occur. In the embodiment of the present disclosure, transitions within the same section (local transitions) and transitions to different sections (global transitions) are distinguished, and only a specified number of transitions having the lowest cost are selected for each. Specifically, the following calculation is executed for each frame in the standby state.
(1) Select N _L of transitions to the same section with the lowest cost.
(2) Of the transitions to different sections, select _NG ones with the lowest cost.
However, select so that the transition destination sections do not overlap.

For example, N _L = 10 and _NG = 5 can be used. If there are frames that you do not want to use due to recording failures, you may select candidate frames from the frame group that excludes those frames.

3.4. Transition to the utterance state The transition from the standby state to the utterance state occurs by an instruction from the dialogue engine. At that time, it is not always necessary to make a transition immediately after the instruction, and it is natural that it takes about 1 second for the "reaction" by the subject image. There may be a time grace between the instruction and the actual transition. In that case, the number of transition timing candidates increases, and a more natural transition can be selected. In the present embodiment, k is the maximum number of frames that can be spent from the time when the instruction is given until the first frame of the utterance state is reached. When the transition to the utterance state starting from the jth frame is instructed during the display of the i-th frame, the transition occurs as shown below.

However, a + b ≦ k. In the pre-calculation, a and b are obtained so that C _{i + a and jb} are minimized for each frame and each utterance section.

As described above, the data structure used in the moving image generation system of the present embodiment can be prepared in advance based on the information obtained by the prior calculation. Such a data structure shows the similarity between a moving image clip, which is a sequence of a plurality of frames including a subject image, and each frame contained in a frame group selected from the plurality of frames and each other frame. It is the similarity information, and the similarity has the similarity information defined based on the feature amount of the subject image in each frame.

4. Movie generation When a movie is generated, the dialogue agent is drawn using the recorded data and the result of pre-calculation. When the moving image generation system in the present embodiment is operated, the synthetic moving image in the standby state continues to be generated unless instructed by the dialogue engine.

The video generation system in standby mode accepts instructions to speak any of the registered lines. Then, the best transition to the instructed line is selected and transitioned, and the video is played. When the playback of the utterance section is finished, the system returns to the standby mode again. When transitioning to either the standby state or the utterance state, frame complementation enables a smoother transition.

4.1. Frame interpolation at transition Frame interpolation is performed when transitioning between non-adjacent frames. As a method of complementation, a method using cross-dissolve, morphing, an optical flow, or a method obtained by further developing it can be considered.

As for morphing, if you prepare some feature lines for each frame, you can draw at high speed in real time. Crosssolve is the simplest method of performing linear interpolation on a pixel-by-pixel basis. In this embodiment, a cross dissolve is used for frame interpolation.

In this embodiment, the transition between frames is performed as shown in FIG. Consider a case where the frame i is displayed at the time t and the frame j (j ≠ i + 1) transitions to the time t + 1. In this case, the frame i + k and the frame j + k-1 are combined and displayed at the ratio of d + 1-k: k at the time t + k. However, d is a value defined in "3.2. Transition cost" and satisfies the relationship of 1 ≦ k ≦ d.

となるように、事前計算で選んだＮ_L＋Ｎ_G個の中から遷移先を選ぶ。σ₀は小さいほどコストの小さい遷移に確率が偏る。ここまでは非特許文献に開示されている方法と全く同じである。フレームｊ（ｊ≠ｉ＋１）が選ばれた場合、時間ｔ＋１からｔ＋ｄの間クロスディゾルブ処理を行い、ｔ＋ｄ＋１で再び通常の状態に戻る。なお、待機状態では頻繁にフレーム遷移が起こるため、録画時の音声は再生しない。 4.2. Standby state The standby state is divided into two types: a section in which a transition by cross-dissolve is performed and a section in which a transition is not performed. The “section in which no transition is performed” includes, for example, a moving image clip reproduction section before the transition and a moving image clip reproduction section after the transition. It is assumed that the user is in the i-th frame in the moving image clip playback section (time t) before the transition. However, t is represented by the number of frames at the same frame rate as the recorded data. At this time, the probability of transitioning to the jth frame at time t + 1 is

Select the transition destination from the _NL + _NG items selected in the pre-calculation so that. The smaller σ ₀ , the more the probability is biased toward the transition with lower cost. Up to this point, the method is exactly the same as that disclosed in the non-patent document. When the frame j (j ≠ i + 1) is selected, the cross-dissolve process is performed for the time t + 1 to t + d, and the normal state is restored again at t + d + 1. Since frame transitions occur frequently in the standby state, the audio at the time of recording is not reproduced.

The above-mentioned number (6) is an example of a monotonous decrease function regarding the degree of similarity. Other monotonic decreasing functions may be used.

4.3. Transition to the utterance state The moving image generation system in the present embodiment switches from the standby mode to the utterance mode when it receives an instruction to utter a line from the dialogue engine. In the moving image clip, since the same line is included in a plurality of moving image parts, the parameter σ1 for adjusting the probability distribution is introduced, and the probability is calculated in the same manner using Equation 6. The above-mentioned value is used as the transition cost at that time. After making a transition with cross dissolve, simply play back the utterance state from the start to the end with voice. In the present embodiment, the standby state is recorded after the utterance state. Therefore, it naturally transitions to the standby state.

An HTTP (Hypertext Transfer Protocol) server may be implemented in the moving image generation system in a certain embodiment. In that case, the utterance instruction can be given as an HTTP request.

4.4 Local Repeat Countermeasures in Standby State According to this embodiment, by adopting the above configuration, an infinite sequence without repetition can be generated for the entire moving image. However, locally, a finite number of unnatural fine repetitions may occur. For example, if the most probable transition from a certain frame is a transition that returns to the d frame, there is a high possibility that the transition will loop many times between them.

As a method of avoiding this, there are a method of removing such a transition by pre-calculation and a method of avoiding it at the time of moving image generation. In this embodiment, an aspect of avoiding when generating a moving image is adopted. That is, when selecting the transition destination of the standby state, the frames that have visited _KL times or more within the past _TL frames are excluded from the candidates. For example, _TL can be the number of frames corresponding to 10 seconds, _KL = 2. FIG. 10 shows the case where, for example, when transitioning from the N + Xth frame (X is an integer of 2 or more) to the next frame, a frame used for display twice in the past 10 seconds exists among the candidate frames. Is schematically shown. In this case, the frame (marked with an X in FIG. 10) is excluded from the candidate frames. However, if there are no transition candidates due to this, such exclusion is not performed. Record the history of the past _KL visits in each frame.

4.5 Effective use of the entire video If a sufficient number of global transitions are prepared by pre-calculation, there is a possibility that any waiting section will be reproduced stochastically. However, when actually playing back, some waiting sections continue to come and go, and the used sections may be extremely biased. There are two possible ways to avoid this, pre-calculation and reproduction, but in this embodiment, the latter is adopted. That is, the rule "Among the past _TG times of global transitions, the section selected as the transition destination of _KG times or more is excluded from the global transition destination" is applied. For example, _TG = 10 and _KG = 2 can be set.

<Example>
In this embodiment, a human (actor) was taken as a subject and a photograph was taken using a Kinect (registered trademark) camera. RGB data of 1280 × 960 pixels was acquired at 12 FPS, and depth data of 640 × 480 pixels was acquired at 30 FPS in the proximity mode. For preview and playback during recording, the central 640 x 480 of the 1280 x 960 pixels was cut out and used. This is because posture recognition and face recognition do not work if the distance between the camera and the actor is small.

A total of 62 lines were prepared and entered into the system in advance, including greetings such as "Good morning" and "Nice to meet you", and lines used for demonstration dialogue. Each line was recorded 2-3 times, and a total of 170 utterances including failures were recorded. After each line was spoken, it was recorded as a standby state for about 5 to 10 seconds. The recording was divided into three times with a break in between. The time was 7 minutes 46 seconds, 25 minutes 42 seconds, 10 minutes 20 seconds, respectively, for a total of 43 minutes 48 seconds. The total number of frames was 31315 frames.

At the time of resumption after each break, the posture was determined by the method described with reference to FIG. By displaying the depth direction of the posture data on the screen of the display and showing it to the actor, or by using the actual recorded image for the posture control, the posture change of the subject image can be further reduced. In this embodiment, a frame is formed by recording with a large size and cutting out the central portion. The range of the central portion to be cut out may be adjusted based on the posture data.

The program in the system of this embodiment is basically written in C #. In this embodiment, the cost matrix is not stored in the memory in the computer, but in a hard disk drive (HDD) that operates as another recording device. For this reason, the number of processes with many disk accesses is increasing. The pre-calculated computer is equipped with a central processing unit (CPU) operating at 3.2 GHz and 8 GB of memory, and is connected to a 7200 rpm, 500 GB HDD. 52 minutes for image conversion, 82 minutes for copying and disassembling recorded data, which is a huge single file, 2 minutes and 20 seconds for removing unnecessary frames, which will be described later, and 6 hours for calculating and writing cost matrices. It took 6 minutes, 4 minutes to select the final transition candidate, and about 8 hours and 30 minutes in total.

During recording, there was a mistake such as the actor talking while recording in the standby state, so some frames that make up the recording clip in the standby state were excluded from the candidate frames for the transition destination. In this embodiment, the frames in which the feature points of the upper lip and the lower lip of face recognition are separated by a certain distance or more and the frames around them are mechanically detected, and 4708 frames are removed from the approximately 21551 frames in the standby state.

The playback system of this embodiment is realized by a tablet (with a CPU operating at 1.33 GHz, 2 GB of memory, and 32 GB of storage media) on which Windows 8.1 (registered trademark) is installed. In this embodiment, since the composite moving image is generated by using the frame of the video image with a human as the subject as it is, the subject image is more natural than that of the conventional technique. Also, I was worried about the frame transition unless I was careful.

Modification 1
The moving image generation system of the present disclosure may further include a line-of-sight tracking device that tracks the line of sight of a subject and outputs line-of-sight information related to this line of sight. In this case, the computer calculates an additional cost that is smaller as the line-of-sight direction of the subject specified based on the line-of-sight information is closer, and becomes larger as the line-of-sight direction of the subject is farther. The above-mentioned transition cost is corrected by utilizing such an additional cost. According to such a modification, it is possible to prevent the generation of a composite image in which the line-of-sight direction of the subject image changes discontinuously.

Modification 2
In the above embodiment, the positions of the feature points are used to calculate the similarity of the subject image, but the similarity of the subject is not limited to this example. The subject image can be represented by linearly combining the product of each of the plurality of base images prepared in advance and each of the plurality of coefficients. When the subject is a human face, such a basal image is sometimes referred to as an "eigenface". When the subject image is represented by a linear combination of the basal images, the feature amount of the subject image can be defined by a combination of these plurality of coefficients. For example, when expressing the "face" appearing in the i-th frame by linearly superimposing a plurality of face images, the set of weighting factors that define the face in the i-th frame and the face in the j-th frame are defined. The "similarity" between frames may be defined by the distance or difference from the set of weighting factors.

Modification 3
FIG. 11 is a diagram showing a modified example of the moving image display system. In the moving image display system of this example, the moving image generation system 100 can distribute a part or all of the generated synthetic moving image to a plurality of mobile terminal devices 500 via the communication network 300.

The moving image generation system 100 includes a system-side communication circuit 120 that transmits data of a synthetic moving image generated by the computer 20. Further, the mobile terminal device 500 includes a device-side communication circuit 520 that receives data of a synthetic moving image transmitted from the moving image generation system 100, and a display 550 that displays the synthetic moving image.

The system-side communication circuit 120 can receive an instruction signal transmitted from the outside, for example, via the communication network 300. A part of the communication network 300 may be an internet line. The instruction signal is transmitted in response to a user's voice input or the like by, for example, a dialogue engine (not shown). The dialogue engine may be provided in the moving image generation system 100. The computer 20 generates synthetic moving image data until the system-side communication circuit 120 receives the instruction signal. After receiving the instruction signal, the computer 20 selects a partial clip that is a part of the moving image clip according to the instruction signal. The system-side communication circuit 120 transmits the composite moving image and the partial clip data via the communication network 300. The display 550 of the mobile terminal device 500 displays a synthetic moving image and a partial clip received by the device-side communication circuit 520.

Modification 4
FIG. 12 is a diagram schematically showing an exemplary embodiment of the robot system according to the present disclosure. The robot system 400 shown in this figure includes the above-mentioned moving image generation system 100 and a robot 90 provided with a plurality of actuators (electric motors) 92 for changing at least one of facial expressions and postures. The robot 90 includes a drive circuit 94 that changes at least one of the facial expressions and postures of the robot 90 by driving a plurality of actuators 92 following changes in at least one of the facial expressions and postures of the subject image in the synthetic moving image. There is. The drive circuit 94 supplies power to an appropriate actuator 92 so that the mouth of the robot 90 opens when, for example, the subject image of the synthetic moving image opens its mouth. The robot 90 is equipped with a speaker (not shown) used for utterance, and the dialogue selected by the dialogue engine 70 is uttered as voice.

According to such a robot system 400, the robot 90 having a three-dimensional structure interacts with the user instead of the image displayed on the display. The robot 90 does not have to be humanoid, and may be animal, for example.

Flow chart FIG. 13 is a flowchart showing an example of a processing procedure to be performed in advance.

In this example, first, in step S10, the data of the moving image clip including the waiting section and the utterance section is stored in the recording device. This recording device can typically be a recording device different from the recording device included in the moving image generation system. In step S12, a group of frames excluding unnecessary frames is selected from a large number of frames constituting the moving image clip. This selection can typically be made by selecting and specifying frames that are not needed by humans. Next, in step S14, the computer used for the pre-calculation calculates the cost matrix described above. The calculation result is stored in the recording device in the form of a table as described with reference to FIG.

In this way, data having a moving image clip used in the moving image generation system and the moving image display system of the present disclosure and a cost matrix are prepared.

Next, refer to FIG. FIG. 14 is a flowchart showing an example of a moving image generation and moving image display processing procedure.

In this example, after the start of the process, in step S20, it is determined whether or not the input device has received the input from the user. If there is no input, the process proceeds to step S30 to execute the operation in the standby mode. In the standby mode, the cost matrix prepared in advance and stored in the recording device is referred to, and a plurality of candidate frames including the next frame transitioning from the current display frame are selected. In step S32, one frame is stochastically determined from the candidate frames according to the above-mentioned principle. In step S34, the next frame determined is displayed on the display. After that, the process proceeds to step S20. In this way, a probabilistic inter-frame transition is realized.

When it is determined that the input device has received an input from the user in step S20, the process proceeds to step S40 to execute the operation of the utterance mode. In step S40, the utterance section corresponding to the input is selected. When there are a plurality of utterance sections, the first frame in any of the utterance sections is stochastically selected by the same operation as the operation in the standby mode. In step S42, the selected first frame is displayed on the display. In step S42, the moving image of the utterance section including the first frame is played. In this moving image, the frames change according to the frame sequence in the moving image clip. When the reproduction of the moving image according to the utterance content is completed, the process returns to step S20.

FIG. 15 is a diagram showing an example of the internal configuration of the computer 20. The computer 20 includes a CPU 22, a memory 24, a GPU 26, and an interface (I / F) terminal 28, which are connected to each other via a bus so as to be communicable with each other.

FIG. 15 also shows a recording device 10 connected to the computer 20. The recording device 10 is, for example, a solid state drive (SSD) or an HDD.

The CPU 22 is an arithmetic circuit integrated on one chip. The memory 24 is a ROM and / or RAM that stores a computer program composed of instructions defining processing by the processor 22. A computer program is a group of instructions for realizing the above-mentioned processing (for example, FIG. 14). When the computer program is executed, the computer program stored in the ROM is expanded in the RAM. The CPU 22 sequentially reads the instructions included in the computer program from the RAM, interprets them, and executes them.

The GPU 26 is an image processing circuit called a so-called graphics processing unit. The GPU 26 performs image processing when generating the above-mentioned moving image or image processing when displaying the moving image. The GPU 26 may perform a process for displaying the image obtained as a result of the process on the display 50 (FIG. 3).

The I / F terminal 28 is a connection terminal for the computer 20 to exchange information with and from a device provided outside the computer 20. The example shown in FIG. 3 will be described. It is assumed that the recording device 10, the display 20, and the dialogue engine 70 are provided outside the computer 20. Regarding the connection between the computer 20 and the recording device 10, the I / F terminal 28 may be, for example, a serial ATA standard connection terminal. With respect to the connection between the computer 20 and the display 50, the I / F terminal 28 may be, for example, a DisplayPort®, HDMI® or DVI video connection terminal. Regarding the connection between the computer 20 and the dialogue engine 70, the I / F terminal 28 is, for example, a communication terminal capable of performing wired communication of Ethernet (registered trademark) standard or IEEE 1394 standard, or Wi-fi (registered trademark). ) A communication circuit capable of wireless communication such as standards.

The moving image generation system and the moving image display system of the present disclosure can be used for moving image generation and display of an anthropomorphic dialogue agent in, for example, a visitor response system and a reservation reception system operated by using a computer. Further, the moving image generation system of the present disclosure can also be used in a system in which a robot operates by following a change in at least one of a facial expression and a posture of a subject image in a synthetic moving image.

10 ... recording device, 20 ... computer, 50 ... display, 60 ... interface device, 70 ... dialogue engine, 80 ... shooting device, 90 ... robot, 92 ... actuator, 94 ... drive circuit, 100 ... video generation system, 120 ... system Side communication circuit, 200 ... Movie display system, 300 ... Communication network, 400 ... Robot system, 500 ... Terminal device, 520 ... Device side communication circuit, 550 ... Display

Claims (25)

A recording device that records a moving image clip that is a sequence of multiple frames including a subject image, and
A computer that reconstructs the plurality of frames based on the similarity between each frame included in the frame group selected from the plurality of frames and each other frame to generate data of a synthetic moving image, and a computer.
Equipped with
The similarity is defined based on the feature amount of the subject image in each frame.
The computer
A plurality of frames selected from the frame group based on the similarity with the Nth frame as the N + 1th frame which is the next frame of the Nth frame (N is a positive integer) of the composite video. While deciding one frame from the candidate frames of
The computer is a moving image generation system that selects the N + 1st frame according to a probability distribution including a transition cost when transitioning from the Nth frame to each of the candidate frames as a parameter. A recording device that records a moving image clip that is a sequence of multiple frames including a subject image, and
A computer that reconstructs the plurality of frames based on the similarity between each frame included in the frame group selected from the plurality of frames and each other frame to generate data of a synthetic moving image, and a computer.
Equipped with
The similarity is defined based on the feature amount of the subject image in each frame.
The computer
A plurality of frames selected from the frame group based on the similarity with the Nth frame as the N + 1th frame which is the next frame of the Nth frame (N is a positive integer) of the composite video. While deciding one frame from the candidate frames of
The plurality of candidate frames are selected according to the transition cost calculated based on the similarity.
Further equipped with a line-of-sight tracking device that tracks the line of sight of the subject image and outputs line-of-sight information related to the line of sight.
The computer
The additional cost, which is specified based on the line-of-sight information, is smaller as the direction of the line of sight of the subject image is closer and larger as the direction of the line of sight of the subject image is farther is calculated.
Modifying the transition cost using the additional cost,
Video generation system. The moving image generation system according to claim 1 or 2, wherein the computer determines one frame from a plurality of candidate frames selected from the frame group based on the similarity. The moving image generation system according to claim 1 or 2, wherein the computer stochastically determines one frame from a plurality of candidate frames selected from the frame group based on the monotonous decreasing function of the similarity. .. The moving image generation system according to any one of claims 1 to 4, wherein the feature amount of the subject image is the position of a plurality of feature points included in the subject image. The moving image generation system according to claim 5, wherein the feature amount of the subject image further includes at least one of the luminance and the color of the plurality of feature points. The similarity between any first frame included in the frame group and a second frame other than the first frame included in the frame group is each of the object images in the first frame. The moving image generation system according to claim 5 or 6, which is defined by the arrangement relationship between the feature points and the corresponding feature points in the second frame. The similarity between any first frame included in the frame group and a second frame other than the first frame included in the frame group is each of the object images in the first frame. The moving image generation system according to claim 5 or 6, which is a function of the distance between the position of the feature point and the position of each corresponding feature point in the second frame. The transition cost when transitioning from the first frame to the second frame is the first set of a predetermined number of consecutive frames including the first frame in the frame group, and the second frame in the frame group. The moving image generation system according to claim 7 or 8, wherein the second set of the predetermined number of consecutive frames including the above is a linear combination of the similarities between the corresponding frames in the same order. The moving image generation system according to any one of claims 1 to 9, wherein the frame group includes a plurality of interpolation frames prepared in advance in addition to a frame selected from the plurality of frames. The moving image generation system according to any one of claims 1 to 10, wherein the computer displays the synthesized moving image on a display. The moving image generation system according to claim 11, wherein the computer performs cross-dissolving according to the degree of similarity between the Nth frame and the N + 1th frame. The moving image generation system according to any one of claims 1 to 12, wherein at least a part of the subject image includes at least one of a human face and a body. The subject image is represented by linearly combining the products of each of a plurality of base images prepared in advance and each of a plurality of coefficients.
The moving image generation system according to any one of claims 1 to 4, wherein the feature amount is defined by a combination of the plurality of coefficients. A recording device that records a moving image clip that is a sequence of multiple frames including a subject image, and
A computer that reconstructs the plurality of frames based on the similarity between each frame included in the frame group selected from the plurality of frames and each other frame to generate data of a synthetic moving image, and a computer.
Equipped with
The computer
A plurality of frames selected from the frame group based on the similarity with the Nth frame as the N + 1th frame which is the next frame of the Nth frame (N is a positive integer) of the composite video. A video generation system that selects one frame after excluding some frames from the candidate frames. The moving image generation system according to claim 15, wherein the computer selects some of the frames according to the order of the frames constituting the synthesized moving image and excludes them from the candidate frames. When the computer has a frame ( _KL is an integer of 1 or more) contained in a predetermined number of TL frames ( _TL is an integer of 2 or more) before the Nth frame by a predetermined number of _KL times or more ( _KL is an integer of 1 or more). The moving image generation system according to claim 16, wherein the frame is selected as the partial frame and excluded from the candidate frames. The motion generation system according to any one of claims 15 to 17, wherein the computer determines one frame from a plurality of candidate frames selected from the frame group based on the similarity. 13. Video generation system. The moving image generation system according to any one of claims 1 to 19, further comprising a display for displaying the synthesized moving image. It also has an interface device that accepts user input.
The computer performs one of the generation of the data of the composite moving image and the selection of the partial clip which is a part of the moving image clip in response to the input of the user.
The moving image generation system according to claim 20, wherein the display displays the generated synthetic moving image or the selected partial clip. 21. The moving image generation system according to claim 21, further comprising a housing for accommodating the recording device, the computer, the display, and the interface device. The moving image generation system according to any one of claims 1 to 19.
With the terminal device
It is a video display system equipped with
The moving image generation system further includes a system-side communication circuit that transmits data of the synthesized moving image generated by a computer.
The terminal device is
The device-side communication circuit that receives the composite video data transmitted from the video generation system, and
A display that displays the composite video and
A video display system equipped with. The system-side communication circuit can receive an instruction signal transmitted from the outside, and can receive the instruction signal.
Until the instruction signal is received, the computer generates the data of the composite moving image.
After receiving the instruction signal, the computer selects a partial clip that is a part of the moving image clip in response to the instruction signal.
The system-side communication circuit transmits the composite video and the data of the partial clip, and receives the data.
23. The moving image display system according to claim 23, wherein the display of the terminal device displays the synthesized moving image and the partial clip received by the device side communication circuit. The moving image generation system according to any one of claims 1 to 12 and 15 to 19.
With a robot equipped with multiple actuators to change at least one of facial expression and posture,
A drive circuit that changes at least one of the facial expressions and postures of the robot by driving the plurality of actuators following changes in at least one of the facial expressions and postures of the subject image in the composite moving image.
A robot system equipped with.

Images