JP4492299B2 - Video apparatus, video display method, and program - Google Patents

Description

  The present invention relates to a video apparatus, a video display method, and a program, and can be applied to, for example, a video apparatus that displays a continuous motion video in which a plurality of videos are connected.

  Conventionally, in order to dynamically create highly realistic video content, for example, a method of preparing a plurality of live-action video files in which an actual person's movement is photographed and connecting them as necessary is available. is there.

  For example, in Patent Document 1, when a plurality of single motion images are connected to create a continuous motion image, the motion of the subject between the connected single motion images is smoothly connected in the single motion image. A technique for correcting a shift of a subject and correcting a shift of a subject between single motion images is disclosed.

Patent Document 1 and Non-Patent Document 1 disclose a technique called warping that corrects the deviation of the feature point position. Specifically, feature points are manually set on the video of a live-action person, the amount of movement of the feature points in the video file is calculated, and the amount of movement is used to perform image composition processing in the video file. There have been proposed methods for improving the quality of an image by correcting the position of the person by performing processing so that the position of the person becomes constant.
JP 2003-69900 A Satoru Yokoyama and Masaki Hayashi, “Examination of two-dimensional virtual actors by live action”, Proceedings of National Conference of Information Processing Society of Japan, March 2001

  However, the technique described in Patent Document 1 corrects a subject shift within a single motion image and between single motion images, and simply connects a plurality of highly realistic video frames in a fragmentary manner. However, if the subject's shape and operating speed are different in the connected video part, even if the subject's displacement is corrected, the subject's position will be misaligned, causing inconsistencies in the connected part, giving an unnatural impression. There is a problem that sometimes.

  Further, as pointed out in Non-Patent Document 1, the technique shown in Non-Patent Document 1 is to process all frames in a video file, that is, pixels on the entire screen, as processing for correcting the position of a person. Therefore, there is a problem that a large amount of processing time is required at the time of synthesis.

  Therefore, in order to solve the above problems, the composition information of the display video in the motion video is used during continuous motion video synthesis, thereby reducing the processing required during synthesis and creating a high-quality composite video. Provided are a video device, a video display method, and a program that can be displayed.

In order to solve such a problem, a plurality first video apparatus of the present invention, in the video apparatus that outputs a continuous operation picture operation image showing an operation in accordance with some information more linked to, continuous (1) operation and video storage means, (2) the information, and the read information of the operation image file, one or more connection criteria information displayed on the operation picture file which stores a plurality of configured operating video file from the frame And (3) retrieval of readout information of a plurality of corresponding motion video files from the connection reference information storage means based on the inputted plurality of input information having a temporal order. and, an operation image acquisition means for taking a plurality of operation images from the behavior video storage means based on the read information of a plurality of operation image file, (4) operation image acquisition unit For the plurality of motion video files extracted from the time frame, the last frame of the previous motion video file and the first frame of the subsequent motion video file, which are temporally mixed, are obtained, and the last frame read from the connection reference information storage unit is acquired. Intermediate data creation means for creating intermediate data of the video displayed in the last frame and the first frame based on the linkage reference information and the above-mentioned linkage reference information of the first frame; and (5) intermediate data from the linkage reference information storage means (6) a reference frame search unit that searches for a reference frame having connection standard information that approximates the intermediate data generated by the generation unit; and (6) a reference frame searched by the reference frame search unit. A motion video that interpolates the intermediate motion video between the last frame and the first frame of the subsequent motion video file Comprising synthesizing means for synthesizing, and output means for outputting an operation image synthesized by (7) combining means, operation image acquisition means, when the search result of the input information are a plurality, at least of the input information The operation video file corresponding to the input information is selected by comparing the connection reference information of the immediately preceding motion video file with the connection reference information of each searched motion video file .

The video display method according to the second aspect of the present invention corresponds to the video apparatus according to the first aspect of the present invention. In other words, the video display method of the second aspect of the present invention is a video display method for outputting a continuous motion video by connecting a plurality of motion videos representing motions according to certain information. (1) Consists of a plurality of continuous frames associates the operation image storage means for storing a plurality of operation image files, (2) to each information, and the read information of the operation image files, and one or more connection criteria information displayed on the operation picture file And (3) the motion picture acquisition means stores a plurality of motion picture files corresponding from the connection reference information storage means on the basis of the inputted plurality of input information in time order. Find the read information, it takes out a plurality of operation images from the behavior video storage means based on the read information of a plurality of operation image file, and (4) the intermediate data generating means For a plurality of motion video files extracted by the motion video acquisition means, obtain the last frame of the previous motion video file and the first frame of the subsequent motion video file that are temporally mixed and read out from the connection reference information storage means Intermediate data of the video displayed in the last frame and the first frame is created based on the connection standard information of the last frame and the connection standard information of the first frame, and (5) the reference frame search means is the cooperation standard information storage means (6) The synthesis means uses the reference frame searched by the reference frame search means to search for a reference frame having connection standard information that approximates the intermediate data created by the intermediate data creation means. Interpolated intermediate motion video between the last frame of the video file and the first frame of the subsequent motion video file Was synthesized image, (7) the output means outputs an operation image synthesized by the synthesizing means, the operation image acquisition means, when the search result of the input information are a plurality of immediately preceding information for at least the input information The operation video file corresponding to the input information is selected by comparing the connection reference information of the operation video file and the connection reference information of each searched operation image file .

Further, the program of the third aspect of the present invention is for outputting a continuous operation image by connecting a plurality operation image showing an operation according to some information, the operation image file composed of a plurality of consecutive frames and operating the image storing means storing a plurality of, in each information, and the read information of the operation image file, linking criterion information storage means for storing in association with one or more connection criteria information displayed on the operation picture file (1) search for the readout information of a plurality of operation video files corresponding from the connection reference information storage means based on a plurality of input information in time order input, and a plurality of operation video files operation image acquisition means for extracting a plurality of the operation image from the behavior video storage means on the basis of the read information, retrieved by (2) operation image acquisition unit For a plurality of motion video files, the last frame of the previous motion video file that is temporally mixed and the first frame of the subsequent motion video file are acquired, and the link reference information of the last frame read from the link criterion information storage means And intermediate data creation means for creating intermediate data of the video displayed in the last frame and the first frame based on the link reference information of the first frame and the first frame, and (3) created by the intermediate data creation means from the linkage reference information storage means (4) using the reference frame searched by the reference frame search means, and using the reference frame searched by the reference frame search means, the last frame of the previous motion video file, A synthesizing means for synthesizing the motion video obtained by interpolating the intermediate motion video between the first frame of the motion video file, 5) to function as output means for outputting the operation image synthesized by the synthesizing means, the operation image acquisition means, when the search result of the input information are multiple connection operation picture file immediately preceding information for at least the input information This is a program for functioning to select an operation video file corresponding to the input information by comparing the reference information with the connection reference information of each searched operation video file .

  With the video device, video display method and program of the present invention, the composition information of the display video in the motion video is used when synthesizing the continuous motion video, thereby reducing the processing required at the time of synthesis and high quality synthesis. Can create and display video.

  Embodiments of a video apparatus, a video display method, and a program according to the present invention will be described below with reference to the drawings.

(A) 1st Embodiment This embodiment demonstrates the case where it applies to the video display apparatus which displays the sign language expressing the meaning of an input sentence as a continuous video.

(A-1) Configuration of First Embodiment FIG. 1 is a functional block diagram of a video display device of the present embodiment. The video display device according to the present embodiment is realized in hardware by an information processing device such as a workstation or a personal computer including an input / output device, a central processing unit, and a storage device. For convenience of explanation of the functional configuration, FIG. 1 shows functional blocks.

  In FIG. 1, the video display device 1 includes an input unit 10, a morphological analysis unit 11, a translation unit 12, a data selection unit 13, a word dictionary 14, a video file group 15, a selected video file 16, and a display unit 17.

  The input unit 10 captures a sentence meaning that the user wants to display a video (in this embodiment, Japanese), and gives the captured sentence to the morphological analysis unit 11. Examples of realizing the function of the input unit 10 include various forms such as character input reception by a keyboard, voice text input reception by converting voice input from a microphone into text, or data stream input reception by broadcast radio waves captured by an antenna. Is assumed.

  The morphological analysis unit 11 receives an input sentence from the input unit 10, performs morphological analysis on the input sentence, decomposes the input sentence into morphemes, and assigns parts of speech to the morphemes using a dictionary (not shown). The morpheme analysis unit 11 gives the analyzed word to the translation unit 12.

  The translation unit 12 receives the word analyzed by the morpheme analysis unit 11, and performs processing such as changing the word order of words, removing unnecessary words, and adding necessary words in accordance with word information and word part-of-speech information. is there. The translation unit 12 gives the processing result to the data selection unit 13.

  When the data selection unit 13 receives the adjusted word from the translation unit 12, information associated with the word from the word dictionary 14 (described later, for example, video file name, metadata in the video, operation, etc. And a video file necessary for expressing the word meaning from the video file group 15 is searched using the video file name and metadata as search keys. The data selection unit 13 stores the searched video file in the selected video file 16. In addition, when the input word is not registered in the word dictionary 14, the data selection unit 13 searches the video showing the finger character expressing the sound of the word in sign language and gives it to the selected video file 16.

  Here, the video file is a video obtained by shooting a person who is sign language and separated by a part expressing each word meaning and given a name. The video file is configured as a sequence of a plurality of still images, and each still image is called a frame.

  The word dictionary 14 stores, for each word, the video file name of the video file including the video representing the word meaning, the operation duration in the video file, the metadata in the video, and the like in association with each other. It is an area. The word dictionary 14 can be newly registered, added, deleted, and changed.

  FIG. 2 shows a configuration example of the word dictionary 14 of the present embodiment. As shown in FIG. 2, the word dictionary 14 associates each word with a reading, a part of speech, a file name, a creation date / time, and a duration, and also associates metadata of each frame constituting each video file. It has been. The metadata is composed of data indicating the position, shape, and orientation of a person's hand shown in the video.

  The word name indicates the name of a registered word. Reading indicates the reading of the word. The part of speech indicates the part of speech name of the word. The file name indicates the name of the video file representing the word. The creation date and time indicates the date and time when the data of the created video file was registered in the word dictionary 14. The duration indicates the duration of the operation in the video file, and the unit in FIG. 2 is seconds.

  Metadata is data obtained by extracting data necessary for video composition and video connection from the content of a video file in units of frames. The configuration of the metadata can be different depending on the content of the video file and the content of the video to be output. Note that the metadata is an example of connection standard information in the claims.

  This connection reference information is at least information related to a display image for adjusting to eliminate blurring of image display between connected frames at the time of combining the images to be connected.

  In the present embodiment, since the output video is a sign language video, the word dictionary 14 includes the position of the person's hand in each frame of the sign language video representing the meaning of the input word, information (symbol) indicating the shape of the hand, The information (symbol) indicating the direction of the movement, the motion speed vector, and the like are used as metadata.

  In FIG. 2, the start point refers to the metadata of the first frame of the video file, and the end point refers to the metadata of the last frame of the video file. Further, the metadata of other frames has the same structure as the start point and end point.

  Next, items in the metadata will be described with reference to the drawings.

  The position of the hand represents the position of the center of gravity of the person's hand area appearing in the video, expressed by two-dimensional coordinates in the video. Although not shown in FIG. 2, an operation speed vector obtained by differentiating the hand position with respect to time may be provided.

  In the screen example shown in FIG. 3, the vertical direction is the y-axis, the horizontal direction is the x-axis, and the intersection of the y-axis and the x-axis is the reference point (0, 0). In this case, the position of the right hand in the frame can be the coordinates (x, y) and the position of the left hand can be the coordinates (x0, y0).

  The hand shape indicates a finger shape in which a person who is sign language is stretching or bending the finger of the hand. Usually, in sign language, about 80 types of hand shapes are distinguished, and in this embodiment, a symbol indicating the distinguished hand shape is determined in advance and the symbol is used. FIG. 4 shows an example of a hand shape and a corresponding symbol.

  The direction of the hand is a combination of the direction in which the hand is facing and the direction in which the palm is facing. Here, the direction in which the hand is facing is the direction in which a straight line drawn from the elbow of the person to the wrist is facing. Usually, in sign language, there are about 20 to 28 types of directions in which the hand is facing, and about 6 to 8 types of palm directions are distinguished. Use a combination of symbols.

  5, FIG. 6, and FIG. 7 show examples of correspondence between the direction of the hand and the symbol, and FIG. 8 shows examples of correspondence between the palm direction and the symbol.

  FIG. 5 shows a direction on a plane when a person is seen from the front, that is, a so-called frontal plane. For example, the direction in which the person's right hand is pointing in FIG. The x-axis and y-axis in FIG. 5 are the same as those described in FIG. FIG. 6 shows six directions on a plane when the person is seen from the side, so-called sagittal plane. The direction in which the person's right hand is pointing in FIG. The z-axis in FIG. 6 is an axis that is perpendicular to the x-axis and y-axis in FIG. 5 and is directed from the person's body toward the face. FIG. 7 shows four directions on a plane in which a person is viewed from directly above, a so-called horizontal plane. The direction in which the person's right hand is pointing in FIG. Each axis in FIG. 7 is the same as that in FIGS. The axis v in FIG. 8 is an axis indicating the direction in which the hand is facing as described above. A thin arrow indicates a direction in which a palm arranged in a plane perpendicular to the axis faces. The direction in which the palm faces in FIG. 8 is represented as aG.

  In this embodiment, the metadata is configured as described above in order to be applied to a video display device that displays a sign language video. However, the content information, video frequency information, color histogram, etc. A configuration may be adopted in which a part of information of the video itself is used as metadata.

  Returning to FIG. 1, the video file group 15 is used to represent each Japanese word to be expressed and Japanese sound in sign language, which are elements that will eventually form the continuous video displayed on the display unit 16. A plurality of video files including videos in which an action corresponding to the finger character action which is the action is taken.

  The selected video file 16 receives a video file for each word searched by the data selection unit 13 and holds these video files.

  The display unit 17 is a functional unit that continuously displays each video file of the selected video file 16.

(A-2) Operation of the First Embodiment Next, the operation of the video display device 1 of the present embodiment will be described with reference to the drawings. FIG. 9 is a flowchart of the video display operation of the present embodiment.

  In FIG. 9, first, a sentence to be expressed by a user in sign language video is input to the input unit 10 (S90, S91, S92). At this time, for example, the user can apply a method of inputting using a text input means such as a keyboard (S90) or a method of inputting voice by speaking toward a microphone (S91). In the case of voice input, the input voice is subjected to text conversion (S92).

  When the input unit 10 captures the input sentence, the input sentence is decomposed into morphemes by the morpheme analysis unit 11 (S93). The decomposed morpheme is assigned a part of speech by using a general Japanese dictionary (not shown) in the morpheme analysis unit 11, and the morpheme to which the part of speech is assigned (this is called a word) is given to the translation unit 12. (S94).

  When the words of the input sentence are given to the translation unit 12, the input unit divided into words is parsed by the translation unit 12 (S95), and is expressed as a state transition between words as in the existing research. The order is changed in accordance with the existing grammar (S96).

  When the structure of the input sentence is adjusted by the translation unit 12, each word is read one by one by the data selection unit 13, and the data selection unit 13 uses the word name and the assigned part of speech as a key for the word dictionary 14. Is searched to see if the registered word is in the word dictionary 14 (S97).

  If the word is not registered in the word dictionary 14 in S97, the process proceeds to S98, and a finger character search process is performed to search for a video file obtained by photographing a finger character that expresses the sound of the word in sign language (S98).

  Here, the finger character search process will be described with reference to the flowchart of FIG.

  First, the sound of an input word expressed by finger characters is decomposed into character units (S981), and characters constituting the word are read one by one (S982). The name of the file representing the finger character operation corresponding to the character is searched from the word dictionary 14 (S983), and the file name is stored in finger character data (S984).

  The processing is repeated until the processing of S982 to S984 is completed for all characters (S985), and finally the file name to be called is determined using the finger character data (S986). In S987, the file is read from the video file group 15 (S987), and the read video file is added to the selected video file (S913).

  In this way, it is possible to search for an image showing a finger character expressing the sound of the input word in sign language.

  Returning to FIG. 9, when the input word is registered in the word dictionary 14 in S97, it is determined how many registrations of the input word are (S99).

  At this time, if the number of registered input words is 1, the process proceeds to S912, and the file name item of the search result is used to read the video file from the video file group 15 (S912), and the read video file is selected. It adds to the video file 16 (S913).

  On the other hand, when there are a plurality of registered input words, a file to be read from the candidates is determined using the metadata in the word dictionary 14 (S910, S911).

  First, the data selection unit 13 refers to the metadata of the video file representing the previous word (S910).

  For example, a specific example of when metadata can be referred will be described with reference to FIG. Here, it is assumed that the current word is “Meet” and the previous word is “Tomorrow”. At this time, since the video file name of the previous word “Tomorrow” is “01.avi”, the video file indicated by this file name is selected.

  As shown in FIG. 2, since there are a plurality of video file candidates representing “Meet”, it is necessary to select which file to use. At this time, first, the coordinates of the right hand position and the left hand position at the end of the metadata of the video file “01.avi” representing the previous word “Tomorrow” are obtained (S910).

  Then, the coordinates of the right-hand position and the left-hand position at the start are obtained from the metadata of a plurality of candidate files representing “Meet”.

  Here, the sum of right and left of the Euclidean distance between the hand position at the end and the hand position at each start is obtained, and the file with the smallest value is determined as the selected file (S911).

In this example, when “01.avi” and “04.avi” are compared, the sum of the Euclidean distances of the left and right hands is
Right hand: ((50-50) ² + (50-60) ² ) ^1/2 = 10
Left ^{hand: ((100-80)} 2 ^{+ (90-60)} 2) 1/2 = 36.06
Therefore, 10 + 36.06 = 46.06.

Similarly, comparing "01.avi" and "02.avi", the sum of the Euclidean distances of the left and right hands is
Right hand: ((50-100) ² + (50-60) ² ) ^1/2 = 50.99
Left hand: ((100-130) ² + (90-60) ² ) ^1/2 = 58.21
Therefore, 50.99 + 58.21 = 109.30.

  Therefore, as a result of these comparisons, “04.avi” having a small sum of the Euclidean distances of the left and right hands is selected.

  By this process, a video file with a small shift in the position of the hand at the start of the current word from the end of the previous word is selected as a continuous file, so the shift in the position of the hand at the connecting portion between words is selected. The effect that a small image is displayed is obtained.

  If the metadata of the selected file cannot be referred to, the new data with the updated date and time may be output as the output result.

  Thereafter, the data selection unit 13 reads the selected video file from the video file group 15 (S912), and adds the read video file to the selected video file 16 (S913). The processes of S97 to S913 are repeated for all words constituting the input sentence.

  The display unit 17 reads the video file held in the selected video file 16 and outputs a continuous video.

  FIG. 11 shows an image of the display screen of the display unit 17. In FIG. 11, an input sentence 1901 is displayed by the user. After the input, for example, when the start button 1903 is pressed, the above-described video display operation is started, and the sign language is continuously displayed on the display unit 1902. An image is displayed.

  In this embodiment, in order to display the sign language image, the above processing is performed. However, some of the information of the image itself, such as information on the contents of the image, information on the image frequency, color histogram, etc. As a configuration used as metadata, it is possible to search for and use similar videos by comparing their values.

  For two video files, it is possible to compare the velocity vector near the end of each of the video files with the motion velocity vector near the start and select the ones with similar similarity.

(A-3) Effect of First Embodiment As described above, according to the present embodiment, the input sentence is decomposed into words, and correspondence is made from a video file group obtained by photographing a person who is sign language with reference to a dictionary. A video file representing a sign language word can be determined.

  At that time, the meta information such as the position of the hand at the motion end part of the person in the preceding video file represented in the dictionary is used as an index for determining the video file representing the next word. In this case, it is possible to reduce the shift of the connection portion between the front and rear images.

  Furthermore, according to the present embodiment, when there are a plurality of video file candidates to be connected and displayed, the video file metadata is used to reduce the movement deviation of the person in the preceding and following files. Since it is possible to select, it is possible to obtain an effect that the connected video as an output result becomes smoother and easier to see. At this time, if the reference is made with reference to the position information and the velocity vector information, the probability that words that were originally continuous at the time of recording the video data will be high, and connection with almost no gap becomes possible.

(B) Second Embodiment Next, a second embodiment of the video apparatus, video display method, and program of the present invention will be described with reference to the drawings.

  In the present embodiment, a case will be described in which the present invention is applied to a video display device that displays a sign language video as in the first embodiment. In addition, in this embodiment, by using a part of the selected video file and synthesizing a video file that complements a person's motion between words between the two video files, it is possible to obtain an easier-to-view video effect. The system configuration will be described.

(B-1) Configuration of Second Embodiment FIG. 12 is a functional block diagram of a video display apparatus according to the second embodiment. The video display device 2 of the second embodiment is different from the first embodiment in that a synthesis unit 101 is newly added.

  Therefore, in FIG. 12, the same and corresponding components as those in FIG. Further, the detailed functional description of the corresponding constituent elements has been described in the first embodiment, and will be omitted.

  The synthesizing unit 101 performs a process of extracting two preceding and following video files from the stored selected video file 16 and synthesizing an intermediate video file between the two video files.

(B-2) Operation | movement of 2nd Embodiment Next, operation | movement of the video display apparatus 2 of this embodiment is demonstrated with reference to drawings. Since basic operations such as input sentence input processing, input sentence morpheme / syntax analysis processing, and video file selection processing are the same as those in the first embodiment, the synthesis unit of the video display device 2 is used here. The description will focus on the creation and synthesis processing of an intermediate frame 101.

  FIG. 13A shows an operation flowchart of the synthesis unit 101, and FIG. 13B shows an image corresponding to the operation flow.

  In FIG. 13A, the compositing unit 101 includes a preceding video file 1107 representing a preceding word that has not yet been processed among the video files stored in the selected video file 16, and a subsequent video file 1109 immediately thereafter. Is extracted (S1101).

  Next, the synthesizing unit 101 extracts the last frame 1108 of the extracted previous video file 1107 and the first frame 1110 of the subsequent video file 1109, respectively (S1102).

  When the final frame 1108 and the leading frame 1110 are extracted, the synthesis unit 101 performs a predetermined morphing process based on the final frame 1108 and the leading frame 1110 to create a plurality of intermediate frames 1111 (S1103).

  Here, the morphing process is a process for creating an intermediate frame between two frames by gradually changing the ratio of the video between the two frames. For example, the existing techniques such as cross dissolve and warping are used. It is possible to use a method.

  When a plurality of intermediate frames are created, time information corresponding to the number of created intermediate frames is added to the created intermediate frames, and a video file 1106 for complementation is created between the last frame 1108 and the first frame 1110. (S1104).

  When the video file 1106 is created, it is added to the selected video file 16 (S1105). At this time, the selected video file 16 is stored as a video file to be inserted in an intermediate portion between the preceding video file 1107 and the subsequent video file 1109.

  This process is repeated until the process is completed for all the video files in the selected video file 16.

(B-3) Effects of the Second Embodiment As described above, according to the present embodiment, by adding a process for synthesizing an intermediate video of video files to be connected and displayed, it is relatively possible to connect the two videos to be connected. Even if there is a large gap, it is possible to obtain an effect that the connected video as an output result becomes smoother and easier to see.

(C) Third Embodiment Next, a third embodiment of the video apparatus, video display method, and program of the present invention will be described with reference to the drawings.

  Similarly to the first and second embodiments, this embodiment is a case where the present embodiment is applied to a video display device that displays a sign language video. Also, in this embodiment, when synthesizing a video file, an appropriate reference frame is sequentially extracted from the video files stored in the video file group using the word dictionary metadata, and the video frame is also synthesized using the reference frame. To do.

(C-1) Configuration of Third Embodiment FIG. 14 is a functional block diagram of a video display apparatus according to the third embodiment. In FIG. 14, the same or corresponding components as those in FIG. Also, the functional description of the configuration requirements described in the first embodiment is omitted.

  In FIG. 14, the function of the synthesis unit 1201 is different from those of the first and second embodiments. The synthesizing unit 1201 performs processing for extracting two preceding and following video files from the stored selected video file and synthesizing a video file that fills the middle of the two video files. At this time, the synthesis unit 1201 searches the video file group using the metadata in the word dictionary 14, acquires an appropriate frame in the video file, and uses it for synthesis.

(C-2) Operation of Third Embodiment Next, the operation of the video display device 3 of the present embodiment will be described with reference to the drawings. Since basic operations such as input sentence input processing, input sentence morpheme / syntax analysis processing, and video file selection processing are the same as those in the first embodiment, the synthesis unit of the video display device 3 is used here. Description will be made centering on the creation and synthesis processing of an intermediate frame in 1201.

  FIG. 15A shows an operation flowchart of the synthesizing unit 1201, and FIG. 15B shows an image corresponding to the operation flow.

  In FIG. 15, processes corresponding to those in FIG. 13 are denoted by corresponding reference numerals, and detailed description thereof is omitted.

  In FIG. 15A, the composition unit 1201 extracts the blooming video 1107 and the subsequent video file 1109 from the selected video file 16 (S1101), and extracts the final frame 1108 and the top frame 1110, as in the second embodiment. (S1102).

  Simultaneously with this processing, the synthesis unit 1201 searches the word dictionary 14 using the file names of the previous video file 1107 and the subsequent video file 1109 as keys (S1301), and extracts metadata corresponding to the file names (S1302).

  When the metadata corresponding to each file name is acquired, the synthesis unit 1201 creates search data by performing predetermined processing using the metadata (S1303). This search data refers to data for searching for a reference frame.

  Here, although various methods can be considered as the predetermined processing for creating the search data, in this embodiment, in order to achieve a smooth movement of the sign language operation, it is close to an intermediate portion between the final frame 1108 and the first frame 1110. Let the frame be the reference frame. Therefore, in order to search for such a reference frame, search data is created by the following method.

  For example, the synthesizing unit 1201 acquires the data of the left and right hand positions and hand orientations of the person performing sign language from the metadata of the last frame 1108 and the metadata of the first frame (S1301, S1302). The intermediate position of the hand position is obtained as search data (S1303).

  First, when the coordinate position of the right hand of the last frame is (x, y) and the coordinate position of the right hand of the first frame is (x1, y1), the coordinate position (m, n) of the right hand of the search data is ( (X + x1) / 2, (y + y1) / 2). The left hand position is similarly determined.

  Next, regarding the orientation of the hand, when the orientation of the right hand of the final frame is a3 in FIG. 5 and the orientation of the right hand of the first frame is a5, the orientation of the hand of the search data is roughly estimated as a4. To do. Thus, the search data of this embodiment is created.

  When the search data is created in S1303, the synthesizing unit 1201 searches the word dictionary 14 using the created search data as a key (S1304), and selects a frame having data closest to the search data from the video file group 15 as one. (S1305). This extracted frame is referred to as a reference frame 1307.

  When the synthesizing unit 1201 acquires the reference frame 1307, the synthesizing unit 1201 creates a plurality of intermediate frames 1111 by morphing processing using the final frame 1108, the reference frame 1307, and the first frame 1110 (S1306).

  At this time, the synthesizing unit 1201 creates a plurality of intermediate frames that gradually change from the final frame 1108 to the reference frame 1307, and also creates a plurality of intermediate frames that gradually change from the reference frame 1307 to the first frame 1110. To do. Note that morphing, cross dissolve, and the like can be applied to the method of creating the intermediate frame, as in the second embodiment.

  Subsequent operations are the same as those in the first and second embodiments, and are therefore omitted.

(C-3) Effect of Third Embodiment As described above, according to the present embodiment, when combining intermediate frames, a search is performed using the result of processing metadata of frames of previous and subsequent video files, and shooting is performed. By adding a reference frame with actual video data as a frame to be used for synthesis, a frame that is synthesized when normal synthesis is performed when changes in hand position or hand orientation are significant between the last frame and the first frame. Therefore, the influence of unnaturalness such as distortion peculiar to the composite image can be reduced.

(D) Fourth Embodiment Next, a fourth embodiment of the video apparatus, video display method, and program of the present invention will be described with reference to the drawings.

This embodiment is also applied to a video display device that displays a sign language video. In addition, the present embodiment adds an element called a dictionary creation unit 1401 to the system, analyzes a video file that captures a person expressing a sign language sentence, and divides it into video files corresponding to words. A system that is stored in a video file group and that can add metadata necessary for search to a word dictionary will be described.

  With the system configuration of the present embodiment, it becomes easy to easily add necessary metadata and the like to the word dictionary simply by inputting a video file of sign language into the system.

(D-1) Configuration of the Fourth Embodiment FIG. 16 is a functional block diagram of the video display apparatus of the present embodiment. The fourth embodiment is different from the first embodiment in that a dictionary creation unit 1401 is newly added. Therefore, in FIG. 16, the same or corresponding components as those in FIG. Also, the functional description of the configuration requirements described in the first embodiment is omitted.

  The dictionary creation unit 1401 analyzes and parses a video file that captures a person representing a sign language sentence, divides it into video files corresponding to words, stores them in a video file group, and stores them in the word dictionary. It is a functional unit characterized by adding metadata.

(D-2) Operation of the Fourth Embodiment Next, the operation of the video display device 4 of the present embodiment will be described with reference to FIG. The basic operations such as input sentence input processing, input sentence morpheme / syntactic analysis processing, and video file selection processing are the same as those in the first embodiment. The dictionary creation process by the unit 1401 will be mainly described.

  In FIG. 17, a video file 1510 is a video file obtained by photographing a person expressing a certain sentence in sign language. Hereinafter, a case will be described in which the video file 1510 is cut into words and added to the video file group 15 and the word dictionary 14.

  First, the dictionary creation unit 1401 takes in a video file 1510 to be added, and divides a plurality of video frames constituting the video file 1510 into a predetermined frame group 1511 (S1500). The frame group 1511 is a collection of a plurality of still images, and can be, for example, a time-series frame group obtained by dividing a video file every predetermined time.

  When the video frame 1510 is divided into a plurality of frame groups 1511, the dictionary creation unit 1401 extracts one frame group (S1501), and sets the extracted frame group as a processing frame 1512.

  Further, the dictionary creation unit 1401 recognizes the positions of human hands and elbows in the processing frame 1512 (S1502). As a method for recognizing the position of the hand or elbow on the screen, for example, the position of the person's hand or elbow in the processing frame 1512 using the existing algorithm such as color detection or centroid detection processing is displayed on the screen. A method of recognizing as two-dimensional coordinates is conceivable. Thereby, the position data 1513 in the processing frame 1512 can be created.

  When the position data of each processing frame is created, the dictionary creation unit 1401 recognizes the orientation of the hand from the positional relationship between the hand and the elbow of the person in the processing frame 1512 using the information of the position data 1513, and recognizes the recognition. The direction of the hand is classified (S1503).

  Next, pixel information of a certain area around the recognized hand position is acquired, the shape of the hand is recognized based on the pixel information, and the recognized hand shape is classified into symbols (S1504). At this time, as a method for recognizing the shape of the hand, for example, a method such as neural network or high-order local autocorrelation feature calculation, which is an existing image recognition algorithm, for pixels of a certain area around the hand position. Applicable.

  In this way, the dictionary creation unit 1401 adds information indicating the display time of the processing frame 1512 in the video file 1510 and adds hand position, hand orientation, and hand shape data to the time-series data ( S1505).

  Here, FIG. 18 shows a structural example of time-series data. As shown in FIG. 18, the time-series data is composed of metadata (hand position, hand orientation, hand shape) for each predetermined time (in FIG. 18, the time unit is milliseconds).

  Note that the processing of S1501 to S1505 is repeated until all the frame groups 1511 are completed.

  When the processing is completed for all the frame groups 1511, the dictionary creation unit 1401 applies a hidden Markov model (for example, a hand position, an operation speed vector that is a time derivative thereof, a hand orientation, etc.) to the created time-series data. Sign language recognition processing is performed using an existing recognition method such as HMM), and time-series data is divided into portions corresponding to sign language words (S1507).

  Further, the dictionary creation unit 1401 attaches a label of the sign language word name of the recognition result to the divided time series data (S1508), reads the video file 1510, and time information of the time series data for each label attached to the time series data. Is used to divide the video file into labels (S1509).

  Then, the dictionary creation unit 1401 assigns a name to the divided video file 1510 and stores it in the video file group 15, and performs processing for adding time series data separated from the video file name to the word dictionary 14 as a new record ( S1514).

(D-3) Effects of Fourth Embodiment As described above, according to the present embodiment, the same effects as those of the first to third embodiments can be achieved.

  In addition, according to the present embodiment, since the dictionary creation unit 1401 is provided, it is possible to easily add metadata to a video file, so that an operator can save time for adding metadata to a large number of video files. Can be eliminated.

(E) Fifth Embodiment Next, a fifth embodiment of the video apparatus, the video display method, and the program of the present invention will be described with reference to the drawings.

  This embodiment is also applied to a video display device that displays a sign language video. In addition, the present embodiment is characterized in that the data relating to the video file created by the dictionary creation unit is determined whether or not it is registered in the word dictionary using the similarity of the data and the like, and the dictionary contents are maintained. . Thereby, excessive registration of the same word can be avoided, and the word dictionary search burden can be reduced. As a result, the selectivity of the video file can be improved.

(E-1) Configuration of Fifth Embodiment FIG. 19 is a functional block diagram of a video display device of the present embodiment. The fifth embodiment is different from the fourth embodiment in that a dictionary maintenance unit 1701 connected to the dictionary creation unit 1401 is newly added. Accordingly, in FIG. 19, the same or corresponding components as those in FIGS. 1 and 16 are denoted by the corresponding reference numerals. Also, the functional description of the constituent elements described in the first and fourth embodiments is omitted.

  When the dictionary creation unit 1401 adds new data from the video file and the position data to the word dictionary 14 and the video file group 15, the dictionary maintenance unit 1701 compares the word name of the new data with the data of the existing word dictionary. New data is added or discarded, or existing data and new data are integrated.

(E-2) Operation of Fifth Embodiment Next, the operation of the video display device 5 of the present embodiment will be described with reference to FIG. Since basic operations such as input sentence input processing, input sentence morpheme / syntax analysis processing, and video file selection processing are the same as those in the first embodiment, the dictionary maintenance of the video display device 5 is performed here. Description will be made centering on maintenance processing of all 1701 word dictionaries 14.

  FIG. 20 shows the operation of the dictionary maintenance unit 1701 when the dictionary creation unit 1401 sets the video file data for each sign language word as new data.

  First, new data output by the dictionary creation unit 1401 is given to the dictionary maintenance unit 1701 (S1801). When new data is given to the dictionary maintenance unit 1701, the dictionary maintenance unit 1701 searches the word dictionary 14 using the word name of the new data as a search key (S1802), and acquires existing data (S1803).

  Here, the new data is a set of a divided video file output from the dictionary creation unit 1401 and metadata added to the word dictionary 14. The existing data may be empty when there is no corresponding data.

  The dictionary maintenance unit 1701 checks how many video files are included in the video file group 15 as a whole (S1804), and if it is greater than a certain number, the process proceeds to S1805, and the degree of matching between the new data and the existing data Calculate

  On the other hand, if the number of video files in the video file group is less than a certain number, the dictionary maintenance unit 1701 checks the number of existing data in the word dictionary 14 acquired in S1803 (S1804b), and if it is less than a certain number. In step S1810, the dictionary maintenance unit 1701 adds data (S1810). Even if the existing data is not in the word dictionary 14, the data is added as it is (S1810). In S1804b, if the number of existing data in the word dictionary 14 is equal to or greater than a certain number, the process proceeds to S1805, and the degree of matching between the new data and the existing data is calculated.

  Here, the fitness is a value representing the consistency between new data and existing data. Various methods can be considered as a method for calculating the fitness as long as new data and existing data can be compared.

  For example, first, a method for calculating the fitness based on the duration will be described. In this case, based on the duration of new data and the average value of duration of existing data, the absolute value of the rate of change is used as the fitness.

Goodness of fit = | (n−m) / m | (1)
n = the duration of new data and m = the average value of the duration of existing data.

  At this time, if the duration of the operation of the new data is significantly shorter or longer than the average value of the duration of the existing data, the fitness of the new data is lowered.

  For example, the case where the average value of acceleration is used as the fitness is described. In this case, first, the difference between the frames of the operation speed vector of the new data is obtained, and the acceleration vector is calculated. And the magnitude | size of the acceleration vector in all the frames is calculated, and the average value is calculated | required. Similarly, the average value of the acceleration vectors is obtained for existing data.

  Then, based on the average value of acceleration of new data and the average value of acceleration of existing data, the absolute value of the change rate is set as the fitness.

  At this time, when the amount of change in the movement of the new data is larger than that of the existing data, the fitness of the new data is low.

  Next, the degree of matching calculated in S1805 is determined using a threshold (S1806). If the fitness level is lower than the threshold, the dictionary maintenance unit 1701 discards new data (S1809). Here, the threshold value is, for example, a value that is 80% of the average value of the fitness of the existing data.

  On the other hand, when the fitness of the new data is equal to or greater than the threshold, the similarity between the new data and the existing data is obtained (S1807).

  Here, the similarity is a value representing the degree of similarity between new data and existing data.

  For example, add the absolute value of the difference between the hand position in each frame of new data and the hand position in the same frame of the existing data to be compared until either data reaches the end point from the start point of both data Then, it can be calculated from a value such as an average value d obtained by dividing the added value by the number of added frames.

  If the reciprocal of this value d is the similarity S, the larger the similarity value is, the higher the similarity is.

d = (| r_1-R_1 | + | r_2-R_2 | + ...
+ | R_n-1-R_n-1 | + | r_n-R_n |) / n (2)
Here, r_t is the hand position (represented by a vector) of new data in frame t, R_1 is the hand position (represented by a vector) of existing data in frame t, and n is the frame of the existing data. The smaller value of the number and the number of frames of new data.

S = 1 / d (3)
Next, when the similarity S between new data and existing data is obtained in S1807, the similarity is determined using a threshold (S1808). Here, the threshold is, for example, a value that is 80% of the average value of the similarity of existing data.

  If the degree of similarity is greater than the threshold, it can be determined that the new data is very similar to the existing data, and the dictionary maintenance unit 1701 discards the new data (S1809).

  On the other hand, if the similarity is equal to or less than the threshold value, it can be determined that there is no existing data similar to the new data, so the dictionary maintenance unit 1701 adds the new data to the word dictionary 14 and the video file group 15 (S1810).

(E-3) Effects of Fifth Embodiment As described above, according to the present embodiment, by providing the dictionary maintenance unit 1701, a candidate for a video file corresponding to one word can be obtained using the similarity and the suitability. A certain number and a certain quality can be maintained. Thereby, it is possible to prevent the video data from becoming unnecessarily large or the search speed from being lowered.

(F) Other Embodiments (F-1) In the first to fifth embodiments, a method such as a keyboard is used as the input unit. This is a voice input, a GUI-like interface, a button, a barcode reader, etc. Another input device may be used.

(F-2) In the first to fifth embodiments, it has been described that the video file 15 and the word dictionary 14 are configured as separate entities, but metadata is embedded in the header of the video file, etc. The header of the video file group may be searched as needed as a word dictionary.

(F-3) In the first to fifth embodiments, the metadata of the video file is the position of the hand, the direction of the hand, and the shape of the hand in order to express the sign language. For example, another data such as facial expressions may be used.

(F-4) In the first embodiment, the evaluation criterion for selecting a video file has been described using the Euclidean distance, but this may be another value that provides the same effect.

(F-5) In the second embodiment, cross dissolve and warping are given as examples of morphing methods, but these are not exclusive, and one or both may be used in combination. Other synthesis methods may also be used.

(F-6) In the fourth embodiment, the position data 1513 is obtained by the image processing method, but this is acquired by using a motion capture device or the like as another data at the time of shooting, and is synchronized with the frame data at the time of data analysis. It is good also as composition which takes.

(F-7) In the fifth embodiment, an example of a method for calculating the degree of matching and similarity of an existing file with respect to new data has been described. Good.

(F-8) In the first to fifth embodiments, the video display device that displays the sign language video has been described. However, the video display device includes not only the sign language but also a video having a meaning that can be associated with a word, a sentence, or a symbol. Any device that connects a plurality of video images and displays continuous motion video images can be applied to a system that displays video images of motions for expressing a specific meaning, such as a hand flag signal, pantomime, and dance.

(F-9) In the first to fifth embodiments, Japanese-language dictionary search is performed. However, by changing the dictionary data, a sign language display system in other foreign languages can be obtained.

(F-10) In the first to fifth embodiments, the configuration of each dictionary is shown as a table, but different data structures such as a tree may be used.

(F-11) In the fourth and fifth embodiments, the video display device 1 described in the first embodiment has been described as having the dictionary creation unit 1401 and the dictionary maintenance unit 1701. The present invention can also be applied to the video display devices 2 and 3 of the third embodiment.

(F-12) The constituent elements constituting the present invention may exist on one PC, or a part or all of the elements such as a database may be a server on a network.

It is a functional block diagram of the video display device of a 1st embodiment. It is a data structure figure of the word dictionary of a 1st embodiment. It is explanatory drawing which shows the position of the person's hand in the screen of 1st Embodiment. It is explanatory drawing explaining the relationship between the hand shape and symbol of 1st Embodiment. It is explanatory drawing explaining the relationship between the direction of the hand of 1st Embodiment, and a symbol. It is explanatory drawing explaining the relationship between the direction of the hand of 1st Embodiment, and a symbol. It is explanatory drawing explaining the relationship between the direction of the hand of 1st Embodiment, and a symbol. It is explanatory drawing explaining the relationship between the direction of the hand of 1st Embodiment, and a symbol. It is an operation | movement flowchart of the video display apparatus of 1st Embodiment. It is an operation | movement flowchart of the finger character search of 1st Embodiment. The example of the display screen of the display part in 1st Embodiment is shown. It is a functional block diagram of the video display apparatus of 2nd Embodiment. It is an operation | movement flowchart of the video display apparatus of 2nd Embodiment. It is a functional block diagram of the video display apparatus of 3rd Embodiment. It is an operation | movement flowchart of the video display apparatus of 3rd Embodiment. It is a functional block diagram of the video display apparatus of 4th Embodiment. It is an operation | movement flowchart of the video display apparatus of 4th Embodiment. It is a figure which shows the structural example of the time series data of 4th Embodiment. It is a functional block diagram of the video display apparatus of 5th Embodiment. It is an operation | movement flowchart of the video display apparatus of 5th Embodiment.

Explanation of symbols

1, 2, 3, 4, 5 ... video display device, 10 ... input unit, 11 ... morphological analysis unit,
12 ... Translation unit, 13 ... Data selection unit, 14 ... Word dictionary, 15 ... Video file group,
16 ... Selected video file, 17 ... Display section, 101, 1201 ... Composition section,
1401 ... Dictionary creation unit, 1701 ... Dictionary maintenance unit.

Claims (7)

In a video device that outputs a continuous motion image by connecting a plurality of motion images representing operations according to certain information,
Motion video storage means for storing a plurality of motion video files composed of a plurality of consecutive frames ;
To the above information, and the read information of the operation image file, and the linking criterion information storage means for storing in association with one or more linking criterion information is displayed on the operation picture file,
Based on the plurality of input information that is input temporal order to find the reading information of a plurality of the operation image file corresponding from the connection reference information storage unit, the based on the read information of the plurality of the operation image file Action video acquisition means for retrieving the plurality of action video files from the action video storage means;
For the plurality of motion video files extracted by the motion video acquisition means, obtain the last frame of the previous motion video file and the first frame of the subsequent motion video file that are temporally mixed, and store the connection reference information Intermediate data creating means for creating intermediate data of the video displayed in the last frame and the first frame based on the connection standard information of the last frame read from the means and the connection standard information of the first frame;
Reference frame search means for searching for a reference frame having concatenation standard information that approximates the intermediate data created by the intermediate data creation means from the linkage standard information storage means;
Using the reference frame searched by the reference frame search means, a motion video obtained by interpolating an intermediate motion video between the last frame of the previous motion video file and the first frame of the subsequent motion video file is synthesized. A synthesis means to
Output means for outputting the motion image synthesized by the synthesis means ,
When there are a plurality of search results of the input information, the operation video acquisition means includes at least the connection reference information of the operation video file immediately preceding the input information and the connection reference of each of the searched operation video files. A video apparatus, wherein the operation video file corresponding to the input information is selected by comparison with information. A frame group generating means for generating a frame group composed of a plurality of processing frames by dividing an input operation video file input as an addition target at predetermined time intervals;
Time-series data generating means for recognizing the position, orientation and shape of the video displayed in each processing frame divided by the frame group generating means and generating time-series data of the position, orientation and shape for each predetermined time When,
By the predetermined information recognition process using each time-series data generated by the time-series data generating means, the time-series data constituting these parts is divided into portions corresponding to one information in the frame group. File dividing means for associating the information with each other and dividing the input operation video file for each information;
The input motion video file is added with read information and added to the motion video storage means, and the time series data of each part is used as the link reference information, and the link criterion is associated with the read information of the input motion video file. Registration means to be added to the information storage means;
Video apparatus according to claim 1, characterized in that it comprises a. When adding by the registration means, it is determined whether or not existing information equivalent to the information to be added this time exists in the connection standard information storage means, and if present, the information to be added this time is discarded without being registered. The video apparatus according to claim 2 , further comprising registration maintenance means. The video apparatus according to claim 1, wherein the connection reference information is metadata indicating configuration information of a display video displayed in the operation video. The motion video file is a sign language video file expressing the meaning of a word,
The connection reference information, the position of the hand of the person sign language, the hand orientation, image apparatus according to any one of claims 1 to 4, wherein the meta data including the shape of a hand. In a video display method for outputting a continuous motion image by connecting a plurality of motion images representing motion according to certain information,
Motion video storage means for storing a plurality of motion video files composed of a plurality of consecutive frames ;
To the above information, comprising: a readout of the operation image file, and a connection reference information storage means for storing in association with one or more linking criterion information is displayed on the operation picture file,
The motion video acquisition means retrieves the readout information of the plurality of motion video files corresponding from the connection reference information storage means based on the plurality of input information in time order, and the plurality of motion video files Based on the read information, a plurality of the motion video files are extracted from the motion video storage means,
The intermediate data creation means acquires the last frame of the previous motion video file and the first frame of the subsequent motion video file for the plurality of motion video files extracted by the motion video acquisition means. Based on the connection reference information of the last frame read from the connection reference information storage means and the connection reference information of the first frame, intermediate data of the video displayed in the last frame and the first frame is created. ,
The reference frame search means searches the reference frame having connection standard information that approximates the intermediate data created by the intermediate data creation means from the linkage standard information storage means,
The synthesizing means interpolates an intermediate motion video between the last frame of the previous motion video file and the first frame of the subsequent motion video file using the reference frame retrieved by the reference frame retrieval means. Synthesize motion video,
The output means outputs the operation video synthesized by the synthesis means ,
When there are a plurality of search results of the input information, the operation video acquisition means includes at least the connection reference information of the operation video file immediately preceding the input information and the connection reference of each of the searched operation video files. A video display method comprising: selecting the operation video file corresponding to the input information by comparison with information. Concatenating a plurality of motion images representing operations according to certain information and outputting a continuous motion image ,
Motion video storage means for storing a plurality of motion video files composed of a plurality of consecutive frames ;
To the above information, and the read information of the operation image file, and the linking criterion information storage means for storing in association with one or more linking criterion information is displayed on the operation picture file
A video device comprising
Based on the plurality of input information that is input temporal order to find the reading information of a plurality of the operation image file corresponding from the connection reference information storage unit, the based on the read information of the plurality of the operation image file Motion video acquisition means for extracting a plurality of the motion video files from the motion video storage means;
For the plurality of motion video files extracted by the motion video acquisition means, obtain the last frame of the previous motion video file and the first frame of the subsequent motion video file that are temporally mixed, and store the connection reference information Intermediate data creating means for creating intermediate data of the video displayed in the last frame and the first frame based on the connection reference information of the last frame read from the means and the connection reference information of the first frame;
Reference frame search means for searching for a reference frame having concatenation standard information that approximates the intermediate data created by the intermediate data creation means from the linkage standard information storage means,
Using the reference frame searched by the reference frame search means, a motion video obtained by interpolating an intermediate motion video between the last frame of the previous motion video file and the first frame of the subsequent motion video file is synthesized. Synthesis means to
Function as an output means for outputting the motion video synthesized by the synthesis means ,
When there are a plurality of search results of the input information, the operation video acquisition means includes at least the connection reference information of the operation video file immediately preceding the input information and the connection reference of each of the searched operation video files. A program for causing a function to select the operation video file corresponding to the input information by comparison with the information.

Images