KR102636708B1 - Electronic terminal apparatus which is able to produce a sign language presentation video for a presentation document, and the operating method thereof - Google Patents

Abstract

The present invention provides an electronic terminal device capable of producing a sign language presentation video for a presentation document and a method of operating the same, thereby supporting hearing-impaired people to more easily understand the contents of the presentation document through the sign language presentation video.

Description

Electronic terminal device capable of producing sign language presentation video for presentation document and its operating method {ELECTRONIC TERMINAL APPARATUS WHICH IS ABLE TO PRODUCE A SIGN LANGUAGE PRESENTATION VIDEO FOR A PRESENTATION DOCUMENT, AND THE OPERATING METHOD THEREOF}

The present invention relates to an electronic terminal device capable of producing a sign language presentation video for a presentation document and a method of operating the same.

Recently, as computers, smartphones, and tablet PCs have become widely available, various types of electronic document-related programs that allow users to view, create, and edit electronic documents using these electronic terminal devices have been released. These electronic document-related programs include word processors that support the creation and editing of basic documents, spreadsheets that assist in data input, arithmetic operations, and data management, and presentation programs that assist presenters in their presentations.

In particular, since presentation programs are mainly used when creating documents for use in presentations, users sometimes create presentation recordings of the slides to help readers easily understand the contents of the slides that make up the presentation document. many.

In this way, recorded presentation videos of slides can be very useful in that they help readers intuitively understand the content of the slide.

However, since these recorded videos mainly consist of audio, there is a problem in that it is difficult for hearing impaired people to understand the contents of the slides using only the recorded videos.

If technology is introduced to extract voice from a recorded video of a slide, convert the extracted voice into text, and produce a sign language presentation video for the slide based on the text, through this sign language presentation video, Even hearing-impaired users will be able to easily understand the contents of the slides.

Therefore, research is needed on technology that can produce sign language presentation videos for presentation documents.

The present invention proposes an electronic terminal device capable of producing a sign language presentation video for a presentation document and a method of operating the same, thereby supporting hearing-impaired people to more easily understand the contents of the presentation document through the sign language presentation video.

An electronic terminal device capable of producing a sign language presentation video for a presentation document according to an embodiment of the present invention uses a microphone mounted on the electronic terminal device when a command to record a presentation video for a presentation document is issued by a user. , a recorded video generator that receives the user's presentation voice for each of the plurality of slides constituting the presentation document and generates a recorded video for each of the plurality of slides based on the presentation voice, the user When a command to add a sign language video for the first slide, which is one of the plurality of slides, is applied, voice is extracted from the first recorded video for the first slide, and a preset STT (Speech To Text) is applied. ) A text conversion unit that converts the extracted voice into text by performing voice recognition on the extracted voice based on the model, and when the extracted voice is converted into text, the text is converted into sign language. An avatar image generator that generates a first avatar image having an expressive motion, and when the first avatar image is generated, inserts the first avatar image into a preset insertion position of the first recorded image and synthesizes it into one image. By doing so, it includes a sign language image generating unit that generates a sign language image, and a video insertion unit that inserts the recorded video for each slide as an object into each of the plurality of slides, and additionally inserts the sign language image as an object in the first slide. do.

In addition, a method of operating an electronic terminal device capable of producing a sign language presentation video for a presentation document according to an embodiment of the present invention involves, when a command to record a presentation video for a presentation document is given by a user, the electronic terminal device Receiving the user's presentation voice for each of the plurality of slides constituting the presentation document through a mounted microphone, and generating a recorded video for each of the plurality of slides based on the presentation voice, When a command to create an additional sign language video for the first slide, which is one of the plurality of slides, is issued by the user, audio is extracted from the first recorded video for the first slide, and a preset STT model is used. As a basis, converting the extracted voice into text by performing voice recognition on the extracted voice, and when the extracted voice is converted into text, a first avatar having an operation of expressing the text in sign language Generating an image, when the first avatar image is generated, inserting the first avatar image into a preset insertion position of the first recorded image and compositing it into one image, thereby generating a sign language image, and the plurality of In each of the slides, inserting the recorded video for each slide as an object, and additionally inserting the sign language video as an object in the first slide.

The present invention provides an electronic terminal device capable of producing a sign language presentation video for a presentation document and a method of operating the same, thereby supporting hearing-impaired people to more easily understand the contents of the presentation document through the sign language presentation video.

Figure 1 is a diagram showing the structure of an electronic terminal device capable of producing a sign language presentation video for a presentation document according to an embodiment of the present invention.
Figures 2 and 3 are diagrams for explaining the operation of an electronic terminal device capable of producing a sign language presentation video for a presentation document according to an embodiment of the present invention.
Figure 4 is a flowchart showing a method of operating an electronic terminal device capable of producing a sign language presentation video for a presentation document according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the attached drawings. This description is not intended to limit the present invention to specific embodiments, but should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. In describing each drawing, similar reference numerals are used for similar components, and unless otherwise defined, all terms used in this specification, including technical or scientific terms, are within the scope of common knowledge in the technical field to which the present invention pertains. It has the same meaning as generally understood by those who have it.

In this document, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary. Additionally, in various embodiments of the present invention, each component, functional block, or means may be composed of one or more subcomponents, and the electrical, electronic, and mechanical functions performed by each component may be electronic. It may be implemented with various known elements or mechanical elements such as circuits, integrated circuits, and ASICs (Application Specific Integrated Circuits), and may be implemented separately or by integrating two or more into one.

Meanwhile, the blocks in the attached block diagram or the steps in the flow chart are computer program instructions that are mounted on the processor or memory of equipment capable of data processing, such as general-purpose computers, special-purpose computers, portable laptop computers, and network computers, and perform designated functions. It can be interpreted to mean. Because these computer program instructions can be stored in a memory provided in a computer device or in a computer-readable memory, the functions described in the blocks of a block diagram or the steps of a flow diagram can be produced as a manufactured product containing instruction means to perform them. It could be. In addition, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). Additionally, it should be noted that in some alternative embodiments, it is possible for functions mentioned in blocks or steps to be executed in a different order. For example, two blocks or steps shown in succession may be performed substantially simultaneously or in reverse order, and in some cases, some blocks or steps may be omitted.

Figure 1 is a diagram showing the structure of an electronic terminal device capable of producing a sign language presentation video for a presentation document according to an embodiment of the present invention.

Referring to FIG. 1, the electronic terminal device 110 according to an embodiment of the present invention includes a recorded image generator 111, a text converter 112, an avatar image generator 113, and a sign language image generator 114. ) and an image insertion unit 115.

When a command to record a presentation video for a presentation document is issued by the user 130, the recorded video generator 111 generates a plurality of slides constituting the presentation document through a microphone mounted on the electronic terminal device 110. The presentation voice of the user 130 for each of the slides is input, and a recorded video for each of the plurality of slides is generated based on the presentation voice.

When a command to create an additional sign language video for a first slide, which is one of the plurality of slides, is applied by the user 130, the text converter 112 generates an audio signal from the first recorded video for the first slide. and converts the extracted voice into text by performing voice recognition on the extracted voice based on a preset STT (Speech To Text) model.

When the extracted voice is converted into text, the avatar image generator 113 generates a first avatar image that has the action of expressing the text in sign language.

At this time, according to an embodiment of the present invention, the avatar image generator 113 is a specific configuration for generating the first avatar image, and includes an avatar image clip storage unit 116, a word extractor 117, and a word matching unit. It may include a unit 118 and a generation processing unit 119.

In the avatar video clip storage unit 116, a plurality of preset representative words for sign language translation and a pre-produced avatar video clip with an operation to express each of the plurality of representative words in sign language are stored in correspondence with each other. .

For example, the avatar video clip storage unit 116 includes a plurality of preset representative words for sign language translation, as shown in Table 1 below, and a pre-made avatar that has an operation to express each of the plurality of representative words in sign language. Video clips may be stored in correspondence with each other.

representative plural words Avatar video clip word 1 Avatar video clip 1 word 2 Avatar video clip 2 word 3 Avatar video clip 3 ... ... word 10 Avatar video clip 10

When the extracted voice is converted into text by the text conversion unit 112, the word extraction unit 117 extracts at least one word constituting the text from the text.

When the at least one word is extracted, the word matching unit 118 matches each of the at least one word with a similar representative word that has the maximum degree of similarity to each word among the plurality of representative words.

At this time, according to one embodiment of the present invention, the word matching unit 118 is configured to match, for each of the at least one word, one by one similar representative words that have the maximum degree of similarity to each word among the plurality of representative words. As a specific configuration, it may include a dictionary database 120, a similarity calculation unit 121, and a matching processing unit 122.

The dictionary database 120 contains a plurality of preset words and a preset embedding vector corresponding to each of the plurality of words (the embedding vector corresponding to each of the plurality of words is based on the similarity between words). As a preset vector, the vector similarity between the embedding vectors corresponding to the words is calculated to be greater as the meaning between the words is similar to each other.) is stored.

Here, as the vector similarity, cosine similarity according to Equation 1 below or Euclidean Distance according to Equation 2 below may be used.

In Equation 1 above, S is the cosine similarity between vector A and vector B and has a value between -1 and 1, A _i refers to the ith component of vector A, and B _i refers to the ith component of vector B. do. In relation to this, the greater the cosine similarity between two vectors, the more similar the two vectors can be considered to be.

In Equation 2 above, D is the Euclidean distance between vector A and vector B, A _i is the ith component of vector A, and B _i is the ith component of vector B. In relation to this, the smaller the Euclidean distance between two vectors, the more similar the two vectors can be considered to be.

For example, the dictionary database 120 may store a plurality of preset words and preset embedding vectors corresponding to each of the plurality of words, as shown in Table 2 below.

plural words Embedding vector word 1 Vector 1 word 2 Vector 2 word 3 Vector 3 ... ... 1000 words Vector 1000

When the at least one word is extracted by the word extraction unit 117, the similarity calculation unit 121 refers to the dictionary database 120 and generates, for each of the at least one word, an embedding vector corresponding to each word, By calculating the vector similarity between the embedding vectors corresponding to each of the plurality of representative words, the similarity between the plurality of representative words for each of the at least one word is calculated.

The matching processing unit 122 matches each of the at least one word with a similar representative word that has the maximum degree of similarity to each word among the plurality of representative words, one by one.

The generation processing unit 119 extracts an avatar video clip corresponding to a similar representative word matching each word from the avatar video clip storage unit 116 for each of the at least one word, After extracting the corresponding avatar video clip, the avatar video clip corresponding to each of the at least one word is concatenated according to the order in which each word is located in the text to generate the first avatar image.

Hereinafter, the recorded image generator 111, the text converter 112, the avatar video clip storage unit 116, the word extractor 117, the similarity calculation unit 121, the matching processor 122, and the generation processor 119. )'s operation will be explained in detail using an example.

First, assume that a command to record a presentation video for a presentation document is granted to the electronic terminal device 110 by the user 130 . At this time, let's assume that the presentation document consists of a plurality of slides such as 'Slide 1, Slide 2, Slide 3, Slide 4, and Slide 5'.

Then, the recorded image generator 111 generates the user 130's presentation voice for each of 'Slide 1, Slide 2, Slide 3, Slide 4, and Slide 5' through the microphone mounted on the electronic terminal device 110. Upon receiving the input, based on the above presentation voice, the recorded video for each of 'Slide 1, Slide 2, Slide 3, Slide 4, and Slide 5' is recorded as 'Recorded Video 1, Recorded Video 2, Recorded Video 3, Recorded Video 4, Recorded Video. It can be created as shown in video 5'.

At this time, let us assume that a command to create an additional sign language image for 'Slide 2' is applied to the electronic terminal device 110 by the user 130.

Then, the text converter 112 extracts the voice from 'recorded video 2', which is the first recorded video for 'slide 2', and performs voice recognition on the extracted voice based on a preset STT model. , the extracted voice can be converted into text.

Then, the word extractor 117 may extract at least one word constituting the text from the text.

As a result, let's assume that 'word 12, word 13, and word 15' are extracted as the at least one word.

Then, the similarity calculation unit 121 refers to the dictionary database 120 as shown in Table 2 above, and for each of 'word 12, word 13, and word 15', an embedding vector corresponding to each word, and the plurality of representative words. By calculating the vector similarity between the embedding vectors corresponding to each of 'Word 1, Word 2, Word 3, ..., Word 10', 'Word 1, Word 2' for each of 'Word 12, Word 13, and Word 15' , word 3, ..., the similarity with word 10' can be calculated. That is, the similarity calculation unit 121 can calculate the similarity with 'word 1, word 2, word 3, ..., word 10' for 'word 12', and for 'word 13', 'word The similarity with '1, word 2, word 3, ..., word 10' can be calculated, and for 'word 15', the similarity with 'word 1, word 2, word 3, ..., word 10' Similarity can be calculated.

As a result, for 'word 12', among 'word 1, word 2, word 3, ..., word 10', the similar representative word with the maximum similarity is called 'word 4', and 'word 13' is For 'Word 1, Word 2, Word 3, ..., Word 10', the similar representative word with the maximum similarity is called 'Word 1', and for 'Word 15', it is called 'Word 1, Word 2,' Let's assume that among 'word 3,..., word 10', the similar representative word with the maximum similarity is 'word 8'.

Then, the matching processor 122 can match 'word 4, word 1, and word 8' one by one to each of 'word 12, word 13, and word 15.'

At this time, in the avatar video clip storage unit 116 as shown in Table 1, avatar images corresponding to 'word 4, word 1, and word 8', which are similar representative words that match each of 'word 12, word 13, and word 15' Since the clips are 'Avatar video clip 4, Avatar video clip 1, and Avatar video clip 8', the creation processing unit 119 selects 'Avatar video clip 4, Avatar video clip' from the avatar video clip storage unit 116 as shown in Table 1 above. By extracting '1, avatar video clip 8', the avatar video clips corresponding to each of 'word 12, word 13, and word 15' can be extracted as 'avatar video clip 4, avatar video clip 1, and avatar video clip 8'. there is.

Then, the generation processing unit 119 generates 'Avatar video clip 4, Avatar video clip 1, and Avatar video clip 8', which are avatar video clips corresponding to each of 'Word 12, Word 13, and Word 15', and each word in the text. You can create 'Avatar Video 1' by splicing them according to the order in which they are located.

In relation to this, if the order in which 'word 12, word 13, and word 15' are located in the text is 'word 12, word 13, word 15', the generation processing unit 119 generates 'avatar video clip 4, avatar video You can create 'Avatar video 1' by concatenating 'Clip 1, Avatar video clip 8' in the order of 'Avatar video clip 4, Avatar video clip 1, and Avatar video clip 8'.

In this way, when the first avatar image is generated by the avatar image generator 113, the sign language image generator 114 inserts the first avatar image into the preset insertion position of the first recorded image to create one image. By compositing, a sign language image can be created.

At this time, according to an embodiment of the present invention, when the first avatar image is generated by the avatar image generator 113, the sign language image generator 114 generates a first start signal in which a voice starts from the first recorded image. The time of the viewpoint and the time of the first end point where the voice ends are confirmed, and based on the time of the first start point and the time of the first end point, the first avatar image is generated at the first start point and the first end point. After adjusting the playback speed of the first avatar image so that it starts at the same point in time and ends at the same point in time as the first end point, the first avatar image with the adjusted playback speed is displayed at the preset point of the first recorded video. The sign language image can be generated by inserting it at the insertion position and combining it into one image.

The video insertion unit 115 inserts the recorded video for each slide as an object into each of the plurality of slides, and additionally inserts the sign language video as an object into the first slide.

Hereinafter, the operations of the sign language image generator 114 and the image insertion unit 115 will be described in detail using an example.

First, let's assume that 'Avatar Image 1' is created by the avatar image generator 113, as in the above-described example.

Then, the sign language video generator 114 can check the first start time when the voice starts and the first end time when the voice ends in 'Recorded Video 2', which is the recorded video for 'Slide 2'. .

As a result, if the time of the first start point is '5 seconds' and the time of the first end point is '1 minute and 47 seconds', '5 seconds', which is the time of the first start point, and the Based on the first ending time of '1 minute and 47 seconds', the playback speed of 'Avatar Video 1' is adjusted so that 'Avatar Video 1' starts at '5 seconds' and ends at '1 minute and 47 seconds'. It can be adjusted.

Then, the sign language image generator 114 inserts 'Avatar Image 1' with the playback speed adjusted at the preset insertion position of 'Recorded Image 2' and synthesizes it into one image, thereby generating the sign language image. there is.

In relation to this, if 'avatar video 1' is said to be the same as the picture indicated by reference numeral 210 in FIG. 2 and 'recorded video 2' is the same as the picture indicated by reference numeral 220 in FIG. 2, the sign language image generator 114 As shown in the figure indicated by reference numeral 230 in FIG. 2, a sign language video can be created by inserting 'Avatar Image 1 (210)' at the preset insertion position of 'Recorded Image 2 (220)' and compositing it into one image. there is.

Then, the video insertion unit 115 inserts the recorded video for each slide as an object into each of 'Slide 1, Slide 2, Slide 3, Slide 4, and Slide 5', and inserts the sign language video into 'Slide 2'. Additional objects can be inserted.

For example, the video insertion unit 115 inserts the recorded video for each slide into 'Slide 1, Slide 2, Slide 3, Slide 4, and Slide 5' as an object (as shown by reference numeral 310 in FIG. 3). 301), but in 'Slide 2', a sign language image can be additionally inserted as an object 302, as shown in the drawing 320.

According to one embodiment of the present invention, the electronic terminal device 110 may further include a playback inquiry unit 123, an image selection inquiry unit 124, and an image playback unit 125.

The playback query unit 123 is operated by the video insertion unit 115 after the recorded video for each slide is inserted as an object into the plurality of slides and the sign language video is additionally inserted as an object into the first slide. When a slideshow execution command for the presentation document is granted by the user 130, a query message asking whether to play the recorded video inserted as an object in the plurality of slides is displayed on the screen.

When a playback command instructing to play a recorded video is applied by the user 130, the video selection query unit 124 records a video inserted as an object in the plurality of slides according to the page order of the plurality of slides. Videos are played sequentially and displayed on the screen, and when it is the turn to play the first recorded video inserted into the first slide among the plurality of slides, whether to play the first recorded video or the sign language video. A video selection query message is displayed on the screen asking whether the video is available.

When a recorded video playback command instructing to play the first recorded video is applied by the user 130, the video playback unit 125 plays the first recorded video and a sign language instructing to play the sign language video. When a video playback command is applied, the sign language video is played.

Hereinafter, the operations of the playback query unit 123, the video selection query unit 124, and the video playback unit 125 will be described in detail using examples.

First, as in the above-mentioned example, the recorded video for each slide is inserted as an object in each of 'Slide 1, Slide 2, Slide 3, Slide 4, and Slide 5' by the video insertion unit 115, and 'Slide Assume that the sign language image is additionally inserted as an object in 2', and thereafter, a slideshow execution command for the presentation document is granted to the electronic terminal device 110 by the user 130.

Then, the playback query unit 123 can display a query message on the screen asking whether to play the recorded video inserted as an object in 'Slide 1, Slide 2, Slide 3, Slide 4, and Slide 5'. there is.

At this time, let us assume that a play command instructing to play the recorded video is applied to the electronic terminal device 110 by the user 130.

Then, the image selection query unit 124 selects the objects in 'Slide 1, Slide 2, Slide 3, Slide 4, and Slide 5' according to the page order of 'Slide 1, Slide 2, Slide 3, Slide 4, and Slide 5'. The recorded videos inserted in 'Recorded Video 1, Recorded Video 2, Recorded Video 3, Recorded Video 4, and Recorded Video 5' can be played sequentially and displayed on the screen.

At this time, when it is the turn to play 'recorded video 2' inserted in 'recorded video 2' inserted in 'slide 2', the video selection query unit 124 determines whether to play 'recorded video 2' or the sign language video. A video selection message inquiring whether or not the video is available can be displayed on the screen.

At this time, when a recorded video playback command instructing to play 'recorded video 2' is applied to the electronic terminal device 110 by the user 130, the video playback unit 125 plays 'recorded video 2'. When it is said that a sign language video playback command instructing to play the sign language video is applied to the electronic terminal device 110 by the user 130, the video playback unit 125 can play the sign language video.

Figure 4 is a flowchart showing a method of operating an electronic terminal device capable of producing a sign language presentation video for a presentation document according to an embodiment of the present invention.

In step S410, when a command to record a presentation video for a presentation document is issued by the user, the user's presentation for each of a plurality of slides constituting the presentation document is performed through a microphone mounted on the electronic terminal device. An audio input is received, and a recorded video for each of the plurality of slides is generated based on the presentation audio.

In step S420, when a command to add a sign language video for a first slide, which is one of the plurality of slides, is commanded by the user, a voice is extracted from the first recorded video for the first slide, The extracted voice is converted into text by performing voice recognition on the extracted voice based on a preset STT model.

In step S430, when the extracted voice is converted into text, a first avatar image is generated that has the action of expressing the text in sign language.

In step S440, when the first avatar image is generated, the first avatar image is inserted into a preset insertion position of the first recorded image and synthesized into one image to generate a sign language image.

In step S450, the recorded video for each slide is inserted as an object into each of the plurality of slides, and the sign language video is additionally inserted as an object into the first slide.

At this time, according to an embodiment of the present invention, in step S430, a pre-produced avatar video clip having a plurality of preset representative words for sign language translation and an operation of expressing each of the plurality of representative words in sign language is provided. maintaining an avatar video clip storage unit stored in correspondence with each other; when the extracted voice is converted to the text, extracting at least one word constituting the text from the text; the at least one word When extracted, for each of the at least one word, matching a similar representative word with the maximum similarity to each word among the plurality of representative words one by one; and for each of the at least one word, the avatar video clip. From the storage unit, by extracting an avatar video clip corresponding to a similar representative word matching each word, the avatar video clip corresponding to each of the at least one word is extracted, and then the avatar video clip corresponding to each of the at least one word is extracted. It may include generating the first avatar image by concatenating each word in the text according to the order in which they are located.

At this time, according to an embodiment of the present invention, the matching step includes a plurality of preset words and a preset embedding vector corresponding to each of the plurality of words (the embedding vector corresponding to each of the plurality of words is , a preset vector based on the similarity between words; the more similar the meaning between words are, the greater the vector similarity between the embedding vectors corresponding to the words is calculated) is maintained in a dictionary database that is stored. When the at least one word is extracted, referring to the dictionary database, for each of the at least one word, between an embedding vector corresponding to each word and an embedding vector corresponding to each of the plurality of representative words. calculating similarity with the plurality of representative words for each of the at least one word by calculating vector similarity; and for each of the at least one word, similarity with each word among the plurality of representative words. It may include the step of matching similar representative words with a maximum one by one.

Additionally, according to an embodiment of the present invention, in step S440, when the first avatar image is generated, a first start time and a first end time when the sound ends in the first recorded image By checking the time, based on the first start time and the first end time, the first avatar image starts at the same time as the first start time, and the first end time and After adjusting the playback speed of the first avatar image so that it ends at the same point in time, the first avatar image with the adjusted playback speed is inserted into the preset insertion position of the first recorded image and synthesized into one image, The sign language image can be generated.

In addition, according to an embodiment of the present invention, in the method of operating the electronic terminal device, in step S450, the recorded image for each slide is inserted as an object into the plurality of slides, and the recorded image for each slide is inserted into the first slide. After a sign language video is additionally inserted as an object, if a slideshow execution command for the presentation document is granted by the user, a query message inquiring whether to play the recorded video inserted as an object in the plurality of slides. Displaying on the screen, when a play command instructing to play the recorded video is issued by the user, the recorded video inserted as an object in the plurality of slides is displayed according to the page order of the plurality of slides. They are played sequentially and displayed on the screen, and when it is the turn to play the first recorded video inserted into the first slide among the plurality of slides, whether to play the first recorded video or the sign language video. displaying a video selection query message querying a screen, and when a recorded video playback command instructing to play the first recorded video is applied by the user, playing the first recorded video, and playing the sign language video. When a sign language video playback command instructing to play is applied, the step of playing the sign language video may be further included.

Above, a method of operating an electronic terminal device according to an embodiment of the present invention has been described with reference to FIG. 4 . Here, the method of operating the electronic terminal device according to an embodiment of the present invention may correspond to the configuration of the operation of the electronic terminal device 110 described using FIGS. 1 to 3, so a detailed description thereof is omitted. I decided to do it.

The method of operating an electronic terminal device according to an embodiment of the present invention may be implemented as a computer program stored in a storage medium to be executed through combination with a computer.

Additionally, the method of operating an electronic terminal device according to an embodiment of the present invention may be implemented in the form of computer program instructions for execution through combination with a computer and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and constructed for the present invention or may be known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , those skilled in the art can make various modifications and variations from this description.

Accordingly, the spirit of the present invention should not be limited to the described embodiments, and the scope of the patent claims described below as well as all modifications that are equivalent or equivalent to the scope of this patent claim shall fall within the scope of the spirit of the present invention. .

110: Electronic terminal device
111: Recorded video generation unit 112: Text conversion unit
113: Avatar image generation unit 114: Sign language image generation unit
115: video insertion unit 116: avatar video clip storage unit
117: word extraction unit 118: word matching unit
119: Generation processing unit 120: Dictionary database
121: Similarity calculation unit 122: Matching processing unit
123: Playback query part 124: Video selection query part
125: Video playback unit
130: user

Claims (12)

In the electronic terminal device capable of producing a sign language presentation video for a presentation document,
When a command to record a presentation video for a presentation document is issued by a user, the user's presentation voice for each of a plurality of slides constituting the presentation document is input through a microphone mounted on the electronic terminal device, a recorded video generator that generates a recorded video for each of the plurality of slides based on the presentation audio;
When a command to create an additional sign language video for the first slide, which is one of the plurality of slides, is applied by the user, voice is extracted from the first recorded video for the first slide, and a preset STT (Speech a text conversion unit that converts the extracted voice into text by performing voice recognition on the extracted voice based on a (To Text) model;
an avatar image generator that generates a first avatar image having an action to express the text in sign language when the extracted voice is converted into text;
When the first avatar image is generated, a sign language image generator generates a sign language image by inserting the first avatar image into a preset insertion position of the first recorded image and compositing it into one image; and
A video insertion unit that inserts the recorded video for each slide as an object into each of the plurality of slides, and additionally inserts the sign language video as an object into the first slide.
Electronic terminal device including. According to paragraph 1,
The avatar image generator
an avatar video clip storage unit in which a plurality of preset representative words for sign language translation and pre-produced avatar video clips having an operation for expressing each of the plurality of representative words in sign language are stored in correspondence with each other;
When the extracted voice is converted into the text, a word extractor for extracting at least one word constituting the text from the text;
When the at least one word is extracted, a word matching unit that matches each of the at least one word with a similar representative word that has the maximum similarity to each word among the plurality of representative words one by one; and
For each of the at least one word, an avatar video clip corresponding to a similar representative word matching each word is extracted from the avatar video clip storage unit, thereby extracting an avatar video clip corresponding to each of the at least one word. , a generation processing unit that generates the first avatar image by splicing the avatar image clips corresponding to each of the at least one word according to the order in which each word is located in the text.
Electronic terminal device including. According to paragraph 2,
The word matching unit
A plurality of preset words, and a preset embedding vector corresponding to each of the plurality of words - the embedding vector corresponding to each of the plurality of words is a vector preset based on similarity between words, The vector similarity between the embedding vectors corresponding to the words is set to be calculated as a large value as the meaning between words is similar to each other. - A dictionary database in which is stored;
When the at least one word is extracted, by referring to the dictionary database, for each of the at least one word, the vector similarity between the embedding vector corresponding to each word and the embedding vector corresponding to each of the plurality of representative words is calculated. a similarity calculation unit that calculates the similarity between each of the at least one word and the plurality of representative words; and
A matching processing unit that matches, for each of the at least one word, one similar representative word with the maximum degree of similarity to each word among the plurality of representative words.
Electronic terminal device including. According to paragraph 1,
The sign language image generator
When the first avatar image is generated, the first start time when the voice starts and the first end time when the voice ends in the first recorded video are checked, and the first start time and the first end time are confirmed. 1 Based on the end time, the playback speed of the first avatar image is adjusted so that the first avatar image starts at the same time as the first start time and ends at the same time as the first end time. Then, the electronic terminal device is characterized in that the sign language image is generated by inserting the first avatar image with the playback speed adjusted at the preset insertion position of the first recorded image and compositing it into one image. According to paragraph 1,
After the recorded video for each slide is inserted as an object into the plurality of slides by the video insertion unit, and the sign language video is additionally inserted as an object into the first slide, the user enters the presentation document. a playback query unit that displays a query message on the screen inquiring whether to play a recorded video inserted as an object in the plurality of slides when a slideshow execution command is applied;
When a play command instructing to play a recorded video is issued by the user, the recorded video inserted as an object in the plurality of slides is sequentially played and displayed on the screen according to the page order of the plurality of slides. However, when it is the turn to play the first recorded video inserted into the first slide among the plurality of slides, a video selection query message inquiring whether to play the first recorded video or the sign language video is sent. An image selection query unit to be displayed on the screen; and
When a recorded video playback command instructing to play the first recorded video is applied by the user, the first recorded video is played, and when a sign language video playback command instructing to play the sign language video is applied, the sign language video is played. Video playback unit that plays video
An electronic terminal device further comprising: In a method of operating an electronic terminal device capable of producing a sign language presentation video for a presentation document,
When a command to record a presentation video for a presentation document is issued by a user, the user's presentation voice for each of a plurality of slides constituting the presentation document is input through a microphone mounted on the electronic terminal device, generating a recorded video for each of the plurality of slides based on the presentation audio;
When a command to create an additional sign language video for the first slide, which is one of the plurality of slides, is applied by the user, voice is extracted from the first recorded video for the first slide, and a preset STT (Speech Converting the extracted voice into text by performing voice recognition on the extracted voice based on a (To Text) model;
When the extracted voice is converted into the text, generating a first avatar image having an action of expressing the text in sign language;
When the first avatar image is generated, generating a sign language image by inserting the first avatar image into a preset insertion position of the first recorded image and compositing it into one image; and
Inserting the recorded video for each slide as an object into each of the plurality of slides, and additionally inserting the sign language video as an object into the first slide.
A method of operating an electronic terminal device comprising: According to clause 6,
The step of generating the first avatar image is
Maintains an avatar video clip storage unit in which a plurality of preset representative words for sign language translation and pre-produced avatar video clips with actions to express each of the plurality of representative words in sign language are stored in correspondence with each other. steps;
When the extracted voice is converted into the text, extracting at least one word constituting the text from the text;
When the at least one word is extracted, matching each of the at least one word with a similar representative word having the maximum degree of similarity to each word among the plurality of representative words one by one; and
For each of the at least one word, an avatar video clip corresponding to a similar representative word matching each word is extracted from the avatar video clip storage unit, thereby extracting an avatar video clip corresponding to each of the at least one word. , generating the first avatar image by splicing the avatar image clips corresponding to each of the at least one word according to the order in which each word is located in the text.
A method of operating an electronic terminal device comprising: In clause 7,
The matching step is
A plurality of preset words, and a preset embedding vector corresponding to each of the plurality of words - the embedding vector corresponding to each of the plurality of words is a vector preset based on similarity between words, Maintaining a dictionary database in which the vector similarity between the embedding vectors corresponding to the words is set to be calculated as a large value as the meaning between the words is similar to each other;
When the at least one word is extracted, by referring to the dictionary database, for each of the at least one word, the vector similarity between the embedding vector corresponding to each word and the embedding vector corresponding to each of the plurality of representative words is calculated. calculating the degree of similarity between each of the at least one word and the plurality of representative words by calculating; and
For each of the at least one word, matching similar representative words with the maximum degree of similarity to each word among the plurality of representative words one by one.
A method of operating an electronic terminal device comprising: According to clause 6,
The step of generating the sign language image is
When the first avatar image is generated, the first start time when the voice starts and the first end time when the voice ends in the first recorded video are checked, and the first start time and the first end time are confirmed. 1 Based on the end time, the playback speed of the first avatar image is adjusted so that the first avatar image starts at the same time as the first start time and ends at the same time as the first end time. Afterwards, the first avatar image with the playback speed adjusted is inserted into the preset insertion position of the first recorded image and synthesized into one image, thereby generating the sign language image. . According to clause 6,
By the inserting step, the recorded video for each slide is inserted as an object in the plurality of slides, and the sign language video is additionally inserted as an object in the first slide, and then added to the presentation document by the user. When a slideshow execution command is approved, displaying a query message on the screen asking whether to play a recorded video inserted as an object in the plurality of slides;
When a play command instructing to play a recorded video is issued by the user, the recorded video inserted as an object in the plurality of slides is sequentially played and displayed on the screen according to the page order of the plurality of slides. However, when it is the turn to play the first recorded video inserted into the first slide among the plurality of slides, a video selection query message inquiring whether to play the first recorded video or the sign language video is sent. displaying on a screen; and
When a recorded video playback command instructing to play the first recorded video is applied by the user, the first recorded video is played, and when a sign language video playback command instructing to play the sign language video is applied, the sign language video is played. Steps to play video
A method of operating an electronic terminal device further comprising: A computer-readable recording medium recording a computer program for executing the method of any one of claims 6 to 10 through combination with a computer. A computer program stored in a storage medium for executing the method of any one of claims 6 to 10 through combination with a computer.

Images