KR102488623B1 - Method and system for suppoting content editing based on real time generation of synthesized sound for video content - Google Patents

Abstract

Disclosed is a method and system for supporting content editing based on real-time generation of synthesized sound for video content. A content editing support method according to an embodiment includes extracting a plurality of snapshots from video content selected through a content editing tool, displaying the plurality of extracted snapshots according to a timeline through the content editing tool, converting the input text into a voice of the selected voice type based on the text input through the content editing tool and the voice type selected through the content editing tool, and a time point selected for the timeline by the content editing tool It may include adding the converted voice to.

Description

Method and system for supporting content editing based on real-time generation of synthesized sound for video content

The following description relates to a method and system for supporting content editing based on real-time generation of synthesized sound for video content.

There are various technologies for editing video content. For example, Korean Patent Laid-open Publication No. 10-2007-0013122 relates to a video subtitle editing method, in which a DVD recorder plays a video file recorded on an optical disc while editing the subtitle part of a subtitle file linked to the video in real time. A method for editing video subtitles is disclosed.

However, while it is relatively easy to edit video, subtitles, audio, etc. already possessed by video content, it is relatively easy to add a new synthesized sound, such as a specific voice or sound effect, to what the user wants to video content. Not only is it not easy to do this, but it is also not easy to add the acquired synthesized sound to a desired part of the video content.

Provided is a method and system for supporting content editing in which a synthesized sound desired by a user can be generated in real time and added to a desired part of video content.

A content editing support method for a computer device including at least one processor, comprising: extracting, by the at least one processor, a plurality of snapshots from video content selected through a content editing tool; displaying, by the at least one processor, the plurality of extracted snapshots according to a timeline through the content editing tool; converting, by the at least one processor, the input text into a voice of the selected voice type based on text input through the content editing tool and a voice type selected through the content editing tool; and adding, by the at least one processor, the converted voice at a time point selected for the timeline in the content editing tool.

According to one side, the step of adding the converted voice may be characterized in that the converted voice is added at a point in time selected by dragging a time indicator indicating a specific point in time on the timeline.

According to another aspect, the method for supporting content editing may include providing, by the at least one processor, a copy or cut function for the added voice; and adding, by the at least one processor, the audio copied or cut through the function at a time point selected for the timeline in the content editing tool.

According to another aspect, the selected voice type may be selected from a plurality of voice types predefined according to the harmony of voice and emotion and provided through the content editing tool.

According to another aspect, the content editing support method further includes providing, by the at least one processor, a function of generating a user's voice type through a content editing tool, wherein the function includes a predefined voice type. It may be characterized in that it includes a function of generating a new voice type by adjusting at least one of a speech speed and a speech volume.

According to another aspect, at least some of the contents of the text corresponding to the added voice may be displayed in association with a time point selected for the timeline in the content editing tool.

According to another aspect, according to a user's input to the region where the at least part of the content is displayed, among the content of the text corresponding to the added voice, the voice type corresponding to the added voice, and the start time of the added voice. It may be characterized in that a function for editing at least one is provided.

According to another aspect, the method for supporting content editing may include receiving, by the at least one processor, selecting one of a plurality of sound effects provided through the content editing tool; and adding, by the at least one processor, the selected sound effect at a time point selected for the timeline in the content editing tool.

According to another aspect, information on the added sound effect is displayed in association with a time point selected for the timeline in the content editing tool, and the type of the added sound effect according to a user's input to the area where the information is displayed. And it may be characterized in that a function for editing at least one of the start time is provided.

According to another aspect, the method for supporting content editing may further include recommending, by the at least one processor, a viewpoint on the timeline to add the converted audio by analyzing the video content.

According to another aspect, the content editing support method analyzes an image of a time point selected for the timeline in the video content by the at least one processor, and recommends at least one of a voice type and a sound effect for the selected time point. It may further include steps to do.

According to another aspect, the converting may include checking a length of a section to which the converted voice can be added in a timeline based on a selected time point on the timeline; and limiting the length of the input text or adjusting the speech speed of the converted voice according to the length of the checked section.

According to another aspect, the converting may include checking a volume of a background sound for a time point selected for the timeline in the video content; and adjusting the speech volume of the converted voice according to the confirmed volume of the background sound.

According to another aspect, the adding of the converted voice may include adding the converted voice after removing a voice of a speaker at a time point selected for the timeline from the video content.

According to another aspect, the method for supporting content editing may further include extracting audio from the video content and converting it into text, by the at least one processor; converting the converted text into a voice of a voice type selected through the content editing tool, by the at least one processor; and adding, by the at least one processor, the converted audio to a section in which audio is extracted from the video content.

According to another aspect, the converting may include providing at least one of a first function of correcting the input text and a second function of translating the input text into characters of another language; and converting the text corrected through the first function or translated through the second function into a voice of the selected voice type.

According to another aspect, the content editing support method provides, by the at least one processor, a thumbnail corresponding to a time point indicated by a time indicator on the timeline and a voice in syllable units corresponding to a time point indicated by the time indicator. The method may further include providing at least one of text information about.

A computer program stored in a computer readable recording medium is provided in combination with a computer device to execute the method on the computer device.

A computer readable recording medium having a program for executing the method in a computer device is recorded.

It includes at least one processor implemented to execute instructions readable by a computer, and by the at least one processor, a plurality of snapshots are extracted from video content selected through a content editing tool, and the plurality of snapshots are extracted. is displayed according to a timeline through the content editing tool, and based on the text input through the content editing tool and the voice type selected through the content editing tool, the input text is converted into a voice of the selected voice type, , It provides a computer device characterized in that for adding the converted voice at the time point selected for the timeline in the content editing tool.

A synthesized sound desired by the user can be generated in real time and added to a desired part of video content.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
2 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention.
3 is a diagram illustrating an example of a content editing support environment according to an embodiment of the present invention.
4 is a diagram showing an example of a screen of a content editing tool according to an embodiment of the present invention.
5 is a diagram showing an example of a sound effect editing function in one embodiment of the present invention.
6 is a diagram illustrating an example of a voice editing function according to an embodiment of the present invention.
7 is a diagram illustrating an example of a function of editing a voice list according to an embodiment of the present invention.
8 is a diagram illustrating an example of a function of editing a voice type according to an embodiment of the present invention.
9 is a diagram illustrating an example of a function for generating a user's own voice type according to an embodiment of the present invention.
10 is a flowchart illustrating an example of a content editing support method according to an embodiment of the present invention.
11 is a diagram illustrating an example of providing information about a time point indicated by a time indicator according to an embodiment of the present invention.

Hereinafter, an embodiment will be described in detail with reference to the accompanying drawings.

A content editing support system according to embodiments of the present invention may be implemented by at least one computer device, and a content editing support method according to embodiments of the present invention includes at least one computer device included in the content editing support system. can be performed through A computer program according to an embodiment of the present invention may be installed and driven in a computer device, and the computer device may perform a content editing support method according to embodiments of the present invention under the control of the driven computer program. The above-described computer program may be combined with a computer device and stored in a computer readable recording medium to execute a content editing support method on a computer.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110 , 120 , 130 , and 140 , a plurality of servers 150 and 160 , and a network 170 . 1 is an example for explanation of the invention, and the number of electronic devices or servers is not limited as shown in FIG. 1 . In addition, the network environment of FIG. 1 only describes one example of environments applicable to the present embodiments, and the environment applicable to the present embodiments is not limited to the network environment of FIG. 1 .

The plurality of electronic devices 110, 120, 130, and 140 may be fixed terminals implemented as computer devices or mobile terminals. Examples of the plurality of electronic devices 110, 120, 130, and 140 include a smart phone, a mobile phone, a navigation device, a computer, a laptop computer, a digital broadcast terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). ), and tablet PCs. As an example, FIG. 1 shows the shape of a smartphone as an example of the electronic device 110, but in the embodiments of the present invention, the electronic device 110 substantially uses a wireless or wired communication method to transmit other information via the network 170. It may refer to one of various physical computer devices capable of communicating with the electronic devices 120 , 130 , and 140 and/or the servers 150 and 160 .

The communication method is not limited, and short-distance wireless communication between devices as well as a communication method utilizing a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, and a broadcasting network) that the network 170 may include may also be included. For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , one or more arbitrary networks such as the Internet. In addition, the network 170 may include any one or more of network topologies including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, and the like. Not limited.

Each of the servers 150 and 160 communicates with the plurality of electronic devices 110, 120, 130, and 140 through the network 170 to provide commands, codes, files, contents, services, and the like, or a computer device or a plurality of computers. Can be implemented in devices. For example, the server 150 provides a service (eg, content providing service, group call service (or voice conference service)) to a plurality of electronic devices 110, 120, 130, and 140 connected through the network 170. , messaging service, mail service, social network service, map service, translation service, financial service, payment service, search service, etc.).

2 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention. Each of the plurality of electronic devices 110 , 120 , 130 , and 140 or each of the servers 150 and 160 described above may be implemented by the computer device 200 shown in FIG. 2 .

As shown in FIG. 2 , the computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input/output interface 240. The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device 200 as a separate permanent storage device distinct from the memory 210 . Also, an operating system and at least one program code may be stored in the memory 210 . These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 210 through the communication interface 230 rather than a computer-readable recording medium. For example, software components may be loaded into memory 210 of computer device 200 based on a computer program installed by files received over network 170 .

The processor 220 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 220 by memory 210 or communication interface 230 . For example, processor 220 may be configured to execute received instructions according to program codes stored in a recording device such as memory 210 .

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (eg, storage devices described above) through the network 170 . For example, a request, command, data, file, etc. generated according to a program code stored in a recording device such as the memory 210 by the processor 220 of the computer device 200 is controlled by the communication interface 230 to the network ( 170) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 170 . Signals, commands, data, etc. received through the communication interface 230 may be transferred to the processor 220 or the memory 210, and files, etc. may be stored as storage media that the computer device 200 may further include (described above). permanent storage).

The input/output interface 240 may be a means for interface with the input/output device 250 . For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input/output interface 240 may be a means for interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 250 and the computer device 200 may be configured as one device.

Also, in other embodiments, computer device 200 may include fewer or more elements than those of FIG. 2 . However, there is no need to clearly show most of the prior art components. For example, the computer device 200 may be implemented to include at least some of the aforementioned input/output devices 250 or may further include other components such as a transceiver and a database.

3 is a diagram illustrating an example of a content editing support environment according to an embodiment of the present invention. 3 shows a content editing support server 300 , a plurality of users 310 and a content editing tool 320 .

The content editing support server 300 may be implemented by at least one computer device 200, and provides a content editing tool 320 to a plurality of users 310 so that the plurality of users 310 can edit the content. A plurality of users 310 may be supported to edit their own image content using the tool 320 .

A plurality of users 310 may receive the content editing tool 320 from the content editing support server 300 and edit their own video content. In this case, each of the plurality of users 310 may be physical electronic devices that substantially access the content editing support server 300 through the network 170 and receive the content editing tool 320 . Each of these physical electronic devices may also be implemented by the computer device 200 described above with reference to FIG. 2 .

The content editing tool 320 may be provided to a plurality of users 310 in a web method or an app method. In the web method, a plurality of users 310 visit a web page in which the function of the content editing tool 320 is implemented and provided by the content editing support server 300, and a function for editing content through the corresponding web page is provided. It can mean the way it is provided. The app method may refer to a method in which a function for content editing is provided by accessing the content editing support server 300 through an application installed and driven in each of physical electronic devices corresponding to a plurality of users 310. there is. According to embodiments, each of the physical electronic devices corresponding to the plurality of users 310 may independently process content editing through an application including a content editing function.

The content editing tool 320 according to this embodiment may include a function of extracting snapshots from video content selected by a user and displaying the extracted snapshots according to a timeline. At this time, the content editing tool 320 may provide a function for the user to select a desired time point or section on the timeline, and provide a user interface capable of associating arbitrary text desired by the user with the selected time point or section. can do. When any text is associated with the selected time or section, the content editing tool 320 automatically converts the linked text into voice and adds the converted voice to the selected time or section, so that users can easily and conveniently It can support dubbing the audio of the content.

4 is a diagram showing an example of a screen of a content editing tool according to an embodiment of the present invention. FIG. 4 shows a screen example 400 of the content editing tool 320 described with reference to FIG. 3 . The configuration of the content editing tool 320 according to this embodiment is one embodiment, and the configuration may be variously changed according to the embodiment.

The user can access the content editing tool 320 through his or her electronic device, and the content editing tool 320 has a content playback function 410 for reproducing the video content selected by the user and the selected video content to another video. A content change function 411 for changing to content may be included. In this case, the video content selected by the user may include video content stored in a local storage of an electronic device used by the user to access the content editing tool 320 . Depending on the embodiment, the video content selected by the user may be video content located on the web instead of the local storage of the electronic device.

In addition, the content editing tool 320 may include a text-to-speech conversion function 420 for converting text into voice and adding it to a timeline for video content. The text-to-speech conversion function 420 may include a text input function 421 for receiving text from a user and a speech type selection function 422 for selecting a voice type to convert input text to. The voice type may be identified according to a combination of a specific specific voice (eg, the voice of a predefined character 'K1') and emotion (eg, basic emotion, sadness, joy, etc.). In FIG. 4 , only “default”, “sadness”, and “joy” are shown as examples of emotions, but more diverse emotions (eg, “anger”, “frustration”, etc.) may be used according to embodiments. Depending on the embodiment, the voice type may be differently identified according to the speed and volume of speech. In other words, voice types having different speech speeds or volumes may be identified as different voice types, even if voice types have the same emotion and the same voice. Actual text-to-speech conversion may be performed through an automatic text-to-speech conversion technology such as TTS (Text-To-Speech). In addition, the text-to-speech conversion function 420 includes a pre-listening function 423 for pre-listening by converting the input text into a voice of a selected voice type, and a video content by converting the input text into a voice of the selected voice type. It may further include a voice add function 424 for adding to the timeline for. For example, the voice add function 424 is selected by the user (for example, in a PC environment, the user clicks an area corresponding to the voice add function 424 with a mouse, or in a touch screen environment, the user uses a finger to add a voice function. (424), the content editing tool 320 converts the text input by the user through the text input function 421 into a voice of the voice type selected through the voice type selection function 422. After that, you can add it to the timeline for video content. Where the corresponding voice is added to the timeline will be described in more detail later.

In addition, the content editing tool 320 may include a sound effect add function 430 for adding pre-produced sound effects to the timeline for video content. The sound effect addition function 430 may include functions for displaying a list of a plurality of pre-produced sound effects, pre-listening to the sound effects, or adding a sound effect at a specific time on the timeline. If necessary, the user can add the desired sound effect from an external file or create it directly.

In addition, the content editing tool 320 may include a timeline display function 440 for visually expressing a timeline for image content. The timeline display function 440 may include a function for displaying thumbnails extracted from video content selected by a user according to time. The time applied to the timeline may correspond to the time of video content, or may be limited to a predefined time (eg, 5 minutes) according to embodiments. For example, the content editing tool 320 may extract thumbnails from frames of 5 minutes in video content and display them according to time through the timeline display function 440 . At this time, the user can add synthesized sounds such as voices or sound effects to a desired time point or section while viewing thumbnails of the timeline display function 440 . For example, a user's desired time point may be set by the time indicator 450 . The user may set a desired viewpoint on the timeline by moving the time indicator 450 by dragging or clicking a desired position on the timeline. In FIG. 4 , “02:16.46” displayed in association with the time indicator 450 may indicate a point in time indicated by the time indicator 450 on the timeline.

When the user selects the voice addition function 424 included in the text-to-speech conversion function 420, the text input through the text input function 421 is voiced according to the voice type selected through the voice type selection function 422. In this case, the converted voice may be added at the time indicated by the time indicator 450.

As another embodiment, when the user selects the voice add function 424, the type indicator 450 indicates the test input through the text input function 421 and the voice type selected through the voice type selection function 422. It can be stored in response to the time of doing it. Using such information, inputted text can be converted into voice according to the selected voice type and provided when necessary (for example, when pre-listening or video creation is requested).

In addition, the content editing tool 320 may display a section in the timeline to which synthesized sounds such as voices or sound effects are added. For example, a dotted line box 460 indicates a section for voices added to the timeline, and a dotted line box 470 indicates a section for sound effects added to the timeline. For example, a section corresponding to “text 1” in the dotted line box 460 may mean a section to which a voice converted according to text 1 input by a user through the text input function 421 is added. At least a part of the content of the actual “Text 1” is displayed at the position of the corresponding section, so that the user can easily recognize the content of the added voice.

In this way, users can edit the information already included in the video content by easily adding desired synthesized sound to the video content by generating a voice in real time through the content editing tool 320 or using a pre-generated sound effect. Instead, content editing such as adding a new synthesized sound can be easily processed.

In addition, the content editing tool 320 may further include a storage function 480 for storing video content to which synthesized sounds are added and a download function 490 for downloading the video content to which synthesized sounds are added to the user's electronic device. there is.

Hereinafter, more various functions that the content editing tool 320 can provide will be described.

5 is a diagram showing an example of a sound effect editing function in one embodiment of the present invention. FIG. 5 shows a portion 500 of the screen example 400 described above with reference to FIG. 4 . When the user selects a sound effect displayed in association with the timeline in the content editing tool 320, a sound effect editing function for the selected sound effect may be provided. The embodiment of FIG. 5 shows an example in which a sound effect editing function is provided, such as a dotted line box 520, when the user selects sound effect 2 (510). For example, when a user in a PC environment places a mouse on an area where sound effect 2 510 is displayed, a sound effect editing function may be provided, such as a dotted line box 520 . The sound effect editing function includes a viewpoint change function (521) for changing the starting point of the sound effect 2 (510) on the timeline and an editing function (522) for copying, cutting and/or deleting the sound effect 2 (510). can include Sound effects copied or cut can be pasted at different points in the timeline. For example, the copied or cut sound effect 2 510 can be pasted according to the user's input at the time indicated by the time indicator 450 described in FIG. 4 .

6 is a diagram illustrating an example of a voice editing function according to an embodiment of the present invention. FIG. 6 shows a portion 600 of the screen example 400 described above with reference to FIG. 4 . When the user selects a voice displayed in association with the timeline in the content editing tool 320, a sound effect editing function for the selected voice may be provided. The embodiment of FIG. 6 shows an example in which a voice editing function is provided, such as a dotted line box 620 when the user selects text 3 610 . Practically, the user can edit the voice by editing the content of text 3 (610). For example, when the user changes the content of text 3 610, the content editing tool 320 may replace the existing voice by reconverting the changed text into a voice of a previously selected voice type.

As another example, conversion of text to voice may be delayed until voice provision is required (eg, when a pre-listening or video creation request is requested). In this case, the user can freely change the content of the text by updating text and voice types corresponding to a specific time point.

The function of editing the contents of the text may be performed through the text input function 421 of the text-to-speech function 420 described above with reference to FIG. 4 or through a separate voice list as shown in FIG. 7 described later. there is.

In addition, the user may change the voice type as well as the content of the text. Depending on the embodiment, the content of the text may be reconverted into the content of the changed voice type to replace the existing voice, or the text and voice types corresponding to a specific point in time may be updated and stored.

Such voice type change may be made through the voice type selection function 422 of the text-to-speech conversion function 420 described with reference to FIG. 4 or may be made in a separate voice list as shown in FIG. 7 described later. As an example, the content editing tool 320 displays text 3 610 when the user selects another voice type through the voice type selection function 422 or selects another voice type from the voice list described in FIG. 7 . The contents can be reconverted according to the selected voice type, and the existing voice can be replaced with the reconverted voice. Depending on embodiments, functions for copying, cutting, and/or deleting voices may be provided.

7 is a diagram illustrating an example of a function of editing a voice list according to an embodiment of the present invention. 7 shows a screen example 700 of an editing function of a voice list that can be provided by the content editing tool 320 . The first dotted line box 710 represents the voice type (for example, the voice font of FIG. 7) set for each voice included in the voice list, and the second dotted line box 720 represents the start time of the individual voices on the timeline. , The third dotted line box 730 indicates the contents of the text corresponding to the individual voices, and the fourth dotted line box 740 indicates the length of the individual voices. In this case, the content editing tool 320 may provide a function for changing the voice type for each individual voice displayed in the first dotted line box 710 to another voice type. In addition, the content editing tool 320 may provide a function for editing the content of the text displayed in the third dotted line box 730 . Also, a fifth dotted line box 750 represents a function for pre-listening to individual voices, and a sixth dotted line box 760 represents a function for deleting individual voices. Also, the editing function of the voice list may include a function for generating a new voice. A seventh dotted line box 770 represents a function for setting a voice type, a start time, and text content in order to generate a new voice.

8 is a diagram illustrating an example of a function of editing a voice type according to an embodiment of the present invention. 8 shows a screen example 800 of a voice-type editing function that the content editing tool 320 can provide. The function of editing a voice type may include a function for selecting a voice type to be mainly used among various voice types and a function for listening to a voice of a specific voice type. For example, when the user selects the voice type K1-joy 810, inputs text through the text input function 820, and presses the preview button 830, the input text is converted to the voice type K1-joy 810. According to the joy 810, the converted voice can be listened to in advance.

7 and 8 show a function of editing a voice list and a function of editing a voice type, respectively, but according to an embodiment, the function of editing the voice list of FIG. 7 and the function of editing the voice type of FIG. By implementing through a window or a page, the user can more easily edit the text content and voice type for each voice.

9 is a diagram illustrating an example of a function for generating a user's own voice type according to an embodiment of the present invention. The function of generating a voice type includes a function of receiving the name of the voice type to be created (910), a function of selecting a desired voice type from among predefined voice types (920), and a function of adjusting the speed of speech (930). , a function 940 for adjusting the volume of speech, and a function 950 for pre-listening to the generated voice type. The generated voice type may be stored in association with the corresponding user, and the user may check and/or select the voice type generated by the user in 'my voice font' shown in FIG. 8 , for example.

10 is a flowchart illustrating an example of a content editing support method according to an embodiment of the present invention. The method for supporting content editing according to the present embodiment may be performed by the computer device 200 that provides a service for supporting content editing through the content creation tool 300 . In this case, the processor 220 of the computer device 200 may be implemented to execute a control instruction according to an operating system code included in the memory 210 or a code of at least one computer program. Here, the processor 220 controls the computer device 200 so that the computer device 200 performs steps 1010 to 1060 included in the method of FIG. 10 according to a control command provided by a code stored in the computer device 200. can control.

In step 1010, the computer device 200 may extract a plurality of snapshots from video content selected through a content editing tool. The content editing tool may correspond to the content editing tool 320 described above with reference to FIG. 3 . When a user selects video content through a content editing tool, the computer device 200 may extract a plurality of snapshots from the video content. For example, the computer device 200 may extract frames at regular intervals from video content as a plurality of snapshots or analyze the video content to extract frames including characters or specific scenes or actions. . As another example, the computer device 200 may extract a frame of a section in which audio changes from video content as a snapshot.

In step 1020, the computer device 200 may display the plurality of extracted snapshots according to a timeline through a content editing tool. For example, in FIG. 4 , an example in which a plurality of snapshots are displayed according to a timeline through the timeline display function 440 has been described.

In step 1030, the computer device 200 may convert the input text into a voice of the selected voice type based on the text input through the content editing tool and the voice type selected through the content editing tool. The voice type may be selected from a plurality of voice types predefined according to the harmony of voice and emotion and provided through a content editing tool. For example, when a user selects one of a plurality of voice types displayed through a content editing tool, the computer device 200 may check the selected voice type. Converting text to voice may be performed through an automatic text-to-speech conversion technology such as TTS, as described above.

Depending on the embodiment, the computer device 200 checks the length of a section to which the voice converted in the timeline can be added based on the selected time point on the timeline, limits the length of text input according to the length of the checked section, or Alternatively, the speech rate of the converted voice may be adjusted. For example, when text is input to add a voice at a time point selected on the timeline, the length of the text is too long, so the length of the converted voice may exceed the length of an addable section. In this case, the computer device 200 may control the length of the converted voice so that it does not exceed the length of the additional section by limiting the length of the input text or adjusting the speech speed of the converted voice relatively faster. In addition, the length of text that can be input in the corresponding section and the expected synthesized voice time may be provided to the user in advance.

In another embodiment, the computer device 200 may check the volume of the background sound for a time point selected on the timeline in the video content, and adjust the speech volume of the converted voice according to the checked volume of the background sound. For example, when the size of the background sound in the video content is too large, there may be cases in which the added sound is difficult to hear. Accordingly, the computer device 200 may convert text into voice so that the added voice increases in volume as the background sound increases.

In another embodiment, the computer device 200 may provide at least one function of a first function of correcting the input text and a second function of translating the input text into characters of another language. The text corrected or translated through the second function may be converted into a voice of the selected voice type. In other words, the computer device 200 may provide the user with a function for correcting the text or converting the text into another language through a content editing tool before converting the text into voice. In the case of the first function of correcting the input text, it corrects spelling or grammatical errors in the input text, or inputs in a form that is more advantageous for speech synthesis (for example, adding spaces to necessary parts, or words that cannot be synthesized well). It can include a function of recognizing prohibited words such as profanity or profanity and excluding them so that synthesis is impossible.

In step 1040, the computer device 200 may add the converted voice at the time point selected for the timeline in the content editing tool. For example, the computer device 200 may add the converted voice at a selected point in time by dragging a time indicator indicating a specific point in time on the timeline. The time indicator may correspond to the time indicator 450 described above with reference to FIG. 4 .

Meanwhile, at least some of the contents of the text corresponding to the added voice may be displayed in association with a time point selected for the timeline in the content editing tool. For example, FIG. 4 shows an example in which at least some of text contents are displayed in association with a time point selected for the corresponding text on a timeline through a dotted line box 460 . At this time, a function for editing at least one of text content corresponding to the added voice, voice type corresponding to the added voice, and start time of the added voice according to a user's input to an area where at least some of the content is displayed is provided. can be provided. As an example, in FIG. 6 , an example in which a voice editing function is provided according to a mouse-over input on an area where at least a part of content is displayed has been described.

Also, in another embodiment, the computer device 200 may remove a speaker's voice for a time point selected on the timeline from video content and then add the converted voice. In other words, the speaker's voice in the video content can be replaced with the converted voice.

In step 1050, the computer device 200 may receive a selection of at least one sound effect from among a plurality of sound effects provided through the content editing tool. For example, in FIG. 4 , functions for displaying a list of a plurality of pre-produced sound effects through the sound effect add function 430, pre-listening to the sound effects, or adding sound effects at a specific time on the timeline are provided. can include

In step 1060, the computer device 200 may add the selected sound effect at the time selected for the timeline in the content editing tool. Depending on the embodiment, these steps 1050 and 1060 may be performed in parallel with the previously described steps 1030 and 1040 or may be omitted. Alternatively, steps 1030 and 1040 may be omitted.

In addition, according to an embodiment, the computer device 200 may provide a copy or cut function for the added voice, and copy or cut the voice through the function at the time selected for the timeline in the content editing tool. can also be added.

In another embodiment, the computer device 200 may provide a function of generating a user's voice type through a content editing tool. In this case, the function may include a function of generating a new voice type by adjusting at least one of a speech speed and a speech volume of a predefined voice type.

In another embodiment, the computer device 200 may recommend a viewpoint on the timeline to add the converted voice by analyzing the video content. For example, the computer device 200 may analyze video content to find a time point when a character appears, a time point when a specific scene or action appears, or a section where voice characteristics change. Using this analyzed information, it is recommended to the user as a time to add voice or sound effects, provided as a search result for a specific keyword (eg character name, scene content to be searched for, etc.), or inserted content. When the indicator to be moved, it can be easily moved based on the point in time.

In another embodiment, the computer device 200 may recommend at least one of a voice type and a sound effect for the selected time point by analyzing an image of a time point selected for a timeline in video content. For example, when a user selects a specific point in time through a time indicator, the computer device 200 analyzes an image of the selected point in time and recommends to the user an audio type for a voice to be added at that point in time or an effect sound to be added at that point in time. can do. As a more specific example, the computer device 200 may recommend a voice type according to the character's gender, age group, body shape or appearance analyzed in the image, or recommend an appropriate sound effect according to a specific scene or place analyzed in the image.

In another embodiment, the computer device 200 may extract audio from video content, convert it into text, and convert the converted text into audio of a selected audio type through a content editing tool. Thereafter, the computer device 200 may add the converted audio to a section in which the audio is extracted from the video content. In other words, the computer device 200 may provide a function of replacing voices and/or emotions of voices included in video content with other voices and/or emotions. In this case, different voices and/or different emotions may be determined according to a voice type selected through a content editing tool or a voice type selected by the computer device 200 itself.

In another embodiment, the computer device 200 may provide at least one of text information on voice in units of syllables corresponding to a time point indicated by the time indicator and a thumbnail corresponding to the time point.

11 is a diagram illustrating an example of providing information about a time point indicated by a time indicator according to an embodiment of the present invention. FIG. 11 shows a portion 1100 of the screen example 400 described with reference to FIG. 4 . As already described, the user can select a specific viewpoint on the timeline by dragging the time indicator 450 or selecting a specific viewpoint (position) on the timeline.

For example, in a PC environment, the user can move the time indicator 450 by clicking and dragging the time indicator 450 with a mouse, and in a touch screen environment, the user can move the time indicator 450 with a finger or a pen. By dragging while touching, the time indicator 450 can be moved. At this time, when the time indicator 450 is in a clicked or touched state, the content editing tool 320 displays a thumbnail 1110 corresponding to the time point indicated by the time indicator 450 and a syllable unit corresponding to the time point. Text information 1120 for voice may be provided. At least a part of the corresponding text may also be displayed in the dotted line box 460 described in FIG. 4, but this is for the user to easily recognize the contents of the voice added to the corresponding section.

Meanwhile, in the present embodiment, the text information 1120 for the voice in syllable units provided corresponding to the time point indicated by the time indicator 450 corresponds to the voice in syllable unit to be output at the time point indicated by the actual time indicator 450. Can contain text content. In the embodiment of FIG. 11, the thumbnail 1110 corresponding to the time of “01:17.12” indicated by the time indicator 450 selected by the user (for example, click or touch), and the thumbnail 1110 actually added at that time An example of displaying the text "Hello" for the voice in units of syllables is shown. If the time indicator 450 is selected by the user and dragged to another time point, the thumbnail corresponding to the other time point and the text for the voice of the syllable unit actually added at the other time point can be displayed. will be easily understandable. In this case, while dragging the time indicator 450, the user can simply check the thumbnails for various points of time and/or the text for the voice added in units of syllables for the points of time.

As such, according to embodiments of the present invention, a synthesized sound desired by a user can be generated in real time and added to a desired part of video content.

The system or device described above may be implemented as a hardware component or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The medium may continuously store programs executable by a computer or temporarily store them for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, and is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc. configured to store program instructions. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (13)

A content editing support method of a computer device including at least one processor,
extracting, by the at least one processor, a plurality of snapshots from frames included in video content selected through a content editing tool;
displaying, by the at least one processor, the plurality of extracted snapshots according to the timeline in an area of the content editing tool for visually expressing the timeline for the video content;
converting text input through the content editing tool into voice by the at least one processor; and
adding, by the at least one processor, the converted voice at a time point selected for the timeline of an area for visually expressing the timeline in the content editing tool;
Content editing support method comprising a. According to claim 1,
providing a copy or cut function for the added voice by the at least one processor; and
Adding, by the at least one processor, a voice copied or cut through the function at a time point selected for the timeline in the content editing tool.
Content editing support method further comprising a. According to claim 1,
The method of supporting content editing, characterized in that at least some of the content of the text corresponding to the added voice is displayed in association with a time point selected for the timeline in the content editing tool. According to claim 3,
Editing at least one of a text content corresponding to the added voice, a voice type corresponding to the added voice, and a start time of the added voice according to a user's input to an area where the at least part of the content is displayed. Content editing support method characterized in that the function is provided. According to claim 1,
receiving, by the at least one processor, selecting at least one sound effect from among a plurality of sound effects provided through the content editing tool; and
adding, by the at least one processor, the selected sound effect at a time point selected for the timeline in the content editing tool;
Content editing support method further comprising a. According to claim 5,
Information on the added sound effect is displayed in association with the time point selected for the timeline in the content editing tool,
and a function for editing at least one of a type and a start time of the added sound effect according to a user's input to the region where the information is displayed is provided. According to claim 1,
The conversion step is
and converting the input text into a voice of the selected voice type based on the text input through the content editing tool and the voice type selected through the content editing tool. According to claim 1,
The conversion step is
checking the length of a section to which the converted voice can be added on a timeline based on a selected time point on the timeline; and
Limiting the length of the input text or adjusting the speech speed of the converted voice according to the length of the checked section
Content editing support method comprising a. According to claim 1,
In the step of adding the converted voice,
The method of supporting content editing, characterized in that, after removing a speaker's voice for a selected time point on the timeline from the video content, the converted voice is added. According to claim 1,
Providing, by the at least one processor, at least one of a thumbnail corresponding to a time point indicated by a time indicator on the timeline and text information about voice in units of syllables corresponding to a time point indicated by the time indicator
Content editing support method further comprising a. A computer program stored in a computer readable recording medium to be combined with a computer device to execute the method of any one of claims 1 to 10 in the computer device. A computer readable recording medium on which a computer program for executing the method of any one of claims 1 to 10 is recorded on a computer device. at least one processor implemented to execute computer readable instructions
including,
by the at least one processor,
Extracting a plurality of snapshots from frames included in video content selected through a content editing tool;
Displaying the plurality of extracted snapshots according to the timeline in an area of the content editing tool for visually expressing a timeline for the video content;
Converting text input through the content editing tool into voice;
Adding the converted voice at a time point selected for the timeline in an area for visually expressing the timeline in the content editing tool.
Characterized by a computer device.

Images