JP2022145617A - Method and system for generating video content based on voice synthesis for image - Google Patents

Abstract

To provide a method and system for generating video content based on voice synthesis for an image.SOLUTION: A video content generation method includes: a step of extracting a snapshot of an image uploaded trough a content editing tool; a step of displaying the extracted snapshot along a timeline through the content editing tool; a step of providing a length adjustment function for adjusting the length of the snapshot displayed through the content editing tool; a step of adjusting a running time of the snapshot with the length adjusted by the length adjustment function, in accordance with the adjusted length; and a step of generating video synthesis for a text to be input trough the content editing tool and adding it at a point of time selected on the timeline.SELECTED DRAWING: Figure 4

Description

The following description relates to methods and systems for generating video content based on speech synthesis to images.

When applying a sound source (including text-to-speech (TTS)) to a material containing images, for example, in the case of a PowerPoint-based material, one sound source must be added to each slide. The troublesome work that must be done was required. At this time, there is a restriction that only one sound source can be added to each slide, and there is also the problem that the playback start time cannot be freely edited.

As such, the conventional video production technology using voice synthesis in the current market where the needs for production and consumption of video content have increased has the problem that it can only be provided in a cumbersome and limited form.

Korean Patent Publication No. 10-2014-0147401 (publication date: December 30, 2014)

A voice synthesis desired by the user can be generated in real time for a number of images and dubbed at a reproduction start time desired by the user, and the generated voice synthesis can be dubbed to generate and reproduce video contents. A method and system for generating video content is provided.

1. A method of generating video content for a computing device comprising at least one processor, comprising: extracting, by said at least one processor, a snapshot of an image uploaded to a content editing tool; displaying the displayed snapshot along a timeline with the content editing tool; and providing, by the at least one processor, a length adjustment function to the content editing tool to adjust the length of the displayed snapshot. adjusting, by the at least one processor, the running time of the snapshot whose length has been adjusted by the length adjustment function by the adjusted length; and, by the at least one processor, providing the content editing tool with A video content generation method is provided, comprising generating a speech synthesis for an input text and adding it to the selected time point of the timeline.

According to one aspect, the length of the displayed snapshot is proportional to the running time, which is the time occupied on the timeline by the image corresponding to the displayed snapshot, along the timeline. The displaying step may be characterized by displaying the extracted snapshot in the content editing tool for a length proportional to a default running time.

According to another aspect, the step of providing the length adjustment function includes a user's touch-and-drag or It may be characterized by providing the ability to increase or decrease the length of the first snapshot by clicking and dragging.

According to yet another aspect, the step of providing the length adjustability comprises adjusting the left side of the first snapshot while a user's touch or click on the left side region or the right side region of the first snapshot is maintained. It may be characterized by displaying the point in time on the timeline for the end portion or the right end portion.

According to another aspect, the step of adjusting the running time according to the adjusted length is the time occupied on the timeline by an image corresponding to the length-adjusted snapshot. is increased or decreased proportionally to the extent to which said length is adjusted.

According to yet another aspect, generating the speech synthesis and adding to the timeline at the selected time includes generating speech synthesis for the text according to a speech type selected in the content editing tool. may be

According to yet another aspect, the step of generating and adding the speech synthesis to the selected point in time on the timeline includes: It may be characterized by adding the generated speech synthesis at a specific point in time.

According to yet another aspect, the method for generating video content includes, by the at least one processor, moving a position on the timeline of the speech synthesis added to the timeline based on user input. May contain more.

According to yet another aspect, the method for generating video content includes selecting, by the at least one processor, one sound effect from among a plurality of sound effects provided by the content editing tool; adding the selected sound effect to the timeline at the selected time point with the content editing tool, by a single processor.

According to yet another aspect, the video content generation method may further include providing, by the at least one processor, functionality for reordering the displayed snapshots.

According to another aspect, the image may be uploaded in the form of a file containing a plurality of pages that can be imaged.

According to yet another aspect, the step of generating and adding the speech synthesis to the timeline at the selected point in time includes adding a first speech synthesis to be added to the timeline to a first speech synthesis that has already been added to the timeline. The first speech synthesis may be added to the timeline as an audio channel different from the second speech synthesis when at least part of the running time overlaps with the second speech synthesis.

According to yet another aspect, generating and adding the speech synthesis to the timeline at the selected point includes providing an indicator to the speech synthesis added at the timeline at the selected point to the content editing tool. It may be characterized by displaying.

In still another aspect, the indicator may display at least a portion of the text.

In still another aspect, the length of the indicator may be proportional to the length of the speech synthesis.

According to yet another aspect, generating the speech synthesis and adding it to the timeline at the selected point includes information about the speech type used to generate the speech synthesis based on user input to the indicator. , information about the length of said speech synthesis, and said text.

A computer program is provided for causing a computer device to perform the method.

A computer-readable recording medium is provided in which a program for causing a computer device to execute the method is recorded.

at least one processor implemented to execute computer readable instructions, said at least one processor extracting a snapshot of an image uploaded to a content editing tool; displaying along a timeline in a content editing tool, providing a length adjustment function for adjusting the length of the displayed snapshot in the content editing tool, and a snap whose length is adjusted by the length adjustment function; A computer device that adjusts the running time of a shot according to the adjusted length, generates speech synthesis for the text input to the content editing tool, and adds it to the selected time point on the timeline. I will provide a.

A voice synthesis desired by a user can be generated in real time for a plurality of images and dubbed at a reproduction start time desired by the user, and the generated voice synthesis is used for video contents using a plurality of dubbed images. can be generated and provided.

1 is a diagram showing an example of a network environment in one embodiment of the present invention; FIG. 1 is a block diagram illustrating an example of a computing device, in accordance with one embodiment of the present invention; FIG. 1 is a diagram showing an example of a video content generation system in one embodiment of the present invention; FIG. FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; FIG. 4 is a diagram showing an example of a content editing tool screen in one embodiment of the present invention; 4 is a flow chart illustrating an example of a method for generating video content, according to an embodiment of the present invention;

Embodiments will be described in detail below with reference to the accompanying drawings.

A content generation system according to embodiments of the present invention may be implemented by at least one computer device, and a content generation method according to an embodiment of the present invention may be implemented by at least one computer device that implements the content generation system. good. A computer program according to an embodiment of the present invention may be installed and executed in a computer device, and the computer device executes a content generation method according to an embodiment of the present invention under control of the executed computer program. you can The computer program described above may be recorded in a computer-readable recording medium in order to combine with a computer device and cause the computer device to execute the content generation method.

FIG. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment of FIG. 1 illustrates an example including multiple electronic devices 110 , 120 , 130 , 140 , multiple servers 150 , 160 , and a network 170 . Such FIG. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited as in FIG. Also, the network environment of FIG. 1 is merely an example for explaining the environment applicable to this embodiment, and the environment applicable to this embodiment is not limited to the network environment of FIG.

The plurality of electronic devices 110, 120, 130, 140 may be fixed terminals or mobile terminals implemented by computing devices. Examples of the plurality of electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, PCs (personal computers), notebook PCs, digital broadcasting terminals, PDAs (Personal Digital Assistants), and PMPs (Portable Multimedia Players). ), tablets, etc. As an example, FIG. 1 shows a smart phone as an example of the electronic device 110, but in embodiments of the present invention, the electronic device 110 substantially utilizes a wireless or wired communication scheme and communicates with other devices via the network 170. may refer to one of a wide variety of physical computing devices capable of communicating with the electronic devices 120, 130, 140 and/or the servers 150, 160.

The communication method is not limited, and not only the communication method using the communication network that can be included in the network 170 (eg, mobile communication network, wired Internet, wireless Internet, broadcast network, satellite network, etc.), but also the device It may also include short-range wireless communication between For example, the network 170 includes a PAN (personal area network), a LAN (local area network), a CAN (campus area network), a MAN (metropolitan area network), a WAN (wide area network), a BBN (broadband network), and the Internet. Any one or more of the networks may be included. Additionally, network 170 may include any one or more of network topologies including, but not limited to, bus networks, star networks, ring networks, mesh networks, star-bus networks, tree or hierarchical networks, and the like. will not be

Each of servers 150, 160 is implemented by one or more computing devices that communicate with a plurality of electronic devices 110, 120, 130, 140 over network 170 to provide instructions, code, files, content, services, etc. good. For example, the server 150 provides services (for example, content provision service, group call service (or voice conference service), messaging service, mail service) to a plurality of electronic devices 110, 120, 130, and 140 connected via the network 170. , social network services, map services, translation services, financial services, payment services, search services, etc.).

FIG. 2 is a block diagram illustrating an example computing device, in accordance with one embodiment of the present invention. Each of the plurality of electronic devices 110, 120, 130 and 140 and each of the servers 150 and 160 described above may be realized by the computer device 200 shown in FIG.

Such a computing device 200 may include memory 210, processor 220, communication interface 230, and input/output interface 240, as shown in FIG. The memory 210 is a computer-readable storage medium and may include random access memory (RAM), read only memory (ROM), and permanent mass storage devices such as disk drives. Here, a permanent mass storage device such as a ROM or disk drive may be included in computer device 200 as a separate permanent storage device separate from memory 210 . Also stored in memory 210 may be an operating system and at least one program code. Such software components may be loaded into memory 210 from a computer-readable medium separate from memory 210 . Such other computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, memory cards, and the like. In other embodiments, software components may be loaded into memory 210 through communication interface 230 that is not a computer-readable medium. For example, software components may be loaded into memory 210 of computing device 200 based on computer programs installed by files received over network 170 .

Processor 220 may be configured to process computer program instructions by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 220 by memory 210 or communication interface 230 . For example, processor 220 may be configured to execute received instructions according to program code stored in a storage device, such as memory 210 .

Communication interface 230 may provide functionality for computer device 200 to communicate with other devices (eg, the recording device described above) via network 170 . As an example, processor 220 of computing device 200 can transmit requests, commands, data, files, etc. generated according to program code recorded in a recording device such as memory 210 to other devices via network 170 under the control of communication interface 230 . device. Conversely, signals, instructions, data, files, etc. from other devices may be received by computing device 200 through communication interface 230 of computing device 200 over network 170 . Signals, commands, data, etc., received through the communication interface 230 may be transmitted to the processor 220 and the memory 210, and files, etc., may be recorded in a recording medium (the permanent recording device described above) that the computing device 200 may further include. you can

Input/output interface 240 may be a means for interfacing with input/output device 250 . For example, input devices may include devices such as a microphone, keyboard, or mouse, and output devices may include devices such as displays, speakers, and the like. As another example, input/output interface 240 may be a means for interfacing with a device that integrates functionality for input and output, such as a touch screen. Input/output device 250 may consist of at least one computing device 200 and one device.

Also, in other embodiments, computing device 200 may include fewer or more components than the components of FIG. However, most prior art components need not be explicitly shown in the figures. For example, computing device 200 may be implemented to include at least some of the input/output devices 250 described above, and may also include other components such as transceivers, databases, and the like.

FIG. 3 is a diagram showing an example of a video content generation system in one embodiment of the present invention. FIG. 3 shows a content generation server 300, a plurality of users 310, and a content editing tool 320. As shown in FIG.

The content generation server 300 may be implemented in at least one computing device 200 and provides a plurality of users 310 with a content editing tool 320 such that each of the plurality of users 310 can use the content editing tool 320 while the plurality of users 310 use the content editing tool 320. It may support dubbing speech synthesis into images to generate video content.

Here, "image" may include a plurality of individual images, a batch of images, or a batch of images and at least one individual image. A bundle of images may also include images of pages contained in a single file, such as a PDF file.

A plurality of users 310 may receive content editing tools 320 from the content generation server 300 to generate video content from images. At this time, each of the plurality of users 310 may be substantially a physical electronic device that accesses the content generation server 300 via the network 170 and receives provision of the content editing tool 320 . Each such physical electronic device may also be embodied in the computing device 200 described with reference to FIG.

The content editing tool 320 may be provided to the plurality of users 310 in a web-based or app-based manner. In the web method, a plurality of users 310 visit a web page that implements the function of the content editing tool 320 and is provided from the content generation server 300, and perform functions for generating video content through the corresponding web page. It may mean a method of receiving an offer. The application method is a method of connecting to the content generation server 300 through an application installed and executed in each physical electronic device corresponding to a plurality of users 310 and receiving a function for generating video content. can mean Depending on the embodiment, each physical electronic device corresponding to a plurality of users 310 may autonomously process the generation of video content using an application that includes a function for generating video content.

In one embodiment, the content generation server 300 may display thumbnails of images uploaded to the content editing tool 320 by the user along a timeline on the content editing tool 320 . When the user uploads a file made up of multiple pages, the content generation server 300 may image the multiple pages and display thumbnails of the imaged pages on the content editing tool 320 along a timeline. .

At this time, the content editing tool 320 may provide a function for the user to adjust the order of the images on the timeline. Users may determine the order of their uploaded images using the appropriate function. The order of the images on the timeline may correspond to the order in which the images appear in the final generated video content.

Also, the content editing tool 320 may provide a function for the user to delete a desired image among the images on the timeline. In other words, the user can use the corresponding function to delete unnecessary images among the images uploaded by the user.

Also, the content editing tool 320 may provide a function for the user to adjust the time (or interval) occupied by each image on the timeline. The adjusted time may correspond to the time (or section) at which the image appears in the video content that is finally generated. For example, the horizontal length (or vertical length) of the thumbnail displayed in content editing tool 320 may correspond to the time (or interval) that the image occupies on the timeline. As an example, content editing tool 320 may initially display thumbnails with a length corresponding to a time (or interval) of four seconds. At this time, the content editing tool 320 may provide functionality for increasing or decreasing the length of the thumbnail by the user clicking or touching and then dragging the left and/or right end portions of the thumbnail. In this case, the time occupied by the image on the timeline may be increased or decreased according to the adjusted thumbnail length.

In addition, the content editing tool 320 may provide a function for the user to select a desired time point or section on the timeline, and associate any text desired by the user with the selected time point or section. may provide a user interface for If any text is linked to the selected time point or section, the content generation server 300 automatically converts the linked text into speech and adds the converted voice to the selected time point or section. Therefore, it is possible to easily and conveniently dub the voice of the user's desired contents into the image.

4 to 19 are diagrams showing examples of content editing tool screens in one embodiment of the present invention.

FIG. 4 shows an example first screen 400 of the content editing tool 320 described with reference to FIG. The configuration of the content editing tool 320 according to the present embodiment is only an example, and the configuration may be variously changed according to the embodiment.

A user may access the content editing tool 320 using their electronic device, and the content editing tool 320 may provide functionality 410 for the user to upload images. Although the first screen example 400 of FIG. 4 describes an example of uploading a video or PDF file, the content editing tool 320 can be used to create a single file containing multiple individual images, multiple images, or a single file containing multiple images. You may provide functionality for uploading a single file and a combination of multiple images. At this time, the images uploaded by the user may include images stored in the local storage location of the electronic device that the user used to access the content editing tool 320 . Depending on the embodiment, the image uploaded by the user may be an image located on the web rather than a local storage location on the electronic device.

Content editing tools 320 may also provide functionality 420 for adding dubbing to images. As an example, functions 420 may include voice selection function 421 and text input function 422 . Voice selection function 421 may be a function for selecting one of a wide variety of predefined voice types, and text input function 422 generates text-to-speech (TTS). It may be a function for entering text for As an example, suppose the user selects the voice type “Voice 1” in voice selection function 421 and enters the text “Hello” in text input function 422 . At this time, when the trial listening button 423 or the dubbing addition button 424 is selected (for example, by clicking in a PC environment or by touching in a touch screen environment), the input text "Hello" and the selected voice type "Voice 1" are selected. ” may be communicated from the content editing tool 320 to the content generation server 300 . In this case, the content generation server 300 may generate speech synthesis for the text "hello" using the speech type "speech 1", and transmit the generated speech synthesis from the content editing tool 320 to the user's electronic device. you can At this time, speech synthesis may be output from the electronic device's speakers in response to selection of preview button 423 , and speech synthesis associated with the image uploaded by function 410 may be output in response to selection of add dubbing button 424 . may be added to the timeline. More specifically, the content editing tool 320 may include a timeline display function 440 for visually representing a timeline for the video content that is finally generated. Where speech synthesis is added to the timeline at this time will be discussed in more detail below.

Depending on the embodiment, the voice selection function 421 may be implemented to select one of the voice types that the user has favorited. At this time, the user may be provided with a user interface for bookmarking a specific voice type among all voice types. As an example, if the user selects "Display All" shown in the dubbing addition function 420, a user interface may be provided for displaying the overall audio type to the user, and the user may select from the provided user interface , at least one desired voice type out of all voice types may be favorited. In this case, voice selection function 421 may be implemented to select one of the voices that the user has favorited.

The content editing tool 320 may also provide a sound effect addition function 430 for adding pre-made sound effects to the timeline in association with the image. The sound effect addition function 430 displays a list of many pre-made sound effects, performs a trial listening to the sound effects, or adds a sound effect to a specific time on the timeline. may contain. If desired, user-desired sound effects may be added from an external file or generated directly.

Content editing tool 320 may also display a time indicator 450 that indicates a particular point in time on the timeline. FIG. 4 shows an example where the time indicator 450 is at the default time of 00:00.00.

Also, a save button 460 shown in the content editing tool 320 of FIG. 4 may provide functionality for saving the edits of the current project, and a download button 470 may generate video content for download to the user's electronic device. may provide functionality for downloading to

FIG. 5 shows an example second screen 500 of the content editing tool 320 . A second screen example 500 in FIG. 5 shows an example in which an image is uploaded by the function 410 described with reference to FIG. ing. At this time, each thumbnail is displayed on the timeline display function 440 so as to correspond to a preset time interval (a time interval of 4 seconds in the embodiment of FIG. 5). Also, a click-and-drag (or a touch-and-drag or swipe gesture for touch screen environments) on the area of the timeline display function 440 allows exploration of the timeline and thumbnails.

FIG. 6 shows an example of displaying another area of the timeline display function 440 by clicking and dragging the area of the timeline display function 440 as a third screen example 600 of the content editing tool 320 . In the third example screen 600, the last thumbnail, thumbnail 10, indicates that the user has uploaded 10 images. As noted above, the 10 images may be uploaded in the form of individual images or a single file containing pages that can be imaged into 10 images, or can be imaged into n images. A file containing a page and m individual images (here, n and m are natural numbers, n+m=10) may be uploaded in a combined form. It will be readily appreciated that combinations of two or more files and individual images can be used.

FIG. 7 shows an example in which the time interval between thumbnails is adjusted as a fourth screen example 700 of the content editing tool 320 . For example, in the fourth screen example 700 of FIG. 7, the horizontal length of the thumbnail displayed in the area of the timeline display function 440 may correspond to the time (or interval) that the image occupies on the timeline. . At this time, the fourth screen example 700 shows an example in which the user extends the length of the thumbnail while dragging in the right direction after clicking the right end portion of the thumbnail 2 . In this case, according to the length of the thumbnail 2 that has been extended, the time that the image corresponding to the thumbnail 2 occupies on the timeline (hereinafter referred to as running time) may increase. At this time, in the fourth screen example 700, while the user is clicking the right end portion of thumbnail 2, the time point (9.9 seconds) on the timeline corresponding to the right end portion of thumbnail 2 is displayed. A user interface 710 is shown. Accordingly, the user may adjust the length of thumbnail 2 based on the time displayed in such user interface 710 . On the other hand, the start points of the thumbnails after thumbnail 2 (thumbnails 3 to 10, for example) may be changed by the length of thumbnail 2 . In the embodiment of FIG. 7, an example in which the running time of the image corresponding to the thumbnail is adjusted by adjusting the length of the thumbnail 2 has been described. It will be readily understood that it is applicable to

FIG. 8 shows an example of a fifth screen example 800 of the content editing tool 320 in which the time interval between thumbnails 4 is reduced. A fifth screen example 800 shows an example in which the user shortens the length of the thumbnail by clicking the right end portion of the thumbnail 4 and then dragging it leftward. At this time, the running time of the image corresponding to the thumbnail 4 may be reduced according to the shortened length of the thumbnail 4 . In this case, in the fifth screen example 800, while the user is clicking the right end portion of thumbnail 4, the time point (17 seconds) on the timeline corresponding to the right end portion of thumbnail 4 is displayed. An interface 810 is shown. On the other hand, the starting points of the thumbnails after the thumbnail 4 (thumbnails 5 to 10, for example) may be changed by the amount corresponding to the reduced length of the thumbnail 4 .

In the embodiments of FIGS. 7 and 8, the running time of the image corresponding to the thumbnail is increased or decreased by increasing or decreasing the length of the thumbnail while the user clicks on the right end portion of the thumbnail and then drags left or right. explained. With this explanation, in some embodiments, the content editing tool 320 can increase or decrease the length of the thumbnail by clicking on the left end portion of the thumbnail and then dragging in the horizontal direction to increase or decrease the running time of the image corresponding to the thumbnail. It will be readily appreciated that it is also possible to provide the ability to increase or decrease.

FIG. 9 shows an example in which the thumbnail order is changed as a sixth screen example 900 of the content editing tool 320 . Content editing tools 320 may provide functionality for a user to change the order of thumbnails by clicking on a particular thumbnail and then dragging (touch then drag in a touchscreen environment). As an example, the user may change the order of thumbnail 1 and thumbnail 2 in the fifth screen example 800 by clicking thumbnail 1 and then dragging it to the right. A sixth screen example 900 shows how the order of thumbnail 1 and thumbnail 2 has been changed.

FIG. 10 shows an example in which a specific thumbnail is deleted as a seventh screen example 1000 of the content editing tool 320 . The content editing tool 320 may provide functionality for the user to delete a particular thumbnail after it has been selected. As an example, a user interface may be displayed for a user to delete a particular thumbnail by generating a mouseover event on the thumbnail. can be deleted. It will be readily appreciated that a variety of methods for such thumbnail deletion can be provided. As an example, a user may delete a selected thumbnail by pressing a "Del" key on the keyboard after selecting a particular thumbnail with a mouse click.

11 and 12 show examples of adding dubbing as an eighth screen example 1100 and a ninth screen example 1200 of the content editing tool 320. FIG. As noted above, time indicator 450 indicates a particular point in time on the timeline. For example, the user may move the time indicator 450 by dragging the time indicator 450 or clicking a desired timeline position. The time '00:06.00' displayed in association with the time indicator 450 in the eighth screen example 1100 may indicate the time indicated by the current time indicator 450 on the timeline.

Further, the eighth screen example 1100 shows an example in which the text "Hello, I am AAA." At this time, if the user selects the add dubbing button 424, the speech synthesis indicator 1210 for the first speech synthesis corresponding to the text "Hello, I am AAA." It may be displayed in association with the thumbnail in the display function 440 area. At this time, the first synthesized speech may be generated by the content generation server 300 and transferred to the content editing tool 320 as described above. On the other hand, text-to-speech indicator 1210 includes at least a portion of the corresponding text "Hello, I am AAA." An identifier for the voice type (eg, identifier (1) 1220 for voice type “voice 1”) may be displayed.

The length of the speech synthesis indicator 1210 may correspond to the length of the first speech synthesis, and the amount of displayed text may vary according to the length of the speech synthesis indicator 1210 . At this time, the time of the time indicator 450 shown in the eighth screen example 1100 is "00:06.00", and the time of the time indicator 450 shown in the ninth screen example 1200 is "00:09.56". is. In other words, it can be seen that the length of the speech synthesis indicator 1210 for the first speech synthesis is 3.56 seconds (00:09.56-00:06.00=00:03.56).

On the other hand, if the user selects listen button 423 in eighth example screen 1100, a first speech synthesis corresponding to the text "Hello, I am AAA" may be output from the speaker of the user's electronic device. In other words, the electronic device may output the first speech synthesis from the speaker under control of the content editing tool 320 .

FIG. 13 shows, as a tenth screen example 1300 of the content editing tool 320, when the user generates an input such as a mouseover on the speech synthesis indicator 1210, the position of the mouse pointer (in a touch screen environment, the position of the speech synthesis indicator 1210). 13 shows an example in which text-to-speech information 1310 is displayed in association with the location of the touch (between touching the location and placing the touch). The speech synthesis information 1310 includes the speech type (speech 1) used to generate the speech synthesis, the speech synthesis length (3.56 seconds (00:03.56)), the input text (Hello, I am AAA ) may be included.

FIG. 14 shows an eleventh screen example 1400 of the content editing tool 320 when the user reduces the length of the thumbnail 3 to match the time indicator 450 . In this case, the length of thumbnail 3 may be 1.56 seconds for the length of the first speech synthesis, and the running time of the image corresponding to thumbnail 3 on the timeline for video content may be 1.56 seconds. I understand.

FIG. 15 shows an example of a twelfth screen example 1500 of the content editing tool 320 in which the user changes the start point of the first speech synthesis. In other words, it can be seen that in the twelfth screen example 1500 , the position of the speech synthesis indicator 1210 has changed compared to the eleventh screen example 1400 . As an example, the user may change the position of the text-to-speech indicator 1210 by clicking and dragging the text-to-speech indicator 1210 to the left or to the right in the content editing tool 320 , and such repositioning of the text-to-speech indicator 1210 may be changed. may change the starting point of the first speech synthesis. On the other hand, the position of the voice synthesis indicator 1210 may be changed by inputting direction keys on the keyboard while the corresponding voice synthesis indicator 1210 is selected (eg, clicked). Such position changes can be applied in a common manner not only to speech synthesis indicator 1210 but also to each of the various indicators provided by content editing tool 320 . Also, multiple indicators may be selected from one group. As an example, multiple indicators may be selected as a group by sequentially selecting (eg, clicking) multiple indicators while holding down the "Shift" key on the keyboard. In this case, the user may change the positions of many indicators belonging to the group at once by dragging or inputting direction keys on the keyboard.

16 and 17 show examples of adding dubbing as a thirteenth screen example 1600 and a fourteenth screen example 1700 of the content editing tool 320. FIG.

In the thirteenth screen example 1600, after the user moves the time indicator 450 to the position of "00:05.78", the voice selection function 421 selects the voice type "voice 2", and the text input function 422 selects the text " Nice to meet you.” is input. At this time, when the user selects the add dubbing button 424, the speech synthesis indicator 1710 for the second speech synthesis corresponding to the text "Nice to meet you." may be displayed in association with the thumbnail in the As described above, the text-to-speech indicator 1710 includes at least a portion of the corresponding text "Nice to meet you" ("Hajime" in the fourteenth screen example 1700) and an identifier for the type of speech used to generate the second speech synthesis ("Nice to meet you"). As an example, identifier (2) 1720) for voice type "voice 2" may be displayed.

The length of the speech synthesis indicator 1710 may correspond to the length of the second speech synthesis, and the amount of displayed text may vary according to the length of the speech synthesis indicator 1710 . At this time, the time of the time indicator 450 shown in the thirteenth screen example 1600 is "00:06.00", and the time of the time indicator 450 shown in the fourteenth screen example 1700 is "00:08.24". is. In other words, it can be seen that the length of the speech synthesis indicator 1710 for the second speech synthesis is 2.24 seconds (00:08.24-00:06.00=00:02.24).

On the other hand, if the user selects the listen button 423 in the thirteenth example screen 1600, a second speech synthesis corresponding to the text "nice to meet you" may be output from the speaker of the user's electronic device. In other words, the electronic device may output the second speech synthesis from the speaker under control of the content editing tool 320 .

FIG. 18 shows an example of adding a sound effect as a fifteenth screen example 1800 of the content editing tool 320 . In the fifteenth screen example 1800, the user selects the sound effect 2 using the sound effect addition function 430 (for example, clicks the plus button in the dotted frame 1810), and the current time indicator 450 is set as the start time. An example is shown in which a sound 2 indicator 1820 is added. At this time, the duration of the sound effect 2 indicator 1820 may be 2.46 seconds as indicated by the dotted frame 1810 . Such an indicator 1820 can also be moved to another point in time by the user by clicking and dragging.

In the above embodiment, information for generating video content is displayed on the timeline using a total of three channels: one channel for thumbnails, one channel for voice synthesis, and one channel for sound effects. An example of enumerating along is explained. However, in some embodiments, more than one channel for speech synthesis and/or more than one channel for sound effects may be used.

FIG. 19 shows an example sixteenth screen 1900 of the content editing tool 320 using two or more channels for speech synthesis. A sixteenth screen example 1900 shows an example in which two speech synthesis indicators 1210 and 1710 are partially overlapped and displayed. This indicates that it is also possible for two speech syntheses to be output simultaneously on at least some timelines. Although the embodiment of FIG. 19 shows that two channels are used for speech synthesis, it will be readily appreciated that more than two channels may be used. It will also be readily apparent that more than one channel for sound effects can be used.

FIG. 20 is a flow chart illustrating an example of a video content generation method in one embodiment of the present invention. The video content generation method according to the present embodiment may be performed by the computer device 200 that provides services for content editing support by the content editing tool 320 . At this time, the processor 220 of the computing device 200 may be implemented to execute control instructions according to the operating system code and the at least one computer program code contained in the memory 210 . Here, processor 220 may control computing device 200 such that computing device 200 performs steps 2010-2090 included in the method of FIG. 20 according to control instructions provided by code recorded in computing device 200. .

At step 2010, computing device 200 may extract a snapshot of the image uploaded to the content editing tool. As noted above, images may be uploaded in the form of individual images, a file containing images, or a combination of a file and images. In certain embodiments, images may be uploaded in the form of files containing multiple pages that can be imaged. As an example, if a PDF file is uploaded, the computing device 200 may extract images from the PDF file and store them as multiple image files, and may extract snapshots for each of the multiple image files.

At step 2020, the computing device 200 may display the extracted snapshots along a timeline with a content editing tool. Here, the length of the displayed snapshot may be proportional to the running time of the time occupied on the timeline by the image corresponding to the displayed snapshot. At this time, the computing device 200 may display the extracted snapshot on the content editing tool with a length proportional to the default running time. FIG. 5 illustrates an example of displaying a snapshot with a length proportional to the default running time of 4 seconds.

At step 2030, computing device 200 may provide functionality for changing the order of the displayed snapshots. As an example, FIG. 8 and FIG. 9 describe an example of changing the positions of thumbnail 1 and thumbnail 2 . Depending on the embodiment, computing device 200 may further provide functionality for deleting a particular thumbnail.

At step 2040, computing device 200 may provide a length adjustment function to adjust the length of the snapshot displayed in the content editing tool. As an example, the computing device 200 may display the length of the first snapshot by the user's touch-and-drag or click-and-drag on a preset left area or right area for the first snapshot among the displayed snapshots. may be provided with the ability to increase or decrease the Also, while the user's touch or click on the left region or right region of the first snapshot is maintained, the computing device 200 displays the point in time on the timeline for the left end portion or right end portion of the first snapshot. you can As an example, FIG. 7 and FIG. 8 describe an example in which the thumbnail length is increased or decreased, and the time point on the timeline is displayed at the right end portion of the corresponding snapshot at this time.

At step 2050, the computing device 200 may adjust the running time of the snapshot whose length is adjusted by the length adjustment function according to the adjusted length. As an example, computing device 200 increases or decreases the running time of the time that an image corresponding to the length-adjusted snapshot occupies on the timeline in proportion to the extent to which the length of the snapshot was adjusted. You can

At step 2060, computing device 200 may generate a speech synthesis for the text entered into the content editing tool and add it to the timeline at the selected point in time. At this time, the computing device 200 may generate speech synthesis for the text according to the speech type selected in the content editing tool. A large number of voice types are pre-generated according to age, gender, language (Korean, English, Chinese, Japanese, Spanish, etc.), emotion (joy, sadness, etc.), etc. A user may select a particular voice type to utilize for speech synthesis from a content editing tool. Also, the computing device 200 may add the generated speech synthesis to a specific point in time on the timeline selected by moving a time indicator indicating the specific point in time on the timeline. 11 and 12, 16 and 17 described examples of adding speech synthesis at the time selected by the time indicator 450. FIG.

According to an embodiment, if the first speech synthesis to be added to the timeline overlaps at least part of the running time of the second speech synthesis that has already been added to the timeline, the computer device 200 replaces the first speech synthesis with the second speech synthesis. It may be added to the timeline as an audio channel separate from speech synthesis. In other words, dubbing may be performed such that two or more audio syntheses are output simultaneously in the generated video content. FIG. 19 has described an example in which two speech syntheses are added to the timeline as different channels.

Computing device 200 may also display an indicator for speech synthesis added at the selected point in the timeline in the content editing tool. In some embodiments, the indicator may display at least a portion of the text, and the length of the indicator may be proportional to the length of the speech synthesis. Here, the speech synthesis length may mean the time during which the speech synthesis is output.

Additionally, computing device 200 may generate speech synthesis information including at least one of information about the type of speech used to generate the speech synthesis, information about the length of speech synthesis, and text based on the user input to the indicator. can be output. Speech synthesis information may be displayed in association with the location where user input to the indicator occurs. As an example, in FIG. 13, an example of displaying the speech synthesis information 1310 regarding the speech synthesis indicator 1210 has been described.

At step 2070, computing device 200 may move the position on the timeline of the speech synthesis added to the timeline based on the user's input. As an example, in FIGS. 14 and 15, an example of moving the position of speech synthesis by input such as click-and-drag or touch-and-drag by the user has been described.

At step 2080, the computing device 200 may receive a selection of one sound effect from multiple sound effects provided by the content editing tool. As an example, in FIG. 18, multiple sound effects are provided to the user by the sound effect addition function 430, and the user selects one of the multiple sound effects.

At step 2090, computing device 200 may add the selected sound effect at the time selected to the timeline with the content editing tool. As an example, in FIG. 18, the example of adding the sound effect 2 at the time selected by the time indicator 450 has been described.

At this time, depending on the embodiment, at least some of steps 2010-2090 may be performed in parallel. As an example, steps 2040 and 2050 may be triggered by user input for length adjustment, steps 2060 and 2070 may be triggered by user input for adding speech synthesis, and step 2080. and stage 2090 may be triggered by user input for the addition of sound effects. Accordingly, the order of steps 2040-2090 may be changed by user input.

Thereafter, when the user requests the generation of video content, the computer apparatus 200 may generate a moving image after leveling the image to a size suitable for the video content. Depending on the embodiment, the computing device 200 may insert watermarks and/or subtitles into the video content. Computing device 200 may then insert voice synthesis and/or sound effects into the video to match the timeline to generate the final video content.

As described above, according to the embodiment of the present invention, it is possible to generate a speech synthesis desired by the user for a number of images in real time and dub at a playback start time desired by the user, and the generated speech synthesis is Video content can be generated and provided by multiple dubbed images.

The systems or devices described above may be realized by hardware components or a combination of hardware and software components. For example, the devices and components described in the embodiments may include, for example, processors, controllers, ALUs (arithmetic logic units), digital signal processors, microcomputers, FPGAs (field programmable gate arrays), PLUs (programmable logic units), microcontrollers, It may be implemented using one or more general purpose or special purpose computers, such as a processor or various devices capable of executing instructions and responding to instructions. The processing unit may run an operating system (OS) and one or more software applications that run on the OS. The processor may also access, record, manipulate, process, and generate data in response to executing software. For convenience of understanding, one processing device may be described as being used, but those skilled in the art will appreciate that a processing device may include multiple processing elements and/or multiple types of processing elements. You can understand that. For example, a processing unit may include multiple processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include computer programs, code, instructions, or a combination of one or more of these, to configure a processor to operate at its discretion or to independently or collectively instruct a processor. You can Software and/or data may be embodied in any kind of machine, component, physical device, virtual device, computer storage medium or device for interpretation on or for providing instructions or data to a processing device. may be changed. The software may be stored and executed in a distributed fashion over computer systems linked by a network. Software and data may be recorded on one or more computer-readable recording media.

The method according to the embodiments may be embodied in the form of program instructions executable by various computer means and recorded on a computer-readable medium. The computer-readable media may include program instructions, data files, data structures, etc. singly or in combination. The medium may be a continuous recording of the computer-executable program or a temporary recording for execution or download. In addition, the medium may be various recording means or storage means in the form of a combination of single or multiple hardware, and is not limited to a medium that is directly connected to a computer system, but is distributed over a network. It may exist in Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc., and may be configured to store program instructions. Other examples of media include recording media or storage media managed by application stores that distribute applications, sites that supply or distribute various software, and servers. Examples of program instructions include high-level language code that is executed by a computer, such as using an interpreter, as well as machine language code, such as that generated by a compiler.

As described above, the embodiments have been described based on the limited embodiments and drawings, but those skilled in the art will be able to make various modifications and variations based on the above description. For example, the techniques described may be performed in a different order than in the manner described and/or components such as systems, structures, devices, circuits, etc. described may be performed in a manner different from the manner described. Appropriate results may be achieved when combined or combined, opposed or substituted by other elements or equivalents.

Accordingly, different embodiments that are equivalent to the claims should still fall within the scope of the appended claims.

300: Content Generation Server 310: User 320: Content Editing Tool

Claims (19)

A method of generating video content for a computing device comprising at least one processor, comprising:
extracting, by the at least one processor, a snapshot of an image uploaded through a content editing tool;
displaying, by the at least one processor, the extracted snapshots along a timeline through the content editing tool;
providing, by the at least one processor, a length adjustment function to adjust the length of the displayed snapshot through the content editing tool;
adjusting, by the at least one processor, a running time of the snapshot whose length has been adjusted by the length adjusting function by the adjusted length; and inputting by the at least one processor through the content editing tool. generating a speech synthesis for the text to be generated and adding it to the selected time point of the timeline. the length of the displayed snapshot is proportional to the running time of the time occupied on the timeline by the image corresponding to the displayed snapshot;
The step of displaying along the timeline includes:
2. The video content generation method of claim 1, wherein the extracted snapshot is displayed through the content editing tool for a length proportional to a default running time. The step of providing the length adjustability comprises:
A function of increasing or decreasing the length of the first snapshot among the displayed snapshots by a user's touch-and-drag or click-and-drag on a preset left area or right area. 3. A video content generation method according to claim 1 or 2, characterized in that it provides . The step of providing the length adjustability comprises:
displaying a point in time on the timeline for the left end portion or the right end portion of the first snapshot while a user's touch or click on the left side region or the right side region of the first snapshot is maintained. 4. A video content generation method as claimed in claim 3. Adjusting the running time according to the adjusted length includes:
The running time of the time occupied by the image corresponding to the length-adjusted snapshot on the timeline is increased or decreased in proportion to the extent to which the length is adjusted. The video content generation method according to any one of claims 1 to 4. generating the speech synthesis and adding it to the timeline at the selected point in time;
The video content generation method according to any one of claims 1 to 5, wherein speech synthesis is generated for the text according to the audio type selected through the content editing tool. generating the speech synthesis and adding it to the timeline at the selected point in time;
wherein said generated speech synthesis is added at a specific point in time on said timeline selected by moving a time indicator representing a specific point in time on said timeline. The video content generation method according to any one of . further comprising, by the at least one processor, moving a position on the timeline of the speech synthesis added to the timeline based on user input. The video content generation method according to the item. selecting, by the at least one processor, one sound effect from among a plurality of sound effects provided through the content editing tool; and 9. The video content generation method according to any one of claims 1 to 8, further comprising: adding the selected sound effect at a point in time selected by . 10. The video of any one of claims 1 to 9, further comprising: providing, by said at least one processor, functionality for changing the order of said displayed snapshots. Content generation method. The video content generation method according to any one of claims 1 to 10, wherein the image is uploaded in the form of a file containing a plurality of pages that can be imaged. generating the speech synthesis and adding it to the timeline at the selected point in time;
If the first speech synthesis to be added to the timeline overlaps at least a part of the running time of the second speech synthesis already added to the timeline, the first speech synthesis is different from the second speech synthesis. A video content generation method according to any one of claims 1 to 5, characterized in that it is added to said timeline as an audio channel. generating the speech synthesis and adding it to the timeline at the selected point in time;
The video content generation method according to any one of claims 1 to 5, characterized in that an indicator for speech synthesis added at the selected time point of the timeline is displayed through the content editing tool. . 14. The method of claim 13, wherein the indicator causes at least part of the text to be displayed. 14. The method of claim 13, wherein the length of the indicator is proportional to the length of the voice synthesis. generating the speech synthesis and adding it to the timeline at the selected point in time;
displaying at least one of information of the speech type used to generate the speech synthesis, information of the length of the speech synthesis, and the text, based on user input to the indicator; 14. The video content generation method according to claim 13. A computer program for causing a computer device to perform the method according to any one of claims 1-16. at least one processor implemented to execute computer readable instructions;
by the at least one processor;
Extract snapshots of images uploaded through content editing tools,
displaying the extracted snapshot along a timeline through the content editing tool;
providing a length adjustment function for adjusting the length of the displayed snapshot through the content editing tool;
adjusting the running time of the snapshot whose length has been adjusted by the length adjustment function according to the adjusted length;
A computer device that generates a speech synthesis for text input through the content editing tool and adds it to the timeline at selected points. to provide the length adjustment functionality, by the at least one processor:
A function of increasing or decreasing the length of the first snapshot among the displayed snapshots by a user's touch-and-drag or click-and-drag on a preset left area or right area. 19. A computing device as claimed in claim 18, characterized in that it provides:

Images