KR102465870B1 - Method and system for generating video content based on text to speech for image - Google Patents

Abstract

Disclosed are a method and a system for generating video content based on speech synthesis for an image. The method for generating image content according to an embodiment includes extracting snapshots of images uploaded through a content editing tool, displaying the extracted snapshots along a timeline through a content editing tool, and displaying the extracted snapshots through a content editing tool. A step of providing a length adjustment function to adjust the length of the snapshots, a step of adjusting the running time of a snapshot whose length is adjusted through the length adjustment function according to the adjusted length, and a voice for text input through a content editing tool It may include creating a compositing and adding it to a selected point in the timeline.

Description

Method and system for generating video contents based on speech synthesis for images

The following description relates to a method and system for generating video content based on speech synthesis for images.

In order to apply a sound source (including Text To Speech, TTS) to material containing images, for example, in the case of a PowerPoint-formed material, a cumbersome task of adding one sound source to each slide was required. In this case, there is a limitation that only one sound source can be added for each slide, and there is a problem that the playback start time cannot be freely edited.

As such, in the market where the need for production and consumption of video content has increased, the conventional video production technology using voice synthesis has a problem of providing only a cumbersome and limited form.

[Prior art literature]

Korea Patent Publication No. 10-2014-0147401 (published date: 2014.12.30)

An image capable of generating a voice synthesis desired by a user for a plurality of images in real time and dubbing it at a playback start time desired by the user, and generating and providing video content through a plurality of images dubbed with the generated voice synthesis A content creation method and system are provided.

A method for generating image content in a computer device including at least one processor, the method comprising: extracting, by the at least one processor, snapshots of images uploaded through a content editing tool; displaying, by the at least one processor, the extracted snapshots according to a timeline through the content editing tool; providing, by the at least one processor, a length adjustment function for adjusting the length of the displayed snapshots through the content editing tool; adjusting, by the at least one processor, a running time of a snapshot whose length is adjusted through the length adjustment function according to the adjusted length; and generating, by the at least one processor, a voice synthesis for text input through the content editing tool, and adding it to the selected time point of the timeline.

According to one side, the length of the displayed snapshots is proportional to the running time, which is a time occupied by images corresponding to the displayed snapshots on the timeline, and displaying according to the timeline includes: It may be characterized in that the shots are displayed through the content editing tool with a length proportional to the default running time.

According to another aspect, the providing of the length adjustment function may include: according to a user's touch-and-drag or click-and-drag a user's touch-and-drag or click-and-drag on a predetermined left area or right area for a first snapshot among the displayed snapshots. It may be characterized in that it provides a function to increase or decrease the length of the snapshot.

According to another aspect, the providing of the length adjustment function may include, while the user's touch or click on the left area or the right area of the first snapshot is maintained, the left end portion of the first snapshot Alternatively, it may be characterized in that a viewpoint on the timeline for the right end is displayed.

According to another aspect, in the step of adjusting the running time according to the adjusted length, the running time, which is a time occupied by an image corresponding to the length-adjusted snapshot, on the timeline, is adjusted by the length. It may be characterized in that it increases or decreases in proportion to the degree.

According to another aspect, the generating and adding the speech synthesis to the selected time point of the timeline may include generating a speech synthesis for the text according to a speech type selected through the content editing tool. .

According to another aspect, the step of generating the speech synthesis and adding it to the selected point in time of the timeline may include moving a time indicator indicating a specific point in time on the timeline to move the generated speech at a specific point in the timeline. It may be characterized by adding synthesis.

According to another aspect, the method for generating image content may further include moving, by the at least one processor, a position on the timeline of the voice synthesis added to the timeline based on a user input. can

According to another aspect, the method for generating image content includes: receiving, by the at least one processor, one of a plurality of sound effects provided through the content editing tool; and adding, by the at least one processor, the selected sound effect at a time selected for the timeline in the content editing tool.

According to another aspect, the method for generating image content may further include providing, by the at least one processor, a function for changing the order of the displayed snapshots.

According to another aspect, the images may be uploaded in the form of a file including a plurality of imageable pages.

According to another aspect, the step of generating the speech synthesis and adding it to the selected time point of the timeline includes a first speech synthesis to be added to the timeline, a second speech synthesis already added to the timeline, and a running time When at least a portion of the syntheses overlap, the first voice synthesis may be added to the timeline as a different voice channel than the second voice synthesis.

According to another aspect, the step of generating the speech synthesis and adding it to the selected time point of the timeline includes displaying an indicator for the speech synthesis added at the selected time point of the timeline through the content editing tool. can do.

According to another aspect, at least a part of the text may be displayed through the indicator.

According to another aspect, the length of the indicator may be proportional to the length of the speech synthesis.

According to another aspect, the step of generating the speech synthesis and adding it to the selected time point of the timeline includes information on a speech type used to generate the speech synthesis based on a user input for the indicator, and the speech synthesis. It may be characterized in that at least one of information on the length of and the text is displayed.

Provided is a computer program stored in a computer-readable recording medium in combination with a computer device to execute the method on the computer device.

It provides a computer-readable recording medium in which a program for executing the method in a computer device is recorded.

at least one processor implemented to execute computer-readable instructions, wherein the at least one processor extracts snapshots of images uploaded through a content editing tool, and edits the extracted snapshots into the content It displays according to the timeline through a tool, provides a length adjustment function for adjusting the length of the displayed snapshots through the content editing tool, and adjusts the running time of the snapshot whose length is adjusted through the length adjustment function Provided is a computer device characterized in that the audio synthesis is adjusted according to the length of the text inputted through the content editing tool and added to the selected time point of the timeline.

A user's desired voice synthesis for a plurality of images may be generated in real time and dubbed at a playback start time desired by the user, and video content may be generated and provided through the plurality of images to which the generated voice synthesis has been dubbed.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention.
2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention.
3 is a diagram illustrating an example of an image content generation system according to an embodiment of the present invention.
4 to 19 are diagrams illustrating examples of screens of a content editing tool according to an embodiment of the present invention.
20 is a flowchart illustrating an example of a method for generating image content according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The content generating system according to the embodiments of the present invention may be implemented by at least one computer device, and the content generating method according to the embodiments of the present invention is performed through at least one computer device implementing the content generating system. can be The computer program according to an embodiment of the present invention may be installed and driven in the computer device, and the computer device may perform the content creation method according to the embodiments of the present invention under the control of the driven computer program. The above-described computer program may be stored in a computer-readable recording medium in order to be combined with a computer device to cause the computer device to execute the content creation method.

1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 shows an example including a plurality of electronic devices 110 , 120 , 130 , 140 , a plurality of servers 150 , 160 , and a network 170 . 1 is an example for explaining the invention, and the number of electronic devices or the number of servers is not limited as in FIG. 1 . In addition, the network environment of FIG. 1 only describes one example of environments applicable to the present embodiments, and the environment applicable to the present embodiments is not limited to the network environment of FIG. 1 .

The plurality of electronic devices 110 , 120 , 130 , and 140 may be a fixed terminal implemented as a computer device or a mobile terminal. Examples of the plurality of electronic devices 110 , 120 , 130 , 140 include a smart phone, a mobile phone, a navigation device, a computer, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). ), tablet PCs, etc. As an example, in FIG. 1 , the shape of a smartphone is shown as an example of the electronic device 110 , but in embodiments of the present invention, the electronic device 110 is substantially configured to be different through the network 170 using a wireless or wired communication method. It may refer to one of various physical computer devices capable of communicating with the electronic devices 120 , 130 , 140 and/or the servers 150 and 160 .

The communication method is not limited, and not only a communication method using a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 170 may include, but also short-range wireless communication between devices may be included. For example, the network 170 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , the Internet, and the like. In addition, the network 170 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network, etc. not limited

Each of the servers 150 and 160 communicates with the plurality of electronic devices 110 , 120 , 130 , 140 and the network 170 through a computer device or a plurality of computers providing commands, codes, files, contents, services, etc. It can be implemented in devices. For example, the server 150 provides a service (eg, a content providing service, a group call service (or a voice conference service) to the plurality of electronic devices 110 , 120 , 130 , and 140 connected through the network 170 ). , a messaging service, a mail service, a social network service, a map service, a translation service, a financial service, a payment service, a search service, etc.).

2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Each of the plurality of electronic devices 110 , 120 , 130 , 140 or the servers 150 and 160 described above may be implemented by the computer device 200 illustrated in FIG. 2 .

As shown in FIG. 2 , the computer device 200 may include a memory 210 , a processor 220 , a communication interface 230 , and an input/output interface 240 . The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-volatile mass storage device such as a ROM and a disk drive may be included in the computer device 200 as a separate permanent storage device distinct from the memory 210 . Also, the memory 210 may store an operating system and at least one program code. These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, the software components may be loaded into the memory 210 through the communication interface 230 instead of a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computer device 200 based on a computer program installed by files received through the network 170 .

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230 . For example, the processor 220 may be configured to execute a received instruction according to a program code stored in a recording device such as the memory 210 .

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (eg, the storage devices described above) through the network 170 . For example, a request, command, data, file, etc. generated by the processor 220 of the computer device 200 according to a program code stored in a recording device such as the memory 210 is transmitted to the network ( 170) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 170 . A signal, command, or data received through the communication interface 230 may be transmitted to the processor 220 or the memory 210 , and the file may be a storage medium (described above) that the computer device 200 may further include. persistent storage).

The input/output interface 240 may be a means for an interface with the input/output device 250 . For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or a speaker. As another example, the input/output interface 240 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 250 may be configured as one device with the computer device 200 .

Also, in other embodiments, the computer device 200 may include fewer or more components than those of FIG. 2 . However, there is no need to clearly show most of the prior art components. For example, the computer device 200 may be implemented to include at least a portion of the above-described input/output device 250 or may further include other components such as a transceiver and a database.

3 is a diagram illustrating an example of an image content generation system according to an embodiment of the present invention. 3 shows a content creation server 300 , a plurality of users 310 , and a content editing tool 320 .

The content creation server 300 may be implemented by at least one computer device 200 , and provides a content editing tool 320 to a plurality of users 310 so that the plurality of users 310 can use the content editing tool. Using 320, each of the plurality of users 310 may support the creation of video content by dubbing audio synthesis on images.

Here, “images” may include a plurality of individual images, a bundle of images, or a bundle of images and at least one individual image. Also, the bundle of images may include images of pages included in one file, such as a PDF file.

The plurality of users 310 may receive the content editing tool 320 from the content creation server 300 to generate image content from the images. In this case, each of the plurality of users 310 may be physical electronic devices that access the content creation server 300 through the network 170 and receive the content editing tool 320 . Each of these physical electronic devices may also be implemented by the computer device 200 described above with reference to FIG. 2 .

The content editing tool 320 may be provided to the plurality of users 310 by a web method or an app method. In the web method, a plurality of users 310 implement the function of the content editing tool 320 and visit a web page provided by the content generation server 300 to generate image content through the web page. It can mean the way in which it is provided. The app method may refer to a method in which a function for image content creation is provided by accessing the content creation server 300 through an application installed and driven in each of the physical electronic devices corresponding to the plurality of users 310 . have. According to an embodiment, each of the physical electronic devices corresponding to the plurality of users 310 may independently process image content creation through an application including a function for generating image content.

In an embodiment, the content creation server 300 may display thumbnails of images uploaded by the user through the content editing tool 320 on the content editing tool 320 along a timeline. If the user uploads a file formed of a plurality of pages, the content creation server 300 may image the plurality of pages and display thumbnails of the imaged pages on the content editing tool 320 along a timeline. have.

In this case, the content editing tool 320 may provide a function for the user to adjust the order of images on the timeline. Users can use this function to determine the order of their uploaded images. The order of the images on the timeline may correspond to the order in which the images appear in the image content to be finally created.

In addition, the content editing tool 320 may provide a function for the user to delete a desired image from among the images on the timeline. In other words, users can delete unnecessary images from among their uploaded images by using the corresponding function.

In addition, the content editing tool 320 may provide a function for the user to adjust the time (or section) occupied by each image on the timeline. The adjusted time may correspond to a time (or section) at which images appear in the image content to be finally created. For example, the horizontal length (or vertical length) of the thumbnails displayed on the content editing tool 320 may correspond to the time (or section) the images occupy on the timeline. For example, the content editing tool 320 may display thumbnails with a length corresponding to the first 4 seconds (or section). In this case, the content editing tool 320 may provide a function to increase or decrease the length of the thumbnail by dragging the user after clicking or touching the left and/or right ends of the thumbnail. In this case, the time the image occupies on the timeline may increase or decrease according to the length of the extended or decreased thumbnail.

In addition, the content editing tool 320 may provide a function for the user to select a desired time point or section on the timeline, and provides a user interface capable of associating any text desired by the user with the selected time point or section can do. When any text is associated with the selected time point or section, the content creation server 300 automatically converts the associated text into a voice and adds the converted voice to the selected time point or section, so that users can easily and conveniently add the desired text to the images. It can support the voice dubbing of the content.

4 to 19 are diagrams illustrating examples of screens of a content editing tool according to an embodiment of the present invention.

FIG. 4 shows an example of a first screen 400 of the content editing tool 320 described with reference to FIG. 3 . The configuration of the content editing tool 320 according to the present embodiment is one embodiment, and the configuration may be variously changed according to the embodiment.

The user may access the content editing tool 320 through his/her electronic device, and the content editing tool 320 may provide a function 410 for uploading images of the user. Although an example of uploading a video or PDF file is described in the first screen example 400 of FIG. 4 , the content editing tool 320 includes a plurality of individual images or a single file including a plurality of images or a single file and A function for uploading a combination of a plurality of images may be provided. In this case, the images uploaded by the user may include images stored in a local storage of the electronic device used by the user to access the content editing tool 320 . According to an embodiment, the images uploaded by the user may be images located on the web rather than the local storage of the electronic device.

Also, the content editing tool 320 may provide a function 420 for adding a dubbing to an image. As an example, the function 420 may include a voice selection function 421 and a text input function 422 . The voice selection function 421 may be a function for selecting one from among various types of predefined voice types, and the text input function 422 inputs text for generating a Text To Speech (TTS). It may be a function to receive. For example, the user may select a voice type “Voice 1” in the voice selection function 421 and input the text “Hello” in the text input function 422 . At this time, when the preview button 423 or the dubbing add button 424 is selected (for example, by clicking in a PC environment or by a touch in a touch screen environment), the input text "hello" and the selected voice type The identifier of “voice 1” may be transmitted to the content creation server 300 through the content editing tool 320 . In this case, the content creation server 300 may generate a voice synthesis for the text “hello” with the voice type “voice 1” and deliver the generated voice synthesis to the user's electronic device through the content editing tool 320 . can At this time, in response to the selection of the preview button 423 , the voice synthesis may be output from the electronic device through the speaker, and the dubbing add button 424 is associated with the image uploaded through the function 410 in response to the selection. Speech synthesis can be added to the timeline. More specifically, the content editing tool 320 may include a timeline display function 440 for visually representing a timeline for the image content to be finally generated. In this case, where the speech synthesis is added to the timeline will be described in more detail later.

According to an embodiment, the voice selection function 421 may be implemented to receive one of voice types registered as a favorite by the user. In this case, a user interface for registering a specific voice type among all voice types as a favorite may be provided to the user. As an example, when the user selects “view all” displayed in the dubbing add-on function 420 , a user interface for displaying all voice types may be provided to the user, and the user may select all voice types through the provided user interface. At least one desired voice type may be registered as a favorite. In this case, the voice selection function 421 may be implemented so that one of voices registered as a favorite is selected by the user.

Also, the content editing tool 320 may provide a sound effect adding function 430 for adding a pre-made sound effect to the timeline in association with the image. The sound effect adding function 430 may include functions for displaying a list of a plurality of pre-produced sound effects, performing a preview of the sound effects, or adding the sound effect at a specific time in the timeline. If necessary, the user can add the desired sound effect from an external file or create it directly.

Also, the content editing tool 320 may display a time indicator 450 indicating a specific point in the timeline. 4 shows an example in which the time indicator 450 indicates the time of 00:00.00 by default.

In addition, the save button 460 shown in the content editing tool 320 of FIG. 4 may provide a function for saving the edit of the current project, and the download button 470 generates image content and sends it to the user's electronic device. A function for downloading may be provided.

5 shows an example of a second screen 500 of the content editing tool 320 . In the second screen example 500 of FIG. 5 , as images are uploaded through the function 410 described in FIG. 4 , some of the thumbnails of the uploaded images are displayed through the timeline display function 440 . At this time, each thumbnail is displayed in the timeline display function 440 to correspond to a preset time interval (a time interval of 4 seconds in the embodiment of FIG. 5 ). In addition, the timeline and thumbnails may be searched for by clicking and dragging the area of the timeline display function 440 (or using a touch and drag or swipe gesture for a touch screen environment).

6 shows an example in which another area of the timeline display function 440 is displayed by clicking and dragging the area of the timeline display function 440 as a third screen example 600 of the content editing tool 320, have. In the third screen example 600, it can be seen that 10 images have been uploaded by the user through the thumbnail 10, which is the last thumbnail. As already described, 10 images are uploaded in the form of individual images or one file including pages imageable as 10 images, or a file containing pages imageable as n images and m images. Individual images (here, n and m are natural numbers n+m=10) may be uploaded in a combined form. It will be readily appreciated that a combination of two or more files and individual images may be used.

7 is an example of a fourth screen 700 of the content editing tool 320, and shows an example in which the time interval of the thumbnails is adjusted. For example, in the fourth screen example 700 of FIG. 7 , the horizontal length of thumbnails displayed in the area of the timeline display function 440 may correspond to the time (or section) the images occupy on the timeline. In this case, the fourth screen example 700 shows an example in which the user extends the length of the thumbnail by clicking the right end of the thumbnail 2 and dragging it in the right direction. In this case, the time (hereinafter, running time) that the image corresponding to the thumbnail 2 occupies on the timeline may increase according to the length of the extended thumbnail 2 . At this time, in the fourth screen example 700, while the user clicks on the right end of the thumbnail 2, a time point on the timeline corresponding to the right end of the thumbnail 2 (the time point of 9.9 seconds) is displayed. User interface 710 is appearing Accordingly, the user may adjust the length of the thumbnail 2 based on the time displayed on the user interface 710 . Meanwhile, as the length of thumbnail 2 increases, the start time of thumbnails after thumbnail 2 (eg, thumbnail 3 to thumbnail 10) may be changed. In the embodiment of FIG. 7, an example of adjusting the running time of an image corresponding to a thumbnail by adjusting the length of thumbnail 2 has been described, but this description can be equally applied to each thumbnail of the timeline display function 440 It will be easy to understand.

FIG. 8 is an example of a fifth screen 800 of the content editing tool 320, showing an example in which the time interval of thumbnail 4 is reduced. In the fifth screen example 800 , the user clicks the right end of the thumbnail 4 and then drags it to the left to reduce the length of the thumbnail. In this case, the running time of the image corresponding to the thumbnail 4 may be reduced according to the reduced length of the thumbnail 4 . In this case, in the fifth screen example 800 , while the user clicks on the right end of the thumbnail 4, a user interface 810 in which a time point on the timeline corresponding to the right end of the thumbnail 4 (a time point of 17 seconds) is displayed. ) is shown. Meanwhile, as the length of thumbnail 4 is reduced, the start time of thumbnails after thumbnail 4 (eg, thumbnail 5 to thumbnail 10) may be changed.

In the embodiments of FIGS. 7 and 8 , the user clicks on the right end of the thumbnail and drags left and right to increase or decrease the length of the thumbnail, thereby increasing or reducing the running time of the image corresponding to the thumbnail. explained. Through this description, according to the embodiment, the content editing tool 320 clicks the left end of the thumbnail and drags it left and right to increase or decrease the length of the thumbnail, thereby increasing or decreasing the running time of the image corresponding to the thumbnail. It will be easy to understand that functions may be provided.

FIG. 9 is a sixth screen example 900 of the content editing tool 320 , and shows an example in which the order of thumbnails is changed. The content editing tool 320 may provide a function for changing the order of thumbnails by a user clicking and dragging a specific thumbnail (touch and drag in a touch screen environment). For example, the user may change the order of thumbnail 1 and thumbnail 2 by clicking and dragging thumbnail 1 to the right in the fifth screen example 800 . The sixth screen example 600 shows a state in which the order of the thumbnail 1 and the thumbnail 2 is changed.

10 is a seventh screen example 1000 of the content editing tool 320, and shows an example in which a specific thumbnail is deleted. The content editing tool 320 may provide a function for the user to select a specific thumbnail and then delete it. For example, when a user generates a mouse-over event for a specific thumbnail, a user interface for deleting the corresponding thumbnail may be displayed, and the user may delete the corresponding thumbnail using the displayed user interface. It will be easily understood that various methods for deleting such a thumbnail may be provided. For example, the user may select a specific thumbnail by clicking it with a mouse, and then press a "Del" key on the keyboard to delete the selected thumbnail.

11 and 12 are examples of an eighth screen 1100 and a ninth screen example 1200 of the content editing tool 320, illustrating examples in which dubbing is added. As already described, the time indicator 450 may indicate a specific point in the timeline. For example, the user may move the time indicator 450 by dragging the time indicator 450 or clicking a desired location of the timeline. The time “00:06.00” displayed in association with the time indicator 450 in the eighth screen example 1100 may indicate a time point indicated by the current time indicator 450 in the timeline.

Also, in the eighth screen example 1100 , the text “Hello. I am AAA” is input through the text input function 422 of the dubbing add function 420 . At this time, when the user selects the add dubbing button 424, as in the ninth screen example 1200, the voice synthesis indicator 1210 for the first voice synthesis corresponding to the text "Hello. I am AAA." It may be displayed in association with thumbnails in the area of the timeline display function 440 . In this case, the first voice synthesis may be generated by the content creation server 300 and transmitted to the content editing tool 320 as described above. On the other hand, the speech synthesis indicator 1210 includes at least a part of the corresponding text "Hello. I am AAA" ("Hello. I" in the ninth screen example 1200) and the voice used to generate the first speech synthesis. The type identifier (eg, the identifier ① 1220 of the voice type “voice 1”) may be displayed.

The length of the speech synthesis indicator 1210 may correspond to the length of the first speech synthesis, and the amount of displayable text may vary according to the length of the speech synthesis indicator 1210 . In this case, the time of the time indicator 450 shown in the eighth screen example 1100 is “00:06.00”, and the time of the time indicator 450 shown in the ninth screen example 1200 is “00:09.56”. In other words, it can be seen that the length of the speech synthesis indicator 1210 for the first speech synthesis is 3.56 seconds (00:09.56 - 00:06.00 = 00:03.56).

Meanwhile, when the user selects the preview button 423 in the eighth screen example 1100, the first voice synthesis corresponding to the text “Hello. I am AAA” will be output through the speaker of the user's electronic device. can In other words, the electronic device may output the first voice synthesis through the speaker under the control of the content editing tool 320 .

13 is an example of a tenth screen 1300 of the content editing tool 320. When the user generates an input such as a mouse over on the voice synthesis indicator 1210, the position of the mouse pointer (speech synthesis in a touch screen environment) An example in which the voice synthesis information 1310 is displayed in association with the position of the indicator 1210 and the position of the touch while the position of the touch is touched) is shown. The speech synthesis information 1310 may include a speech type (voice 1) used to generate speech synthesis, a length of speech synthesis (3.56 seconds (00:03.56)), and input text (hello, I am AAA). have.

FIG. 14 is an eleventh screen example 1400 of the content editing tool 320 and shows an example in which the user shortens the length of thumbnail 3 to fit the time indicator 450 . In this case, it can be seen that the length of the thumbnail 3 is 1.56 of the length of the first voice synthesis, and the running time of the image corresponding to the thumbnail 3 in the timeline for video content is 1.56 seconds.

FIG. 15 is a twelfth screen example 1500 of the content editing tool 320 and shows an example in which the user changes the start time of the first voice synthesis. In other words, it can be seen that the position of the voice synthesis indicator 1210 is changed in the twelfth screen example 1500 compared to the eleventh screen example 1400 . As an example, the user may change the position of the voice synthesis indicator 1210 by clicking and dragging the voice synthesis indicator 1210 to the left or right in the content editing tool 320, such a voice synthesis indicator 1210. The start time of the first speech synthesis may be changed according to a change in the position of . On the other hand, the change of the position of the voice synthesis indicator 1210 may be made through the input of the direction key of the keyboard while the corresponding voice synthesis indicator 1210 is selected (eg, clicked). This change of position may be applied in a common method not only to the voice synthesis indicator 1210 but also to each of the various indicators provided by the content editing tool 320 . Also, a plurality of indicators may be selected as one group. For example, as a plurality of indicators are sequentially selected (eg, clicked) while a “Shift” key on the keyboard is pressed, a plurality of indicators may be selected as a group. In this case, the user may change the positions of a plurality of indicators belonging to a corresponding group at once through dragging or inputting a direction key on the keyboard.

16 and 17 are examples of a thirteenth screen 1600 and a fourteenth screen example 1700 of the content editing tool 320, illustrating examples in which dubbing is further added.

In the thirteenth screen example 1600, after the user moves the time indicator 450 to the position of “00:05.78”, the user selects the voice type “Voice 2” through the voice selection function 421, and a text input function ( 422) to input the text "Nice to meet you" is shown. At this time, when the user selects the add dubbing button 424, as in the 14th screen example 1700, the speech synthesis indicator 1710 for the second speech synthesis corresponding to the text “Nice to meet you” is displayed on the timeline. It may be displayed in association with thumbnails in the area of the function 440 . As already described, the speech synthesis indicator 1710 includes at least a part of the corresponding text "Nice to meet you" ("Nice to meet you" in the 14th screen example 1700) and the speech type used to generate the second speech synthesis. An identifier of (eg, an identifier ② 1720 of the voice type “Voice 2”) may be displayed.

The length of the speech synthesis indicator 1710 may correspond to the length of the second speech synthesis, and the amount of displayable text may vary according to the length of the speech synthesis indicator 1710 . In this case, the time of the time indicator 450 shown in the thirteenth screen example 1600 is “00:06.00”, and the time of the time indicator 450 shown in the fourteenth screen example 1700 is “00:08.24”. In other words, it can be seen that the length of the speech synthesis indicator 1710 for the second speech synthesis is 2.24 seconds (00:08.24 - 00:06.00 = 00:02.24).

Meanwhile, when the user selects the preview button 423 on the thirteenth screen example 1600 , a second voice synthesis corresponding to the text “Nice to meet you” may be output through the speaker of the user's electronic device. In other words, the electronic device may output the second voice synthesis through the speaker under the control of the content editing tool 320 .

18 is a fifteenth screen example 1800 of the content editing tool 320, and shows an example of adding sound effects. In the fifteenth screen example 1800, the current time indicator 450 starts as the user selects sound effect 2 through the sound effect selection function 430 (eg, clicks the plus button in the dotted line box 1810). An example in which the indicator 1820 of sound effect 2 is added as a viewpoint is shown. In this case, the length of the indicator 1820 of the sound effect 2 may be 2.46 seconds as shown in the dotted line box 1810 . The indicator 1820 may also be moved to another viewpoint by the user by clicking and dragging.

In the above embodiments, information for generating image content is listed along a timeline through a total of three channels: one channel for thumbnails, one channel for voice synthesis, and one channel for sound effects. have been explained. However, two or more channels for voice synthesis and/or two or more channels for sound effects may be used according to an embodiment.

19 is an example of a sixteenth screen 1900 of the content editing tool 320, showing an example of using two or more channels for voice synthesis. In the sixteenth screen example 1900, two voice synthesis indicators 1210 and 1710 may be partially overlapped and displayed. This indicates that two speech synthesis may be simultaneously output in at least some timelines. Although it is shown that two channels for speech synthesis can be used in the embodiment of FIG. 19, it will be easily understood that three or more channels may be used. In addition, it will be easily understood that two or more channels for sound effect may be used.

20 is a flowchart illustrating an example of a method for generating image content according to an embodiment of the present invention. The image content creation method according to the present embodiment may be performed by the computer device 200 that provides a service for content editing support through the content creation tool 320 . In this case, the processor 220 of the computer device 200 may be implemented to execute a control instruction according to a code of an operating system included in the memory 210 or a code of at least one computer program. Here, the processor 220 causes the computer device 200 to perform the steps 2010 to 2090 included in the method of FIG. 20 according to a control command provided by the code stored in the computer device 200 . can control

In operation 2010, the computer device 200 may extract snapshots of images uploaded through a content editing tool. As already described, the images may be uploaded in the form of a plurality of individual images, a single file including a plurality of images, or a combination of a single file and a plurality of images. In certain embodiments, images may be uploaded in the form of a file including a plurality of imageable pages. For example, when a PDF file is uploaded, the computer device 200 may extract an image from the PDF file and store it as a plurality of image files, and may extract snapshots for each of the plurality of image files.

In operation 2020, the computer device 200 may display the extracted snapshots according to a timeline through a content editing tool. Here, the length of the displayed snapshots may be proportional to the running time, which is a time occupied by images corresponding to the displayed snapshots on the timeline. In this case, the computer device 200 may display the extracted snapshots with a length proportional to the default running time through the content editing tool. Previously, in FIG. 5 , an example of displaying snapshots with a length proportional to a default running time of 4 seconds has been described.

In step 2030, the computer device 200 may provide a function for changing the order of the displayed snapshots. As an example, an example of changing the positions of the thumbnails 1 and 2 has been described in FIGS. 8 and 9 . According to an embodiment, the computer device 200 may further provide a function for deleting a specific thumbnail.

In operation 2040, the computer device 200 may provide a length adjustment function for adjusting the length of the snapshots displayed through the content editing tool. For example, the computer device 200 increases or decreases the length of the first snapshot according to the user's touch and drag or click and drag on the left area or the right area preset for the first snapshot among the displayed snapshots. function can be provided. In addition, the computer device 200 displays a viewpoint on the timeline for the left or right end of the first snapshot while the user's touch or click on the left or right area of the first snapshot is maintained. can do. As an example, in FIGS. 7 and 8 , the length of the thumbnail can be increased or decreased, and an example in which a viewpoint on the timeline is displayed at the right end of the corresponding snapshot has been described.

In operation 2050 , the computer device 200 may adjust the running time of the snapshot whose length is adjusted through the length adjustment function according to the adjusted length. For example, the computer device 200 may increase or decrease the running time, which is a time occupied by an image corresponding to the length-adjusted snapshot on the timeline, in proportion to the degree to which the length of the snapshot is adjusted.

In operation 2060, the computer device 200 may generate a voice synthesis for text input through the content editing tool and add it to the selected time point of the timeline. In this case, the computer device 200 may generate a speech synthesis for the text according to the speech type selected through the content editing tool. A number of voice types can be created in advance and provided to users through a content editing tool according to people of all ages, languages (Korean, English, Chinese, Japanese, Spanish, etc.), emotions (joy, sadness, etc.) In the tool, you can select a specific speech type to use for speech synthesis. Also, the computer device 200 may add the voice synthesis generated at a specific point in the selected timeline by moving a time indicator indicating a specific point in time on the timeline. Previously, examples of adding speech synthesis at a time point selected through the time indicator 450 have been described in FIGS. 11 and 12 and FIGS. 16 and 17 .

According to an embodiment, when the first voice synthesis to be added to the timeline overlaps with the second voice synthesis already added to the timeline at least a part of the running time, the computer device 200 performs the first voice synthesis to the second voice synthesis. and other voice channels can be added to the timeline. In other words, dubbing may be performed so that two or more voice syntheses are simultaneously output from the generated image content. In FIG. 19, an example in which two speech synthesis is added to the timeline through different channels has been described.

Also, the computer device 200 may display an indicator for voice synthesis added at a selected time in the timeline through the content editing tool. According to an embodiment, at least a portion of the text may be displayed through the indicator, and the length of the indicator may be proportional to the length of the speech synthesis. Here, the length of the speech synthesis may mean a time during which the speech synthesis is output.

In addition, the computer device 200 outputs speech synthesis information including at least one of information on a speech type used to generate a speech synthesis, information on a length of speech synthesis, and text based on a user input to the indicator. can do. The speech synthesis information may be displayed in association with a location where a user input to the indicator occurs. For example, in FIG. 13 , an example of displaying the voice synthesis information 1310 for the voice synthesis indicator 1210 has been described.

In operation 2070, the computer device 200 may move the position on the timeline of the voice synthesis added to the timeline based on the user's input. As an example, examples in which the position of speech synthesis can be moved based on an input such as a user's click and drag or touch and drag have been described in FIGS. 14 and 15 .

In operation 2080, the computer device 200 may receive a selection of one sound effect from among a plurality of sound effects provided through the content editing tool. As an example, in FIG. 18 , it has been described that a plurality of sound effects can be provided to the user through the sound effect addition function 430 , and that the user can select one of the plurality of sound effects.

In operation 2090, the computer device 200 may add the selected sound effect to the timeline at the selected time point in the content editing tool. For example, in FIG. 18 , an example of adding sound effect 2 at a time point selected through the time indicator 450 has been described.

In this case, according to an embodiment, at least some of the steps 2010 to 2090 may be performed in parallel. As an example, steps 2040 and 2050 may be triggered by a user input for length adjustment, and steps 2060 and 2070 may be triggered by a user input for addition of speech synthesis. Also, steps 2080 and 2090 may be triggered by a user input for adding a sound effect. Accordingly, the order of the steps 2040 to 2090 may be changed according to a user's input.

Later, when the user requests to generate image content, the computer device 200 may generate a moving image after normalizing the images to a size tailored to the image content. According to an embodiment, the computer device 200 may insert a watermark and/or a caption into the image content. Thereafter, the computer device 200 may generate final video content by inserting voice synthesis and/or sound effects in the video according to the timeline.

As described above, according to embodiments of the present invention, a voice synthesis desired by a user can be generated in real time for a plurality of images and dubbed at a playback start time desired by the user, and the generated voice synthesis is dubbed in a plurality of images. video contents can be created and provided through them.

The system or apparatus described above may be implemented as a hardware component or a combination of a hardware component and a software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The medium may continuously store a computer executable program, or may be a temporary storage for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, or servers. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

A method for generating image content in a computer device including at least one processor, the method comprising:
extracting, by the at least one processor, snapshots of images uploaded through a content editing tool;
displaying, by the at least one processor, the extracted snapshots according to a timeline through the content editing tool;
providing, by the at least one processor, a length adjustment function for adjusting the length of the displayed snapshots displayed in the content editing tool through the content editing tool;
adjusting, by the at least one processor, a running time of a snapshot whose length is adjusted through the length adjustment function according to the adjusted length; and
generating, by the at least one processor, a speech synthesis for text input through the content editing tool, and adding it to the selected time point of the timeline;
Video content creation method comprising a. According to claim 1,
The length at which the displayed snapshots are displayed in the content editing tool is proportional to the running time, which is a time occupied by images corresponding to the displayed snapshots on the timeline,
The step of displaying according to the timeline,
Displaying the extracted snapshots with a length proportional to the default running time through the content editing tool
A method of generating video content, characterized in that According to claim 1,
The step of providing the length adjustment function,
and providing a function of increasing or decreasing the length of the first snapshot according to a user's touch and drag or click and drag with respect to the left or right area preset for the first snapshot among the displayed snapshots. How to create video content with 4. The method of claim 3,
The step of providing the length adjustment function,
While the user's touch or click on the left area or the right area of the first snapshot is maintained, a time point on the timeline for the left end or right end of the first snapshot is displayed. How to create video content with According to claim 1,
The step of adjusting the running time according to the adjusted length,
The method of claim 1, wherein the running time, which is a time occupied by an image corresponding to the length-adjusted snapshot, on the timeline, is increased or decreased in proportion to the degree to which the length is adjusted. According to claim 1,
The step of generating the speech synthesis and adding it to the selected time point of the timeline comprises:
and generating a voice synthesis for the text according to the voice type selected through the content editing tool. According to claim 1,
The step of generating the speech synthesis and adding it to the selected time point of the timeline comprises:
and adding the generated voice synthesis at a specific point in the timeline selected through movement of a time indicator indicating a specific point in time on the timeline. According to claim 1,
moving, by the at least one processor, a position on the timeline of the speech synthesis added to the timeline based on a user input;
Image content generation method further comprising a. According to claim 1,
receiving, by the at least one processor, one of a plurality of sound effects provided through the content editing tool; and
adding, by the at least one processor, the selected sound effect at a time point selected for the timeline in the content editing tool
Image content generation method further comprising a. According to claim 1,
providing, by the at least one processor, a function for changing the order of the displayed snapshots;
Image content generating method, characterized in that it further comprises. According to claim 1,
The images are image content creation method, characterized in that it is uploaded in the form of a file including a plurality of imageable pages. According to claim 1,
The step of generating the speech synthesis and adding it to the selected time point of the timeline comprises:
When the first voice synthesis to be added to the timeline overlaps with the second voice synthesis already added to the timeline at least a part of the running time, the first voice synthesis is transferred to a different voice channel than the second voice synthesis. A method of creating video content, characterized in that it is added to the timeline. According to claim 1,
The step of generating the speech synthesis and adding it to the selected time point of the timeline comprises:
and displaying an indicator for voice synthesis added at the selected time point of the timeline through the content editing tool. 14. The method of claim 13,
At least a part of the text is displayed through the indicator. 14. The method of claim 13,
The length of the indicator is proportional to the length of the voice synthesis method. 14. The method of claim 13,
The step of generating the speech synthesis and adding it to the selected time point of the timeline comprises:
and displaying at least one of information on a voice type used to generate the voice synthesis, information on a length of the voice synthesis, and the text based on a user input to the indicator. A computer program stored in a computer-readable recording medium in combination with a computer device to cause the computer device to execute the method of any one of claims 1 to 16. 17. A computer-readable recording medium in which a computer program for executing the method of any one of claims 1 to 16 in a computer device is recorded. at least one processor implemented to execute computer-readable instructions
including,
by the at least one processor,
Extract snapshots of uploaded images through the content editing tool,
Displaying the extracted snapshots along a timeline through the content editing tool,
Through the content editing tool, the displayed snapshots provide a length adjustment function for adjusting the displayed length in the content editing tool,
Adjusting the running time of the snapshot whose length is adjusted through the length adjustment function according to the adjusted length,
Creating a speech synthesis for text input through the content editing tool and adding it to the selected time point of the timeline
A computer device characterized by a. 20. The method of claim 19,
to provide the length adjustment function, by the at least one processor,
Providing a function of increasing or decreasing the length of the first snapshot according to a user's touch and drag or click and drag on a left or right region preset for the first snapshot among the displayed snapshots
A computer device characterized by a.

Images