KR102360919B1 - A host video directing system based on voice dubbing - Google Patents

Abstract

A voice dubbing-based host video editing system according to the present invention comprises: an image database for receiving a plurality of images including the face of a host and setting and storing a representative image; a region detection module for detecting a face region and a mouth region from the face region from the images stored in the image database; an object extraction module for extracting and storing a mouth-shaped object for each pronunciation of a voice from the mouth region; a voice management module including a voice input unit for receiving an information voice including guide information and a pronunciation analysis unit for analyzing pronunciation of the information voice; a synthesis module for generating a synthesized image by synthesizing the mouth-shaped object into the mouth region of the representative image depending on the pronunciation of the information voice; and an output module for outputting the synthesized image by dubbing the information voice. Accordingly, it is possible to provide a more natural image.

Description

A host video editing system based on voice dubbing {A HOST VIDEO DIRECTING SYSTEM BASED ON VOICE DUBBING}

The present invention relates to a moderator video editing system based on voice dubbing. More specifically, the present invention synthesizes and processes informational voice that delivers information to the moderator's video, and allows the moderator's mouth shape to be modified to match the pronunciation of the information voice. It relates to a moderator video editing system based on voice dubbing that enables the actual moderator to transmit information even if they do not shoot all the videos.

News, more preferably TV news, is about 5W 1H, that is, Who, When, Where, What, Why This is a video that guides you according to the (Why) and How (How) principles.

In such news, a presenter represented by an announcer appears. For this purpose, in the conventional news, the presenter used to film and provide a video providing information on the news in a studio.

However, in order to proceed with such a conventional news, since each time a studio shoots and provides an image in which the presenter delivers information, there is a limitation in space and time, and there is a limitation that a presenter is always needed to provide news.

In addition, in recent years, deepfake technology, meaning a fake video produced using artificial intelligence (AI) technology or the production process itself, has been widely used. It is a compound word of fake). It is a technology that creates by superimposing existing photos or videos on the original using machine learning (ML) technology called Generative Adversarial Network (GAN).

Accordingly, techniques for creating various additional images by superimposing existing photos or images on the originals and modifying the originals have been introduced and used.

As a prior art for this, Korean Patent Registration No. 10-0682889 discloses a 'image-based realistic three-dimensional face modeling method and apparatus'.

The present invention relates to a realistic three-dimensional face modeling method and apparatus based on an image, the method comprising: a method for generating a three-dimensional realistic face model expressed by a textured polygonal mesh model, the front face of an input image and detecting facial features in the side face image. generating a three-dimensional face model by transforming an initial model using detected facial features to determine a three-dimensional shape of a specific face; generating a realistic texture from the input image; and mapping the texture onto the 3D model.

However, in the case of the prior art described above, as a three-dimensional face model is generated and a texture is generated, the entire face must be modeled based on facial features, so it takes a long time to generate an image and there is a problem that the amount of information to be processed is large. there was.

Therefore, in order to solve the above-mentioned problems, the moderator can provide a composite image using an existing image without taking an image every time to provide information, but to provide a change in the shape of the mouth only. Therefore, there is a need to develop an audio dubbing-based moderator video editing system that reduces the amount of information processing.

The main purpose of the present invention is to create and provide a synthesized image in which the shape of the mouth is synthesized according to the pronunciation of the information voice based on the existing image, even if the moderator who appears on the image and provides information does not shoot the image every time. The purpose.

Another object of the present invention is to provide a more realistic synthetic image by allowing a change of a face according to an emotion to be reflected in the synthetic image generation.

Another object of the present invention is to perform image correction processing to provide a more natural image.

It is a further object of the present invention to enable the composition of the background as well as the composition of the presenter's mouth shape, thereby eliminating the constraint of the location.

In order to achieve the above object, an audio dubbing-based moderator video editing system according to the present invention includes: an image database for receiving a plurality of images including a face of the moderator, setting and storing representative images; a region detection module configured to detect a face region and a mouth region from the face region from the plurality of images stored in the image database; an object extraction module for extracting and storing a mouth-shaped object for each pronunciation of a voice from the mouth region; a voice management module comprising: a voice input unit for receiving an information voice including guide information; and a pronunciation analysis unit for analyzing pronunciation of the information voice; a synthesis module for generating a synthesized image by synthesizing the mouth-shaped object into the mouth region of the representative image according to the pronunciation of the information voice; and an output module for outputting the information voice by dubbing the synthesized image.

Furthermore, the voice input unit further includes an additional information input part for receiving the emotion information of the guide information, and the object extraction module includes an emotion classification unit for classifying the plurality of mouth-shaped objects by emotion, and the guide information and an object extracting unit for extracting the mouth-shaped object matching the emotion information of It is characterized in that it generates a composite image.

In addition, the synthesis module includes a skin pixel detection part for detecting skin pixels at a boundary between the mouth region and the face region belonging to the synthesized image, a boundary correction part for correcting saturation and brightness of the boundary region, and , It characterized in that it further comprises a boundary correction unit including a smudging part for smudging the boundary portion.

In addition, the image database receives and stores a background image including any one of indoor and outdoor places, and the region detection module includes, in the plurality of images, an object region including the moderator and an object region other than the object region. and a background detection unit for discriminating a background region that is a region, and the synthesizing module includes a background synthesizing unit for synthesizing the background image with the background region.

A presenter video editing system based on voice dubbing according to the present invention,

1) A synthesized image is created by synthesizing a mouth shape that matches the pronunciation of the inputted information voice from the previously recorded image of the presenter, and providing it together with the information voice, allowing the presenter to provide information without the need to shoot a video every time and at the same time to reduce the amount of information processing,

2) Even with the same pronunciation, it is possible to create a composite image reflecting the emotional state to reflect the mouth shape that varies depending on the emotional state, thereby enhancing the sense of reality.

3) It provides a more natural image by enabling blurring and boundary correction around the synthesized mouth shape.

4) By allowing various backgrounds to be synthesized together, it has the effect of providing a more realistic and realistic image.

1 is a conceptual diagram showing a schematic configuration of a system of the present invention.
Fig. 2 is a block diagram showing the overall configuration of the system of the present invention;
3 is a conceptual diagram illustrating a face region and a mouth region;
4 is a conceptual diagram illustrating an example of a screen of a composite image including an object region and a background region according to the present invention.
5 is a conceptual diagram illustrating a highlight area, a shadow area, a point area, and a shadow area according to the present invention;
6 is a conceptual diagram illustrating a gradation area calculated based on a virtual connecting line and a virtual progress line.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The accompanying drawings are not drawn to scale, and like reference numbers in each drawing refer to like elements.

1 is a conceptual diagram showing a schematic configuration of a system of the present invention.

Referring to FIG. 1 , the moderator video editing system of the present invention includes a moderator 1 and a central control server 2 .

The moderator 1 is a person who provides guidance information by appearing on a video, and may be a news announcer or anchor, a weather forecaster for a weather forecast, a show host for home shopping, etc., and a moderator 1 for other TV programs.

The presenter 1 is actually a person who appears in the video and the synthesized image generated through synthesis, and as will be described later, the presenter 1 may also be the person providing the information voice. Therefore, the moderator 1 can be said to be a person who provides a face by appearing in an image providing guide information.

Therefore, the moderator (1) must provide a plurality of images taken by himself/herself to the system of the present invention in order to generate a composite image. It means the recorded video.

The central control server 2 is a subject that generates a composite image through the presenter's video editing system of the present invention, and generates a composite image by synthesizing the mouth shape according to the mouth shape of the information voice to the representative image, and the information voice to the composite image. It performs the function provided by dubbing.

The central control server 2 is a series of subjects for realizing the system of the present invention, and includes a server PC and a network communication network together. In addition, the central control server 2 has a central processing unit (CPU) and a program that can be executed in the central processing unit on a hardware basis having storage means such as memory and hard disk, that is, software is installed and this software can be executed. A series of specific configurations of such software will be described later as structural units such as 'modules', 'parts', and 'parts'.

The configuration of these 'modules' or 'parts' or 'interfaces' or 'parts' is installed and stored in the storage means of the central control server (2), and software or FPGA or ASIC executed through the CPU and memory. It means a piece of hardware. In this case, the configuration of 'module', 'unit', and 'interface' is not limited to hardware, and may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. As an example, 'module' or 'part' or 'interface' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, and properties. It includes fields, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables.

Functions provided by these 'modules' or 'units' or 'interfaces' may be combined into a smaller number of components and 'units' or 'modules' or additional components and 'units' or 'modules' can be further separated.

In addition, the central control server 2 means all kinds of hardware devices including at least one processor, and may be understood as encompassing software configurations operating in the corresponding hardware devices according to embodiments. For example, a computing device as an example of a server may be understood as meaning including a smartphone, a tablet PC, a desktop, a notebook computer, and a user client and an application running in each device, and is not limited thereto.

Hereinafter, the detailed configuration of the presenter's video editing system based on voice dubbing of the present invention based on the configuration of the central control server 2 will be described with drawings.

2 is a block diagram showing the overall configuration of the system of the present invention, and FIG. 3 is a conceptual diagram illustrating a face region and a mouth region.

Referring to Figs. 2 and 3, the present invention's audio dubbing-based video editing system for a presenter is described. It is characterized in that it basically includes an image database 100 , a region detection module 200 , an object extraction module 300 , a voice management module 400 , a synthesis module 500 , and an output module 600 .

The image database 100 performs a function of receiving and storing a plurality of images in which the presenter 1 appears to capture the face of the presenter 1, and sets a representative image among the images. Here, the plurality of images refers to an image in which the face of the presenter 1 is captured and includes the face of the presenter 1, and as described above, it may be a news image, a weather forecast image, a TV show program image, a home shopping image, etc. , or, of course, there is no limitation as long as it is a video in which the presenter 1 can appear and provide information.

Therefore, the input of an image can be referred to as a pre-generated image taken by the presenter 1, or it is also possible to store the image in the image database 100 whenever an image in which the presenter 1 appears is created. do.

The number of images to be stored is also not limited, but preferably, it should include at least one image including a representative image, and as the number of images increases, the material for synthesis increases. The number may be tens to hundreds, thousands, or even tens of thousands. In order to generate a more realistic synthetic image, it is preferable that the number of images stored in the image database 100 is a number that can achieve big data.

Since there is no restriction on the shooting location of the image, there is no restriction in studio shooting, outdoor shooting, set shooting, etc., so all images taken at various places and times are stored in the image database 100 and used for producing a composite image to be described later. .

In addition, the image database 100 sets and stores at least one of the images as a representative image. Here, the representative image is the original image before generating the synthesized image, and can be called an original for mouth shape synthesis. Of course, all images can be representative images, but most preferably, the image of the host 1 in a studio with a solid background for easy synthesis may be the representative image, which is used for conventional weather forecast or news shooting. Think of a video shot in a studio.

Therefore, in the case of the representative image that is the source of the synthesis, it is preferably an image taken indoors or in a studio rather than an image taken outdoors, and an indoor studio with a solid background so that it is easy to distinguish between the object (the presenter 1) and the background. can be filmed in Here, the representative image may be one or plural, and the number is not limited.

The region detection module 200 detects the face region 10 of the presenter 1 and the mouth region 11 within the face region 10 of the image captured by the presenter 1 stored in the image database 100 . perform the function At this time, there is no limitation in the detection method of the face region 10 and the mouth region 11, but preferably, by detecting the outline of the face, the region within the closed curve formed by the outline is detected as the face region 10, and the face region 10 ) by detecting the outline of the lips, the inner region formed by the outline of the lips may be detected as the mouth region 11 .

Since it is said that the face of the presenter 1 is captured and the image is included in the image, only the face region 10 of the presenter 1 is detected from the image, and the mouth region ( 11) performs the function of detecting only.

The object extraction module 300 extracts and stores a mouth-shaped object for each pronunciation of a voice with respect to the mouth region 11 detected in each of the plurality of images stored in the image database 100 .

For example, since each image is an image that transmits information, the moderator 1 transmits information based on voice, and at this time, the mouth area 11 moves according to the pronunciation of the voice. For example, if the information content is ‘hello’, the pronunciation corresponding to the text composing the information content such as ‘an’, ‘nyeong’, ‘ha’, ‘se’, and ‘yo’ is pronounced.

At this time, the mouth shape is different for each pronunciation of each letter, and the object extraction module 300 extracts and stores the mouth shape object for each pronunciation. That is, only the mouth area 11 is detected from each image, cut out, and only the mouth area 11 is stored, but the mouth shape for each pronunciation is stored. For example, the part of the image in which only the mouth region 11 corresponding to the mouth shape pronouncing 'in' in the image is cut out, that is, the part where only the mouth region 11 is edited, is a mouth-shaped object for the pronunciation of 'in'. will become

In this case, the mouth shape object for each pronunciation may be preferably stored for each pronunciation of the Korean language. Furthermore, it is also possible to cut and store only the mouth area 11 from the image for each pronunciation of the language in a foreign language and store it as a mouth shape object for each pronunciation. .

Furthermore, since there is a standard pronunciation method in Korean, the content of information to be provided from the input image is converted into text, written according to the standard pronunciation method, and divided into syllables to form the mouth region of the image corresponding to the syllable. It is also possible to cut only (11) and extract it as a mouth-shaped object for each pronunciation and save it.

The voice management module 400 includes a voice input unit 410 and a pronunciation analyzer 420 to input an information voice to be provided through a newly produced synthetic image, and analyze the pronunciation of the information voice.

The voice input unit 410 performs a function of receiving an information voice including guide information, where the guide information is the content of information to be provided. For example, if the synthetic video to be provided is a weather forecast, the information corresponding to 'tomorrow's weather will be sunny' can be guide information, and the information voice refers to the voice that reads the content.

In this case, the person providing the information voice may preferably be the moderator 1, but it is also possible for someone other than the moderator 1 to provide the information voice. can do. Alternatively, it is also possible to input the guide information as text, convert it to voice through a text to sound (TTS) function, and input the information as voice information.

The pronunciation analysis unit 420 performs a function of analyzing the pronunciation of the information voice, and analyzes the pronunciation of the information voice, which is a voice for the guidance information, to the extent that the text corresponding to the guidance information and the actual pronunciation may be different.

For example, as described above, when 'tomorrow's weather will be sunny' is the guide information, the pronunciation of the information voice becomes 'tomorrow's weather was 鞭求蔑?'. In this case, the pronunciation analysis of the information voice is preferably based on the analysis based on the standard pronunciation method.

The synthesis module 500 generates a synthesized image by synthesizing the mouth-shaped object corresponding to the pronunciation in the mouth region 11 of the representative image according to the pronunciation of the information voice analyzed by the pronunciation analyzing unit 420 .

That is, in accordance with the pronunciation and playback speed of the information voice, a mouth-shaped object corresponding to the pronunciation is synthesized in the mouth region 11 detected in the representative image, which is the source of synthesis, to generate a synthesized image in which the mouth shape is changed in the representative image. will do

Here, if there is one representative image, a mouth-shaped object is synthesized in the corresponding representative image, and if there are a plurality of representative images, the mouth-shaped object is synthesized in any one of the plurality of representative images. In this case, the subject who selects one of the representative images may preferably be a system administrator, or may randomly select one representative image on the system.

As described above, when 'tomorrow's weather will be sunny' is the guide information, the pronunciation of the information voice is 'tomorrow's weather was horse? The mouth region ( 11), the mouth-shaped object is synthesized so as to correspond to the contents and pronunciation of the information voice, the reproduction speed of the information voice, and the length of the voice.

Finally, through the output module 600, the synthesized image is dubbed with information voice and output. At this time, if a sound such as a voice is inserted in the representative image, which is the original of the synthesized image, the corresponding sound, that is, the sound is muted, and the synthesized image Overlay the sound of the information voice so that the information voice is output as sound together with the composite image.

Therefore, through such an image editing system, the information voice to be transmitted is separately input, and only the mouth area 11 in the previously photographed image of the presenter 1 is changed to a mouth shape corresponding to the pronunciation of the information voice. After creating a synthesized image synthesizing the shape of the mouth, the information voice input to the synthesized image can be dubbed and output, so that the presenter 1 appears and You can make it look like you are providing guidance information.

Furthermore, since only the mouth shape is synthesized at this time, the amount of information processing is simple compared to deepfake processing of the entire face of the presenter (1), so video production can be simplified, and many synthetic images are generated and provided at a faster rate Of course you can.

In order to provide a more realistic feeling, a more realistic composite image is provided by reflecting the fact that the mouth shape may be slightly different depending on the emotional state even if the mouth shape is based on the same pronunciation. can make it For example, even if the same 'ah' is pronounced, it is possible to indicate that the corner of the mouth is slightly raised in the mouth area 11 when it is a positive emotion, and that the corner of the mouth is drooping downward when it is a negative emotion. it will reflect

Accordingly, it may further include a configuration capable of changing the shape of the mouth, preferably the height of the tail of the mouth, by reflecting the content of the information voice, that is, the emotional state of the guide information. To this end, first, the voice input unit 410 inputs additional information A part 411 may be further included.

The additional information input part 411 performs a function of receiving the emotional information of the guide information, and this is when the guide information represents a specific emotion, for example, if the information is bright and joyful, it is emotional information as 'positive mood, lively' .

That is, if the guide information contains content that represents a specific emotion, the emotional state of the guide information, which may be difficult to determine only with text or voice, can be separately inputted when inputting information and voice, so that the system can provide information that corresponds to information voice. It is possible to understand the emotional state of the guide information. In this case, the emotional information may be preferably input by a person who recorded the information voice, or may be input by a system administrator.

Furthermore, the object extraction module 300 may further include an emotion classification unit 310 and an object extraction unit 320 in order to classify the mouth-shaped object by emotion and to extract the mouth-shaped object matching the guide information.

The emotion classification unit 310 classifies a plurality of stored mouth-shaped objects according to pronunciations for each emotion. At this time, classification by emotion can be preferably identified with a change in the corner of the mouth. Even for a mouth-shaped object corresponding to the same pronunciation, if the corner of the mouth is raised, it generally indicates a positive emotion, so ' It can be classified into emotions such as 'joy', 'joy', 'brightness', 'cheerfulness', and 'happiness', and even if the pronunciation is the same, a mouth-shaped object with a raised corner can generally respond to negative emotions, so it is 'sad'. It can be classified into emotions such as depression, tears, and unhappiness. If the corners of the mouth are flat without any change, it can be classified as expressing neutral emotions such as 'no emotion' and 'dumbness'.

In addition, the emotion classification unit 310 may be classified into such detailed emotions, but may be classified more simply by classifying only positive, negative, and neutral emotions such as 'positive', 'negative', and 'no emotion'.

In this case, the classification may be performed by a system administrator, or emotions may be recognized through the entire identification of the face region 10 of the image on the system, and emotions may be given to the mouth-shaped object based on this and classified by emotions.

The object extraction unit 320 extracts only the mouth-shaped object matching the emotion information input through the additional information input part 411 as an emotion-matching object from among all the stored mouth-shaped objects.

For example, when the emotional information is a positive emotion, only the mouth-shaped object classified as representing a positive emotion is extracted as an emotion matching object, and when the emotion information contains a corrective emotion, the mouth classified as representing a negative emotion is extracted. It is to extract the shape object as an emotion matching object.

Furthermore, in the case of the synthesis module 500, as in the above-described basic configuration, a mouth-shaped object is synthesized in the mouth region 11 of the representative image according to the pronunciation and playback speed of the information voice to generate a synthesized image. In reflection, the emotion matching object is synthesized in the mouth area 11 of the representative image according to the pronunciation and reproduction speed of the information voice.

For example, even if a mouth-shaped object corresponding to the pronunciation of the same 'an', 'nyeong', 'ha', 'se', and 'yo' is synthesized, if the corresponding content is a positive emotion, the emotion classified as representing a positive emotion Synthesis processing is performed among the matching objects, and when the corresponding content contains negative emotions, the synthesis processing may be performed among the emotion matching objects classified as representing negative emotions.

Through this, the emotional state to be conveyed through the guide information can be expressed in the mouth shape synthesis, enabling expression of the emotional state based on the mouth area 11, especially the corner of the mouth, thereby creating a more realistic and realistic synthetic image. can be created and provided.

In addition, in order to generate a more realistic synthetic image, boundary surface smudging at the boundary surface of the mouth region 11 into which the mouth-shaped object is inserted, that is, the boundary surface blurring process is performed so that the synthesized part is not visible. Also possible, for this purpose, the synthesis module 500 may further include a boundary corrector 510 , wherein the boundary corrector 510 includes a skin pixel detection part 511 , a boundary correction part 512 , and a smudging part. (513).

The skin pixel detection part 511 detects a skin pixel on a boundary between the mouth area 11 and the face area 10 in the generated composite image, that is, a boundary surface corresponding to the outline of the mouth area 11 to be synthesized. Here, the skin pixel refers to detecting a pixel representing a skin color, that is, a general skin color spanning ivory to apricot color, and further brown, and the color of the skin among pixels close to the boundary of the mouth region 11 to be synthesized, that is, the outline boundary. To detect pixels with .

The boundary correction part 512 corrects the saturation and brightness of the detected boundary region. That is, by correcting color and brightness, a mouth-shaped object is synthesized to reduce the difference in chroma and brightness that may occur between the face area 10 and the mouth area 11, and to reduce the tone in the skin pixels in the boundary area. to create a natural boundary surface.

In the case of the smudging part 513, the boundary is blurred by smudging, that is, blurring the synthesized boundary surface by smudging, that is, the boundary of the synthesized part is not recognizable. will do Through this, the skin of the face region 10 can look cleaner and at the same time, a more natural composite image can be provided as the boundary surface is blurred and the boundary of the synthesized portion is difficult to recognize.

4 is a conceptual diagram illustrating an example of a screen of a composite image including an object region and a background region according to the present invention.

In the case of the synthesized image of the present invention, an object, that is, the face region 10 of the presenter 1 is detected from the representative image, and synthesis is performed only on the mouth region 11 among the face regions 10 to generate a synthesized image. However, as the representative image is preferably an indoor studio image that is easy to synthesize as described above, there is a limitation in that it can be limited to indoors.

In order to supplement this, as can be seen with reference to FIG. 4 , background synthesis may also be possible in the generation of a composite image.

To this end, first, the image database 100 may further receive and store a background image photographed indoors and outdoors, and the background image is stored separately from the image photographed by the presenter 1, It can be said that it is a video of only a landscape or a specific place. At this time, in the case of the outdoors, it is possible to photograph by time such as day, night, afternoon, evening, and morning, or it is also possible to photograph according to weather.

Therefore, by capturing various indoor or outdoor spaces in the form of images, receiving and storing background images including any one of indoor and outdoor places, it can be used as a database for the background.

In addition, the region detection module 200 further includes a background detection unit 210 and includes an object region 20 including the presenter 1, that is, representative The object area 20, which is the area of the presenter 1, which is an object, and the background area 30, which is an area other than the object area 20, can be distinguished.

In the image of the moderator 1, the moderator 1 can sit still or stand, but can also move, so that the moderator 1, a representative object of the video, is recognized, and the object area ( 20) and the background area 30 excluding it can be divided and detected.

In addition, the synthesizing module 500 may further include a background synthesizing unit 520, so that the background image is synthesized in the background area 30 of the representative image to change the mouth shape of the presenter 1 in the representative image. It is also possible to change the background.

Therefore, by changing only the background of the representative image shot in the indoor studio, it is possible to produce a composite image as if it was filmed in various places according to the content of the information voice, thereby providing the image without time and space restrictions.

Further, the background image may be taken not only indoors but also outdoors. In the case of a background image taken outdoors, there may be changes according to weather and time. That is, even if the background image was taken at the same outdoor location, the sense of presence may vary depending on the weather in the same time zone, whether it is sunny, cloudy, or raining, or the feeling of the background may change according to time, such as morning, lunch, evening, or night. have.

Therefore, preferably, even in the same outdoor location in the background image, if it is possible to provide by repeatedly shooting according to time and weather, it is possible to generate and provide a more realistic and realistic synthetic image. In addition, the sense of presence and realism can be maximized if the background to be synthesized is the background of the time and weather that coincides with the time when the synthesized image is provided to the viewer.

Therefore, in order to realize such a realistic image, the image editing system of the present invention may further include an environment identification module 700, wherein the environment identification module 700 includes an external environment identification unit 710 and a background classification unit. 720 may be included.

The external environment determining unit 710 determines the output time of the composite image through the output module 600, that is, the weather information of the place where the background image synthesized in the composite image was recorded based on the timing at which the composite image is provided to the viewer, and Get time information. In the case of time information, it can be determined that the output time of the composite image is identified, and the weather information can be confirmed by checking the weather forecast of the Korea Meteorological Administration for the corresponding time zone and place.

The background classification unit 720 performs a function of classifying a plurality of background images stored in the image database 100 of the present invention according to weather information and time information. That is, even with a background image taken at the same place, there may be differences according to weather information and time information. , rainy weather video at 5pm, snowy weather video at 5pm, sunny weather video at 6am, rainy weather video at 6am, rainy weather video at 8pm, 8pm It can be divided into images of snowy weather in

That is, even in the background image taken at the same place, weather information and time information can be reflected, so that the time and weather at which the background image was taken can be grasped, and background synthesis is possible based on this. In addition, it can be said that the more the number of background images is, the more effective the configuration is to be made effectively.

In addition, when the weather information and time information of the background image are identified as described above, the background synthesizer 520 is a background image that matches the weather information and time information of a place corresponding to the output time of the synthesized image through the output module 600 . is synthesized in the background area 30 of the representative image.

That is, in providing guide information in the form of an image, it is basic that it is desirable to provide a background image corresponding to the guide information. Assuming that it is output at 1:30 pm on May 5, 2020, it identifies the weather information of Gangnam-gu, Seoul at 1:30 pm on May 5, and represents the background image of Gangnam-gu, Seoul with the same weather information and time information. Compositing is performed on the background area 30 of the image.

As another example, if it is determined that the weather at 1:30 on May 5 in Gangnam-gu, Seoul will be clear, the background image of the sunny weather taken outdoors in Gangnam-gu, Seoul at around 1:00 pm in May is synthesized and processed. To the viewer, it is possible to provide a sense of realism as if the video was recorded and provided in real time.

Also, this is the same for foreign examples. If a story about May in Paris is provided as guide information, it is preferable to synthesize a background image taken outdoors in Paris in May. Here, if the generated composite image is output to the viewer at 1:30 on May 5, a background image identical to the weather in Paris at 1:30 on May 5, especially a background image with the same time zone and sense of seasons, is synthesized Thus, it is possible to provide the same effect as if the host 1 went to Paris and took the corresponding composite image.

In addition, when the background image is synthesized separately from the presenter 1 as described above, the brightness of the object region 20 and the object region 20 represented by the presenter 1 in the representative image preferably shot indoors are excluded. There may be a problem in that the brightness of the background area 30 is too different or the naturalness is deteriorated.

This can be more conspicuous in a composite image in which the background image taken outdoors is synthesized. When the background is night but only the host (1) is too bright, or the background is day but the host (1), the object, is relatively dark. , the realism may be reduced.

To this end, the brightness difference between the background area 30 and the object area 20 is reduced by adjusting the brightness of the object area 20 according to the brightness of the background image in the produced composite image, and furthermore, a more natural image is generated. For this, the synthesis module 500 may further include an image additional correction unit 530 . The image additional correction unit 530 is configured to include a brightness detection part 531 and a brightness adjustment part 532 .

The brightness detecting part 531 performs a function of recognizing a background brightness that is an average brightness of the background area 30 and an object brightness that is an average brightness of the object area 20 in the generated composite image. This is obtained by calculating the average brightness of pixels constituting the background area 30 and the average brightness of pixels constituting the object area 20 , where brightness refers to brightness, and Da Vinci Resolve work for image editing You can find the average values of gain, gamma, and lift based on the Sea Wavescope, or find and understand the values of highlight, exposure, and contrast in a video editing tool such as Premiere Pro.

Most preferably, the brightness value is an image element value existing in an image according to the definition of the luminance value, and a value used to indicate a discontinuous amount of brightness may be used. It can have a value ranging from 0 to 255.

Here, preferably, the background area 30 and the object area 20 are detected for each frame constituting the synthesized image, and the background brightness and the object brightness are respectively determined in the detected background area 30 and the object area 20, respectively. After the calculation, the average of the calculated background brightness and the average of the object brightness in the entire frame are calculated and compared.

The brightness control part 532 controls the brightness of the object according to the difference between the background brightness and the object brightness detected in the background area 30 and the object area 20 , that is, the function of adjusting the brightness of the object area 20 . carry out That is, the object area 20 matches the brightness of the background area 30 so that the object area 20 naturally blends with the background area 30. It can be implemented by adjusting each of the gain, gamma, and lift values, or by adjusting the values of highlight, exposure, and contrast in a video editing tool such as Premiere Pro.

Accordingly, it is possible to adjust the brightness of the object area 20 to match the brightness of the background area 30 by this configuration, so that the object area 20 , that is, the area in which the presenter 1 is photographed, is the background area 30 . ), and it becomes possible to create a more natural composite image.

5 is a conceptual diagram illustrating a highlight area, a shadow area, a point area, and a shadow area according to the present invention, and FIG. 6 is a conceptual diagram illustrating a gradation area calculated based on a virtual connecting line and a virtual progress line.

Furthermore, in the case of an outdoor background, in general, the main light source is sunlight, or in order to provide a natural and no backlight situation, it is common for light to naturally pour in from a light source located at a distance.

In this case, it is natural in terms of brightness that light comes down from the top of the object, which is generally referred to as the presenter 1, and provides a screen without backlight. When such an outdoor background is provided, the object area (20), that is, if the brightness of the presenter 1 can also be adjusted in the same way as the flow of light, it is possible to create a more realistic and natural synthetic image.

To this end, as shown in FIG. 5 , a highlight area 40 and a shadow area 50 are set in the background area 30 of the composite image, and the corresponding point area 60 and shadow area are also set in the object area 20 . By setting (70) to differentially control the brightness of the point area 60 and the shadow area 70, the object area 20 can be given the effect of receiving natural light with a more natural gradation of brightness. The additional correction unit 530 may include a highlight setting part 533 , a shadow setting part 534 , a shadow setting part 535 , and an additional adjustment part 536 .

The highlight setting part 533 divides the background area 30 constituting the composite image into a plurality of peripheral portions located outside the synthesized image, that is, the background region 30 located on the edge side of the synthesized image, and divides them. A part of the background area 30 located on the upper side of the background area 30 , that is, located on the upper edge, is set as the highlight area 40 , and a function of determining the brightness and position of the highlighted area 40 is performed.

Here, the highlight area 40 refers to an area with high brightness. Preferably, in the case of a background image shot outdoors, light pouring in from a part of the upper edge is the most natural light source, so the It can be said that setting

If the light source is sunlight as in the example of FIG. 5 , the upper peripheral portion of the background area 30 that receives sunlight most directly has the highest brightness. Therefore, this is set as the highlight area 40, and as described above, gain, gamma, and lift values based on the wave scope are identified in Da Vinci Resolve, respectively, or highlight and exposure used in video editing tools such as Premiere Pro. , it is possible to grasp the brightness by grasping the value of the contrast. In the case of the location, it is possible to determine the location by checking the location information of the area set within the image frame.

The shadow setting part 534 is to set a lower part of the periphery of the background area 30 located in a diagonally opposite direction from the highlight area 40 as the shadow area 50 . Here, since the shadow region 50 means the opposite end of the light source, it may be referred to as a region having the lowest brightness. Preferably, an object, that is, a lower portion of the background area 30 in the direction in which the toe of the presenter 1 is located, may be set as the shadow area 50 .

In addition, the diagonally opposite direction means that when the highlight area 40 is located at the upper left side of the composite image screen as in FIG. 5 , the shadow area 50 is generally located at the lower right side of the background area 30 . This means that when the highlight area 40 is located at the upper right corner of the composite image screen, the shadow area 50 is located at the lower left corner.

In addition, the shade setting part 535 sets the point area 60 and the shadow area 70 in the object area 20 , where the point area 60 is a highlight zone within the object area 20 . It can be said that the brightness is the highest among the object area 20 . The point area 60 may preferably be set according to the position of the highlight area 40 , and the part located closest to the highlight area 40 in the object area 20 is preferably set as the point area 60 . can be In general, the highlight area 40 may be the most natural and common in the vicinity of the face area 10 even within the object area 20 .

Conversely, the shadow area 70 may be referred to as an area having the lowest brightness within the object area 20 , and is preferably set according to the position of the shadow area 50 . Most preferably, even within the object area 20 , a portion located closest to the shadow area 70 may be set as the shadow area 70 . In general, the shadow area 70 may be the most natural and common in the object area 20 to be near the toe of the presenter 1 .

The additional adjustment part 536 adjusts the brightness of the point area 60 and the shadow area 70 according to the object brightness, the background brightness, the brightness of the highlight area 40, and the brightness of the shadow area 50. Each performs a control function.

Here, since it is said that the brightness of the point area 60 becomes the brightest highlight zone in the object area 20, it should be higher than the object brightness, which is the average brightness of the object area 20, but is adjusted to be lower than the brightness of the highlight area 40 which is the light source. desirable. In addition, if there is too much difference from the background brightness, a sense of difference may be heard, so it is preferable to adjust the adjustment within a range similar to the background brightness.

In addition, since the brightness of the shadow area 70 is the darkest zone in the object area 20 , it should be darker than the object brightness, which is the average brightness of the object area 20 , but the shadow which is the darkest area among the background areas 30 . It is preferable to adjust the brightness higher than the brightness of the region 70 . In addition, if there is too much difference from the background brightness, a sense of difference may be heard, so it is preferable to adjust the adjustment within a range similar to the background brightness.

The brightness of the point area 60 and the shadow area 70 can be adjusted by the video editing system administrator, and it can be a video editing tool such as Premiere Pro as described above or a program such as Da Vinci Resolve to be used. have.

Accordingly, the point area 60 and the shadow area 70 are set according to the object brightness, the background brightness, the brightness of the highlight area 40, and the brightness of the shadow area 50, and the point area 60 and the shadow area ( 70) can be differentially adjusted to create contrast in the object area 20, and through this, it is possible to provide a more realistic feeling as if the host 1 actually went to an outdoor place and filmed it. .

In addition, referring to FIG. 6 , when the highlight area 40 and the shadow area 70 are set, the composite image screen is divided based on the highlight area 40 so that the brightness can be gradually reduced, and further based on this Thus, it can be configured to enable detailed correction of the point region 60 and the shadow region 70 within the object region 20. It may further include a setting part 538 , a virtual progress line setting part 539 , a virtual call setting part 540 , and a gradation setting part 541 .

The virtual connecting line generation part 537 connects the center point of the highlight area 40 and the center point of the shadow area 50 as in FIG. 6 described above to form the highlight area 40 and the shadow area 50 in the composite image. Creates a virtual connecting line 81 connecting across. This may preferably be a line that crosses the composite image diagonally.

The central angle setting part 538 performs a function of setting the light transmission central angle according to the brightness of the highlight area 40 . Here, the central angle of light transmission is indicated by theta (θ) in the drawing, and as the brightness of the highlight area 40 increases, the light spreads widely. Therefore, the central angle of light transmission may be reduced. Preferably, the light transmission center angle may be adjusted within the range of 30 to 180°, which may be set by a system administrator.

The virtual progress line setting part 539 serves to set two virtual progress lines 82 symmetrical with respect to the virtual connection line 81 as shown in FIG. 6 . Each virtual travel line 82 starts at the center point of the highlight area 40 , and the two virtual travel lines 82 are spaced apart from each other with a light transmission center angle. That is, the virtual connecting line 81 and the virtual traveling line 82 each have an angle of θ/2, and the angle formed by the two virtual traveling lines 82 may be the light transmission central angle θ.

The virtual call setting part 540 sets up a plurality of virtual arcs 83 (arc) connecting two virtual running lines 82 to each other. In this case, a plurality of virtual arcs 83 are set, and they are spaced apart from each other by a brightness interval set on the system. In this case, the brightness interval is set based on the distance between the highlight area 40 and the shadow area 70, preferably based on the distance between the center point of the highlight area 40 and the center point of the shadow area 70. . Preferably, the narrower the distance is, the narrower the brightness interval is, and the greater the distance is, the wider the brightness interval is. Therefore, preferably, the virtual arc 83 in the composite image screen may be set in the range of 3 to 5. Therefore, each virtual arc 83 can be said to be a part of the circumference of concentric circles spaced apart from each other by the brightness interval.

The gradation setting part 541 defines a gradation area, which is an area set on the screen of the composite image as the virtual progress line 82 and the virtual arc 83 are set, and divides the virtual gradation area accordingly. carry out

When the virtual progress line 82 and the virtual arc 83 are drawn on the screen, an area, that is, an area, is set by the virtual arc 83 and the virtual progress line 82 . This can be said to represent the gradual spread of light from the highlight region 40, which can be said to the region of the light source, to the shadow region 70. In the present invention, the virtual progress line 82 and the virtual arc 83 The area divided by is referred to as a gradation area.

A plurality of gradation areas may be set, preferably 3 to 5 depending on the number of virtual arcs 83 . The closer the gradation area to the highlight area 40 is, the narrower the area is, and the closer to the shadow area 70 . The more the gradation area is, the wider the area is divided.

When the setting of the gradation area is completed in this way, the additional adjustment part 536 includes object brightness, background brightness, brightness of highlight area 40, brightness of shadow area 50, as well as point area 60 and shadow area 70 ), the brightness of the point area 60 and the shadow area 70 is adjusted by reflecting the gradation area to which each belongs.

This reflects the fact that the size of the object region 20 can vary widely in the composite image, and when the gradation areas A1 to A4 are formed as shown in the figure, it is confirmed that the point region 60 belongs to A2. It can be confirmed that the shadow area 70 belongs to the A4 area.

Therefore, when adjusting the brightness of the point area 60, the closer the gradation area to which the point area 60 belongs to the highlight area 40, that is, the closer to A1, the closer the brightness of the point area 60 is to the brightness of the highlight area 40. It is brightened at a similar level, and the closer it is to A4 (the farther it is from the highlight area), the darker it is processed than the highlight area 40 .

Also in adjusting the brightness of the shadow area 70, as the gradation area to which the shadow area 70 belongs is closer to the shadow area 50, that is, closer to A4, the brightness of the shadow area 70 is the same as the brightness of the shadow area 50. It is darkened at a similar level, and the closer to A1 (the farther from the shadow area 50), the brighter the shadow area 50 is.

In addition, more preferably, in the brightness control of the point area 60 and the brightness control of the shadow area 70, a point value and a shadow value may be calculated respectively, in which case the point value is based on Equation 1 below. , the shadow value may be calculated based on Equation 2 below.

Equation 1,

Equation 2,

는 포인트 수치,

는 섀도우 수치,

는 기준 포인트 수치,

는 기준 섀도우 수치,

는 포인트 영역(60)이 속한 그라데이션 에어리어에 부여된 그라데이션 가중치,

는 섀도우 영역이 속한 그라데이션 에어리어에 부여된 그라데이션 가중치,

는 배경 밝기,

는 객체 밝기,

는 하이라이트 영역의 밝기,

는 그림자 영역의 밝기)(here,

is the number of points,

is the shadow figure,

is the reference point number,

is the reference shadow figure,

is the gradation weight given to the gradation area to which the point area 60 belongs,

is the gradient weight given to the gradient area to which the shadow area belongs,

is the background brightness,

is the object brightness,

is the brightness of the highlight area,

is the brightness of the shadow area)

Here, the background brightness, the object brightness, the brightness of the highlight area 40, and the brightness value of the shadow area 50 are all numerical values. Values used to represent quantities are available. It has a value ranging from 0 to 255.

Since the reference point value and the reference shadow value are also values corresponding to brightness, they may be set within a range of 0 to 255. In this case, the reference point value and the reference shadow value may be set by a system administrator. Preferably, the reference point value may have a value of 200 to 255, and the reference shadow value may have a value of 0 to 100.

In the case of the gradation weight, it is characterized in that it is given to each gradation area, and in the case shown in the drawing, a gradation weight of a different value is given to each of the gradation areas A1 to A4. In this case, the gradation weight can be adjusted within the range of 0.8 to 1.2. Preferably, the gradation area closer to the highlight area 40 has a larger gradation weight, and the gradation area closer to the shadow area 50 has a smaller gradation weight.

In addition, these point numbers and shadow values have characteristics of values that appear non-linearly rather than on a linear graph. Since the shape is common, a hyperbolic showing the graph of the corresponding shape was taken to calculate the point value and shadow value reflecting the natural dimming effect.

In the case of point values, the brightness of the highlight area 40 becomes the most important point, and at the same time, the difference between the background brightness and the object brightness, and the brightness difference between the highlight area 40 and the shadow area 50 can be compared with each other, and In this case, the brightness of the shadow area 50 becomes the most important point, and at the same time, the difference between the background brightness and the object brightness and the brightness difference between the highlight area 40 and the shadow area 50 can be compared with each other.

If a simple numerical example is given, the reference point value is 200, the reference shadow value is 130, the gradation weight given to the gradation area to which the point area 60 belongs is 1, and the gradation weight to which the shadow area 70 belongs. Assuming that the given gradation weight is 0.8, the background brightness is 150, the object brightness is 130, the brightness of the highlight area 40 is 220, and the brightness of the shadow area 50 is 120,

can be calculated as

Therefore, based on the numerical comparison value as described above, the shadow value and point value are calculated, and the differential correction for the brightness of the point area 60 and the shadow area 70 is possible depending on the height of the shadow value and the point value. The higher the numerical value, the higher the brightness of the point area 60 is, and the higher the shadow value, the lower the brightness of the shadow area 70 can be adjusted, so that detailed contrast control is possible.

As described so far, the configuration and operation of the voice dubbing-based presenter video editing system according to the present invention are expressed in the above description and drawings, but this is merely an example, and the spirit of the present invention is not limited to the above description and drawings. It goes without saying that various changes and modifications can be made without departing from the technical spirit of the present invention.

1: Moderator 2: Central control server
10: face area 11: mouth area
20: object area 30: background area
40: highlight area 50: shadow area
60: point area 70: shadow area
81: virtual connection line 82: virtual progress line
83: virtual call 100: image database
200: area detection module 210: background detection unit
300: object extraction module 310: emotion classification unit
320: object extraction unit 400: voice management module
410: voice input unit 411: additional information input part
420: pronunciation analysis unit 500: synthesis module
510: boundary correction unit 511: skin pixel detection part
512: boundary correction part 513: smudging part
520: background synthesizing unit 530: image additional correction unit
531: brightness grasp part 532: brightness control part
533: highlight setting part 534: shadow setting part
535: shade setting part 536: additional adjustment part
537: Virtual connection line creation part 538: Center angle setting part
539: virtual progress line setting part 540: virtual call setting part
541: gradation setting part 600: output module
700: environment identification module 710: external environment identification unit
720: background classification unit

Claims (9)

As an audio dubbing-based moderator video editing system,
an image database for receiving a plurality of images including the face of the host, setting and storing a representative image, and receiving and storing a background image including any one of indoor and outdoor places;
A background for detecting a face region and a mouth region from the face region in a plurality of the images stored in the image database, and separating an object region including the moderator and a background region excluding the object region in the plurality of images a region detection module including a detection unit;
an object extraction module for extracting and storing a mouth-shaped object for each pronunciation of a voice from the mouth region;
a voice management module comprising: a voice input unit for receiving an information voice including guide information; and a pronunciation analyzer for analyzing pronunciation of the information voice;
a synthesis module comprising a background synthesizing unit for synthesizing the mouth-shaped object to the mouth region of the representative image according to the pronunciation of the information voice to generate a synthesized image, and synthesizing the background image to the background region of the representative image;
Including; an output module for outputting the information voice by dubbing the synthesized image,
The synthesis module is
a brightness detecting part for respectively detecting background brightness and object brightness, which are average brightnesses of the background area and the object area in the composite image, and a brightness adjusting part for adjusting object brightness according to the difference between the background brightness and the object brightness, and the background; A highlight setting part for determining the brightness and position of the highlight area by dividing the peripheral part into a plurality of areas and setting a highlight area that is a high-brightness area above the divided background area; and A shadow setting part for determining a position and brightness of the shadow region by setting a shadow region, which is a low-brightness region, below the background region, and a position of the highlight region and the position of the shadow region within the object region A shadow setting part for setting the vicinity of a highlight area as a high-brightness point area and setting a shadow area with low brightness near the shadow area, and the object brightness, the background brightness, the brightness of the highlight area, and the brightness of the shadow area. an additional adjustment part for respectively adjusting the brightness of the point area and the shadow area; a virtual connection line generation part for creating a virtual connection line connecting the center point of the highlight area and the center point of the shadow area; A central angle setting part for setting the light transmission central angle accordingly, and a virtual traveling line setting part for setting two virtual traveling lines that start at the center point of the highlight area and spread out with the light transmission central angle as symmetrical with respect to the virtual connection line; , a virtual call setting part that connects the two virtual progress lines to each other and sets up a plurality of virtual calls spaced apart from each other by a brightness interval set based on the distance between the highlight area and the shadow area; An image additional correction unit including a gradation setting part for setting a plurality of gradation areas according to a line and the virtual arc,
The additional adjustment part,
Adjusting the brightness of the point area and the shadow area according to the object brightness, the background brightness, the brightness of the highlight area, the brightness of the shadow area, and the gradation area to which the point area and the shadow area belong, respectively , a moderator video editing system. The method of claim 1,
The voice input unit,
Including an additional information input part for receiving the emotional information of the guide information,
The object extraction module,
an emotion classification unit for classifying the plurality of mouth-shaped objects according to emotions; and
An object extraction unit for extracting the mouth shape object matching the emotion information of the guide information as an emotion matching object,
The synthesis module is
The presenter video editing system, characterized in that the synthesized image is generated by synthesizing the emotion matching object with the mouth region of the representative image according to the pronunciation of the information voice. The method of claim 1,
The synthesis module is
a skin pixel detection part for detecting a skin pixel at a boundary between the mouth region and the face region belonging to the composite image;
a boundary correction part for correcting the saturation and brightness of the boundary region;
and a boundary correction unit including a smudging part for performing smudging processing on the boundary region. The method of claim 1,
The system is
An environment identification module comprising: an external environment identification unit configured to determine the weather information and time information of the location based on the output time of the output module; and a background classification unit configured to classify the background image according to weather information and time information; and ,
The background synthesizing unit,
The presenter image editing system, characterized in that the background image matching the weather information and time information of the place is synthesized in the background area. The method of claim 1,
The additional adjustment part,
Calculate a point value based on Equation 1 below, and adjust the brightness of the point area according to the calculated point value,
A moderator image editing system, characterized in that a shadow value is calculated based on Equation 2 below, and the brightness of the shadow area is adjusted according to the calculated shadow value.
Equation 1,

Equation 2,

(here,

is the number of points,

is the shadow figure,

is the reference point number,

is the reference shadow figure,

is the gradient weight given to the gradient area to which the point area belongs,

is the gradient weight given to the gradient area to which the shadow area belongs,

is the background brightness,

is the object brightness,

is the brightness of the highlight area,

is the brightness of the shadow area)
delete delete delete delete

Images