KR20220036210A - Device and method for enhancing the sound quality of video - Google Patents

Abstract

A device and method for improving the sound quality of an image are provided. A method for a device to improve the sound quality of an image includes acquiring an image; Obtaining sound and image from the acquired image; Obtaining a sound source image representing at least one sound source from the acquired image; Obtaining at least one unit sound data corresponding to the at least one sound source from the acquired sound; Applying a preset sound-image matching model to match the at least one sound source image and the at least one unit sound data; tracking the movement of the at least one sound source from the sound source image; and individually adjusting the loudness of the unit sound data according to the movement of the tracked sound source.

Description

Device and method for improving the sound quality of video {DEVICE AND METHOD FOR ENHANCING THE SOUND QUALITY OF VIDEO}

The present disclosure relates to a device and method for improving the sound quality of an image, and more specifically, to a device and method for improving the overall sound quality of an image by separating sound by sound source and individually adjusting the volume of the separated unit sound data. It's about.

Videography is the act of capturing the world around us. All modern mobile devices equipped with cameras have video recording capabilities. As mobile devices such as smart phones become widely available, the number of individuals filming and watching videos is increasing. Although the quality of video recording through mobile devices has improved over time, most of the focus has been on improving the quality of recorded visual images or improving the visual user experience. In contrast, there is little discussion of improving the quality of sound.

Additionally, with the spread of mobile devices, rather than everyone watching the same video on TV at home, individual viewers can watch different videos while moving on public transportation, in the office, or in the bathroom. It is often watched on mobile devices. When watching a video using a personal mobile device, a headset or earphone is generally used to avoid disturbing others and to focus on the video. Headsets and earphones support stereo sound with different sounds played on the left and right sides. Therefore, even in the case of sound recorded as mono audio through a single microphone, it is necessary to convert it to stereo format or other multi-channel format to improve sound quality.

In general mobile devices, in order to improve the sound quality of the video, in addition to the built-in microphone, a separate microphone such as a shotgun microphone or lapel microphone is used, or after filming, the video is transferred to a device such as a computer to compress the video. This is done through separate manual post-processing operations such as noise removal. Separate professional microphone equipment is expensive, and it is inconvenient to have to bring it with you every time you shoot. A separate post-processing process to improve sound quality requires professional knowledge of video editing programs and programs, and it is difficult to edit video directly on mobile devices such as smartphones with small screens. Therefore, it is not easy for general users who want to shoot and distribute video with a smart phone to improve the sound quality of the video.

Accordingly, the sound quality of images captured through cameras and microphones included in mobile devices such as smart phones is automatically improved within the mobile device, without requiring separate audio equipment or requiring separate post-processing operations. Skills that can do it are required.

In one embodiment of the present disclosure, a sound source image representing at least one sound source is obtained from a video image, the sound of the video is separated into unit sound data depending on whether it occurs from the same sound source, and the sound source image and A device that can adjust the number of channels of output sound regardless of the number of channels of input sound and improve the sound quality of output video by matching unit sound data and adjusting the loudness of each unit sound data, and A method can be provided.

In addition, in one embodiment of the present disclosure, an image is captured through an input unit included in a mobile device, and a processor included in the mobile device automatically performs sound processing of the captured image, thereby providing separate sound to improve sound quality. Devices and methods may be provided in which no equipment is required and the user does not manually perform post-processing operations.

A method of improving the sound quality of an image by a device, disclosed as a technical means for achieving the above-described technical problem, includes the steps of acquiring an image; Obtaining sound and image from the acquired image; Obtaining a sound source image representing at least one sound source from the acquired image; Obtaining at least one unit sound data corresponding to the at least one sound source from the acquired sound; Applying a preset sound-image matching model to match the at least one sound source image and the at least one unit sound data; tracking the movement of the at least one sound source from the sound source image; and individually adjusting the loudness of the unit sound data according to the movement of the tracked sound source. The sound-image matching model may include matching information between the image of a specific sound source and the sound generated by the specific sound source.

A device disclosed as a technical means for achieving the above-described technical problem includes an input unit for acquiring an image; An output unit that outputs an output image; Memory that stores a program containing one or more instructions; and at least one processor executing one or more instructions stored in the memory. The at least one processor controls the input unit to obtain an image, acquire sound and an image from the acquired image, and generate at least one sound source from the acquired image. Acquire a sound source image representing the sound source, obtain at least one unit sound data corresponding to the at least one sound source from the acquired sound, and apply a preset sound-image matching model to obtain the at least one sound source image and the It is possible to match at least one unit sound data, track the movement of the at least one sound source from the sound source image, and individually adjust the loudness of the unit sound data according to the movement of the tracked sound source. The sound-image matching model may include matching information between the image of a specific sound source and the sound generated by the specific sound source.

As a technical means for achieving the above-described technical problem, a computer-readable recording medium may store a program for executing at least one of the embodiments of the disclosed method on a computer.

1 is a schematic diagram of a method by which a device improves the sound quality of an image according to an embodiment of the present disclosure.
Figure 2 is a block diagram of a device according to an embodiment of the present disclosure.
Figure 3 is a flowchart of a method for improving the sound quality of an image according to an embodiment of the present disclosure.
Figure 4 is a flowchart of a method for improving the sound quality of an image according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating an operation of a device acquiring additional sound through an auxiliary input unit according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating an operation of a device acquiring at least one sound source image from an image according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating an operation in which a device acquires at least one unit sound data from sound according to an embodiment of the present disclosure.
FIG. 8 is a diagram illustrating an operation in which a device according to an embodiment of the present disclosure separates sound according to a sound source image and matches the separated unit sound data to each sound source image.
FIG. 9 is a diagram illustrating a specific example of an operation in which a device individually adjusts the volume of unit sound data according to the movement of a tracked sound source according to an embodiment of the present disclosure.
FIG. 10A is a diagram illustrating an example of a device acquiring output sound having multi-channels according to an embodiment of the present disclosure.
FIG. 10B is a diagram illustrating an example of a device obtaining output sound having multi-channels according to an embodiment of the present disclosure.
FIG. 10C is a diagram illustrating an example in which a device according to an embodiment of the present disclosure obtains output sound having multiple channels.
FIG. 11 is a diagram illustrating an example in which a device individually adjusts the volume of unit sound data according to an embodiment of the present disclosure.
FIG. 12 is a diagram illustrating an example in which a device individually adjusts the volume of unit sound data according to the movement of a tracked sound source according to an embodiment of the present disclosure.
FIG. 13 is a diagram illustrating an example in which a device according to an embodiment of the present disclosure adjusts the volume of unit sound data according to the movement of a tracked sound source and obtains output sound with multi-channel from the adjusted unit sound data.
FIG. 14 is a diagram illustrating an example in which a device according to an embodiment of the present disclosure obtains additional sound through an auxiliary input unit and obtains output sound having multiple channels.

Below, with reference to the attached drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present disclosure in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

The terms used in the embodiments of the present disclosure have selected general terms that are currently widely used as much as possible while considering the function of the present disclosure, but this may vary depending on the intention or precedent of a person working in the art, the emergence of new technology, etc. . In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant embodiment. Therefore, the terms used in this specification should be defined based on the meaning of the term and the overall content of the present disclosure, rather than simply the name of the term.

Singular expressions may include plural expressions, unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary skill in the technical field described herein.

Throughout the present disclosure, when a part “includes” a certain element, this means that it may further include other elements rather than excluding other elements, unless specifically stated to the contrary. In addition, terms such as “... unit” and “... module” used in this specification refer to a unit that processes at least one function or operation, which is implemented as hardware or software or as a combination of hardware and software. It can be implemented.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

The expression “configured to” used in this specification may mean, for example, “suitable for,” “having the capacity to,” depending on the situation. ”, “designed to”, “adapted to”, “made to”, or “capable of”. The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware. Instead, in some situations, the expression “system configured to” may mean that the system is “able to” work with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored in memory. It may refer to a general-purpose processor (e.g., CPU or application processor) that can perform the corresponding operations.

In this disclosure, ‘video’ refers to audio-visual material including auditory sounds and visual screens. The visual composition of a video can be described as ‘image’, ‘visual data’ or ‘picture’, and the auditory composition of a video can be described as ‘audio’ or ‘sound’. )', 'acoustic', or 'sound data'.

In this disclosure, ‘sound quality’ refers to the quality of sound. Sound quality may vary depending on various acoustic factors. For example, the amount of noise may be the standard for sound quality, and the sound quality may vary depending on the flatness of the frequency and volume of the sound.

In this disclosure, ‘sound source’ refers to the source of sound from which sound is generated. For example, people, animals, musical instruments, or any object can be a sound source if sound is generated therefrom.

In this disclosure, ‘sound corresponding to a specific sound source’ refers to a sound generated from that specific sound source. For example, a sound corresponding to a specific person may refer to the voice made by that person, and a sound corresponding to a specific animal may refer to the animal's cry.

In this disclosure, ‘screen area’ refers to the area on the screen where the visual screen of the image is displayed. For example, the screen range may be an area defined by the border of an image captured from a video at a specific point in time.

In this disclosure, ‘monaural, monophonic: mono’ audio refers to audio consisting of one channel. Mono audio is recording through one microphone, and may correspond to the sound heard through one speaker. Even if the sound is recorded or played through multiple speakers, it can be mono audio if the sound is connected to only one channel. In mono audio, the same sound is played from all connected speakers.

In this disclosure, ‘multi-channel’ audio refers to audio consisting of two or more channels. For example, when stereo audio, a type of multi-channel audio, is listened to through one speaker, signals from two channels are synthesized and played as one, but it is played through two speakers (e.g., headset, earphones). When playing through an earphone, etc., different sounds are played from both speakers, and the sound can be reproduced with a sense of space and richness compared to mono audio.

Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

Figure 1 is a schematic diagram of a method by which a device 1000 improves the sound quality of an image according to an embodiment of the present disclosure.

Referring to FIG. 1, an image input to the device 1000 may include an image 110 and sound 150. In one embodiment, image 110 may be recorded through an input device such as a camera, and sound 150 may be recorded through an input device such as a microphone. The device 1000 may obtain sound source images SS1, SS2, and SS3 separated into individual sound sources from the recorded image 110. In one embodiment, the sound source image may be divided into a speaker (person) (SS1, SS2) and a background (eg, including a person other than the speaker, a desk, a chair, paper, and background images) (SS3).

The device 1000 may obtain a unit sound data set 160 that divides the sound 150 into sounds generated from each sound source. In one embodiment, the device 1000 may classify the separated unit sound data set 160 into human voices (UA1, UA2) or background sounds (noise) (UA3).

The device 1000 may apply a preset sound-image matching model to match the classified unit sound data set 160 to the separated sound source images SS1, SS2, and SS3. For example, unit sound data (UA1, UA2) classified as a human voice can be matched to sound source images (SS1, SS2) determined to be human, and other sounds (UA3) can be matched to the background image (SS3). It can be matched. When two or more people are talking, the device 1000 separates the separated unit sound data (UA1, UA2) through the 'person's face and voice corresponding to the face' information included in the sound-image matching model. It can be matched to the sound source images (SS1, SS2).

The device 1000 can individually adjust the volume of each of the separated unit sound data (UA1, UA2, and UA3). For example, the volume of unit sound data UA3 corresponding to noise can be reduced, and the volume of unit sound data (UA1, UA2) corresponding to the speaker's conversation can be adjusted to the level corresponding to the input signal or to a preset level. there is.

The device 1000 may resynthesize the output sound 170 from the adjusted sound data of each unit. In one embodiment, the output sound 170 may be synthesized in a stereo format, a type of multi-channel sound, for output to an output device such as headphones. For example, the output sound 170 may include a first channel 171 output to the left speaker (LC) and a second channel 173 output to the right speaker (RC).

The device 1000 may determine a rendering channel for the unit sound data UA1 and UA2 corresponding to each sound source image SS1 and SS2, depending on the relative positions within the screen range of the sound source images SS1 and SS2. For example, since the first sound source image (SS1) is located on the left within the screen range, the first unit sound data (UA1) corresponding to the first sound source image (SS1) is output to the left speaker (LC) through the first channel ( 171). Additionally, the second unit sound data UA2 corresponding to the second sound source image SS2 located on the right side within the screen range may be rendered on the second channel 173 output to the right speaker RC. In one embodiment, the device 1000 can adjust the number of channels of the output sound 170 regardless of the number of channels of the input sound 150, and improve the sound quality of the image to enable output of a spatial and rich sound. there is.

Figure 2 is a block diagram of a device 1000 according to an embodiment of the present disclosure.

Referring to FIG. 2 , the device 1000 may include an input unit 1100, a processor 1300, a memory 1500, an output unit 1700, and a motion sensor 1900. Not all of the components shown in FIG. 2 are essential components of the device 1000. The device 1000 may be implemented with more components than those shown in FIG. 2 , or the device may be implemented with fewer components than the components shown in FIG. 2 .

The input unit 1100 can acquire images from the outside. In one embodiment, the input unit 1100 may include a recording unit that acquires a visual image and a recording unit that acquires an auditory sound. For example, the recording unit may include a camera, and the recording unit may include a microphone (microphone). In one embodiment, the input unit 1100 may be a single component that is not physically separated into a recording unit and a recording unit.

The output unit 1700 may output an output image to the outside. The output unit 1700 may include a display 1710 and an audio output unit 1720.

The display 1710 can display and output a visual image to the outside. In one embodiment, display 1710 may include a panel. The display 1710 may be, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, 3 It may consist of at least one of a 3D display and an electrophoretic display.

The audio output unit 1720 can reproduce and output auditory sound to the outside. In one embodiment, the audio output unit 1720 may include a speaker. The audio output unit 1720 may include, for example, a single speaker, two or more speakers, a mono speaker, a stereo speaker, a surround speaker, a headset, and earphones ( It may consist of at least one of earphones.

In one embodiment, the display 1710 and the audio output unit 1720 of the output unit 1700 may be a single structure that is not physically separated.

The memory 1500 may store a program to be executed by a processor 1300, which will be described later, to control the operation of the device 1000. The memory 1500 may store a program including at least one instruction for controlling the operation of the device 1000. The memory 1500 may store instructions and program codes that the processor 1300 can read. In one embodiment, the processor 1300 may be implemented to execute instructions or codes of a program stored in the memory 1500. The memory 1500 may store data input to or output from the device 1000.

The memory 1500 may include, for example, flash memory, hard disk, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, It may include at least one type of storage medium among a magnetic disk and an optical disk.

Programs stored in the memory 1500 can be classified into a plurality of modules according to their functions. For example, the memory 1500 includes an acoustic image separation module 1510, a sound source image acquisition module 1520, a unit acoustic data acquisition module 1530, a matching module 1540, a sound source motion tracking module 1550, and a volume May include an adjustment module 1560. Additionally, the memory 1500 may include an audio-image matching model 1570, a deep neural network (DNN) 1580, and a database 1590.

The processor 1300 may control the overall operation of the device 1000. For example, the processor 1300 executes programs stored in the memory 1500, thereby generating an input unit 1100, an output unit 1700 including a display 1710 and an audio output unit 1720, and a motion sensor 1900. and memory 1500, etc. can be controlled overall.

The processor 1300 may be comprised of hardware components that perform arithmetic, logic, input/output operations, and signal processing. The processor 1300 may include, for example, a Central Processing Unit, a microprocessor, a Graphics Processing Unit, Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), and Digital Signal Processors (DSPDs). It may consist of at least one of Signal Processing Devices (PLDs), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but is not limited thereto.

The processor 1300 may acquire an image through the input unit 1100 by executing at least one command stored in the memory 1500. The video may include images, which are visual data, and sounds, which are auditory data.

The processor 1300 can acquire sound and image from the acquired image by executing at least one instruction constituting the sound image separation module 1510 among the programs stored in the memory 1500. there is. In one embodiment, an image composed of a single mono file can be separated into a sound file, which is auditory data, and an image file, which is visual data.

The processor 1300 executes at least one command constituting the sound source image acquisition module 1520 among the programs stored in the memory 1500, thereby generating a sound source image representing at least one sound source from the acquired image. It can be obtained. An image can be composed of one continuous, undivided screen. In this continuous screen, each object such as a person, animal, or object can be separated. Each separated object can be a sound source that generates sound. In one embodiment, at least one sound source image may be obtained from an image using a deep neural network (DNN) or a database storing image files.

The processor 1300 executes at least one command constituting the unit sound data acquisition module 1530 among the programs stored in the memory 1500, thereby executing at least one command determined depending on whether it occurs in the same sound source from the acquired sound. Unit acoustic data can be acquired. The unit sound data acquisition module 1530 may separate input sound consisting of one channel into a plurality of channels of unit sound data generated from different sound sources. In one embodiment, the unit sound data acquisition module 1530 may separate sounds into unit sound data using a deep neural network (DNN) or a database in which audio information is accumulated. Information on the separated unit sound data is transferred to and stored in the database 1570 stored in the memory 1500, and the database 1570 can be updated. In one embodiment, a model for separating acoustic data may be preset and stored in a database.

The processor 1300 applies a preset sound-image matching model 1570 by executing at least one command constituting the matching module 1540 among the programs stored in the memory 1500 to generate at least one sound source image. and at least one unit sound data may be matched, respectively.

The sound-image matching model 1570 may include matching information between the image of a specific sound source and the sound generated by the specific sound source. The matching information between sound and image includes sound information according to the information of the sound source image (for example, a barking sound corresponding to an image of a dog, a rustling sound corresponding to a tree image, or the sound of a specific person when two or more people are talking). Features of sound according to a specific image, such as voice information according to a face image) and information of a sound source image according to a specific sound (for example, the mouth shape of the image when a specific sound occurs) may be included.

In one embodiment, the sound-image matching model 1570 may be set using a deep neural network (DNN) or a database in which image files and audio information are accumulated.

In one embodiment, individual sound source images separated from the image and individual unit sound data separated from the sound may be matched based on the correspondence information between the sound and the image included in the sound-image matching model 1570, respectively. The information of each sound source image and unit sound data matched according to the preset sound-image matching model 1570 is transferred back to the sound-image matching model 1570 stored in the memory 1500 and stored, and sound-image matching is performed. The model 1570 can be updated, or it can be transferred to the database 1590 and stored, and the database 1590 can be updated.

The processor 1300 may track the movement of at least one sound source from the sound source image by executing at least one command constituting the sound source motion tracking module 1550 among the programs stored in the memory 1500. In videos containing moving images, the state or location of a specific sound source image may change over time. In one embodiment, the sound source motion tracking module 1550 analyzes the image (screen) to analyze the moving direction, speed, change in shape of the sound source image, etc. of a specific sound source and obtains a movement profile for the specific sound source. can do. In one embodiment, the obtained motion profile can be used to individually adjust the volume of each unit acoustic data in a subsequent step.

The processor 1300 executes at least one command constituting the volume control module 1560 among the programs stored in the memory 1500, thereby individually adjusting the loudness of the unit sound data according to the movement of the tracked sound source. It can be adjusted. In one embodiment, the volume of unit sound data may be adjusted so that the output sound maintains a constant volume throughout the entire image. For example, when recording a person speaking, as the sound source (the person speaking) moves during recording and becomes relatively distant from the device 1000, the volume of the input unit sound data decreases. In this case, based on the volume when the sound source is close to the device 1000, the volume can be adjusted so that the unit sound data has a constant volume throughout the entire image. In this way, based on the motion profile of the sound source previously acquired by the sound source motion tracking module 1550, the volume of unit sound data corresponding to the sound source can be individually adjusted.

In one embodiment, the processor 1300 individually adjusts the volume of unit sound data according to the type of sound source by executing at least one command constituting the volume control module 1560 among the programs stored in the memory 1500. It may be possible. For example, if specific unit sound data is classified as noise, the volume of the unit sound data can be adjusted to be low. In one embodiment, when the output of only a specific type of sound is required, only the specific type of sound to be output among the unit sound data is filtered, and the volume of other types of unit sound data may be adjusted to be low, such as noise. there is.

A deep neural network (DNN) 1580 stored in the memory 1500 is a type of artificial neural network and may have the characteristic of consisting of multiple hidden layers between the input layer and the output layer. DNN (deep neural network) 1580 can model complex nonlinear relationships like a general artificial neural network. For example, in a deep neural network structure for an object identification model, each object can be expressed as a hierarchical composition of basic image elements. At this time, additional layers can integrate the characteristics of gradually gathered lower layers. This feature of the DNN (deep neural network) 1580 allows complex data to be modeled with fewer units. The DNN (deep neural network) 1580 can be applied to the field of image recognition or voice recognition, and can be used for processing to separate images and match the separated images with each voice information, as in the present disclosure.

The database 1590 stored in the memory 1500 may be composed of a vast amount of data. In one embodiment, the database 1590 may include sound information corresponding to a specific sound source image and image information corresponding to a specific sound. In one embodiment, the database 1590 may be used to establish a sound-image matching model 1570 by obtaining matching information representing the correspondence between sounds and images. Additionally, the database 1590 can be used to match unit sound data and sound source images or to adjust the volume of each unit sound data.

The processor 1300 may obtain output sound from unit sound data whose volume is individually adjusted and obtain an output image from the output sound and image by executing at least one command stored in the memory 1500. In one embodiment, final output sound data may be obtained by re-synthesizing unit sound data whose volume has been individually adjusted. In one embodiment, the processor 1300 may classify and render unit sound data into two or more channels in order to obtain multi-channel output sound, such as a stereo format. For example, unit sound data corresponding to sound source images placed on the left side of the display screen may be rendered in a first channel, and unit sound data corresponding to sound source images placed on the right side of the display screen may be rendered in a second channel. there is. Afterwards, output sound that reproduces the first channel from the left speaker and the second channel from the right speaker can be obtained. In one embodiment, the output sound may be in a stereo format, as well as a surround format, Ambisonic format, or other multi-channel format.

In one embodiment, device 1000 may further include a motion sensor 1900. The motion sensor 1900 may include an accelerometer 1910, a gyroscope 1920, and a magnetometer 1930. The motion sensor 1900 can detect movement of the device 1000. If there is movement of the device 1000 itself, which includes the input unit 1100 for acquiring an image, it may be recognized that there is movement of the sound source image in the acquired image even if there is no actual movement of the object. In one embodiment, based on movement information of the device 1000 obtained from the motion sensor 1900, a motion profile of the sound source may be additionally obtained through relative changes on the screen of the sound source image. The acquired additional sound source motion profile can be used to adjust the volume of unit sound data matched to the corresponding sound source image.

Conventionally, in order to obtain an image containing high-quality sound, one had to use professional sound recording equipment or record with general sound equipment and then go through an image post-processing process. With the development of the Internet and social networks, the number of creators who personally film, edit, and distribute videos is increasing. Rather than using professional equipment, these individual creators often shoot videos using the camera and microphone included in mobile devices such as smartphones. There has been much improvement in video shooting using mobile devices in the processing area of images, which are visual data, but there has been no significant improvement in the area of processing sound, which is auditory data. Improving sound quality is important for watching more realistic videos.

The device 1000 according to an embodiment of the present disclosure converts the image acquired from the input unit 1100 included in the device 1000 into visual data by having the processor 1300 execute one or more instructions stored in the memory 1500. Separate the visual image and audio data into sound, obtain a sound source image representing at least one sound source from the image of the video, and separate the sound of the video into unit sound data depending on whether it occurs from the same sound source. , the sound quality of the output video can be improved by matching the sound source image and unit sound data and adjusting the loudness of each unit sound data.

Therefore, even when recording using the input unit 1100, such as a microphone, which is basically included in a mobile device such as a smart phone, the captured image is stored inside the device 1000 through the processor 1300 included in the device 1000. It can be post-processed immediately. In this case, separate audio equipment is not required to improve the sound quality of the captured video, and the mobile device automatically performs sound post-processing even if the user does not have professional video post-processing technology, thereby providing high-quality sound. Video can be obtained.

In addition, the device 1000 according to an embodiment of the present disclosure has the processor 1300 execute one or more instructions stored in the memory 1500, thereby rendering the separated individual unit sound data into two or more different channels. can do. Therefore, even when recorded as mono audio through a single microphone, the output image may have multi-channel stereo sound, surround sound, or ambisonic sound. In this way, the number of channels of output sound can be adjusted regardless of the number of channels of input sound, and high-quality sound for a more realistic image can be obtained.

Figure 3 is a flowchart of a method for improving the sound quality of an image according to an embodiment of the present disclosure.

In step S300, an image may be acquired. A video may refer to an audiovisual representation drawn on a two-dimensional plane. Video may refer to a moving video. In one embodiment, images can be acquired through an input unit including a microphone that acquires sound and a camera that acquires images.

In step S310, sound can be obtained from the image. For example, sound may include human voices, animal sounds, sounds generated from objects, noise, etc. In one embodiment, the sound may be a single channel of mono audio recorded from a single microphone, or it may be multi-channel audio recorded from a plurality of microphones.

In step S320, an image may be obtained from the video. For example, an image may be visual data recorded from a camera. In one embodiment, the image may include sound source images of various sound sources.

In step S330, at least one unit sound data corresponding to at least one sound source may be obtained from the sound. For example, at least one unit sound data determined depending on whether it occurs from the same sound source can be obtained. In one embodiment, sound consisting of a single channel of mono audio can be separated into a plurality of unit sound data generated from different sound sources. In one embodiment, images may be used when separating sound into a plurality of unit sound data.

In step S340, a sound source image representing at least one sound source may be obtained from the image. For example, from an image composed of continuous visual data, objects such as people, animals, objects, and backgrounds, each of which can be a sound source that generates sound, can be separated.

In step S350, at least one sound source image and at least one unit sound data can be matched, respectively, by applying a preset sound-image matching model. In one embodiment, the sound-image matching model may include matching information between the image of a specific sound source and the sound generated by the specific sound source. In one embodiment, the sound-image matching model may be preset through a deep neural network (DNN). Matching information between sound and image may include sound information according to sound source image information and sound source image information according to a specific sound.

In one embodiment, sound source images separated from the image and unit sound data separated from the sound may be matched one-to-one, many-to-one, or many-to-many based on a sound-image matching model, respectively. In one embodiment, when matching unit sound data and sound source image, movement of the sound source image and changes in the sound source image may be considered.

In step S360, the movement of at least one sound source can be tracked from the sound source image. Movement can be tracked for each sound source image. The motion profile of the sound source can be used to individually adjust the volume of each unit sound data in a subsequent step. In one embodiment, the movement of the sound source can be calculated and tracked through changes in the sound source image on the screen. In one embodiment, the movement of the sound source is determined by relative changes on the screen of the sound source image using device movement information obtained from a motion sensor including an accelerometer, gyroscope, and magnetometer. It can also be calculated and tracked through

In step S370, the loudness of unit sound data can be individually adjusted according to the movement of the tracked sound source. In one embodiment, the volume of each unit sound data separated from the sound can be reduced, increased, or adjusted to have a constant volume throughout the entire image. By individually adjusting the volume of each unit sound data, optimization and tuning of the overall sound is possible.

In step S380, output sound can be obtained from unit sound data whose volume is individually adjusted. In one embodiment, final output sound data may be obtained by re-synthesizing unit sound data whose volume has been individually adjusted. In one embodiment, unit sound data may be classified into two or more channels and rendered, and multi-channel output sound such as stereo format may be obtained. For example, the output sound can be re-synthesized by adjusting the volume of the unit sound data corresponding to noise to be low, or can be re-synthesized to have multiple channels for rich sound. Therefore, the final output sound can be based on the initial input sound. The sound quality may be improved compared to

In step S390, an output image can be obtained from the output sound and image. When compared to the input image, the output image has the same image (screen) but contains adjusted unit sound data, so the sound quality can be improved.

Figure 4 is a flowchart of a method for improving the sound quality of an image according to an embodiment of the present disclosure.

In step S400, an image including sound and an image can be acquired, and in steps S410 and S420, the sound and image can be separated from the image and obtained.

In step S430, a sound source image representing at least one sound source may be obtained from the image. For example, from an image composed of continuous visual data, objects such as people, animals, objects, and backgrounds, each of which can be a sound source that generates sound, can be separated.

In step S440, at least one sound source image and a portion of the sound may be matched by applying a preset sound-image matching model. In one embodiment, the sound-image matching model may include matching information between the image of a specific sound source and the sound generated by the specific sound source.

In step S450, at least one unit sound data corresponding to at least one sound source may be obtained from the sound and the separated sound source image. For example, at least one unit sound data determined depending on whether it occurs from the same sound source can be obtained. In one embodiment, a sound source image can be used when separating sound consisting of a single channel of mono audio into a plurality of unit sound data generated from different sound sources. For example, when separating sound into individual unit sound data, it is possible to determine in advance which sound source exists by referring to the previously separated sound source image, and to preferentially separate the sound from that sound source. For example, part of the sound matched to each sound source image can be separated into each unit sound data.

The operation of preferentially separating the sound source image from the image and separating unit sound data from the sound using the matched sound source image may be useful when there is a sound source that does not appear in the image of the video. For example, after the sounds corresponding to each separate sound source image are separated into respective unit sound data, unit sound data corresponding to a sound source that does not appear in the image of the video can be obtained from the remaining sound data.

In step S460, the movement of at least one sound source can be tracked from the sound source image of the sound source. The movement of the sound source can be tracked for each sound source image. The motion profile of the sound source can be used to individually adjust the volume of each unit sound data in a subsequent step.

Thereafter, similarly to the above-described embodiment, the loudness of the unit sound data can be individually adjusted according to the movement of the tracked sound source in step S470, and the volume is output from the individually adjusted unit sound data in step S480. Sound can be obtained. The sound quality of the output sound may be improved compared to the initially input sound.

FIG. 5 is a diagram illustrating an operation of the device 1000 acquiring additional sound through the auxiliary input unit 2100 according to an embodiment of the present disclosure.

The device 1000 according to an embodiment of the present disclosure may itself include an input unit including a microphone for acquiring sound and a camera for acquiring an image.

In one embodiment, the device 1000 may obtain additional sound through the auxiliary input unit 2100 external to the device 1000. For example, the auxiliary input unit 2100 may include an auxiliary microphone such as a lapel microphone. The sound acquired directly by the device 1000 and the sound acquired through the auxiliary input unit 2100 may be sounds generated from the same sound source (SS). In one embodiment, the sound directly acquired by the device 1000 and the sound obtained through the auxiliary input unit 2100 may be sound generated from the same sound source (SS). The sound obtained through 2100 is sound generated from different sound sources and may constitute multi-channel sound.

If the sound acquired directly by the device 1000 and the sound acquired through the auxiliary input unit 2100 are sounds generated from the same sound source (SS), the sound acquired directly from the device 1000 and the sound acquired through the auxiliary input unit 2100 A sound may only have differences in loudness or signal-to-noise ratio. In this case, the sound input through the auxiliary input unit 2100 may be transmitted to the device 1000 and used for post-processing of the image together with the sound acquired by the device 1000. For example, if the sound input through the auxiliary input unit 2100 has a better signal-to-noise ratio, the sound input through the auxiliary input unit 2100 may be used to remove acoustic noise from the image acquired by the device 1000. You can. The sound acquired through the auxiliary input unit 2100 can assist the sound acquired by the device 1000 itself, and does not completely replace the sound acquired by the device 1000.

If the sound acquired directly by the device 1000 and the sound acquired through the auxiliary input unit 2100 are sounds of different channels and constitute a multi-channel sound, each sound is post-processed together or independently to create a new mono channel. Output sound can be obtained in format or multi-channel format.

FIG. 6 is a diagram illustrating an operation of the device 1000 acquiring at least one sound source image from the image 610 according to an embodiment of the present disclosure.

The device 1000 may obtain a sound source image representing at least one sound source from the image 610.

Referring to FIG. 6, the acquired video image 610 may be composed of continuous visual data. From continuous visual data images, it is possible to separate objects such as people, animals, objects, and backgrounds, each of which can be a sound source that generates sound. In one embodiment, image analysis may be done through deep learning, a deep neural network (DNN) technology, or artificial intelligence, which allows for high accuracy and recognition of various objects.

Artificial Intelligence (AI) image recognition technology can classify images into various patterns and learn pattern-type data to determine what an image is when given a new image. In one embodiment, device 1000, through a deep neural network (DNN) or artificial intelligence (AI), selects images 610 from human images (H1, H2, H3, H4, H5, H6) and dog images (D1). , D2) can be separated. Separated human images (H1, H2, H3, H4, H5, H6) and dog images (D1, D2) can each be sound source images. In this way, the device 1000 can separate and obtain at least one sound source image from the image 610.

FIG. 7 is a diagram illustrating an operation of the device 1000 acquiring at least one unit sound data set 760 from the sound 750 according to an embodiment of the present disclosure.

Unit sound data 760 may be determined depending on whether it occurs from the same sound source. Sound has three elements: intensity, timbre, and pitch. These three elements correspond to the amplitude, waveform, and frequency of the sound wave, respectively. The larger the amplitude of the wave, the greater the intensity of the sound, and the higher the frequency of the wave, the higher the height of the sound. The timbre of a sound is determined by its waveform, and the reason why the sounds of a piano, a person, a violin, etc. are different even if the sound is the same, is because the waveforms of the sound are different.

Additionally, the envelope can be considered when distinguishing sounds. Envelope is the change in sound over time, and refers to the time for the sound to reach its peak, the time for the sound to stabilize, the time for the sound to last, and the time until the sound disappears. The envelope can vary depending on how the sound source produces the sound. In one embodiment, whether a sound originates from the same sound source may be determined based on the three elements and envelope of the sound.

In one embodiment, the operation of separating and acquiring the unit acoustic data set 760 from the sound 750 includes converting the sound 750 into at least one unit acoustic data 761 according to amplitude, frequency, phase, waveform, and spectrum. 762, 763, 764). For example, from the sound 750 in which the sounds of four instruments are synthesized, four sounds are separated into sounds by each instrument according to the three elements and envelope of the sound depending on amplitude, frequency, phase, waveform, spectrum, etc. Unit sound data (761, 762, 763, 764) can be obtained.

In one embodiment, when the amplitude, frequency, phase, waveform, and spectrum of two or more unit sound data are all the same, they can be separated into each unit sound data using the sound source image. For example, if an image includes split screens and the same person is speaking simultaneously on each split screen, the unit sound data corresponding to each split screen can be separated by referring to the shape of the person's mouth on each split screen. You can. For example, when there are two or more instruments of the same type, unit sound data for each instrument can be separated by referring to the hand shape of the person playing the instrument. In this way, when it is difficult to separate unit sound data for each sound source due to sound characteristics, image data can be additionally used.

FIG. 8 shows that the device 1000 according to an embodiment of the present disclosure separates the sound 850 according to the sound source images 821 and 822, and divides the separated unit sound data 861 and 862 into each sound source image 821. , 822) This is a diagram to explain the matching operation.

Referring to FIG. 8 , the image may include an image 810 and sound 850 including sound source images 821 and 822 separated into individual sound sources. The sound 850 may include a synthesis of sounds A1 and A2 generated from each sound source image 821 and 822. In one embodiment, the sound 850 can be separated into unit sound data 861 and 862 by applying a sound-image matching model that includes voice information corresponding to each sound source image 821 and 822. . The separated unit sound data 861 and 862 may be matched with each sound source image 821 and 822 according to voice information.

In one embodiment, the operation of matching the sound source image and unit sound data may additionally use information obtained from the sound source image. For example, when the same person is speaking at the same time on a split screen, unit sound data can be matched to the split screen by referring to the shape of the person's mouth on each split screen. For example, when there are two or more instruments of the same type, unit sound data for each instrument can be matched by referring to the hand shape of the person playing the instrument.

In one embodiment, when the output of only a specific sound is required, only the specific type of sound to be output among the unit sound data 861 and 862 is filtered and the volume of other types of unit sound data is adjusted to be low. It may be possible.

For example, referring to FIG. 8, when it is desired to output only the unit sound data 861 corresponding to the sound source image 821, the unit sound data 862 corresponding to the sound source image 822 may be muted. Additionally, when it is desired to output only the unit sound data 862 corresponding to the sound source image 822, the unit sound data 861 corresponding to the sound source image 821 can be muted.

FIG. 9 is a diagram illustrating a specific example of an operation (S370) in which the device 1000 individually adjusts the volume of unit sound data according to the movement of a tracked sound source according to an embodiment of the present disclosure.

In step S910, a volume curve of the entire execution time of each unit sound data can be obtained. For example, the level of the detected sound volume for each unit sound data can be calculated over time.

In step S920, a volume correction curve containing adjustment information to be performed for each unit sound data may be obtained. In one embodiment, the volume correction curve may include information about whether to reduce or increase the volume of unit sound data at a specific time within the entire running time of the video. For example, when it is desired to keep the sound volume constant within the entire running time of the video, the volume correction curve can be calculated as the difference between the volume curve and the preset output volume value.

In step S930, the volume of each unit sound data can be individually adjusted over time based on the volume correction curve.

10A, 10B, and 10C are diagrams illustrating an example in which the device 1000 obtains output sound having multi-channels according to an embodiment of the present disclosure.

Referring to FIG. 10A, in one embodiment, the device 1000 may capture an image including two sound sources SS101 and SS102.

Referring to Figure 10b, the recorded input sound is mono audio, and when played back without post-processing, the input sounds (IA101, IA102) generated from two sound sources are played in the left channel (LC) and right channel (RC), respectively. Can be played simultaneously. In this case, the two sound sources (SS101, SS102) can be recognized as being in the same place. As such, in mono audio having a single channel, the user cannot recognize the directions of the two sound sources (SS101 and SS102).

Referring to FIG. 10C, when the method for improving the sound quality of an image according to an embodiment of the present disclosure is applied to the recorded input sounds (IA101 and IA102), the device 1000 transmits the sounds to the respective sound sources (SS101 and SS102). Depending on this, it can be separated into unit sound data, and the separated unit sound data can be rendered into the left channel (LC) or right channel (RC) depending on the location on the screen of each sound source image. For example, the unit sound data corresponding to the sound source SS101 located on the left side of the screen output sound to the left channel (LC), and the unit sound data corresponding to the sound source SS102 located on the right side of the screen output sound to the right channel (RC). Can be rendered. Therefore, the output sound can be implemented as multi-channel audio with two channels (LC, RC).

FIG. 11 is a diagram illustrating an example in which the device 1000 individually adjusts the volume of unit sound data according to an embodiment of the present disclosure.

In one embodiment, a person carrying the device 1000 and taking pictures may directly become a sound source SS111 that generates sound. The person carrying the device 1000 and taking pictures may appear on the screen, but may not appear on the screen in other cases.

In the operation of separating sound into unit sound data, if the sound source image of the sound source SS111 exists on the screen, the sound-image matching model can be used. If the sound source SS111 does not appear on the screen and there is no sound source image, the unit sound data A2 corresponding to the other sound source SS112 that appears on the screen can be separated and the remaining sound data can be determined as unit sound data A1 corresponding to the sound source SS111. .

Referring to (a) of FIG. 11, in the input sound, unit sound data A1 generated by the sound source SS111 located close to the device 1000 is unit sound data generated by the sound source SS112 located far from the device 1000. The volume may be louder than A2.

Referring to (b) of FIG. 11, in order to improve the sound quality of the video, the device 1000 reduces the volume of unit sound data A1 and increases the volume of unit sound data A2 to adjust the volume of A1 and A2 to the same level. You can. If the volume of unit sound data A1 and A2 is adjusted to the same level, the overall sound volume of the image can be maintained constant, and thus the sound quality of the image can be improved.

FIG. 12 is a diagram illustrating an example in which the device 1000 individually adjusts the volume of unit sound data according to the movement of a tracked sound source according to an embodiment of the present disclosure.

In one embodiment, the subject (sound source) SS120 that the device 1000 is photographing may be moving while generating sound. For example, the subject may have an initial position (SS120i) and a final position (SS120f). In one embodiment, the subject may move in a direction away from the device 1000. At this time, the initial position (SS120i) of the subject may be relatively close to the device 1000, and the final position (SS120f) of the subject may be relatively far from the device 1000.

Referring to (a) of FIG. 12, the volume of the initial input sound (Ai) generated at the initial position (SS120i) of the subject is high, and the volume may become low as the sound source moves away from the device 1000. The volume of the final input sound (Af) generated at the final position of the subject (SS120f) may be relatively small.

Referring to (b) of FIG. 12, in one embodiment, in order to improve the sound quality of the image, the device 1000 reduces the volume of the initial input sound (Ai), increases the volume of the final input sound (Af), etc. , a volume correction curve containing volume adjustment information over time can be obtained. The device 1000 can adjust the volume of the sound using the obtained volume correction curve and keep the volume of the output sound at the same level throughout the entire running time of the video.

Figure 13 shows an example in which the device 1000 according to an embodiment of the present disclosure adjusts the volume of unit sound data according to the movement of the tracked sound source and obtains output sound with multi-channel from the adjusted unit sound data. It is a drawing.

In one embodiment, the subject (sound source) SS130 that the device 1000 is photographing may move relative to the device 1000 while generating sound. At the initial time (Ti), the subject may be located far to the right of the device 1000, and may move to the left closer to the device 1000 as the final time (Tf) approaches. At this time, the initial position (SS130i) of the subject may be relatively far from the device 1000, and the final position (SS130f) of the subject may be relatively close to the device 1000.

Referring to (a) of FIG. 13, at the initial time Ti, the volume of the initial input sound Ai generated from the initial position SS130i may be low. As the sound source (SS130) approaches the device 1000, the volume increases. Referring to (c) of FIG. 13, the volume of the final input sound (Af) generated from the final position (SS130f) at the final time (Tf) is It can be relatively large.

In one embodiment, the device 1000 includes information on adjusting the volume over time, such as increasing the volume of the initial input sound (Ai) and decreasing the volume of the final input sound (Af), in order to improve the sound quality of the image. A volume correction curve can be obtained. The device 1000 can adjust the volume of the sound using the obtained volume correction curve and keep the volume of the output sound at the same level throughout the entire running time of the video.

To obtain more realistic sound quality, the device 1000 may render output sound as multi-channel audio according to the location of the sound source. For example, near the initial time (Ti) when the sound source (SS130i) is located on the right side of the screen, the volume of the right channel (RCi) can be increased and rendered. Referring to (b) of FIG. 13, at the initial time Ti, the output sound can be adjusted so that the volume of the right channel (RCi) is high and the volume of the left channel (LCi) is low.

Referring to (d) of FIG. 13, near the final time (Tf) when the sound source (SS130f) is located on the left side of the screen, the right channel (LCf) is used so that the sound of the left channel (LCf) can be heard better than the sound of the right channel (RCf). RCf) can be rendered by reducing the volume. For example, at the final time Tf, the output sound may be adjusted so that the volume of the right channel (RCf) is low and the volume of the left channel (LCf) is high.

FIG. 14 is a diagram illustrating an example in which the device 1000 acquires additional sound through the auxiliary input unit 2200 and obtains output sound having multiple channels according to an embodiment of the present disclosure.

In one embodiment, the device 1000 may acquire sound including sound A1 obtained directly through the input unit and sound A2 acquired through the auxiliary input unit 2200 external to the device 1000. For example, the auxiliary input unit 2200 may be a wearable device including a microphone.

Referring to (a) of FIG. 14, when the sound source SS140 is located far from the device 1000, the sound A1 directly obtained from the input unit of the device 1000 has a low volume and a low signal-to-noise ratio. You can. Meanwhile, since the auxiliary input unit 2200 is always located close to the sound source SS140, the sound A2 obtained through the auxiliary input unit 2200 is loud and clear, and has a high signal-to-noise ratio.

Signal-to-Noise Ratio (SNR) is the ratio of the intensity of the signal and the intensity of noise. In one embodiment, in terms of signal-to-noise ratio, a signal may mean valid acoustic data. A higher signal-to-noise ratio means less noise.

In one embodiment, in order to improve the sound quality of the image, the device 1000 reduces sound noise using the sound A2 obtained from the auxiliary input unit 2200 and adjusts the volume of the output sound to a preset level. You can.

To obtain more realistic sound quality, the device 1000 may render output sound as multi-channel audio according to the location of the sound source SS140. For example, referring to FIG. 14, the sound source SS140 may be located on the right side of the screen. Referring to (b) of FIG. 14, in this case, the output sound can be rendered by adjusting the volume of the left channel (LC) to be low and the volume of the right channel (RC) to be high.

In one embodiment of the present disclosure, a sound source image representing at least one sound source is separated from a video image, and the sound of the video is separated into unit sound data depending on whether it occurs from the same sound source, so that the input sound Sound can be processed regardless of the number of channels. Additionally, by rendering the separated unit sound data into a single channel or multi-channel, the number of channels of the output sound can be adjusted regardless of the channel of the input sound. In one embodiment of the present disclosure, the sound quality of the output image can be improved by matching the separated sound source image and the unit sound data and adjusting the loudness of each unit sound data.

In addition, in one embodiment of the present disclosure, an image is captured through an input unit included in a mobile device, and a processor included in the mobile device automatically performs sound processing of the captured image, thereby providing a separate device to improve sound quality. No audio equipment is required, and the user may not manually perform post-processing operations.

An embodiment of the present disclosure may also be implemented in the form of a recording medium containing instructions executable by a computer, such as program modules executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Computer-readable media may also include computer storage media and communication media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Communication media typically may include computer readable instructions, data structures, or other data such as modulated data signals, program modules.

Additionally, computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage medium' only means that it is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is semi-permanently stored in a storage medium and temporary storage media. It does not distinguish between cases where it is stored as . For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or via an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smartphones) or online. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) is stored on a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be temporarily stored or created temporarily.

Additionally, in this specification, “unit” may be a hardware component such as a processor or circuit, and/or a software component executed by the hardware component such as a processor.

Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may consist of one or multiple processors. At this time, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-specific processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-specific processor such as an NPU. One or more processors control input data to be processed according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, when one or more processors are dedicated artificial intelligence processors, the artificial intelligence dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

Predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned using a large number of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform the desired characteristics (or purpose). It means burden. This learning may be performed on the device itself that performs the artificial intelligence according to the present disclosure, or may be performed through a separate server and/or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

An artificial intelligence model may be composed of multiple neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and neural network calculation is performed through calculation between the calculation result of the previous layer and the plurality of weights. Multiple weights of multiple neural network layers can be optimized by the learning results of the artificial intelligence model. For example, a plurality of weights may be updated so that loss or cost values obtained from the artificial intelligence model are reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, etc., but are not limited to the examples described above.

Artificial intelligence models can be created through learning. Here, being created through learning means that the basic artificial intelligence model is learned using a large number of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform the desired characteristics (or purpose). It means burden. An artificial intelligence model may be composed of multiple neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and neural network calculation is performed through calculation between the calculation result of the previous layer and the plurality of weights.

The foregoing description of the present disclosure is for illustrative purposes, and a person skilled in the art to which the present disclosure pertains will understand that the present disclosure can be easily modified into another specific form without changing its technical idea or essential features. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as single may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present disclosure. do.

Claims (21)

In a method for a device to improve the sound quality of an image,
Acquiring an image;
Obtaining sound and image from the acquired image;
Obtaining a sound source image representing at least one sound source from the acquired image;
Obtaining at least one unit sound data corresponding to the at least one sound source from the acquired sound;
Applying a preset sound-image matching model to match the at least one sound source image and the at least one unit sound data;
tracking the movement of the at least one sound source from the sound source image; and
individually adjusting the loudness of the unit sound data according to the movement of the tracked sound source;
Including,
The sound-image matching model includes matching information between an image of a specific sound source and a sound generated by the specific sound source. According to paragraph 1,
The step of acquiring the image is,
Including acquiring an image through an input unit included in the device,
The input unit includes a microphone for acquiring sound and a camera for acquiring an image. According to paragraph 1,
The step of acquiring the image is,
Including acquiring an image through an input unit included in the device and an auxiliary input unit outside the device,
The input unit includes a microphone for acquiring sound and a camera for acquiring an image,
The method wherein the auxiliary input unit includes an auxiliary microphone for acquiring additional sound. According to paragraph 1,
Obtaining at least one unit sound data corresponding to the at least one sound source from the acquired sound,
comprising separating the sound into at least one unit sound data according to amplitude, frequency, phase, waveform and spectrum,
When two or more unit sound data having the same amplitude, frequency, phase, waveform and spectrum exist, a method comprising separating the two or more unit sound data into individual unit sound data using the sound source image. . According to paragraph 1,
The step of applying the preset sound-image matching model to match the at least one sound source image and the at least one unit sound data, respectively,
A method comprising matching the at least one sound source image and the at least one unit sound data by additionally using information obtained from the sound source image. According to paragraph 1,
The step of tracking the movement of the at least one sound source from the sound source image includes tracking the movement of the sound source through a change in the state of the sound source image. According to paragraph 1,
The step of tracking the movement of the at least one sound source from the sound source image uses movement information of the device obtained from a motion sensor including an accelerometer, a gyroscope, and a magnetometer, A method comprising tracking the movement of a sound source through changes in the state of the sound source image. According to paragraph 1,
The step of individually adjusting the volume of the unit sound data according to the movement of the tracked sound source,
Obtaining a volume curve of the entire execution time of each unit sound data;
Obtaining a volume correction curve including adjustment information to be performed for each unit sound data; and
Method comprising individually adjusting the volume of each unit sound data based on the volume correction curve. According to paragraph 1,
Obtaining output sound from unit sound data whose volume is individually adjusted; and
Obtaining an output image from the output sound and the image;
A method further comprising: According to clause 9,
The step of obtaining output sound from unit sound data whose volume is individually adjusted,
A method comprising classifying and rendering the unit sound data into two or more channels and obtaining output sound having multi-channels. In a device that improves the sound quality of video,
An input unit that acquires an image;
An output unit that outputs an output image;
Memory that stores a program containing one or more instructions; and
At least one processor executing one or more instructions stored in the memory,
The at least one processor,
By controlling the input unit, an image is acquired,
Obtaining sound and images from the acquired image,
Obtaining a sound source image representing at least one sound source from the acquired image,
Obtaining at least one unit sound data corresponding to the at least one sound source from the acquired sound,
Applying a preset sound-image matching model to match the at least one sound source image and the at least one unit sound data, respectively,
Tracking the movement of the at least one sound source from the sound source image,
Individually adjusting the loudness of the unit sound data according to the movement of the tracked sound source,
The sound-image matching model includes matching information between the image of a specific sound source and the sound generated by the specific sound source. According to clause 11,
The input unit is a device including a microphone that acquires sound and a camera that acquires an image. According to clause 11,
The processor executes the one or more instructions,
A device that acquires additional sound through an auxiliary microphone external to the device. According to clause 11,
The processor executes the one or more instructions,
Separating the sound into at least one unit sound data according to amplitude, frequency, phase, waveform and spectrum,
When there are two or more unit sound data having the same amplitude, frequency, phase, waveform and spectrum, by separating the two or more unit sound data into individual unit sound data using the sound source image,
A device that acquires at least one unit sound data corresponding to the at least one sound source from the acquired sound. According to clause 11,
The processor executes the one or more instructions,
By additionally using information obtained from the sound source image to match the at least one sound source image and the at least one unit sound data,
A device that applies the preset sound-image matching model to match the at least one sound source image and the at least one unit sound data, respectively. According to clause 11,
The processor executes the one or more instructions,
A device that tracks the movement of the sound source through changes in the state of the sound source image. According to clause 11,
The processor executes the one or more instructions,
A device that tracks the movement of the sound source through changes in the state of the sound source image, using device movement information obtained from a motion sensor including an accelerometer, gyroscope, and magnetometer. According to clause 11,
The processor executes the one or more instructions,
Obtain a volume curve of the entire execution time of each unit sound data,
Obtaining a volume correction curve containing adjustment information to be performed for each unit sound data,
By individually adjusting the volume of each unit sound data based on the volume correction curve,
A device that individually adjusts the volume of the unit sound data according to the movement of the tracked sound source. According to clause 11,
The processor executes the one or more instructions,
Obtaining output sound from unit sound data whose volume is individually adjusted,
The device further comprising obtaining an output image from the output sound and the image. According to clause 19,
The processor executes the one or more instructions,
A device that classifies and renders the unit sound data into two or more channels and obtains output sound with multi-channel. A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 10 on a computer.

Images