KR102273267B1 - Method and apparatus for processing sound - Google Patents

Abstract

A sound processing method and apparatus are disclosed.
According to an embodiment of the present invention, there is provided a method of processing sound included in content, the method comprising: extracting one or more deep meta data from an image included in the content; selecting object metadata, which is deep metadata related to the target object generating the sound, from among the extracted deep metadata; and processing the sound based on the object metadata.

Description

Sound processing method and apparatus {METHOD AND APPARATUS FOR PROCESSING SOUND}

The present invention relates to a method and an apparatus for processing sound, and more specifically, upscaling a sound corresponding to a corresponding image based on deep metadata extracted from an image or applying various effects to the corresponding sound to give a three-dimensional effect to the sound. and a sound processing method and apparatus for providing a sense of reality.

The content described in this section merely provides background information on the present invention and does not constitute the prior art.

As communication environments and hardware technologies develop, users' demands for high-quality content including high-quality images and high-quality sound are increasing.

In order to satisfy these needs, a method of converting Full HD, 8-bit non-HDR (high dynamic range) images to 4k UHD, 10-bit HDR images through deep learning-based image upscaling technology, and deep learning-based audio upscaling technology Various methods are being developed, such as a method of converting a 128kbps MP3 sound source into a 320kbps sound source.

However, there are no methods for providing a three-dimensional effect to the user by converting the sound included in the content according to the movement of an object in the image, or for more realistically expressing the sound by reflecting the environmental characteristics in which the sound is generated.

An embodiment of the present invention accurately grasps the spatial change of sound or the characteristics of the environment in the image in which the sound occurs based on the deep metadata of the image, and provides three-dimensional and realistic sound based on this. A main object is to provide a sound processing method and apparatus that can meet the needs.

According to an embodiment of the present invention, there is provided a method for processing sound included in content, the method comprising: extracting one or more deep meta data from an image included in the content; selecting object metadata, which is deep metadata related to the target object generating the sound, from among the extracted deep metadata; and processing the sound based on the object metadata.

According to another embodiment of the present invention, there is provided an apparatus for processing sound included in contents, comprising: an extractor configured to extract one or more deep meta data from an image included in the contents; a selection unit for selecting object metadata, which is deep metadata related to the target object generating the sound, from among the extracted deep metadata; and a processing unit that processes the sound based on the object metadata.

Since the present invention is configured to reflect the movement of a target object corresponding to the sound source or the environmental characteristics in which the sound is generated, a more three-dimensional and realistic sound can be provided, thereby providing high-quality content.

In addition, since the present invention is configured to perform sound processing through a mobile edge computing server, it is possible to solve the problem of delay in content transmission, overhead problem of the user terminal, and the problem of price increase of the user terminal at once.

In addition, the present invention can reflect a user's intentions in the selection of deep metadata, selection of a target object, and whether or not environmental characteristics are reflected, so that it is possible to provide content that meets the preferences and needs of individual users.

1 is a diagram illustrating a sound processing apparatus and related components according to an embodiment of the present invention.
2 is a block diagram schematically illustrating a sound processing apparatus according to an embodiment of the present invention.
3 is a flowchart for explaining an embodiment of the present invention for a sound processing method.
4 is a diagram for explaining various examples of the present invention for deep metadata.
5 is a flowchart illustrating an embodiment of the present invention for processing sound based on the movement of a target object.
6 is a flowchart for explaining an embodiment of the present invention in which sound is processed by reflecting the environmental characteristics in which the sound is generated.
7 is a flowchart for explaining an embodiment of the present invention in which sound upscaling based on the movement of a target object and applying an effect reflecting environmental characteristics are organically performed.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the component from other components, and the essence, order, or order of the component is not limited by the term. Throughout the specification, when a part 'includes' or 'includes' a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

1 is a diagram illustrating a sound processing apparatus (hereinafter referred to as a 'processing apparatus') 100 and related components according to an embodiment of the present invention. Hereinafter, the processing apparatus 100 and related components according to the present invention will be described with reference to FIG. 1 .

The sound processing method of the present invention may be implemented in one or more of a contents server 110 , a mobile edge computing (MEC) server 120 , and a user equipment (UE) 130 . . In other words, the processing device 100 of the present invention may be implemented in the form of one or more of the content server 110 , the mobile edge computing server 120 , and the user terminal 130 .

The content server 110 illustrated in FIG. 1 is a configuration that manages and provides various types of contents, and may be referred to by various names such as a contents management system. The content server 110 may correspond to a server owned, managed, or operated by a content provider.

Since the mobile edge computing server 120 and/or the user terminal 130 receives various types of content from the content server 110 , the mobile edge computing server 120 and the user terminal 130 provide a content server from the standpoint of the mobile edge computing server 120 and the user terminal 130 . 110 corresponds to an external data source.

The mobile edge computing server 120 is located between the content server 110 and the user terminal 130 based on the direction in which the content is provided, stores and manages the content transmitted from the content server 110, and This corresponds to a configuration in which content corresponding to the request is provided to the user terminal 130 . The mobile edge computing server 120 may correspond to a server provided by a communication service provider.

Mobile edge computing corresponds to a concept developed to solve problems such as network performance degradation, overload, and delay due to an explosive increase in network traffic. In mobile edge computing, the mobile edge computing server 120 is located close to the network edge (network edge), and data over a certain rate is processed through the server 120 . In other words, mobile edge computing can be seen as an extension of cloud 　 computing and services to the network 　 edge level.

The user terminal 130 is a device owned or managed by a content user, and as shown in FIG. 1 , a mobile terminal, a laptop computer, a desktop computer, etc. may be included in the user terminal 130 . In addition, although not shown in FIG. 1 , the user terminal 130 may include a TV, a vehicle-mounted display device, an augmented reality (AR)/virtual reality (VR) device, and the like.

When the processing apparatus 100 of the present invention is implemented in the form of the content server 110 , the sound processing method proposed in the present invention may be performed in the content server 110 . In this case, the processed sound may be directly transmitted from the content server 110 to the user terminal 130 .

When the processing apparatus 100 of the present invention is implemented in the form of the user terminal 130 , the sound processing method proposed in the present invention may be performed in the user terminal 130 . In this case, the content server 110 may perform only a limited role of transmitting the content corresponding to the user's request to the user terminal 130 without performing sound processing. The user terminal 130 may process the sound of the transmitted content (unprocessed content) by itself and provide it to the user.

When the processing device 100 of the present invention is implemented in the form of the mobile edge computing server 120 , the sound processing method proposed in the present invention may be performed in the mobile edge computing server 120 .

When the present invention is implemented in the mobile edge computing server 120, providing high-quality content through communication between the mobile edge computing server 120 and the user terminal 130 located in a relatively close position (edge end of the network) Since this can be done, it is possible to solve the problem of transmission delay occurring in the conventional cloud computing method.

In addition, since data processing (sound processing) is performed in the mobile edge computing server 120 and the user terminal 130 only plays a role of expressing the processed data, the overhead of the user terminal 130 can be reduced. Accordingly, high-quality sound (sound to which the sound processing method of the present invention is applied) can be implemented even in the user terminal 130 configured with relatively low-spec hardware.

Hereinafter, the present invention will be described with reference to an embodiment in which the processing device 100 is implemented in the form of a mobile edge computing server 120 .

The mobile edge computing server 120, that is, the processing device 100 receives various types of contents from the contents server 110 through a core network, and extracts deep metadata from images included in the contents. Then, it corresponds to a device that processes sound (corresponding to the corresponding image) included in the contents based on the extracted metadata.

The content provided from the content server 110 to the processing device 100 of the present invention includes complex content in which an image and a sound corresponding to the image are combined, such as media content, movie content, game content, drama content, educational content, etc. can

The sound composing the content in a form combined with the image includes the voice of the character appearing in the content, the sound generated by the movement of the character, and the sound generated by the collision between the objects expressed in the image of the corresponding content. etc. may be included.

In addition, sounds that are not generated by characters or objects, such as sounds generated by rain or lightning, but caused by external environmental factors expressed as images, etc. may be included in the sound corresponding to the processing target.

The sound processing proposed by the present invention includes upscaling and deep metadata that applies a three-dimensional effect by applying different or the same weight to each channel of the sound based on the deep metadata extracted from the image and adjusting the volume of each channel. A method of applying various effects to the sound may be included.

The sound processing proposed by the present invention can be understood as channel expansion or dimension upscaling in terms of applying a three-dimensional effect or realism that did not exist in the original sound (unprocessed sound).

The processing apparatus 100 of the present invention may provide high-quality content to the user by transmitting the content including the processed sound to the user terminal 130 through a base station or the like.

2 is a block diagram schematically showing a processing apparatus 100 according to an embodiment of the present invention, and FIG. 3 is a flowchart for explaining an embodiment of the present invention for a sound processing method.

As shown in FIG. 2 , the processing device 100 includes an I/O interface unit 210 , an extraction unit 220 , a selection unit 230 , a processing unit 240 , and a metadata storage unit 250 . can be configured.

First, 'content including an image and a sound corresponding thereto' is obtained through the I/O interface unit 210 (S310). Acquisition of content through the I/O interface unit 210 is implemented through transmission from the content server 110, or stores the content transmitted from the content server 110 in advance in a memory and accesses the memory. It may be implemented through a method or the like.

Metadata refers to data to describe the properties of data, and deep metadata refers to metadata extracted from useful features (metadata) included in content through artificial intelligence (AI), that is, machine learning or deep learning technology. do.

In general, deep metadata extracted from an image may include a person's face, emotion, movement, sound source, situation, dialogue, caption, surrounding environment, etc. expressed from the image. Since these various types of deep metadata indicate useful properties of the video, they can be utilized to enhance content experience and user convenience.

Representative technologies that utilize deep metadata to enhance content experience and improve user convenience include shot identification technology, intro/ending auto detection technology, alternative poster technology, and metadata composition technology.

The shot identification technology corresponds to a technology that distinguishes the boundary at which the camera shot changes based on the similarity between frames included in the image. In this technology, the similarity between frames is determined in consideration of the characteristics of the image, such as the genre of the content (movie, drama, entertainment, etc.) and the situation recognized from the frame (night/day, movement speed, fade in/out, etc.).

The intro/ending auto detection technology corresponds to a technology that automatically detects an intro/ending section of content. This technology is specialized in detecting the intro/ending section using shot identification technology, and it is used for skipping the intro/ending part for the convenience of binge watching, which continuously watches several episodes constituting one drama. can be

The alternative poster technology corresponds to a technology for extracting image candidates that can represent the content from the corresponding content. This technology may be utilized to generate a summary image of the corresponding content by using several image candidates.

The metadata composition technique corresponds to a technique for creating a metadata set by combining multiple metadata having the same or similar meaning. This technology can be utilized to extract and provide frames including the same or similar scenes within the content. Here, a frame including the same or similar scene may correspond to a frame desired by the user to view, and the user's request may be input through voice recognition. Therefore, this technology can be combined with natural language understanding technology.

In addition to the above representative technologies, deep metadata can be utilized for automatic recommendation of cast images for each content, selection by scene of a person or sound source, extraction/search of deep metadata within content, and section search for specific content.

The extractor 220 extracts one or more deep metadata from the acquired image (S320). An example of a method in which the extraction unit 220 extracts deep metadata will be described as follows.

Baseline data that can distinguish/recognize/recognize people, situations, places, etc. from a specific image is converted into a DB. The extractor 220 may be pre-learned using DB-ized baseline data. On the other hand, the image for each frame of the image (video signal) included in the content is converted into DB.

The extractor 220 may classify/recognize/recognize a person, a situation, a place, etc. by comparing the baseline DB and the DB image DB for each frame and applying a co-relation operation. Also, the extraction unit 220 may finish the process of extracting the deep metadata by extracting 'recognized person, situation, place, etc.' from a specific frame. In this process, in order to extract deep metadata from the entire content, a moving window technique may be applied.

The extractor 220 may be configured to extract the deep metadata in units of a preset time period for the entire image or to extract the deep metadata at one or more preset points in time.

The time period unit and time point at which the deep metadata is extracted may be variably set according to the necessity of sound processing. For example, the extraction unit 220 may be configured to extract deep metadata only from images having a need for sound processing such as a character's voice, character movement, collision between objects, etc. among all images included in the content. can

The selection unit 230 selects deep metadata (object metadata) related to the target object from among the extracted deep metadata ( S330 ). Here, the target object means a sound source that generates a sound included in the content, and the target object may include various elements that can generate sound, such as characters, animation characters, game characters, and things appearing in the corresponding content. .

The processing unit 240 processes the sound on the basis of deep metadata (object metadata) related to the target object or as indicated by the object metadata (S340).

When the sound processing is completed, the I/O interface unit 210 transmits the content including the processed sound to the user terminal 130 to provide the corresponding content (high-quality content) to the user (S350).

According to an embodiment, the sound processing method proposed by the present invention may be implemented on the premise that a request for content provision is transmitted from a user, or may be implemented irrespective of whether a request for content provision is transmitted from the user.

The former means a case in which the sound processing method is applied to the sound of the content corresponding to the request after the content provision request is received, and then the sound-processed content is transmitted to the user terminal 130 . The latter means a case in which the content is transmitted to the user terminal 130 only when a request for providing the content is received after applying the sound processing method to the obtained content in advance without receiving the request for providing the content. .

According to embodiments, the sound processing method proposed by the present invention may be selectively implemented according to the necessity of sound processing.

When the necessity to give a three-dimensional effect to the sound generated from the target object is recognized from the image, such as whether the character's voice is generated, whether the character's movement is present, or whether the character is expressed in the image, the need for sound processing is determined to exist. can

In addition, it can be determined that the need for sound processing exists even when the necessity to give an effect corresponding to the sound is recognized from the image, such as whether there is a collision between objects, whether a place is narrow, a change of weather, a change of season, etc. have.

The processing apparatus 100 of the present invention may determine the necessity of sound processing by itself, and selectively apply or implement the sound processing method proposed in the present invention according to the determination result.

According to an embodiment, the sound processing method proposed by the present invention may be selectively implemented according to a user's intention for sound processing. That is, on the premise that the user's intention to execute sound processing is received through the user terminal 130 , the processing apparatus 100 of the present invention may apply or implement the sound processing method.

4 is a diagram for explaining various examples of the present invention for deep metadata.

As described above, the deep metadata extracted from the image may include a person's face, emotion, movement, sound source, situation, dialogue, subtitles, surrounding environment, etc. expressed from the image.

As shown in FIG. 4 , the deep metadata extracted through the extraction unit 220 includes object metadata that is deep metadata related to a target object that generates sound, and background metadata that is deep metadata related to background music of an image. etc. may be included.

The object metadata includes motion metadata, which is deep metadata related to the motion of the target object, context metadata, which is deep metadata related to the context of the image in which the sound is generated, and deep metadata about whether the target object exists in the image. recognition metadata, etc. may be included.

Recognition metadata is deep metadata indicating whether the target object is visually recognized in the image. For example, if the target object is a game character and the game character is not recognized in the image, the recognition metadata may correspond to Off. can

The motion metadata may include metadata indicating whether a character (a person, a game character, an animation character, etc.) appearing in the content moves, the direction of the motion, and the degree of motion (the magnitude of the motion).

As shown in FIG. 4 , the motion metadata can be divided for each character appearing in the content (target object 1, target object 2, etc.), and the movement direction indicated by the motion metadata is a specific point (reference) in the image or screen. Point) based on Up/Down, Left/Right, and Front/Back.

That is, the motion metadata may include metadata indicating Up/Down, Left/Right, and Front/Back, respectively. In addition, the degree of motion indicated by the motion metadata may be expressed as a numerical value for each of Up/Down, Left/Right, and Front/Back.

The situation of the image indicated by the context metadata is an environmental factor that may cause deformation or change in the inherent sound generated from the target object itself, and may be understood as the surrounding environment, surrounding conditions, and the like.

As shown in Fig. 4, context metadata includes deep metadata related to the place where the sound occurs, deep metadata related to the time the sound is generated, and deep metadata related to the weather or season in which the sound occurs (sunny, cloudy). , rain, etc.), deep metadata related to whether the target objects collide or not may be included. In addition, the deep metadata related to the place may include metadata indicating whether indoors/outdoors and metadata indicating the size of the corresponding place.

After the deep metadata including such various sub-meta data is extracted from the image, it may be linked to the corresponding image and stored in the metadata storage unit 250 .

5 is a flowchart illustrating an embodiment of the present invention for processing sound based on the movement of a target object. Hereinafter, an embodiment of the present invention in which a three-dimensional effect is applied to a sound by upscaling the sound based on motion metadata will be described with reference to FIG. 5 .

First, as shown in FIG. 5 , a process ( S510 ) in which content including an image and a sound corresponding thereto is transmitted from the content server 110 to the processing device 100 and extracting one or more deep metadata from the image A process S520 may be performed.

The selection unit 230 selects motion metadata among the extracted deep metadata ( S540 ). According to an embodiment, the selection unit 230 may be configured to selectively select motion metadata for a specific character intended by the user.

To this end, before the process of selecting the motion metadata ( S540 ), a process of receiving a selection signal indicating any one of the characters appearing in the content from the user terminal 130 ( S530 ) may be performed. The selection signal corresponds to a signal or data input by the user through the user terminal 130 .

As such, if the present invention is configured to selectively select motion metadata for a specific character intended by the user, the present invention can implement sound processing (upscaling) for a character that meets the user's intention. Accordingly, the present invention can satisfy the various needs of individual users, thereby realizing a personalized service.

When the selection of the motion metadata is completed, the processing unit 240 upscales the sound generated from the corresponding character based on the selected motion metadata (according to the movement of the character indicated by the motion metadata) (S550) .

For example, if the character moves by +2 in the upper right direction, respectively, Up/Down, Left/Right, and Front/Back of the movement metadata are +2/-2, -2/+2, 0/0. can represent Accordingly, the processing unit 240 applies a weight proportional to +2 to each of the channels corresponding to the right direction and the upper direction, and applies a weight proportional to -2 to each of the channels corresponding to the left direction and the lower direction. can perform sound upscaling.

As another example, when the character moves by +3 in the direction (Front) closer to the user viewing the image, the motion metadata may indicate 0/0, 0/0, and +3/-3. Accordingly, the processing unit 240 may perform upscaling by applying a weight proportional to +3 to the channel corresponding to the front direction and applying a weight proportional to -3 to the channel corresponding to the back direction. .

In the above-described embodiment in which processing is performed based on movement metadata, it is assumed that the movement of the character occurs from the reference point (0, 0) for convenience of explanation and understanding. Accordingly, in the above example, +2, -2, +3, -3, etc. mean the relative magnitude of the movement based on the original position (previous position) of the character.

On the other hand, considering that an image is composed of a plurality of pictures (frames), the continuity of the motion direction and the degree of motion between the motion metadata extracted from the picture of a specific viewpoint and the motion metadata extracted from the picture of the previous viewpoint is maintained. there is a need to secure

Accordingly, the motion metadata extracted at a specific point in time may be expressed as a vector value based on the motion direction and motion information of the motion metadata extracted at a previous point in time. That is, the motion metadata of a specific viewpoint may be expressed as a relative value based on the motion metadata of a previous viewpoint.

When the sound processing is completed, the I/O interface unit 210 transmits the content including the processed (upscaled) sound to the user terminal 130 to provide high-quality content to the user (S560).

According to an embodiment, the process of extracting the deep metadata from the image ( S520 ) may be performed with a time difference from the process of selecting the motion metadata ( S540 ) and the process of upscaling the sound ( S550 ).

For example, the processing device 100 extracts deep metadata from the entire image received from the content server 110 and stores it in the deep metadata storage unit 250 , and then stores it in the image from the user terminal 130 . A process of selecting motion metadata ( S540 ) and a process of upscaling the sound ( S550 ) may be performed on the condition that provision of the data is required.

According to an embodiment, the process of receiving the selection signal ( S530 ) may be performed before the process of extracting the deep metadata ( S520 ). For example, the processing apparatus 100 extracts deep metadata on condition that a selection signal is received from the user terminal 130 ( S520 ), selects movement metadata of a character corresponding to the selection signal ( S540 ), and performs sound upscaling. (S550) may be performed. According to another embodiment, the process of receiving the selection signal ( S530 ) may be performed before the process of receiving the content from the content server 110 ( S510 ).

According to an embodiment, the above-described method (sound upscaling based on motion metadata) may be selectively turned on or off according to a user's intention. If the user turns off the entire sound processing function or turns on only the effect application function, which will be described later, the upscaling method for giving a three-dimensional effect may not be implemented. On the other hand, when the user turns on the entire sound processing function or turns off only the effect application function, the upscaling method may be implemented.

6 is a flowchart for explaining an embodiment of the present invention in which sound is processed by reflecting the environmental characteristics in which the sound is generated. Hereinafter, a method of applying various effects to sound based on context metadata will be described with reference to FIG. 6 .

First, a process of obtaining content including an image and a sound corresponding thereto ( S610 ) and a process of extracting one or more deep metadata from the image ( S620 ) may be performed in the same manner as described above.

The selection unit 230 selects context metadata, which is deep metadata related to an image situation, from among the extracted deep metadata (S640). According to an embodiment, the selection unit 230 may be configured to selectively select context metadata for a specific situation intended by the user using a selection signal received ( S630 ) from the user terminal 130 , through which The present invention can implement a personalization service.

When the selection of the context metadata is completed, the processing unit 240 applies various effects corresponding to the situation indicated by the selected context metadata to the sound (S650).

For example, when the context metadata indicates 'the narrowness of the place where the sound is generated', the processing unit 240 applies a reverb effect to the sound to obtain a reverberation effect (reality) according to the narrowness of the place. or a sense of space).

The context metadata may indicate the narrowness of the place as On or Off, or numerically indicate the degree of disgust or the size of the place. When the context metadata numerically represents the degree of narrowness of a place, the processing unit 240 may apply a reverb effect proportionally to the corresponding numerical value.

As another example, when the context metadata indicates 'collision of a target object', the processing unit 240 applies a high-pass filter to the sound to emphasize a high-frequency band of the sound, so that the sound generated due to the collision of the target object. can be expressed more realistically. The context metadata may numerically represent the size of the collision of the target object. In this case, the processing unit 240 may apply the sound emphasis effect proportionally to the corresponding numerical value.

When the sound processing is completed, the I/O interface unit 210 transmits the content including the processed sound to the user terminal 130 to provide high-quality content to the user (S660).

Similarly to the embodiment described with reference to FIG. 5 , in the embodiment described with reference to FIG. 6 , the process of extracting deep metadata from the image ( S620 ) is a process of selecting context metadata ( S640 ) and a process of applying an effect ( S650 ). ) and with a time difference.

In addition, the process of receiving the selection signal ( S630 ) may be performed before the process of extracting the deep metadata ( S620 ) or before the process of receiving the content from the content server 110 ( S610 ).

In addition, the above-described method (applying an effect to a sound based on context metadata) may be selectively turned on or off according to a user's intention. When the user turns off the entire sound processing function or turns on only the upscaling function described above, the effect application method may not be implemented. On the contrary, when the user turns on the entire sound processing function or turns off only the upscaling function, an effect application method may be implemented.

7 is a flowchart for explaining an embodiment of the present invention in which sound upscaling based on the movement of a target object and applying an effect reflecting environmental characteristics are organically performed.

5 and 6, a method of applying a three-dimensional effect to a sound and a method of applying an effect to a sound have been separately described. Hereinafter, the embodiment described with reference to FIG. 7 corresponds to an embodiment in which a method of applying a three-dimensional effect and a method of applying an effect are simultaneously implemented for the same sound.

First, a process ( S710 ) of obtaining content including an image and a sound corresponding thereto and a process ( S720 ) of extracting one or more deep metadata from the image may be performed in the same manner as described above.

The selection unit 230 selects object metadata from the extracted deep metadata (S730), and the processing unit 240 performs sound upscaling based on the object metadata (S740).

Meanwhile, the selection unit 230 selects context metadata from the extracted deep metadata ( S750 ), and the processing unit 240 applies an effect corresponding to a situation indicated by the context metadata to the upscaled sound ( S750 ). S760).

When sound processing (upscaling and effect application) is completed, the I/O interface unit 210 transmits the content including the processed sound to the user terminal 130 to provide high-quality content to the user (S770).

Similarly to the embodiment described with reference to FIGS. 5 and 6 , in the embodiment described with reference to FIG. 7 , the process of extracting deep metadata from an image ( S720 ) includes the process of selecting metadata ( S730 and S750 ) and sound processing. It may be performed with a time difference from the processes (S740 and S760). In addition, the above-described method (simultaneous performance of upscaling and effect application) may be selectively turned on or off according to a user's intention.

Although it is described that each process is sequentially executed in FIGS. 3, 5, 6 and 7, this is merely illustrative of the technical idea of an embodiment of the present invention. In other words, those of ordinary skill in the art to which an embodiment of the present invention belongs change the order described in FIGS. 3, 5, 6 and 7 without departing from the essential characteristics of an embodiment of the present invention. 3, 5, 6, and 7 are not limited to a time-series order, since various modifications and variations may be applied by executing or executing one or more of the processes in parallel.

Meanwhile, the processes illustrated in FIGS. 3, 5, 6 and 7 may be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system is stored. That is, the computer-readable recording medium includes a storage medium such as a magnetic storage medium (eg, ROM, floppy disk, hard disk, etc.) and an optically readable medium (eg, CD-ROM, DVD, etc.). In addition, the computer-readable recording medium is distributed in a network-connected computer system so that the computer-readable code can be stored and executed in a distributed manner.

The above description is merely illustrative of the technical idea of this embodiment, and various modifications and variations will be possible by those skilled in the art to which this embodiment belongs without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of the present embodiment should be interpreted by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present embodiment.

Claims (9)

A method of processing sound included in content, comprising:
extracting one or more deep meta data from the image included in the content;
selecting object metadata among the extracted deep metadata; and
processing the sound by upscaling the sound or applying an effect corresponding to a situation in the image to the sound based on the object metadata,
The object metadata is
Motion metadata that is deep metadata related to the motion of the target object generating the sound in the image, context metadata that is deep metadata related to the context in the image, and whether the target object exists in the image Among the recognition metadata that is deep metadata, it includes at least one,
The processing step is
and upscaling the sound by applying a weight to each channel of the sound according to the movement of the target object in the image indicated by the movement metadata. delete According to claim 1,
The target object is
It is a character appearing in the content,
The character is
Sound processing method, characterized in that the character indicated by the selection signal input from the user among one or more characters appearing in the content. According to claim 1,
The processing step is
The sound processing method of claim 1, wherein the sound is processed by applying an effect corresponding to the situation in the image indicated by the context metadata to the sound. A computer-readable recording medium recording a program for executing the method of any one of claims 1, 3 and 4 in a computer. An apparatus for processing sound included in content, comprising:
an extractor for extracting one or more deep meta data from the image included in the content;
a selection unit for selecting object metadata among the extracted deep metadata; and
a processing unit for processing the sound by upscaling the sound or applying an effect corresponding to a situation in the image to the sound based on the object metadata,
The object metadata is
Motion metadata that is deep metadata related to the motion in the image of the target object that generates the sound, context metadata that is deep metadata related to the context in the image, and whether the target object exists in the image Among the recognition metadata that is deep metadata, it includes at least one,
The processing unit,
and upscaling the sound by applying a weight to each channel of the sound according to the movement of the target object in the image indicated by the movement metadata. delete 7. The method of claim 6,
The target object is
It is a character appearing in the content,
The character is
Sound processing apparatus, characterized in that the character indicated by the selection signal input from the user among one or more characters appearing in the content. 7. The method of claim 6,
The processing unit,
The sound processing apparatus of claim 1, wherein the sound is processed by applying an effect corresponding to the situation in the image indicated by the context metadata to the sound.

Images