KR102620852B1 - Method and apparatus for providing foley sound using artificial intelligence - Google Patents

Abstract

An artificial intelligence-based poly sound providing device and method are disclosed. An artificial intelligence-based Foley sound providing method according to an embodiment includes obtaining a target image including a subject performing a target action, recognizing the properties of at least one object including the subject from the target image, and generating information related to the target action. Determining context information, determining the duration and intensity of the target action by tracking the posture of the subject from the target image, responding to the target action of the subject based on the determined context information, duration and intensity. It includes generating a poly sound, and providing download data of the generated poly sound or download data in which the generated poly sound and the target image are merged.

Description

Artificial intelligence-based foley sound providing device and method {METHOD AND APPARATUS FOR PROVIDING FOLEY SOUND USING ARTIFICIAL INTELLIGENCE}

The disclosure below relates to an apparatus and method for generating and providing foley sound based on artificial intelligence.

Foley sound is a sound effect that is mixed into a video to improve the quality of sound during video work or to depict a realistic video with sound. It is created using various props and equipment. Mixing Foley sound is an essential task, but creating Foley sound and adding it to the video is done manually.

Meanwhile, the recent development of various neural network models has made more sophisticated image analysis possible. A neural network model is a computer system that implements human-level intelligence and is a model in which machines learn and make decisions on their own. The neural network model consists of machine learning (deep learning) technology that uses an algorithm that classifies/learns the characteristics of input data on its own, and element technologies that mimic the functions of the human brain such as cognition and judgment using machine learning algorithms. Element technologies include, for example, linguistic understanding technology that recognizes human language/characters, visual understanding technology that recognizes objects as if they were human eyes, reasoning/prediction technology that judges information and makes logical inferences and predictions, and human experience information. It may include at least one of a knowledge expression technology that processes knowledge data and a motion control technology that controls the autonomous driving of a vehicle and the movement of a robot.

The background technology described above is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and cannot necessarily be said to be known technology disclosed to the general public before this application.

The disclosure below is intended to improve the problem of having to manually and repeatedly perform all steps of poly sound mixing work, which is frequently performed during video work.

However, technical challenges are not limited to the above-mentioned technical challenges, and other technical challenges may exist.

An artificial intelligence-based Foley sound providing method according to an embodiment includes obtaining a target image including a subject performing a target action, recognizing properties of at least one object including the subject from the target image, and selecting the target. determining context information related to an action, determining a duration of the target action and an intensity of the target action by tracking the posture of the subject from the target image, the determined context information, the duration and the intensity Based on this, it may include generating a Foley sound corresponding to the target behavior of the subject, and providing download data of the generated Foley sound or download data in which the generated Foley sound and the target image are merged. You can.

The step of generating the poly sound includes searching a sound library for a sound corresponding to the determined context information, and when a sound corresponding to the context information is found, adjusting the length of the searched sound based on the duration. It may include steps.

The step of generating the poly sound includes, when a sound corresponding to the context information is not searched, inputting the context information, the duration, and the intensity into a learned neural network model, and output of the learned neural network model. It may further include generating a poly sound corresponding to the input context information, duration, and intensity.

The target action is a human step, the properties of the at least one object include the gender of the subject, the material of the ground on which the subject is walking, and the material of the shoe the subject is wearing, and the context information includes the It may be characterized in that it is determined in text form based on the gender of the recognized subject, the recognized material of the ground surface, and the recognized material of the shoe.

The target action is a person's footsteps, and the step of determining the duration of the target action and the intensity of the target action includes detecting the posture of the subject from the target image, and tracking the detected posture in the target image. It may include acquiring the movement of the subject, and determining the duration of the step and the intensity of the step based on the obtained movement.

Obtaining the movement of the subject may include, in response to detecting the posture of the upper body of the subject, acquiring the movement of the upper body by tracking the detected posture of the upper body, and based on the movement of the upper body It may include the step of acquiring movement of the subject's lower body.

A method according to an embodiment includes receiving an original image from a user device, and inputting the original image into a learned neural network model to classify at least one consecutive frame corresponding to the target action among the frames of the original image. Further comprising the step of acquiring the target image, the classified at least one consecutive frame may be acquired as the target image.

The properties of each of the at least one object are recognized through different neural network models learned to recognize the properties of each object from the image, and the context information is text information connecting the outputs of the different neural network models. You can do this.

A method according to an embodiment includes transmitting the generated Foley sound to a user device, receiving a sound editing command for the Foley sound from a user device that has received the Foley sound, and according to the sound editing command, It may further include the step of editing a poly sound, and the step of providing the download data may include providing download data of the edited poly sound or download data of a target video in which the edited poly sound is merged. there is.

The poly sound providing device according to one embodiment includes a processor and a memory storing instructions to be executed by the processor, and when the instructions are executed by the processor, the processor performs a target action. Obtain a target image including a subject, recognize the properties of at least one object including the subject from the target image, determine context information related to the target behavior, and determine the posture of the subject from the target image. Tracking to determine the duration of the target behavior and the intensity of the target behavior, based on the determined context information, the duration and the intensity, generating a poly sound corresponding to the target behavior of the subject, and generating the Download data of the created Foley sound or download data in which the generated Foley sound and the target image are merged may be provided.

According to the artificial intelligence-based Foley sound providing device and method according to an embodiment of the present disclosure, there is an effect of improving the problem of having to manually repeatedly perform all steps of the Foley sound mixing task frequently performed during video work.

1 is a diagram illustrating an example of a network environment according to an embodiment.
Figure 2 is a block diagram illustrating an example of a computer device according to an embodiment.
Figure 3 is a diagram illustrating an example of a general view of a Foley sound providing system according to an embodiment.
Figure 4 is a block diagram illustrating an example of a Foley sound providing device according to an embodiment.
Figure 5 is a flowchart of a method for providing poly sound according to an embodiment.
Figure 6 is a flowchart of a method for providing foley sound of footsteps to an image of a person's full body or lower body, according to an embodiment.
Figure 7 is a flowchart of a method of providing foley sound of footsteps to an image of a person's upper body in one embodiment.
FIG. 8 is a diagram illustrating an example of a user screen of a method for providing Foley sound according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 is a diagram illustrating an example of a network environment according to an embodiment. The network environment in FIG. 1 shows an example including a plurality of electronic devices 110, 120, and 130, a plurality of servers 150 and 160, and a network 140. Figure 1 is an example for explaining the invention, and the number of electronic devices or servers is not limited as in Figure 1. In addition, the network environment in FIG. 1 only explains one example of environments applicable to the present embodiments, and the environment applicable to the present embodiments is not limited to the network environment in FIG. 1.

The plurality of electronic devices 110, 120, and 130 may be fixed terminals implemented as computer devices or mobile terminals. Examples of the plurality of electronic devices 110, 120, 130 include smart phones, mobile phones, navigation devices, computers, laptops, digital broadcasting terminals, Personal Digital Assistants (PDAs), Portable Multimedia Players (PMPs), There are tablet PCs, etc. For example, in FIG. 1, the shape of a smartphone is shown as an example of the electronic device 110, but in embodiments of the present invention, the electronic device 110 is substantially connected to the server through the network 140 using a wireless or wired communication method. It may refer to one of a variety of physical computer devices capable of communicating with (150, 160).

The communication method is not limited, and may include not only a communication method utilizing a communication network that the network 140 may include (for example, a mobile communication network, wired Internet, wireless Internet, and a broadcast network), but also short-range wireless communication between devices. For example, the network 140 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , may include one or more arbitrary networks such as the Internet. Additionally, the network 140 may include any one or more of network topologies including a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, etc. Not limited.

Each of the servers 150 and 160 is a computer device or a plurality of computer devices that communicate with a plurality of electronic devices 110, 120, 130 and a network 140 to provide commands, codes, files, content, services, etc. It can be implemented as: For example, the server 150 provides a service (for example, a cloud service, a game service, a content provision service, a group call service (or It may be a system that provides (voice conference service), messaging service, mail service, social network service, map service, translation service, financial service, payment service, search service, etc.).

Figure 2 is a block diagram illustrating an example of a computer device according to an embodiment. Each of the plurality of electronic devices 110, 120, and 130 described above or each of the servers 150 and 160 may be implemented by the computer device 200 shown in FIG. 2.

As shown in FIG. 2, this computer device 200 may include a memory 210, at least one processor 220, a communication interface 230, and an input/output interface 240. The memory 210 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, non-perishable large-capacity recording devices such as ROM and disk drives may be included in the computer device 200 as a separate permanent storage device that is distinct from the memory 210. Additionally, an operating system and at least one program code may be stored in the memory 210. These software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 210 through the communication interface 230 rather than a computer-readable recording medium. For example, software components may be loaded into memory 210 of computer device 200 based on computer programs installed by files received over network 140.

The processor 220 may be configured to process instructions (or instructions) of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, processor 220 may be configured to execute received instructions according to program code stored in a recording device such as memory 210.

The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (eg, the storage devices described above) through the network 140. For example, a request, command, data, file, etc. generated by the processor 220 of the computer device 200 according to a program code stored in a recording device such as memory 210 is transmitted to the network ( 140) and can be transmitted to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 200 through the communication interface 230 of the computer device 200 via the network 140. Signals, commands, data, etc. received through the communication interface 230 may be transmitted to the processor 220 or memory 210, and files, etc. may be stored in a storage medium (as described above) that the computer device 200 may further include. It can be stored as a permanent storage device).

The input/output interface 240 may be a means for interfacing with the input/output device 250. For example, input devices may include devices such as a microphone, keyboard, or mouse, and output devices may include devices such as displays and speakers. As another example, the input/output interface 240 may be a means for interfacing with a device that integrates input and output functions, such as a touch screen. The input/output device 250 may be configured as a single device with the computer device 200.

Additionally, in other embodiments, computer device 200 may include fewer or more components than those of FIG. 2 . However, there is no need to clearly show most prior art components. For example, the computer device 200 may be implemented to include at least some of the input/output devices 250 described above, or may further include other components such as a transceiver, a database, etc.

Figure 3 is a diagram illustrating an example of a general view of a Foley sound providing system according to an embodiment.

Foley sound is a sound effect that is mixed into a video to improve the quality of sound during video work or to depict a realistic video with sound. It is created using various props and equipment. Mixing Foley sound is an essential task, but creating Foley sound and adding it to the video is done manually.

Through the Foley sound providing device 320 and method according to an embodiment, Foley sound suitable for the image can be generated and provided, thereby reducing the time and cost required for Foley sound recording and mixing.

Referring to FIG. 3, the Foley sound providing system 300 includes a user device 310 (e.g., electronic devices 110, 120, and 130) and a Foley sound providing device 320 (e.g., servers 150 and 160). ))) may be included. Here, the user device 310 and the Foley sound providing device 320 may each be implemented by at least one computer device 200.

The user can input an image to which he/she wants to add Foley sound through the user device 310. The image input through the user device 310 may be uploaded to the Foley sound providing device 320 through a network.

The Foley sound providing device 320 that receives the uploaded video can generate Foley sound based on artificial intelligence. Specifically, the poly sound providing device 320 may analyze an image using at least one neural network model and generate poly sound corresponding to the image according to the analysis result.

A neural network model may refer to a neural network that includes multiple hidden layers in addition to the input layer and output layer. Using a neural network model, you can identify latent structures in data. In other words, the potential structure of the image (for example, which object is in the photo, the posture of the object, etc.) can be identified. Neural network models include convolutional neural network (CNN), recurrent neural network (RNN), auto encoder, Generative Adversarial Networks (GAN), and restricted Boltzmann machine (RBM). ), deep belief network (DBN), Q network, U network, Siamese network, generative adversarial network (GAN), etc.

The neural network model according to one embodiment may be learned by at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Learning a neural network model may be a process of applying knowledge for the neural network model to perform a specific operation to the neural network. Hereinafter, the term ‘model’ may refer to a neural network model.

For example, referring to Figure 4, an example block diagram of a Foley sound providing device 320 according to one embodiment is shown. The poly sound providing device 320 may include a neural network processor 405, an editor 410, and a sound library 415. In one embodiment, neural network processor 405 and editor 410 may each be implemented by at least one computer device 200. Alternatively, the Foley sound providing device 320 may be composed of a single computer device 200 and the functions of the neural network processor 405 and the editor 410 may be performed by the processor 220 of the computer device 200.

The Foley sound providing device 320 can generate Foley sounds corresponding to the input image using a neural network processor 405, an editor 410, and a sound library 415. The functions of the neural network processor 405, editor 410, and sound library 415 will be described in detail with reference to the block diagram below in FIG. 5.

The Foley sound providing device 320 can provide download data so that the user can download the Foley sound through the user device 310, and the user can download the Foley sound.

Figure 5 is a flowchart of a method for providing poly sound according to an embodiment.

Referring to FIG. 5, the Foley sound providing device 320 may acquire a target image including a subject performing a target action in step 505. The target behavior may be predetermined. For example, the target behavior could be a person's steps. Footstep sounds may need to be added to the target image as foley sounds that correspond to footsteps in the target image.

The target video may be a video uploaded by the user, or a video classified by the Foley sound providing device 320 as corresponding to the target behavior among the videos uploaded by the user.

In one embodiment, the target image is generated by cutting out parts other than the part where the user wants to add footsteps from the original video, and the generated target image is sent to the Foley sound providing device 320 through the user device 310. It may have been uploaded by . The Foley sound providing device 320 can acquire the target image by receiving the target image uploaded by the user.

In another example, the user may upload the original video to the Foley sound providing device 320. The target image may be an image corresponding to a frame that the Foley sound providing device 320 classifies as corresponding to the target action among the original images.

For example, the neural network processor 405 of the Foley sound providing device 320 may use a neural network model (e.g., action classification model (not shown)) learned to classify at least one consecutive frame corresponding to the target action from the input image. ) may include.

The neural network processor 405 may classify at least one consecutive frame corresponding to the target behavior by inputting the original image into a learned behavior classification model (not shown). The Foley sound providing device 320 may acquire at least one continuous frame classified as corresponding to the target action as the target image.

In step 510, the Foley sound providing device 320 may determine context information related to the target behavior from the target image. Context information may include information about the surrounding environment of the subject related to the target behavior. For example, the context information may include properties of at least one object including the subject of the device.

The Foley sound providing device 320 may determine context information related to target behavior by recognizing the properties of at least one object including the subject from the target image. Context information may be information expressing the properties of at least one object including the subject in text.

The properties of objects included in the target image can be obtained using a neural network model. For example, the properties of the object included in the target image may be obtained using the image recognition models 411 to 413 included in the neural network processor 405.

Referring to FIG. 4 , the neural network processor 405 may include at least one image recognition model 411 to 413 learned to recognize the properties of a specific object from an input image. Each image recognition model 411 to 413 may be a model learned to recognize properties of different objects.

The target image acquired in step 510 may be input to each of at least one image recognition model 411 to 413 through the neural network processor 405. The neural network processor 405 may output properties of at least one different object included in the target image using at least one image recognition model 411 to 413.

The Foley sound providing device 320 may determine context information related to target behavior using the properties of the generated object, and may provide Foley sound corresponding to the situation of the target image using the determined context information.

Although not shown in the drawing, in another embodiment of the present invention, the Foley sound providing device 320 may receive context information about the properties of at least one object including the subject of the target image from the user device.

Context information received from the user device may include information about the subject or information about the properties of surrounding objects that affect the Foley sound. For example, if the Foley sound is the sound of footsteps, information about the subject may include, but is not limited to, the subject's gender, age, weight, and clothing. Information about the properties of surrounding objects may include, but is not limited to, the type of shoe, the material of the sole of the shoe, the material of the ground that the shoe touches, the space where the subject is located, and surrounding noise.

The Foley sound providing device 320 can automatically recognize the properties of at least one object including the subject using a neural network model, but the properties of the recognized object may not include sufficient information for generating the Foley sound. .

This may be an example of a case where the object properties required for poly sound generation are located outside the screen in the target image, or are included within the screen due to low resolution of the screen, but are not recognized by the neural network model. Alternatively, in order to realize a higher sense of immersion for the viewer, a case may be provided where the user wishes to artificially input the properties of the object used in generating the poly sound.

In one embodiment, the user device may display a user interface on the screen of the user device through which the user can input properties of an object. A user interface that allows input of object properties may be provided by displaying a blank space so that the user can input a desired word or sentence.

The user device may transmit the word or sentence input by the user to the poly sound providing device 320. The Foley sound providing device 320 may generate a Foley sound based on context information corresponding to a word or sentence input by the user.

In another embodiment, the user device may provide a plurality of object properties with high input frequency as candidates for the user to select, and display the user to select the desired object property.

In another embodiment, the user device may provide highly relevant object properties as candidates for the user to select based on the properties of the object primarily recognized through an image recognition model. That is, the Foley sound providing device 320 may determine the properties of an object that are highly correlated with the properties of the object recognized by the image recognition model as candidates.

For example, if the objects recognized by the image recognition model include shoes and the ground, the Foley sound providing device 320 may determine that the type of Foley sound is a footsteps sound. The Foley sound providing device 320 uses the material of the shoe sole, information about the area, the material of the shoe worn by the subject, and spatial information on the ground, as candidate object properties that affect the sound of footsteps, to the user. Can be provided to the device.

The user device may display the candidate object properties to the user through a user interface. The user device may transmit the properties of the object selected by the user to the poly sound providing device 320. The Foley sound providing device 320 may generate Foley sounds based on context information corresponding to the properties of the object selected by the user.

In step 515, the Foley sound providing device 320 may determine the duration and intensity of the target action by tracking the posture of the subject from the acquired target image. The Foley sound providing device 320 may determine the duration of the target action and the intensity of the target action using the neural network processor 405.

The posture of the subject included in the target image can be detected using a neural network model. For example, the neural network processor 405 may include a posture detection model 420 learned to detect the posture of a subject (eg, a person) from an acquired target image.

For example, the posture detection model 420 may be a skeleton detection model that recognizes joints of the human body. The Foley sound providing device 320 may acquire the movement (or motion) of the subject in the target image by tracking the posture of the subject based on the result of posture detection for each frame of the target image.

The subject's posture can be tracked using various algorithms or neural network models. For example, the subject's posture may be tracked using a Kalman filter-based posture tracking algorithm based on the posture detected from the posture detection model 420. For example, the subject's posture can be tracked using a neural network model learned to track continuous postures based on the posture detected using the posture detection model 420.

In an embodiment in which the target image acquired in step 505 includes the target behavior of the subject, the neural network processor 405 may determine the duration and intensity of the target behavior based on the acquired movement of the subject.

For example, if a target image including the full body or lower body of a walking person is input to the neural network processor 405 and the target action is a person's footsteps, the neural network processor 405 detects the walking person's body through the posture detection model 420. The posture of the whole body or lower body can be detected, and the movement of the whole body or lower body can be obtained by tracking the posture of the person's whole body or lower body based on the posture detection results.

The neural network processor 405 may determine the duration and intensity of footsteps based on the acquired movement of the subject. The neural network processor 405 can determine when the target action starts and ends in the target image according to the movement of the subject, and determines the difference between the two times as the duration of the target action. The neural network processor 405 may determine the intensity of the target action based on the speed at which the subject moves.

Depending on the embodiment, the neural network processor 405 creates a neural network model (e.g., the motion prediction model 435 in FIG. 4) that predicts movement not included in the target image of the subject based on the movement of the subject obtained from the target image. More may be included. For example, in an embodiment in which the target image acquired in step 505 does not include the target behavior of the subject, the motion prediction model 435 is used by the neural network processor 405 to predict the movement corresponding to the target behavior of the subject. may be further included.

For example, when a target image including the upper body of a walking person is input to the neural network processor 405 and the target action is a person's footsteps, the neural network processor 405 detects the upper body of the walking person through the posture detection model 420. It is possible to detect the posture and track the posture of the person's upper body based on the posture detection result to obtain the movement of the person's upper body, but the movement of the person's lower body may not be obtained.

In this case, the neural network processor 405 can predict the movement of the lower body from the acquired movement of the upper body using the motion prediction model 435 learned to predict the movement of the lower body based on the movement of the upper body. The Foley sound providing device 320 may determine the duration and intensity of steps based on the acquired movement of the lower body.

In step 520, the Foley sound providing device 320 generates a Foley sound corresponding to the target behavior of the subject based on the context determined in step 510 and the duration and intensity of the target behavior determined in step 515. You can.

Referring to FIG. 4 , the Foley sound providing device 320 may include a sound library 415 in which various sounds related to target actions are mapped and stored with text representing the corresponding sounds.

In one embodiment, the Foley sound providing device 320 may retrieve a Foley sound corresponding to a target action of a subject from the sound library 415 based on at least one of context, duration of the target action, and intensity of the target action. there is.

For example, the Foley sound providing device 320 may retrieve a Foley sound corresponding to at least one of the context, the duration of the target action, and the intensity of the target action from the sound library 415.

In another example, the poly sound providing device 320 may use a neural network model (e.g., : You can search for Foley sounds using the search model 430 of FIG. 4.

Referring to FIG. 4, the neural network processor 405 uses a search model 430 learned to output one of the sounds stored in the sound library 415 based on at least one of the context, the duration of the target action, and the intensity of the target action. ) may include.

For example, at least one of context, duration of the target behavior, and intensity of the target behavior may be input to the search model 430 of the neural network processor 405. The neural network processor 405 can output one of the sounds stored in the sound library 415 using the search model 430.

The Foley sound providing device 320 uses the Foley sound retrieved from the sound library 415 or the search model 430 to adjust the length of the output sound in consideration of the duration of the target action to produce the Foley sound corresponding to the target action. can be created.

Referring to FIG. 4, the Foley sound providing device 320 may include an editor 410 capable of editing sound. The editor 410 can perform an editing function on the audio track.

In an embodiment of the present invention, the editor 410 can adjust sync so that the generated poly sound corresponds to the screen in the video. Specifically, the editor 410 can adjust the position of the audio track, cut or paste a specific point, and perform sneak in and out functions.

In one embodiment, the Foley sound providing device 320 searches for a sound in the sound library 415 and generates a Foley sound in parallel, and uses a neural network model to generate a Foley sound corresponding to the target action. .

For example, the Foley sound providing device 320 uses the Foley sound generation model 425, which is learned to generate Foley sounds based on at least one of the context, the duration of the target behavior, and the intensity of the target behavior, to target behavior. A corresponding poly sound can be created.

Referring to FIG. 4 , the neural network processor 405 may include a Foley sound generation model 425 trained to generate Foley sounds based on at least one of context, duration of the target action, and intensity of the target action. At least one of the context determined in step 510 and the duration and intensity of the target action determined in step 515 may be input to the Foley sound generation model 425 . The neural network processor 405 may use the Foley sound generation model 425 to generate a Foley sound corresponding to the target action.

In step 525, the Foley sound providing device 320 may provide download data of the generated Foley sound or download data combining the generated Foley sound and the target image to the user device. Merging between the generated Foley sound and the target image may be performed by an editor (eg, editor 410 in FIG. 4). The editor 410 may perform an operation to correspond the generated poly sound to the screen in the video through at least one of the following functions: adjusting the position of the generated poly sound, cutting or pasting, and sneaking in and out on the audio track. Users can download the generated Foley sounds through their device.

Hereinafter, an exemplary method of providing a Foley sound when the target action is a person's footsteps will be described with reference to FIGS. 6 and 7.

An image of a person walking may include the full body, upper body, or lower body of the walking person. When a person's entire body is included in the target image or a person's lower body is included in the target image, the Foley sound providing device 320 may track the person's movements from the target image and provide Foley sounds corresponding to footsteps.

Referring to FIG. 6, in one embodiment, a flowchart of a method for providing poly sound of footsteps, which is a target action, to a target image of a person's full body or lower body is shown.

The Foley sound providing device 320 of one embodiment may acquire a target image including the lower body or whole body of a walking subject (i.e., a person) in step 605.

The target video may be a video uploaded by the user, or a video classified by the Foley sound providing device 320 as corresponding to the target behavior among the videos uploaded by the user.

In one embodiment, the user generates a target image by cutting out a portion including the lower body or whole body of a person walking from the original image, and sends the generated target image to the Foley sound providing device 320 through the user device 310. ) may have been uploaded. The Foley sound providing device 320 can acquire the target image by receiving the target image uploaded by the user.

In another example, the user may upload the original video to the Foley sound providing device 320. The target image may be an image corresponding to a frame classified by the Foley sound providing device 320 as containing the lower body or full body of a walking person among the original images. For example, the neural network processor 405 of the Foley sound providing device 320 may use a neural network model (e.g., action classification) learned to classify at least one continuous frame containing the lower body or full body of a person walking from the input image. A model (not shown) may be included.

The neural network processor 405 may classify at least one consecutive frame corresponding to the target behavior by inputting the original image into a learned behavior classification model (not shown). The Foley sound providing device 320 may acquire at least one continuous frame classified as corresponding to the target action as the target image.

In the example of FIG. 6 , step 605 may be a step included in step 505 of FIG. 5 .

The Foley sound providing device 320 may recognize the properties of at least one object including the subject in steps 610, 615, and 620. At least one object may be an object related to footsteps, which is the target action. At least one object related to the step may be predetermined. In the example of FIG. 6 , at least one object may include a person as the subject, the ground on which the person is walking, and the shoes the person is wearing. However, this is an example, and at least one object related to the target action may include various objects.

The properties of each of at least one object may be acquired using different neural network models (eg, the first image recognition model 411 to the Nth image recognition model 413 in FIG. 4). For example, in step 610, the Foley sound providing device 320 inputs the input image into a first image recognition model learned to recognize the gender of the person, and in step 615, selects the input image from the input image. The image may be input to a second image recognition model trained to recognize the material of the ground, and in step 620, may be input to a third image recognition model learned to recognize the material of the shoe.

The poly sound providing device 320 acquires the gender of the person through the output of the first image recognition model, acquires the material of the ground through the output of the second image recognition model, and obtains the material of the shoe through the output of the third image recognition model. It can be obtained.

In step 625, the Foley sound providing device 320 may determine the context associated with the target action, footsteps, based on the properties of the at least one object obtained in steps 610, 615, and 620. there is.

In one embodiment, context information may be determined in text form based on properties of at least one object. For example, context information may be information expressing the properties of at least one object in text. For example, in step 610, male is recognized as the human gender through a plurality of image recognition models, in step 615, concrete is recognized as the material of the ground, and in step 620, leather is recognized as the material of the shoe. If recognized, the context information may be text information of “Man-Concrete-Leather” linking the output of the image recognition models. In the example of FIG. 6 , steps 610 , 615 , 620 , and 625 may be steps included in step 510 of FIG. 5 .

In step 630, the Foley sound providing device 320 may determine the duration and intensity of the target action by tracking the posture of the subject from the acquired target image. In one embodiment, the Foley sound providing device 320 uses a posture detection model (e.g., the posture detection model 420 of FIG. 4) learned to recognize footsteps from an image, and determines the duration and duration of footsteps, which is the target action. intensity can be determined. For example, the Foley sound providing device 320 may detect and track the posture of the lower body or whole body of a person by inputting the acquired target image into the posture detection model.

The Foley sound providing device 320 may determine the duration and intensity of steps by tracking the subject's behavior based on the results of posture detection in response to detecting the posture of the person's lower body or whole body. In the example of FIG. 6 , step 630 may be a step included in step 515 of FIG. 5 .

In step 635, the Foley sound providing device 320 sounds a Foley sound corresponding to the subject's steps based on at least one of the context determined in step 625 and the duration and intensity of the steps determined in step 630. You can search in a library (e.g., sound library 415 in FIG. 4). For example, the Foley sound providing device 320 may search for a Foley sound corresponding to at least one of the context, the duration of footsteps, and the intensity of footsteps from the sound library.

In another example, the Foley sound providing device 320 may use a search model learned to output any one of the sounds stored in the sound library based on at least one of the context, the duration of the footsteps, and the intensity of the footsteps (e.g., the search in FIG. 4 You can search for Foley sounds using the model 430).

In step 640, the Foley sound providing device 320 may generate a Foley sound corresponding to the subject's footsteps by adjusting the length of the Foley sound retrieved in step 635 in consideration of the duration determined in step 630. there is.

In step 645, in parallel with steps 635 to 640, the foley sound providing device 320 learns to generate foley sounds based on at least one of context, duration of footsteps, and intensity of footsteps. A Foley sound corresponding to the subject's footsteps can be generated using the Foley sound generation model (e.g., the Foley sound generation model 425 of FIG. 4). In the example of Figure 6, steps 635, 640, and 645 may be included in step 520 of Figure 5.

Although not shown in FIG. 6, in the method of providing Foley sound according to an embodiment, the Foley sound providing device synchronizes the target image and Foley sound using an editor (e.g., editor 410 in FIG. 4) before step 650. It may further include a step of adjusting. In the step of adjusting the sync, the Foley sound providing device may generate one Foley sound by combining the Foley sound generated through steps 635 to 640 and the Foley sound generated through step 645. .

In Figure 6, steps 635 through 640 are shown as being performed in parallel with step 645, but in other embodiments, step 635 is performed first and the result of step 635 is A step (not shown) of determining whether to perform step 645 depending on whether a set condition is satisfied may be further included. For example, in determining whether to perform step 645 (not shown), the Foley sound providing device may determine whether the Foley sound retrieved in step 635 is appropriate. For example, the Foley sound providing device may determine the difference between the length of the retrieved Foley sound and the duration of the footsteps to be appropriate if it is less than or equal to a preset value, and may determine it to be inappropriate if it exceeds the preset value.

For example, if the duration of the retrieved Foley sound is 1 second and the duration of the subject's footsteps is 3 seconds, if the Foley sound of 1 second is extended to 3 seconds in step 640, the quality of the Foley sound may deteriorate, so the Foley sound may be deteriorated. The sound providing device may determine that the searched Foley sound is not appropriate. Step 645 may be performed when it is determined that the retrieved Foley sound is not appropriate. If the retrieved Foley sound is determined to be appropriate, step 640 may be performed.

Again in step 650 of FIG. 6, the Foley sound providing device 320 may provide download data of the Foley sound generated in at least one of steps 640 and 645 to the user device 310. there is. In one embodiment, the Foley sound providing device 320 may provide download data of the generated Foley sound based on a user input, or may provide download data that merges the generated Foley sound with the target image. Merging between the generated Foley sound and the target image may be performed by an editor (eg, editor 410 in FIG. 4). In the example of Figure 6, step 650 may be included in step 525 of Figure 5.

Referring to FIG. 7, unlike the case of FIG. 6, a flowchart of a method of providing poly sound of footsteps, which is a target action, to a target image in which a person's upper body is captured is shown.

The Foley sound providing device 320 of one embodiment may acquire a target image including the upper body of a walking subject (i.e., a person) in step 705.

The target video may be a video uploaded by the user, or a video classified by the Foley sound providing device 320 as corresponding to the target behavior among the videos uploaded by the user.

In one embodiment, the user creates a target image by cutting out a portion including the upper body of a person walking from the original image, and sends the generated target image to the Foley sound providing device 320 through the user device 310. It may have been uploaded. The Foley sound providing device 320 can acquire the target image by receiving the target image uploaded by the user.

In another example, the user may upload the original video to the Foley sound providing device 320. The target image may be an image corresponding to a frame classified by the Foley sound providing device 320 as containing the upper body of a walking person among the original images. For example, the neural network processor 405 of the Foley sound providing device 320 may use a neural network model (e.g., a behavior classification model ( (not shown)) may be included.

The neural network processor 405 may classify at least one consecutive frame corresponding to the target behavior by inputting the original image into a learned behavior classification model (not shown). The Foley sound providing device 320 may acquire at least one continuous frame classified as corresponding to the target action as the target image.

In the example of FIG. 7 , step 705 may be a step included in step 505 of FIG. 5 .

In step 710, the Foley sound providing device 320 may recognize the properties of at least one object including the subject. At least one object may be an object related to footsteps, which is the target action. At least one object related to the step may be predetermined. In the example of FIG. 7, at least one object may be a subject. However, this is an example, and at least one object related to the target action may include various objects.

The properties of each of at least one object may be acquired using different neural network models (eg, the first image recognition model 411 to the Nth image recognition model 413 in FIG. 4). For example, in step 710, the Foley sound providing device 320 may input input into a first image recognition model learned to recognize a person's gender from an input image. The Foley sound providing device 320 may obtain the person's gender as an output of the first image recognition model.

In step 715, the Foley sound providing device 320 may determine a context related to footsteps, which is the target action, based on the property of the object (i.e., the gender of the person) obtained in step 710. In one embodiment, context information may be determined in text form based on properties of at least one object. For example, context information may be information expressing the properties of at least one object in text. For example, if male is recognized as the person's gender in step 710, the context information may be “male.” However, a person's gender is only an example, and context information expressing the properties of any object that affects the Foley sound may be included. In the example of FIG. 7 , steps 710 and 715 may be steps included in step 510 of FIG. 5 .

Although not shown in the drawing, the Foley sound providing device 320 may receive context information about the properties of at least one object including the subject of the target image from the user device.

There are cases where it is difficult to generate foley sound that accurately corresponds to the target image because the context information required for generating foley sound is located outside the screen of the target image. As in the embodiment of FIG. 7, when trying to generate a footsteps sound, only the upper body of the subject may be displayed on the screen. Accordingly, there may be a limitation in predicting information about the subject from the movement of the upper body or predicting the properties of the object from the surrounding environment surrounding the subject.

Accordingly, the Foley sound providing device 320 may generate the Foley sound based on context information about the properties of the object directly input by the user. To this end, the Foley sound providing device 320 may provide a user interface that can receive context information through a user device. The user can enter context information about the properties of the object. The user device may transmit the input context information to the Foley sound providing device 320.

In an embodiment of the present invention, the Foley sound providing device 320 may receive context information about the material of the shoe and the material of the ground from the user device. The Foley sound providing device 320 generates footsteps based on context information about the material of the shoe and the material of the ground directly input by the user, along with context information about the person's gender determined through the first image recognition model described above. can be created.

In another embodiment, the Foley sound providing device 320 may predict the properties of an object that affects the sound of footsteps but is not displayed on the screen, based on information recognizing objects in the screen of the target image. The properties of the predicted object may be provided to the user in the form of context information expressed as text.

The user can select context information appropriate for the target image among context information that is not displayed on the screen but affects Foley sound. The Foley sound providing device 320 may receive context information selected by the user and generate Foley sound based on this.

In step 720, the Foley sound providing device 320 may predict the movement of the subject's lower body by tracking the posture of the subject's upper body from the target image. For example, the Foley sound providing device 320 may detect the posture of the subject's upper body from the target image through a posture detection model (e.g., the posture detection model 420 of FIG. 4). In response to detecting the posture of the subject's upper body, the Foley sound providing device 320 may obtain movement of the upper body by tracking the posture of the upper body based on the result of posture detection. The Foley sound providing device 320 uses a neural network model (e.g., motion prediction model 435 in FIG. 4) learned to predict lower body movement based on upper body movement, and acquires lower body movement from the obtained upper body movement. can do.

In step 725, the Foley sound providing device 320 may determine the duration and intensity of a step, which is the target action, based on the movement of the lower body predicted in step 720. For example, the Foley sound providing device 320 can determine when steps start and end in the target image according to the movement of the lower body, and determine the difference between the two times as the duration of steps. The Foley sound providing device 320 may determine the intensity of footsteps based on the speed at which the subject moves. In the example of FIG. 7 , steps 720 and 725 may be steps included in step 515 of FIG. 5 .

In step 730, the Foley sound providing device 320 sounds a Foley sound corresponding to the subject's steps based on at least one of the context determined in step 715 and the duration and intensity of the steps determined in step 725. You can search in a library (e.g., sound library 415 in FIG. 4). For example, the Foley sound providing device 320 may search for a Foley sound corresponding to at least one of the context, the duration of footsteps, and the intensity of footsteps from the sound library.

In another example, the Foley sound providing device 320 may use a search model learned to output any one of the sounds stored in the sound library based on at least one of the context, the duration of the footsteps, and the intensity of the footsteps (e.g., the search in FIG. 4 You can search for Foley sounds using the model 430).

In step 735, the Foley sound providing device 320 may generate a Foley sound corresponding to the subject's footsteps by adjusting the length of the Foley sound retrieved in step 730 by considering the duration determined in step 725. there is.

In step 740, in parallel with steps 730 to 735, the foley sound providing device 320 learns to generate foley sounds based on at least one of context, duration of footsteps, and intensity of footsteps. A Foley sound corresponding to the subject's footsteps can be generated using the Foley sound generation model (e.g., the Foley sound generation model 425 of FIG. 4). In the example of Figure 7, steps 730, 735, and 740 may be included in step 520 of Figure 5.

Although not shown in FIG. 7, the Foley sound providing method according to one embodiment involves the Foley sound providing device synchronizing the target image and the Foley sound using an editor (e.g., the editor 410 of FIG. 4) before step 745. It may further include the step of adjusting . In the step of adjusting the sync, the Foley sound providing device may generate one Foley sound by combining the Foley sound generated through steps 730 to 735 and the Foley sound generated through step 740. .

In Figure 7, steps 730 through 735 are shown as being performed in parallel with step 740, but in other embodiments, step 730 is performed first and the result of step 730 is A step (not shown) of determining whether to perform step 740 depending on whether a set condition is satisfied may be further included. For example, in determining whether to perform step 740 (not shown), the Foley sound providing device may determine whether the Foley sound retrieved in step 730 is appropriate. For example, the Foley sound providing device may determine the difference between the length of the searched Foley sound and the duration of the footsteps to be appropriate if it is less than or equal to a preset value, and may determine it to be inappropriate if it exceeds the preset value. For example, if the duration of the retrieved Foley sound is 1 second and the duration of the subject's footsteps is 3 seconds, if the Foley sound of 1 second is extended to 3 seconds in step 735, the quality of the Foley sound may deteriorate, so the Foley sound may be deteriorated. The sound providing device may determine that the searched Foley sound is not appropriate.

Step 740 may be performed when it is determined that the retrieved Foley sound is not appropriate. If the retrieved Foley sound is determined to be appropriate, step 735 may be performed.

Again in step 745 of FIG. 7, the Foley sound providing device 320 may provide download data of the Foley sound generated in at least one of steps 735 and 740 to the user device 310. there is. In one embodiment, the Foley sound providing device 320 may provide download data of the generated Foley sound based on a user input, or may provide download data that merges the generated Foley sound with the target image. Merging between the generated Foley sound and the target image may be performed by an editor (eg, editor 410 in FIG. 4). In the example of Figure 7, step 745 may be included in step 525 of Figure 5.

FIG. 8 is a diagram illustrating an example of a user screen of a method for providing Foley sound according to an embodiment.

In one embodiment, the user can use the Foley sound providing service provided by the Foley sound providing device 320 through the user device 310. Referring to FIG. 8 , an example of the user screen 800 displayed on the user device 310 when the user uses the Foley sound providing service through the user device 310 is shown.

The user screen 800 includes a project upload area 805 that receives the project file, a video viewer area 810 that displays the uploaded video, a video track of the uploaded video, and a poly sound generated by the poly sound providing device 320. A timeline area 815 that displays the soundtrack of the sound, a control area 820 that adjusts the sound volume and equalizer (EQ), and a text tag area 825 that displays the context created by the poly sound providing device 320. ) and other areas 830.

For example, context information of a target image recognized through a Foley sound providing device may be displayed in the text tag area 825. The other area 830 may include, for example, at least one of a user interface (UI) element commanding the download of a poly sound and a UI element commanding the download of an image in which the poly sound is merged.

The user can upload a video for mixing Foley sound through the project upload area 805 or the video viewer area 810. The Foley sound providing device 320 may generate Foley sounds by performing the methods of FIGS. 4 to 7 on the uploaded video and transmit the generated Foley sounds to the user device 310. The sound track of the Foley sound transmitted to the user device 310 may be displayed in the timeline area 815 of the user screen 800 along with the video track of the uploaded image.

For example, the poly sound displayed in the timeline area 815 of the user device 310 is the download data of step 525 in FIG. 5, step 650 in FIG. 6, and step 745 in FIG. 7. It may be a Foley sound displayed on the user device 310 before being provided. The user can edit the foley sound by manipulating the sound track displayed in the timeline area 815. When the user edits the Foley sound through the user device, the Foley sound providing device 320 may receive a sound editing command according to the user's manipulation from the user device. Editing of the Foley sound according to the sound editing command can be performed by the Foley sound providing device 320 through an editor (e.g., the editor 410 of FIG. 4) of the Foley sound providing device 320.

When the user edits the Foley sound, the Foley sound providing device 320 downloads or edits the edited Foley sound in step 525 of FIG. 5, step 650 of FIG. 6, and step 745 of FIG. 7. The poly sound can provide download data for the merged video.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions.

The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may store program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. there is. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

110, 120, 130: Electronic devices
140: network
150, 160: Server
210: memory
220: processor
230: communication interface
240: input/output interface
250: input/output device
300: Foley sound provision system
310: user device
320: Foley sound providing device

Claims (12)

In a method of providing artificial intelligence-based poly sound performed by a computer device,
Obtaining a target image including a subject performing a target action;
determining context information related to the target behavior from the target image;
From the target image, determining the duration of the target action and the intensity of the target action by tracking the posture of the subject;
generating a Foley sound corresponding to a target action of the subject based on the determined context information, the duration, and the intensity; and
Providing download data of the generated Foley sound or download data in which the generated Foley sound and the target image are merged.
Including,
The step of determining the context information is,
The properties of at least one object including the subject are recognized by the artificial intelligence and provided as candidates for the user to select, or based on information that initially recognizes the object in the screen of the target image by the artificial intelligence, A method comprising predicting the properties of other objects that affect the poly sound but not displayed on the screen and providing them as candidates for the user to select. According to paragraph 1,
The step of generating the poly sound is,
searching for a sound corresponding to the determined context information from a sound library; and
When a sound corresponding to the context information is searched, adjusting the length of the searched sound based on the duration.
Method, including. According to paragraph 2,
The step of generating the poly sound is,
When a sound corresponding to the context information is not searched, inputting the context information, the duration, and the intensity into a learned neural network model; and
As an output of the learned neural network model, generating a poly sound corresponding to the input context information, duration, and intensity.
A method further comprising: According to paragraph 1,
The target action is a person's footsteps,
The properties of the at least one object are:
A method including the gender of the subject, the material of the ground on which the subject is walking, and the material of the shoes the subject is wearing. According to paragraph 4,
The context information is,
Characterized in that the method is determined in text form based on the gender of the recognized subject, the recognized material of the ground surface, and the recognized material of the shoe. According to paragraph 1,
The target action is a person's footsteps,
The step of determining the duration of the target behavior and the intensity of the target behavior is,
detecting the posture of the subject from the target image;
acquiring movement of the subject in the target image by tracking the detected posture; and
determining the duration of the step and the intensity of the step based on the obtained movement.
Method, including. According to clause 6,
The step of acquiring the movement of the subject is,
In response to detecting the posture of the subject's upper body, tracking the detected posture of the upper body to obtain movement of the upper body; and
Obtaining movement of the lower body of the subject based on movement of the upper body
Method, including. According to paragraph 1,
The above method is,
Receiving an original image from a user device; and
Inputting the original image into a learned neural network model to classify at least one consecutive frame corresponding to the target action among the frames of the original image,
The step of acquiring the target image is,
A method of obtaining the classified at least one consecutive frame as the target image. According to paragraph 1,
The properties of each of the at least one object are:
It is recognized through different neural network models learned to recognize the properties of each object from images,
The context information is,
A method, characterized in that it is text information connecting the outputs of the different neural network models. According to paragraph 1,
The above method is,
transmitting the generated Foley sound to a user device;
Receiving a sound editing command for the Foley sound from a user device that has received the Foley sound; and
Editing the poly sound according to the sound editing command
It further includes,
The step of providing the download data is,
A method comprising providing download data of the edited Foley sound or download data of a target video into which the edited Foley sound is merged. A computer program combined with hardware and stored in a computer-readable recording medium to execute the method of any one of claims 1 to 10. In computer devices,
processor; and
Includes a memory that stores instructions to be executed by the processor,
When the instructions are executed by the processor, the processor:
Acquire a target image containing a subject performing the target action,
From the target image, determine context information related to the target behavior,
From the target image, track the posture of the subject to determine the duration of the target action and the intensity of the target action,
Based on the determined context information, the duration and the intensity, generate a poly sound corresponding to a target action of the subject, and
Providing download data of the generated Foley sound or download data in which the generated Foley sound and the target video are merged,
The context information is,
The user selects the properties of at least one object including the subject by recognizing it using artificial intelligence, or does not affect the foley sound based on information that initially recognizes the object in the screen of the target image using artificial intelligence. A computer device, in which the user's selection is determined only by predicting the properties of other objects not displayed on the screen.