WO2021221394A1

WO2021221394A1 - Method and electronic device for image augmentation

Info

Publication number: WO2021221394A1
Application number: PCT/KR2021/005171
Authority: WO
Inventors: 쿠마르 체루쿠리사티쉬; 이신준; 하누만타차르야 나가누르라자람; 차크라보르티프라센지트; 쿠마르 나바니사 크리쉬난사랄
Original assignee: 삼성전자 주식회사
Priority date: 2020-04-30
Filing date: 2021-04-23
Publication date: 2021-11-04
Also published as: KR20210134229A

Abstract

Disclosed are an electronic device and method for augmenting a virtual object in an image. The electronic device for augmenting a virtual object in an image may comprise: a communication unit; a display; a memory for storing a program including one or more instructions; and at least one processor which executes the one or more instructions of the program stored in the memory. The at least one processor can identify one or more objects in the image, classify the one or more objects into a main object and sub-objects, acquire a virtual object related to the main object on the basis of the main object and the sub-objects, acquire depth information about the three-dimensional space in which the image was imaged, acquire a trajectory of the virtual object on the basis of the depth information, the trajectory showing the relative three-dimensional movement path with respect to the main object, and control the display to display the movement of the virtual object.

Description

Method and electronic device for image augmentation

Various embodiments relate to a method of augmenting an image in an electronic device.

As technology advances, electronic devices such as mobile devices have media editing functions that allow users to edit images. For example, users of mobile devices will be able to add augmented reality (AR) stickers within images while capturing images.

However, image editing to add stickers is manually performed by the user, and the added stickers also have a limit in providing an improved user experience. In addition, since the stickers are simply overlaid on the image, the edited image has a limit in providing a visually attractive output to the user.

In addition, the function of adding an AR sticker to give an AR effect to the image is to detect a horizontal plane and a vertical plane in the preview image of the camera and overlay objects on the detected plane. It is difficult to give an AR effect to an image.

Accordingly, when editing media, it is possible to provide visually outstanding and original media output by editing media using information acquired through technologies such as image/video depth segmentation, object classification, and artificial intelligence. skills are required

As a technical means for achieving the above technical problem, a method for an electronic device to augment a virtual object in an image according to an embodiment includes: identifying one or more objects in the image; classifying the one or more objects into a main object and a sub-object; obtaining a virtual object related to the main object based on the main object and the sub-object; obtaining depth information of a three-dimensional space in which the image is captured; obtaining a trajectory of the virtual object indicating a three-dimensional movement path relative to the main object, based on the depth information; The method may include moving the virtual object along the trajectory.

In the method of an electronic device augmenting an image with a virtual object according to an embodiment, the moving of the virtual object includes: a size of the virtual object, a direction of the virtual object with respect to the main object, and a direction of the virtual object with respect to the main object. The method may include changing at least one of a position of the virtual object and a perspective view of the virtual object.

In the method of an electronic device augmenting a virtual object on an image, according to an embodiment, the moving of the virtual object comprises: a size of the virtual object and the main object based on at least one of a size, a location, and a category of the main object. The method may further include determining a direction of the virtual object with respect to the object, a position of the virtual object with respect to the main object, and a perspective view of the virtual object.

In the method for augmenting a virtual object on an image by an electronic device according to an embodiment, the acquiring of the virtual object includes: a relationship between sub-objects identified in the image, a relationship between a main object and sub-objects, an interest identifying at least one of a region and a context of the image; Generating a context connection tree representing a connection relationship between the main object and the sub-objects based on at least one of a relationship between the sub-objects, a relationship between a main object and sub-objects, a region of interest, and a context of the image ; and identifying the virtual object based on the context connection tree.

According to an embodiment, a method for an electronic device to augment a virtual object on an image, the method comprising: obtaining virtual object candidates to be added to the image based on the identified one or more objects; and obtaining, for each of the virtual object candidates, a trajectory cluster including trajectory candidates corresponding to each of the virtual object candidates, wherein the obtaining of the virtual object includes: selecting a virtual object to be added to the image, wherein obtaining the trajectory includes: selecting a trajectory indicating a movement path of the selected virtual object within the trajectory cluster corresponding to the selected virtual object may include

In the method of augmenting a virtual object on an image by an electronic device according to an embodiment, the depth information may include foreground information and background information of the image.

In a method by an electronic device to augment a virtual object on an image according to an embodiment, the moving the virtual object may include controlling visibility of the virtual object based on the depth information. have.

In the method of an electronic device augmenting an image with a virtual object according to an embodiment, the moving of the virtual object includes: the depth information, a direction of the virtual object with respect to the main object, and the virtual object from the main object The method may further include changing the perspective view based on the distance of .

In the method for augmenting a virtual object on an image by an electronic device according to an embodiment, the acquiring the trajectory includes acquiring a user input guiding a movement path of the virtual object from a user of the electronic device can do.

In the method for augmenting a virtual object on an image by an electronic device according to an embodiment, the acquiring the trajectory includes applying the virtual object to an artificial intelligence model to acquire a trajectory corresponding to the virtual object can do.

Also, according to an embodiment, an electronic device for augmenting a virtual object on an image includes: a communication unit; display; a memory storing a program including one or more instructions; at least one processor executing one or more instructions of the program stored in the memory, wherein the at least one processor identifies one or more objects in the image and converts the one or more objects into main objects and sub-objects. classify, obtain a virtual object related to the main object based on the main object and the sub-object, obtain depth information of a three-dimensional space in which the image is captured, and based on the depth information, with respect to the main object The electronic device may obtain a trajectory of the virtual object indicating a relative three-dimensional movement path and control the display to display the movement of the virtual object.

1 is a block diagram illustrating a configuration of an electronic device according to an exemplary embodiment.

2 is a block diagram illustrating a configuration of an object controller according to an exemplary embodiment.

3 is a flowchart illustrating a method in which an electronic device generates an augmented image by identifying a virtual object and a trajectory of the virtual object, according to an exemplary embodiment.

FIG. 4 is a diagram for describing a method in which an electronic device generates a moving trajectory of a virtual object using an artificial intelligence model, according to an exemplary embodiment.

5 is a flowchart illustrating a method for an electronic device to determine a virtual object and a trajectory by using object identification information, image depth information, and context information, according to an exemplary embodiment.

FIG. 6 is a diagram for describing a method for an electronic device to recognize an object in an image and acquire a virtual object, according to an exemplary embodiment.

7 is a diagram for describing a method of an electronic device moving a virtual object in an image based on depth information, according to an exemplary embodiment.

8 is a diagram for describing a method of an electronic device moving a virtual object in an image based on a user input, according to an exemplary embodiment.

FIG. 9 is a diagram for describing a method of obtaining, by an electronic device, a virtual object to be augmented on an image and a trajectory of the virtual object, according to an exemplary embodiment;

FIG. 10 is a diagram for describing a method of obtaining, by an electronic device, a virtual object based on information obtained from an image, according to an exemplary embodiment.

11 is a diagram for describing a method of obtaining, by an electronic device, a virtual object by a user input, according to an exemplary embodiment.

12 is a diagram for describing a method of an electronic device recommending a virtual object and a trajectory of the virtual object, according to an exemplary embodiment.

13 is a diagram for describing various methods of augmenting an image by an electronic device using a virtual object, according to an exemplary embodiment.

14 is a diagram for describing a method of augmenting an image based on a user gesture by an electronic device, according to an exemplary embodiment.

15 is a flowchart illustrating a method of augmenting an image by an electronic device according to an exemplary embodiment.

16 is a flowchart illustrating a method for an electronic device to acquire a virtual object, according to an embodiment.

17 is a diagram for describing a method of obtaining, by an electronic device, an animation effect and a trajectory of a virtual object based on a pre-stored mapping table, according to an exemplary embodiment.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

The singular expression may include the plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meanings as commonly understood by one of ordinary skill in the art described herein. Also, terms including an ordinal number such as 'first' or 'second' used in this specification may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a part "includes" a certain element throughout the specification, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as "unit" and "engine" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art can easily carry out the embodiments of the present invention. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the embodiments disclosed below, 'augmenting an image' by the electronic device 100 means that the electronic device 100 adds a virtual object in the original image, and a specific effect (eg, animation effect, virtual object movement, etc.). In this case, a moving picture implementing the above-described animation effect may be generated through 'image augmentation', and each frame of the generated moving picture may be composed of a plurality of augmented images.

In the embodiments disclosed below, 'augmentation of an image' may be equally applied and expanded to a method of 'augmentation of a video'. For convenience of explanation, the following description will be based on 'image'.

The electronic device 100 according to an embodiment may augment a virtual object in an image according to the disclosed embodiments. The electronic device 100 may be, for example, a smart watch, a smart phone, a smart TV, a smart robot, an artificial intelligence speaker, a personal digital assistant (PDA), a tablet computer, a laptop computer, a music player, a video player, etc., but is limited thereto. it's not going to be

The electronic device 100 according to an embodiment may be configured to include at least a processor 110 , a communication unit 120 , a memory 130 , a display 140 , and an object controller 150 .

The processor 110 according to an embodiment may execute one or more instructions of a program stored in the memory 130 . The processor 110 may be composed of hardware components that perform arithmetic, logic, input/output operations and signal processing.

Processor 110 is, for example, a central processing unit (Central Processing Unit), a microprocessor (microprocessor), a graphic processor (Graphic Processing Unit), ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), Application Processors (APs), Neural Processing Units, or artificial intelligence-only processors designed with a hardware structure specialized for processing artificial intelligence models. It may consist of at least one, but is not limited thereto.

The communication unit 120 according to an embodiment may perform data communication with an external device (eg, a server or other electronic device) under the control of the processor 110 .

The communication unit 120 is, for example, a wired LAN, a wireless LAN, a Wi-Fi, a Bluetooth, a Zigbee, a Wi-Fi Direct (WFD), an infrared communication (IrDA). Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), WiBro (Wireless Broadband Internet, Wibro), WiMAX (World Interoperability for Microwave Access, WiMAX), SWAP (Shared Wireless Access Protocol), WiGig ( Data communication with an external device may be performed using at least one of data communication methods including Wireless Gigabit Alliance (WiGig) and RF communication.

In the memory 130 according to an embodiment, instructions, data structures, and program codes readable by the processor 110 may be stored. In the disclosed embodiments, operations performed by the processor 110 may be implemented by executing instructions or codes of a program stored in the memory 130 .

The memory 130 may include, for example, a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg, SD or XD memory). etc.), RAM (Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic It may include at least one type of storage medium among a memory, a magnetic disk, and an optical disk.

The display 140 according to an embodiment may display information processed by the electronic device 100 . For example, the display 140 may display the augmented image generated by the electronic device 100 . In this case, the augmented image may be an image in which a virtual object generated by the electronic device 100 is rendered in the image. Also, by using a plurality of images augmented by rendering the virtual object, a video in which the virtual object moves within the image and reproduces various animation effects may be generated. Hereinafter, for convenience of explanation, a method of augmenting an image by the electronic device based on any one of the respective augmented images will be described. However, the disclosed embodiments may include an embodiment in which an augmented video is generated using a plurality of augmented images.

The object controller 150 according to an embodiment may perform various operations for the electronic device 100 to generate an augmented image.

In an embodiment, the object controller 150 may acquire a virtual object and a trajectory indicating a path along which the virtual object moves in the image. In this case, the trajectory may be a trajectory of a virtual object indicating a movement path relative to the main object. The object controller 150 may move the virtual object in the image along the trajectory of the virtual object. However, the present invention is not limited thereto, and the object controller 150 may augment the image by applying only a predetermined animation effect without moving the virtual object in the image.

The object controller 150 according to an embodiment may obtain depth information from an image. In order to augment the virtual object in the image, the object controller 150 selects at least one of the size, direction and position of the virtual object at different depths in the three-dimensional space of the image including depth information, and a perspective view. can be changed In this case, the trajectory on which the virtual object moves may be a 3D trajectory generated based on the acquired trajectory and depth information.

The object controller 150 is implemented as hardware that enables the electronic device 100 to perform a series of operations for generating an augmented image, or is implemented in the form of software and stored in the memory 130 of the electronic device 100 . can be

A detailed configuration and operation of the object controller 150 will be further described in the description of FIG. 2 .

The electronic device 100 according to an embodiment may augment the virtual object in the image acquired by the electronic device using the object controller 150 .

Referring to FIG. 2 , the object controller 150 according to an embodiment includes at least an object identification and classification engine 202 , a depth segmentation engine 204 , a context analyzer 206 , and one or more artificial intelligence models 208 . may include The object controller 150 may perform overall operations for controlling the virtual object in the image.

In the disclosed embodiment, the main object may be an object determined as a subject to be photographed from among the objects identified in the image. The electronic device 100 according to an embodiment may classify an object determined to be an important object among the objects identified in the image as a main object. The identified main object may be plural.

In the disclosed embodiment, the sub-object may be an object that is not a target of photographing among objects identified in the image, but exists in the image. The electronic device 100 according to an embodiment may classify an object that is not classified as a main object in an image as a sub-object. The identified sub-objects may be plural.

In the disclosed embodiment, the virtual object may be an object for generating an augmented image by adding an animation effect to the acquired image. The electronic device 100 according to an embodiment may acquire a virtual object to be added to an image based on at least a part of the main object, the sub-object, and the context information of the image.

In one embodiment, the object controller 150 may use the object identification and classification engine 202 to identify and classify objects within the image. The object identification and classification engine 202 may identify at least one object within the image. In this case, the object identification and classification engine 202 may classify the identified at least one object into a main object and a sub-object. When there is one object in the image, one identified object may be classified as a main object, and in this case, the main object may be treated as a main object and a sub-object in the disclosed embodiments. The object identification and classification engine 202 may identify at least one object existing in the image by using the object recognition model among the artificial intelligence models 208 .

In one embodiment, the object identification and classification engine 202 may classify at least one object identified within the image. For example, when there are a plurality of identified at least one object, the object identification and classification engine 202 classifies the identified objects into groups of the same or similar category, thereby identifying a relationship between the objects. The information indicating the relationship between the identified objects may include a degree of similarity between categories of objects classified into groups, and the number of objects included in the group. Also, the information indicating the relationship between the identified objects may include relationship information between the main object and the sub-objects and relationship information between the sub-objects. However, the present invention is not limited thereto. The electronic device 100 according to an embodiment may acquire a virtual object to be added to an image and a trajectory of the virtual object based on the identified objects and classification information of the objects.

In an embodiment, the electronic device 100 may obtain depth information of a 3D space in which an image is captured by using a depth segmentation engine 204 .

For example, the depth segmentation engine 204 recognizes an object (eg, a person) within an image, and segments the image into a foreground image and a background image based on the recognized object. can do. In this case, the foreground image and the background image may be one or more images having different depths. For example, the depth segmentation engine 204 divides the image into a background image and a first foreground image having different depths, a second foreground image, . . . , can be divided into the N-th foreground image. The depth segmentation engine 204 may obtain segmentation levels of segmented images having different depths as depth information.

In an embodiment, the electronic device 100 may acquire depth information of a 3D space in which an image is captured in another method. For example, the electronic device 100 may acquire depth information from an image captured by using at least two or more cameras. In another example, the electronic device 100 may obtain depth information from an image including depth information captured using an RGB-Depth (RGB-D) camera.

The electronic device 100 according to an embodiment may give perspective to the image and the virtual object by enabling three-dimensional movement when the virtual object moves along a trajectory in the image by using the obtained depth information.

The electronic device 100 according to an embodiment generates a 3D trajectory to move the virtual object in an image by using the acquired trajectory and depth information of the virtual object, and moves the virtual object along the generated 3D trajectory. can

In an embodiment, the electronic device 100 may obtain context information of an image using the context analyzer 206 and analyze the obtained context information. The context information may include, for example, at least one of a pose, emotion, action, scene of an image, an event in the image, and a location at which the image is captured, of an object in the image, but is not limited thereto. The context analyzer 206 may identify at least context information existing in the image by analyzing the image using the context information acquisition model among the artificial intelligence models 208 . The electronic device 100 according to an embodiment may acquire a virtual object and a trajectory of the virtual object based on a result of analyzing the identified context information.

The electronic device 100 according to an embodiment may analyze the identified context information and combine at least a portion of the identified context information in various ways to identify a virtual object and a trajectory of the virtual object.

In an embodiment, the electronic device 100 may identify a pose of an object in an image and analyze the identified pose to identify an action that the object in the image is taking. The electronic device 100 may identify a virtual object related to the action of the object and a trajectory of the virtual object based on the action of the object.

In an embodiment, the electronic device 100 may identify a scene/event in an image, and analyze the identified scene and event to identify a theme of the image. The electronic device 100 may identify a virtual object related to the theme of the image and a trajectory of the virtual object based on the theme of the image.

In an embodiment, the electronic device 100 identifies a place (eg, a travel destination, a restaurant, etc.) where an image was captured, and based on the identified place where the image was captured, a virtual object related to the place where the image was captured and a trajectory of the virtual object may be identified.

In the above, embodiments in which the electronic device 100 identifies and analyzes context information to identify a virtual object and a trajectory have been described. However, this should be understood only as an exemplary embodiment, and the electronic device 100 may use at least a portion of the context of an image or Based on the combination, it is possible to determine the virtual object and trajectory to be added to the image.

In one embodiment, one or more artificial intelligence models 208 may include various artificial intelligence models trained to perform a particular action.

For example, the AI models 208 may include a model for identifying an object in an image, a model for analyzing the context of an image, a model for determining a virtual object to be added in the image, and a model for identifying a trajectory of the virtual object. It may be a model, an artificial intelligence model for recommending a virtual object and a trajectory to a user, but is not limited thereto.

In another example, the one or more AI models 208 may be an AI model that is implemented as a single AI model and trained to perform all of the various operations and functions described above.

In operation S310, the electronic device 100 according to an embodiment may acquire an image. In an embodiment, the electronic device 100 may acquire an image by photographing it using a camera. In this case, there may be more than one camera, and may be an RGB-D camera capable of acquiring depth information together. In an embodiment, the electronic device 100 may obtain an image from an external device using the communication unit 120 .

In operation S320, the electronic device 100 may identify at least one object in the image and classify the identified object.

In an embodiment, the electronic device 100 may identify at least one object in the image by applying the obtained image to the artificial intelligence model. When there are a plurality of objects in the image, the electronic device 100 may classify the identified objects into a main object to which the virtual object is to be augmented and sub-objects for determining the virtual object.

The electronic device 100 may generate an object tag 322 by assigning a unique tag (eg, object identification information, category information, etc.) to each of the objects identified in the image.

In an embodiment, the electronic device 100 may identify a relationship between the objects by classifying the identified objects into groups of the same or similar category.

In operation S330 , the electronic device 100 may acquire depth information 332 by performing depth division on the acquired image.

In an embodiment, the electronic device 100 may divide the image into a foreground and a background based on at least one object identified in the acquired image. In this case, the foreground image and the background image may be one or more images having different depths. For example, the depth segmentation engine 204 divides the image into a background image and a first foreground image having different depths, a second foreground image, . . . , can be divided into the N-th foreground image. The depth segmentation engine 204 may obtain segmentation levels of segmented images having different depths as depth information 332 .

However, the method by which the electronic device 100 acquires the depth information 332 is not limited thereto, and the electronic device 100 acquires the depth information 332 from an image photographed using at least two or more cameras, or Depth information 332 may be obtained from an image including depth information photographed using an RGB-Depth (RGB-D) camera.

In operation S340 , the electronic device 100 may acquire a virtual object to be added to the image. In this case, the virtual object may be an object for applying an animation effect to the main object identified in step S320. In addition, the virtual object may be an object for imparting an animation effect to another region in the image obtained by the identified main object.

In an embodiment, the electronic device 100 may determine the virtual object 352 to be added to the image based on the object tags of the main object and the sub-objects identified in operation S320 . The electronic device 100 may determine the virtual object 352 from among a plurality of virtual object candidates stored in the memory of the electronic device 100 or in an external device (eg, a server).

In an embodiment, the electronic device 100 determines an object similar to the main object as the virtual object 352 based on the object tag of the main object, obtains the determined virtual object 352, or an animation effect suitable for the main object An object to which can be assigned is determined as a virtual object 352 , and the determined virtual object 352 may be obtained.

In an embodiment, the electronic device 100 determines an object similar to the sub-object as the virtual object 352 based on the object tag of the sub-object, obtains the determined virtual object 352, or stores the object tag of the sub-objects. Based on the virtual object 352 , an object capable of imparting an animation effect suitable for the main object may be determined and the determined virtual object 352 may be obtained.

In an embodiment, the main object may not exist in the image acquired by the electronic device 100 . In this case, the electronic device may determine the virtual object 352 to be added to at least a partial area in the image based on the context information of the image.

In another example, the electronic device 100 determines a virtual object 352 to be added to the image based on a combination of at least two or more of the object tag of the main object, the object tag of the sub-object, and context information of the image, and the determined virtual An object 352 may be obtained.

The obtained virtual object 352 may be added to an image to give an animation effect to a main object in the image, or may be used to create an augmented image that animates at least some regions in the image.

In operation S350 , the electronic device 100 according to an embodiment may obtain a trajectory 354 indicating a movement path of the virtual object 352 . In this case, the trajectory 354 may be a trajectory revolving around the main object based on the identified main object. However, the present invention is not limited thereto, and the trajectory 354 may be a trajectory indicating a relative movement path of the virtual object 352 with respect to the main object with respect to the main object.

In an embodiment, the electronic device 100 may obtain a user input guiding a movement path of the virtual object 352 from the user. For example, the electronic device 100 obtains a user input for drawing a trajectory of a virtual object on an input device (eg, a touch screen, etc.) of the electronic device 100 , and uses the obtained user input as the trajectory of the virtual object. (354) can be determined. In another example, the electronic device 100 may acquire the trajectory 354 from the user's other external device (eg, a smart watch, a stylus pen, etc.). In this case, the other external device may include a plurality of sensors (motion sensor, gyro sensor, etc.). The user's other external device may identify an operation in which the user draws a trajectory of the virtual object 352 using another external device, generate a trajectory corresponding to the identified operation, and transmit the generated trajectory to the electronic device 100 . In another example, the electronic device 100 may obtain the trajectory 354 indicating the movement path of the virtual object by using the artificial intelligence model learned to generate the trajectory corresponding to the obtained virtual object 352 . .

In the disclosed embodiments, the obtained trajectory 354 of the virtual object includes depth information and may be a trajectory corresponding to a movement path of the virtual object in a 3D space. Also, the obtained trajectory 354 of the virtual object may be a trajectory corresponding to a movement path of the virtual object on a two-dimensional plane.

In operation S360 , the electronic device 100 according to an embodiment may move the obtained virtual object 352 on the trajectory 354 . In this case, the electronic device 100 may move the virtual object 352 by using the depth information 332 together.

In an embodiment, the electronic device 100 may move the virtual object 352 along the trajectory 354 in the image. While the electronic device 100 moves the virtual object 352 along the trajectory, the size of the virtual object 352, the direction of the virtual object 352 with respect to the main object, the position of the virtual object with respect to the main object, and the virtual object At least one of the perspective views of 352 may be changed.

In an embodiment, when the virtual object 352 moves on the trajectory 354 , the electronic device 100 moves the virtual object 352 to a deeper region of the three-dimensional space in the image based on the two-dimensional plane of the image. Adjust the size of the virtual object 352 to become smaller in order to express (352) can be adjusted to increase the size.

In an embodiment, when the virtual object 352 moves on the trajectory 354 , the electronic device 100 displays the direction of the virtual object 352 to represent that animation effects are applied based on the main object. and position can be adjusted. For example, the electronic device 100 may adjust the direction of the virtual object 352 with respect to the main object and the position of the virtual object with respect to the main object for each image corresponding to every frame of a moving picture in which the virtual object moves. .

In an embodiment, the electronic device 100 may provide only an animation effect without moving the virtual object 352 even if the trajectory 354 does not exist or even if the trajectory 354 exists.

In operation S370 , the electronic device 100 according to an embodiment may store the augmented image including the obtained virtual object 352 . The electronic device 100 may generate a video in which the virtual object 352 moves on the trajectory 354 by generating a plurality of images in which the obtained virtual object 352 is augmented. In this case, the moving picture in which the virtual object 352 moves on the trajectory 354 may be composed of images in which respective frames of the moving picture are augmented.

Referring to FIG. 4 , the electronic device 100 according to an embodiment may obtain a plurality of images or an input 402 in the form of a video including the plurality of images as frames.

In operation S404, the electronic device 100 according to an embodiment may perform video analysis. The electronic device 100 may obtain the input 402 and perform object identification, object classification, and context analysis. The video analysis operation of step S404 may correspond to the function and operation of the object controller 150 described with reference to FIG. 2 . Input 402 may be in the form of video and images. Hereinafter, for convenience of description, an image will be used as a reference. The electronic device 100 according to an embodiment may generate a list of data required to identify a virtual object and generate a trajectory based on the object identification, object classification, and context analysis results.

In operation S406, the electronic device 100 according to an embodiment may generate a list of information obtained by performing the video analysis of operation S404. For example, the electronic device 100 may obtain a list of main objects, sub-objects, virtual object candidates to be added to the image, and actions related to the identified objects identified in the image. Also, the electronic device 100 may identify motions of virtual objects related to objects in the image and scenes based on the analysis result of context information acquired in the image. In this case, the identified movement of the virtual object may be a general movement of the virtual object determined by the electronic device 100, and the virtual object is created using information such as the location and size of the main object, sub-object, and scene in the image. It may be a movement that allows a natural movement within the image.

In operation S408, the electronic device 100 may obtain metadata about a main object, a sub-object, virtual object candidates, a trajectory point, and a scene from the list generated in operation S406. In this case, the trajectory point may be coordinates in space in the image corresponding to the trajectory indicating the movement path of the virtual object, and the scene metadata may indicate the characteristics of the image scene, such as the subject, atmosphere, season, and time of the image scene. It may be information, but is not limited thereto.

In step S410 , the electronic device 100 may apply the acquired data to the artificial intelligence model. The electronic device 100 may apply at least a portion of the virtual object candidates, the trajectory point, and the metadata about the scene to the artificial intelligence model 208 . The electronic device 100 may perform trajectory clustering using the artificial intelligence model 208 . The electronic device 100 may identify candidates of trajectories on which the virtual object can move through trajectory clustering, and perform scene-to-virtual object analysis of the image to identify trajectories on which the virtual object moves in the image from among the candidate trajectories. The electronic device may determine the identified trajectory as a contextual trajectory suitable for a scene, and generate identification information of the contextual trajectory.

In operation S412, the electronic device 100 according to an embodiment may generate a context connection tree and a scene-virtual object connection tree.

In an embodiment, the electronic device 100 may generate a context connection tree indicating a connection relationship between the main object and the sub-objects.

The electronic device 100 according to an embodiment may identify at least one of a relationship between a main object and sub-objects, a relationship between sub-objects, an ROI of an image, and obtained context information in an image. In this case, the relationship between the objects indicates the relevance of the identified objects in the image based on whether the identified object exists in the region of interest, the similarity of the categories of the identified objects, the relevance of the identified objects with context information, and the like. It may be a score, but is not limited thereto.

The electronic device 100 indicates a connection relationship between the main object and the sub-objects based on at least one of the identified relationship between the main object and the sub-objects, the relationship between the sub-objects, the region of interest of the image, and the obtained context information. You can create a context-connected tree.

The connection relationship between the main object and the sub-object is, for example, connected when the main object and the sub-object belong to the same or similar category and thus has a high relevance between objects, and the main object and the sub-object belong to different categories, so the relevance between objects is high. If not, it may not be connected.

However, the method by which the electronic device 100 generates the context connection tree is not limited thereto, and when the electronic device 100 creates the context connection tree, the electronic device 100 indicates the connection relationship between the main object and the sub-object based on various criteria. can create

The electronic device 100 may determine a virtual object to be added to the image based on the connection relationship between the main object and the sub-objects. For example, the electronic device 100 may determine, as a virtual object, an object belonging to the same or similar category to the main object and sub-objects connected to the main object, based on the main object and sub-objects connected to the main object. . However, the present invention is not limited thereto.

In an embodiment, when an object related to the main object is identified as a virtual object by determining the virtual object based on the category of the main object, the electronic device 100 may connect the main object and the virtual object. The electronic device 100 according to an embodiment may identify a contextual trajectory corresponding to a main object and a virtual object connected to the main object. In this case, the contextual trajectory may be identified based on the contextual trajectory identification information output using the artificial intelligence model 208 .

In an embodiment, the electronic device 100 may generate a scene-virtual object connection tree indicating a connection relationship between a scene of an image and a virtual object. The connection relationship between the scene and the virtual object is, for example, connected when the scene of the image and the virtual object to be added to the image belong to the same or similar category and are highly related, and the scene of the image and the virtual object to be added to the image are in different categories If it belongs to and is not highly related, it may not be connected. However, the present invention is not limited thereto, and the scene-virtual object connection tree may be set in various ways to indicate a connection relationship between the scene and the virtual object.

The electronic device 100 may determine a virtual object to be added to the image based on the connection relationship between the scene and the virtual object. For example, the electronic device 100 may determine a virtual object connected to a scene as a virtual object to be added to the image. However, the present invention is not limited thereto. In this case, since the determined virtual object is not related to the main object, but to the scene of the image, unlike the above-described embodiment, the virtual object may not be connected to the main object. The electronic device 100 according to an embodiment may identify a contextual trajectory corresponding to the determined virtual object. In this case, the contextual trajectory may be identified based on the contextual trajectory identification information output using the artificial intelligence model 208 .

5 is a flowchart illustrating a method for an electronic device to determine a virtual object and a trajectory by using object identification information, image depth information, and context information, according to an embodiment.

In operation S510 , the electronic device 100 according to an embodiment may obtain an input 502 and perform object identification, object classification, and context analysis. Step S510 may correspond to step S404 of FIG. 4 .

In operation S520, the electronic device 100 may obtain information for determining a virtual object and a trajectory based on the result of operation S510. For example, the electronic device 100 may obtain a main object, a sub-object, virtual object candidates to be added to the image, actions related to the identified objects, trajectory candidates, and scene metadata identified in the image. In this case, the scene metadata may be information capable of representing characteristics of an image scene, such as a subject, atmosphere, season, and time of the image scene, but is not limited thereto.

In operation S530, the electronic device 100 may determine a trajectory by applying the acquired data to the AI model. For example, the electronic device 100 may input the main object, sub-object, virtual object candidates, and trajectory candidates to the AI model 208 to obtain trajectory clusters for each of the virtual object candidates. In this case, the artificial intelligence model 208 may be a model trained to output a trajectory suitable for each virtual object.

For example, the cluster of candidate trajectories corresponding to the first candidate virtual object may be T1, and the cluster of candidate trajectories corresponding to the second candidate virtual object may be T2. The electronic device 100 may analyze the obtained trajectory clusters and determine the trajectory of the candidate virtual object within each trajectory cluster. The electronic device 100 may analyze a relationship with a main object or a sub-object for each of the candidate virtual objects. For example, the electronic device 100 may determine whether the candidate virtual object has high relevance to the main object or whether the candidate virtual object has high relevance to the sub-object.

In operation S540, the electronic device 100 may select at least some of the main object and the sub-object. The method for the electronic device 100 to select an object includes a method of selecting only a main object, a method of selecting only a sub-object, a method of selecting using scene metadata, a method of selecting using preset conditions, and other methods. and the like, and is not limited to any one method.

The electronic device 100 may select a virtual object corresponding to the selected object from among the candidate virtual objects. For example, when the electronic device 100 selects the main object, the virtual object may be an object corresponding to the main object from among the candidate virtual objects. As another example, when the electronic device 100 selects a sub-object, the sub-object may be an object corresponding to the sub-object from among the candidate virtual objects.

The electronic device 100 according to an embodiment may augment an image by using a selected object (a main object or a sub-object), a selected virtual object, and a trajectory determined for the selected virtual object. Also, the electronic device 100 may train the artificial intelligence model 208 by using the object selected with respect to the input image and the selected virtual object as training data. When a specific image is input, the electronic device 100 may recommend an object selected by the electronic device 100 to augment the image, a virtual object corresponding to the selected object, and a trajectory corresponding to the virtual object. Intelligence models can be trained.

In an embodiment, the electronic device 100 may acquire an image 602 . The image 602 may be received from an external device or obtained by photographing the electronic device 100 through a camera.

In an embodiment, the electronic device 100 may identify one or

more objects

604 and 606 within the image 602 . The electronic device 100 may classify the identified

objects

604 and 606 . For example, based on a result of analyzing the image 602 , the electronic device 100 identifies that a main subject is a person when capturing an image, classifies the identified first object 604 'person' as a main object, and , the identified second object 606 'butterfly printed on a person's clothes' may be classified as a sub-object.

In an embodiment, the electronic device 100 may acquire depth information by performing depth division on the image 602 . The electronic device 100 may divide the image 602 into a foreground image 608 and a background image 610 based on the object recognized in the image 602 . Although the foreground image 608 and the background image 610 are shown as one each in FIG. 6 , the foreground image 608 or the background image 610 may be one or more images each having different depths. The electronic device 100 may acquire depth information corresponding to the foreground image 608 as first depth information and depth information corresponding to the background image 610 as second depth information.

In an embodiment, the electronic device 100 may acquire a virtual object 612 to be added to the image 602 .

For example, the electronic device 100 may acquire the virtual object 612 to be added to the image 602 through an external device or a manual input of a user.

In another example, the electronic device 100 may determine the virtual object 612 to be added to the image based on the main object 604 , the sub-object 606 , context information of the image, or a combination thereof. Specifically, the electronic device may determine a 'butterfly' similar to the sub-object 606 as the virtual object 612 based on the sub-object 606 .

In an embodiment, the electronic device 100 may obtain a trajectory cluster for the virtual object 'butterfly'. For example, the electronic device 100 bounces, flies, rotates, or moves (eg, from the left side) to a cluster of candidate trajectories corresponding to the virtual object 612 'butterfly'. to the right or from top to bottom), etc. The electronic device 100 determines a trajectory corresponding to the virtual object 612 in the cluster of obtained candidate trajectories based on the main object 604 , the sub-object 606 , context information of the image, or a cluster analysis result, etc. can

The electronic device 100 according to an embodiment obtains the virtual object 612 and the trajectory of the virtual object, and renders the plurality of virtual objects on the plurality of images to generate a plurality of augmented images, thereby the main object 604 . It is possible to generate an augmented video in which the virtual object 612 moves along a trajectory around the .

The electronic device 100 according to an embodiment may control visibility of the virtual object moved in the image 602 based on the acquired depth information and the virtual object. For example, the electronic device 100 may move the obtained virtual object 612 to rotate around the main object 604 based on the main object 604 . In this case, the electronic device 100 overlays the main object 604 on the virtual object 612 to represent a state in which the virtual object 612 is closer than the main object 604 based on the acquired depth information. Thus, the virtual object 612 may be displayed. In addition, in order to represent a state in which the virtual object 612 is farther than the main object 604 (ie, when it is behind the main object 604 ), the electronic device 100 determines that at least a part of the virtual object 612 is Visibility of the virtual object 612 may be controlled by controlling the main object 604 so that it is not displayed.

In an embodiment, when the virtual object 612 moves on the trajectory, the electronic device 100 determines the size of the virtual object 612, the direction of the virtual object with respect to the main object, the position of the virtual object with respect to the main object, and the virtual object. at least one of the perspective views of This will be further described with reference to FIG. 7 .

Referring to FIG. 7 , the electronic device 100 according to an embodiment may obtain trajectory(s) indicating a movement path of the virtual object 710 . The trajectory indicating the movement path of the virtual object 710 is the coordinates of all points on the movement path of the virtual object 710 from the point where the movement of the virtual object 710 starts to the point where the movement of the virtual object 710 ends. It may be a set of these, but is not limited thereto. For example, the trajectory of the virtual object 710 may be a set of vectors indicating the distance and direction in which the virtual object 710 moves within one image frame for each of the augmented image frames included in the augmented video. have.

For convenience of explanation, a trajectory of the virtual object 710 will be described below by dividing it into a first trajectory 720 and a second trajectory 722 . In this case, the first trajectory is a trajectory indicating a movement path in which the virtual object 710 rotates forward of the main object 730 , and the second trajectory indicates that the virtual object 710 rotates backward of the main object 730 . It may be a trajectory indicating a moving path. As shown in FIG. 7 , the virtual object 710 may change in size, direction, position, perspective, and the like as it moves along a trajectory in the image.

In an embodiment, when the virtual object 710 moves on the first trajectory 720 , the electronic device 100 determines the size of the virtual object 710 , the direction of the virtual object 710 with respect to the main object 730 , At least one of a position of the virtual object with respect to the main object and a perspective view of the virtual object 710 may be changed.

For example, when the virtual object 710 moves along the first trajectory 720 , the electronic device 100 causes the virtual object 710 to move in front of the main object 730 and to be expressed as being closer to the main object 730 . For this purpose, the size of the virtual object 710 may be adjusted to be larger. The electronic device 100 may adjust the position and direction of the virtual object 710 so that the virtual object 710 is expressed as if it rotates in front of the main object 730 while looking at the main object 730 . Also, the electronic device 100 may adjust a perspective view of the virtual object 710 in order to give a lively effect to the movement of the virtual object 710 .

In an embodiment, the electronic device 100 may use the main object, the virtual object, and the depth information when adjusting the perspective view. The electronic device 100 may change based on depth information, the direction of the virtual object 710 with respect to the main object 730 , the position of the virtual object with respect to the main object, and the distance from the main object to the virtual object 710 . have.

In an embodiment, when the virtual object 710 moves on the second trajectory 722 , the electronic device 100 determines the size of the virtual object 710 , the direction of the virtual object 710 with respect to the main object 730 , At least one of a position of the virtual object with respect to the main object, a perspective view and visibility of the virtual object 710 may be changed.

For example, when the virtual object 710 moves along the second trajectory 722 , the electronic device 100 causes the virtual object 710 to move backward of the main object 730 and be expressed as if it were farther away. For this purpose, the size of the virtual object 710 may be adjusted to be smaller. The electronic device 100 may adjust the location and direction of the virtual object 710 so that the virtual object 710 is expressed as if it rotates backward of the main object 730 while looking at the main object 730 . The electronic device 100 may adjust a perspective view of the virtual object 710 in order to give a lively effect to the movement of the virtual object. In this case, the perspective view may be changed based on depth information, the direction of the virtual object 710 with respect to the main object 730 , the position of the virtual object with respect to the main object, and the distance from the main object to the virtual object 710 . .

In addition, the electronic device 100 controls the visibility of the virtual object 710 to represent that the virtual object 710 is obscured by the main object 730 as it moves to the rear of the main object 730 . At least a portion of the virtual object 710 may be controlled so that it is not visible because it is obscured by the main object 730 .

In an embodiment, the electronic device 100 determines a size of the virtual object 710 and a direction of the virtual object 710 with respect to the main object 730 based on at least one of a size, a location, and a category of the main object 730 . , a position of the virtual object with respect to the main object, and at least one of a perspective view of the virtual object 710 may be determined.

For example, as shown in FIG. 7 , when the main object 730 is classified as a category 'person' and the virtual object 710 is classified as 'butterfly', the electronic device 100 sets the main object 730 to 'Butterfly'. The size of the virtual object 710 may be determined based on a ratio of the actual size of the virtual object 710 to . Also, based on the location of the main object 730 , the electronic device 100 may adjust the direction and location of the virtual object 710 so that the virtual object 710 is naturally displayed in the image.

In an embodiment, the electronic device 100 determines the size of the virtual object 710 , the direction of the virtual object 710 with respect to the main object 730 , the location of the virtual object with respect to the main object, and the virtual object 710 . at least one of the perspective views of

The electronic device 100 according to an embodiment may acquire a trajectory corresponding to the movement path of the virtual object based on a user input of touching an image displayed on the display of the electronic device. In this case, the trajectory may be a trajectory of a virtual object indicating a movement path relative to the main object. For example, the electronic device 100 may obtain a first trajectory 820 from the user's first touch input 810 and acquire a second trajectory 822 from the second touch input 812 . However, the present invention is not limited thereto, and the electronic device 100 may obtain a trajectory including the first trajectory 820 and the second trajectory 822 from the user's one-time touch input.

In an embodiment, the electronic device 100 may perform depth division on an image, divide the image into a foreground image 830 and a background image 832 , and obtain depth information. Since this has been described in the above-described embodiments, a description thereof will be omitted.

The electronic device 100 may move the virtual object based on the first trajectory 820 and the second trajectory 822 obtained from the user's input. In this case, depth information obtained from the foreground image 830 and the background image 832 may be used. Since the method of the electronic device 100 moving the virtual object along the trajectory has been described in the above-described embodiments, the same description will be omitted.

Referring to FIG. 9 , the electronic device 100 according to an embodiment may acquire various images, and acquire a virtual object corresponding to the acquired image and a trajectory of the virtual object.

In an embodiment, the electronic device 100 may acquire the first image 910 . The first image 910 may be an image of 'a child holding a toy'. According to the above-described embodiments, the electronic device 100 may identify objects 'child' and 'toy' in the first image 910 . The electronic device 100 may determine the first virtual object 912 to be added to the first image 910 based on at least a part of object tags of objects identified in the image and context information of the image. For example, the electronic device 100 may acquire a 'doll toy' as a first virtual object 912 to be augmented in a first image 910 that is an image of a 'child holding a toy'.

Also, the electronic device 100 may acquire a first trajectory 914 on which the first virtual object 912 moves within the first image 910 . In an embodiment, the first trajectory 914 may be obtained based on a mapping table pre-stored in the electronic device 100 or an external device (eg, a server). In another embodiment, the first trajectory 914 may be obtained from a user input to the electronic device 100 or a user input using an external device (eg, a smart watch or a stylus pen). In another embodiment, the first trajectory 914 may be obtained based on an artificial intelligence model trained to output a trajectory corresponding to the object. The first trajectory 914 may be output from the AI model by inputting the first object 912 into the AI model.

In an embodiment, the electronic device 100 may acquire the second image 920 . The second image 920 may be an image of 'a man near the sea'. The electronic device 100 may identify an object in the second image 920 according to the above-described embodiments. The electronic device 100 may determine the second virtual object 922 to be added to the second image 920 based on at least a part of an object tag of an object identified in the image and context information of the image. For example, the electronic device 100 may acquire 'dolphin' as the second virtual object 922 to be augmented in the second image 920 that is an image of 'a man near the sea'.

Also, the electronic device 100 may acquire a second trajectory 924 on which the second virtual object 922 moves within the second image 920 . Since this has been described in the embodiment in which the electronic device 100 acquires the trajectory of the first object of the first image, a description thereof will be omitted.

In an embodiment, the electronic device 100 may acquire a third image 930 . The third image 930 may be an image of 'a girl near the beach'. The electronic device 100 may identify an object in the third image 930 according to the above-described embodiments. The electronic device 100 may determine a third virtual object 932 to be added to the third image 930 based on at least a part of an object tag of an object identified in the image and context information of the image. For example, the electronic device 100 may acquire the 'sailboat' as the third virtual object 932 to be augmented in the third image 930 that is an image of the 'girl near the beach'.

Also, the electronic device 100 may acquire a third trajectory 934 on which the third virtual object 932 moves within the third image 930 . Since this has been described in the embodiment in which the electronic device 100 acquires the trajectory of the first object of the first image, a description thereof will be omitted.

The electronic device 100 according to an embodiment may augment the images based on the acquired virtual object and the trajectory of the virtual object for each of the images. For example, the electronic device 100 adds a first virtual object 912 to a first image 910 that is an image of a 'child holding a toy', and creates the first virtual object 912 along a first trajectory. By moving and applying an animation effect, the first image 910 may be augmented. In another example, the electronic device 100 adds a third virtual object 932 to a third image 930 that is an image of a 'girl near the beach', and adds the third virtual object 932 along the third trajectory. By moving and applying an animation effect, the third image 930 may be augmented.

Referring to FIG. 10 , the electronic device 100 according to an embodiment may obtain various information to identify a virtual object to be added to an image.

In an embodiment, the electronic device 100 may identify objects in an image and classify the identified objects. The electronic device 100 may classify the identified objects into a main object and a sub-object, and obtain an object tag of each object. The object tag is information for identifying each object, and may include object identification information, category information, and the like, but is not limited thereto.

For example, the electronic device 100 may identify a 'toy' and a 'child' in the first image 1010 that is a 'child holding a toy'. In this case, the electronic device 100 may classify 'child' as a main object and classify 'toy' as a sub-object. However, the present invention is not limited thereto, and the electronic device 100 may classify 'toy' as a main object and classify 'child' as a sub-object.

In an embodiment, the electronic device 100 may identify a relationship between the objects by classifying the identified objects into groups of the same or similar category. For example, the electronic device 100 may classify 'child' and 'toy' into a 'kids' group, and identify objects in the image as objects related to the 'kids' category.

In an embodiment, the electronic device 100 may obtain context information of an image. The context information may include, for example, at least one of a pose, emotion, action, scene of an image, an event in the image, and a location at which the image is captured, of an object in the image, but is not limited thereto.

In an embodiment, the electronic device 100 may identify the pose of the main object in the image to identify the action the main object is taking in the image. For example, the electronic device 100 classifies the 'child' as a main object in the first image that is the 'child holding a toy', identifies the child's pose, and takes the action of the child holding the toy. It can be identified that there is Also, the electronic device 100 may identify the child's current emotion, 'happiness' by analyzing the child's expression. The electronic device 100 according to an exemplary embodiment includes an object tag and an object of the objects identified in the image. A virtual object to be added in the image may be determined based on at least a part of a relationship between the two and the context information of the image. In this case, the virtual object determined by the electronic device 100 may be an object similar to the object identified in the image, or an object that is not similar to the object identified in the image, but suitable for the subject, atmosphere, etc. of the image.

For example, the electronic device 100 performs object identification and image analysis on a first image 1010 that is an image of a 'child holding a toy', and the object of the identified objects in the first image 1010 . You can acquire tags, relationships between objects, and context information of images. As a first virtual object 1012 to be added to the first image 1010 , the electronic device 100 sets a 'doll toy', which is an object similar to a 'toy' existing in the first image 1010 , to the first object 1012 . ) can be determined.

In another example, the second image 1020 may be 'a man near the sea'. The electronic device 100 performs object identification and image analysis on the second image 1020 to obtain object tags of objects identified in the second image 1020, relationships between objects, and context information of images. can The electronic device 100 is a second virtual object 1022 to be added to the second image 1020, which is not similar to 'man', which is an object existing in the second image 1020 , but has 'the sea' as a background. Objects suitable for the second image 1020 , such as 'dolphins' and 'sailboats', may be determined as the second object 1022 .

In another example, the third image 1030 may be a 'flower'. The electronic device 100 performs object identification and image analysis on the third image 1030 to obtain object tags of objects identified in the second image 1030, relationships between objects, and context information of images. can The electronic device 100 is a third virtual object 1032 to be added to the third image 1030 , which is not similar to 'flower', which is an object existing in the third image 1030 , but is a third image of 'flower'. 'Butterfly' and 'bee', which are objects suitable for the 1030 , may be determined as the third object 1032 .

Referring to FIG. 11 , the electronic device 100 according to an embodiment may acquire an image 1110 of a 'basketball player holding a basketball ball'.

In an embodiment, the electronic device 100 may identify the region of interest 1120 within the acquired image 1110 . The electronic device 100 may identify the ROI 1120 by analyzing the image 1110 , but may also identify the ROI 1120 based on a user's input for selecting the ROI. For example, the electronic device 100 may use an external device (eg, a stylus pen) 1130 to identify the 'basketball ball' as the region of interest based on the region where the user draws a circle in the image. have.

In an embodiment, the electronic device 100 may identify an object in the ROI and acquire the virtual object 1150 based on the identified object in the ROI. For example, the electronic device 100 may identify a 'basketball ball' that is an object in the ROI as the virtual object 1150 . However, the present invention is not limited thereto, and the electronic device 100 may identify an object similar to a basketball or an object suitable for basketball as the virtual object 1150 based on a 'basketball ball' that is an object in the region of interest. For example, the electronic device 100 may identify the 'basketball ball' as the virtual object 1150, but is not limited thereto.

In an embodiment, the electronic device 100 may acquire the trajectory 1140 of the virtual object 1150 . The electronic device 100 may receive an input in which a user draws a trajectory 1140 on the touch screen of the electronic device 100 using the external device 1130 . However, the present invention is not limited thereto, and the electronic device 100 allows a user to draw a trajectory 1140 in an empty space using an external device 1130 including various sensors (eg, a motion sensor, a gyro sensor, etc.). It can also receive input.

The electronic device 100 according to an embodiment may move the virtual object 1150 in the image based on the obtained virtual object 1150 and the trajectory 1140 .

Referring to FIG. 12 , the electronic device 100 according to an embodiment may acquire an image 1202 obtained by photographing a skydiving activity.

In an embodiment, the electronic device 100 may determine the virtual object to be augmented in the image 1202 and the trajectory of the virtual object based on the object tags of the objects identified in the image 1202 and context information of the image. Since the method for the electronic device 100 to determine the virtual object and the trajectory of the virtual object has been described in the above-described embodiments, a description thereof will be omitted.

In one embodiment, the electronic device 100 receives a user's input (eg, the user's gesture 1206, the user's touch input 1208) to augment the image 1202, A user interface 1204 that recommends virtual objects and trajectories determined as candidates to be added to the image according to the above information may be displayed. For example, the electronic device 100 may determine a helicopter, a parachute, a person in skydiving, etc. as a virtual object candidate to be added to the skydiving image 1202 and recommend it to the user. Also, the electronic device 100 may determine a trajectory of a rectangle, a circle, a straight line, a reciprocating arrow, etc. as a trajectory candidate of the virtual object and recommend it to the user. In this case, the trajectory candidates may indicate a movement path in a 3D space including depth information.

In an embodiment, the electronic device 100 may recognize the user's gesture 1206 and move the virtual object added to the image or modify the trajectory of the virtual object based on the recognized gesture.

The electronic device according to an embodiment may acquire a virtual object to be added to an image and a trajectory of the virtual object, and change at least one of a size, a direction, a position, and a perspective view of the virtual object while moving the virtual object on the trajectory. Since this has been described through the above-described embodiments, a description thereof will be omitted herein.

In an embodiment, the electronic device 100 may acquire an image 1310 of a girl on the beach and determine a virtual object to be added to the image. The electronic device 100 may augment the image 1310 by determining the sailboat 1312 as a virtual object with respect to the image 1310 of the girl on the beach and automatically moving the virtual object sailboat 1312 on the trajectory. have.

In an embodiment, the electronic device 100 may acquire an image 1320 of a child holding a toy and determine a virtual object to be added to the image. With respect to the image 1320 of the child holding the toy, the electronic device 100 determines the doll toy 1322 as a virtual object, and automatically moves the virtual object doll toy 1322 on the trajectory to generate the image 1320. can be augmented.

In an embodiment, the electronic device 100 may acquire an image 1330 of a skydiving activity and determine a virtual object to be added to the image. The electronic device 100 receives an input for selecting the helicopter 1332 as the virtual object 1332 with respect to the image 1330 obtained by photographing the skydiving activity, and manually adds the virtual object helicopter 1332 to the image. and the virtual object helicopter 1332 may be moved on the acquired trajectory.

In an embodiment, the electronic device 100 may acquire an image 1340 of a man near the sea and determine a virtual object to be added to the image. The electronic device 100 may augment the image 1340 by determining the dolphin 1342 as a virtual object and moving the virtual object dolphin 1342 on a trajectory with respect to the image 1340 of a man near the sea. . In this case, the path that the virtual object dolphin 1342 moves on the image may be manually set by the user.

In an embodiment, the electronic device 100 may augment the image by adding a virtual object butterfly 1352 to the image and moving the virtual object on the trajectory with respect to the selfie image 1350 of the man. In this case, the butterfly 1352, which is a virtual object, may give an animation effect while turning around a 'man' classified as a main object in the image.

Referring to FIG. 14 , the electronic device 100 according to an embodiment may identify a gesture for augmenting an image of a user. The electronic device 100 may identify that the user makes a gesture for augmenting an image based on whether a preset event (eg, a user's call command, execution of an image augmentation mode, etc.) occurs. In this case, the user's gesture for augmenting the image may be preset and stored in the electronic device 100 , but is not limited thereto.

In an embodiment, when the user makes a gesture 1410 of raising the index finger of his right hand, the electronic device 100 may augment the image by displaying a twinkling animation effect.

In an embodiment, when the user makes a gesture 1420 of raising the index finger and the little finger of the left hand, the electronic device 100 may augment the image by displaying an animation effect in which a rainbow appears in the background.

In operation S1510, the electronic device 100 according to an embodiment may acquire a virtual object to be added to an image.

In an embodiment, the electronic device 100 may acquire a virtual object to be added to an image through an external device or a manual input of a user.

In an embodiment, the electronic device 100 may determine a virtual object to be added to the image based on the main object, the sub-object, context information of the image, or a combination thereof, and obtain the determined virtual object.

In an embodiment, the electronic device 100 may acquire the virtual object based on at least a part of a connection relationship between the main object and sub-objects, a context connection tree, and a scene-virtual object connection tree.

Methods for the electronic device 100 to obtain a virtual object according to an embodiment have been described in the description of the above-described embodiments, and thus the same description will be omitted.

In operation S1520, the electronic device 100 according to an embodiment may acquire a trajectory indicating a movement path of the obtained virtual object.

In an embodiment, the electronic device 100 may acquire the trajectory corresponding to the movement path of the virtual object by using the artificial intelligence model learned to output the trajectory corresponding to the object.

In an embodiment, the electronic device 100 may obtain a trajectory corresponding to the movement path of the virtual object based on a user input (eg, a touch input, a motion input, a gesture input, etc.).

Methods for the electronic device 100 to obtain the trajectory of the virtual object according to an embodiment have been described in the description of the above-described embodiments, and thus the same description will be omitted.

Also, according to an embodiment, the electronic device 100 may acquire a trajectory corresponding to the acquired virtual object based on a mapping table pre-stored in the electronic device 100 or an external device. This will be further described in the description of FIG. 17 .

In operation S1530, the electronic device 100 according to an embodiment may move the virtual object along the trajectory.

The electronic device 100 according to an embodiment moves the virtual object along a trajectory in the image, and the size of the virtual object, the direction of the virtual object with respect to the main object, the position of the virtual object with respect to the main object, and the position of the virtual object with respect to the main object. At least one of the perspective views may be changed. Since the description of how the electronic device 100 moves the virtual object in the image has been described through the above-described embodiments, the same description will be omitted.

In operation S1610, the electronic device 100 according to an embodiment may identify one or more objects in the image.

In an embodiment, the electronic device 100 may identify and classify an object in an image using an artificial intelligence model. In this case, the identified object may be classified into a main object and a sub-object. Since the description of the method for the electronic device 100 to identify the object in the image has been described through the above-described embodiments, the same description will be omitted.

In operation S1620, the electronic device 100 according to an embodiment may identify at least one of information among a relationship between objects, an ROI of an image, and a context of the image. The electronic device 100 may identify a relationship between identified sub-objects from among the identified objects, a relationship between a main object and sub-objects, and identify an ROI of an image and a context of the image.

In operation S1630, the electronic device 100 according to an embodiment may generate a context connection tree based on at least one of a relationship between objects, an ROI of an image, and obtained context information. In this case, the context connection tree may indicate a connection relationship between the main object and sub-objects. Since the description of the method in which the electronic device 100 generates the context connection tree has been described through the above-described embodiments, the same description will be omitted.

In operation S1640, the electronic device 100 according to an embodiment may identify a virtual object to be added to the image in order to augment the image based on the context connection tree. Since the description of the method for the electronic device 100 to identify the virtual object has been described through the above-described embodiments, the same description will be omitted.

In an embodiment, the electronic device 100 may acquire animation effects and trajectories corresponding to the virtual objects based on the mapping table 1700 . In this case, the electronic device 100 may use the mapping table 1700 stored in the electronic device 100 or may use the mapping table 1700 stored in the external device (eg, a server).

Referring to FIG. 17 , the mapping table 1700 may include a list of virtual objects 1710 obtainable by the electronic device and an animation type 1720 corresponding to each virtual object. For example, when the virtual object that the electronic device 100 wants to add to the image is a watch, the electronic device 100 rotates the clock hand, which is an animation corresponding to the virtual object 'clock', based on the mapping table 1700 . An image may be augmented by acquiring an effect and applying the acquired animation effect to the virtual object 'clock'.

Meanwhile, the block diagrams of the electronic device 100 and the object controller 150 illustrated in FIGS. 1 and 2 are block diagrams for an exemplary embodiment. Each component in the block diagram may be integrated, added, or omitted according to the specifications of each device that is actually implemented. That is, two or more components may be combined into one component, or one component may be subdivided into two or more components as needed. In addition, the function performed in each block is for describing the embodiments, and the specific operation or device does not limit the scope of the present invention.

The method of augmenting a virtual object on an image by the electronic device 100 according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

In addition, a method for the electronic device 100 to augment a virtual object on an image according to the disclosed embodiments may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities.

The computer program product may include a S/W program and a computer-readable storage medium in which the S/W program is stored. For example, computer program products may include products (eg, downloadable apps) in the form of software programs distributed electronically through manufacturers of electronic devices or electronic markets (eg, Google Play Store, App Store). have. For electronic distribution, at least a portion of the S/W program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing a SW program.

The computer program product, in a system including a server and the electronic device 100 , may include a storage medium of a server or a storage medium of a client device. Alternatively, when there is a third device (eg, a smart phone) that is communicatively connected to the server or the electronic device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S/W program itself transmitted from the server to the client device or the third device, or transmitted from the third device to the client device.

In this case, one of the server, the electronic device 100 and the third device may execute a computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the electronic device, and the third device may execute the computer program product to distribute the method according to the disclosed embodiments.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims are also included in the scope of the present invention. belongs to

Claims

A method for an electronic device to augment a virtual object on an image, comprising:

identifying one or more objects within the image;

classifying the one or more objects into a main object and a sub-object;

obtaining a virtual object related to the main object based on the main object and the sub-object;

obtaining depth information of a three-dimensional space in which the image is captured;

obtaining a trajectory of the virtual object indicating a three-dimensional movement path relative to the main object, based on the depth information;

moving the virtual object along the trajectory.
According to claim 1,

Moving the virtual object comprises:

changing at least one of a size of the virtual object, a direction of the virtual object with respect to the main object, a position of the virtual object with respect to the main object, and a perspective view of the virtual object.
3. The method of claim 2,

The step of obtaining the virtual object comprises:

identifying at least one of a relationship between identified sub-objects in the image, a relationship between a main object and sub-objects, a region of interest, and a context of the image;

Generating a context connection tree representing a connection relationship between the main object and the sub-objects based on at least one of a relationship between the sub-objects, a relationship between a main object and sub-objects, a region of interest, and a context of the image ; and

and identifying the virtual object based on the context connection tree.
3. The method of claim 2,

obtaining virtual object candidates to be added to the image based on the identified one or more objects; and

obtaining, for each of the virtual object candidates, a trajectory cluster including trajectory candidates corresponding to each of the virtual object candidates;

The step of obtaining the virtual object comprises:

selecting a virtual object to be added to the image from among the virtual object candidates;

The step of obtaining the trajectory is

and selecting a trajectory indicating a movement path of the selected virtual object from within the trajectory cluster corresponding to the selected virtual object.
According to claim 1,

Moving the virtual object comprises:

and controlling visibility of the virtual object based on the depth information.
3. The method of claim 2,

Moving the virtual object comprises:

The method further comprising: changing the perspective view based on the depth information, the orientation of the virtual object with respect to the main object, and the distance of the virtual object from the main object.
According to claim 1,

The step of obtaining the trajectory is

and obtaining a user input guiding a movement path of the virtual object from a user of the electronic device.
An electronic device for augmenting a virtual object in an image, comprising:

communication department;

display;

a memory storing a program including one or more instructions;

at least one processor executing one or more instructions of the program stored in the memory;

the at least one processor,

identify one or more objects within the image;

Classifying the one or more objects into a main object and a sub-object,

obtaining a virtual object related to the main object based on the main object and the sub-object;

Obtaining depth information of the three-dimensional space in which the image was taken,

obtaining a trajectory of the virtual object indicating a three-dimensional movement path relative to the main object based on the depth information;

and controlling the display to indicate movement of the virtual object.
9. The method of claim 8,

the at least one processor,

An electronic device for changing at least one of a size of the virtual object, a direction of the virtual object with respect to the main object, a position of the virtual object with respect to the main object, and a perspective view of the virtual object.
10. The method of claim 9,

the at least one processor,

identify at least one of a relationship between identified sub-objects in the image, a relationship between a main object and sub-objects, a region of interest, and a context of the image;

Based on at least one of a relationship between the sub-objects, a relationship between a main object and sub-objects, a region of interest, and a context of the image, a context connection tree indicating a connection relationship between the main object and the sub-objects is generated,

and identify the virtual object based on the context connection tree.
10. The method of claim 9,

the at least one processor,

obtaining virtual object candidates to be added to the image based on the identified one or more objects;

obtaining, for each of the virtual object candidates, a trajectory cluster including trajectory candidates corresponding to each of the virtual object candidates;

selecting a virtual object to be added to the image from among the virtual object candidates;

and selecting a trajectory indicating a movement path of the selected virtual object from within the trajectory cluster corresponding to the selected virtual object.
9. The method of claim 8,

the at least one processor,

and controlling visibility of the virtual object based on the depth information.
10. The method of claim 9,

the at least one processor,

and changing the perspective view based on the depth information, a direction of the virtual object with respect to the main object, and a distance of the virtual object from the main object.
9. The method of claim 8,

the at least one processor,

Obtaining a user input for guiding a movement path of the virtual object from a user of the electronic device.
A computer-readable recording medium in which a program for executing the method of any one of claims 1 to 7 on a computer is recorded.