KR20210091033A - Electronic device for estimating object information and generating virtual object and method for operating the same - Google Patents

Abstract

Disclosed are an electronic device for estimating object information and generating a virtual object and an operating method of the electronic device. The disclosed operating method of the electronic device includes the steps of: acquiring an image; acquiring class characteristics, posture characteristics, and relational characteristics of objects included in the image; correcting the class characteristics, posture characteristics, and relational characteristics, respectively, by using the class characteristics, posture characteristics, and relational characteristics of the object; and acquiring class information, posture information, and relationship information of the object based on the corrected class characteristics, the corrected posture characteristics, and the corrected relational characteristics, respectively.

Description

An electronic device for estimating object information and generating a virtual object, and an operating method of the electronic device {ELECTRONIC DEVICE FOR ESTIMATING OBJECT INFORMATION AND GENERATING VIRTUAL OBJECT AND METHOD FOR OPERATING THE SAME}

The following embodiments relate to an electronic device for estimating object information and generating a virtual object and an operating method of the electronic device.

Object detection may be a technique for recognizing various objects in an input image. As part of various efforts to improve the accuracy of object recognition, research is being conducted to detect an object included in an image by using the entire information of the image.

According to an embodiment, a method of operating an electronic device includes: acquiring an image; acquiring class characteristics, posture characteristics, and relational characteristics of objects included in the image; correcting the class feature, the posture feature, and the relationship feature, respectively, by using the class feature, the posture feature, and the relation feature of the object; and obtaining class information, posture information, and relationship information of the object based on the corrected class feature, the corrected posture feature, and the corrected relationship feature, respectively.

In the method of operating an electronic device according to an embodiment, the correcting may include applying a predetermined weight to the class feature, the posture feature, and the relation feature of the object, so as to select one of the class feature, the posture feature, and the relation feature. Either one can be corrected.

In the method of operating an electronic device according to an embodiment, the obtaining of the class feature, the posture feature, and the relationship feature may include acquiring each of the class feature, the posture feature, and the relationship feature from an intermediate layer of a corresponding subnetwork. can do.

In the method of operating an electronic device according to an embodiment, intermediate layers of a plurality of sub-networks may be connected to each other so that the class feature, the posture feature, and the relation feature may be shared with other sub-networks.

In the method of operating an electronic device according to an embodiment, the obtaining of the class information, the posture information, and the relationship information of the object corresponds to each of the corrected class feature, the corrected posture feature, and the corrected relationship feature. The class information, the posture information, and the relationship information may be obtained from an output layer of the corresponding subnetwork in response to being input to a next layer of the intermediate layer of the sub-network.

In the method of operating an electronic device according to an embodiment, the class information includes information on which object the object detected in the image is, and the posture information includes information indicating a rotation angle of the object detected in the image. and, the relation feature may include action information of an object detected in the image and/or connection information with another object.

According to an embodiment, a method of operating an electronic device may include determining location information, posture information, and motion information of a virtual object to be created in the image based on class information, posture information, and relationship information of the object; and adding the virtual object to the image based on location information, posture information, and motion information of the virtual object.

In the method of operating an electronic device according to an embodiment, the adding of the virtual object may include adding the virtual object to information selected by the user from among the plurality of pieces of information when at least one of the determined location information, posture information, and operation information for the virtual object is plural. based on the virtual object may be added to the image.

In the method of operating an electronic device according to an embodiment, the location information of the virtual object includes information indicating a position at which the virtual object can be rendered in an image, and the posture information includes information indicating a rotation angle of the virtual object. and the operation information may include information indicating the operation of the virtual object.

In the method of operating an electronic device according to an embodiment, the image may be an RGB-D image.

An electronic device according to an embodiment includes one or more processors, wherein the one or more processors acquire an image, acquire class features, posture features, and relational features of an object included in the image, and obtain the class feature of the object , correcting the class feature, the posture feature, and the relationship feature, respectively, using the posture feature and the relation feature, based on the corrected class feature, the corrected posture feature, and the corrected relation feature, respectively Acquire class information, posture information, and relationship information of an object.

1 is a diagram for describing an electronic device according to an exemplary embodiment.
2 to 4 are diagrams for explaining a process of recognizing an object based on an entire image according to an exemplary embodiment.
5 is a diagram for describing a process of estimating information on an object included in an image in an electronic device according to an exemplary embodiment.
6 is a diagram for explaining mutual correction based on sub-networks according to an embodiment.
7 is a diagram for explaining a process of acquiring object information according to an embodiment.
8 and 9 are diagrams illustrating examples of object information obtained according to an embodiment.
10 is a diagram for describing a process of generating a virtual object based on acquired object information, according to an exemplary embodiment.
11 is a diagram for describing a process of generating a virtual object in an image in an electronic device according to an exemplary embodiment.
12 to 14 are diagrams for explaining a process of generating a virtual object by estimating information on an object included in an image according to an exemplary embodiment.
15 is a diagram illustrating an electronic device according to an exemplary embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the actual implemented form is not limited to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 is a diagram for describing an electronic device according to an exemplary embodiment.

Referring to FIG. 1 , the electronic device 100 may detect an object included in an image to estimate object information, or may generate a virtual object in a corresponding image based on the estimated object information. The image is an image of one or more objects included in one scene, and may be a red-green-blue depth (RGB-D) image including a color image and a depth image. For example, the image may be an image captured by a camera built into the electronic device 100 or an image received from an external camera device. The object information estimated by the electronic device 100 may include class information of an object included in an image, posture information, and relationship information with other objects. The relationship information may be expressed as a scene graph. A scene graph may be expressed as a matrix of size NxN, where N is the number of recognized objects, each row and each column of the matrix corresponds to an object, and each element of the matrix may correspond to a relationship between objects . Also, the electronic device 100 may generate a virtual object suitable for one or more objects based on at least one of class information, posture information, and relationship information of one or more objects included in the image. Hereinafter, it will be described in detail with reference to the drawings.

2 to 4 are diagrams for explaining a process of recognizing an object based on an entire image according to an exemplary embodiment.

Referring to FIG. 2 , an object 210 located in a local area of an image is illustrated by way of example. In some cases, it may be difficult to accurately estimate information about a specific object using only a partial local area because other types of objects having similar shapes may exist or may be obscured by other objects. It is difficult to clearly distinguish whether the detected object 210 is a TV or a picture frame only with the local area exemplarily shown in FIG. 2 . If the entire image shown in FIG. 3 can be considered, it may be possible to more accurately determine the object 310 detected through the analysis of the relationship between the objects included in the entire image. For example, if the information that the object 310 to be estimated is hung on the wall and behind the sofa is used, it can be easily determined that the object 310 is not a TV but a picture frame.

Referring to FIG. 4 , another example of estimating the detected object 410 in consideration of the entire image is illustrated. Considering that the object 410 is located on the left side of the bed and under the back, it can be easily determined that it is a bedside table. In this way, if the relationship with other objects included in the entire image is considered, the accuracy of object detection can be improved.

5 is a diagram for describing a process of estimating information on an object included in an image in an electronic device according to an exemplary embodiment.

Referring to FIG. 5 , a method of estimating object information performed by a processor included in an electronic device is illustrated. The electronic device may recognize the class, posture, and relationship of one or more objects included in the image by using the plurality of subnetworks. Each of the plurality of subnetworks may include one or more convolutional layers and a fully connected layer.

In operation 510, the electronic device may acquire an image. The image may be a color image (eg, an RGB-D image) including depth information. For example, the electronic device may acquire an image through a built-in camera module or receive an image photographed from an external camera device. Also, the image may be obtained from an image collecting device for an augmented reality (AR) device.

In operation 520 , the electronic device may acquire class characteristics, posture characteristics, and relation characteristics of one or more objects included in the image. The class feature is a feature used to determine class information about which object the detected object is specifically, and the posture feature is a feature used to determine posture information indicating a rotation angle of an object in a target map. , and the relationship feature may be a feature used to determine relationship information including action information of the detected object or connection information with another object. For example, 'reading' in 'a person reading a book' may indicate relationship information between a book and a person, and 'hanging' in 'a picture on a wall' may indicate relationship information between a wall and a picture.

The electronic device may input an image to a neural network including a plurality of sub-networks. The plurality of subnetworks may include a class recognition network for class information recognition, a posture recognition network for posture information recognition, and a relationship recognition network for relationship information recognition. A class feature, a posture feature, and a relationship feature may be output from an intermediate layer of each of the plurality of subnetworks. The intermediate layer may correspond to one of one or more hidden layers included in the subnetwork.

In operation 530 , the electronic device may respectively correct the class feature, the posture feature, and the relationship feature by using the class feature, the posture feature, and the relationship feature. A plurality of subnetworks are jointed with each other, so that features output from an intermediate layer of each subnetwork can be exchanged. For example, the electronic device may correct the class feature by using the class feature, the posture feature, and the relation feature. Also, the electronic device may correct the posture characteristic by using the class feature, the posture feature, and the relation feature. In addition, the relational characteristic may be corrected by using the class characteristic, the posture characteristic, and the relational characteristic.

In operation 540, the electronic device may obtain class information of the object based on the corrected class feature. Also, the electronic device may determine the posture information of the object based on the corrected posture characteristics. Also, the electronic device may determine the relationship information of the object based on the corrected relationship feature. By determining object information using different types of features, class information, posture information, and relationship information of an object in consideration of the entire image can be recognized with high accuracy.

In the present specification, for convenience of description, class characteristics and class information may be referred to as category characteristics and category information, respectively, and class information and posture information may be collectively referred to as attribute information.

6 is a diagram for explaining mutual correction based on sub-networks according to an embodiment.

Referring to FIG. 6 , a process in which object information with high accuracy is determined by correcting each feature through data exchange between a plurality of sub-networks 621 , 622 , and 623 included in the neural network 620 is illustrated.

The image 610 may be input to the neural network 620 and recognition may be performed. The neural network 620 may include a class recognition network 621 , a posture recognition network 622 , and a relationship recognition network 623 . A class feature, a posture feature, and a relationship feature may be output from an intermediate layer of the plurality of subnetworks 621 , 622 , and 623 , respectively. The plurality of sub-networks 621 , 622 , and 623 may exchange data with each other, and through this, respective characteristics may be corrected.

For example, the class feature output from the middle layer of the class recognition network 621 is to be corrected using the posture feature output from the middle layer of the posture recognition network 622 and the relationship feature output from the relationship recognition network 623 . can

If the class feature correction is described in more detail, the corrected class feature may be obtained based on the class feature, the posture feature, the relation feature, and a preset first weight coefficient array. The first weighting coefficient array may include a weighting coefficient of a class characteristic, a weighting coefficient of a posture characteristic, and a weighting coefficient of a relational characteristic used in the process of correcting the class characteristic.

If the first weighting coefficient array _{is expressed as [a 11} , a ₁₂ , a ₁₃ ], the corrected class feature may be expressed as follows.

은 보정된 클래스 특징을 나타낼 수 있다.In Equation 1 above, A ₁ may indicate a class characteristic, A ₂ may indicate a posture characteristic, and A ₃ may indicate a relational characteristic. In addition, a ₁₁ may represent a weighting coefficient of a class feature applied to the correction process, a ₁₂ may represent a weighting coefficient of a posture feature applied to the correction process, and a ₁₃ may represent a weighting coefficient of a relational feature applied to the correction process. there is. Also,

may represent the corrected class feature.

Similarly, the corrected posture characteristic may be determined based on the class characteristic, the posture characteristic, the relational characteristic, and the second weighting coefficient array, the corrected relational characteristic being determined based on the class characteristic, the posture characteristic, the relational characteristic and the third weighting coefficient array. can be decided.

The first weighting coefficient array, the second weighting coefficient array, and the third weighting coefficient array may be determined according to the importance of the class characteristic, the posture characteristic, and the relational characteristic in correcting each characteristic.

Accuracy of class information, posture information and relational information of an object is improved through a neural network by obtaining corrected class characteristics, corrected posture characteristics, and corrected relational characteristics through mutual correction between class characteristics, posture characteristics, and relational characteristics can do it Even if mutual correction between different features is performed, the parameters of each sub-network are not changed.

The corrected class feature is input to the next layer of the intermediate layer from which the class feature is output from the class recognition network 621 , and is processed through the remaining layers included in the class recognition network 621 , and is an output layer of the class recognition network 621 . class information may be output from . Similarly, the corrected posture feature is input to the next layer of the intermediate layer of the posture recognition network 622 and processed through the remaining layers included in the posture recognition network 622 , and posture information is obtained from the output layer of the posture recognition network 622 . can be output. In addition, the corrected relationship features are input to the next layer of the intermediate layer of the relationship recognition network 623 and processed through the remaining layers included in the relationship recognition network 623 , and posture information in the output layer of the relationship recognition network 623 . can be output.

The category recognition network 621 , the posture recognition network 622 , and the relationship recognition network 623 are respectively a Convolutional Neural Networks (CNN), a Faster Fast Region-based Convolutional Neural Network (RCNN), and You Only Look Once: Unified (Yolo). Real-Time Object Detection), but the network type is not limited.

According to an embodiment, the neural network 620 may be trained based on a plurality of sample images. The plurality of sample images are training data used for training the neural network 620 , and may include correct answer class information, correct correct posture information, and correct answer relationship information of one or more objects included in each image.

Inference class information, reasoning posture information, and reasoning relationship information may be obtained from the neural network 620 by inputting each of a plurality of sample images in which correct class information, correct posture information, and correct answer relationship information are set to the neural network 620 . . The parameters of the neural network 620 may be adjusted based on the loss between the inference class information and the correct answer class information, the loss between the inference posture information and the correct posture information, and the loss between the inference relationship information and the correct answer relationship information. In the training process, a weight parameter applied to data exchange between the plurality of sub-networks 621 , 622 , and 623 may also be adjusted. By adjusting the parameters of the neural network 620 until each loss is less than a preset threshold, a trained neural network 620 may be obtained.

It is possible to acquire the trained cognitive neural network 620 by training the neural network 620 a preset number of times, but the training method of the neural network 620 is not limited thereto.

By jointly training the plurality of subnetworks 621 , 622 , and 623 for three tasks of object class recognition, posture recognition, and relation recognition, it is possible to effectively improve information accuracy. A gated message passing system may be applied to exchange data between the plurality of sub-networks 621 , 622 , and 623 , and recognition may be performed through feature refinement based thereon.

7 is a diagram for explaining a process of acquiring object information according to an embodiment.

Referring to FIG. 7 , a process of extracting object information from an input image is illustrated.

In step 710 , an image to be recognized is obtained, and shared features may be obtained by extracting image features based on the VGG16 network. The VGG16 network may be a convolutional network including 16 convolutional layers and an entire connection layer. The VGG16 network can simplify the structure of a neural network. Based on the Faster R-CNN network, an object is recognized in a shared feature, and feature regions can be obtained. The Faster R-CNN network may be a neural network including a convolution layer, a region proposal network (RPN), a region of interest pooling, and a classification and regression network. However, the network type used in the process of extracting image features and recognizing shared features is not limited thereto.

Based on the feature region, the candidate object region, the candidate object peripheral region, and the region in which the related object pair is located may be cropped, respectively. The candidate object region may indicate a region in which an object is located among the feature regions, the region around the candidate object may indicate a region around the object among the feature regions, and the region in which the related object pair is located may indicate a region in which the related object pair is located.

In step 720, the cropped selected object region, the region around the candidate object, and the region in which the pair of related objects are located are respectively input to the class recognition network to obtain class characteristics of the object, and input to the posture recognition network to obtain the posture characteristics of the object obtained, and input to the relation-aware network to obtain relational characteristics of the object. In this specification, for convenience of description, the relation feature may also be referred to as a scene graph feature.

In operation 730 , each of the class feature, the posture feature, and the relationship feature may be corrected through data exchange between the class recognition network, the posture recognition network, and the relationship recognition network.

In operation 740 , class information of an object, for example, information about a person, a hat, and a kite, may be output from the class recognition network. Also, the posture information of the object may be output from the posture recognition network. In addition, relation information between objects, for example, a scene graph such as a person wearing a hat, a person flying a kite, and a person standing on the grass, may be output from the relationship recognition network.

8 and 9 are diagrams illustrating examples of object information obtained according to an embodiment.

The class recognition network, the posture recognition network, and the relationship recognition network are jointed with each other to correct features of each other during the recognition process, so that object information in the input image can be recognized more accurately. The understanding of a 3D scene based on object information, including object detection, posture estimation, and relationship recognition between objects, enables high-accuracy information acquisition by maximizing the relationship between the entire scene and objects. The recognized object information may be used in various fields such as smart home, autonomous driving, and security as well as augmented reality systems.

In addition, the object information may be provided as information necessary for other applications. For example, as shown in FIG. 8 , the smart home may recognize an event such as 'person 810-fall-floor' based on the recognized object information, and may sound an alarm to notify the user.

In addition, when a specific object is obscured by another object, the class and posture of the obscured object may be better recognized using information on the surrounding object. A large area of chair 2 on the right side of FIG. 9 is covered by the table in front and chair 1 on the left. By using the object recognition technique described above, class information and 3D posture information of an obscured object can be recognized more accurately.

10 is a diagram for describing a process of generating a virtual object based on acquired object information according to an exemplary embodiment.

Based on the class, posture, and relation of the real object in the scene, a possible position and posture of the virtual object to be added to the scene and the relation between the posture and surrounding objects may be predicted. Through this, the added virtual object may be able to interact realistically and naturally with the surrounding environment.

For example, when there is a bookshelf next to a chair in a real scene, when the augmented reality system adds a virtual character to the scene, a virtual character sitting in a chair and reading a book for natural interaction with the real scene may be created. Alternatively, if the chair is facing the table on which the laptop is placed in the real scene, the virtual character can sit on the chair and use the computer on the table. If the chair is with the table back and facing the TV, the virtual character can sit on the chair and watch TV. As described above, a possible position, posture, and motion of a virtual object are estimated according to a class, posture, and relationship of a real object in a real scene, and a natural interaction between the virtual and the real may be implemented based on the estimated result.

When the virtual object 1020 is added to the real scene 1010 shown in FIG. 10 , the virtual object 1020 standing on the sofa as in the first case 1030 is a real object included in the real scene 1010 . It can be unnatural with people. If the object information recognized from the real scene 1010 described above (eg, a sofa next to a bookshelf) is used, as in the second case 1040 , by adding a virtual object 1030 that reads a book while sitting on the sofa, more realistic It is possible to create scenes that are naturally fused with real objects. Hereinafter, virtual object creation will be described in detail with reference to the drawings.

11 is a diagram for describing a process of generating a virtual object in an image in an electronic device according to an exemplary embodiment.

Referring to FIG. 11 , a method of generating a virtual object in a processor included in an electronic device is illustrated.

In operation 1110, the electronic device may determine location information, posture information, and motion information of a virtual object to be created in the image based on object information including class information, posture information, and relationship information of the object included in the image. . For example, the electronic device may obtain location information, posture information, and motion information of a virtual object that can be rendered as an image by inputting object information into the rendering prediction network. In the present specification, for convenience of description, location information, posture information, and motion information of a virtual object may be referred to as virtual object information.

The location information of the virtual object may indicate a position where the virtual object can be rendered in an image, the posture information may indicate a rotation angle of the virtual object, and the motion information may indicate the operation of the virtual object. The virtual object may include a virtual character or a virtual object. When a virtual object is rendered in an image, a realistic and natural scene can be obtained by utilizing predicted position information, posture information, and motion information.

The rendering prediction network may include three sub-networks each predicting location information, posture information, and motion information of a virtual object. The three subnetworks may include a position regression network, a posture prediction network, and a motion candidate network.

A position regression network can predict an appropriate position of a virtual object through a convolutional layer, a pooling layer, and a fully connected layer using object features as input. The pose prediction network may be a regression network used to estimate the 3D pose of a virtual object in a scene. The motion candidate network may output a scene graph including the virtual object and the real object by predicting a relationship between the virtual object and the surrounding object.

In operation 1120 , the electronic device may add the virtual object to the image based on location information, posture information, and motion information of the virtual object.

The position information of the virtual object obtained from the rendering prediction network may include at least one position, the posture information may include different postures of the virtual object at each position, and the motion information may include at least one operation of the virtual object. may include If several positions, postures and motions are predicted, the user can select one position, posture and motion from among various predicted positions, postures and motions, and the virtual object is created according to the selected position, posture and motion. It can be rendered into an image.

In an embodiment, in response to input of class information, posture information, and relationship information of an object recognized prior to the rendering prediction network, by obtaining location information, posture information, and motion information of a virtual object that can be rendered in the image, the image Depending on the class, posture, and relationship of my real object, I can create a virtual object that interacts naturally with the real object.

The three subnetworks in the rendering prediction network may be combined with each other, and in the process of predicting location information, posture information, and motion information of a virtual object, each information may be exchanged and used to correct other information, through which A virtual object having a more natural interaction with a real object may be obtained.

12 to 14 are diagrams for explaining a process of generating a virtual object by estimating information on an object included in an image according to an exemplary embodiment.

Referring to FIG. 12 , a process of extracting object information from an input image and adding a virtual object to an image based on the object information is illustrated. The electronic device may acquire an RGB-D image 1210 that is a color image including a depth image. The electronic device may input the RGB-D image 1210 into the joint estimation module 1220 to estimate object information 1230 included in the image. For example, the joint estimation module 1220 outputs a result of classifying the object, a 3D posture of the object, and a scene graph as estimated scene information 1230 , and based on this, class information and posture of the object Object information 1240 including information and relationship information may be obtained. The object information 1240 is input to the virtual object prediction module 1250 to obtain position information, posture information, and motion information of the virtual object 1260 that can be rendered in the RGB-D image 1210 . The virtual object prediction module 1250 may also be referred to as a rendering prediction network.

Hereinafter, a training technique of the rendering prediction network will be described. From each of the plurality of sample images for training of the rendering prediction network, the remaining scene parts except for the preset object may be obtained. The rendering prediction network may be trained so that, when class information, posture information, and relationship information of an object of the remaining scene portion are input, preset object position information, posture information, and motion information are output.

For example, if the sample image includes a person sitting in a chair, the chair and the person are separated to obtain the chair attribute information and the relationship information between the chair and the ground, and the location information, posture information, and motion information of the person are obtained. , the rendering prediction network may be trained to output position information, posture information, and motion information of a person when the attribute information of the chair and the relationship information between the chair and the ground are input.

To implement this, training data may be generated as follows. For example, first, an image including a person is selected from an existing image set, and class information, posture information, and relationship information of a preset object (that is, a person) can be extracted from the selected image through the joint estimation module. . Object information of a person may be separated from other information and used as correct answer learning data, and information of other objects may be used as input learning data.

Referring to FIG. 13 , an example of rendering a virtual object on an image is illustrated. Class information, posture information, and relationship information of a real object included in the image 1310 may be acquired. When the electronic device adds the virtual object 1330 to the image 1310 , information on the objects in the image 1310 may be acquired through the joint estimation module. Using such object information, the electronic device may predict the renderable position and posture of the virtual object 1330 and the relationship 1320 with the surrounding real objects. An image 1340 in which the virtual object is naturally rendered may be obtained based on the prediction result.

Referring to FIG. 14 , the AR device 1410 may capture an RGB-D image. The joint estimation module 1420 may estimate class information, posture information, and relationship information of each of one or more objects in the image obtained from the AR device 1410 . When the AR device 1410 receives a rendering command (that is, a control command) for a virtual object from a user or a designer, class information, posture information, and relationship information of the object are input to the virtual object prediction module 1430 to obtain an image Position information, posture information, and motion information of a virtual object to be rendered may be output to the . The CG engine 1440 may render a virtual object in an image based on location information, posture information, and motion information of the virtual object.

15 is a diagram illustrating an electronic device according to an exemplary embodiment.

Referring to FIG. 15 , an electronic device 1500 includes a processor 1510 and a memory 1520 . Optionally, the electronic device 1500 may further include a transceiver 1530 . The processor 1510 , the memory 1520 , and the transceiver 1530 may be connected to each other through a bus 1540 . The electronic device 1500 includes various computing devices such as smart phones, tablets, laptops and personal computers, various wearable devices such as smart watches, smart glasses, and smart clothing, various home appliances such as smart speakers, smart TVs, and smart refrigerators, smart cars, and smart devices. It may include a kiosk, an Internet of Things (IoT) device, a Walking Assist Device (WAD), a drone, a robot, and the like.

The processor 1510 may include a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a transistor logic device, and a hardware configuration. elements or any combination thereof. It may be implemented or implemented in combination with the various illustrative logical blocks, modules, and circuits described herein. Processor 1510 may also be a combination that realizes computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, and the like.

Memory 1520 may be read only memory (ROM) or other type of static storage that may store static information and instructions, random access memory (RAM), or other type of dynamic storage that may store information and instructions, and , or Electronically Erasable Programmable Read Only Memory (EEPROM), Compact Disc Read Only Memory (CD-ROM), or other optical disc storage device, optical disc storage device such as compact disc, laser disc, optical disc, digital universal optical disc; Blu-ray disc, etc.), a disk storage medium or other magnetic storage device, or any other medium that is used to carry or store the desired program code in the form of instructions or data structures and which can be accessed by a computer, but is not limited thereto. does not

The memory 1520 is used to store application program codes for performing the above-described operations, and execution may be controlled by the processor 1510 . The processor 1510 may be used to implement the above-described operations by executing the application program code stored in the memory 1520 .

The bus 1540 may include a path for transmitting information between components. The bus 1540 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 1540 may be classified into an address bus, a data bus, a control bus, and the like. For convenience of expression, only one bold line is used in FIG. 15, but this does not indicate that there is only one bus or that there is only one bus type.

In addition, the above-described operation may be processed with respect to the electronic device 1500 .

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and a software application running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium are specially designed and configured for the embodiment, or are known and available to those skilled in the art of computer software. may be Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, a person skilled in the art may apply various technical modifications and variations based thereon. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

acquiring an image;
acquiring class characteristics, posture characteristics, and relational characteristics of objects included in the image;
correcting the class feature, the posture feature, and the relationship feature, respectively, by using the class feature, the posture feature, and the relation feature of the object; and
obtaining class information, posture information, and relationship information of the object based on the corrected class feature, the corrected posture feature, and the corrected relationship feature, respectively
containing
A method of operation of an electronic device.
According to claim 1,
The correcting step is
correcting any one of the class feature, the posture feature, and the relationship feature by applying a predetermined weight to the class feature, the posture feature, and the relation feature of the object,
A method of operation of an electronic device.
According to claim 1,
The acquiring of the class feature, the posture feature, and the relation feature includes:
obtaining each of the class feature, the posture feature, and the relation feature from an intermediate layer of a corresponding subnetwork,
A method of operation of an electronic device.
4. The method of claim 3,
Intermediate layers of a plurality of sub-networks are connected to each other and the class feature, the posture feature, and the relation feature are shared by other sub-networks,
A method of operation of an electronic device.
According to claim 1,
The step of obtaining the class information, the posture information, and the relationship information of the object
In response to each of the corrected class feature, the corrected posture feature, and the corrected relational feature being input to a next layer of an intermediate layer of a corresponding subnetwork, the class information and the posture information from an output layer of a corresponding subnetwork and obtaining the relationship information,
A method of operation of an electronic device.
According to claim 1,
The class information includes information about which object the object detected in the image is,
The posture information includes information indicating the rotation angle of the object detected in the image,
The relation feature includes action information of an object detected in the image and/or connection information with another object,
A method of operation of an electronic device.
According to claim 1,
determining position information, posture information, and motion information of a virtual object to be created in the image based on class information, posture information, and relationship information of the object; and
adding a virtual object to the image based on location information, posture information, and motion information of the virtual object
further comprising,
A method of operation of an electronic device.
8. The method of claim 7,
The step of adding the virtual object is
adding the virtual object to the image based on information selected by a user from among the plurality of pieces of information when at least one of location information, posture information, and motion information determined for the virtual object is plural;
A method of operation of an electronic device.
8. The method of claim 7,
The location information of the virtual object includes information indicating a location where the virtual object can be rendered in an image,
The posture information includes information indicating the rotation angle of the virtual object,
The operation information includes information indicating the operation of the virtual object,
A method of operation of an electronic device.
According to claim 1,
The image is an RGB-D image,
A method of operation of an electronic device.
A computer-readable storage medium in which a program for executing the method of any one of claims 1 to 10 is recorded.
one or more processors;
the one or more processors
acquire an image,
Obtaining class characteristics, posture characteristics, and relational characteristics of the object included in the image,
correcting the class feature, the posture feature, and the relationship feature, respectively, by using the class feature, the posture feature, and the relation feature of the object,
Obtaining class information, posture information, and relationship information of the object based on the corrected class feature, the corrected posture feature, and the corrected relationship feature, respectively
electronic device.
13. The method of claim 12,
the one or more processors
correcting any one of the class feature, the posture feature, and the relationship feature by applying a predetermined weight to the class feature, the posture feature, and the relation feature of the object,
electronic device.
13. The method of claim 12,
the one or more processors
obtaining each of the class feature, the posture feature, and the relation feature from an intermediate layer of a corresponding subnetwork,
electronic device.
15. The method of claim 14,
Intermediate layers of a plurality of sub-networks are connected to each other and the class feature, the posture feature, and the relation feature are shared by other sub-networks,
electronic device.
13. The method of claim 12,
the one or more processors
In response to each of the corrected class feature, the corrected posture feature, and the corrected relational feature being input to a next layer of an intermediate layer of a corresponding subnetwork, the class information and the posture information from an output layer of a corresponding subnetwork and obtaining the relationship information,
electronic device.
13. The method of claim 12,
The class information includes information about which object the object detected in the image is,
The posture information includes information indicating the rotation angle of the object detected in the image,
The relation feature includes action information of an object detected in the image and/or connection information with another object,
electronic device.
13. The method of claim 12,
the one or more processors
determining position information, posture information, and motion information of a virtual object to be created in the image based on class information, posture information, and relationship information of the object;
adding a virtual object to the image based on location information, posture information, and motion information of the virtual object
electronic device.
19. The method of claim 18,
the one or more processors
adding the virtual object to the image based on information selected by a user from among the plurality of pieces of information when at least one of location information, posture information, and motion information determined for the virtual object is plural;
electronic device.
19. The method of claim 18,
The location information of the virtual object includes information indicating a location where the virtual object can be rendered in an image,
The posture information includes information indicating the rotation angle of the virtual object,
The operation information includes information indicating the operation of the virtual object,
electronic device.

Images