KR102597074B1 - Image generating device for generating 3d images corresponding to user input sentences and operation method thereof - Google Patents

Abstract

An embodiment of the present disclosure discloses an image generating device that generates a 3D image corresponding to a user's input sentence and a method of operating the image generating device. The image generating device includes a display providing an input interface, a memory storing at least one instruction, and at least one processor executing at least one instruction stored in the memory, and the at least one processor executes at least one instruction. Acquire the user's input sentence through the input interface by executing, segment at least one sentence component included in the obtained input sentence, and generate a 3D image based on the at least one segmented sentence component. 3D images can be provided through the display.

Description

Image generating device that generates a 3D image corresponding to a user's input sentence and its operating method {IMAGE GENERATING DEVICE FOR GENERATING 3D IMAGES CORRESPONDING TO USER INPUT SENTENCES AND OPERATION METHOD THEREOF}

The present invention relates to an image generating device, and more specifically, to an image generating device and a method of operating the same for generating a 3D image corresponding to a user's input sentence.

Recently, technology has been developed to receive sentences such as text as input and generate images or videos based on the sentences.

In particular, with the development of AI (Artificial Intelligence) technology, models that generate images or videos, such as GAN (Generative Adversarial Networks), VAE (Variational AutoEncoder), and Transformer, are being developed.

Through this, many companies are providing services that create webtoons, pictures, videos, etc. by entering sentences.

Registration No. 2081229 (Registration Date: February 19, 2020)

When creating an image based on an input sentence, if there are multiple people included in the sentence, it may be difficult to create the image by considering the interaction between the people when creating the image.

Additionally, when creating an image, it may be difficult to create an image with an appropriate view by considering the number of people included in the sentence or the movements of the people when determining the view of the image.

Additionally, when inputting a sentence to create an image, the user may have difficulty composing the sentence that must be entered to create the desired image.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

According to the present disclosure for achieving the above-described technical problem, an image generating device that generates a 3D image corresponding to a user's input sentence can be provided. The image generating device may include a display that provides an input interface. The image generating device may include a memory that stores at least one instruction and at least one processor that executes the at least one instruction stored in the memory. At least one processor may obtain a user's input sentence through an input interface by executing at least one command. At least one processor may segment at least one sentence component included in the obtained input sentence by executing at least one instruction. At least one processor may generate a 3D image based on at least one segmented sentence component by executing at least one instruction. At least one processor can provide the generated 3D image through a display by executing at least one instruction.

Additionally, according to the present disclosure, a method of operating an image generating device that generates a 3D image corresponding to a user's input sentence can be provided. A method of operating an image generating device may include providing an input interface through a display. A method of operating an image generating device may include obtaining a user's input sentence through an input interface. The method of operating the image generating device may include segmenting at least one sentence component included in the obtained input sentence. A method of operating an image generating device may include generating a 3D image based on at least one segmented sentence component. A method of operating an image generating device may include providing a generated 3D image through a display.

In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to the means for solving the above-described problem of the present disclosure, an image generating device that generates a 3D image corresponding to a user's input sentence extracts sentence components included in the input sentence and creates a 3D image based on the extracted sentence components. By creating 3D images, the accuracy of 3D image creation can be increased.

Additionally, when there are multiple people included in the input sentence, a 3D image can be generated by considering the interaction between the plurality of people, and an appropriate view can be determined when generating the 3D image.

In addition, user convenience can be improved by providing an input interface that allows users to sequentially input sentence elements necessary for creating a 3D image.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below.

1 is a block diagram showing the configuration of an image generating device according to an embodiment of the present disclosure.
Figure 2 is a diagram for explaining the operation of an image generating device according to an embodiment of the present disclosure.
FIG. 3A is a diagram for explaining the operation of an image generating device that generates a 3D image based on changes in character elements according to an embodiment of the present disclosure.
FIG. 3B is a diagram for explaining the operation of an image generating device that generates a 3D image based on changes in character elements according to an embodiment of the present disclosure.
FIG. 4A is a diagram for explaining the operation of an image generating device that generates a 3D image based on the number of people and motion elements included in the person element according to an embodiment of the present disclosure.
FIG. 4B is a diagram for explaining the operation of an image generating device that generates a 3D image based on the number of people and motion elements included in the person element according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating the operation of an image generating device that generates a 3D image based on the movement and direction of each of a plurality of recognized people according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating the operation of an image generating device that generates a 3D image by considering interactions between a plurality of recognized people according to an embodiment of the present disclosure.
FIG. 7A is a diagram for explaining the operation of an image generating device that generates a background image according to an embodiment of the present disclosure.
FIG. 7B is a diagram for explaining the operation of an image generating device that generates a person image according to an embodiment of the present disclosure.
FIG. 7C is a diagram for explaining the operation of an image generating device that generates a motion image of a person according to an embodiment of the present disclosure.
Figure 8 is a diagram for explaining an operation of providing a 3D image of a single frame corresponding to a motion found in a video including a plurality of frames.
Figure 9 is a diagram for explaining an operation of determining a view of a 3D image based on the number of people and motion elements included in a user's input sentence according to an embodiment of the present disclosure.
FIG. 10 is a diagram illustrating the operation of an image generating device that receives a plurality of sentence components through a plurality of interfaces according to an embodiment of the present disclosure.
FIG. 11A is a diagram for explaining an operation of editing a generated 3D image according to an embodiment of the present disclosure.
FIG. 11B is a diagram for explaining an operation of editing a generated 3D image according to an embodiment of the present disclosure.
FIG. 11C is a diagram for explaining an operation of editing a generated 3D image according to an embodiment of the present disclosure.
FIG. 12 is a flowchart illustrating a method of operating an image generating device according to an embodiment of the present disclosure.

Like reference numerals refer to like elements throughout this disclosure. The present disclosure does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the present disclosure pertains is omitted. The term 'unit, module, member, block' used in the specification may be implemented as software or hardware, and depending on the embodiment, a plurality of 'unit, module, member, block' may be implemented as a single component, or It is also possible for one 'part, module, member, or block' to include multiple components.

Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Terms such as first and second are used to distinguish one component from another component, and the components are not limited by the above-mentioned terms.

Singular expressions include plural expressions unless the context clearly makes an exception.

The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is.

In this specification, 'image generating device according to the present disclosure' includes all various devices that can perform computational processing and provide results to the user. For example, the device according to the present disclosure may include all of a computer, a server, and a portable terminal, or may take the form of any one.

Here, the computer may include, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

The server is a server that processes information by communicating with external devices, and may include an application server, computing server, database server, file server, game server, mail server, proxy server, and web server.

The portable terminal is, for example, a wireless communication device that guarantees portability and mobility, such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), and PDA. (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone ), all types of handheld wireless communication devices, and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-device (HMD). may include.

Hereinafter, the operating principle and embodiments of the present disclosure will be described with reference to the attached drawings.

1 is a block diagram showing the configuration of an image generating device according to an embodiment of the present disclosure.

Referring to FIG. 1 , in one embodiment, the image generating device 100 may generate a 3D image corresponding to a sentence input by a user using the image generating device 100. The image generating device 100 may provide the generated 3D image to the user. However, the present disclosure is not limited thereto, and it goes without saying that the image generating device 100 can generate a 2D image corresponding to the input sentence. Hereinafter, for convenience of explanation, it will be described that the image generating device 100 generates a 3D image corresponding to an input sentence. Additionally, a 3D image may include both an image corresponding to one frame or a video corresponding to multiple frames.

In one embodiment, the image generating device 100 may include a display 110, a memory 120, and at least one processor 130. The components shown in FIG. 1 are not essential for implementing the image generating device 100 according to the present disclosure. In one embodiment, the image generating device 100 described herein may have more or fewer components than those listed above. The display 110, the memory 120, and at least one processor 130 may each be electrically and/or physically connected to each other.

In one embodiment, the display 110 may display information processed by the image generating device 100. The display 110 may display an input interface provided by the image generating device 100.

In one embodiment, the display 110 displays execution screen information of an application program (for example, an input interface) running on the image generating device 110, or a user interface (UI) or graphic user interface (GUI) according to the execution screen information. Interface) information can be displayed.

In one embodiment, at least one processor 130 may control the display 110 to display an input interface, thereby providing the input interface to a user using the image generating device 100. A user may provide a user input sentence to the image generating device 100 through an input interface displayed on the display 110.

In one embodiment, at least one processor 130 may control the display 110 to display a 3D image generated to correspond to a user input sentence. In one embodiment, at least one processor 130 may provide the generated 3D image to the user through the display 110.

In one embodiment, the memory 120 may store data supporting various functions of the image generating device 100 and a program for operation of at least one processor 130, and may store input/output data (e.g. For example, sentences, music files, still images, videos, etc.), a plurality of application programs (application programs or applications) running on the device, at least one data for operation of the device, at least One instruction can be stored. At least some of these applications may be downloaded from an external server via wireless communication.

The memory 120 includes a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), and a multimedia card micro type. micro type), card type memory (e.g. SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), EEPROM (electrically erasable) It may include at least one type of storage medium among programmable read-only memory (PROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. Additionally, the memory is separate from the image generating device 100, but may be a database connected wired or wirelessly.

In one embodiment, at least one processor 130 may control overall operations of the image generating device 100. In one embodiment, at least one processor 130 may control the operation of the image generating device 100 by executing at least one instruction stored in the memory 120. In one embodiment, at least one processor 130 may execute at least one command stored in the memory 120 to generate a 3D image corresponding to a user input sentence.

In one embodiment, the image generating device 100 may further include a communication interface. In one embodiment, the communication interface may perform data communication between the image generating device 100 and an external server or other nearby electronic devices.

In one embodiment, the communication interface may include one or more components that enable communication with an external server or other nearby electronic devices, for example, a wireless communication module, a short-range communication module, or a location information module. It can contain at least one.

In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, wireless communication modules include GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and UMTS (universal mobile telecommunications system). ), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, etc. may include a wireless communication module that supports various wireless communication methods.

The wireless communication module may include a wireless communication interface including an antenna and a transmitter that transmits a data signal. In addition, the wireless communication module further includes a data signal conversion module that modulates a digital control signal output from at least one processor 130 into an analog wireless signal through a wireless communication interface under the control of the at least one processor 130. It can be included.

The wireless communication module may include a wireless communication interface including an antenna and a receiver for receiving data signals. Additionally, the wireless communication module may further include a data signal conversion module for demodulating an analog wireless signal received through a wireless communication interface into a digital control signal.

The short-range communication module is for short-range communication and includes Bluetooth™ RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, and NFC (Near Field Communication). , short-distance communication can be supported using at least one of Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

In one embodiment, the image generating device 100 may further include a user interface. The user interface is for receiving information from the user. When information is input through the user interface, at least one processor 130 may control the operation of the image generating device 100 to correspond to the input information.

In one embodiment, the input interface displayed through the display 110 may be an interface for displaying information provided through a user input interface.

In one embodiment, the user interface may include touch keys. As an example, the touch key consists of a virtual key, soft key, or visual key displayed on a touch screen-type display unit through software processing, or is displayed on the touch screen. It may be composed of touch keys placed in other parts. The user can input a user input sentence using the touch keys.

Additionally, the user interface may include devices such as a mouse and keyboard. In one embodiment, a user may input a user input sentence using a keyboard or the like.

Additionally, the user interface may include a voice input interface such as a microphone. In one embodiment, the user may input a user input sentence using a microphone or the like.

In one embodiment, the image generating device 100 may communicate with an external server or surrounding electronic device through a communication interface. In one embodiment, external servers may include application servers, computing servers, database servers, file servers, game servers, mail servers, proxy servers, and web servers.

In one embodiment, a model including a method for generating a 3D image in response to a user's input sentence described in this disclosure may be included in an external server or nearby electronic device. The image generating device 100 may provide a user input sentence obtained through a user interface to an external server or nearby electronic device through a communication interface.

In one embodiment, the image generating device 100 may provide a 3D image provided from an external server or a nearby electronic device to the user through the display 110. A user may use a 3D image generation model included in an external server or external electronic device through the image generation device 100.

Hereinafter, for convenience of explanation, the operation of generating a 3D image in response to a user's input sentence will be described as being performed in the image generating device 100.

Figure 2 is a diagram for explaining the operation of an image generating device according to an embodiment of the present disclosure.

Referring to FIGS. 1 and 2 , in one embodiment, FIG. 2 corresponds to a user input sentence provided to the image generating device 100 through the input interface 300 provided to the user through the image generating device 100. The generated 3D image 400 is displayed. At this time, the input interface 300 and the 3D image 400 may be included in one integrated interface 200 and provided to the user.

In one embodiment, a user of the image generating device 100 may provide an input sentence to the image generating device 100 through the input interface 300. In one embodiment, at least one processor 130 may obtain a user's input sentence through the input interface 300.

In one embodiment, at least one processor 130 may extract at least one sentence component included in the obtained input sentence. At least one processor 130 may extract at least one sentence component included in the input sentence using natural language processing (NLP). In one embodiment, at least one processor 130 may extract at least one sentence component included in the input sentence using a Recurrent Neural Network (RNN), Long Short Term Memory (LSTM), Transformer, etc.

In one embodiment, at least one sentence component may include a sentence background element 310, such as a classroom, school, park, etc. In one embodiment, at least one sentence component may include a person element 320 of the sentence, such as a man, a woman, a name, an animal, an object, etc. In one embodiment, the at least one sentence component is an action element (330, 340, 350) of the sentence, such as fighting, reading, eating, taking an object, singing, etc. may include.

In one embodiment, the at least one processor 130 processes the background element 310, the character element 320, and the action element included in at least one sentence element through automatic segmentation and extraction ( 330, 340, 350) can be divided, and information of the input sentence can be extracted based on the divided elements. In one embodiment, referring to Figure 2, at least one processor 130 based on the segmented elements, the background of the input sentence is "classroom", the person is "woman and man", and the action is "football". Information such as “movement to steal the ball” and “fighting movement” can be extracted.

In one embodiment, at least one processor 130 may generate a 3D image 400 based on at least one extracted sentence component. The 3D image 400 may include at least one image component each corresponding to at least one extracted sentence component. In one embodiment, at least one image component may include a background image 410 corresponding to the background element 310 of the sentence. At least one image component may include person images 420 and 430 corresponding to the person element 320 of the sentence. At least one image component may include an action image 440 corresponding to the action elements 330, 340, and 350 of the sentence. At this time, the motion image 440 may be an element representing the motion of the person images 420 and 430.

FIG. 3A is a diagram for explaining the operation of an image generating device that generates a 3D image based on changes in character elements according to an embodiment of the present disclosure. FIG. 3B is a diagram for explaining the operation of an image generating device that generates a 3D image based on changes in character elements according to an embodiment of the present disclosure. Hereinafter, the same reference numerals will be assigned to the same components as those described in FIG. 2, and overlapping descriptions will be omitted.

Referring to FIGS. 1, 3A, and 3B, the person element 320 included in the input sentence shown in FIG. 3A is “A is B”, and the person element 321 included in the input sentence shown in FIG. 3B ) is “the younger brother is the older brother.”

In one embodiment, the at least one processor 130 determines from the person element 320 of “A is B” shown in FIG. 3A that the people in the input sentence are “A” and “B” and the operation of the input sentence. Information with the subject “A” can be extracted. At this time, since additional information such as age, gender, etc. of “” and “B” cannot be obtained, at least one processor 130 creates a random person image 420 corresponding to “A” based on the extracted information. A 3D image that performs an action on a random person image 421 corresponding to “B” can be created.

In one embodiment, the at least one processor 130 determines from the person element 321 of “younger brother” shown in FIG. 3B that the people in the input sentence are “younger brother” and “older brother” and determines the operation of the input sentence. Information that the subject is “younger brother” can be extracted. At this time, based on the additional information that the characters in the input sentence are “younger brother” and “older brother,” at least one processor 130 selects the person image 420 corresponding to “younger brother” to the person image corresponding to “older brother” ( 422), a 3D image that performs an operation can be generated. At this time, the shape of the person image 422 corresponding to “older brother” may be larger than the shape of the person image 420 corresponding to “younger brother.”

The present disclosure is not limited thereto, and when the background element or action element included in the input sentence is changed, the at least one processor 130 may generate a 3D image by reflecting the information included in the changed background element or action element. there is.

FIG. 4A is a diagram for explaining the operation of an image generating device that generates a 3D image based on the number of people and motion elements included in the person element according to an embodiment of the present disclosure. FIG. 4B is a diagram for explaining the operation of an image generating device that generates a 3D image based on the number of people and motion elements included in the person element according to an embodiment of the present disclosure. Hereinafter, the same reference numerals will be assigned to the same components as those described in FIG. 2, and overlapping descriptions will be omitted.

1, 4A, and 4B, in one embodiment, at least one processor 130 estimates the pose of the segmented person element 320 based on the segmented motion element. can do. In one embodiment, the action element of the input sentence shown in Figure 4A is “reclining,” and the person element 320 is “a man.”

In one embodiment, at least one processor 130 may estimate the pose of the segmented human element 320 using a pose estimation algorithm. In one embodiment, at least one processor 130 may use a pose estimation algorithm to recognize the joint positions of the human element 320 and identify the skeleton shape of the human element 320. In one embodiment, the pose estimation algorithm may use OpenPose, AlphaPose, or DensePose.

In one embodiment, in Figure 4A, at least one processor 130 may estimate the pose of “a man.” At least one processor 130 may determine the skeleton form of “a man.”

In one embodiment, at least one processor 130 includes a “reclining” operational element and

Based on the character elements of “a man”, the pose of “a man” on the 3D image corresponding to the user’s input sentence can be estimated. In one embodiment, at least one processor 130 may estimate the skeleton form of “a man” in a reclining form within the 3D image.

In one embodiment, at least one processor 130 may determine position and direction information of a person element that performs a segmented action element, based on the estimated pose of the person element. At least one processor 130 determines the position and direction information of the person image corresponding to the person element in the 3D image based on the position and direction information of the determined person element, and generates a 3D image based on the information. can do.

In one embodiment, at least one processor 130 may determine location and orientation information of “a man” based on a pose in which “a man” is leaning. In one embodiment, when the input sentence includes a background element, at least one processor 130 may determine the location information of “a man” as standing against the background element. At least one processor 130 may determine orientation information such that “a man” is standing facing forward with his back to a background element.

At this time, at least one processor 130 may determine the location and direction information of “one man” using a pre-learned correction algorithm based on learning information including location and direction information according to the poses of various people. .

In one embodiment, person element 320 in Figure 4A includes one person. In one embodiment, when the person element 320 includes one person, the at least one processor 130 determines the position and orientation of the person element at which the one person performs the segmented motion element based on the estimated pose. information can be determined. At least one processor 130 may determine location and direction information of a person image included in a 3D image based on the pose of one person. Accordingly, when the person element included in the user's input sentence includes one person, at least one processor 130 can determine the position and direction information of the person image included in the 3D image by considering the pose of one person. there is.

In one embodiment, the person element 321 in FIG. 4B includes a plurality of people. In one embodiment, when the person element 321 includes a plurality of people, at least one processor 130 performs a segmented motion element by considering the interaction between the estimated poses of each of the plurality of people. Position and direction information of human elements can be determined. At least one processor 130 may determine location and direction information of a person image included in a 3D image based on the poses of a plurality of people.

In one embodiment, at least one processor 130 may estimate the respective poses of “man and woman 321.” At least one processor 130 may estimate the poses of each “man and woman” using multi-person pose estimation. At least one processor 130 may estimate the respective poses of “man and woman” based on the motion elements of “side by side” (340) and “reclining” and the person elements of “man and woman” (321). there is.

In one embodiment, at least one processor 130 uses a pre-learned correction algorithm based on learning information including position and direction information according to the poses of various people to determine each of the “man and woman 321”. By considering the interaction between the estimated poses, the position and orientation information of “man and woman” can be determined. Accordingly, when the person element included in the user's input sentence includes a plurality of people, at least one processor 130 uses the position and direction information of the person image included in the 3D image to determine the pose of each of the plurality of people. It can be decided by considering interactions.

FIG. 5 is a diagram illustrating the operation of an image generating device that generates a 3D image based on the movement and direction of each of a plurality of recognized people according to an embodiment of the present disclosure. Hereinafter, the same reference numerals will be assigned to the same components as those described in FIG. 2, and overlapping descriptions will be omitted.

In one embodiment, referring to FIG. 5, at least one processor 130 may generate a 3D image 400 including a plurality of frames corresponding to a user input sentence provided through the input interface 300. . At this time, the 3D image 400 including a plurality of frames may be a video.

In one embodiment, at least one processor 130 may generate a 3D image of a single frame to correspond to an input sentence and then assign identifiers to people included in the frame. In one embodiment, the input sentence includes “two men 320,” and at least one processor 130 generates a single frame of a 3D image to include a person image corresponding to “two men 320.” When generated, at least one processor 130 may assign an identifier to each of the two male images 420 and 421 included in the person image.

In one embodiment, at least one processor 130 may use an identifier to track each of the men when an image of two men is “fighting 340” in a plurality of frames. . Based on this, the movement lines of the two male images can be tracked over a plurality of frames, and based on this, the location and direction information of the people included in the 3D image can be determined.

FIG. 6 is a diagram illustrating the operation of an image generating device that generates a 3D image by considering interactions between a plurality of recognized people according to an embodiment of the present disclosure. Hereinafter, the same reference numerals will be assigned to the same components as those described in FIGS. 2 and 6, and overlapping descriptions will be omitted.

Referring to FIGS. 2, 4B, 5, and 6, in one embodiment, at least one processor 130 determines the number of people in the person element 320 included in the input sentence. Different methods for determining location and direction information can be applied.

In one embodiment, when the people element 320 includes a single person, the at least one processor 130 can interact with the people element 320 and the motion element 340 without having to consider the interaction of the people element 320. ), the pose of the person element 320 may be estimated, and the location and direction information of the person element 320 may be determined based on the estimated person element 320.

In one embodiment, when the person element 320 includes a plurality of people, at least one processor 130 may determine the location and direction information of the person element 320 by considering the interaction between the plurality of people. there is. In one embodiment, the at least one processor 130 estimates the pose of each of the plurality of people based on the person element 320 and the motion element 340, and calculates the pose of each of the plurality of people based on the estimated pose. Location and direction information can be determined.

At least one processor 130 may correct the position and direction information of each of the plurality of people using an appropriate modeling algorithm according to the number of the plurality of people. At this time, the corrected location and direction information for each of the plurality of people may take into account the interaction between the plurality of people.

In one embodiment, the modeling algorithm may include a Social force model or Attention mechanism. Social force model may be a model that considers interactions between multiple people based on behavioral science. Attention mechanism may be a model that analyzes the degree to which a plurality of people influence each other when each person performs an action.

In one embodiment, referring to Figure 6, figure element 320 is shown to include “three men” and motion element 340 is shown to be “struggling.” At least one processor 130 may estimate the poses of three men fighting based on the person element 320 and the motion element 340, and may assign an identifier to each of the three men.

At least one processor 130 may extract location and direction information for each of the three men based on their respective estimated poses and identifiers. At least one processor 130 may correct the extracted position and direction information of each of the three men using a modeling algorithm and determine the corrected position and direction information by considering the interaction between the three men. At least one processor 130 may generate the 3D image 400 based on the corrected position and direction information. For example, if one of the three men is left-handed and the other two are right-handed, the position and direction information is corrected so that the struggle between the left-handed man and the right-handed man appears natural, taking into account their interaction. can do.

Compared to generating a 3D image by determining the position and direction information of the person element 320 regardless of the number of people included in the person element 320, when a plurality of people are included in the person element 320, When generating a 3D image based on position and direction information corrected by taking into account the interaction between a plurality of people, it is possible to generate images of people performing more natural movements. Additionally, a 3D image can be generated so that the actions of multiple people are all expressed in one scene.

FIG. 7A is a diagram for explaining the operation of an image generating device that generates a background image according to an embodiment of the present disclosure. FIG. 7B is a diagram for explaining the operation of an image generating device that generates a person image according to an embodiment of the present disclosure. FIG. 7C is a diagram for explaining the operation of an image generating device that generates a motion image of a person according to an embodiment of the present disclosure.

Referring to FIGS. 1, 7A, 7B, and 7C, in one embodiment, at least one processor 130 generates a 3D image corresponding to an input sentence, a background image 410, and a person image 420. ) and motion images 440 can be generated sequentially.

Referring to FIG. 7A, at least one processor 130 divides a background element in an input sentence and loads a background image corresponding to the divided background element from a background asset dump containing a plurality of basic background images. can do. In one embodiment, at least one processor 130 may extract background information from the divided background elements and load the background image 410 corresponding to the background information extracted from the background asset pile. In one embodiment, the first scene 500 including only the background image 410 may be provided to the user through the display.

Referring to FIG. 7B, at least one processor 130 may segment a person element in an input sentence and load a person image corresponding to the divided person element from a person asset pile including a plurality of basic person images. In one embodiment, at least one processor 130 may extract person information from the segmented person elements and load the person image 420 corresponding to the person information extracted from the person asset pile. In one embodiment, the second scene 510 including only the background image 410 and the person image 420 may be provided to the user through the display. At this time, the person information may include information such as the person's occupation, the person's clothing, the person's appearance, and the person's gender.

Referring to FIG. 7C, at least one processor 130 may segment action elements in an input sentence and determine pose, location, and direction information of the loaded person image 420. In one embodiment, at least one processor may extract operation information from the segmented operation elements. Motion information may include a person's pose, location, and direction information. At least one processor 130 may change the pose, position, and direction information of the loaded person image using 3D modeling, based on the extracted motion information. However, the present disclosure is not limited thereto. At least one processor 130 loads motion data corresponding to motion elements divided from a motion asset pile containing poses, motions, etc. of a plurality of people, and uses the loaded motion data to create a pose of the loaded person image, Location and direction information can also be determined.

At this time, the background asset pile, the character asset pile, and the motion asset pile may be stored in the memory 120. Additionally, the image generating device 100 may receive background asset piles, character asset piles, and motion asset piles stored in an external server or nearby electronic device through a communication interface.

Figure 8 is a diagram for explaining an operation of providing a 3D image of a single frame corresponding to a motion found in a video including a plurality of frames.

Referring to FIG. 8, the 3D image 400 generated to correspond to the input sentence entered into the input interface 300 may be a video representing a plurality of actions over a plurality of frames. In one embodiment, when the 3D image 400 is a video, at least one processor 130 uses the display 110 to display a search interface 360 that can search for the motion of a person included in the 3D image 400. It can be provided through. The user enters a search action element, for example, “uppercut”, including the action to be searched, into the search interface 360 to detect a 3D image in one frame corresponding to the corresponding action among the 3D images 400. You can.

In one embodiment, at least one processor 130 may obtain a search operation element through the search interface 360. At least one processor 130 may obtain a search operation image corresponding to the operation to be searched, based on the search operation element. At this time, the search operation image may be generated by at least one processor 130 based on the search operation element. In one embodiment, at least one processor 130 may generate a search operation image using Generative Adversarial Networks (GAN). In one embodiment, at least one processor 130 may provide search operation elements as input to the GAN to generate a search operation image including the operation to be searched.

Additionally, the at least one processor 130 may provide search operation elements to an external server or nearby electronic device and then obtain a search operation image generated from the external server or nearby electronic device.

At least one processor 130 compares the search motion image and a 3D image including a plurality of frames, and selects an action element included in the search motion image among a plurality of motions each corresponding to a plurality of frames included in the 3D image. A 3D image of one frame containing a motion corresponding to can be detected.

At least one processor 130 may provide a 3D image of one detected frame to the user through the display 110.

Through this, the user can easily find the 3D image of the frame corresponding to the desired motion within the 3D image containing a plurality of frames without the inconvenience of manually searching for the frame of the motion corresponding to the desired motion.

Figure 9 is a diagram for explaining an operation of determining a view of a 3D image based on the number of people and motion elements included in a user's input sentence according to an embodiment of the present disclosure.

Referring to FIG. 9, in one embodiment, the at least one processor 130 generates a 3D image based on at least one of the segmented background element, the segmented character element 320, and the segmented motion element 340. At least one of the number of lights (400), the location of lights, the color of lights, weather, sound, and the point of view of the generated 3D image can be determined. At this time, the viewpoint of the 3D image may mean the position or composition of the camera looking at the 3D image.

In one embodiment, when there is only one person included in the divided person element 320, the at least one processor 130 may determine the viewpoint of the 3D image as a bust shot, close-up, full-body shot, half-body shot, etc. there is. When there are multiple people included in the divided person element 320, at least one processor 130 may determine the viewpoint of the 3D image as a full shot, long shot, top view, etc.

In one embodiment, when the at least one processor 130 determines the viewpoint of the 3D image, not only the number of people included in the segmented person element 320 but also the segmented motion element 340 may be considered.

In one embodiment, at least one processor 130 may determine the viewpoint of the 3D image by considering not only the position, posture, and distance of the people included in the person element 320, but also the motion elements of the people. In one embodiment, when a plurality of people included in the person element 320 are physically fighting, the viewpoint of the 3D image may be determined so that all the plurality of people fighting are expressed in a full shot. Additionally, if the input sentence is “students fighting and fighting in the classroom,” the viewpoint of the 3D image will be determined so that the overall atmosphere of the classroom and the students’ fight are clearly visible and recognized as a single scene, rather than focusing on the fight between the “students.” It may be possible.

At this time, at least one processor 130 may use LSTM to determine the viewpoint of the 3D image based on the posture, motion, person, etc. of the character elements included in the input sentence.

Additionally, at least one processor 130 may vary the location, number, color, weather, sound, etc. of lights to vary the effect of the 3D image according to conditions included in the input sentence. In one embodiment, if the input sentence includes background elements that affect the effect of the 3D image, such as “in front of a cold corpse,” “on a dreary night,” “in a rainy park,” etc., at least one processor (130 ) can vary the location, number, color, weather, sound, etc. of lights based on the relevant factors.

In one embodiment, at least one processor 130 uses a Convolutional Neural Network (CNN) to extract features of images corresponding to background elements that affect the effect of the 3D image, and performs processing based on the extracted images. This allows you to determine the location, number, color, weather, sound, etc. of lights in the 3D image. In addition, at least one processor 130 performs segmentation of images corresponding to background elements that affect the effect of the 3D image using FCN (Fully Convolutional Network), and divides the 3D image based on the segmented images. You can determine the location, number, color, weather, sound, etc. of lights.

FIG. 10 is a diagram illustrating the operation of an image generating device that receives a plurality of sentence components through a plurality of interfaces according to an embodiment of the present disclosure.

Referring to FIG. 10 , in one embodiment, the input interface may include a first input interface 370, a second input interface 380, and a third input interface 390.

In one embodiment, the first input interface 370 may be an interface that can input a background element, for example, “classroom.” The second input interface 380 may be an interface that can receive character elements, for example, “student” or “center”, which is the location of the character. The third input interface 390 may be an interface that can receive input of action elements, for example, “hitting and fighting.”

In one embodiment, a user may sequentially input sentence elements of an input sentence through the first to third interfaces 370, 380, and 390. Through this, the information that the user unnecessarily provides to the image generating device 100 can be reduced, and the user's convenience in using the image generating device 100 can be increased.

Additionally, the user can select and modify an element he or she wants to modify among the generated 3D images 400 through the first to third interfaces 370, 380, and 390. In one embodiment, the 3D image 400 may include a background image, a person image 420, 421, and a motion image 440, respectively corresponding to the background element, the person element, and the motion element. The user may use the first to third interfaces 370, 380, and 390 to edit and input a sentence element corresponding to at least one image element that he or she wants to modify among the 3D image 400. Through this, the user's freedom of modification can be increased.

FIG. 11A is a diagram for explaining an operation of editing a generated 3D image according to an embodiment of the present disclosure. FIG. 11B is a diagram for explaining an operation of editing a generated 3D image according to an embodiment of the present disclosure. FIG. 11C is a diagram for explaining an operation of editing a plurality of components included in a 3D image generated by dividing them into layers according to an embodiment of the present disclosure.

Referring to FIGS. 1 and 11A , in one embodiment, at least one processor 130 may provide an editing display 600 that can edit the generated 3D image 400 through the display 110. . In one embodiment, a user may provide input to modify the generated 3D image through the editing interface 600.

At least one processor 130 acquires correction input information through the editing interface 600, and based on the obtained correction input information, at least one of the position, direction information, and movement of the person included in the 3D image 400 You can edit one.

In one embodiment, a user may resize, change the position, delete, or add assets included in the 3D image 400, such as background images, portrait images, or motion images, through the editing interface 600. You can.

In one embodiment, when the 3D image 400 includes a plurality of frames, the user can delicately control changes in the components of the image over time through the editing interface 600. At this time, the editing interface 600 may include a time-bar.

Referring to FIG. 11C, the editing interface 610 may display a plurality of operation models. A user may select one of a plurality of motion models to modify the motion of a person included in a 3D image.

Referring to FIG. 11C, the editing interface may include a plurality of editing layers 620, 630, and 640. At this time, each of the plurality of editing layers (620, 630, 640) divides the plurality of components included in the 3D image by layer, and the user uses the plurality of editing layers (620, 630, and 640) to edit each element. Components divided into layers can also be edited individually. Through this, you can edit only the components included in the layer you want to edit, without affecting the components in other layers.

FIG. 12 is a flowchart for explaining a method of operating an image generating device according to an embodiment of the present disclosure.

Referring to FIGS. 1, 2, and 12, the method of operating the image generating device 100 to generate a 3D image 400 corresponding to a user's input sentence includes providing an input interface 300 (S100). It can be included. In one embodiment, a user using the image generating device 100 may input an input sentence into the image generating device 100 through the input interface 300.

In one embodiment, the method of operating the image generating device 100 may include obtaining a user's input sentence through the input interface 300 (S200).

In one embodiment, the method of operating the image generating device 100 may include segmenting at least one sentence component included in the obtained input sentence (S300). In one embodiment, in the step of segmenting at least one sentence component (S300), information on the at least one segmented sentence component may be extracted.

In one embodiment, a method of operating the image generating device 100 may include generating a 3D image 400 based on at least one segmented sentence component (S400). In one embodiment, in step S400 of generating the 3D image 400, the 3D image may be generated based on information on at least one extracted sentence component.

In one embodiment, the method of operating the image generating device 100 may include providing the generated 3D image 400 to the user through the display 110 (S500). Accordingly, the image generating device 100 may generate and provide a 3D image corresponding to the input sentence to the user who inputs the input sentence.

The method according to an embodiment of the present disclosure described above may be implemented as a program (or application) and stored in a medium in order to be executed in combination with a server, which is hardware.

The above-mentioned program is C, C++, JAVA, machine language, etc. that can be read by the processor (CPU) of the computer through the device interface of the computer in order for the computer to read the program and execute the methods implemented in the program. It may include code coded in a computer language. These codes may include functional codes related to functions that define the necessary functions for executing the methods, and include control codes related to execution procedures necessary for the computer's processor to execute the functions according to predetermined procedures. can do. In addition, these codes may further include memory reference-related codes that indicate at which location (address address) in the computer's internal or external memory additional information or media required for the computer's processor to execute the above functions should be referenced. there is. In addition, if the computer's processor needs to communicate with any other remote computer or server to execute the above functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes regarding whether communication should be performed and what information or media should be transmitted and received during communication.

The storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers that the computer can access or on various recording media on the user's computer. Additionally, the medium may be distributed to computer systems connected to a network, and computer-readable code may be stored in a distributed manner.

The steps of the method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which this disclosure pertains.

Above, embodiments of the present disclosure have been described with reference to the attached drawings, but those skilled in the art will understand that the present disclosure can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

Claims (10)

A display providing an input interface;
a memory storing at least one instruction; and
At least one processor executing the at least one instruction stored in the memory,
The at least one processor,
By executing the at least one command,
Obtain the user's input sentence through the input interface,
Dividing at least one sentence component included in the obtained input sentence, wherein the at least one sentence component includes a background element of the sentence, a person element of the sentence, and an action element of the sentence,
Generating a 3D image based on the at least one segmented sentence component,
The generated 3D image is provided through the display,
When the person element includes a person, estimate the pose of the divided person element based on the divided motion element,
Generate the 3D image by determining position and direction information of the person element that performs the segmented action element based on the estimated pose of the person element,
When the person element includes a plurality of people, the position and direction information of the plurality of people performing the divided action element is determined by considering the interaction between the plurality of estimated poses of each of the plurality of people. do,
Assigning an identifier to each person included in the image frame of the generated 3D image,
An image generating device, characterized in that the location and direction information of each person included in the 3D image is determined by tracking the movement line of each person using the identifier.
According to paragraph 1,
When the person element includes one person,
The at least one processor, by executing the at least one instruction, determines position and direction information of the person performing the segmented action element based on the estimated pose of the person,
When the person element includes multiple people,
By executing the at least one instruction, the at least one processor determines the positions of the plurality of people performing the segmented action element, taking into account interactions between the plurality of estimated poses of each of the plurality of people, and An image generating device that determines orientation information.
According to paragraph 1,
When the person element includes one person,
The at least one processor, by executing the at least one instruction, determines position and direction information of the person performing the segmented action element based on the estimated pose of the person,
When the person element includes multiple people,
By executing the at least one instruction, the at least one processor determines the positions of the plurality of people performing the segmented action element, taking into account interactions between the plurality of estimated poses of each of the plurality of people, and An image generating device that determines orientation information.
According to paragraph 1,
The input interface is,
A first input interface for inputting the background element, a second input interface for inputting the character element, and a third input interface for inputting the action element,
The at least one processor executes the at least one instruction,
An image generating device that acquires the background element through the first input interface, the character element through the second input interface, and the action element through the third input interface.
According to paragraph 1,
The at least one processor executes the at least one instruction,
Loading a background image corresponding to the divided background element from a background asset dump containing a plurality of basic background images,
Loading a person image corresponding to the divided person element from a person asset pile containing a plurality of basic person images,
An image generating device that generates the 3D image by determining pose, position, and direction information of the loaded person image based on the divided motion elements.
According to paragraph 1,
The at least one processor executes the at least one instruction,
Based on at least one of the divided background element, the divided character element, and the divided motion element, the number of lights, the position of the light, the color of the light, weather, sound, and the generated 3D image of the generated 3D image. An image generating device that determines at least one of the points of view.
According to paragraph 1,
The at least one processor executes the at least one instruction,
Providing an editing interface for editing the generated 3D image through the display,
Obtain correction input information for modifying the generated 3D image through the editing interface,
An image generating device that generates and provides a 3D image in which at least one of the position, direction information, size, and motion of the person in the 3D image is modified based on the correction input information.
According to paragraph 1,
When the generated 3D image is a video showing multiple movements of a person included in the 3D image over multiple frames,
The at least one processor executes the at least one instruction,
Providing a search interface through the display that can search for movements of people in the 3D image,
Through the search interface, obtain a search sentence including a search action element for searching the movement of the person included in the 3D image,
Based on the search action elements, generate a search action image,
Comparing the generated search motion image and the 3D image to detect a 3D image of one frame representing a motion corresponding to a search motion element among the plurality of motions included in the 3D image,
An image generating device that provides a 3D image of the single frame through the display.
In a method of operating an image generating device that generates a 3D image corresponding to a user input sentence,
providing, by the image generating device, an input interface through a display;
acquiring, by the image generating device, the input sentence of the user through the input interface;
The image generating device divides at least one sentence component included in the obtained input sentence, wherein the at least one sentence component includes a background element of the sentence, a person element of the sentence, and an action element of the sentence. Component division step;
generating, by the image generating device, the 3D image based on the at least one segmented sentence component; and
A step of the image generating device providing the generated 3D image through the display,
The image generating device,
When the person element includes a person, estimate the pose of the divided person element based on the divided motion element,
Generate the 3D image by determining position and direction information of the person element that performs the segmented action element based on the estimated pose of the person element,
When the person element includes a plurality of people, considering the interaction between the plurality of estimated poses of each of the plurality of people, determine the position and direction information of the plurality of people performing the divided action element. do,
Assigning an identifier to each person included in the image frame of the generated 3D image,
A method of operating an image generating device, characterized in that the location and direction information of each person included in the 3D image is determined by tracking the moving line of each person using the identifier.
A computer-readable recording medium coupled to a computer and storing a program for executing the method of claim 9.

Images