KR102533209B1 - Method and system for creating dynamic extended reality content - Google Patents

Abstract

The present disclosure provides a dynamic XR content generation method, performed by at least one processor of a user terminal. The method includes the steps of: receiving a plurality of images taken from different directions of a target object located in a specific space; receiving user input associated with XR content from the user; and playing the XR content generated based on the plurality of images and information associated with the user input on a display.

Description

Dynamic extended reality (XR) content creation method and system {METHOD AND SYSTEM FOR CREATING DYNAMIC EXTENDED REALITY CONTENT}

The present disclosure relates to a method and system for generating dynamic augmented reality (XR) content, and more specifically, to a method and system for generating XR content generated using a neural radiance field for a target object.

Extended reality (XR) refers to virtual convergence technology, and various technologies such as AR (augmented reality), VR (virtual reality), and MR (mixed reality) immersively expand the user's experience in a virtual space similar to reality. It can refer to the technique of giving. Extended reality content can be provided through various devices such as Head Mounted Display (HMD), smart phone, and lightweight augmented reality glasses. In addition, the extended reality content may be a generic term for interfaces that allow users to interact in a virtual environment within the extended reality as if in reality.

On the other hand, in order to produce a promotional video for a product, a lot of money and effort are consumed. For example, in a studio producing a product promotion video, a screen on which a product to be promoted and a background screen are reproduced must be prepared. In addition, a mechanical device such as a camera rail must be prepared in advance so that a camera photographing a product moves along a camera path. After that, a process of editing the captured video and inserting promotional phrases and video effects is required. In this process, if the background screen or shooting path and angle need to be modified, there is the hassle of having to shoot the product again in the studio.

The present disclosure provides a method and apparatus (system) for generating dynamic augmented reality (XR) content to solve the above problems.

The present disclosure may be implemented in a variety of ways, including a method, apparatus (system) or computer program stored on a readable storage medium.

According to an embodiment of the present disclosure, a method for generating dynamic augmented reality (XR) content includes receiving a plurality of images of a target object located in a specific space from different directions, and a user input related to XR content from a user. Receiving and reproducing XR contents generated based on information associated with a plurality of images and a user input on a display.

According to an embodiment of the present disclosure, XR content may include a plurality of view synthesis images of a target object viewed from various positions, generated using a neural radiance field of the target object. . Here, the neuroradiation field for the target object may be generated based on a plurality of images.

According to an embodiment of the present disclosure, a neuroradiation field of a target object may be learned based on a plurality of images and positions and poses of each image.

According to an embodiment of the present disclosure, receiving a user input includes receiving a first user input associated with a camera path, and the XR content moves a target object in a space different from the specific space to the camera path. A composite image of a plurality of viewpoints viewed along may be included.

According to an embodiment of the present disclosure, the first user input may be a user input for selecting one of a plurality of predefined camera paths.

According to an embodiment of the present disclosure, the first user input may be a user input generated by a user moving a user terminal.

According to an embodiment of the present disclosure, receiving the user input further includes receiving a second user input associated with the background space, and the XR content includes a plurality of views of a target object located in the background space along a camera path. It may include a composite image of a viewpoint.

According to an embodiment of the present disclosure, the receiving of the user input may further include receiving a third user input associated with the text, and the text may be overlaid on a composite image of a plurality of viewpoints and displayed.

According to an embodiment of the present disclosure, a computer program stored in a computer-readable recording medium may be provided to execute a method for generating dynamic augmented reality (XR) content in a computer.

According to an embodiment of the present disclosure, an apparatus for providing a chat room background screen in an instant messaging service is provided. An apparatus for providing a chat room background screen in an instant messaging service includes a communication module, a memory, a display, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and includes at least one processor. The program receives a plurality of images of a target object located in a specific space from different directions, receives a user input related to XR content from a user, and generates a generated image based on the plurality of images and information related to the user input. It may include instructions for playing XR content on a display.

According to an embodiment of the present disclosure, a method for generating dynamic augmented reality (XR) content includes receiving a plurality of images of a target object located in a specific space from a user terminal in different directions, the XR content from the user terminal. It may include receiving information associated with and generating XR content based on a plurality of images and information.

According to various embodiments of the present disclosure, a user can easily produce high-quality dynamic XR content of a target object using a general user terminal (eg, a smartphone) without specialized equipment. Accordingly, the user can efficiently produce dynamic XR content, such as a promotional video of a target object, without space and time constraints.

According to various embodiments of the present disclosure, a user can easily create dynamic XR content using a plurality of images of a target object and a few settings in a dynamic XR content creation application. In addition, the user can create and view various dynamic XR contents by changing settings (eg, various camera paths, various background spaces) in the dynamic XR content creation application. As a result, it is possible to effectively reduce the cost and time required to create and edit dynamic XR contents such as product promotional videos.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clear to those skilled in the art (referred to as "ordinary technicians") from the description of the claims. will be understandable.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein like reference numbers indicate like elements, but are not limited thereto.
1 shows an example in which dynamic augmented reality (XR) content is generated according to an embodiment of the present disclosure.
2 is a schematic diagram showing a configuration in which an information processing system is communicatively connected with a plurality of user terminals in order to generate dynamic augmented reality (XR) content according to an embodiment of the present disclosure.
3 is a block diagram showing internal configurations of a user terminal and an information processing system according to an embodiment of the present disclosure.
4 is a diagram illustrating an example of photographing a target object to generate dynamic XR content according to an embodiment of the present disclosure.
5 is a diagram illustrating an example of setting a camera path for generating dynamic XR content according to an embodiment of the present disclosure.
6 is a diagram illustrating an example of setting a background space for generating dynamic XR contents according to an embodiment of the present disclosure.
7 is a diagram illustrating an example of setting text for generating dynamic XR content according to an embodiment of the present disclosure.
8 is a diagram illustrating an example of outputting dynamic XR content generated according to an embodiment of the present disclosure on a display.
9 is a diagram illustrating an example of a method of generating a neural radiation field for a target object according to an embodiment of the present disclosure.
10 is a flowchart illustrating an example of a method for generating dynamic XR contents according to an embodiment of the present disclosure.

Hereinafter, specific details for the implementation of the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the gist of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, identical or corresponding elements are given the same reference numerals. In addition, in the description of the following embodiments, overlapping descriptions of the same or corresponding components may be omitted. However, omission of a description of a component does not intend that such a component is not included in an embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the following embodiments in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms, but only the present embodiments make the present disclosure complete, and the present disclosure does not extend the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary according to the intention of a person skilled in the related field, a precedent, or the emergence of new technology. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms and the general content of the present disclosure, not simply the names of the terms.

Expressions in the singular number in this specification include plural expressions unless the context clearly dictates that they are singular. Also, plural expressions include singular expressions unless the context clearly specifies that they are plural. When it is said that a certain part includes a certain component in the entire specification, this means that it may further include other components without excluding other components unless otherwise stated.

Also, the term 'module' or 'unit' used in the specification means a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not meant to be limited to software or hardware. A 'module' or 'unit' may be configured to reside in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, a 'module' or 'unit' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, and attributes. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, or variables. Functions provided within components and 'modules' or 'units' may be combined into a smaller number of components and 'modules' or 'units', or further components and 'modules' or 'units'. can be further separated.

According to one embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and a memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some circumstances, 'processor' may refer to an application specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), or the like. 'Processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or a combination of any other such configurations. You may. Also, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' includes random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), It may also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor can read information from and/or write information to the memory. Memory integrated with the processor is in electronic communication with the processor.

In the present disclosure, a 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may consist of one or more server devices. As another example, a system may consist of one or more cloud devices. As another example, the system may be operated by configuring a server device and a cloud device together.

In this disclosure, 'display' may refer to any display device associated with a computing device, for example, any display device capable of displaying any information/data provided or controlled by the computing device. can refer to

In the present disclosure, 'each of a plurality of A' or 'each of a plurality of A' may refer to each of all components included in a plurality of A's, or each of some components included in a plurality of A's. .

In some embodiments of the present disclosure, 'a plurality of images' may refer to a video including a plurality of images, and 'video' may refer to a plurality of images included in the video.

1 shows an example in which dynamic augmented reality (XR) content is generated according to an embodiment of the present disclosure. As shown, the user may photograph the target object 120 from various angles through the user terminal 110 . In this case, dynamic XR contents 130 may be created based on a plurality of captured images of the target object 120 .

In one embodiment, the user may photograph the target object 120 located on a specific space (eg, on a table) from different directions through the user terminal 110 . In addition, the user may input various setting values related to dynamic XR content in the dynamic XR content creation application. Here, settings associated with dynamic XR content may include a camera path, background space, text settings, and the like.

In one embodiment, target object images in the dynamic XR content 130 may be generated based on a plurality of captured images of the target object 120 . Specifically, a neural radiation field for the target object 120 may be generated based on a plurality of captured images of the target object 120 . Thereafter, a plurality of view synthesis images viewing the target object 120 from various positions may be generated using the generated neuroradiation field. The dynamic XR content 130 may include a plurality of view synthesis images.

In an embodiment, the dynamic XR content 130 may be generated based on a plurality of captured images of the target object 120 and information related to setting of the dynamic XR content. For example, the dynamic XR content 130 may be a video in which the image of the target object 120 generated according to the position and pose corresponding to the camera path set by the user and the image of the background space set by the user are merged. . Additionally, the text input by the user may be overlaid and displayed on the dynamic XR content 130 . The user can check the generated dynamic XR contents 130 on the display of the user terminal 110 .

Through this configuration, a user can easily create high-quality dynamic XR contents of a target object using a general user terminal (eg, a smartphone) without specialized equipment. Accordingly, the user can efficiently produce dynamic XR content, such as a promotional video of a target object, without space and time constraints.

2 is a schematic diagram showing a configuration in which an information processing system 230 is communicatively connected to a plurality of user terminals 210_1, 210_2, and 210_3 in order to generate dynamic augmented reality (XR) content according to an embodiment of the present disclosure. am. As shown, a plurality of user terminals 210_1, 210_2, and 210_3 may be connected to an information processing system 230 capable of providing a dynamic augmented reality (XR) content creation service through a network 220. Here, the plurality of user terminals 210_1 , 210_2 , and 210_3 may include terminals of users receiving a dynamic augmented reality (XR) content generation service.

In one embodiment, the information processing system 230 is one or more servers capable of storing, providing, and executing computer-executable programs (eg, downloadable applications) and data associated with the provision of dynamic augmented reality (XR) content creation services. device and/or database, or one or more distributed computing devices and/or distributed database based cloud computing services.

The dynamic augmented reality (XR) content creation service provided by the information processing system 230 is a dynamic extended reality (XR) content creation service application web browser or web browser installed in each of the plurality of user terminals 210_1, 210_2, and 210_3. It may be provided to the user through a browser extension program or the like. For example, the information processing system 230 transmits information corresponding to a dynamic extended reality (XR) content creation request received from the user terminals 210_1, 210_2, and 210_3 through a dynamic extended reality (XR) content creation service application. It may provide or perform corresponding processing.

A plurality of user terminals 210_1 , 210_2 , and 210_3 may communicate with the information processing system 230 through the network 220 . The network 220 may be configured to enable communication between the plurality of user terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of a wireless network such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and the user terminals (210_1, 210_2, 210_3) as well as a communication method utilizing a communication network (eg, mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network 220 may include ), short-range wireless communication between them may also be included.

In FIG. 2, a mobile phone terminal 210_1, a tablet terminal 210_2, and a PC terminal 210_3 are illustrated as examples of user terminals, but are not limited thereto, and the user terminals 210_1, 210_2, and 210_3 may perform wired and/or wireless communication. This may be any computing device capable of running a dynamic augmented reality (XR) content creation service application or a web browser, etc. For example, user terminals include AI speakers, smartphones, mobile phones, navigation devices, computers, laptop computers, digital broadcasting terminals, personal digital assistants (PDA), portable multimedia players (PMPs), tablet PCs, game consoles, It may include a wearable device, an internet of things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, a set-top box, and the like. In addition, although three user terminals 210_1, 210_2, and 210_3 are shown in FIG. 2 to communicate with the information processing system 230 through the network 220, it is not limited thereto, and a different number of user terminals may be connected to the network ( 220) to communicate with the information processing system 230.

Figure 3 is a block diagram showing the internal configuration of the user terminal 210 and the information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 may refer to any computing device capable of executing applications, web browsers, etc. and capable of wired/wireless communication, for example, the mobile phone terminal 210_1 of FIG. 2, the tablet terminal 210_2, It may include a PC terminal 210_3 and the like. As shown, the user terminal 210 may include a memory 312 , a processor 314 , a communication module 316 and an input/output interface 318 . Similarly, the information processing system 230 may include a memory 332 , a processor 334 , a communication module 336 and an input/output interface 338 . As shown in FIG. 3, the user terminal 210 and the information processing system 230 are configured to communicate information and/or data through the network 220 using respective communication modules 316 and 336. It can be. In addition, the input/output device 320 may be configured to input information and/or data to the user terminal 210 through the input/output interface 318 or output information and/or data generated from the user terminal 210.

The memories 312 and 332 may include any non-transitory computer readable media. According to one embodiment, the memories 312 and 332 are non-perishable mass storage devices such as read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. can include As another example, a non-perishable mass storage device such as a ROM, SSD, flash memory, or disk drive may be included in the user terminal 210 or the information processing system 230 as a separate permanent storage device separate from memory. Also, an operating system and at least one program code may be stored in the memories 312 and 332 .

These software components may be loaded from a computer readable recording medium separate from the memories 312 and 332 . A recording medium readable by such a separate computer may include a recording medium directly connectable to the user terminal 210 and the information processing system 230, for example, a floppy drive, a disk, a tape, a DVD/CD- It may include a computer-readable recording medium such as a ROM drive and a memory card. As another example, software components may be loaded into the memories 312 and 332 through the communication modules 316 and 336 rather than computer-readable recording media. For example, at least one program is loaded into the memories 312 and 332 based on a computer program installed by files provided by developers or a file distribution system that distributes application installation files through the network 220 . It can be.

The processors 314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processors 314 and 334 by memories 312 and 332 or communication modules 316 and 336 . For example, processors 314 and 334 may be configured to execute instructions received according to program codes stored in a recording device such as memory 312 and 332 .

The communication modules 316 and 336 may provide configurations or functions for the user terminal 210 and the information processing system 230 to communicate with each other through the network 220, and the user terminal 210 and/or information processing. System 230 may provide configurations or functions for communicating with other user terminals or other systems (eg, separate cloud systems, etc.). For example, a request or data generated by the processor 314 of the user terminal 210 according to a program code stored in a recording device such as the memory 312 (eg, a dynamic augmented reality (XR) content creation request, a captured images, etc.) may be transferred to the information processing system 230 through the network 220 under the control of the communication module 316 . Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 passes through the communication module 336 and the network 220 and through the communication module 316 of the user terminal 210. It may be received by the user terminal 210 .

The input/output interface 318 may be a means for interfacing with the input/output device 320 . As an example, the input device may include a device such as a camera, keyboard, microphone, mouse, etc. including an audio sensor and/or image sensor, and the output device may include a device such as a display, speaker, haptic feedback device, or the like. can As another example, the input/output interface 318 may be a means for interface with a device in which a configuration or function for performing input and output is integrated into one, such as a touch screen. For example, when the processor 314 of the user terminal 210 processes a command of a computer program loaded into the memory 312, information and/or data provided by the information processing system 230 or other user terminals are used. A service screen or the like configured as described above may be displayed on the display through the input/output interface 318 . Although the input/output device 320 is not included in the user terminal 210 in FIG. 3 , it is not limited thereto, and the user terminal 210 and the user terminal 210 may be configured as one device. In addition, the input/output interface 338 of the information processing system 230 is connected to the information processing system 230 or means for interface with a device (not shown) for input or output that the information processing system 230 may include. can be In FIG. 3, the input/output interfaces 318 and 338 are shown as separate elements from the processors 314 and 334, but are not limited thereto, and the input/output interfaces 318 and 338 may be included in the processors 314 and 334. there is.

The user terminal 210 and the information processing system 230 may include more components than those shown in FIG. 3 . However, there is no need to clearly show most of the prior art components. In one embodiment, the user terminal 210 may be implemented to include at least some of the aforementioned input/output devices 320 . In addition, the user terminal 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, when the user terminal 210 is a smartphone, it may include components that are generally included in a smartphone, for example, an acceleration sensor, a gyro sensor, a microphone module, a camera module, and various physical Various components such as a button, a button using a touch panel, an input/output port, and a vibrator for vibration may be implemented to be further included in the user terminal 210 .

While a program for a dynamic augmented reality (XR) content creation service application is running, the processor 314 includes a touch screen connected to the input/output interface 318, a keyboard, a camera including an audio sensor and/or an image sensor, a microphone, and the like. Text, image, video, voice, and/or motion input or selected through an input device may be received, and the received text, image, video, voice, and/or motion may be stored in the memory 312 or the communication module ( 316) and the information processing system 230 through the network 220.

The processor 314 of the user terminal 210 manages, processes, and/or stores information and/or data received from the input/output device 320, other user terminals, the information processing system 230, and/or a plurality of external systems. can be configured to Information and/or data processed by processor 314 may be provided to information processing system 230 via communication module 316 and network 220 . The processor 314 of the user terminal 210 may transmit and output information and/or data to the input/output device 320 through the input/output interface 318 . For example, the processor 314 may display the received information and/or data on the screen of the user terminal 210 .

The processor 334 of the information processing system 230 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals 210 and/or a plurality of external systems. Information and/or data processed by the processor 334 may be provided to the user terminal 210 through the communication module 336 and the network 220 .

4 is a diagram illustrating an example of photographing a target object 410 to generate dynamic XR content according to an embodiment of the present disclosure. In one embodiment, by selecting the photographing button 420 in the photographing interface 400 of the dynamic XR content creation application through a touch input or the like, the user may photograph the target object 410 located in a specific space from various angles and directions. can Alternatively, the user may input multiple previously captured images/videos into the dynamic XR content creation application.

In one embodiment, the target object 410 may be masked and extracted from a plurality of images obtained by capturing the target object 410 . Thereafter, a plurality of view synthesis images viewing the target object 410 from various positions may be generated based on the plurality of images including the extracted target object 410 .

In one embodiment, a neuroradiation field for the target object 410 may be generated based on a plurality of images obtained by capturing the target object 410 . In addition, a plurality of view synthesis images viewing the target object 410 from various positions may be generated using the neuroradiation field of the target object 410 .

5 is a diagram illustrating an example of setting a camera path for generating dynamic XR content according to an embodiment of the present disclosure. Users can set camera paths in dynamic XR content creation applications. Here, the camera path may refer to a path of a position and pose in which a virtual camera captures the target object 510 in dynamic XR content. Additionally, the user can set the speed at which the virtual camera moves along the path. Specifically, the user may set the virtual camera to move at a predetermined speed along the camera path or to move faster or slower in a specific section. In addition, the user can set the virtual camera to move gradually faster or slower along one section or the entire section of the camera path.

In one embodiment, the user may set the camera path by selecting the camera path setting button 520 in the dynamic XR contents creation application with a touch input or the like. In this case, the user may select one of the predefined camera path sets 530 on the display. Alternatively, the user may create a camera path to capture the target object 510 by moving the user terminal. Specifically, the movement path of the user terminal may be recorded through a gyro sensor or the like mounted in the user terminal. In this case, the user may set a camera path to capture the target object 510 by moving the user terminal.

In one embodiment, when a user selects one camera path, the selected camera path may be displayed on the display of the user terminal. For example, when a user selects a first predefined camera path 532 from among a set of predefined camera paths 530, a camera path preview 540 corresponding thereto may be displayed on the display. The user can easily check predefined camera paths and select a desired camera path by checking the camera path preview 540 displayed on the display. In one embodiment, the user may modify and use the camera path preview 540 displayed on the display through touch input, drag input, and the like.

If the user selects the “create” button after selecting the camera path, dynamic XR content may be created based on the selected camera path. For example, if the user selects the "create" button after selecting the first predefined camera path 532, the virtual camera descends and rotates according to the first predefined camera path 532 and Dynamic XR content including a composite image of a plurality of viewpoints, such as a photograph of the object 510, may be created. In the case of FIG. 5 , since the user has not set a background space for the dynamic XR content, the dynamic XR content is a case in which a virtual camera captures a target object 510 along a first predefined camera path 532 in an empty space. It may include a plurality of viewpoint synthesis images such as

6 is a diagram illustrating an example of setting a background space for generating dynamic XR contents according to an embodiment of the present disclosure. Users can set the background space in dynamic XR content creation applications. Here, the background space may refer to a 3D space serving as a background of the target object 610 in dynamic XR content.

In one embodiment, the user may set the background space by selecting the background space setting button 620 in the dynamic XR contents creation application with a touch input or the like. In this case, the user can select any one of the set of predefined background spaces 630 on the display. Alternatively, the user may use background spatial data previously stored in the user terminal.

In one embodiment, a user may set a background space captured using a user terminal as a background space of dynamic XR content. Here, the background space photographed by the user may be a space photographed in an environment separate from the target object 610 . In this case, the background space may be spatial data generated by a view synthesis method based on a neuroradiation field of a space captured in a separate environment.

In one embodiment, when the user selects one background space, the selected background space may be displayed on the display of the user terminal. For example, when the user selects a first predefined background space 632 from the set of predefined background spaces 630, a corresponding background space preview 640 may be displayed on the display. By checking the background space preview 640 displayed on the display, the user can easily check predefined background spaces and select a desired background space. In one embodiment, the user may modify and use the background space preview 640 displayed on the display through touch input, drag input, and the like.

After selecting the camera path and the background space, if the user selects the “create” button, dynamic XR content may be created based on the selected camera path and background space. For example, the user may select a “create” button after selecting a first predefined camera path (eg, 532 in FIG. 5 ) and a first predefined background space 632 . In this case, while the virtual camera descends and rotates along the first predefined camera path (eg, 532 in FIG. 5 ), the target object 610 disposed in the first predefined background space 632 is captured. Dynamic XR content including a composite image of a plurality of viewpoints as captured may be created.

7 is a diagram illustrating an example of setting text for generating dynamic XR content according to an embodiment of the present disclosure. The user may set the text 720 in the dynamic XR content by selecting the text setting button 710 in the dynamic XR content creation application with a touch input or the like. In this case, the user may set the phrase of the text, the size of the text, the location of the text, the font of the text, and the like.

After selecting the camera path, background space, and text, if the user selects the "create" button 730, dynamic XR content may be created based on the selected camera path, background space, and text. For example, the user selects a first predefined camera path (eg, 532 in FIG. 5 ) and a first predefined background space (eg, 632 in FIG. 6 ), and writes text 720 . After inputting, the "create" button 730 can be selected. In this case, the virtual camera descends and rotates along the first predefined camera path (eg, 532 in FIG. 5 ) and is disposed in the first predefined background space (eg, 632 in FIG. 6 ). Dynamic XR content including a composite image of a plurality of viewpoints, such as a captured target object, may be created. Additionally, the text 720 may be overlaid and displayed on the composite image of a plurality of viewpoints. Alternatively, a virtual camera descends and rotates along a first predefined camera path (eg, 532 in FIG. 5 ) while moving in a first predefined background space (eg, 632 in FIG. 6 ). Dynamic XR content including a composite image of a plurality of viewpoints, such as a photograph of the arranged target object and text 720, may be created.

5 to 7 show that a user sets a camera path, background space, and text to create dynamic XR contents, but is not limited thereto. For example, the user may additionally set background music, visual effects, visual objects, camera zoom-in/zoom-out options, camera rotation options, and lighting options to be included in the dynamic XR content.

8 is a diagram illustrating an example of outputting dynamic XR content generated according to an embodiment of the present disclosure on a display. The user may check the dynamic XR contents generated through the first operation 810 , the second operation 820 , and the third operation 830 . For example, when the dynamic XR content is generated according to FIGS. 5 to 7 , the user moves the virtual camera along a first predefined camera path (eg, 532 in FIG. 5 ) while descending and rotating, 1 It is possible to check a video including a composite image of a plurality of viewpoints, such as a photograph of a target object (eg, a shoe) disposed in a predefined background space (eg, 632 of FIG. 6 ). As shown, text (for example, "NEW ARRIVAL") set by the user may be overlaid and displayed on the created video.

Through this configuration, a user can easily create dynamic XR content with a plurality of images of a target object and a few settings in a dynamic XR content creation application. In addition, the user can create and view various dynamic XR contents by changing settings (eg, various camera paths, various background spaces) in the dynamic XR content creation application. As a result, it is possible to effectively reduce the cost and time required to create and edit dynamic XR contents such as product promotional videos.

9 is a diagram illustrating an example of a method of generating a neural radiation field for a target object according to an embodiment of the present disclosure. First, at least one processor of a user terminal or information processing system may receive a plurality of images/videos (videos) of a target object located in a specific space from different directions (910). When images/videos are received, the processor may estimate positions and poses of each image (920). Here, the position and pose at which each image is taken may refer to the position and direction of a camera that has taken each image. Various estimation methods for estimating a position and pose from an image may be used for position and pose estimation. For example, a photogrammetry technique of extracting feature points from a plurality of images and estimating positions and poses of each image may be used, but is not limited thereto, and various position and pose estimation methods may be used. Alternatively, each image may include information about a camera's shooting position and direction.

Thereafter, the processor may learn a neural radiance field (NeRF) of the target object based on the plurality of images and the positions and poses at which each image was captured (930). Here, the neural radiation field may be a machine learning model (eg, an artificial neural network model). According to an embodiment, the neuroradiation field may be a model learned to receive location information and viewing direction information in a specific space and output color values and volume density values.

In one embodiment, the processor may estimate a color value and a volume density value for an arbitrary location and viewing direction in a specific space where the target object is located using the learned neuroradiation field. Accordingly, the processor may generate a composite image of a plurality of viewpoints viewing the target object from various positions and angles. In addition, the processor may generate a video including the target object by connecting the generated viewpoint synthesis images.

10 is a flowchart illustrating an example of a method 1000 for generating dynamic augmented reality (XR) content according to an embodiment of the present disclosure. In one embodiment, method 1000 may be performed by at least one processor of a user terminal and/or information processing system. The method 1000 may start with a processor receiving a plurality of images of a target object located in a specific space from different directions (S1010).

In one embodiment, the processor may receive a user input related to XR content from the user (S1020). For example, the processor may receive a first user input associated with a camera path. Here, the first user input may be a user input for selecting one of a plurality of predefined camera paths. Alternatively, it may be a user input generated by a user moving a user terminal. In this case, the XR contents may include composite images of a plurality of viewpoints looking at a target object along a camera path in a space different from a specific space.

Additionally or alternatively, the processor may receive a second user input associated with the background space. In this case, the XR contents may include composite images of a plurality of viewpoints looking at a target object located in the background space along a camera path. Additionally or alternatively, the processor may receive third user input associated with text. In this case, text may be overlaid and displayed on a composite image of a plurality of viewpoints.

Thereafter, the processor may reproduce XR contents generated based on information associated with a plurality of images and a user input on the display (S1030). Here, the XR content may include a plurality of view synthesis images of a target object viewed from various positions, generated using a neural radiance field of the target object. Also, a neuroradiation field for a target object may be generated based on a plurality of images. Specifically, the neuroradiation field for the target object may be an artificial neural network model learned based on a plurality of images and positions and poses at which each image was captured.

The above method may be provided as a computer program stored in a computer readable recording medium to be executed on a computer. The medium may continuously store programs executable by a computer or temporarily store them for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, but is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto optical media such as floptical disks, and Anything configured to store program instructions may include a ROM, RAM, flash memory, or the like. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server.

The methods, acts or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or combinations thereof. Those skilled in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and the design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs) ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be incorporated into a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or may be implemented or performed in any combination of those designed to perform the functions described in A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In firmware and/or software implementation, the techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on a computer readable medium, such as programmable read-only memory (EPROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage device, or the like. It can also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

If implemented in software, the techniques may be stored on or transmitted over as one or more instructions or code on a computer readable medium. Computer readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or desired program code in the form of instructions or data structures. It can be used for transport or storage to and can include any other medium that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium.

For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable , fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. Disk and disc as used herein include CD, laser disc, optical disc, digital versatile disc (DVD), floppy disk, and blu-ray disc, where disks are usually magnetic data is reproduced optically, whereas discs reproduce data optically using a laser. Combinations of the above should also be included within the scope of computer readable media.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium can be coupled to the processor such that the processor can read information from or write information to the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and storage medium may reside within an ASIC. An ASIC may exist within a user terminal. Alternatively, the processor and storage medium may exist as separate components in a user terminal.

Although the embodiments described above have been described as utilizing aspects of the presently-disclosed subject matter in one or more stand-alone computer systems, the disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Further, aspects of the subject matter in this disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure that can be understood by those skilled in the art. Moreover, such modifications and variations are intended to fall within the scope of the claims appended hereto.

110: user terminal
120: target object
130: dynamic XR content

Claims (11)

In the dynamic augmented reality (XR) content generation method, performed by at least one processor of a user terminal,
receiving a plurality of images of a target object located in a specific space from different directions;
Receiving a user input related to XR content from a user; and
Playing XR contents generated based on the plurality of images and information associated with the user input on a display;
including,
The XR content includes a plurality of view synthesis images of the target object viewed from various positions, generated using a neural radiance field of the target object,
wherein the neuroradiation field for the target object is generated based on the plurality of images.
delete According to claim 1,
The neuroradiation field for the target object is learned based on the plurality of images and the position and pose at which each image was taken, dynamic XR content creation method.
According to claim 1,
Receiving the user input,
Receiving a first user input associated with a camera path
including,
Wherein the XR content includes a plurality of viewpoint synthesis images viewed from the target object along the camera path in a space different from the specific space.
According to claim 4,
Wherein the first user input is a user input for selecting one of a plurality of predefined camera paths.
According to claim 4,
The first user input is a user input generated by a user moving the user terminal, dynamic XR content creation method.
According to claim 4,
Receiving the user input,
receiving a second user input associated with the background space;
Including more,
Wherein the XR content includes a plurality of viewpoint composite images viewed from the target object located in the background space along the camera path.
According to claim 4,
Receiving the user input,
receiving a third user input associated with the text;
Including more,
Dynamic XR content generation method in which the text is overlaid and displayed on the composite image of the plurality of viewpoints.
A computer program stored in a computer-readable recording medium to execute the method according to any one of claims 1, 3 to 8 on a computer.
As a user terminal,
communication module;
Memory;
display; and
at least one processor connected to the memory and configured to execute at least one computer readable program contained in the memory
including,
The at least one program,
Receiving a plurality of images of a target object located in a specific space from different directions;
Receive user input related to XR content from the user;
Includes instructions for playing on a display XR content generated based on the plurality of images and information associated with the user input;
The XR content includes a plurality of view synthesis images of the target object viewed from various positions, generated using a neural radiance field of the target object,
A neuroradiation field for the target object is generated based on the plurality of images, the user terminal.
In the dynamic augmented reality (XR) content creation method, performed by at least one processor,
Receiving a plurality of images of a target object located in a specific space from different directions from a user terminal;
Receiving information related to XR content from the user terminal; and
Generating XR content based on the plurality of images and the information
including,
The XR content includes a plurality of view synthesis images of the target object viewed from various positions, generated using a neural radiance field of the target object,
wherein the neuroradiation field for the target object is generated based on the plurality of images.

Images