KR20200143293A - Metohd and appartus for generating augumented reality video for real-time multi-way ar broadcasting - Google Patents

Abstract

The present invention relates to a method for generating an augmented reality image for real-time multi-way augmented reality (AR) broadcasting, and an apparatus thereof. According to the present invention, the method may comprise the steps of: obtaining motion data and first audio data for a target object, respectively; generating virtual image data including information on a virtual object corresponding to the target object by using the obtained motion data; acquiring real image data and second audio data about the real object; generating an augmented reality image based on the virtual image data, the real image data, the first audio data, and the second audio data; and transmitting the generated augmented reality image in real time.

Description

Augmented reality video generation method and device for real-time multi-way AR broadcasting {METOHD AND APPARTUS FOR GENERATING AUGUMENTED REALITY VIDEO FOR REAL-TIME MULTI-WAY AR BROADCASTING}

The present invention relates to a method and apparatus for generating an augmented reality image for real-time multi-source AR broadcasting.

Virtual reality (VR) refers to a specific environment or situation that is similar to reality created by artificial technology using computers, but is not real. It is one of the new paradigms in the field of information activities that allows people to indirectly experience situations that cannot be directly experienced in the real world due to spatial and physical constraints through action.

Augmented reality (AR) is a field of virtual reality (VR) and refers to a computer graphic technique that synthesizes virtual objects or information in an actual environment to make it look like an object existing in the original environment. Unlike virtual reality, which presupposes a complete virtual world, it is a technology that provides information based on reality, and can further increase the effect of reality by combining virtual objects. In other words, the actual appearance of the real world is added to the virtual reality. Augmented reality can be provided by synthesizing virtual objects on the basis of the real world by reinforcing additional information that is difficult to obtain with the real world alone or delivered for marketing purposes. Due to these characteristics, augmented reality can be applied to various reality environments, unlike existing virtual reality that can be applied only limited to fields such as games.

With the recent development of information and communication technology, as information processing and data transmission capabilities have increased dramatically, an augmented reality image obtained by combining a virtual image with a real image captured through a camera can be generated and provided in real time.

The present invention is to provide a method and apparatus for generating an augmented reality image for real-time multiple AR broadcasting.

According to an embodiment of the present invention, a method of generating an augmented reality image includes: obtaining motion data and first audio data, respectively, about a target object; Generating virtual image data including information on a virtual object corresponding to the target object by using the received motion data; Acquiring real image data and second audio data about the real object; Generating an augmented reality image based on the virtual image data, the real image data, the first audio data, and the second audio data; And transmitting the generated augmented reality image in real time.

According to an embodiment of the present invention, the apparatus for generating an augmented reality image includes a communication unit, a memory storing a program for generating the augmented reality image, and a program stored in the memory, thereby providing motion data of a target object. And obtaining first audio data, respectively, and generating virtual image data including information on a virtual object corresponding to the target object using the received motion data, and real image data and second audio related to the real object A processor for acquiring data, generating an augmented reality image based on the virtual image data, the real image data, the first audio data, and the second audio data, and transmitting the generated augmented reality image in real time. I can.

In a computer-readable recording medium according to an embodiment of the present invention, a program for executing an augmented reality image generation method on a computer may be recorded.

According to an embodiment of the present invention, a method and apparatus for generating an improved augmented reality image for real-time multi-source AR broadcasting can be provided.

1 and 2 are diagrams illustrating a system for providing a method of generating an augmented reality image according to an embodiment of the present invention.
3 is a diagram illustrating an augmented reality image generating apparatus according to an embodiment of the present invention.
4 is a flowchart illustrating a method of generating an augmented reality image.
5 and 6 are flowcharts illustrating an embodiment of step 440 of FIG. 4.
7 is a flow chart illustrating an embodiment of step 520 of FIG. 5.
8 illustrates an example of synchronizing reproduction times of virtual image data, real image data, and audio data in a method of generating an augmented reality image according to an embodiment of the present invention.
9 is a diagram illustrating an operation of a sensing module for estimating a position and angle of a camera in a method of generating an augmented reality image according to an embodiment of the present invention.
10 illustrates an example in which lyrics image data and actual image data are matched according to an operation of a camera in a method of generating an augmented reality image according to an embodiment of the present invention.
11 illustrates an example of an augmented reality image generated by a method of generating an augmented reality image according to an embodiment of the present invention.

The technical idea of the present invention is that various changes may be made and various embodiments may be provided. Specific embodiments are illustrated in the drawings and will be described in detail through detailed description. However, this is not intended to limit the technical idea of the present invention to a specific embodiment, it should be understood to include all changes, equivalents, and substitutes included in the scope of the technical idea of the present invention.

In describing the technical idea of the present invention, when it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description of the present specification are merely identification symbols for distinguishing one component from another component.

In addition, in the present specification, when one component is referred to as "connected" or "connected" to another component, the one component may be directly connected or directly connected to the other component, but specially It should be understood that as long as there is no opposing substrate, it may be connected or may be connected via another component in the middle.

In addition, terms such as "~ unit", "~ group", "~ character", and "~ module" described in the present specification mean a unit that processes at least one function or operation, which is a processor or microcomputer. Processor (Micro Processor), Application Processor (Application Processor), Micro Controller (Micro Controller), CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerate Processor Unit), DSP (Digital Signal Processor), ASIC ( Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), etc. may be implemented in hardware or software, or a combination of hardware and software.

In addition, it is intended to clarify that the division of the constituent parts in the present specification is merely divided by the main function that each constituent part is responsible for. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more according to more subdivided functions. In addition, each of the constituent units to be described below may additionally perform some or all of the functions of other constituent units in addition to its own main function, and some of the main functions of each constituent unit are different. It goes without saying that it may be performed exclusively by.

Hereinafter, embodiments according to the technical idea of the present invention will be sequentially described in detail.

1 and 2 are diagrams illustrating a system for providing a method of generating an augmented reality image according to an embodiment of the present invention.

A system for providing a method of generating an augmented reality image according to an embodiment may include a motion/audio data generating device 110, a server 120, and an augmented reality image generating device 130 according to the function. . In one embodiment, the motion/audio data generating device 110, the server 120, and the augmented reality image generating device 130 may be connected to a network. However, this is only an example, and the configuration of a system for providing a method of generating an augmented reality image is not limited thereto. In particular, only one motion/audio data generating device 110 is shown in FIG. 1, but in order to implement a plurality of virtual objects (ie, characters), a plurality of motion/audio data generating devices 110 are provided to be connected to the server 1120. I can. In addition, a system for providing a method of generating an augmented reality image may further include other configurations, may not include some of the configurations disclosed in FIGS. 1 and 2, or may be configured differently from FIGS. 1 and 2. For example, a system for providing a method for generating an augmented reality image is configured to directly transmit and receive data without the server 120, or according to an embodiment, a system for providing a method for generating an augmented reality image is a single device. It may be implemented so that there is no need for a separate network connection.

In one embodiment, the motion/audio data generating device 110 may be provided in a character studio (ie, a studio for acquiring motion and audio of a character actor playing a virtual object), and generating an augmented reality image The device 130 may be provided in a broadcasting studio that photographs an actual actor (ie, an entity object). In addition, in one embodiment, the server 120 is a server supporting 5th generation (5G), through which motion data (MDATA) and audio data (ADATA) are divided and transmitted in the present invention. Therefore, it is possible to minimize the occurrence of a time delay between both data.

The motion/audio data generating device 110 may include a motion data acquisition unit 111, an audio data acquisition unit 212, a communication unit 213, a control unit 214, etc., but this is only an example. / The configuration of the audio data generating device 110 is not limited thereto.

The motion data acquisition unit 111 may recognize a motion of a target object (eg, an actor) and generate motion data MDATA representing a motion (ie, movement) of the target object according to a recognition result. In an embodiment, the motion data acquisition unit 211 may acquire motion data MDATA using a sensor, image analysis, motion capture, or the like. For example, motion data (MDATA) may be generated by irradiating light (eg, infrared), or motion data (MDATA) may be generated using sensor data received from an electrode attached to an actor, but limited to this. It does not become. Here, the motion data MDATA may mean three-dimensional coordinates of each part of the target object, a velocity, acceleration, or moment of inertia of each part of the target object, but is not limited thereto.

In an embodiment, the motion data MDATA may include information on the motion of the target object acquired from at least one angle. For example, a plurality of sensors or cameras are installed at different angles around the target object, and motion data corresponding to each angle may be obtained by detecting the motion of the target object at one or more angles through this. Preferably, the motion data acquisition unit 111 includes the number of cameras used for photographing and the location of the cameras in order to acquire the actual image data for the real object in the image data acquisition unit 131 of the augmented reality image generating device 130. Motion data (MDATA) for the target object may be obtained through sensors or cameras disposed in the same number and at the same location.

The audio data acquisition unit 112 may measure sound (eg, voice) generated from the target object, and may generate audio data ADATA according to the measurement result. Here, the target object may be the same object as the object acquiring the motion data MDATA, but is not limited thereto, and may be acquired from another target object such as a voice actor. In addition, according to an embodiment, the audio data ADATA may include data on various sounds such as background sounds and effect sounds, as well as sounds directly produced by the target object. For example, the audio data acquisition unit 112 may acquire audio data ADATA through an input device such as a microphone, but is not limited thereto.

The communication unit 113 may communicate with an external device. Motion data MDATA and audio data ADATA acquired by the motion data acquisition unit 111 and the audio data acquisition unit 112 may be transmitted to an external device. In an embodiment, the communication unit 113 may separate motion data MDATA and audio data ADATA and transmit them to the server 120, respectively.

The controller 114 controls the overall operation of the motion/audio data generating device 110. For example, the control unit 114 acquires motion data (MDATA) using the motion data acquisition unit 111, and uses the audio data acquisition unit 112 to obtain audio data (ADATA) of a target object (eg, an actor). ) Is obtained, and an operation of transmitting data generated through the communication unit 113 to the server 120 may be controlled.

The server 120 may be a computing device capable of sending and receiving data according to a wired or wireless communication protocol with the motion/audio data generating device 110 and the augmented reality image generating device 130. According to embodiments, the server 120 may receive data, process the received data, and transmit the processed data. The server 120 may store a program including instructions for performing a specific operation in a non-temporary storage medium, load the program, and perform various operations or operations according to the control of the loaded program.

The server 120 receives the separately transmitted motion data MDATA and audio data ADATA using, for example, 5th generation mobile communication (5G), and receives the separately transmitted motion data MDATA and the audio data ADATA. 130). However, this communication method is exemplary and may be implemented to transmit motion data (MDATA) and audio data (ADATA) through various wireless communication methods according to an embodiment to which the present invention is applied.

The augmented reality image generation device 130 includes an image data acquisition unit 131, an audio data acquisition unit 132, a camera operation determination unit 133, a virtual image generation unit 134, an augmented reality image generation unit 135, The communication unit 136, the control unit 137, etc. may be included, but this is only an example, and the configuration of the augmented reality image generating apparatus 130 is not limited thereto.

The image data acquisition unit 131 may acquire real image data by photographing an actual object through a camera. The image data acquisition unit 131 may include, for example, a camera and a digital encoder and a codec program that digitally encode an image acquired from the camera. In an embodiment, the image data acquisition unit 131 may acquire actual image data captured through at least one camera. Preferably, the image data acquisition unit 131 is the same position as the motion data MDATA acquired by the motion data acquisition unit 111 (ie, the angle_) of the camera so that the actual image data for the real object can be acquired. The number and the location of the cameras may be set, thereby generating an augmented reality image with higher accuracy.

The audio data acquisition unit 132 may measure sound (eg, voice) generated from an actual object and generate audio data according to the measurement result. Depending on the embodiment, the audio data may include data about various sounds such as background sound and effect sound, as well as sound directly produced by an actual object. For example, the audio data acquisition unit 132 may acquire audio data through an input device such as a microphone, but is not limited thereto.

The camera operation determination unit 133 may determine an operation of a camera for photographing an actual object. In one embodiment, the camera operation determination unit 133 uses at least one camera by using a plurality of markers installed on the floor, ceiling, etc. of the shooting space of an actual object and a detection module attached to each of at least one camera. Can judge the operation of In addition, in an embodiment, the camera operation determination unit 133 may determine an operation of at least one or more cameras by analyzing an actual image captured by at least one or more cameras. Here, the operation of the camera may include a camera photographing position, a camera angle, a zoom in, a zoom out, a focus, and an image conversion from the first camera to the second camera.

The virtual image data generating unit 134 may generate virtual image data based on motion data MDATA and/or audio data ADATA received from the server 120. Here, the virtual image data may include data on at least one or more of a display position, angle, motion, size, direction, focus, and depth of the virtual object.

The augmented reality image generator 135 may synchronize virtual image data, real image data, audio data about a target object received from the server 120, and a playback time of audio data about the real object. In one embodiment, synchronization may be performed based on a PTS (Presentation Time Stamp). For example, the augmented reality image generator 135 may perform synchronization by adjusting the PTS of other data based on the PTS of the data having the slowest reproduction time among the data. According to the shooting operation of the camera determined by the camera operation determination unit 133, the combined image data is adaptively matched with the virtual object based on the virtual image data, the actual image data, and the information about the synchronized playback time. Can be generated. According to an embodiment, the virtual image may be changed together as the actual image is changed. For example, when a real person and a space are photographed through a plurality of cameras in a studio shooting an actual object, one camera changes its posture (ie, angle) or zoom in/zoom out, or the first camera When the image is switched to the camera, the display position (ie, coordinates), the display angle, and the size of the virtual object can be changed according to the change. In an embodiment, the augmented reality image generator 135 may be configured to generate combined image data in which a virtual object and a real object are matched based on a virtual camera. For example, a virtual camera corresponding to a real camera is created, and the pose (angle), position, or zoom in/zoom out of the virtual camera is changed in response to the actual camera motion, or the virtual object is actually It can be adaptively matched to the object. In addition, the augmented reality image generator 135 may generate an augmented reality image by combining audio data based on a synchronized playback time with the generated combined image data.

The communication unit 136 may communicate with an external device. In an embodiment, the communication unit 136 may receive motion data MDATA and audio data ADATA from the server 120. In addition, the communication unit 136 may transmit the generated augmented reality image to an external device (eg, a display device).

The controller 137 controls the overall operation of the augmented reality image generating device 130. For example, the control unit 137 acquires actual image data and audio data using the image data acquisition unit 131 and the audio data acquisition unit 132, and the camera operation using the camera operation determination unit 133 And, using the virtual image data generating unit 134 to generate virtual image data for a virtual object, and using the augmented reality image generating unit 135 to synchronize data, combine image data, and augmented reality image You can control the actions you create.

3 is a diagram illustrating an augmented reality image generating apparatus according to an embodiment of the present invention.

Referring to FIG. 3, the augmented reality image generating apparatus 300 according to an embodiment of the present invention may include a communication unit 310, a processor 320, and a memory 330 according to the hardware configuration, This is only an example, and the configuration of the augmented reality image generating apparatus 300 is not limited thereto. For example, the augmented reality image generating apparatus 300 may further include other configurations or may not include some of the configurations disclosed in FIG. 3.

In an embodiment, the communication unit 310 may transmit and receive data or signals with an external device such as a server or a display device or an external server under the control of the processor 320. The communication unit 310 according to an embodiment may perform the function of the communication unit 136 illustrated and described in FIG. 3. For example, the communication unit 310 may receive motion data and audio data from a server through a network, or may transmit the generated augmented reality image to an external device by a display device or the like.

The communication unit 310 may include a wired or wireless communication unit. When the communication unit 310 includes a wired communication unit, the communication unit 310 includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), and a mobile communication network ( mobile radio communication network), a satellite communication network, and one or more components for communicating through a combination thereof. In addition, when the communication unit 310 includes a wireless communication unit, the communication unit 310 may transmit data or signals wirelessly using a cellular communication such as 5G or a wireless LAN (eg, Wi-Fi). Can send and receive.

In an embodiment, the processor 320 may control the overall operation of the augmented reality image generating apparatus 300. The processor 320 may execute one or more programs stored in the memory 330, and the image data acquisition unit 131, audio data acquisition unit 132, camera operation determination unit 133, virtual image data of FIG. 2 At least one of the functions of the generation unit 134, the augmented reality image generation unit 135, and the control unit 137 may be performed.

In an embodiment, the processor 320 may control the communication unit 310 to receive motion data and audio data from a server. Here, the motion data may include 3D coordinates of each part of the target object detected at at least one angle, a velocity, acceleration, or moment of inertia of each part of the target object.

In an embodiment, the processor 320 may acquire actual image data for an actual object photographed through at least one camera.

In an embodiment, the processor 320 may generate audio data according to a measurement result of sound generated from an actual object.

In an embodiment, the processor 320 may determine an operation of a camera that actually photographs an object. Here, the photographing operation of the camera may include at least one of a camera photographing position, a camera angle, a zoom in, a zoom out, a focus, and an image change from the first camera to the second camera.

In an embodiment, the processor 320 may generate virtual image data including information on a virtual object based on motion data. Here, the virtual image data may include data on at least one of a display position, angle, size, direction, focus, and depth of the virtual object.

In an embodiment, the processor 320 may synchronize playback times of at least two or more of virtual image data, real image data, audio data about a target object, and audio data about a real object. For example, the processor 320 may synchronize the playback time based on the PTS of each data.

In an embodiment, the processor 320 may generate combined image data in which the virtual object is adaptively matched with the real object according to an operation of at least one or more cameras photographing a real object. The processor 320 may match the actual object by changing at least one of the display position, angle, size, direction, focus, and depth of the virtual object according to a result of determining the operation of the camera.

In an embodiment, the processor 320 may generate an augmented reality image by combining audio data with the combined image data.

In one embodiment, the processor 320 may transmit the generated augmented reality image to an external device (eg, a display device, a server, a terminal, etc.) in real time.

The memory 330 may store various data, programs, or applications for driving and controlling the augmented reality image generating apparatus 300. A program stored in the memory 330 may include one or more instructions. A program (one or more instructions) or an application stored in the memory 330 may be executed by the processor 320. Also, the memory 330 may include the storage unit 230 illustrated and described in FIG. 2, but is not limited thereto.

Meanwhile, the block diagrams of FIGS. 2 and 3 are block diagrams for an embodiment, and each component may be integrated, added, or omitted according to an actual implemented system. That is, if necessary, two or more components may be combined into a single component, or one component may be subdivided into two or more components and configured. In addition, functions performed by each block are for explaining embodiments, and specific operations or devices thereof do not limit the scope of the present invention.

4 is a flowchart illustrating a method of generating an augmented reality image, FIGS. 5 and 6 are flowcharts illustrating an embodiment of step 440 of FIG. 4, and FIG. 7 is a flowchart illustrating an embodiment of step 520 of FIG. 5 to be.

In step 410, the augmented reality image generating apparatus may receive motion data and first audio data regarding a target object. The motion data and the first audio data are separately transmitted from the motion/audio acquisition device and may be separately received through a server. Here, the motion data may include information on the motion of the target object detected at at least one angle, that is, three-dimensional coordinates of each part of the target object, a velocity, acceleration, or moment of inertia of each part of the target object. . In addition, in an embodiment, the motion data and the first audio data may include information about a reproduction time, that is, information about a presentation time stamp (PTS).

In step 420, the augmented reality image generating apparatus may generate virtual image data including information on a virtual object corresponding to the target object by using the received motion data. For example, the augmented reality image generating apparatus may generate virtual image data including information on a shape and motion of a virtual object corresponding to the motion data based on information and motion data of the virtual object previously stored. In an embodiment, the virtual image data may include data on at least one or more of a display position, angle, size, direction, focus, and depth of the virtual object, and according to the embodiment, information on the playback time, that is, , PTS information may be further included. The PTS of the virtual image data may be the same as the motion data.

According to the present invention, since the motion data and audio data are separately transmitted and received to generate the virtual image data and the augmented reality image, the transmission resource is not transmitted and received. It has a reducing effect.

According to an embodiment, the augmented reality image generating apparatus may generate virtual image data based on both motion data and first audio data.

According to embodiments, the virtual image data may be data that is not rendered. For example, in order to broadcast using virtual image data, an additional rendering operation may be required.

In step 430, the augmented reality image generating apparatus may acquire real image data about an actual object photographed through at least one camera and second audio data measured in relation to the real object. In an embodiment, the actual image data and the second audio data may include information about a reproduction time, that is, information about a PTS.

In step 440, the augmented reality image generating apparatus may generate an augmented reality image based on the generated virtual image data, real image data, first audio data, and second audio data.

Referring to FIG. 5, in an embodiment, step 440 may include steps 510 to 540.

In operation 510, the augmented reality image generating apparatus may synchronize at least two or more of the virtual image data, real image data, first audio data, and second audio data.

In one embodiment, step 510 may be performed based on a presentation time stamp (PTS). For example, step 510 may be performed by adjusting the PTS of other data based on the PTS of the data having the slowest playback time among virtual image data, real image data, first audio data, and second audio data. In this regard, referring to FIG. 8, FIG. 8 shows an example of synchronizing reproduction times of virtual image data, real image data, and audio data. As shown, by adjusting the PST of the virtual image data (VVDATA), the actual image data (AVDATA) and/or the first audio data (ADATA) based on the PTS of the second audio data having the slowest reproduction time, Playback times for at least two or more data can be synchronized.

In step 520, the augmented reality image generating apparatus may obtain information on the operation of at least one camera that photographs an actual object. Here, the information on the operation of the camera may include information on at least one of a camera photographing position, a camera angle, a zoom in, a zoom out, a focus, and an image conversion from the first camera to the second camera.

In an embodiment, information on the motion of the camera may be obtained using a marker installed in a photographing space for an actual object. That is, for example, as shown in FIG. 6, the augmented reality image generating apparatus receives detection data for at least one first marker from a detection module installed in each camera (710), and based on the detection data , At least one of a photographing position and an angle of a camera photographing an actual object may be estimated (720 ).

Here, the first marker may refer to at least some of the plurality of second markers installed in at least one area of the photographing space that the detection module detects (or the signal is received from the detection module). In addition, the sensing data may include at least one piece of information about a unique identification signal corresponding to the first marker and a reception strength and detection time of each of the unique identification signals.

In this regard, referring to FIG. 9, FIG. 9 is a diagram illustrating an operation of a sensing module for estimating a position and angle of a camera.

As shown, a plurality of markers are arranged in a certain arrangement or randomly arranged on the ceiling of the photographing space, and a detection module 920 for detecting such markers is installed to face the ceiling in the camera 910 that photographs an actual object. I can.

In one embodiment, the plurality of markers may be Infrared Rays Emitting Diodes (IRDs) that emit unique identification signals such as different wavelengths or frequencies. In addition, in an embodiment, when light of a specific wavelength is projected, the plurality of markers may have retroreflective properties that reflect different unique identification signals, respectively. In this case, the sensing module 920 may be provided with a lighting module (eg, an LED module) for projecting light of a specific wavelength.

When the horizontal position (X, Y), height (Z) or angle (P (pan), R (roll), T (tilt)) of the camera 910 changes, a unique identification signal received by the detection module 920 and Since the intensity of the corresponding signal changes differently, the detection module 920 detects this and transmits it to the augmented reality image generating device, and the augmented reality image generating device is based on this, among the photographing positions and angles of the camera 910. At least one can be estimated.

In an embodiment, the detection module 920 may include a marker detection camera that photographs a direction different from the camera 910, that is, an installation position (eg, a ceiling) of the marker. In this case, the marker detection camera may include a detection filter capable of detecting a unique identification signal such as infrared rays of different wavelengths. Accordingly, when the marker detection camera photographs a marker such as a ceiling, the marker and/or signal strength may be identified from the image of the photographing portion. In this case, the detection data may include an image of the first marker captured by the marker detection camera.

When a marker detection camera is provided, according to an embodiment, the step of synchronizing the captured image of the camera 910 and the captured image of the marker camera may be included.

Again, returning to FIG. 5, in step 530, the augmented reality image generating apparatus adapts the virtual object to the real object according to a photographing operation of at least one camera that photographs the real object based on the virtual image data and the real image data. Combined image data matching with can be generated. That is, for example, the augmented reality image generating apparatus may determine at least one of a display position, an angle, a size, a direction, a focus, and a depth of the virtual object based on the synchronized playback time and the acquired camera operation information. By changing and matching it to the actual object, combined image data can be generated.

According to the present invention, in particular, when there are a plurality of cameras photographing an actual object, the display position, angle, and size of the virtual object are changed according to the photographing position, angle, and camera switching of each camera to match the actual object. By generating image data, it is possible to provide more realistic real-time AR broadcasting.

In step 540, the augmented reality image generating apparatus combines the first audio data related to the virtual object and the second audio data related to the real object to the combined image data generated based on the synchronized playback time information to generate an augmented reality image. Can be generated.

Referring to FIG. 6, step 440 may be performed through steps 610 to 660. Steps 610, 620, 650, and 660 may be performed in the same manner as steps 510 to 540 described above with reference to FIG. 5, and redundant descriptions will be omitted.

In step 630, the augmented reality image generating apparatus may generate at least one virtual camera corresponding to at least one camera that photographs an actual object. Here, the virtual camera is implemented to be adaptively synchronized with the operation of the actual camera in order to convert the display position (coordinate), angle, size, direction, focus, and depth of the virtual object, and as described below, the actual When the operation of the camera changes, it is possible to change the display position and size of the virtual object by adjusting the shooting position and angle of the virtual camera in response thereto.

Meanwhile, in FIG. 6, step 630 is shown to be performed subsequent to step 620, but is not limited thereto, and according to embodiments, step 630 may be performed simultaneously with or before step 620.

Subsequently, in step 640, the augmented reality image generating apparatus adaptively adjusts the operation of the virtual camera in response to the operation of the actual camera, thereby allowing the virtual camera to interwork with the real camera. That is, the virtual camera corresponds to each real camera, and when the position or angle of the real camera is changed, or when switching between cameras occurs, the position or angle of the virtual camera is adaptively changed or the switching between cameras is the same. Can be done.

In step 650, by changing at least one of the display position (coordinate), angle, size, direction, focus, and depth of the virtual object based on the operation of the virtual camera interlocking with the real camera, and displaying it together with the real object, the combination It creates image data.

When the generation of the augmented reality image is completed, the generated augmented reality image may be transmitted in real time to an external device such as a display device or a user terminal (450). Here, the display device is a mobile phone, desktop, laptop computer, tablet PC, digital camera, camcorder, e-book terminal, digital broadcasting terminal, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), navigation, MP3 Various electronic devices such as a player, an IoT device, a wearable device, a CE (Consumer Elctronics) device (eg, a refrigerator having a display panel, an air conditioner, etc.) may be included, but are not limited thereto.

10 illustrates an example in which lyric image data and actual image data are matched according to an operation of a camera in a method of generating an augmented reality image according to an embodiment of the present invention.

In this case, it is assumed that the cup 20, which is an actual object in reality, is photographed using two cameras located at different positions. In this case, the image captured by the two cameras corresponds to a captured image of an actual object, and the augmented reality image generating apparatus may acquire actual image data that is data about this. The two cameras have different shooting angles, and accordingly, different images will be acquired from both cameras.

Thereafter, the augmented reality image generating apparatus may generate the combined images 1010 and 1020 by matching the virtual object generated from the virtual image data with the real object.

Referring to FIG. 10, a rabbit-shaped character 10 as a virtual object is displayed on an image of an actual object acquired by different cameras. The rabbit-shaped character 10 may be an image that reproduces the movement of an actor, which is a target object. In this case, the virtual image data may include motion data indicating the motion of the target object as an actor. As shown in FIG. 10, it can be seen that in the combined image generated based on two cameras photographing an actual object at different shooting angles, the appearance of the rabbit-shaped character 10 is shown differently corresponding to the angle of the camera. have.

According to an embodiment, as the operation of the camera is changed, an image of a virtual object that matches the real object is changed to provide a more realistic image.

11 illustrates an example of an augmented reality image generated by a method of generating an augmented reality image according to an embodiment of the present invention.

Using the augmented reality image generation method according to an embodiment, for example, combining virtual image data with real image data, but according to a change in the shooting position, angle, etc. By changing the object and matching it with the real object, it is possible to obtain an effect that the viewer watching the video feels as if the virtual character exists in the actual studio without feeling of heterogeneity.

A method of operating a system according to an exemplary embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

In addition, a system providing a service-type function or a method of operating the system according to the disclosed embodiments may be included in a computer program product and provided. Computer program products can be traded between sellers and buyers as commodities.

The computer program product may include a S/W program and a computer-readable storage medium storing the S/W program. For example, the computer program product may include a product (e.g., downloadable app) electronically distributed through an electronic device manufacturer or an electronic market (e.g., Google Play Store, App Store). have. For electronic distribution, at least a part of the S/W program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing an SW program.

The computer program product may include a storage medium of a server or a storage medium of a client device in a system composed of a server and a client device. Alternatively, when there is a third device (eg, a smart phone) communicating with a server or a client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a S/W program itself transmitted from a server to a client device or a third device, or transmitted from a third device to a client device.

In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of a server, a client device, and a third device may execute a computer program product to distribute and implement the method according to the disclosed embodiments.

For example, a server (eg, a cloud server or an artificial intelligence server) may execute a computer program product stored in the server to control a client device communicating with the server to perform the method according to the disclosed embodiments.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention defined in the following claims are also the scope of the present invention. Belongs to.

Claims (13)

In a method for generating an augmented reality image for real-time multiple AR broadcasting,
Obtaining motion data and first audio data for the target object, respectively;
Generating virtual image data including information on a virtual object corresponding to the target object by using the received motion data;
Acquiring real image data and second audio data about the real object;
Generating an augmented reality image based on the virtual image data, the real image data, the first audio data, and the second audio data; And
And transmitting the generated augmented reality image in real time. The method of claim 1,
Generating the virtual image data,
A method of generating the virtual image data including information on a shape and motion of a virtual object corresponding to the motion data, based on the information of the virtual object stored in advance and the motion data. The method of claim 1,
The motion data,
A method comprising information on an operation of the target object obtained from at least one angle. The method of claim 1,
The step of generating the augmented reality image,
And synchronizing playback times of at least two or more of the virtual image data, the real image data, the first audio data, and the second audio data based on a presentation time stamp (PTS). The method of claim 4,
The step of synchronizing the playing time,
Synchronizing at least two playback times based on a PTS of data having a late playback time among the virtual image data, the real image data, the first audio data, and the second audio data. The method of claim 1,
The step of generating the augmented reality image,
Based on the virtual image data and the real image data, generating combined image data in which the virtual object adaptively matches the real object according to a photographing operation of at least one camera that photographs the real object. How to. The method of claim 6,
The step of generating the augmented reality image,
Further comprising the step of obtaining information on the operation of the at least one or more cameras photographing the actual image,
Generating the combined image data,
And changing at least one of the display position, angle, size, direction, focus, and depth of the virtual object according to an operation of the at least one camera to match the actual object. The method of claim 7,
Information on the operation of the at least one camera,
A method comprising information about at least one of a camera shooting position, a camera angle, a zoom in, a zoom out, a focus, and an image transition from the first camera to the second camera. The method of claim 7,
Acquiring information on the operation of the camera,
Receiving detection data for at least one first marker from a detection module installed in each of the at least one camera; And
Estimating at least one of a photographing position and an angle of the camera photographing the real object based on the sensing data,
The first marker is at least a part of a plurality of second markers installed at different locations in at least one area of the shooting space of the real object. The method of claim 9,
The detection data includes information on at least one of a unique identification signal corresponding to each of the first markers and a reception strength of each of the unique identification signals. The method of claim 9,
The detection module includes a marker detection camera for photographing a direction different from the camera,
The detection data includes an image of the first marker captured by the marker detection camera. In an augmented reality image generating apparatus for real-time multi-source AR broadcasting,
Communication department;
A memory for storing a program for generating the augmented reality image; And
By executing the program stored in the memory, motion data and first audio data of a target object are obtained, respectively, and information on a virtual object corresponding to the target object is included using the received motion data. Generate virtual image data, acquire real image data and second audio data about a real object, and an augmented reality image based on the virtual image data, the real image data, the first audio data and the second audio data And a processor that generates and transmits the generated augmented reality image in real time. A computer-readable recording medium in which a program for executing the method of claim 1 on a computer is recorded.

Images