WO2019177181A1 - 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치, 제공 방법 및 상기 방법을 실행하기 위하여 매체에 저장된 컴퓨터 프로그램 - Google Patents
뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치, 제공 방법 및 상기 방법을 실행하기 위하여 매체에 저장된 컴퓨터 프로그램 Download PDFInfo
- Publication number
- WO2019177181A1 WO2019177181A1 PCT/KR2018/002868 KR2018002868W WO2019177181A1 WO 2019177181 A1 WO2019177181 A1 WO 2019177181A1 KR 2018002868 W KR2018002868 W KR 2018002868W WO 2019177181 A1 WO2019177181 A1 WO 2019177181A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- situation
- information
- augmented reality
- neural network
- Prior art date
Links
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 83
- 238000000034 method Methods 0.000 title claims abstract description 83
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 41
- 238000004590 computer program Methods 0.000 title claims description 10
- 238000009877 rendering Methods 0.000 claims abstract description 62
- 230000008859 change Effects 0.000 claims abstract description 35
- 230000008569 process Effects 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims description 60
- 230000015654 memory Effects 0.000 claims description 24
- 239000000284 extract Substances 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 8
- 230000006870 function Effects 0.000 description 13
- 239000013598 vector Substances 0.000 description 11
- 238000010586 diagram Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 6
- 230000003111 delayed effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000005484 gravity Effects 0.000 description 3
- 101100218322 Arabidopsis thaliana ATXR3 gene Proteins 0.000 description 2
- 102100029768 Histone-lysine N-methyltransferase SETD1A Human genes 0.000 description 2
- 102100032742 Histone-lysine N-methyltransferase SETD2 Human genes 0.000 description 2
- 101000865038 Homo sapiens Histone-lysine N-methyltransferase SETD1A Proteins 0.000 description 2
- 101100149326 Homo sapiens SETD2 gene Proteins 0.000 description 2
- LZHSWRWIMQRTOP-UHFFFAOYSA-N N-(furan-2-ylmethyl)-3-[4-[methyl(propyl)amino]-6-(trifluoromethyl)pyrimidin-2-yl]sulfanylpropanamide Chemical compound CCCN(C)C1=NC(=NC(=C1)C(F)(F)F)SCCC(=O)NCC2=CC=CO2 LZHSWRWIMQRTOP-UHFFFAOYSA-N 0.000 description 2
- 101100533304 Plasmodium falciparum (isolate 3D7) SETVS gene Proteins 0.000 description 2
- 101150117538 Set2 gene Proteins 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002194 synthesizing effect Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000005358 geomagnetic field Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 235000011888 snacks Nutrition 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/254—Analysis of motion involving subtraction of images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/211—Selection of the most significant subset of features
- G06F18/2113—Selection of the most significant subset of features by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
- G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V10/422—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation for representing the structure of the pattern or shape of an object therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/758—Involving statistics of pixels or of feature values, e.g. histogram matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/147—Scene change detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/759—Region-based matching
Definitions
- the present invention relates to an apparatus for providing augmented reality that recognizes a situation using a neural network, a providing method, and a computer program stored in a medium for executing the method.
- Augmented Reality is a field of virtual reality that is a computer graphics technique that synthesizes virtual objects or information in the real environment and looks like objects existing in the original environment. It is also called Mixed Reality (MR) because it combines the virtual world with additional information in real time into one image.
- MR Mixed Reality
- a process of adding a virtual object to each frame of the captured image needs to be performed every frame of the image, and thus requires more processing power of the processor depending on the resolution, frame rate, etc. of the captured image.
- Embodiments of the present invention may provide an image reflecting an analysis result in real time using a neural network while providing augmented reality to an image acquired through an image sensor in a computing device of limited performance.
- a method for providing augmented reality that recognizes a situation using a neural network may include obtaining an image by a processor; Rendering, by the processor, the image to analyze the image and to place a virtual object on a plane included in the image; Determining whether to process a situation of the image based on a determination criterion including comparing a current frame included in the image with a previous frame to determine whether to change a scene of the current frame; If it is determined whether the situation recognition processing is true, calculating one or more situation information by analyzing a sensing value received from the image and / or sensor unit using a neural network; And generating additional content to which the contextual information is applied and providing the additional content.
- the determination criteria may further include determining whether the processing capability of the processor exceeds a threshold value.
- the determination criteria may further include determining whether the output frame rate of the image is less than the average frame rate.
- Determining whether or not to switch the scene may include determining a scene change by calculating a distribution chart in which pixels of the current frame are distributed according to color values, and comparing the distribution map with a distribution map of the previous frame.
- the determining of whether to change the scene may include extracting a color and a shape of an object by synthesizing the edge information by using the edge information detected in the current frame, and determining whether to change the scene using the color and the shape of the object. have.
- the calculating of the contextual information may be obtained by limiting the kind of contextual information calculated in response to the input user input according to the user input.
- the image may be image data photographed by a camera or generated and output by another application.
- the providing of the additional content may include providing content including the contextual information, or generating a speech utterance content using the contextual information through the virtual object.
- the calculating of the one or more situation information may be performed in the form of a probability map further including a matching probability for each situation information, and ranking and transferring the one or more situation information based on the matching probability for each situation information. have.
- a memory including instructions executable by one or more processors in accordance with embodiments of the present invention; And one or more processors coupled to the memory and capable of executing instructions, wherein the memory acquires an image, analyzes the image, and places the virtual object on a plane included in the image.
- a rendering module for rendering a;
- a determination module for determining whether to process a situation of the image based on a criterion for determining whether to change a scene of the current frame by comparing a current frame included in the image with a previous frame;
- a situation recognition module for analyzing one or more situation information by analyzing a sensing value received from the image and / or sensor unit by using a neural network when it is determined whether the situation recognition process is true;
- an editing module that generates additional content to which the contextual information is applied and provides the additional content.
- the determination criterion may further include determining whether the processing capability of the processor exceeds a threshold.
- the determination criteria may further include determining whether an output frame rate of the image falls below an average frame rate.
- Determining whether or not to switch the scene may include determining a scene change by calculating a distribution chart in which pixels of the current frame are distributed according to color values, and comparing the distribution map with a distribution map of the previous frame.
- the determining of whether to change the scene may include extracting a color and a shape of an object by synthesizing the edge information by using the edge information detected in the current frame, and determining whether to change the scene using the color and the shape of the object. have.
- the contextual awareness module may acquire and limit the type of contextual information calculated in response to the input user input according to the user input.
- the image may be image data photographed by a camera or generated and output by another application.
- the editing module may provide content including the contextual information, or generate voice speech content in a dialogue format using the contextual information and provide the content via the virtual object.
- the situation recognition module may calculate the form of a probability map further including a matching probability for each situation information, and rank and transmit the one or more situation information based on the matching probability for each situation information.
- a computer program according to an embodiment of the present invention may be stored in a medium for executing any one of the methods according to the embodiment of the present invention using a computer.
- a computer readable recording medium for recording another method for implementing the present invention, another system, and a computer program for executing the method.
- a method for simultaneously performing augmented reality and deep learning based image processing in a limited performance terminal and a computer program stored in a medium for executing the method may perform image processing based on a neural network.
- By adjusting the interval or frequency it is possible to prevent screen output delay due to lack of computing resources of the computing device and to lower the power consumption of the computing device.
- a method for simultaneously performing augmented reality and neural network-based image processing in a limited-performance terminal and a computer program stored in a medium for executing the method may be applied to a thread that provides augmented reality.
- a method for simultaneously performing augmented reality and neural network-based image processing in a limited-performance terminal and a computer program stored in a medium for executing the method may be applied to a thread that provides augmented reality.
- FIG. 1 is a view showing the structure of a system according to embodiments of the present invention.
- FIG. 2 is a block diagram illustrating a structure of an augmented reality application that recognizes a situation using a neural network according to embodiments of the present invention.
- FIG. 3 is a flowchart of a method for providing augmented reality for recognizing a situation using a neural network according to an embodiment of the present invention.
- FIG. 4 is a flowchart for describing an operation of a rendering module.
- 5 is a flowchart for explaining an operation of a determination module.
- FIG. 6 is a diagram for describing a process of processing a plurality of modules by a limited processor.
- 7 to 9 are diagrams for explaining an operation of comparing a current frame and a previous frame by the determination module.
- 10A to 10D are diagrams for describing embodiments of using contextual information acquired through the contextual awareness module.
- a neural network may be a set of algorithms that identify and / or determine objects in an image by extracting and using various attributes in the image by using a result of statistical machine learning.
- the neural network can identify objects in the image by abstracting various attributes contained in the image input to the neural network. In this case, abstracting the attributes in the image may be to detect the attributes from the image and determine a core attribute among the detected attributes.
- the augmented reality providing apparatus disclosed herein may be implemented in hardware or a combination of software and hardware.
- they can be implemented as an operating system kernel, as a separate user process, as a library package defined within network applications, on a specially configured machine, or on a network interface card.
- the techniques disclosed herein may be implemented in software, such as an operating system, or in an application running on an operating system.
- the software / hardware hybrid implementation (s) of at least some of the augmented reality providing embodiment (s) disclosed herein may be implemented on a programmable machine selectively activated or reconfigured by a computer program stored in memory. Can be.
- Such network devices may have multiple network interfaces that may be configured or designed to utilize different types of network communication protocols. General architecture for some of these machines may appear from the descriptions disclosed herein.
- at least some of the features and / or functions of the various augmented reality providing embodiments disclosed herein may be used in connection with an end user computer system, computer, network server or server system, mobile computing device (eg, personal digital).
- Computing device 100 may be, for example, an end user computer system, a network server or server system, a mobile computing device (eg, a smartphone, laptop, tablet computer, or the like), a consumer electronic device, a music player, or any other Suitable electronic devices, or any combination or part thereof.
- Computing device 100 may be configured to communicate with other computing devices, such as clients and / or servers, via a communication network, such as the Internet, using known protocols for such communication, whether wireless or wired. .
- computing device 100 includes a central processing unit (CPU) 110, a memory 130, an input / output device 140, and a camera 150.
- the CPU 110 may be responsible for implementing specific functions associated with the functions of a specially configured computing device or machine.
- the user's terminal 100 functions as an electronic device utilizing the CPU 110, the memory 130, the input / output device (I / O, 140), and the camera 150. It may be configured or designed to.
- CPU 110 is a neural rendering of a virtual object in augmented reality under the control of software modules / components, which may include, for example, an operating system and any suitable application software, drivers, and the like. Network-based context awareness, and one or more of the functions and / or actions.
- CPU 110 may include one or more processor (s) 110, such as, for example, a processor from Qualcomm or Intel based microprocessors or MIPS based microprocessors.
- the CPU 110 may include a GPU that is a processor for graphics processing.
- processor (s) 110 may be specially designed hardware (eg, application-specific integrated circuit (ASIC), electrically erasable programmable read-only memory) to control operations of computing device 100.
- memory 130 eg, nonvolatile RAM and / or ROM
- Memory block 130 may be used for a variety of purposes, for example, caching and / or storing data, programming instructions, and the like.
- one CPU 110 is illustrated in FIG. 1, one or more CPUs 110 may be provided without being limited thereto.
- processor is not limited to only those integrated circuits referred to in the art as processors, but are broadly microcontrollers, microcomputers, programmable logic controllers, ASICs, and any other Refers to a programmable circuit.
- the interfaces 160 are provided as interface cards. In general, they control the transmission and reception of data packets over the computing network and sometimes support other peripherals used with the computing device 100.
- the interfaces that can be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, and the like.
- USB universal serial bus
- serial serial
- Ethernet Firewire
- PCI parallel
- radio frequency RF
- BluetoothTM near field communication
- 802.11 WiFi, Frame Relay, TCP / IP, ISDN, Fast Ethernet Interface, Gigabit Ethernet Interface, Asynchronous Transfer Mode (ATM) Interface, High-speed Serial Interface (HSSI) Interface, Point of Sale Interface, Fiber Data (FDDI)
- ATM Asynchronous Transfer Mode
- HSSI High-speed Serial Interface
- FDDI Fiber Data
- Various types of interfaces may be provided, such as a distributed interface.
- such interfaces 160 may include ports suitable for communicating with the appropriate medium.
- they may also include an independent processor, and in some cases may include volatile and / or nonvolatile memory (eg, RAM).
- FIG. 1 illustrates one specific architecture for the computing device 100 for implementing the techniques of the invention described herein, but it is never at least some of the features and techniques described herein. Is not the only device architecture that can be implemented. For example, architectures with one or any number of processors 110 may be used, and such processors 110 may exist in a single device or may be distributed among any number of devices. In one embodiment, processor 110 handles communications as well as routing operations. In various embodiments, different types of contextual information generating features and / or functions may be used in a client device (eg, a personal smartphone running client software) and server system (s) (eg, a server system described in more detail below. ) May be implemented in a contextual information generation system that includes.
- client device eg, a personal smartphone running client software
- server system eg, a server system described in more detail below.
- the system of the present invention is configured to store (eg, store data, program instructions for general purpose network operations, and / or other information about the functionality of the contextual information generating techniques described herein.
- One or more memories or memory modules (such as memory block 65).
- Program instructions may, for example, control the operation of an operating system and / or one or more applications.
- the memory or memories may also be configured to store data structures, advertisement information, user click and impression information, and / or other specific non-program information described herein.
- Non-transitory machine readable storage media that can be configured or designed to store state information, and the like. Examples of such non-transitory machine readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; Optical media such as CD-ROM disks; Magneto-optical media, such as floptical disks, and hardware devices specifically configured to store and perform program instructions, such as ROM, flash memory, memristor memory, RAM, etc. It is not limited to. Examples of program instructions include, for example, both machine code generated by a compiler and files containing higher level code that can be executed by a computer using an interpreter.
- the augmented reality application 200 may be included in the memory 130 of the computing device 100 and may be performed by the control of the processor 110.
- the augmented reality application 200 may provide an augmented reality that displays an object in real time by overlaying an object on an image photographed by a camera.
- the augmented reality provided according to an embodiment recognizes a plane that is actually present around the computing device 100 and determines the properties of the recognized plane, thereby real-time the virtual object along with the image taken by the camera in real time. I can display it.
- the augmented reality application 200 may be implemented in hardware or software. Some of the components included in the augmented reality application 200 may be implemented in hardware or software, respectively.
- the augmented reality application 200 may include a rendering module 210, a determination module 220, a context aware module 230, an editing module 240, and an output control module 250. have.
- the augmented reality application 200 may receive an image photographed through a camera, a voice input through a microphone, a sensing value obtained through a sensor unit, and the like. In this case, the augmented reality application 200 may convert a voice input through a microphone into text using a voice recognition technology.
- the augmented reality application 200 may analyze the captured image using vision recognition technology.
- the type of space included in the image, the 3D model, the type and position of the object, the position and the feature point of the face, the feature vector, etc. may be derived using the vision recognition technology.
- the augmented reality application 200 may derive the current location information by using the location information obtained through the GPS module.
- the augmented reality application 200 may obtain context information on the user's environment by comprehensively considering the voice, the image, and the sensing value.
- the rendering module 210 performs a function of applying augmented reality to input images and images.
- the rendering module 210 may render the virtual object on the image.
- the rendering module 210 may obtain distances from the sensor unit (distance sensor, direction sensor, etc.) to a plurality of front points, respectively.
- the rendering module 210 obtains respective distances and directions to a plurality of points on the plane when there is a plane in front, and defines plane information including plane information including the plurality of points, for example, plane information. , Plane vectors and the like can be obtained.
- the sensor unit 120 may be an infrared sensor and emit infrared light and receive reflected infrared light to measure a distance to a point.
- the sensor unit 120 may be an ultrasonic sensor, and after receiving the ultrasonic waves, receive the reflected ultrasonic waves and measure the distance to the point where the ultrasonic waves are reflected.
- the sensor unit 120 may be an acceleration sensor, an angular velocity sensor, a geomagnetic field sensor, or a combination of at least two of them.
- the sensor unit 120 may recognize the direction of gravity and measure a direction in which the terminal is directed based on the direction of gravity.
- the rendering module 210 may render the virtual object on the image by using the image photographed by the camera and the plane information acquired through the sensor unit.
- the virtual object is an object implemented to be perceived visually or auditoryly, and there is no limitation in format, such as an image, text, or sound.
- the virtual object may be arranged and represented in the obtained plane.
- the virtual object may be called by voice input or touch input of the user.
- the rendering module 210 may generate and render additional virtual objects determined by a user's voice and touch input.
- the rendering module 210 may generate and render one or more virtual objects in every frame according to the frame rate of the image.
- the processing speed of the rendering module 210 may depend on the processing power of the processor. For example, if the processing power of the processor is increased, the processing speed of the rendering module 210 increases by a percentage of the increased processing power of the processor, and if the processing power of the processor decreases, the processing speed of the rendering module 210 is increased. Decreases by a percentage of the reduced processor's processing power. In another embodiment, the processing speed of the rendering module 210 may be increased or decreased depending on the currently working thread of the computing device 100. For example, the rendering module 210 may operate using the processing power other than the processing power of the processor allocated to the one or more threads currently being worked on.
- the determination module 220 may perform a function of determining whether the situation awareness module is executed, that is, the execution time of the image to which the augmented reality is applied. The determination module 220 may determine whether to execute the situation awareness module in order to prevent output delay of an image due to a lack of processing power of the processor due to simultaneous execution with the rendering module 210. Through this, the execution frequency of the situation awareness module may be reduced. The determination module 220 may determine whether to execute the situation awareness module using the input data, that is, the image, the input voice, and the location information. The determination module 220 infers the processing capability of the processor to be used for the processing of the input data using the input data, and determines whether there is no processing speed delay due to the execution of the situation recognition module within the processing capability of the processor. You can judge. The determination module 220 may determine whether to execute the context aware module using the processing power of the processor to be used for the processing of the input data.
- the determination module 220 may determine whether to execute the context awareness module based on the determination criteria including whether to change the scene of the image acquired using the input data.
- the determination criteria may determine whether to apply the context awareness module to the current frame by comparing the current frame with a frame previously acquired in time (hereinafter, referred to as a previous frame).
- the determination module 220 may use a comparison process between the current frame and the previous frame to determine whether to change the scene.
- the determination module 220 may determine whether the scene is changed by calculating the first histogram of the current frame and the second histogram of the previous frame, and comparing the calculated first and second histograms. have.
- the histogram may be a distribution map in which pixels of the current frame of the image are distributed according to color values. Specifically, the number of pixels having the first color value in the current frame is very small or less than a certain ratio (for example, 50%) or more than the specific ratio (150%) compared to the number of pixels having the first color value in the previous frame. If large, it may be determined that a scene change has occurred in the current frame. By comparing the number of pixels within the first range through the first histogram of the current frame and the number of pixels within the second range via the second histogram, it may be determined that the current frame has a different scene from the previous frame.
- the determination module 220 may determine whether to change scenes using the color and shape of the object detected in the current frame of the image.
- the determination module 220 may obtain edge information in the current frame, and extract color information and shape information of the object by combining the edge information.
- the information on the object in the current frame is not the same as the information on the object in the previous frame, it may be determined that a scene change has occurred.
- Correspondence between objects present in the frames may be determined using a SIFT algorithm.
- the SIFT algorithm may be used to calculate information that does not change even when the size, position, and orientation of each object change.
- the determination criteria including whether to change the scene may be determined using the analysis result of the image, the direction information and the motion information obtained through the sensor unit of the computing device.
- the direction information and motion information of the computing device may be used to three-dimensionally predict the direction of the computing device and determine whether a scene change occurs. For example, when the direction information or motion information of the computing device differs from the previously obtained direction information or motion information by more than a preset threshold, it is determined that the direction or position of the computing device is physically changed, and the determination result It may be determined whether or not the scene transition occurs based on.
- the determination module 220 may determine whether the situation recognition processing is true, and request to calculate the situation information of the image through the situation recognition module.
- the determination module 220 may determine whether to process the situation in consideration of the processing power of the processor.
- the determination module 220 monitors the processor's processing power and the remaining power minus the processing power in the total capacity, and when the processing power of the processor used by a thread, a program, or the like executed exceeds a preset threshold, It may be determined that the remaining capability is not sufficient to execute the context aware module and determine whether to process the context aware as FALSE.
- the determination module 220 may determine whether the situation recognition processing is performed in consideration of the image output speed, that is, the frame rate of the image output. If the image output speed is processed without being delayed, it may mean that the processor or the memory is not overloaded. For example, detecting that the frame rate of the image output is lower than the average frame rate may mean that the computing device is overloaded, and in this case, the determination module 220 determines whether the situation recognition process is FALSE. can do.
- the determination module 220 aggregates input data, whether a scene is changed, performance of a processor, processing capability, and output data, that is, image output speed. It may perform a function of determining whether or not, and execute the situation awareness module 230 only at a selected time point.
- the determination module 220 may transmit a signal for requesting the current frame and the context information to the context awareness module and receive a response thereto.
- the determination module 220 may not perform a process of acquiring situation information when the situation recognition process is false.
- the determination module does not perform a call of the situation awareness module 230 when the situation awareness processing is false.
- the situation awareness module 230 may be executed together with the rendering module 210 by one or more processors.
- the operating system (OS) of the computing device may perform appropriate scheduling for executing the context aware module 230 and the rendering module 210.
- Residual power may refer to processing power that is not used or allocated except for the processing power allocated to programs running on the computing device.
- the situation recognition module 230 is executed by the determination module 220, analyzes a frame and / or sensing value using a neural network, calculates situation information using the analyzed result, and determines the situation information. 220).
- the context awareness module 230 may calculate output data called context information by classifying one or more factors included in a frame and / or a sensing value as input data. In this case, one or more factors included in the frame and / or the sensing value as the input data and the context information as the output data may be learned as a set. Classifying one or more factors included in a frame and / or sensing value that is input data may utilize a connection between one or more factors and output data included in the input data input during the learning process.
- the input data may be connected with hidden data for deduction of the output data.
- the hidden data is not included in the input or output, but refers to a factor or data used in the classification process.
- the output data corresponding to the input data may be calculated using a connection between at least two of the learned input data, hidden data, and output data.
- the input data according to the present embodiment may be various types such as a frame, a sensing value, sound data, and the like.
- the output data according to the present embodiment may be context information including one or more attribute information, and may vary according to input data and / or data requested by a user.
- the neural network used in the context aware module 230 may be pre-trained and generated by an external server, and may be continuously trained and updated through input data and output data requesting context awareness. As the amount of input data increases, the accuracy of the context information acquired using the neural network may increase.
- the connection relationship between at least two of the input data, the hidden data, and the output data included in the neural network may be nonlinear.
- the first connection between the first data and the second data may be set as 0.3, not 0 or 1.
- the second connection between the first data and the third data is set to 0.7 so that the first data can be classified as either second data or third data at a ratio of three to seven.
- the contextual awareness module 230 is called and executed by the determination module 220 and processes each attribute information based on a neural network, and corresponds to a sound data input through a frame and a microphone included in an image.
- the above situation information can be calculated and provided.
- the context aware module 230 may be included in the augmented reality application, may be included in another application, an operating system, or may be included in an external device.
- the determination module 220 may transmit / receive data with the situation awareness module 230 through a network.
- the contextual information may include various attribute information generated around the computing device.
- contextual information may include information about nearby places (name, location, route, etc.), information about things recognized through computing devices (names, product names, details, etc.), and people recognized through computing devices. It may include information about information (such as age, feelings, and similarity).
- the context information may be obtained from at least one of an image photographed using a neural network, a sensing value, and sound data.
- the editing module 240 may generate additional content applying the context information of the current frame by using the neural network.
- the additional content may be graphic data such as a character, a place, a building, an effect set by a user. Specific examples of the additional content will be described with reference to FIGS. 10A, 10B, 10C, and 10D.
- the output control module 250 controls to output image data received through the rendering module 210 and the editing module 240.
- an augmented reality application may generate situation information corresponding to an input image, a sensing value, and sound data while providing augmented reality that adds graphic data to an image under limited computing performance.
- the augmented reality application can prevent overload due to limited computing performance by adjusting the generation frequency of the situation information.
- the augmented reality application may calculate situation information corresponding to the input data even when the communication network is disconnected without using external resources.
- FIG. 3 is a flowchart of a method for providing augmented reality for recognizing a situation using a neural network according to an embodiment of the present invention.
- the augmented reality application acquires an image.
- the image may be taken by a camera in real time.
- the image may be various frame sets generated by other applications.
- the augmented reality application may render a virtual object on an image.
- the detailed operation of this step will be described in FIG.
- the augmented reality application may downsample the image.
- the augmented reality application can convert the image to multiple scales.
- An image converted to a plurality of scales may be a scale space below.
- the augmented reality application determines whether to change scenes in the down-sampled image. Since the operation of S140 is the same as that of the determination module, a detailed description thereof will be omitted.
- the augmented reality application may request context information of the input data from the context awareness module.
- the augmented reality application may edit the image using the received situation information.
- the augmented reality application may further provide visual data using contextual information.
- an editing method of an image may vary according to the type of contextual information received. An image editing method will be described with reference to FIGS. 10A to 10D.
- FIG. 4 is a flowchart for describing an operation of a rendering module.
- the rendering module calculates plane information about a plane including a plurality of points by using the distances of the plurality of points.
- the rendering module may calculate plane information about a plane including a plurality of points using directions and distances to the plurality of points.
- the rendering module obtains a normal vector of the plane using direction information of the terminal measured by the sensor unit and plane information obtained in operation S121.
- the rendering module acquires the direction information of the terminal measured by the direction sensor, and obtains the normal vector of the plane by using the previously obtained plane information and the direction information of the terminal.
- the rendering module converts the reference direction of the plane information from the direction of the terminal (or the direction of the distance sensor installed in the terminal) to the reference direction of the direction sensor.
- the rendering module moves the direction information of the plane obtained based on the direction of the terminal by the direction information of the terminal measured by the direction sensor (based on the gravitational direction recognized by the acceleration sensor), thereby finally the gravity direction.
- the direction information of the gravitational direction reference plane thus obtained may be regarded as a normal vector of the plane.
- the rendering module determines parameters of the virtual object in consideration of the normal vector of the plane.
- the rendering module considers the normal vector of the plane and determines the parameters of the virtual object to display in the plane.
- the object may include a plurality of parameters.
- the parameters may be, for example, the color, tilt, category, type, direction and animation of the virtual object.
- the rendering module may set the tilt of the virtual object to correspond to the direction of the normal vector of the plane.
- the rendering module may classify the attributes of the plane in consideration of the normal vector of the plane. Also, the rendering module may determine the parameter of the virtual object in consideration of the property of the plane.
- the property of a plane is, for example, a type of plane, which can be divided into floors, walls, and ceilings.
- the rendering module may determine parameters of the object differently according to whether the plane is a floor, a wall, or a ceiling. For example, the rendering module may set an animation parameter so that the object moves along a path through the plane when the plane is a wall.
- the rendering module may set animation parameters that move parallel to the floor from the floor when the plane is the floor.
- the rendering module may determine the parameters of the virtual object in consideration of the properties of the plane determined according to the normal vector of the plane and the direction information of the plane (based on the direction of the terminal). For example, the rendering module may determine the first parameter of the object according to the property of the plane and determine the second parameter of the virtual object according to the normal vector of the plane. For example, the rendering module may determine the color of the object according to the property of the plane and determine the inclination of the virtual object according to the direction of the plane. When the object is an icon, the rendering module may determine the color of the icon differently according to the type of plane, and determine the inclination of the icon according to the direction of the plane. The inclination may be an inclination for rendering and displaying three-dimensional icon display information in two dimensions, or may be a horizontal / vertical display ratio of an icon.
- the rendering module displays the object through the connected input / output device.
- the rendering module extracts a plane from the image photographed by the camera and places a virtual object on the plane.
- the display direction of the virtual object may be for rendering and displaying three-dimensional furniture display information in two dimensions, but is not limited thereto and may be for rotating and / or scaling and displaying two-dimensional furniture display information.
- the rendering module applies the animation to display the virtual object. For example, if you display a virtual object that is animated through a wall against the plane of the “Wall” property, the render module overlays the plane with an opaque virtual layer for the object, and forwards the virtual object behind the virtual layer. By moving and displaying, an animation that looks like a virtual object pierces a plane is displayed.
- the virtual layer may be set to be transparent for display other than the virtual object.
- the rendering module may set a virtual object displaying an image of a sun, a moon, and a star with respect to a plane of a “ceiling” attribute.
- the rendering module may set a virtual object moving between a plurality of planes. For example, you can set a droplet object that moves from the plane of the "ceiling" attribute toward the plane of the "floor” attribute.
- the droplet object may include an animation that scatters in the same direction as the plane when the plane reaches the "bottom" property.
- 5 is a flowchart for explaining an operation of a determination module.
- the determination module 220 performs a function of determining a time point of obtaining situation information using the situation recognition module. Through the determination module, the image processing method according to an embodiment of the present invention can efficiently manage the processing capability of the processor in adding a virtual object to the captured image and simultaneously providing additional content according to the situation information of the captured image. .
- the determination module 220 selects a current frame from a frame set included in the image.
- the determination module 220 may determine whether a scene change occurs in the current frame by comparing the current frame with the previous frame. If the number of pixels according to the color value is different from the number of pixels according to the corresponding color value of the previous frame by using the histogram which is the distribution of the color values of the current frame, it is determined whether or not the scene transition occurs in the current frame. Can be.
- the determination module 220 extracts the shape and color of the appearance object of the current frame, determines whether the appearance object of the current frame and the appearance object of the previous frame are the same, and if the appearance objects are the same, a scene change occurs in the current frame. You can decide whether it is true or not.
- the determination module 220 may reduce the execution frequency of the context awareness module so as to smoothly implement and output the augmented reality content according to the present embodiment by using the limited processing power of the processor.
- FIG. 6 is a diagram for describing a process of processing a plurality of modules by a limited processor.
- the first module module 1 is executed by the processor for the first time t1, the executed result is inputted and outputted for the second time t2, and again executed by the processor for the third time t3, and executed.
- the result is inputted and outputted for the fourth time t4.
- each module may be performed by crossing the process processed by the processor and output by the input / output device. As shown in FIG. 6 (b), the input / output of each module may not use the processing power of the processor.
- the processor may be processed by the context aware module (module 2). Thereafter, after the input / output time of the rendering module passes, the processor processes the rendering module (module 1). At this time, the processing of the rendering module 1 may be delayed by tB time for the execution of the context aware module 2.
- the determination module 220 determines whether the situation awareness module 2 is executed at the time when the rendering module 1 is input and output, and executes the situation awareness module 2 only when execution of the determination result is necessary. You can. In the case in which the situation awareness module (module 2) is not executed according to the judgment of the decision module, execution of the rendering module (module 1) may not be delayed.
- the determination module 220 may prevent the execution of the contextual awareness module from being delayed or the execution of the rendering module due to the processing of the contextual awareness module.
- the determination module 220 may determine whether it is time to execute the context aware module, thereby preventing the processing of the modules to be executed due to the limited performance of the computing device.
- 7 to 9 are diagrams for explaining an operation of comparing a current frame and a previous frame by the determination module.
- the captured image may be classified into a first frame set SET1 and a second frame set SET2 in a predetermined time interval.
- the determination module 220 may arbitrarily select the first frame f1 of the first frame set SET1 and arbitrarily select the second frame f2 of the second frame set SET2.
- the determination module 220 generates a distribution chart H1 based on the color values of the pixels of the first frame f1.
- the first distribution chart H1 counts the pixels of the first frame f1 according to color values, and graphically displays the number of pixels for each color value.
- the second distribution H2 is also generated in the same way. As shown in FIG. 7, even when the color value is different, it may be determined that there is no scene change when the distribution patterns of the first distribution map H1 and the second distribution map H2 are compared or matched within 10%.
- the determination module 220 may extract and compare an area of a frame in order to reduce the amount of data to be processed.
- the determination module 220 may determine whether to switch scenes between the first frame and the second frame by comparing one region of the first frame f1 and one region of the second frame f2.
- the determination module 220 extracts the edge of the first frame, extracts the edge of the second frame, and compares the edges extracted from the frames, thereby changing the scene between the first frame and the second frame. Can be determined.
- the determination module 220 obtains a rectangle edge 1 of the appearance object from the edge of the first frame and obtains a cylinder edge 2 of the appearance object from the edge of the second frame and shapes the appearance objects of the frames. By comparing them, it is possible to determine whether to switch scenes between the first frame and the second frame. That is, the determination module 220 may determine that the scene is switched between the first frame and the second frame.
- 10A to 10D are diagrams for describing embodiments of using contextual information acquired through the contextual awareness module.
- the augmented reality application may render augmented reality on an image provided through an input / output device.
- the augmented reality application may recognize a plane present in the captured image and place a virtual object on the plane.
- the augmented reality application may generate and render the obtained plane as a virtual object obj 2, and place a virtual object obj 1 that is a character in the plane.
- the virtual object obj 1 may be generated in response to a preset command signal. For example, when a predetermined command signal “nawara” is input in the form of voice or text, the virtual object obj1 may be rendered.
- the augmented reality application may acquire and provide age, emotion, and similar shape information as situation information on a human face existing in an image.
- the augmented reality application may use vision recognition technology to determine if there is a human face in the captured image.
- the human face areas a2 and a3 may overlap with a rectangle.
- the augmented reality application can make interaction with the virtual object in response to the detection of the human face. For example, in response to the detection of a human face, a virtual object may look for additional content such as' how old do you look? '(A4)', 'I wonder how it feels (a5)' Can be provided by voice.
- the augmented reality application may call the context awareness module and receive the context information according to the selection input for the additional contents. According to the selection input for a4, the augmented reality application may acquire the age of the person calculated through the context awareness module as a probability map. The augmented reality application may output an age and a probability value of the age included in the probability map as a text or a voice of a virtual object. Mood information or similar information may also be obtained through the context awareness module.
- the augmented reality application may provide a name (a7) of the object included in the image in the form of text, image, voice, etc. based on the contextual information obtained through the contextual awareness module.
- the augmented reality application may communicate with a user based on context information acquired through the context awareness module.
- the augmented reality application uses the contextual information called jajangmyeon and the virtual object is called 'jajangmyeon ! It will be delicious.
- the conversation may be made using a place, an object, etc. included in the context information.
- the virtual object may utter a conversation about an area or a location.
- the augmented reality application learns a person's face from the video, and if it detects a person's face more than once, it reflects this situational information and is “nice to see”. I meet often. '
- the virtual object When situation information, such as an obstacle appearing in the captured image, is calculated, the virtual object may be generated to display a surprising expression.
- the virtual object provided by the augmented reality application may be set to have a specific function.
- the virtual object may be set by the user as a 'secretary' that provides output according to the user's command.
- the augmented reality application may generate necessary information based on reality meta information such as vision analysis information, location information, and weather information acquired through an image, and provide the necessary information through a virtual object set as an 'secretary'.
- the augmented reality application may augment a game character set by a user into a virtual object.
- the virtual object that is a game character may be implemented to act according to the age, gender, hobby, disposition, etc. of the preset game character. For example, when a 'cafe' is output from the captured image, the virtual object may be implemented to reflect the output of the 'cafe' to utter a voice for ordering a favorite drink 'coffee' according to the propensity in the game. Can be.
- the augmented reality application may augment the pet set by the user into a virtual object.
- the virtual object set as a pet such as a dog may be implemented to respond to a 'ball' or a 'snack' included in the captured image according to the characteristics of the pet.
- the virtual object may generate various conversations according to context information acquired through the captured image through linkage with another application.
- the virtual object may provide a dialogue according to the context information obtained through the road guidance image in connection with the road guidance application.
- the virtual object may provide information on a building, a road, and the like in the road guide image by voice.
- the virtual object may be provided with a conversational voice utterance that is different from the voice provided by the road guidance application, and may play a role of accompanied friends, family, etc., and may communicate a friendly conversation to the driver.
- the augmented reality application may obtain context information using an image captured in real time and recommend the interior in consideration of the context information (structure of the room, wallpaper color, size, layout of existing furniture, etc.). For example, augmented reality applications can recommend matching furniture, pots, accessories, and the like.
- the apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components.
- the devices and components described in the embodiments are, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs).
- ALUs arithmetic logic units
- FPGAs field programmable gate arrays
- PLU programmable logic unit
- the processing device may execute an operating system (OS) and one or more software applications running on the operating system.
- the processing device may also access, store, manipulate, process, and generate data in response to the execution of the software.
- processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include.
- the processing device may include a plurality of processors or one processor and one controller.
- other processing configurations are possible, such as parallel processors.
- the software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device.
- Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted.
- the software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner.
- Software and data may be stored on one or more computer readable recording media.
- the method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium.
- the computer readable medium may include program instructions, data files, data structures, etc. alone or in combination.
- the program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts.
- Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks.
- Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.
- the hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.
- the present invention relates to an apparatus for providing augmented reality that recognizes a situation using a neural network, a providing method, and a computer program stored in a medium for executing the method.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Signal Processing (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Mathematical Physics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Evolutionary Biology (AREA)
- Human Computer Interaction (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Image Analysis (AREA)
- User Interface Of Digital Computer (AREA)
- Processing Or Creating Images (AREA)
- Position Input By Displaying (AREA)
Abstract
Description
Claims (19)
- 프로세서에 의해 영상을 획득하는 단계;상기 프로세서에 의해, 상기 영상을 분석하여 상기 영상에 포함된 평면에 가상 오브젝트를 배치시키도록 상기 영상을 렌더링하는 단계;상기 영상에 포함된 현재 프레임을 이전 프레임과 비교함으로써, 상기 현재 프레임의 장면 전환 여부를 결정하는 것을 포함하는 판단 기준에 기초하여 상기 영상에 대한 상황 인지 처리 여부를 결정하는 단계;상기 상황 인지 처리 여부가 참으로 결정된 경우, 뉴럴 네트워크를 이용하여 상기 영상 및/또는 센서부로부터 수신된 센싱 값을 분석하여 하나 이상의 상황 정보를 산출하는 단계; 및상기 상황 정보를 적용한 추가 컨텐츠를 생성하고, 상기 추가 컨텐츠를 제공하는 단계;를 포함하는 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 방법.
- 제1항에 있어서,상기 영상에 대한 상황 인지 처리 여부를 결정하는 단계에서, 상기 판단기준은상기 프로세서의 처리 능력이 임계 값을 초과하는지 여부를 결정하는 것을 더 포함하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 방법.
- 제1항에 있어서,상기 영상에 대한 상황 인지 처리 여부를 결정하는 단계에서, 상기 판단기준은상기 영상의 출력 프레임 레이트가 평균 프레임 레이트 미만에 해당하는 지 여부를 결정하는 것을 더 포함하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 방법.
- 제1항에 있어서,상기 장면 전환 여부를 결정하는 것은,상기 현재 프레임의 픽셀들을 컬러값에 따라 분포시킨 분포도를 산출하고, 상기 분포도를 상기 이전 프레임의 분포도와 비교함으로서, 장면 전환 여부를 결정하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 방법.
- 제1항에 있어서,상기 장면 전환 여부를 결정하는 것은,상기 현재 프레임에서 검출한 엣지 정보를 이용하여 상기 엣지 정보를 종합하여 물체의 색상 및 형태를 추출하고, 상기 물체의 색상 및 형태를 이용하여 장면 전환 여부를 결정하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 방법.
- 제1항에 있어서,상기 상황 정보를 산출하는 단계는입력된 사용자 입력에 대응하여 산출하는 상황 정보의 종류를 상기 사용자 입력에 따라 한정하여 획득하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 방법.
- 제1항에 있어서,상기 영상은카메라에 의해 촬영되거나, 다른 애플리케이션에 의해 생성되어 출력되는 화상 데이터인, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 방법.
- 제1항에 있어서,상기 추가 컨텐츠를 제공하는 단계는 상기 상황 정보를 포함하는 컨텐츠를 제공하거나, 상기 상황 정보를 활용한 대화 형식의 음성 발화 컨텐츠를 생성하여 상기 가상 오브젝트를 통해 제공하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 방법.
- 제1항에 있어서,상기 하나 이상의 상황 정보들을 산출하는 단계는각 상황 정보에 대한 매칭 확률을 더 포함하는 확률맵의 형태로 산출하고, 각 상황 정보에 대한 매칭 확률을 기초로 상기 하나 이상의 상황 정보를 순위화하여 전달하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 방법.
- 하나 이상의 프로세서에 의해 실행할 수 있는 명령어를 포함하는 메모리; 및 상기 메모리에 결합되고 명령어들을 실행 가능한 하나 이상의 프로세서를 포함하는 영상 처리 장치에 있어서,상기 메모리는영상을 획득하고, 상기 영상을 분석하여 상기 영상에 포함된 평면에 가상 오브젝트를 배치시키도록 상기 영상을 렌더링하는 렌더링 모듈;상기 영상에 포함된 현재 프레임을 이전 프레임과 비교함으로써, 상기 현재 프레임의 장면 전환 여부를 결정하는 것을 포함하는 판단 기준에 기초하여 상기 영상에 대한 상황 인지 처리 여부를 결정하는 판단 모듈;상기 상황 인지 처리 여부가 참으로 결정된 경우, 뉴럴 네트워크를 이용하여 상기 영상 및/또는 센서부로부터 수신된 센싱 값을 분석하여 하나 이상의 상황 정보를 산출하는 상황 인지 모듈; 및상기 상황 정보를 적용한 추가 컨텐츠를 생성하고, 상기 추가 컨텐츠를 제공하는 편집 모듈을 포함하는 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치.
- 제10항에 있어서,상기 판단 기준은상기 프로세서의 처리 능력이 임계 값을 초과하는지 여부를 결정하는 것을 더 포함하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치.
- 제10항에 있어서,상기 판단기준은상기 영상의 출력 프레임 레이트가 평균 프레임 레이트 미만에 해당하는지 여부를 결정하는 것을 더 포함하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치.
- 제10항에 있어서,상기 장면 전환 여부를 결정하는 것은,상기 현재 프레임의 픽셀들을 컬러값에 따라 분포시킨 분포도를 산출하고, 상기 분포도를 상기 이전 프레임의 분포도와 비교함으로서, 장면 전환 여부를 결정하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치.
- 제10항에 있어서,상기 장면 전환 여부를 결정하는 것은,상기 현재 프레임에서 검출한 엣지 정보를 이용하여 상기 엣지 정보를 종합하여 물체의 색상 및 형태를 추출하고, 상기 물체의 색상 및 형태를 이용하여 장면 전환 여부를 결정하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치.
- 제1항에 있어서,상기 상황 인지 모듈은입력된 사용자 입력에 대응하여 산출하는 상황 정보의 종류를 상기 사용자 입력에 따라 한정하여 획득하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치.
- 제10항에 있어서,상기 영상은카메라에 의해 촬영되거나, 다른 애플리케이션에 의해 생성되어 출력되는 화상 데이터인, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치.
- 제10항에 있어서,상기 편집 모듈은상기 상황 정보를 포함하는 컨텐츠를 제공하거나, 상기 상황 정보를 활용한 대화 형식의 음성 발화 컨텐츠를 생성하여 상기 가상 오브젝트를 통해 제공하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치.
- 제1항에 있어서,상기 상황 인지 모듈은각 상황 정보에 대한 매칭 확률을 더 포함하는 확률맵의 형태로 산출하고, 각 상황 정보에 대한 매칭 확률을 기초로 상기 하나 이상의 상황 정보를 순위화하여 전달하는, 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치.
- 컴퓨터를 이용하여 제1항 내지 제9항 중 어느 한 항의 방법을 실행시키기 위하여 컴퓨터 판독 가능한 저장 매체에 저장된 컴퓨터 프로그램.
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020227024827A KR20220106855A (ko) | 2018-03-12 | 2018-03-12 | 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치, 제공 방법 및 상기 방법을 실행하기 위하여 매체에 저장된 컴퓨터 프로그램 |
PCT/KR2018/002868 WO2019177181A1 (ko) | 2018-03-12 | 2018-03-12 | 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치, 제공 방법 및 상기 방법을 실행하기 위하여 매체에 저장된 컴퓨터 프로그램 |
JP2020548651A JP7295132B2 (ja) | 2018-03-12 | 2018-03-12 | ニューラルネットワークを利用して状況を認知する拡張現実提供装置、提供方法、およびコンピュータプログラム |
KR1020207025178A KR102423903B1 (ko) | 2018-03-12 | 2018-03-12 | 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치, 제공 방법 및 상기 방법을 실행하기 위하여 매체에 저장된 컴퓨터 프로그램 |
US17/016,845 US11501500B2 (en) | 2018-03-12 | 2020-09-10 | Augmented reality (AR) providing apparatus and method for recognizing context using neural network, and non-transitory computer-readable record medium for executing the method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/KR2018/002868 WO2019177181A1 (ko) | 2018-03-12 | 2018-03-12 | 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치, 제공 방법 및 상기 방법을 실행하기 위하여 매체에 저장된 컴퓨터 프로그램 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/016,845 Continuation US11501500B2 (en) | 2018-03-12 | 2020-09-10 | Augmented reality (AR) providing apparatus and method for recognizing context using neural network, and non-transitory computer-readable record medium for executing the method |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019177181A1 true WO2019177181A1 (ko) | 2019-09-19 |
Family
ID=67907288
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/KR2018/002868 WO2019177181A1 (ko) | 2018-03-12 | 2018-03-12 | 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치, 제공 방법 및 상기 방법을 실행하기 위하여 매체에 저장된 컴퓨터 프로그램 |
Country Status (4)
Country | Link |
---|---|
US (1) | US11501500B2 (ko) |
JP (1) | JP7295132B2 (ko) |
KR (2) | KR102423903B1 (ko) |
WO (1) | WO2019177181A1 (ko) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113592699A (zh) * | 2020-04-30 | 2021-11-02 | 辉达公司 | 使用一个或更多个神经网络的内容管理 |
JP2022512525A (ja) * | 2019-10-15 | 2022-02-07 | ベイジン センスタイム テクノロジー デベロップメント カンパニー, リミテッド | Arシーン画像処理方法及び装置、電子機器並びに記憶媒体 |
JP7449672B2 (ja) | 2019-11-05 | 2024-03-14 | ユニ・チャーム株式会社 | 表示制御装置、表示制御方法及び表示制御プログラム |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102472568B1 (ko) | 2020-10-15 | 2022-12-01 | 한국전자기술연구원 | 포즈 정보와 뎁스 정보를 기초로 증강 현실을 제공하기 위한 3차원 맵을 구성하기 위한 장치 및 이를 위한 방법 |
KR102525583B1 (ko) | 2020-10-15 | 2023-04-26 | 한국전자기술연구원 | 경량형 증강현실장치의 스테레오 프레임을 이용한 3차원 맵을 생성하기 위한 장치 및 이를 위한 방법 |
KR20220115433A (ko) * | 2021-02-10 | 2022-08-17 | 삼성전자주식회사 | Ar 객체를 표시하는 방법 및 전자 장치 |
US11798273B2 (en) * | 2021-03-12 | 2023-10-24 | Lawrence Livermore National Security, Llc | Model-based image change quantification |
KR102555524B1 (ko) * | 2021-06-29 | 2023-07-21 | 주식회사 케이에스앤픽 | 영상 콘텐츠 플랫폼을 제공하는 서버 및 그 방법 |
KR20230087952A (ko) * | 2021-12-10 | 2023-06-19 | 세종대학교산학협력단 | 카메라 움직임에 따른 레이 트레이싱 화질조절방법, 이를 수행하는 화질조절장치 및 이를 저장하는 기록매체 |
CN116311519B (zh) * | 2023-03-17 | 2024-04-19 | 北京百度网讯科技有限公司 | 动作识别方法、模型训练方法及装置 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20130060339A (ko) * | 2010-09-20 | 2013-06-07 | 퀄컴 인코포레이티드 | 클라우드 보조 증강 현실을 위한 적응적 프레임워크 |
KR20140082610A (ko) * | 2014-05-20 | 2014-07-02 | (주)비투지 | 휴대용 단말을 이용한 증강현실 전시 콘텐츠 재생 방법 및 장치 |
KR20170022088A (ko) * | 2015-08-19 | 2017-03-02 | 한국전자통신연구원 | 증강대상의 모션에 기반한 미러 디스플레이 상에서의 증강현실 렌더링 방법 및 이를 이용한 장치 |
US20170270582A1 (en) * | 2016-03-16 | 2017-09-21 | Paypal, Inc. | Item recognition and interaction |
KR20170122443A (ko) * | 2016-04-27 | 2017-11-06 | 삼성에스디에스 주식회사 | 공간 관리 시스템 및 방법과 이를 수행하기 위한 모바일 기기 |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2008022442A (ja) | 2006-07-14 | 2008-01-31 | Sony Corp | 画像処理装置および方法、並びにプログラム |
AU2009298438B2 (en) | 2008-10-03 | 2013-10-24 | 3M Innovative Properties Company | Systems and methods for multi-perspective scene analysis |
KR101130734B1 (ko) * | 2010-08-12 | 2012-03-28 | 연세대학교 산학협력단 | 상황 구조 생성 방법 및, 상황 구조 생성 시스템 |
US8873813B2 (en) * | 2012-09-17 | 2014-10-28 | Z Advanced Computing, Inc. | Application of Z-webs and Z-factors to analytics, search engine, learning, recognition, natural language, and other utilities |
US9361730B2 (en) | 2012-07-26 | 2016-06-07 | Qualcomm Incorporated | Interactions of tangible and augmented reality objects |
EP2985706A1 (en) | 2014-08-14 | 2016-02-17 | Samsung Electronics Co., Ltd. | Method and apparatus for providing image contents |
JP6614611B2 (ja) | 2016-02-29 | 2019-12-04 | Kddi株式会社 | 画像間類似度を考慮して物体を追跡する装置、プログラム及び方法 |
KR20170112406A (ko) * | 2016-03-31 | 2017-10-12 | 한국전자통신연구원 | 증강 현실 아바타 사진 촬영 장치 및 방법 |
US11144907B2 (en) * | 2017-04-12 | 2021-10-12 | Visa International Service Association | Resource provider aggregation and context-based content notification system |
SG11201909605VA (en) * | 2017-04-19 | 2019-11-28 | National Science And Tech Development Agency | System for recording, analyzing risk(s) of accident(s) or need of assistance and providing real-time warning(s) based on continuous sensor signals |
KR101834925B1 (ko) | 2017-09-25 | 2018-03-06 | 주식회사 대경바스컴 | 객체 위치 변화를 벡터로 변환하여 영상 및 음향 신호를 동기화한 가상스튜디오 방송 편집 및 송출 기기와 이를 이용한 방법 |
US10659526B2 (en) * | 2018-01-03 | 2020-05-19 | Verizon Patent And Licensing Inc. | Edge compute systems and methods |
-
2018
- 2018-03-12 WO PCT/KR2018/002868 patent/WO2019177181A1/ko active Application Filing
- 2018-03-12 KR KR1020207025178A patent/KR102423903B1/ko active IP Right Grant
- 2018-03-12 KR KR1020227024827A patent/KR20220106855A/ko not_active Application Discontinuation
- 2018-03-12 JP JP2020548651A patent/JP7295132B2/ja active Active
-
2020
- 2020-09-10 US US17/016,845 patent/US11501500B2/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20130060339A (ko) * | 2010-09-20 | 2013-06-07 | 퀄컴 인코포레이티드 | 클라우드 보조 증강 현실을 위한 적응적 프레임워크 |
KR20140082610A (ko) * | 2014-05-20 | 2014-07-02 | (주)비투지 | 휴대용 단말을 이용한 증강현실 전시 콘텐츠 재생 방법 및 장치 |
KR20170022088A (ko) * | 2015-08-19 | 2017-03-02 | 한국전자통신연구원 | 증강대상의 모션에 기반한 미러 디스플레이 상에서의 증강현실 렌더링 방법 및 이를 이용한 장치 |
US20170270582A1 (en) * | 2016-03-16 | 2017-09-21 | Paypal, Inc. | Item recognition and interaction |
KR20170122443A (ko) * | 2016-04-27 | 2017-11-06 | 삼성에스디에스 주식회사 | 공간 관리 시스템 및 방법과 이를 수행하기 위한 모바일 기기 |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2022512525A (ja) * | 2019-10-15 | 2022-02-07 | ベイジン センスタイム テクノロジー デベロップメント カンパニー, リミテッド | Arシーン画像処理方法及び装置、電子機器並びに記憶媒体 |
US11423625B2 (en) | 2019-10-15 | 2022-08-23 | Beijing Sensetime Technology Development Co., Ltd. | Augmented reality scene image processing method and apparatus, electronic device and storage medium |
JP7150894B2 (ja) | 2019-10-15 | 2022-10-11 | ベイジン・センスタイム・テクノロジー・デベロップメント・カンパニー・リミテッド | Arシーン画像処理方法及び装置、電子機器並びに記憶媒体 |
JP7449672B2 (ja) | 2019-11-05 | 2024-03-14 | ユニ・チャーム株式会社 | 表示制御装置、表示制御方法及び表示制御プログラム |
CN113592699A (zh) * | 2020-04-30 | 2021-11-02 | 辉达公司 | 使用一个或更多个神经网络的内容管理 |
Also Published As
Publication number | Publication date |
---|---|
US11501500B2 (en) | 2022-11-15 |
KR102423903B1 (ko) | 2022-07-22 |
JP7295132B2 (ja) | 2023-06-20 |
KR20220106855A (ko) | 2022-07-29 |
JP2021520535A (ja) | 2021-08-19 |
US20200410770A1 (en) | 2020-12-31 |
KR20200108484A (ko) | 2020-09-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2019177181A1 (ko) | 뉴럴 네트워크를 이용하여 상황을 인지하는 증강 현실 제공 장치, 제공 방법 및 상기 방법을 실행하기 위하여 매체에 저장된 컴퓨터 프로그램 | |
CN112379812B (zh) | 仿真3d数字人交互方法、装置、电子设备及存储介质 | |
EP2570994B1 (en) | Image processing device, program, and image processing method | |
JP6684883B2 (ja) | カメラエフェクトを提供する方法およびシステム | |
US20200349975A1 (en) | Video tagging by correlating visual features to sound tags | |
CN108874126B (zh) | 基于虚拟现实设备的交互方法及系统 | |
US10922536B2 (en) | Age classification of humans based on image depth and human pose | |
EP3791573B1 (en) | Puppeteering a remote avatar by facial expressions | |
EP3419020B1 (en) | Information processing device, information processing method and program | |
US11587280B2 (en) | Augmented reality-based display method and device, and storage medium | |
KR20190035116A (ko) | Ar 객체를 표시하기 위한 장치 및 방법 | |
US10943335B2 (en) | Hybrid tone mapping for consistent tone reproduction of scenes in camera systems | |
US20240104744A1 (en) | Real-time multi-view detection of objects in multi-camera environments | |
US10861169B2 (en) | Method, storage medium and electronic device for generating environment model | |
US11474776B2 (en) | Display-based audio splitting in media environments | |
CN111797873A (zh) | 场景识别方法、装置、存储介质及电子设备 | |
CN110431838B (zh) | 提供人脸识别摄像机的动态内容的方法及系统 | |
US11846783B2 (en) | Information processing apparatus, information processing method, and program | |
WO2019124850A1 (ko) | 사물 의인화 및 인터랙션을 위한 방법 및 시스템 | |
KR20150071594A (ko) | 제어 장치를 위한 증강 현실 오버레이 | |
US20180097865A1 (en) | Video processing apparatus and method | |
WO2023026634A1 (ja) | 表示制御装置 | |
US20240087091A1 (en) | Server device and network control method | |
JP2023022157A (ja) | コンピュータプログラム、方法及びサーバ装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18909284 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 20207025178 Country of ref document: KR Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2020548651 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 18909284 Country of ref document: EP Kind code of ref document: A1 |