KR102530509B1 - A terminal device for executing a screen hold program for applying augmented reality content - Google Patents

Abstract

A terminal device executing an augmented reality content hold program according to an embodiment of the present invention captures a captured image of a target object to which AR hold is applied in response to an input signal according to the length or number of user inputs (touches). a screen holding unit that holds the screen on the screen after performing the operation; a screen hold setting unit configured to set a hold method to one of at least one of a basic hold method, a panoramic hold method, and an emergency hold method; and an AR contents execution and editing unit for activating AR contents applicable to the type of photographing target (home equipment, industrial equipment, office equipment, etc.) on the held photographed image.

Description

A terminal device for executing a screen hold program for applying augmented reality content}

The present invention relates to a terminal device that executes an augmented reality content hold program.

Augmented Reality (AR) is a technology that superimposes virtual objects on the real world that users see. It is also called Mixed Reality (MR) because it combines the real world with the virtual world with additional information in real time and shows it as a single image. In augmented reality technology, the user can see the real environment, and the real environment and virtual objects take the form of a mixture. In other words, augmented reality has the distinction of supplementing the real world and showing it to users by superimposing virtual objects on the real world, and provides users with a better sense of reality than Virtual Reality (VR). Marker-based Augmented Reality is a technology that overlaps and outputs real-life images and augmented reality objects by identifying visual markers stored in the system in advance, and is widely used in games and marketing.

Publication No. 10-2020-0110075

An object of the present invention is to provide a terminal device that executes an augmented reality content hold program.

In order to solve the above problem, a terminal device executing an augmented reality content hold program according to an embodiment of the present invention transmits a captured image of a target object to which AR hold is applied according to the length or number of user inputs (touches). a screen holding unit that captures a screen in response to an input signal and then holds it on a screen; a screen hold setting unit configured to set a hold method to one of at least one of a basic hold method, a panoramic hold method, and an emergency hold method; and an AR contents execution and editing unit for activating AR contents applicable to a type of shooting target (home equipment, industrial equipment, office equipment) on a held photographed image, wherein the panorama hold method is performed from a first point to a second point. It is characterized in that it is a hold method for continuously capturing the screens taken up to , expanding the width of the shot images so that different AR content is activated according to the screen transition when the user slides the continuously shot images in a panoramic form.

Therefore, when using a terminal device that executes an augmented reality content hold program according to an embodiment of the present invention, there is an advantage of supporting an AR hold function in which content appearing on the object is continuously displayed even if it moves away from the object under any circumstances. to provide.

1 is a device configuration diagram of a terminal device executing an augmented reality content hold program according to an embodiment of the present invention.
FIG. 2 is a device configuration diagram of a control unit shown in FIG. 1 .
3 to 5 are examples of cases where the augmented reality content holding device shown in FIG. 1 is applied to home appliances, business documents, and industrial sites.

Hereinafter, embodiments of the present specification will be described with reference to the accompanying drawings. However, it should be understood that the technology described herein is not limited to specific embodiments, and includes various modifications, equivalents, and/or alternatives of the embodiments herein. . In connection with the description of the drawings, like reference numerals may be used for like elements. In this specification, expressions such as “has,” “can have,” “includes,” or “can include” indicate the existence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

In this specification, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, Or (3) may refer to all cases including at least one A and at least one B.

Expressions such as "first," "second," "first," or "second," as used herein, may modify various elements regardless of order and/or importance, and may refer to one element as It is used only to distinguish it from other components and does not limit the corresponding components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, a first element may be termed a second element without departing from the scope of rights described herein, and similarly, the second element may also be renamed to the first element.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or connected through another component (eg, a third component). On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), the element and the above It may be understood that other components (eg, a third component) do not exist between the other components.

As used herein, the expression “configured (or configured) to” means, depending on the circumstances, for example, “suitable for,” “having the capacity to” ," "designed to," "adapted to," "made to," or "capable of." The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware. Instead, in some contexts, the phrase "device configured to" may mean that the device is "capable of" in conjunction with other devices or components.

For example, the phrase "a processor configured (or configured) to perform A, B, and C" may include a dedicated processor (eg, embedded processor) to perform the operation, or by executing one or more software programs stored in a memory device. , may mean a general-purpose processor (eg, CPU or application processor) capable of performing corresponding operations.

Terms used in this specification are only used to describe a specific embodiment, and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art described herein. Among the terms used in this specification, terms defined in a general dictionary may be interpreted as having the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in the present specification, an ideal or excessively formal meaning. not be interpreted as In some cases, even terms defined in this specification cannot be interpreted to exclude embodiments in this specification.

Hereinafter, a terminal device executing an augmented reality content hold program according to an embodiment of the present invention will be described in more detail based on the accompanying drawings.

1 is a device configuration diagram of a terminal device executing an augmented reality content hold program according to an embodiment of the present invention, FIG. 2 is a device configuration diagram of a control unit shown in FIG. 1, and FIGS. 3 to 5 are FIG. 1 It is an example of a case where the augmented reality content holding device shown in is applied to home appliances, business documents, and industrial sites.

First, prior to describing the present application, the augmented reality content hold service proposed in the present invention is a service that supports an AR hold function in which content appearing on the object is continuously displayed even if the object is out of the camera angle under any circumstances. .

For reference, due to the characteristics of augmented reality that recognizes a specific marker, image, or 3D object and displays content on the recognized object, content is not displayed when the camera moves away from the object to be recognized.

Therefore, the augmented reality hold program proposed in the present invention may be a program that provides a function of continuously displaying the contents of an originally recognized object even if the object to be recognized is out of the camera angle.

More specifically, the terminal device 100 executing the augmented reality content hold program of the present invention includes a communication unit 110, a storage unit 120, a display unit 130, an audio output unit 140, a control unit 150, and a camera. It may consist of part 160.

Not all of the components of the terminal device 100 shown in FIG. 1 are essential components, and the terminal device 100 that executes the augmented reality hold program will be implemented by more components than those shown in FIG. Alternatively, the terminal device 100 executing the augmented reality hold program may be implemented with fewer components.

The terminal device 100 that executes the augmented reality hold program includes a smart phone, a portable terminal, a mobile terminal, a foldable terminal, and a personal digital assistant. : PDA), PMP (Portable Multimedia Player) terminal, telematics terminal, navigation terminal, personal computer, notebook computer, slate PC, tablet PC, ultrabook (ultrabook), wearable device (including, for example, a watch type terminal (Smartwatch), a glass type terminal (Smart Glass), HMD (Head Mounted Display), etc.), Wibro terminal, IPTV (Internet Protocol Television) ) terminal, smart TV, digital broadcasting terminal, AVN (Audio Video Navigation) terminal, A / V (Audio / Video) system, flexible terminal (Flexible Terminal), it can be applied to various terminals such as digital signage devices.

The communication unit 110 communicates with any internal component or at least one external terminal through a wired/wireless communication network. At this time, the external arbitrary terminal may include a server (not shown), another terminal (not shown), and the like. Here, wireless Internet technologies include Wireless LAN (WLAN), DLNA (Digital Living Network Alliance), Wireless Broadband (Wibro), WiMAX (World Interoperability for Microwave Access: Wimax), HSDPA (High Speed Downlink Packet Access) ), High Speed Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS), etc. , and the communication unit 110 transmits and receives data according to at least one wireless Internet technology within a range including Internet technologies not listed above. In addition, short-range communication technologies include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and Near Field Communication (NFC). , Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and the like may be included. In addition, wired communication technologies may include power line communication (PLC), USB communication, Ethernet, serial communication, optical/coaxial cables, and the like.

In addition, the communication unit 110 may mutually transmit information with an arbitrary terminal through a Universal Serial Bus (USB).

In addition, the communication unit 110 complies with technical standards or communication methods for mobile communication (eg, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (CDMA2000), EV) -DO(Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA(Wideband CDMA), HSDPA(High Speed Downlink Packet Access), HSUPA(High Speed Uplink Packet Access), LTE(Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.) to transmit and receive radio signals with a base station, the server, and other terminals on a mobile communication network.

In addition, the communication unit 110 receives an augmented reality hold program app provided from a server connected through wired/wireless communication under the control of the control unit 150, data related to the corresponding app, and the like.

The storage unit 120 stores various user interfaces (UI), graphic user interfaces (GUI), and the like.

In addition, the storage unit 120 stores data and programs necessary for the operation of the terminal device 100 executing the augmented reality hold program.

That is, the storage unit 120 includes a plurality of application programs (applications) running in the terminal device 100 that executes the augmented reality hold program, and the terminal device 100 that executes the augmented reality hold program. ) can store data and instructions for operation. At least some of these application programs may be downloaded from an external server through wireless communication. In addition, at least some of these application programs may exist on the terminal device 100 that executes the augmented reality hold program from the time of shipment for basic functions of the terminal device 100 that executes the augmented reality hold program. On the other hand, the application program is stored in the storage unit 120, installed in the terminal device 100 that executes the augmented reality hold program, and the terminal device 100 that executes the augmented reality hold program by the control unit 150 It can be driven to perform the operation (or function) of.

In addition, the storage unit 120 may be a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg, SD or XD). memory, etc.), magnetic memory, magnetic disk, optical disk, RAM (Random Access Memory: RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory: ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), It may include at least one storage medium of PROM (Programmable Read-Only Memory). In addition, the terminal device 100 that executes the augmented reality hold program operates a web storage that performs the storage function of the storage unit 120 on the Internet, or operates in relation to the web storage. may be

In addition, the storage unit 120 stores the received word learning app (program) and data related to the corresponding app (program) under the control of the controller 150 .

The display unit (or display unit) 130 uses the user interface and/or graphic user interface stored in the storage unit 120 and the control unit 150 under the control of the control unit 150 to display various menu screens and the like. content can be displayed.

Here, the content displayed on the display unit 130 includes various text or image data (including various information data) and a menu screen including data such as icons, list menus, and combo boxes. Also, the display unit 130 may be a touch screen.

In addition, the display unit 130 may include a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. (Flexible Display), a three-dimensional display (3D Display), e-ink display (e-ink display), it may include at least one of LED (Light Emitting Diode).

In addition, the display unit 130 displays an augmented reality hold execution result screen and the like under the control of the controller 150 .

The audio output unit 140 outputs audio information included in a signal that has been subjected to predetermined signal processing by the control unit 150 . Here, the audio output unit 140 may include a receiver, a speaker, a buzzer, and the like.

In addition, the voice output unit 140 outputs a guide voice generated by the control unit 150 .

In addition, the audio output unit 140 outputs audio information (or sound effects) corresponding to the word learning app execution result screen by the control unit 150 .

The camera unit 160 is a module that supports collecting images as still images or moving images. The camera unit 160 may include an image sensor and an image data processing device, and may basically include components for image collection such as a lens, an aperture and an aperture driving unit, and a lens control unit for supporting camera functions. there is. In particular, the camera unit 160 of the present invention may provide a mode for capturing a panoramic image or an action shot image.

In addition, the camera unit 160 is equipped with a camera operating program and stores various routines necessary for operating the camera of the terminal device 100 . For example, the camera operation program includes a routine for activating the camera unit 160 according to the reception of a turn-on input signal of the camera unit 160 and outputting data collected and transmitted by the activated camera unit 160 as a preview image. A storage routine may be included to synthesize and store the data collected and transmitted by the camera unit 160 as a panoramic image or an action shot image according to a set mode, such as a panoramic image capturing mode or an action shot image capturing mode.

In addition, the camera unit 160 provides camera location information, camera status, angle, direction angle, degree of rotation, camera driving status, camera movement, object movement detection, GPS, geomagnetism, gyro compass, acceleration, angular velocity, level detection, and contact detection. , magnification ratio, focusing information, motion recognition, and a configuration for measuring or operating at least one of a distance to an object.

Next, the controller or microcontroller unit (MCU) 150 executes overall control functions of the terminal device 100 executing the augmented reality hold program.

In addition, the control unit 150 executes overall control functions of the terminal device 100 that executes the augmented reality hold program using the programs and data stored in the storage unit 120 . The controller 150 may include RAM, ROM, CPU, GPU, and a bus, and the RAM, ROM, CPU, and GPU may be connected to each other through a bus. The CPU may access the storage unit 120 and perform booting using the O/S stored in the storage unit 120, and use various programs, contents, data, etc. stored in the storage unit 120. You can perform a variety of actions by doing so.

In addition, the control unit 150 is installed in the terminal device 100 that executes the augmented reality hold program provided from the server.

In addition, when a dedicated app (or application) installed in the terminal device 100 executing the augmented reality hold program is selected (or touched/executed) according to user selection (or user touch/input), the control unit 150 executes the dedicated app and displays the app execution result screen on the display unit 130 .

More specifically, the control unit 150 includes a screen hold unit 151, a screen hold setting unit 152, and an AR contents execution and editing unit 152.

The screen holding unit 151 may be configured to capture and hold a captured image of a target object to which AR hold is applied.

The screen holding unit 151 activates the hold function or releases the activated hold function based on the length or number of user inputs (touches).

The screen hold setting unit 152 may be configured to set a hold method. The hold method includes at least one of ① a basic hold method, ② a panoramic hold method, and ③ an emergency hold method.

The basic hold method may be a hold method for activating AR content on the captured image after simply capturing a captured image taken by a user.

The panoramic hold method continuously captures screens captured from a first point to a second point and expands the width of the captured image so that when the user slides the continuously captured images in a panoramic form, different AR content is displayed according to the screen transition. It may be a hold method to activate.

The emergency hold method may be a hold method for responding to a shake or change of a camera angle, and may be a hold method for automatically capturing a photographing subject when a camera angle shakes or a screen is switched. Therefore, the user can hold the photographed image of the target object by applying any one of the three holding methods described above.

Next, the AR content execution and editing unit 153 may be configured to activate AR content applicable to the type of shooting target (home device, industrial device, office device, etc.) on the held shot image.

Here, the AR content format includes various content formats such as documents, images, 3D contents, video, and audio.

In addition, the AR content execution and editing unit 153 may support an AR manual pop-up and may support a function for editing preset AR content or AR manual pop-up.

Below, we will explain the technology through some examples that can be actually used in AR application services.

3 to 5, ① case of AR manual pop-up for daily life applied to home appliances, ② case of AR management support pop-up for business applied to business document, ③ case of industrial AR guide pop-up applied to industrial site, etc. are given as examples. can

For example, among various home appliances used at home, if the need for manual use is great, an existing paper manual or web manual can be used.

In the case of an AR manual, it is highly likely to be used because it can solve the problem of immediacy and accessibility at once, but the user must hold the camera-attached device with one hand to maintain the content continuously, so both hands are not free in terms of operability, which can cause inconvenience. there is.

In addition, when the camera is shaken violently or the camera angle is switched at high speed, the content may not be displayed properly due to the readjustment of AR recognition, which slows down the recognition speed or reduces the accuracy of recognition. In this case, if the target image is fixed in a capture form using the AR hold function and the content displayed on the target, that is, the augmented content is continuously activated, the inconvenience in terms of operability can be solved and concerns about unstable content display can be eliminated. .

Hereinafter, an example of applying the three AR hold methods described above will be described.

First, basic hold application technology: As a method of instantly capturing the situation to which AR hold is to be applied, various types of objects such as markers, texts, images, and 3D objects are instantly captured and fixed based on images, and then displayed on the target. The content allows it to remain active. The content displayed on the object includes text, image, 3D content, video, sound, etc., and is continuously activated as it is.

Panorama hold application technology: As a form of slightly expanding the AR hold application range, it recognizes a wider range of objects through panorama setting and captures them as an image with an expanded range. When the captured target image is moved by sliding left and right or scrolling up and down, the augmented content suitable for the target is changed and displayed on the screen.

Emergency hold application technology: AR hold application technology to respond to sudden camera angle changes. Through presetting, when the camera shakes abnormally or a sudden screen transition occurs, the subject is captured and augmented content on the subject is displayed. keep it going

Hereinafter, an example of a manipulation method for applying AR hold will be described.

In the case of Basic Hold, the situation to be held is momentarily captured through a momentary touch of the Hold button. In the hold situation, the hold release button is activated so that the user can return to the original situation at any time.

In the case of Panorama hold, when the hold button is pressed in advance and the camera is moved to capture the target image, when the hold button is released, the entire situation where the button was pressed is captured and converted to the hold situation. In the hold situation, the hold release button is activated so that the user can return to the original situation at any time.

In the case of emergency hold, the hold function is automatically applied when an emergency hold situation occurs through presetting. In the hold situation, the hold release button is activated so that the user can return to the original situation at any time.

Therefore, when using a terminal device that executes an augmented reality content hold program according to an embodiment of the present invention, there is an advantage of supporting an AR hold function in which content appearing on the object is continuously displayed even if it moves away from the object under any circumstances. to provide.

“~ unit” used in one embodiment of the present invention may be implemented as a hardware component, a software component, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The foregoing may be modified and modified by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

100: terminal device that executes the augmented reality content hold program
110: communication department
120: storage unit 130: display unit
140: audio output unit 150: control unit
160: camera unit
151: screen holding unit
152: screen hold setting unit
153: AR content execution and editing department

Claims (4)

A screen holding unit that captures a captured image of a target object to which AR hold is applied in response to an input signal according to the length or number of user inputs (touches) and holds it on the screen;
a screen hold setting unit configured to set a hold method to one of at least one of a basic hold method, a panoramic hold method, and an emergency hold method; and
Includes an AR content execution and editing unit that activates AR content applicable to the type of shooting target (home equipment, industrial equipment, office equipment) on the held photographed image ,
The panoramic hold method
Hold for continuously capturing screens taken from the first point to the second point and expanding the width of the shot images so that different AR content is activated according to the screen transition when the user slides the continuously shot images in a panoramic form A terminal device that executes an augmented reality content hold program. According to claim 1,
The basic hold method is
A terminal device for executing an augmented reality content hold program, characterized in that it is a hold method for simply capturing a captured image taken by a user and then activating AR content on the captured image. delete According to claim 1,
The emergency hold method
A terminal device for executing an augmented reality content hold program characterized in that it is a hold method for automatically capturing a subject to be photographed when the camera angle is shaken or the screen is switched as a hold method for responding to shaking or change of the camera angle. .

Images