WO2018155733A1

WO2018155733A1 - Augmented reality system comprising portable terminal device and projection device

Info

Publication number: WO2018155733A1
Application number: PCT/KR2017/001964
Authority: WO
Inventors: 이현민
Original assignee: 이현민
Priority date: 2017-02-22
Filing date: 2017-02-23
Publication date: 2018-08-30
Also published as: KR20180097031A

Abstract

According to various embodiments of the present invention, an augmented reality system comprises: a projection device; and a portable terminal device, wherein the projection device can comprise: a projection module for projecting a virtual image to the outside; an object sensor for recognizing surrounding objects; a first communication circuit for communicating with the portable terminal device; and a first processor for transmitting object information related to the recognized object to the portable terminal device through the first communication circuit, and the portable terminal device can comprise: a second communication circuit for communicating with the projection device; and a processor configured to detect at least one event occurring in the portable terminal device, generate a virtual image in response to the event, and transmit the virtual image to the projection device through the communication circuit such that the projection device projects the virtual image to the outside. In addition, various embodiments are possible.

Description

Augmented Reality System Including Portable Terminal Device and Projection Device

The present embodiment relates to an augmented reality system, and more particularly, an augmentation apparatus including a projection apparatus capable of recognizing external objects and projecting augmented reality contents adjacent to an object, and a portable terminal apparatus wirelessly communicating with the projection apparatus. It is about a real system.

With the development of mobile communication technology and processor technology, portable terminal devices (eg, smartphones) can implement various functions beyond the conventional call function. Recently, personal assistant services / robots have been released to provide users with various user experiences.

In order to utilize various functions of the portable terminal device, users input a lot of information and may receive various information from the portable terminal device. In order to obtain such various pieces of information from the portable terminal device, the user enters an active process of directly inputting the information.

As a method for the user to interact with the portable terminal device, interactions through voice are used in addition to the basic touch input. However, the interaction through voice is limited in the input / output method, and the transmission of unwanted information due to the nature of the sound can cause fatigue to the user.

An object of the present invention for solving the above problems is to provide a projection-based augmented reality system that can improve the user's convenience and provide a variety of user experience.

According to various embodiments of the present disclosure, a portable terminal device may include a display; Memory; Communication circuitry for communicating with an external projection device; And a processor electrically connected to the display, the memory, and the communication circuit, wherein the processor detects at least one event occurring in the portable terminal device, generates a virtual image in response to the event, and generates the virtual image. The projection device may be set to project the virtual image to the outside by transmitting an image to the projection device through the communication circuit.

According to the present invention described above, it is possible to provide a projection-based augmented reality system that can improve the user's convenience and provide a variety of user experience.

1 is a block diagram illustrating components of an augmented reality system according to various embodiments of the present disclosure.

2 is a block diagram of a projection apparatus according to various embodiments of the present disclosure.

3 is a detailed configuration diagram of a projection apparatus according to various embodiments of the present disclosure.

4 illustrates a pattern project according to various embodiments of the present disclosure.

5A to 5B illustrate a focal length of a projection apparatus according to various embodiments of the present disclosure.

6 is a block diagram of a portable terminal device according to various embodiments of the present disclosure.

7 is a detailed block diagram of a second processor of a mobile terminal device according to various embodiments of the present disclosure.

8A to 8C illustrate embodiments of function control of a mobile terminal device according to various embodiments of the present disclosure.

9A is a detailed block diagram of a second processor of a mobile terminal device according to various embodiments of the present disclosure.

9B illustrates each module for generating emotional information according to various embodiments of the present disclosure.

10A through 10C illustrate virtual images including character images according to various embodiments of the present disclosure.

11 illustrates a method of storing a recognized image according to various embodiments of the present disclosure.

12 illustrates functions that may be performed by the secretary robot using the augmented reality system according to various embodiments of the present disclosure.

13 and 14 are flowcharts of a virtual image providing method according to various embodiments of the present disclosure.

Hereinafter, various embodiments of the present disclosure will be described with reference to the accompanying drawings. However, this is not intended to limit the techniques described herein to a particular embodiment and should be understood to include various modifications, equivalents, and / or alternatives to the embodiments herein. .

An augmented reality system according to various embodiments of the present disclosure may include a projection device 110 and a portable terminal device 120. Some configurations and / or functions of the projection device 110 may be implemented on the portable terminal device 120 according to actual implementations, and on the contrary, some components and / or functions of the projection device 110 may be implemented by the projection device 110. It may be implemented in the above.

The function of the projection apparatus 110 may be representative of recognition of the object 140 (eg, a user's hand) and projection of the virtual image 130. To this end, the projection apparatus 110 may include an object recognition module 140 for detecting the presence or movement of the surrounding object 140 and a projection module for projecting an image to the outside.

First, the object sensor is for realizing a vision in augmented reality, may acquire an image of a target area around the projection apparatus 110, and check the existence and movement of the object 140 from the acquired image. For example, the user may install the projection device 110 around the book while reading the book and direct the view of the projection module toward the book, in which case the projection module shoots the book, the user's hand, and the like. You can check the movement.

The object sensor may include a camera and / or an infrared sensor, and according to an embodiment, the object 140 may be recognized in 3D using a plurality of cameras and / or infrared sensors.

The projection module may display the virtual image 130 by projecting it to the outside. For example, the virtual image 130 may be displayed on an area of the book while the user reads the book. Information displayed through the virtual image 130 will not be limited, but, for example, as shown in FIG. 1, weather information may be displayed as a related image and text information, and a predetermined character image may be displayed. have. The user may simultaneously recognize the virtual image 130 (eg, weather information) and reality information projected adjacent to the object 140 (eg, the user's hand).

According to an embodiment, the virtual image 130 may include a predetermined character image. The character image may be changed according to the surrounding environment or the recognized object 140 to express the current state and emotion of the user.

The projection apparatus 110 may include more various components, such as a first communication circuit and a first processor, and a detailed configuration and function of the projection apparatus 110 will be described in detail later with reference to FIGS. 2 to 5.

The mobile terminal device 120 may be implemented as a known electronic device that can be carried by a user, such as a smart phone and a tablet PC, and a detailed embodiment is not limited thereto.

The portable terminal device 120 can wirelessly communicate with the projection device 110 and receives the related information (eg, existence of the object and movement of the object, etc.) of the object 140 recognized by the projection device 110. In addition, the projection apparatus 110 may provide the virtual image 130 to be projected.

The portable terminal device 120 may perform various operations based on the object related information received from the projection device 110. According to an embodiment, the portable terminal device 120 may perform a control operation corresponding to the received object related information. For example, when a user moves a hand, the user may turn over a page of an application (eg, an e-book) currently displayed on the portable terminal device 120 or terminate the application.

According to an exemplary embodiment, the portable terminal device 120 may generate a virtual image 130 to be projected by the projection device 110 and provide it to the projection device 110. Here, the virtual image 130 may include a character image, and the portable terminal device 120 may correct the character image according to an event occurring in an application installed and executed on the portable terminal device 120. According to an embodiment, the mobile terminal device 120 may be equipped with an emotion engine to correct a character image according to the various events, for example, a schedule recorded on a schedule application, a message received from a message application, or the like. Emotion information may be generated based on the emotion information, and the character image may be corrected according to the emotion information.

More detailed configurations and functions of the mobile terminal device 120 will be described in more detail with reference to FIGS. 6 to 11.

Unlike the portable terminal device 120, the projection device 110 is generally installed and used by a user at a fixed position. However, the projection device 110 may have a size similar to that of the portable terminal device 120 for convenience of use.

According to an embodiment, the projection device 110 may receive information (eg, virtual image, weather / time, widget information, etc.) received from the portable terminal device 120 from another device on the Internet. In this case, the projection apparatus 110 may include a processor (eg, a first processor) capable of directly executing an application and rendering a virtual image. In this case, at least some of the functions of the portable terminal device 120 in the augmented reality system may be performed on the projection device 110, and at least some of the functions may be performed on other devices, and the projection device 110 and the other device may be connected to the Internet. Can communicate with.

As shown, the projection apparatus 200 includes a projection module 230, an object sensor 240, a first communication circuit 220, and a first processor 210, some of which are omitted or substituted. If so, there will be no problem in implementing the various embodiments of the present invention. The projection apparatus 200 may be the projection apparatus 110 of FIG. 1.

The first processor 210 is configured to perform operations or data processing related to control and / or communication of each component of the projection apparatus 200. The illustrated projection module 230, the object sensor 240, Each component of the projection apparatus 200 including the first communication circuit 220 may be electrically and / or functionally connected.

The first communication circuit 220 is a configuration for communicating with the portable terminal device using near field communication. The short-range wireless communication used may include, for example, wireless fidelity (WiFi), Bluetooth (bluetooth), Bluetooth low power (BLE), Zigbee, and near field communication (NFC), but are not limited thereto. Standard or nonstandard short range wireless communication may be utilized. According to an exemplary embodiment, the first communication circuit 220 may be connected to a network and may communicate with the mobile terminal device through the network.

According to an embodiment, the projection apparatus 200 may be connected to the Internet through the first communication circuit 220, and may generate and display a virtual image using information obtained on the Internet.

The projection module 230 may project a virtual image (for example, the virtual image 130 of FIG. 1) to the outside and display the same. Here, the virtual image may be an image generated by the first processor 210 or an image received from the portable terminal device through the first communication circuit 220. The virtual image may include a character image and may be projected onto a real area to implement augmented reality. According to an embodiment, the virtual image may be image information obtained from another device on the Internet other than the portable terminal device.

According to an embodiment, the virtual image may include information related to an object recognized by the object sensor 240 to be described later. For example, when a user is reading a book, the object sensor 240 may recognize text in the book, and the projection module 230 may include information related to the recognized text (eg, translation of text, text to the Internet). The virtual image including the searched and retrieved information) can be projected adjacent to the corresponding text.

According to an embodiment, the first processor 210 may directly generate a virtual image. According to another embodiment, the virtual image may be one received by another device on the Internet.

The object sensor 240 may include a software and / or hardware configuration for recognizing an object in the real world (eg, the object 140 of FIG. 1). According to one embodiment, the hardware of the object sensor 240 may include a camera and an infrared sensor, and the software may include a series of logic to detect the coordinates and movement of the object from the captured image. The object sensor 240 may acquire the three-dimensional coordinates of the object by using optical triangulation, time delay measurement, and measurement method using structured light, and detect the presence and movement of the object by detecting the coordinate movement. For example, the object sensor 240 may check the gesture of the user's hand. According to an embodiment, the object sensor 240 may include two or more cameras and / or infrared sensors to obtain three-dimensional coordinates of the object. According to an embodiment, the camera or infrared sensor may include a pattern projector (described in detail later with reference to FIG. 4).

According to an embodiment, at least some of the functions of the object sensor 240 may be performed on the portable terminal device. For example, the object sensor 240 may acquire only an image of a real space through a camera and provide the image to the portable terminal device. The object recognition may be performed by analyzing the image in the mobile terminal device.

Hereinafter, a hardware configuration for transmitting the virtual device to the virtual image will be described in more detail with reference to FIG. 3. 3 is only an example, and various types of projection apparatuses may be used.

The projection module (eg, the projection module of FIG. 2) may include an output 310, a lens 312, and a mirror 314.

The output unit 310 may output the virtual image received from the processor. The output unit 310 may include, for example, a light emitting unit (not shown) for emitting light and a liquid crystal layer (not shown) for selectively transmitting the light emitted from the light emitting unit by using a change in the liquid crystal transmittance to output a virtual image. It may include. In addition, various known techniques may be applied to the output of the virtual image.

At least one lens 312 may be provided on the rear surface of the output unit 310. Light through the output 310 may be refracted by the lens 312 and directed toward the mirror 314. Accordingly, the projection apparatus 300 may be miniaturized by reducing a path through which light passes. The lens 312 may be implemented as a Fresnel lens.

Light refracted by the lens 312 may be output to the outside through the mirror 314. The first processor (eg, the first processor 210 of FIG. 2) may adjust the position and size (or magnification) of the virtual image by adjusting the angle of the mirror 314.

According to an embodiment, the lens 312 and the mirror 314 may function as a short focal length lens capable of projecting an image at a close distance.

The projection apparatus 300 may include an object sensor (eg, the object sensor 240 of FIG. 2) for recognizing an external object. According to one embodiment, the object sensor may include at least one (eg two) cameras 322 and / or at least one (eg two) infrared sensors 324 for obtaining a real image. . Among these, only at least one camera 322 may be included, or at least one infrared sensor 324 may be included.

If the projection device 300 is equipped with two cameras 322 (or two infrared sensors 324), the first camera and the second camera (or the first infrared sensor and the second infrared sensor) may be Images of the same area can be obtained, and since the positions of the first camera and the second camera (or the first infrared sensor and the second infrared sensor) are partially spaced to photograph the corresponding areas at different angles, the three-dimensional object Can be recognized.

According to an embodiment, the camera 322 and / or the infrared sensor 324 may be implemented as a pattern projector.

4 illustrates a pattern projector according to various embodiments of the present disclosure.

According to an embodiment, the object sensor (eg, the object sensor 240 of FIG. 2) of the projection device (eg, the projection device 200 of FIG. 2 and / or the projection device 300 of FIG. 3) may be a pattern projector ( pattern projector). The pattern projector may not recognize the pattern in the eyes of a user using an infrared sensor (for example, the infrared sensor 324 of FIG. 3).

As shown, the pattern projector includes two

light emitters

422 and 424, and the two

light emitters

422 and 424 are control signals of a first processor (eg, the first processor 210 of FIG. 2). Depending on the light output can be selectively. Here, the output light may be light in the infrared region. In FIG. 4, two light emitting

units

422 and 424 are illustrated, but the number of light emitting units is not limited thereto, and one or three light emitting units may be used.

The light output from each of the

light emitting parts

422 and 424 passes through the pattern film 430, and the light output from the pattern film 430 may be repeated by crossing the light and dark portions. To this end, the pattern film 430 may be formed to cross the area passing through the light and the area blocking the parallel to each other.

Light passing through the pattern film 430 may be magnified through the lens 440 to be projected onto the area of the object. As shown in FIG. 4, the light output from each of the first light emitting unit 422 and the second light emitting unit 424 may be patterned and projected, and may be projected at different positions.

An object sensor (eg, the object sensor 240 of FIG. 2) may project the patterned light and photograph the object to generate a 3D map of the object. That is, the object can be recognized using the pattern projector, which can use the configured projector module and can be configured as an additional infrared pattern projector module. The added projection module may be configured to change the phase of the projected image by changing the infrared emitter to obtain three-dimensional coordinates of the object.

According to an embodiment, the object sensor (eg, the object sensor 240 of FIG. 2) may include at least one camera (eg, the first camera 322 and the second camera 324 of FIG. 3). . The projection apparatus may arrange the camera obliquely due to the characteristics of the miniaturized product, and thus the camera may acquire an image at an oblique position with respect to the plane on which the object is provided. In this case, the focal length (or depth of field) of the camera of the input image may not include both the near and far sides, and as a result, it may be difficult to obtain a clear image.

The projection apparatus according to various embodiments of the present disclosure may have a larger focal length by partially tilting the lens of the camera in order to solve the above problems. Referring to the first picture of Figure 5a, the image sensor and the lens is arranged in parallel as in the prior art. In the projection apparatus according to various embodiments of the present disclosure, as a result of arranging the lens by tilting the lens without paralleling the image sensor and the lens, the plane in which the focus is formed may also be tilted. Accordingly, it is possible to solve the structural problem that the camera is obliquely arranged due to the miniaturized product characteristics.

Referring to FIG. 5B, it can be seen that a tilted lens may have a focal length in a wider range than a case of using a half-lens.

As shown, the portable terminal device 600 includes a display 630, a second communication circuit 620, a second processor 610 and a memory 640, even if some of the illustrated configuration is omitted or replaced It will be no problem to implement various embodiments of the present invention. The portable terminal device may be the portable terminal device 120 of FIG. 1.

The display 630 displays an image and may be used as a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or the like. Can be implemented.

The display 630 may include a touch sensor (not shown) capable of sensing a touch input generated on a cover window (not shown) formed in front of the display 630.

The second communication circuit 620 is configured to wirelessly transmit and receive data with an external device (eg, the projection device 110 of FIG. 1) and includes an antenna (not shown) and a communication processor (not shown). can do. The second communication circuit 620 may receive object-related information, transmit a virtual image, or communicate with each other through a network by a projection apparatus and various known short-range wireless communication schemes (eg, Wifi and Bluetooth).

The memory 640 may include known volatile memory and non-volatile memory. The memory 640 may store various instructions that may be performed by the second processor 610. Such instructions may include control instructions such as arithmetic and logical operations, data movement, input / output, and the like that can be recognized by the second processor 610. Various applications that perform functions such as schedules, games, and messages may be stored in the memory 640, and the processor 610 may execute applications stored in the memory 640.

The second processor 610 may be configured to perform operations or data processing related to control and / or communication of components of the mobile terminal device 600, and may include a display 630, a second communication circuit 620, The memory 640 may be electrically and / or functionally connected to the memory 640. The second processor 610 is a hardware or software configuration that performs each function, and includes a gesture recognition module, a function mapping module, and a function control module to be described later with reference to FIG. 7, or a character generation module and emotion to be described later with reference to FIG. 9. Module and a virtual image generation module. Detailed functions of each module will be described in more detail with reference to FIGS. 7 and 9.

According to various embodiments, the second communication circuit 620 may receive object information from the projection apparatus. Here, the object information may include the existence, type (eg, human hand, object, etc.) and movement (eg, movement direction and speed, etc.) of the object (eg, the object 140 of FIG. 1).

The second processor 610 may obtain object information received through the second communication circuit 620. According to an embodiment of the present disclosure, the second processor 610 may generate the object information by receiving the real image photographed by the projection apparatus and performing object recognition therefrom. In another embodiment, the second processor 610 may perform object recognition directly on the projection apparatus. Afterwards, the generated object information may be received.

According to various embodiments of the present disclosure, the second processor 610 may confirm a control operation of the mobile terminal device 600 corresponding to the object information and perform the identified control operation. The memory 640 may store a table that maps functions that may be performed on the mobile terminal device 600 according to each user gesture, and may be allocated to each application. When the object information is received, the processor may find object information to be performed on the executed application (or on another application or on another device) in the table and perform a corresponding operation.

For example, in a state in which the e-book application is executed, the gesture for moving the user's hand to the right may correspond to the operation of turning to the next page and the gesture of moving the left to the previous page. In addition, the gesture to move downward may correspond to an operation of terminating the e-book application. As another example, a gesture of moving a user's hand in the direction in which the user moves his / her hand while the message application is executed may correspond to an operation of outputting a received message as a voice and a gesture of moving the gesture in the opposite direction to delete the corresponding message. Can be. Also, when a user inputs a message by voice after executing a message application, a gesture of moving a hand upward may correspond to an operation of transmitting a message.

Various embodiments related to the function control will be described in more detail with reference to FIGS. 7 to 8.

According to various embodiments, the second processor 610 may generate a virtual image (eg, the virtual image 130 of FIG. 1). The second processor 610 transmits the generated virtual image to the projection apparatus through the second communication circuit 620, and the projection apparatus may project the virtual image into the real world.

According to various embodiments, the virtual image may be a still image or a video, and may be a 3D stereoscopic image. The virtual image may include a character image. The character may utilize various kinds of well-known characters, and the type of the character image may be determined according to a user's selection. The character may be changed according to at least one event occurring on the mobile terminal device 600.

The virtual image may be based on, for example, an image generated by an application or a widget executed by the second processor 610. According to an embodiment, images generated by various applications such as a clock, weather, music, SNS, a schedule, and an email may be included in the virtual image. For example, the second processor 610 may include time information generated by the clock application in the virtual image, and may further include text and character images related to the clock notification.

According to various embodiments, the second processor 610 may detect at least one event occurring in the portable terminal device 600 and generate a virtual image in response to the detected event. In this case, the event may be an event occurring on an executing application.

For example, when a specific schedule time registered in the schedule application arrives, a virtual image including schedule information may be generated and provided to the projection apparatus. In addition, when a notification arrives from the SNS and the message application, a virtual image including the corresponding notification information may be generated and provided to the projection apparatus. According to an embodiment, sound data including event information (eg, message content, notification sound effect, etc.) generated in an application may be provided to the projection apparatus together with the virtual image.

According to various embodiments, the second processor 610 may generate a virtual image based on the object information received from the projection device. For example, the second processor 610 obtains the current state of the user (for example, emotional information, etc.) or the surrounding situation (for example, a place / time estimation based on surrounding objects, etc.) from the recognized object information. You can create a suitable virtual image.

According to various embodiments, the virtual image includes a predetermined character, and the second processor 610 may generate emotion information of the character included in the virtual image and correct the character included in the virtual image according to the emotion information. have.

For example, if the user is in a pleasant mood, the character image may have a bright or cute shape. If the user is depressed, the character image may have a dark expression or a slightly gloomy shape. Can be.

The emotion information may be classified according to a person's emotion type such as white music, and the memory 640 may store various character images corresponding to each emotion information.

According to various embodiments, the second processor 610 may generate the emotion information by referring to the information of the running application. For example, referring to a schedule created in a schedule application, the type and number of schedules can be referred to. If today's schedule is a class, a test, etc., and the number is large, the user's emotion can be predicted by depressed emotion. Accordingly, the second processor 610 may output a character image corresponding to the depressed emotion. As another example, the second processor 610 may estimate the emotion of the user by referring to the text or the emoticon of the message sent or received in the message application.

According to another embodiment, the second processor 610 may generate the emotion information based on the object related information received through the second communication circuit 620. For example, when the object sensor of the projection device photographs the user's face and the facial expression of the user is recognized as a depressed expression, the second processor 610 may determine that the user's emotion is depressed with reference to the object related information. Character images corresponding thereto may be generated.

Various embodiments related to generation and correction of the virtual image will be described in more detail with reference to FIGS. 9 to 10.

As shown, the second processor 710 (eg, the second processor 610 of FIG. 6) may include a gesture recognition module 712, a function mapping module 714, and a function control module 716. . Each module may be a set of programs executed by the second processor 710 as a software module.

The gesture recognition module 712 may recognize the gesture of the user through the object information received from the projection device. For example, the user's gesture may be a gesture of moving a hand down or up, or a gesture having a smiley face. In addition, the gesture recognition module 712 may also obtain object information about an object that is not a user.

The gesture recognition module 712 may map gestures of various types of users to simple operations based on the object information. For example, the form of the user moving the hand down will be very diverse, the gesture recognition module 712 analyzes the object information to classify the various forms as the user moving the hand down, identification corresponding to You can give a code.

The function mapping module 714 may check a control operation mapped with the type of the gesture recognized by the gesture recognition module 712. According to an embodiment of the present disclosure, the portable terminal device may store a table that maps functions that can be performed on the portable terminal device (or another device) in response to each user gesture, and the function mapping module 714 stores the table. You can check the control operation with reference.

According to an embodiment, the mapped functions may be allocated for each application. For example, for the same motion as the user moves his hand down, the music playback application may have a control command mapped to decrease the volume, and the calendar application may display a control command mapped to display the calendar for the next month. have. The function mapping module 714 may identify an application currently being executed by the second processor 710, and identify on the table a gesture of the recognized user and a function mapped to the currently running application.

This may be allocated for each application. When the object information is received, the processor may find object information to be performed on the executed application (or on another application or on another device) in the table and perform a corresponding operation.

According to an embodiment, the function mapping module 714 may identify a corresponding control operation based on the voice data input from the user in addition to the object information. The voice data may be obtained by the projection device or the portable terminal device.

According to an embodiment, the voice data may be for selecting an application or device to perform a mapped function, or for confirming a corresponding control operation in consideration of both object information and voice data.

The function control module 716 may perform the identified control function. According to an embodiment, the function control module 716 may transmit a command to a component (eg, the display 630 or a sound output device) that performs the identified control operation. According to an embodiment, the identified control operation may be an operation performed by an external device (eg, TV, lighting, boiler, electric / gas, washing machine, door lock, etc.) based on IoT, and in this case, the function control module ( 716 may transmit a corresponding command to an external device through a second communication circuit (eg, the second communication circuit 620 of FIG. 6).

8A illustrates a control operation performed on the mobile terminal device 820 according to the movement of a user's hand.

The projection device 810 may photograph an area where the user's hand is located in the real world and perform object recognition. In this case, it is assumed that the user has made an operation 830 for moving the hand from right to left.

The projection device 810 may transmit object information including the user's gesture 830 to the mobile terminal device 820.

A second processor (eg, the second processor 610 of FIG. 6) of the mobile terminal device 820 corresponds to a user's gesture in a table stored on a memory (memory 640 of FIG. 6) using object information. The control operation can be confirmed. According to an embodiment of the present disclosure, the second processor may further check voice data collected by the user using a microphone (not shown), and check a control operation corresponding thereto.

The second processor may perform a control operation corresponding to the gesture of the user in the currently running application. As shown, the control operation corresponding to the gesture 830 in which the user moves his hand to the left in the e-book application may be an operation of turning a page of the e-book.

According to an embodiment of the present disclosure, if there is no control operation corresponding to the recognized user gesture and the application currently running in the foreground (eg, the e-book application of FIG. 8A), the second processor may execute the application running in the background (eg The control operation (eg, change to the next track) corresponding to the music application) and perform the identified control operation for the corresponding application. The priority of an application performing a control operation may be determined according to a user's setting or an attribute of an application running in the background.

8B illustrates an operation for providing information related to a recognized object.

Referring to FIG. 8B, the object information obtained by the projection apparatus 810 may include text or image information in a user's hand and an object (eg, a book, paper, etc.).

The second processor of the mobile terminal device 820 may recognize the user's finger 830 and check the text (eg, cygbot) around the finger 830. The second processor may recognize the text and check the information corresponding to the text (eg, the URL of the cygbot homepage) through the Internet. The portable terminal device 820 may provide the confirmed information to the projection device 810, and the projection device 810 may provide augmented reality by projecting the acquired information around the corresponding text.

According to another exemplary embodiment, the recognized text may be translated and displayed in another language, or the recognized text and related information may be retrieved from information stored in the mobile terminal device 820 (eg, a contact or a memo). When an object is an image, information corresponding to the image may be searched and provided.

8C illustrates an operation of transmitting a command of a control operation performed by another device.

According to an embodiment of the present disclosure, when the control operation corresponding to the object information is a control operation performed on an external device different from the portable terminal device 820, the second processor of the portable terminal device 820 controls the control. Information related to the operation may be transmitted to the other external device 850.

As illustrated, the portable terminal device 820 may be wirelessly connected to another external device 850 (eg, a television). When the user's gesture recognized by the projection device 810 is a control operation (for example, a channel change) performed by the external device 850, the second processor of the portable terminal device 820 may provide a command related to the control operation. It may transmit to the external device 850.

Accordingly, the projection device 810 and the portable terminal device 820 are used to control various electronic devices (eg, TV, lighting, boiler, electric / gas, washing machine, door lock, etc.) based on the Internet of things (IoT). can do.

As shown, the second processor 910 (eg, the second processor 610 of FIG. 6) may include a character generation module 912, an emotion module 914, and a virtual image generation module 916. . Each module may be a set of programs executed by the second processor 910 as a software module.

According to various embodiments, the second processor 910 may include each module (eg, gesture recognition module 712, function mapping module 714, function control module 716) included in FIG. 7 and FIG. 9A. Each module may be included in all or only some of them.

The character generation module 912 may generate a character to be displayed through the virtual image. The character may utilize various kinds of well-known characters, and a type of character to be included in the virtual image may be determined according to a user's selection.

The emotion module 914 may predict an emotion of the user based on various pieces of information identified on the portable terminal device, and accordingly, may perform an operation for reflecting the emotion information on the character.

According to an embodiment, the second processor 910 may generate emotion information of the character in response to at least one event (eg, an event occurring in an application) occurring in the portable terminal device. In addition, the second processor 910 may generate the emotion information of the character further based on the object information received from the projection apparatus. That is, the second processor 910 may predict the current emotional state of the user by using the application installed by the user and the gesture of the user in the portable terminal device.

For example, referring to a schedule created in a schedule application, the type and number of schedules can be referred to. If today's schedule is a class, a test, etc., and the number is large, the user's emotion can be predicted by depressed emotion. For another example, the second processor 910 may estimate the emotion of the user by referring to the text or the emoticon of the message sent or received in the message application.

In addition, when the user's facial expression is recognized as a depressed expression through the object information received from the projection apparatus, the second processor 910 determines that the user's emotion is depressed with reference to the object information and accordingly the character image Can be corrected.

According to an embodiment of the present disclosure, the second processor 910 may obtain emotional information of the user using physical health information obtained from a wearable device (for example, a smart watch, a smart glass, etc.) worn by the user.

The second processor 910 may correct the character included in the virtual image based on the predicted emotion information. For example, if the user is in a pleasant mood, the character image may have a bright or cute shape. If the user is depressed, the character image may have a dark expression or a slightly gloomy shape. Can be.

Emotional information may be classified according to a person's emotion type such as white music, and the memory may store various character images corresponding to each emotion information.

9B illustrates each process for generating emotion information according to various embodiments of the present disclosure.

The illustrated method of generating emotion information may be performed by a second processor of the portable terminal device.

The second processor may acquire behavior pattern information and environment information of the user. At least some of these may be obtained from the projection apparatus. The second processor may recognize the emotional state of the current user using the obtained information.

The second processor may determine attributes (eg, facial expressions, behaviors, etc.) to be applied to the character based on the emotional state of the user and the basic inclination of the character to be used for the virtual image. According to an embodiment, the second processor may determine an attribute to be applied to the character by using the character's characteristic and the predicted user's emotion information on the emotion DB.

The second processor may reflect the determined attribute on the character, generate a virtual image including the character, and provide the same to the projection apparatus. When it is determined that the emotional information of the user is changed while the projection apparatus is projecting the virtual image, the second processor may change the character of the virtual image and provide it to the projection apparatus.

Accordingly, the augmented reality system can communicate emotionally with the user, not just information transmission.

FIG. 10A illustrates a screen on which a virtual image 1000a is generated based on information of a clock widget.

The virtual image 1000a may include time information 1010a generated by the widget. The second processor (eg, the second processor 610 of FIG. 6) may generate text information 1012 a from time information. According to an embodiment, the generated text information 1012a may be output as sound by a sound output device (not shown) at least partially simultaneously with the display of the virtual image 1000a.

A predetermined character 1050a may be displayed in one region of the virtual image 1000a. At least a part of the character 1050a may be changed based on emotion information of the user.

The second processor may provide the generated virtual image 1000a to the projection apparatus, and the projection apparatus may project the virtual image 1000a to the real world.

10B illustrates that the character image is changed according to the emotion information of the user.

According to an embodiment of the present disclosure, the second processor may generate the emotion information of the user with reference to the contents registered in the certain application. For example, if the registered schedule of today and tomorrow is a schedule of generally enjoyable content such as a class reunion, the second processor may predict that the user is a pleasant emotion and correct the character 1050b with a smile.

Accordingly, the user may directly predict the schedule of the day through the expression of the character 1050b without seeing exactly what the schedule 1010b is specified in the schedule application.

10C illustrates that the character image is changed according to the gesture of the user.

According to an embodiment, the second processor may predict the emotional state of the user based on the object information received from the projection apparatus. For example, when it is recognized that the user moves his hand quickly (1070c) or makes an angry facial expression, the facial expression of the character 1050c may be changed to correspond thereto.

10A to 10C are merely exemplary embodiments and may generate emotions of various character images based on various input information in addition to application and object information. According to various embodiments, the mobile terminal device may display at least a portion of the generated virtual image on a display of the mobile terminal device.

According to various embodiments, the second processor may store the recognized object information in a memory. In this case, the second processor may select an application to use for storage based on the user gesture.

As shown, the business card image may be recognized by the projection device. The second processor may recognize that the card is a business card based on the object information, and search for an application (eg, a business card APP, a memo APP, a schedule APP, a word APP) capable of storing a business card image.

The second processor may generate a user interface for selecting one of the searched applications for storing the recognized business card image, and include the user interface in a virtual image and provide the same to the projection apparatus.

The projection device may project the virtual image, recognize the direction of the user's gesture, and provide the same to the portable terminal device. The second processor may store the recognized business card image using an application corresponding to the direction of the user's gesture.

According to various embodiments of the present disclosure, since a two-way interaction is possible using a projection device (eg, the projection device 110 of FIG. 1) and a portable terminal device (eg, the mobile terminal device 120 of FIG. 1), It can be used as a secretary robot that serves as a secretary to the user.

For example, in relation to the time information, the portable terminal device may output a sound for notifying the time at a specific time or a time registered by the user, and the projection device may project a virtual image related to the time information. In this case, the user may input a predetermined control command for controlling the mobile terminal device or another device by performing a gesture registered in advance on the virtual image.

12 illustrates various fields that can be implemented through the augmented reality system as described above, and various embodiments of the present invention are not limited thereto.

13 is a flowchart illustrating a virtual image providing method according to various embodiments of the present disclosure.

The illustrated method may be performed by the projection apparatus (for example, the projection apparatus 110 of FIG. 1) and the portable terminal apparatus (for example, the portable terminal apparatus 120 of FIG. 1) described above. Technical description thereof will be omitted.

In operation 1310, the projection device and the portable terminal device may establish a wireless communication connection. Accordingly, the object information recognized by the projection apparatus may be provided to the portable terminal apparatus, and the portable terminal apparatus may provide a virtual image to the projection apparatus.

In operation 1320, the portable terminal device may detect an event occurring on the portable terminal device. According to an embodiment, the event may include an event (eg, notification) occurring on an application (eg, a schedule, SNS, a message application, etc.) installed and executed in the portable terminal device.

In operation 1330, the mobile terminal device may generate a virtual image. According to an embodiment, the mobile terminal device may generate a virtual image corresponding to the detected event. According to an embodiment of the present disclosure, the virtual image includes a predetermined character image, and the portable terminal device generates user's emotion information by referring to an event occurring on the portable terminal device and / or received object information and the like, and generates the emotion information. Based on this, the character image can be corrected.

In operation 1340, the portable terminal device may provide a virtual image to the projection device, and the projection device may project it to the real space.

14 is a flowchart illustrating a virtual image providing method according to various embodiments of the present disclosure.

In operation 1410, the portable terminal device may receive object information from the projection device.

In operation 1420, the portable terminal device may check a control operation of the portable terminal device (or another device) corresponding to the object information.

In operation 1430, the portable terminal device may perform the identified control operation.

Claims

In a portable terminal device,

display;

Memory;

Communication circuitry for communicating with an external projection device; And

A processor electrically connected with the display, memory, and communication circuitry;

The processor,

Detect at least one event occurring in the portable terminal device,

Generates a virtual image in response to the event,

And transmit the virtual image to the projection apparatus through the communication circuit so that the projection apparatus projects the virtual image to the outside.
The method of claim 1,

The processor,

Executing at least one application stored in the memory,

The mobile terminal device to generate the virtual image according to the event generated in the application.
The method of claim 2,

The virtual image includes at least one character,

The processor,

And generating emotion information of the character according to an event generated in the application, and correcting a character included in the virtual image according to the emotion information.
The method of claim 3, wherein

The communication circuit,

The projection apparatus receives object information generated by recognizing an external object,

The processor,

The mobile terminal device is further configured to generate emotion information of the character based on the received object information.
In the virtual image providing method of a mobile terminal device,

Detecting at least one event occurring in the portable terminal device;

Generating a virtual image in response to the event; And

Transmitting the virtual image to an external projection device to cause the projection device to project the virtual image to the outside.
Projection devices; And

A mobile terminal device,

The projection device,

A projection module for projecting the virtual image to the outside;

An object sensor for recognizing surrounding objects;

A first communication circuit for communicating with the portable terminal device; And

A first processor configured to transmit object information related to the recognized object to the portable terminal device through the first communication circuit,

The portable terminal device,

Second communication circuitry for communicating with the projection device; And

Detects at least one event occurring in the portable terminal device, generates a virtual image in response to the event, and transmits the virtual image to the projection device through the communication circuit so that the projection device externally transmits the virtual image. Augmented reality system comprising a processor set to project to.
In a portable terminal device,

display;

Memory;

Communication circuitry for communicating with an external projection device; And

A processor electrically connected with the display, memory, and communication circuitry;

The processor,

Receiving the object information generated by the projection apparatus recognizes an external object through the communication circuit,

Confirm a control operation of the portable terminal device corresponding to the object information;

And a portable terminal device configured to perform the identified control operation.
The method of claim 7, wherein

The processor,

And when the control operation corresponding to the object information is a control operation performed on another external device, causing the communication circuit to transmit information related to the control operation to the other external device.
The method of claim 7, wherein

The object includes a part of the user's body,

The processor determines the control operation according to the moving direction of the object.
Projection devices; And

A mobile terminal device,

The projection device,

A projection module for projecting the virtual image to the outside;

An object sensor for recognizing surrounding objects;

A first communication circuit for communicating with the portable terminal device; And

A first processor configured to transmit object information related to the recognized object to the portable terminal device through the first communication circuit,

The portable terminal device,

Second communication circuitry for communicating with the projection device; And

And a second processor configured to check a control operation of the portable terminal device corresponding to the object information and to perform the identified control operation when object information is received from the second communication circuit.