KR102218210B1 - Smart glasses capable of processing virtual objects - Google Patents

Abstract

Smart glasses are disclosed that have a display and are capable of handling virtual objects. Smart glasses are a frame assembly that forms a shape of glasses by a plurality of frames, an electrical component mounted on the frame assembly, and a first display controlled by a control device of the electrical component and disposed at the upper center and lower center of the frame assembly. The device, the second display device, and the first main lens and the second main lens by expanding and reducing or focusing the image size while transmitting the first image output from the first display device and the second image output from the second display device Each includes an optical system projecting onto the image, and the first image and the second image are viewed by the user's eyes on the first main lens and the second main lens.

Description

Smart glasses that can process virtual objects{SMART GLASSES CAPABLE OF PROCESSING VIRTUAL OBJECTS}

Embodiments of the present invention relate to smart glasses, and more particularly, to smart glasses that have a camera and a display device and can process virtual objects.

Head Mount Display (HMD) refers to a digital device that is worn on the face like glasses and displays a virtual image on glasses in a form close to the eyeball with multimedia contents.

HMD can provide various conveniences to users by combining augmented reality technology, etc., beyond a simple display function, and can communicate with external digital devices to output relevant contents, as well as users for external digital devices. It also supports receiving inputs or performing tasks linked with the corresponding external digital device. Recently, research and development on HMD has been actively conducted.

Augmented reality is one of the hybrid virtual reality technologies that combine reality and virtual environments by using technology to superimpose 3D virtual objects on the real world. Such augmented reality is gradually becoming a reality, out of the imaginary stage depicted in science fiction novels and movies, as it is used in various fields such as military, entertainment, medical care, learning, film, architectural design, and tourism.

Augmented reality is a field of virtual reality, and virtual reality systems are classified into window systems, mirror systems, vehicle-based systems, and augmented reality systems. Can be.

The virtual reality system includes an input device and an output device. Output devices are devices that allow users to perceive sight, hearing, tactile sense, movement, etc. through sensory channels, and include a visual display device, an auditory display device, a tactile display device, a motion feedback and an image display device. Representative hardware among visual display devices is HMD and smart glasses.

The goal of virtual reality is to allow users to experience telepresence. Presence means a sense of being felt in a certain environment. In this sense, remote presence can refer to being in a certain virtual environment by a communication medium, and a virtual reality system or augmented reality system. Allows users to experience remote presence.

However, conventional smart glasses supply an image to one eyeball, either left or right, and the user observes multimedia content only with one eye, so that side effects such as dizziness occur even after observing only a certain time.

In order to improve this problem, to supply multimedia contents to both eyes, smart glasses having a small projector supplying contents to the left eye and a small projector supplying contents to the right eye, respectively, have been proposed. When content is sent to both eyes of a user, there is a problem that the user feels greatly uncomfortable, such as dizziness or a lot of fatigue because the content delivered to both eyes is not identical to each other in terms of the user's perspective and is not linked to each other.

Korean Patent Publication No. 10-1549074 (2015.08.26.) Korean Registered Patent Publication No. 10-1588402 (2016.01.19.)

The present invention has been conceived to solve the above-described problems, and an object of the present invention is to provide smart glasses having an optical system capable of greatly reducing user fatigue and dizziness by projecting an image from the upper or lower direction of the face. There is.

Another object of the present invention is to reflect the multimedia content provided by the display in both directions through a mirror around the eye, and partially reflect the content magnified by the magnifying lens unit or the auxiliary lens to both eyes through a beam splitter, and partially transmit it to the front of the eyeball. The aim is to provide smart glasses that enable observation of the same image content formed in a virtual image in both eyeballs, and implement 3D content as needed, or easily implement augmented reality or virtual reality using 3D content.

Smart glasses according to an aspect of the present invention for solving the above technical problem, a frame assembly having an eyeglass form; Electrical components mounted on the frame assembly; A display system including a first display device and a second display device supported by the frame assembly, controlled by a control device of the electronic component, and disposed at an upper center side and a lower center side of the frame assembly; And a first image and a first image respectively output from the first display device and the second display device, which are horizontally coupled to the display system through a case assembly fixed to the frame assembly, and are arranged in a vertical direction within the display system. 2 It includes an optical system that transmits the image in both directions in the horizontal direction through a mirror positioned between the first display device and the second display device.

In one embodiment, the optical system reflects the first image input in a first direction from the upper central side toward the lower central side from a first surface and transmits the first image in a second direction orthogonal to the first direction, The second image, which is input from the lower central side toward the upper central side, is reflected from a second surface opposite to the first surface and is orthogonal to the first reverse direction, and is opposite to the second direction. A mirror transmitting in a second reverse direction; A first right lens in the form of a concave lens through which the first image transmitted from the mirror passes; A second right lens in the form of a convex lens for passing the first image passing through the first right lens again; The first main lens reflecting a first image passing through the second right lens and supported on one front side of the frame assembly; A first left lens in the form of a concave lens through which a second image transmitted from the mirror passes; A second left lens in the form of a convex lens for passing a second image passing through the first left lens again; And the second main lens reflecting a second image passing through the second left lens and supported on the other front side of the frame assembly.

In one embodiment, the first main lens or the second main lens, a hexahedral block-shaped body; And a mirror surface, a radial surface, or a beam split disposed in the body to be inclined with a projection direction of the first image passing through the second right lens or the second image passing through the second left lens.

In an embodiment, the control device may control the first display device or the second display device so that the second image is shifted by a predetermined pixel in one direction based on the first image to correspond to binocular parallax.

In one embodiment, the control device may control the first display device or the second display device to sequentially or alternately output the first image and the second image to display a three-dimensional image or a three-dimensional image. .

In one embodiment, the first display device or the second display device includes a liquid crystal display (LCD), a digtal light processing (DLP) projector, a liquid crystal on silicon (LCoS) device, a plasma dispaly panel (PDP), and a FED. (field emission display), ELD (elelctro-luminescent display), and may include any one of OLED (organic light emitting diode) device.

The optical system of smart glasses according to another aspect of the present invention for solving the above technical problem is an optical system of smart glasses that is worn on a user's face in the form of glasses to display an image in front of the user's eyes, and has a frame having an eyeglass form. The first image input in a first direction from the front center upper side of the assembly toward the front center lower side is reflected from the first surface and transmitted in a second direction orthogonal to the first direction. The second image, which is input in a first reverse direction toward the front center upper side, is reflected from a second surface opposite to the first surface, and is transmitted in a second reverse direction, which is orthogonal to the first reverse direction and opposite to the second direction. A mirror; A first right lens in the form of a concave lens through which the first image reflected from the mirror passes; A second right lens in the form of a convex lens for passing the first image passing through the first right lens again; The first main lens reflecting a first image passing through the second right lens and supported on a left front side of the frame assembly; A first left lens in the form of a concave lens through which a second image reflected from the mirror passes; A second left lens in the form of a convex lens for passing a second image that has passed through the first left lens again; And the second main lens reflecting a second image passing through the second left lens and supported on the front right side of the frame assembly.

In one embodiment, the combination structure of the first right lens and the second right lens or the combination structure of the first left lens and the second left lens is viewed from the mirror side, the front or one side is concave and the rear or The other surface may have a convex single lens shape.

In one embodiment, the first main lens or the second main lens, a hexahedral block-shaped body; And a mirror surface, a radial surface, or a beam split disposed in the body to be inclined with a projection direction of the first image passing through the second right lens or the second image passing through the second left lens.

In one embodiment, the smart glasses may include a camera or an infrared projector mounted on the frame assembly; And a control board or processor mounted or supported in the frame assembly. The control board or processor recognizes an object including any one of a palm, a back of a hand, a forearm, a desk, a door, and a wall, and displays a virtual object or a user interface (UI) image based on the structural characteristics of the object. The area may be searched, and the virtual object or UI image may be generated in the searched area along with a background.

In one embodiment, the control board or processor recognizes an end of another object that is close to the virtual object or UI image, and further performs a preset operation on the virtual object or UI image corresponding to the end of the other object. can do. In addition, when the other object is the other hand, the control board or the processor may recognize a gesture of the hand by extracting an outline of the other hand or extracting a joint.

Smart glasses according to another aspect of the present invention for solving the above technical problem, a frame assembly having a spectacle frame shape; A display device supported by the frame assembly and positioned on a central portion of the face when the frame assembly is worn on a user's face; And an optical system supported by the frame assembly and transmitting an image of the display device in a direction in which both eyes of the user are located when the frame assembly is worn on the user's face, wherein the optical system comprises the At least one main lens partially reflecting and partially transmitting an image transmitted from the display device in front of the user's eyes, and the image visible to the user is an image formed on a virtual image surface a predetermined distance from the back of the main lens To do.

An optical system that can be employed in smart glasses according to another aspect of the present invention for solving the above technical problem is an optical system of smart glasses that is worn on a user's face in the form of a glasses frame to provide an image to the user's eyes, and includes a display A mirror reflecting the image of the device; An auxiliary lens to magnify the image reflected from the mirror; A main lens partially reflecting and partially transmitting the image magnified by the auxiliary lens; And a frame assembly in the form of a spectacle frame supporting the mirror, the auxiliary lens, and the main lens, and when the frame assembly is worn on the user's head, the mirror, the auxiliary lens, and the main lens are It is characterized in that it is located in the center of the face, the brow or above the nose.

In the case of employing the smart glasses and the optical system thereof according to the present invention, it is possible to significantly reduce fatigue and dizziness of the user's eyes, thereby improving the fit and convenience of use. In addition, it is possible to provide an excellent fit by minimizing weight and designing with an ergonomic structure and design. In addition, there is an advantage in that it is possible to easily supply power for a long time use by a battery replacement method.

In addition, by preventing fatigue and dizziness and increasing the fit, it is used for a long time when using contents such as virtual reality (VR), augmented reality (AR), converged reality (MR), and mixed reality (MR) using smart glasses. You can provide an environment. In addition, it can be applied effectively or easily to various fields such as working or working without a monitor, shopping, using as a navigation device, playing AR, VR or MR games, watching videos, or making video calls. There is this.

In addition, according to the present invention, there is an advantage of securing a side viewing angle by configuring to provide independent images to both eyes on one or two displays disposed at the front center of the smart glasses.

In addition, there is an effect of effectively implementing 2D and 3D images with one smart glasses, and implementing augmented reality and virtual reality.

In addition, by providing the same multimedia content to both eyes on one or two displays, there is an effect that the user does not feel uncomfortable even when used for a long time.

In addition, by reducing the thickness of the front main lens, the device can be lightened and the feeling of use can be improved.

In addition, by recognizing the user's eyelids through a rear camera or sensor and generating vibrations or alarms through actuators, actuators, or speakers housed in the device according to the recognition result, the user, in particular, a drowsy driver can be warned of danger.

In addition, it is possible to provide operational stability and reliability for smart glasses by effectively discharging heat that may be generated when the device is used for a long time through a heat dissipation structure or a heat sink disposed on the front top of the main frame.

In addition, when the battery is removed from the smart glasses and power is supplied to the smart glasses through the battery to an external control and power device, the smart glasses can be effectively lightened and the fit and convenience of use can be increased.

In addition, augmented reality service is provided using a stereo RGB camera, stereo infrared camera, or infrared projector mounted on smart glasses, or a user command is input through a user interface image or virtual object displayed in augmented reality, and a preset operation is performed accordingly. Can be done. According to this configuration, there is an advantage of providing an augmented reality service and various 3D image services using augmented reality while maximizing user convenience.

1 is a perspective view of smart glasses according to an embodiment of the present invention.
2 is an exploded perspective view of the smart glasses of FIG. 1.
3 is a partial exploded perspective view for explaining the main operating principle of the smart glasses of FIG. 1.
4 is a perspective view showing an enlarged structure of the rear case of the smart glasses of FIG. 3.
5 is a block diagram illustrating a main configuration of a main control board that can be employed in smart glasses according to another embodiment of the present invention.
6 is a perspective view of smart glasses according to another embodiment of the present invention.
7 is a perspective view of smart glasses according to another embodiment of the present invention.
8 is a front view of the smart glasses of FIG. 7.
9 is a partial projection front view for explaining the internal structure of the smart glasses of FIG. 8.
10 is a right side view of the smart glasses of FIG. 7.
11 is a partial projection right side view showing the internal structure of the smart glasses of FIG. 10.
12 is a partial exploded perspective view for explaining the components of the smart glasses of FIG. 7.
13 and 14 are views for explaining the optical system of the smart glasses of FIG. 7.
15 is a block diagram of a control board that can be employed in the smart glasses of FIG. 7.
16 is a perspective view of smart glasses and a control system according to another embodiment of the present invention.
17 is a block diagram illustrating a main configuration of the smart glasses of FIG. 16.
18 is a block diagram illustrating a control module that can be employed in the smart glasses of FIG. 16.
19 and 20 are flowcharts illustrating a main operating principle of the smart glasses of FIG. 16.
21 is a view for explaining a user interface that can be employed in smart glasses according to another embodiment of the present invention.
22 and 23 are exemplary diagrams for explaining a modified example of a user interface that can be employed in smart glasses according to another embodiment of the present invention.
24 is a diagram for explaining the operating principle of a user interface that can be employed in the smart glasses of the present embodiment.
25 and 26 are flowcharts illustrating a signal processing process of a user interface employed in smart glasses according to another embodiment of the present invention.
27 is a perspective view of smart glasses according to another embodiment of the present invention.
28 is a front view of the smart glasses of FIG. 27.
29 and 30 are diagrams illustrating a state of use of the smart glasses of FIG. 28.
31 is a front view of smart glasses according to another embodiment of the present invention.
32 and 33 are diagrams illustrating a state of use of the smart glasses of FIG. 31.
34 and 35 are perspective views showing the appearance of a device for powering and controlling smart glasses according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same reference numerals shown in each drawing indicate the same member.

1 is a perspective view of smart glasses according to an embodiment of the present invention. 2 is an exploded perspective view of the smart glasses of FIG. 1. 3 is a partial exploded perspective view for explaining the main operating principle of the smart glasses of FIG. 1. And Figure 4 is a perspective view showing an enlarged structure of the rear case of the smart glasses of Figure 3;

1 to 4, the smart glasses 100 according to the present embodiment include a frame assembly 10, an electronic component, a display system 30, and an optical system 50. The display system 30 may include a case assembly 60 and an optical system 50 in addition to the first display device 31 and the second display device 32.

The display devices, the optical system 50, and the case assembly 60 may be cross-arranged in a form arranged on three different axes in a Cartesian coordinate system, and a mirror 51 is disposed at the center of this cross-array structure, The first display device 31 and the second display device 32 may be disposed on both sides with the mirror 51 interposed therebetween.

The frame assembly 10 forms a shape of glasses by a plurality of frames. The frame assembly includes an upper frame (11), a lower frame (12), a body frame (13), a right cover frame (14) and a left cover frame (15). ) Can be included. A nose pad 18 may be coupled to the body frame 13.

The lower frame 12 accommodates the main control board 21 in the central receiving portion. The central receiving portion may be covered by the upper frame 11. The lower frame 12 includes a first printed circuit board 23 and a first battery pack 24 connected to the first printed circuit board 23 in a first receiving part in a right side extension connected to the right side of the central receiving part. ) Can be stored. The first accommodating part may be covered by the right cover frame 14. A button-type power button 22 is installed on the right cover frame 14, and the power button 22 may be connected to the first printed circuit board 23. The first printed circuit board 23 may connect the battery management system in the first battery pack 24 and the control device 210 of the main control board 21.

In addition, the lower frame 12 is a second printed circuit board 25 and a second battery pack connected to the second printed circuit board 25 in the second receiving portion in the left side extension connected to the left side of the central receiving portion. (26) can be stored. The second storage unit may be covered by the left cover frame 15. The second printed circuit board 25 may connect the battery management system in the second battery pack 26 and the control device 210 of the main control board 21.

The body frame 13 is coupled to the lower portion of the lower frame 12. The body frame 13 is installed between the lower frame 12 and the case system 60 to support their coupling. In addition, the body frame 13 is coupled to the lower portion of the lower frame 12 to complement and increase the overall durability of the smart glasses.

The electronic components may be mounted or accommodated in the frame assembly 10. Electrical components include a main control board (21), a power switch (22), a first control board (23), a first battery pack (24), a second control board (25) and a second battery. Pack 26 may be included. The first control board 23 may correspond to a first printed circuit board, the second control board 25 may correspond to a second printed circuit board, and the battery pack may be simply referred to as a battery.

The first display device 31 and the second display device 32 are controlled by a control device of an electronic component, and may be disposed at an upper center side and a lower center side of the frame assembly, respectively.

In this embodiment, the first display device 31 or the second display device 32 includes an LCD (liquid crystal display). In this case, the first display device 31 and the second display device 32 may include backlight units 33 and 34.

On the other hand, the display device of the present invention is not limited to LCD, and reflective micro-display (liquid crystal on silicon, LCoS), plasma dispaly panel (PDP), field emission display (FED), electro-luminescent display (ELD), and OLED It can be replaced with any one of (organic light emitting diode).

The optical system 50 transmits the first image output from the first display device 31 and the second image output from the second display device 32 to both eyes of the user and controls the image size to control the first main lens ( 54) and the second main lens 57, respectively.

The optical system 50 includes a mirror 51, the first right lens 52, the second right lens 53, the first main lens 54, and A first left lens (55), a second left lens (56), and a second main lens (the second main lens) 57 may be included.

The mirror 51 reflects a first image input in a direction (first direction) from the front center of the smart glasses or from the upper side of the center of the optical system toward the lower side of the center (first direction), and reflects the second image orthogonal to the first direction. Pass in the direction. In addition, the mirror 51 reflects the second image input in a direction from the lower side of the center toward the upper side of the center (a first reverse direction facing the first direction) from a second surface opposite to the first surface. It can be transmitted in a second reverse direction orthogonal to the first reverse direction and opposite to the second direction.

The first right lens 52 may have a concave lens shape and may pass the first image transmitted from the mirror 51. The first right lens 52 may first pass the first image. The second right lens 53 may have a convex lens shape and may secondly pass the first image passing through the first right lens 52. The first right lens 52 and the second right lens 53 may form a single lens shape in which light incident through the concave surface of one end is emitted through the concave surface of the other surface.

The first main lens 54 may reflect and transmit the first image that has passed through the second right lens 54. The first main lens 54 may be supported by a frame assembly or a cover and may be disposed on one front side of the smart glasses. The first main lens 54 may be arranged in front of the user's right eye.

The first left lens 55 has a concave lens shape and may pass a second image transmitted from the mirror 51. The first left lens 55 may first pass the second image in a direction opposite to the first right lens 52. The second left lens 56 may have a convex lens shape and may secondly pass the second image passing through the first left lens 55. The first left lens 55 and the second left lens 56 may correspond to a single lens structure in which light incident through one concave surface is emitted through the convex surface of the other surface, but is not limited thereto.

The second main lens 57 may reflect and transmit the second image passing through the second left lens 56. The second main lens 57 may be supported by a frame assembly or a cover and may be disposed on one front side of the smart glasses. The second main lens 57 may be arranged in front of the user's left eye.

In order to form a reliable structure of the above-described display devices and the optical system 50, in this embodiment, the two display devices of the optical system are orthogonal to the second direction (or horizontal direction) in which the components of the optical system 50 are arranged. Arrange in the first direction (or vertical direction). The two display devices 31 and 32 may be disposed above and below the mirror 51 of the optical system 50 at the center. The mirror 51 is also located in the center of the components of the optical system 50 arranged in the second direction, thereby having the components of the optical system 50 arranged to transmit image signals in two directions opposite to each other. .

To this end, the rear case 62 of the present embodiment, as shown in FIG. 4, the first seating groove 620 in which the mirror 51 is seated, and the first upper seating for seating the first display device 31 It includes a groove 621 and a first lower seating groove 622 for seating the second display device 32. When the first display device 31 is an LCD panel, a second upper mounting groove 631 for accommodating the backlight unit 33 may be further provided above the first upper mounting groove 621. Similarly, a second lower seating groove for accommodating the backlight unit 34 of the second display device 32 may be provided under the first lower seating groove 622.

Further, as shown in FIG. 4, the rear case 62 includes a second seating groove 625 for seating the first right lens 52 and the second right lens 53, and a first main lens 54. It may be provided with a third seating groove 626 for receiving and supporting. Similarly, the rear case 62 has a fourth seating groove 627 for seating the first left lens 55 and the second left lens 56, and for receiving and supporting the second main lens 57. A fifth seating groove 628 may be provided. The first main lens 54 and the second main lens 57 are primarily supported by the front case 61 and the rear case 62, and a front cover 63 surrounding the outside of the front case 61 Wow, it can be firmly supported by the coupling of the rear cover 64 surrounding the outside of the rear case 62.

In addition, the rear case 62 may include a fastening portion 629 for fastening with the body frame 13. The fastening part 629 may have a screw hole.

The front cover 63 may be referred to as a first outer case, in which case the front case 61 may be referred to as a first inner case. Similarly, the back cover 64 may be referred to as a second outer case, in which case the back case 63 may be referred to as a second inner case.

In this embodiment, the optical system does not provide the first image and the second image from the side of the user's face, but transmits the first image and the second image from the forehead and nose side to the glabellar side of the user's face. The image size is adjusted and focused through the concave lens and the convex lens while being transferred to the front, and the first main lens 54 and the second main lens 57 in front of the user's eyes are visible based on the user's perspective. One image and a second image can be displayed.

The smart glasses 100 according to the present embodiment can implement a 3D program without causing substantial inconvenience to the user. That is, according to the present embodiment, gesture recognition, 3D implementation, virtual reality image playback, augmented reality service, and the like can be provided using smart glasses having a simplified structure. In particular, it is convenient for users to use and simple structure of the device to reduce manufacturing cost by solving the problems of use that many existing head mounted display users feel, such as dizziness and nausea while using 3D content using smart glasses. There is an advantage.

The basic signal processing technologies for gesture recognition, 3D realization, virtual reality image playback, and augmented reality service among the services provided by the smarton glasses according to the present embodiment are well known in the art, and thus detailed descriptions thereof will be omitted. do.

5 is a block diagram illustrating a main configuration of a main control board that can be employed in smart glasses according to another embodiment of the present invention.

Referring to FIG. 5, in the main control board 21 according to the present embodiment, the control device 210 may include a communication unit 211, a processor 212, and a memory 213. The control device 2120 may include a controller or a computing device. The processor 212 may be referred to as a control unit.

The communication unit 211 connects the control device 210 to a network. The communication unit 211 may access the Internet or various servers on the network through a network, and may operate in cooperation or interworking with various servers. The server may include a server supporting virtual reality, a server supporting augmented reality, and the like.

The communication unit 211 may include one or more wired and/or wireless communication subsystems supporting one or more communication protocols. The wired communication subsystem is a public switched telephone network (PSTN), ADSL (Asymmetric Digital SubscriberLine) or VDSL (Very high-data rate Digital SubscriberLine) network, a subsystem for PES (PSTN Emulation Service), and an IP (internet protocol) multimedia subsystem. System (IMS), and the like. The wireless communication subsystem may include a radio frequency (RF) receiver, an RF transmitter, an RF transceiver, an optical (eg, infrared) receiver, an optical transmitter, an optical transceiver, or a combination thereof.

The wireless network basically refers to Wi-Fi, but is not limited thereto. In this embodiment, the communication unit 211 includes various wireless networks, for example, Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), and W-Code Division Multiple Access (W-CDMA). , LTE (Long Term Evolution), LET-A (LET-Advanced), OFDMA (Orthogonal Frequency Division Multiple Access), WiMax, Wi-Fi (Wireless Fidelity), may be implemented to support at least one selected from Bluetooth. .

The processor 212 may provide various services through smart glasses by executing a software module or program stored in the internal memory or the memory 213. The processor 212 may be referred to as a microprocessor.

The processor 212 may be implemented as a processor or microprocessor including at least one central processing unit (CPU) or core. The central processing unit or core is a register storing instructions to be processed, an arithmetic logical unit (ALU) in charge of comparison, judgment, and operation, and the CPU internally for interpretation and execution of instructions. It may be provided with a control unit (control unit) to control, and an internal bus connecting them. The central processing unit or core may be implemented as a system on chip (SOC) in which a micro control unit (MCU) and a peripheral device (an integrated circuit for an external expansion device) are disposed together, but is not limited thereto.

Further, the processor 212 may include one or more data processors, image processors, or codecs, but is not limited thereto. The control unit 212 may include a peripheral device interface and a memory interface. The peripheral device interface may connect the processor 212 and an input/output system such as an output device or other peripheral devices, and the memory interface may connect the processor 212 and the memory 213.

In this embodiment, the processor 212 includes the first battery pack 24 and the second battery in order to secure a relatively long operating time for a single charge of the battery even under an operating condition that consumes a relatively large amount of power by the image display. Controls the pack 26. To this end, the processor 212 may be connected to and communicate with the battery management system (BMS) 240 of each battery pack. In addition, the processor 212 may be connected to a control device or a timing controller of at least one or both of the two display devices of the display system 30, and control the operation of the display device through this.

The memory 213 may store a software module for providing services such as object recognition, gesture recognition, virtual reality, and augmented reality through smart glasses. For example, the software module includes: a first module 215 for recognizing a control destination; A second module 216 for providing a 3D image or controlling 3D; A third module 217 for providing a display image service of virtual reality; It may include a fourth module 218 and the like for providing a display image service of an augmented reality that adds a virtual object to the currently visible real world image. Such a module may be implemented by the processor 212 and implemented as at least one task processor that performs a specific function or operation within the processor 212.

The above-described memory 213 is a semiconductor memory such as non-volatile random access memory (NVRAM), a typical volatile memory such as dynamic random access memory (DRAM), a hard disk drive (HDD), and optical storage. It can be implemented as a device, flash memory, or the like. In addition, the memory 56 may store an operating system, a program, an instruction set, and the like, in addition to software modules for implementing the print production service method.

Meanwhile, in the above-described embodiment, components of the smart glasses device may be implemented as functional blocks or modules mounted on various computing devices based on nonvolatile memory (NVRAM). For example, the software modules stored in the memory of the processor of FIG. 5 are stored in a computer-readable medium (recording medium) in the form of software for implementing a series of functions performed by them, or in a storage device in a remote server device in the form of a carrier. It may be implemented to operate on a specific computing device that is stored and connected via a network with the server device. Here, the computer-readable medium may include a plurality of computer devices connected through a network or a memory or storage device of a cloud system, and at least one of the plurality of computer devices or cloud systems provides various services in the smart glasses of the present embodiment. You can save the program or source code to implement.

Further, the computer-readable medium may be implemented in a form of singly or in combination with program instructions, data files, and data structures. The programs recorded on the computer-readable medium may include those specially designed and configured for the present invention or those known and usable to those skilled in computer software.

In addition, the computer-readable medium may include a hardware device specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Here, the program instruction may include a high-level language code that can be executed by a computer using an interpreter or the like as well as a machine language code that is generated by a compiler. The hardware device may be configured to operate as at least one software module to implement all services that can be provided through the smart glasses of the present embodiment, and vice versa.

Meanwhile, the control device 210 of the present embodiment may be implemented as a controller including a control unit, a memory, and a communication unit in a single substrate or a single housing for convenience of description, but is not limited thereto, and includes a control unit and a memory or only a control unit. It can be implemented in a form.

6 is a perspective view of smart glasses according to another embodiment of the present invention.

Referring to FIG. 6, the smart glasses 100a according to the present embodiment include a frame assembly 10 in the form of a glasses frame; And a case assembly 60 integrally supporting a display that outputs an image from the front central portion of the user's face when the user wears smart glasses, and an optical system that reflects, enlarges, partially reflects, and partially transmits the image of the display. .

In this embodiment, the first main lens (see 54 in FIG. 2) and the second main lens (see 57 in FIG. 2) denoted by reference numeral 50a are made of plate-shaped reflective and transmissive members, unlike the above-described embodiment. , The first main lens and the second main lens are disposed to be inclined in a predetermined angle range with the central axis of each. The first main lens and the second main lens may be a beam splitter in the form of a plate member having a thickness of several millimeters or less.

The first main lens and the second main lens partially reflect and partially transmit the image so that the user wearing the smart glasses sees the image formed on the virtual image so that a relatively large image can be viewed through the smart glasses.

7 is a perspective view of smart glasses according to another embodiment of the present invention. 8 is a front view of the smart glasses of FIG. 7. 9 is a partial projection front view for explaining the internal structure of the smart glasses of FIG. 8. 10 is a right side view of the smart glasses of FIG. 7. 11 is a partial projection right side view showing the internal structure of the smart glasses of FIG. 10. And FIG. 12 is a partial exploded perspective view for explaining the components of the smart glasses of FIG. 7.

In this embodiment, for convenience of description, the same reference numeral 100 as the smart glasses 100 or 100a of the above-described embodiments will be designated.

The smart glasses 100 according to the present embodiment include a main frame 110, a first display 131, a second display 132, a first mirror 133, a second mirror 134, and a first main lens. (137) and a second main lens (138). In addition, the smart glasses 100 may further include a first auxiliary lens 135 and a second auxiliary lens 136.

In addition, the smart glasses 100 includes a controller including a processor (refer to 210 of FIG. 15), a display interface 160 for the first and second displays 131 and 132, and a flexible print connecting them. A flexible printed circuit board (FPCB) 164 may be provided. The controller may be referred to as a control and power device, and may be built-in or have separate communication and power boards. The communication and power board may include a connector or a means or component for relaying or interconnecting communication or power.

In addition, the smart glasses 100 may include one or more front cameras 151 and one or more rear cameras 153. When the front camera 151 includes a first front camera and a second front camera that are spaced apart from each other by a predetermined interval, it is possible to easily input or apply 3D content. The front or rear camera is a kind of sensor or can function as a sensor. In particular, the rear camera 153 or the sensor may be a means for detecting a user's eyelid motion or a device for performing a function corresponding to such a means.

In addition, the smart glasses 100 may further include an actuator or actuator 194 that vibrates in response to signals from a camera or sensor or a control and power supply device, or a speaker that outputs an acoustic signal in response to the signal. I can. The driver 194 may include a vibration motor.

In addition, the smart glasses 100 may further include a heat dissipation structure or a heat sink for dissipating heat generated from the first display 131, the second display 132, and the display interface 160. At least one of the first display 131 and the second display 132 may be referred to as a display unit. The display simply refers to a display device, and when the display is a liquid crystal display device, the display may include a backlight unit.

To describe each component in more detail, the main frame 110 is formed of a solid material having an approximately U-shape and has a shape of a frame or a frame of a frame. The main frame 110 may be formed of a single structure, and in that case, the main frame 110 includes a central frame portion 110c and a left frame portion 110a and a right frame portion ( 110b) may be provided. The left and right sides may be opposite sides from the user's point of view.

On the other hand, in this embodiment, the main frame 110 is a combination structure of a support frame disposed in the middle for ease of manufacture and weight reduction, a first side frame coupled to the left side of the support frame, and a second side frame coupled to the right side. It can be provided. In this case, the support frame may be detachably formed as the first support frame 120a and the second support frame 120b in order to facilitate support, arrangement, and assembly of the display, mirror, auxiliary lens, and main lens.

Here, for convenience of explanation, the left frame portion and the first side frame are denoted by a single reference numeral 110a, and the right frame portion and the second side frame are denoted by a single reference numeral 110b. The support frame 120 may refer to a form in which the first support frame 120a and the second support frame 120b are combined.

The display interface 160 is inserted in the central portion of the support frame 120 and may be electrically connected to the controller through a terminal of a flexible printed circuit board (FPCB) 164 disposed on the upper central side. In addition, the display interface 160 is connected to the first display 131 and the second display 132.

The FPCB 164 may be installed extending from the terminal at the central front portion to the first side frame 110a and the second side frame 110b along the upper surface of the main frame 110 or the support frame 120.

The first display 131 and the second display 132 are disposed on both side surfaces of the box-shaped central portion of the support frame 120 to output a first image and a second image in opposite directions. The first display 131 or the second display 132 is a thin film transistor liquid crystal display (TFT LCD), an organic light emitting diode (OLED), a liquid crystal on silicon (LCoS), It may include at least one selected from a digital light processing (DLP) based display.

A first mirror 133 supported by a support structure portion on the left inclined surface of the support frame 120 is installed in the front of the first display 131, and the first image of the first display 131 is substantially perpendicular to the left direction. It can reflect in the first downward direction. Similarly, a second mirror 134 supported by a support structure portion of the right inclined surface of the support frame 120 is installed on the front of the second display 132 and the second image of the second display 132 is directed to the right. It may reflect in a second downward direction approximately orthogonal to. The second downward direction may be a direction parallel to each other with a distance approximately equal to a distance between two eyes of the user.

The first image reflected by the first mirror 133 is enlarged by the first auxiliary lens 135. The first auxiliary lens 135 may be supported by being inserted into a support structure having an uneven structure of the support frame 120. The first auxiliary lens 135 magnifies the first image and transmits it to the first main lens 137.

Similarly, the second image reflected by the second mirror 134 is enlarged by the second auxiliary lens 136. The second auxiliary lens 136 may be supported by being inserted into a support structure having an uneven structure of the support frame 120. The second auxiliary lens 136 magnifies the second image and transmits it to the second main lens 138.

The first auxiliary lens 135 or the second auxiliary lens 136 is a stacked structure or overlapping of a single-sided convex lens (the other-sided concave lens), a double-sided convex lens, and a single-sided convex lens when the image is viewed from the input end side. May have an arrangement. The above-described first and second auxiliary lenses 135 and 136 are for enlarging the first image and the second image to a desired size and removing spherical aberration, and various modifications are possible according to implementation.

The first main lens 137 is disposed under the first auxiliary lens 135. The first main lens 137 may correspond to a first main lens or a second main lens indicated by reference numeral 50 in FIG. 1 or 50a in FIG. 6.

The first main lens 137 may have an arrangement structure in which a relatively thin plate-shaped lens member is installed to be inclined with a central axis of the first auxiliary lens 135. The first main lens 137 may be coupled to the lower side of the support frame 120 or the central frame portion 110c (hereinafter, briefly the central frame) at the lower side of the first auxiliary lens 135. The first main lens 137 is a beam splitter and may reflect a first image or a first image passing through the first auxiliary lens 135 toward the user's eyes at a substantially right angle.

For example, a first image descending from the first mirror 133 in a substantially vertical direction forms an image directed toward the user's eyes and a virtual image toward the front of the user's eyes, centered on the reflective surface of the first main lens 137. The distance from the reflective surface to the location where the bed forms or the image plane may be about 20 meters.

Similarly, the second main lens 138 is disposed under the second auxiliary lens 136. The second main lens 138 may have an arrangement structure in which a relatively thin plate-shaped lens member is installed to be inclined with a central axis of the second auxiliary lens 136. The second main lens 138 may be coupled to a lower side of the support frame 120 or the central frame 110c from the lower side of the second auxiliary lens 136. The second main lens 138 is a beam splitter and may reflect a second image that has passed through the second auxiliary lens 136 toward the user's eye at a substantially right angle.

For example, the second image descending from the second mirror 134 in a substantially vertical direction forms an image directed toward the user's eye and a virtual image toward the front of the user's eye based on the reflective surface of the second main lens 138.

The convex surfaces of the auxiliary lenses 135 and 136 described above may be installed to include three aspherical lenses to enlarge an image or content and eliminate spherical aberration.

In addition, the nose ring member 140 may be connected to the lower central side of the support frame 120. At least a portion of the nose ring member 140 may be made of a relatively soft material. The soft material may include synthetic resin.

The upper cover 113 is coupled to the central upper side of the main frame 110, the front upper cover 114 is coupled to the central front upper side, and the front auxiliary cover 115 may be coupled to the upper center of the central front. The front upper cover 114 may include the first and second support frames 120a and 120b, and may be coupled to the central frame 120c by fastening means such as a rear cover 112c and screws or bolts. The front auxiliary cover 115 may cover the exposed lens portion of the front camera 151, and in that case, at least a portion of the front auxiliary cover 115 may be formed of a transparent or translucent material. In addition, depending on implementation, the front auxiliary cover 115 may have an opening or a through hole for exposing the lens of the front camera 151.

One end of the first side frame 110a may be connected to the left side of the support frame 120, and the other end may be connected to the first side connection frame 121. The first side connection frame 121 may be connected to a first side end frame 122. The first side end frame 122 has an inner space for accommodating the first battery 172, and the opening of the inner space facing the second side end frame 125 to be described later is in the first flexible cover 123 It can be covered detachably. The first side connection frame 121 and the first flexible cover 123 may be formed of rubber or a soft synthetic resin material in order to improve a fit with at least a surface portion thereof in contact with a user's ear and a rear portion thereof.

Similarly, one end of the second side frame 110b may be connected to the right side of the support frame 120 and the other end of the second side frame 110b may be connected to the second side connection frame 124. The second side connection frame 124 may be connected to a second side end frame 125. The second side end frame 125 has an inner space for accommodating the second battery 171, and the opening of the inner space facing the first side end frame 122 or the first flexible cover 123 described above is It may be detachably closed by the second flexible cover 126. The second side connection frame 124 and the second flexible cover 126 may be formed of rubber or a soft synthetic resin material in order to improve the fit with at least a portion of the surface portion in contact with the left ear and the rear of the user. The first battery 171 and the second battery 172 may be referred to as a power supply device, and depending on implementation, one battery may be mounted or a battery may not be mounted and all may be omitted.

A first side cover 112a may be installed on an outer surface of the first side frame 110a to accommodate a first printed circuit board 161. The first printed circuit board 161 may be connected to the first battery 172 and may be connected to one end of the FPCB 164. Similarly, a second side cover 112b may be coupled to an outer surface of the second side frame 110b to accommodate a second printed circuit board (a second PCB) 162. The second printed circuit board 162 may be connected to the second battery 171 and may be connected to the other end of the FPCB 164.

The front side of the first side frame 110a and the second side frame 110b may be provided with a protrusion or convex portion for securing a coupling force and designating a position when the shield (refer to 180 in FIG. 16) is coupled. In addition, instead of the uneven portion, a magnet or a metallic material to which the magnet is attached may be provided in the vicinity of the formed surface, thereby facilitating coupling with the shield and maintaining a stable coupling state.

The first printed circuit board 161 may include at least one port or a first connector for transmitting/receiving data to/from the outside or receiving power. The second printed circuit board 162 may include at least one port or a second connector for transmitting/receiving data to/from the outside or receiving power. In addition, the first or second printed circuit board 161; 162 may include an earphone terminal.

The front camera 151 may include a lens exposed to the front of the main frame 110 or the central frame 120c. Two front cameras 151 may be installed, but the present invention is not limited thereto. The lens of the front camera 151 or a peripheral portion thereof may be protected by the front auxiliary cover 115. The front camera 151 may be electrically connected to another end or extension of the FPCB 164.

The rear camera 153 may include a lens exposed to the rear of the main frame 110 or the support frame 120. One rear camera 153 may be installed, but is not limited thereto. The rear camera 153 may be electrically connected to another end or extension of the FPCB 164.

The processor (see 212 of FIG. 5) may be disposed on the first or second printed circuit boards 161 and 162. The processor may operate and manage an operation or function of the smart glasses 100 by controlling the components of the smart glasses 100. The processor may include an application processor (AP). The processor may include a display interface.

The display interface may control timing of signals transmitted to the first display 131 and the second display 132 under control of the processor. The signal may include an image signal. In this embodiment, the display interface is a dual and 3D MIPI DSI (Dual & 3D Mobile Industry Processor Interface Digital Serial Interface) suitable for application to the first and second displays 131 and 132 and 3D content (hereinafter referred to as DSI for short). ) (See FIG. 17 210A) or a means or device that performs a similar function.

In addition, a driver 194 may be incorporated in the first or second side frames 110a and 110b. The driver 194 may be electrically connected to the first or second printed circuit boards 161 and 162 and operate according to signals from the front camera 151, the rear camera 153, a sensor, or a processor to generate vibration. . As an example, when the user's eyelid is recognized through the rear camera 153 and it is recognized that the eyelid is covering the eyeball for a predetermined period of time or longer, a vibration for the anti-drowsiness alarm may be generated according to a signal from the rear camera 153.

In addition, a heat dissipation structure may be installed on the upper portion of the support frame 120 or at a lower portion or a portion of the upper cover 113. One end of the heat dissipation structure may be connected to the display interface 160, the first display 131, the second display 132, a backlight for an LCD, and the like to emit heat generated from at least one of the above components. For effective heat dissipation, the heat dissipation structure may extend along an outline in the longitudinal direction of the upper end of the central frame 120c, and may be installed in a state in which the cross-sectional area is expanded by providing a plurality of fins on at least one surface. . In addition, depending on the implementation, the heat dissipation structure may be connected to a material having excellent conductivity installed in a predetermined width along the length direction of the upper cover 113 or may be formed integrally with the highly conductive material.

13 and 14 are diagrams for explaining an optical system that can be employed in the smart glasses of FIG. 7.

In the present embodiment, the description is focused on the optical system including the first mirror 133, the first auxiliary lenses 135a, 135b, and 1372, and the first main lens 137, but the second mirror 134, the second auxiliary lens It goes without saying that the same can be applied to an optical system including the lens 136 and the second main lens 138.

13 and 14, the smart glasses according to the present embodiment include a first mirror 133, a first auxiliary lens (135a, 135b, 1372), and a first or right side including a first main lens (137). An optical system, and a second or left optical system (see 134, 136 and 138 in FIG. 9) including a second mirror, a second auxiliary lens, and a second main lens.

In this embodiment, the first main lens and the second main lens are formed of plate-shaped reflective and transmissive members, and are disposed to be inclined at a predetermined angle range with the central axis of each of the first and second main lenses. The first main lens and the second main lens may be a beam splitter in the form of a plate member having a thickness of several millimeters or less.

In addition, the smart glasses of the present embodiment may include at least one display device, preferably two display devices, and may include one mirror or two mirrors 133 and 134 depending on implementation.

The display device is arranged to output an initial image in a front direction or a face direction of a user wearing smart glasses, or is arranged to output an initial image in a vertical direction from the upper side of the face to the lower side or from the lower side to the upper side. It may be arranged to output an initial image toward both eyes between the eyes (glabellar) or above the nose. The distance between both eyes may approximately correspond to the distance between the centers of the first and second main lenses.

In the smart glasses according to the present embodiment, the first main lens 137 faces the first auxiliary lenses 135a, 135b, and 1372 and is inclined in a range of approximately 30 to 60 degrees with the center line of the optical path of the first auxiliary lens. Is placed. The first auxiliary lens 137 may include a first one-sided convex lens 135a, a first two-sided convex lens 135b, and a first one-sided convex lens 1372.

One surface of the first one-sided convex lens 135a has a convex surface, and the other surface has a concave surface. The curvature of the concave surface of the first convex lens 135a may be set to be substantially similar to or equal to the curvature of the convex surface facing the first convex lens 135b.

The first double-sided convex lens 135b has a convex surface on both its one side and the other side. The radius of curvature of the convex surface of one surface and the convex surface of the other surface may be the same, but is not limited thereto.

The first convex lens 1372 may have a convex surface facing the convex surface of the other surface of the first double-convex lens 135b, and a surface opposite to the convex surface may have a planar shape.

According to the above configuration, the first image and the second image output from the first and second displays of the smart glasses according to the present embodiment are reflected from the first mirror 133 and the second mirror, respectively, and then the first auxiliary After being enlarged by a combination of the convex surface of each of the lens and the second auxiliary lens, the reflection surface of the first main lens 137 and the reflection surface of the second main lens are partially reflected and partially transmitted. The reflective surface may be referred to as a beam split surface.

Here, the first image and the second image are enlarged by the combination of the convex surface of the first auxiliary lens and the convex surface of the second auxiliary lens, and the spherical surface by the aspherical convex surface of the first auxiliary lens and the second auxiliary lens. The aberration can be eliminated.

In addition, the first image and the second image to be focused at a predetermined position on the front surface of the first and second main lenses may form a predetermined single image plane or a virtual image plane. Of course, in the case of a stereoscopic (3D) image, a virtual reality image, an augmented reality image, and the like, a multilayer image surface or a virtual image surface may be provided.

In addition, in the smart glasses of the present embodiment, each of the main lenses has an empty space between the side frame (see 112b in FIG. 7 ). As described above, by not disposing any components outside the side surfaces of each main lens, there is an advantage in that the maximum wide viewing angle can be provided without limiting the viewing angle of the smart glasses wearer.

In addition, according to the present embodiment, the content output from the screen A1 of the display can be enlarged using the auxiliary lens. In such a case, the distance from the auxiliary lens to the magnified image can be adjusted as necessary. That is, the image enlarged by the auxiliary lens is a virtual image B1, not a real image, and its size or resolution can be easily adjusted so that the user's eye C1 can see it through the smart glasses.

The distance (li) to the virtual image felt by the user through the main lens may be adjusted by a relationship between the focal length (f) of the main lens and the distance (lo) from the display to the reflective surface of the main lens. The relationship between the distances satisfies Equation 1.

According to this embodiment, it can be designed and manufactured so that an enlarged image can be formed at a location desired by the user. In addition, it can be configured to implement content of about 20 inches in front of the smart glasses about 2.4 meters. That is, about 20 inches of content can be provided about 2 meters in front of the smart glasses, and users can easily implement and use augmented reality or virtual reality as needed.

15 is a block diagram of a control board that can be employed in the smart glasses of FIG. 7.

Referring to FIG. 5, the smart glasses 100 according to the present embodiment may include a controller 210 connected to the first and second displays D1 and D2 131 and 132. In addition, the smart glasses 100 may include a power supply unit 170 and a communication unit 190. The power supply unit 170 may be referred to as a power supply device or a power supply device, and may include a battery or a power supply terminal. In addition, the communication unit 190 may include a subsystem for wired or wireless communication. At least a portion of the above-described power supply unit 170 and the communication unit 190 may be mounted on a printed circuit board. In that case, such a printed circuit board may be referred to as a communication and power board.

The controller 210 may include a processor and a memory 220. The controller 210 may control the operation of the power supply unit 170 and the communication unit 190 by a program stored in the memory 220 and control the operation of the smart glasses 100. In this embodiment, the controller 210 and the power supply unit 190 are integrally formed in the smart glasses 100 in a storage form, but are not limited thereto.

In the above-described embodiment, the display interface is described as being a separate configuration from the controller 210, but the present invention is not limited to such a configuration, and the display interface may be implemented to be integrally formed with the controller.

In addition, the smart glasses 100 according to the present embodiment includes one small battery or auxiliary battery as an auxiliary power supply unit, and a separate main power supply device including a large, large capacity or main battery is disposed outside the smart glasses. Then, it may be configured to be connected through a data and power line (see 192 in FIG. 26).

16 is a perspective view of smart glasses and a control system according to another embodiment of the present invention. 17 is a block diagram illustrating a main configuration of the smart glasses of FIG. 16.

Referring to FIGS. 16 and 17, the smart glasses 100 according to the present embodiment may be implemented to have a display interface and to place a control device connected to the display interface outside the smart glasses 100. In addition, the smart glasses 100 are implemented to place substantially all batteries outside the smart glasses 100.

Here, the display interface may be a dual and 3D MIPI DSI (Dual & 3D Mobile Industry Processor Interface Digital Serial Interface, 210a) (hereinafter referred to as a DSI for short) or a means or device that performs a similar function. Using such a display interface, it is suitable for implementing 3D content with the first and second displays 131 and 132.

For the above-described configuration, the smart glasses 100 include a first display 131, a second display 132, a display interface 160, a camera 150, and a communication and power board 164.

The camera 150 may include a front or rear camera, and the communication and power board 164 is installed at an installation position of the first or second printed circuit board, and an external main power supply (MPS) 240) and the application processor (AP, 260). In this case, the external device 300 may externally control and monitor the operation of the smart glasses 100 as a control and power device.

In the present embodiment, the first battery 172 and the second battery 171 may be omitted along with most of the controller, the first printed circuit board 161 and the second printed circuit board 162. That is, a communication and power board 164 having a relatively simple structure is installed at the location of the second printed circuit board 162 to relay communication and power connection between the smart glasses 100 and the external device 300, and Most of the components for the part can be omitted.

According to the present embodiments, the weight of the smart glasses 100 can be significantly reduced, and there is an advantage of increasing the usability. In addition, it is possible to increase the battery capacity by a relatively certain amount according to the use of external batteries, and by receiving power through commercial power or other auxiliary batteries through the power terminal as necessary, the necessary power for long-term use of smart glasses can be smoothly supplied. There are advantages to supply.

18 is a block diagram illustrating a control module that can be employed in the smart glasses of FIG. 16.

Referring to FIG. 18, the control device for smart glasses according to the present embodiment may be referred to as an application processor, and includes a video interface 261, a tracking module 262, a rendering module 263, and a measurement module 264. Can include. The tracking module 262, the rendering module 263, and the measurement module 264 may be connected to a user interface (U/I, 265).

The video interface 261 may receive a video image stream input from the video cameras 151 or 152 and transmit an image of a desired section to the tracking module 262.

The tracking module 262 may receive image data from the video interface 261 and transmit a measurement image and a pose estimation result to the rendering module. In this case, the tracking module 262 may receive adjustment information including adjustment parameters, etc. through the user interface 265 and recognize the marker using marker information stored in the marker information database 266. The marker functions as a medium between the real image and the virtual object. The marker information may include information on the size and pattern of the marker. The marker information database 266 may be replaced with marker information stored in a predetermined storage unit.

The rendering module 263 is responsible for creating and removing virtual objects. The rendering module 263 may obtain adjustment information or the like through the user interface 265. The adjustment information may include load or unload, rotation, movement (coordinate movement, etc.), scaling, and the like. The rendering module 263 may interwork with an engine supporting 3D (3D) modeling (briefly, a 3D modeling engine 268) or a means for performing such a function and a content resource database 267. The 3D modeling engine 268 may generate a virtual object of a measurement image based on a virtual reality modeling language (VRML). The rendering module 263 may receive a virtual object from the content resource database 267 and perform rendering on the measured image.

The measurement module 264 may determine and process a distance between virtual objects, a distance and a direction between the generated coordinate systems, and whether there is interference between virtual objects. The measurement module 264 may receive an augmented image from the rendering module 263 and provide an augmented reality image to the display device according to object information input through the user interface 265. The object information may include information on positive or negative matching, 3D point, collision, and the like.

19 and 20 are flowcharts illustrating a main operating principle of the smart glasses of FIG. 16.

Referring to FIG. 19, the control device for smart glasses according to the present embodiment may acquire a stereoscopic image through a camera (S201). The control device may generate a depth map image based on the stereoscopic image.

Next, the control device may detect or extract the hand image (S203). The control device may extract a hand image from the depth map image.

Next, the control device may extract a command corresponding to a series of hand images from a storage unit or a database (S205). The control device provides a vector component indicated by a series of hand images created by the hand images extracted for a predetermined period of time, and an image area corresponding to the start and end points of the vector component, or a preset command corresponding to an object located in the image area. I can recognize it. Here, the object may have different types of menus, icons, etc. in response to a preset command.

According to the present embodiment, the control device of the smart glasses may detect hand motions through image processing and control an image viewed in the smart glasses based on this. For example, the image control may include forward or reverse an image, fast forward an image, rewind an image, and execute a preset command by recognizing a touch on a specific area of an image.

In addition, the control device recognizes a hand in an image frame input for hand motion recognition, detects an image in which the hand is located, or recognizes or predicts the movement direction of the hand, and places content on the screen or moves the current image accordingly. Alternatively, the current image can be arranged by overlapping with other images. Predicting hand motion may include recognizing a hand, a finger, or a combination thereof. Such gesture recognition may be performed based on image processing and preset gesture information.

In addition, according to the present embodiment, it is possible to provide a user interface using hand gestures in not only augmented reality, but also virtual reality, mixed reality, etc., thereby providing various contents of smart glasses. Can be used effectively, and user convenience can be greatly improved.

As an example, according to the present embodiment, when working at a desk without a monitor, wearing smart glasses when shopping, performing payment through augmented reality, using a navigation function through augmented reality, or playing an augmented reality game, , You can watch virtual reality videos, make video calls in augmented reality or virtual reality, work in the field with augmented reality while the worker wears smart glasses, or easily manipulate and use the functions of the smartphone on the screen of the smart glasses. have.

In addition, referring to FIG. 20, the control device of the smart glasses according to the present embodiment acquires a stereoscopic image through a camera (S201), detects or extracts a hand image based thereon (S203), and then In response to a series of hand images, screen mirroring of a mobile computing device such as a portable terminal or a notebook may be performed (S206).

Screen mirroring may be referred to as screen mirroring, and refers to a function of allowing a screen of a mobile computing device to be viewed on a screen of a smart glasses wirelessly without connecting a separate line.

For screen mirroring, the control device of the smart glasses or the display interface of the control board receives and decodes the screen information when the mobile computing device encodes the screen information and transmits the screen information through the display. Can be printed on. Wireless includes Bluetooth, but is not limited thereto, and other short-range wireless communication methods may be used.

If you use screen mirroring, you can receive calls, phone calls, video calls, etc. from a mobile terminal with a communication function in smart glasses, or receive text messages and check text contents to conveniently provide services such as phone calls and text messages from mobile terminals. You can make it easy to use.

21 is a view for explaining a user interface that can be employed in smart glasses according to another embodiment of the present invention. 22 and 23 are exemplary diagrams for explaining a modified example of a user interface that can be employed in smart glasses according to another embodiment of the present invention. 24 is a diagram for explaining the operating principle of a user interface that can be employed in the smart glasses of the present embodiment. 25 and 26 are flowcharts illustrating a signal processing process of a user interface employed in smart glasses according to another embodiment of the present invention.

21 to 23, the smart glasses according to the present embodiment may be implemented as a user interface in a form in which a virtual touch panel is displayed on the surface of a hand, a back of a hand, a forearm, a desk, and a visit to perform an interaction function. have. In this case, a sensor or a camera (see 150 of FIG. 17 and 151 and 152 of FIG. 18) for recognizing a corresponding body part or object may be mounted on the smart glasses.

More specifically, in order to recognize an object in a three-dimensional space, the camera has two infrared sensors (cameras) for three-dimensional analysis and stereo functions, and when the infrared projector projects infrared rays into the space, By reading infrared rays, it is possible to detect the depth of an object, more specifically, an object. Using an infrared projector, it is possible to easily determine the depth or distance of a space.

Infrared projectors can use two main types of devices. One is the so-called time of flight, which has a so-called structured light structure, which irradiates infrared rays into a specific space in the form of a certain pattern or pattern, or judges the sensitivity of infrared rays by having a different three-dimensional distance from an object. (time of flight) structure. Meanwhile, the present invention can implement a depth sensor for two cameras without using infrared projection. In particular, it is also possible to use only two camera sensors to read a gesture of a close hand. There is a method that can recognize depth with only one camera sensor, but the method is complicated to implement and use, so it will be omitted here.

When describing the depth perception using a sensor or a camera according to the present embodiment, first, a shape of a space and coordinates of an object can be obtained. Of course, this step can be omitted if implemented on the body surface.

Next, you can find the location of the space where the virtual object will be drawn. In this step, the depth sensor can be used to find an optimal location for the virtual object. In this case, the user interface control device applied to the smart glasses may be implemented to recognize a corresponding body such as a hand, a back of a hand, and a forearm as at least some components of the control device. In addition, it may be implemented to recognize a corresponding object in order to place an object on a refrigerator door or a surface of a visitor.

Next, a virtual object can be created to have properties of size, direction, curved surface, and perspective. In this step, when the location of the virtual object is determined, it is possible to draw the corresponding object by naturally reflecting the properties of the object along the image surface of the user interface. Objects can be drawn along the same plane or curve to have an appropriate size, direction, and perspective.

Next, the finger of another hand approaching to touch the object or the tip of a pointer corresponding thereto may be recognized. In this step, it may be set to recognize by specifying a specific finger according to the implementation (see FIG. 34). In this case, operation errors such as overlapping touches can be prevented.

Next, a virtual contact (touch) may be determined by recognizing whether a specific area corresponding to the virtual object and a designated fingertip are substantially at the same position. In this step, when a finger is overlapped and recognized on a specific object while a virtual object is drawn on a virtual image, the processor may perform a preset command according to a gesture of the movement.

To this end, a processor connected to a sensor or camera can detect a body part or object to recognize its surface, size, and floor, and measure the distance between feature points such as edges or points using functions such as motion tracking. I can. In addition, the processor may be implemented to generate a point cloud for texturing, mashing, or the like by referring to color image data or to search for a color of the point cloud.

In addition, sensors, cameras, processors, or a combination thereof can be designed to work best indoors and at distances of the order of 0.5 to 4M. This configuration can contribute to exhibiting good performance while maintaining a balance of power required for infrared illumination and depth processing. However, for objects that do not reflect infrared rays well, scanning performance may be deteriorated.

In addition, in the present embodiment, additional information may be reinforced so that the virtual object is displayed to the user as if it is naturally mixed with the real object. For example, it is possible to adjust the size according to the position of the virtual object, adjust the position or size of a three-dimensional surface, or adjust the color or brightness of the lighting. In addition, if the object on which the virtual object is displayed is in a moving state, it is possible to provide a more realistic feel to the user by adjusting the above conditions.

In addition, a processor connected to a sensor or a camera may be implemented to process stereopsis. The phantom fusion region of stereoscopic vision may include a user interface region for binocular stereopsis used by the human eye. The user interface area may be a single vision area. In this case, a stereoscopic vision processing system such as software running on the processor may determine the user's current focus area within the user's field of view. To this end, the processor may perform eye tracking processing based on data captured by an eye tracking camera for each eye. In the eye tracking process, a current focus area of the user is acquired, and a processing operation may be performed based on this. Convergence between the eyes can be used to triangulate with a focal curve and a focal point on a horopter, along with data representing the position of a user's face. The focus area adjustment module in the processor may adjust the variable focus lens disposed to focus at least one eye in the current user's focus area through the control circuit.

As an example, a method of drawing a virtual object in three dimensions at a specific location in real space is as follows.

First, a surface of a user interface image on which a virtual object is displayed may be set as a default virtual image position. In addition, a virtual object to be drawn in space may be drawn in front or behind the basic virtual image position. At this time, the location and size of the virtual object are different when viewed from the user's left eye and right eye according to the location. That is, the 2D image of the virtual object reflected in each eye is transmitted to the user's brain to recognize the 3D virtual object. In the reverse calculation here, if the image that the left eye will see and the image that the right eye will see are drawn as separate images, the user's brain can recognize them as three-dimensional.

Accordingly, in this embodiment, as shown in FIGS. 21A to 21C and FIGS. 22 and 23, the palm 510, the back of the hand 520, the forearm 530, the desk ( 540), refrigerator door (door), visitor (550), track image output targets, such as walls, determine the location and size of the actual space and objects, create a coordinate system based on this, and based on the virtual object (600, 610) ) Can be placed. The virtual object may contain a user interface image or vice versa. In addition, the stereoscopic vision processing system in the processor of the present embodiment, the virtual object control unit, or a combination thereof may perform a preset command or a series of operations by recognizing the overlap of instruction pointers approaching the virtual object.

Looking more at the process of detecting the palm using the depth sensor, as shown in FIG. 34, the palm detection system in the processor extracts the outline 562 of the hand from the image acquired by the sensor or the camera, or 550) and the location of the fingertip 566 may be extracted. That is, the outline 562 of the hand may be extracted or the joint 556 of the hand and the tip 566 of the finger may be extracted based on the point cloud obtained based on the acquired image.

In addition, it is preferable to use joint extraction when implementing a method in which a virtual object is also shown or covered when the palm is not opened, but when a fist is clenched or when the palm is turned over. Using joint extraction, a gesture for a hand motion can be determined in detail.

A virtual object 600 is displayed on the recognized palm, and the virtual object 600 may function as a user interface for acquiring an input signal from the user by a combination of the user's other hand or a pointer corresponding thereto. .

Of course, the recognized palm may be used as another hand and also as a pointer, and in that case, it may be used to perform a preset command according to various gesture detailed actions set in advance starting from palm recognition.

The stereoscopic vision processing system described above, the virtual object controller, a combination thereof, or a processor including them may perform a series of operations as illustrated in FIGS. 25 and 26.

First, the processor mounted on the smart glasses may recognize an object having structural characteristics of a hand or a wrist (S301). Objects may include palms, back of hands, forearms, objects, walls, doors, and the like.

Next, it may be determined whether a plurality of objects are found (S302). When a plurality of objects are found, the processor may select one of the plurality of objects according to a preset processing policy (S303). The selection criterion may be set to include a criterion such as a priority for an object, a center of an image, or occupying the largest area in an image.

Next, an area in which a user interface (UI) image is to be drawn may be searched based on the structural characteristics of the recognized or selected object (S304). The UI image may correspond to or include a virtual object, and vice versa.

Next, the processor may determine or adjust an appropriate size of an area to draw a UI image for the searched area (S305). The appropriate size, position, or arrangement direction of the UI image may be determined according to a rule or setting preset by the type of the recognized or selected object or the UI image arrangement area accordingly.

Next, a UI image of the adjusted size may be generated in the searched area. At this time, the UI image may be generated together with a 3D plane or a curved surface serving as the background (S306). The 3D plane or curved surface may function to allow the user to better see the virtual object or UI image by contrast with the virtual object.

Next, the processor mounted on the smart glasses may recognize an object having structural features of the other hand that is close to the UI image emitted on the object (S311). The other hand or object may be included in or referred to as a pointer.

Next, the processor may determine whether a plurality of pointers are found (S312). If there are a plurality of found pointers, the processor may select any one of the plurality of pointers according to a pre-stored processing policy (S313).

Next, the processor may determine whether the fingertip is close to a specific object in the UI within a specific range (S314). Proximity may include overlapping, and overlapping in a recognition value indicating a degree of proximity may have a negative value when the state in which two objects are in contact with each other is 0 based on the distance between the two objects. If the fingertip is not close within a certain range, the processor may maintain the step of recognizing another hand or object or return to the step.

If the fingertip approaches within a specific range, the processor may determine whether there are a plurality of objects corresponding to the tip of one fingertip or the corresponding pointer (S315).

As a result of the determination in step S315, if a plurality of objects exist, the processor may determine the nearest proximity according to a preset proximity priority policy (S316). The closest determination can be determined based on the distance or overlapping distance between the pointer tip and the object. In addition, the closest determination may be determined based on a preset priority among a plurality of objects, a weight, a difference between proximity distances, or a combination thereof.

Next, the processor may determine that the object corresponding to the tip of the finger or the tip of the pointer is touched (touch determination), return the identifier (ID) of the corresponding UI object determined to be touched, and then execute a preset touch reaction (S317). ).

In the above-described embodiment, a stereo RGB camera mounted on smart glasses may be used, or an infrared projector or a stereo infrared camera may be used. According to this embodiment, augmented reality using binocular stereoscopic vision may be implemented, and a preset operation may be executed by detecting a touch signal or an event input through a user interface according to a user's gesture. Therefore, it is possible to increase user convenience and provide a variety of augmented reality services and smart glasses services combining them.

27 is a perspective view of smart glasses according to another embodiment of the present invention. 28 is a front view of the smart glasses of FIG. 27. 29 and 30 are diagrams illustrating a state of use of the smart glasses of FIG. 28.

Smart glasses according to the present embodiment may have an appearance as shown in FIGS. 27 to 28. The smart glasses may provide images to both eyes of a user through both main lenses, and may be connected to a device for power and control by wire or wirelessly, as shown in FIG. 29. In addition, as shown in FIG. 30, the smart glasses may include a component to which a shield for blocking UV rays, a filter, and a color addition function may be combined.

31 is a front view of smart glasses according to another embodiment of the present invention. 32 and 33 are diagrams illustrating a state of use of the smart glasses of FIG. 31.

The smart glasses according to the present embodiment may have an appearance as shown in FIG. 31, and may be installed to provide an image to only one main lens, or may be configured to have the other main lens detachably provided. . In addition, as shown in Figs. 32 and 33, it goes without saying that the device for power and control may be connected by wire or wirelessly. In addition, as shown in FIG. 33, the smart glasses may include a component such as a coupling structure to which a shield may be additionally coupled.

34 and 35 are perspective views showing the appearance of a device for powering and controlling smart glasses according to an embodiment of the present invention.

34 and 35, the device for powering and controlling smart glasses may have a neck band type, may be referred to as a control board, a control system, etc., and may include a camera, a speaker, and It may be provided with earphones, etc., and may be provided with a button for remote control (remote control), a component such as a jog shuttle.

In the above, preferred embodiments of the present invention have been described, but those of ordinary skill in the technical field of the present invention can variously modify the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims. It will be appreciated that it can be modified and changed.

Claims (7)

A frame assembly having the shape of a spectacle frame and on which electrical components are mounted, and
A display device that is supported by the frame assembly, and when the frame assembly is worn on a user's face, a first display device is vertically provided at a center upper side of the face and a second display device is vertically provided at a center lower side of the face;
An optical system that is supported by the frame assembly and transmits an image of the display device in a direction in which both eyes of the user are positioned when the frame assembly is worn on a user's face; And
And a rear case that couples the display device and the optical system in a cross-arranged structure, and components of the optical system face each other at the center of the cross-arranged structure of the display device and the optical system. Includes; a case assembly in which a mirror transmitting in both horizontal directions is located
In the optical system, elements for transmitting the image transmitted from the display device in a direction in which both eyes of the user are located are arranged horizontally,
A first auxiliary lens for enlarging a first image of a first display device belonging to the display device to a desired size through an aspherical convex surface and removing spherical aberration and a first image transmitted from the first auxiliary lens are partially reflected and A first optical system having a first main lens that partially transmits,
A second auxiliary lens for enlarging a second image of a second display device belonging to the display device to a desired size through an aspherical convex surface and removing spherical aberration and a second image transmitted from the second auxiliary lens are partially reflected and Including a second optical system having a second main lens partially transmitting,
Smart glasses, characterized in that the image transmitted to the user's eyes is an image formed on a virtual image surface a predetermined distance from the back of the main lens. delete delete The method according to claim 1,
The first image is output from the first display device to a first mirror disposed on the user's eyebrow or nose and positioned above one eye of the user, and is output from the upper eye by the first mirror. It is reflected under the eye and incident on the first auxiliary lens, is enlarged from the first auxiliary lens and transmitted to the first main lens, is reflected and transmitted from the first main lens to be seen by the user's eyes,
The second image is output from the second display device to a second mirror disposed above the user's eyebrow or nose and positioned above the other eye of the user, and from the second mirror It is reflected under the eye and incident on the second auxiliary lens, is enlarged from the second auxiliary lens and transmitted to the second main lens, reflected and transmitted from the second main lens to be seen by the user's eyes. glasses. The method according to claim 1,
Each of the first main lens and the second main lens has an empty space between adjacent side frames. The method according to claim 1,
Further comprising a communication and power board accommodated in the center frame of the frame assembly or a left or right frame portion of the main frame and connected to at least one of the first display device and the second display device, the communication and power board Smart glasses that are connected to an external control and power device by wire and transmit signal and data communication between the control and power device and the first and second displays. The method of claim 6,
Further comprising a control device connected to the communication and power board,
The control device shifts and outputs the second image by a predetermined pixel in one direction based on the first image to correspond to binocular parallax.

Images