KR102578119B1 - Smart glasses operation method interworking to mobile device - Google Patents

Abstract

A method of operating smart glasses in conjunction with a mobile device is disclosed. The method of operating smart glasses includes sensing information obtained from IMU sensors mounted on smart glasses and information from one or more sensors mounted on the mobile device in an application installed on a mobile device for linking with smart glasses. In addition, it provides a method of collecting network service information, imaging real-time data based on the mode set in the mode setting stage, creating a mode image suitable for the situation, and displaying it on smart glasses through HMD projection. Users can efficiently visualize and view information according to each situation.

Description

How to operate smart glasses that link with mobile devices {SMART GLASSES OPERATION METHOD INTERWORKING TO MOBILE DEVICE}

The present invention relates to a method of operating smart glasses, and more specifically, to a method of operating smart glasses that generates and displays images according to the operation mode in real time while linking with a mobile device.

In general, a head mounted display (HMD) refers to a digital device that is worn on the head like glasses and displays virtual images in a position close to the eyeball. Recently, HMD has gone beyond simple display functions and is developing into a form of combination with augmented reality technology. For example, the HMD provides various conveniences to the user through the screen in front of the user, communicates with external digital devices, outputs the relevant content, and also receives user input through the external digital device or links with the external digital device. It also performs one task.

HMD refers to a digital device that is worn on the face like glasses and displays virtual images on the glasses in a form that displays multimedia content close to the eyeball.

These HMDs go beyond simple display functions and can provide various conveniences to users by combining with Augmented Reality (AR) technology, and can not only output the content by communicating with external digital devices, but also communicate with external digital devices. It also supports receiving user input through or performing tasks linked to the relevant external digital device. Recently, research and development on HMDs has been actively conducted.

However, when performing a task linked to an external digital device, conventional smart glasses determine the user's geographic location (a coordinate system such as latitude, longitude, altitude, or a coordinate system indicating an arbitrary physical location and direction) appropriate for the current situation. There were limitations in generating and displaying real-time images suitable for the user's situation, such as the direction and speed of movement, the direction the user is looking at, and the user's posture.

As a conventional technology for improving the above-mentioned problems, a vehicle HMD system and its operating method are known in Korean Patent Publication No. 10-2014-0145332 (December 23, 2014).

However, the prior art relates to a vehicle HMD system including a glasses-type projection HMD linked to AVN (audio video navigation) that provides vehicle operation information and navigation information, and an operating method thereof, which determines the location, direction, and speed of the user's movement. , Rather than generating and displaying real-time images suitable for the user's situation, such as the user's viewing direction and posture, it simply includes a glasses-type projection HMD that displays driving information and navigation information through a display window.

The present invention was derived to solve the problems of the prior art described above, and the purpose of the present invention is to provide a method of operating smart glasses in conjunction with a mobile device that generates and displays images according to the operating mode in real time. It is done.

In addition, another object of the present invention is to provide sensing information obtained from IMU (Inertial Measurement Unit) sensors mounted on smart glasses in an application installed on a mobile device and information with one or more sensors mounted on the mobile device. Rather, it provides a method of collecting network service information, imaging real-time data based on the mode set in the mode setting step, creating a mode image appropriate for the situation, and displaying it on smart glasses through HMD projection.

In a method of operating smart glasses in conjunction with a mobile device running a specific application for linking with smart glasses according to an aspect of the present invention for solving the above technical problem, the Inertial Measurement Unit (IMU) operating inside the smart glasses IMU data acquisition step of acquiring sensing data in real time through Unit); An IMU data wireless transmission step of transmitting the sensing data to a mobile device; Receiving the IMU data from the mobile device, collecting and processing sensing information obtained from the IMU data based on the mode preset in the mode setting step, information on one or more sensors mounted on the mobile device, and information on network services A mode image generation step of generating a mode image suitable for the situation; A mode image transmission step of compressing and encrypting the mode image in the mobile device and transmitting it to the smart glasses; A mode image reception step of receiving, decoding, and decompressing the mode image from the smart glasses; and a mode image projection step of displaying the mode image on the smart glasses using a head mounted display (HMD) projection method.

The IMU of the present invention may include one or more of an acceleration sensor, a gyro sensor, a geomagnetic sensor, and a barometric pressure sensor (or altitude sensor) for determining altitude, which are built into the smart glasses.

Additionally, in the mode setting step of the present invention, any one of hang gliding, cycling, running, skiing, navigation, golf, constellation finding, and weight training can be set in the application of the mobile device.

Additionally, the setting mode in the mode setting step may be additionally set according to the operation situation of the smart glasses in the application of the mobile device.

In addition, the mode setting step of the present invention is directly set by the UI of the application of the mobile device, or a mode setting signal is generated by a voice recognition means or a gesture recognition means in the smart glasses and transmitted to the mobile device to the application. It may be characterized as being configurable.

By the method of operating smart glasses in conjunction with a mobile device running a specific application for interworking with the above-mentioned smart glasses, a mode image according to each set mode is projected onto the smart glasses, so that the user of the smart glasses It has the effect of efficiently visualizing and viewing information according to each situation.

In addition, the method of operating smart glasses in conjunction with the mobile device of the present invention allows the user to check real-time images appropriate for the situation and use them as a tool to improve records, and to avoid health risks or predictable physical risks, such as hang gliding mode. If you continue like this, you will receive a warning about the possibility of falling and hitting the ground soon.

1 is a configuration diagram of smart glasses that work with a mobile device according to an embodiment of the present invention.
Figure 2 is a flowchart of a method of operating smart glasses in conjunction with a mobile device according to an embodiment of the present invention.
Figure 3 is an exemplary diagram showing an example of operation of smart glasses in hang gliding mode according to the first embodiment of the present invention.
Figure 4 is an exemplary diagram showing an example of operation of smart glasses in cycling mode according to the second embodiment of the present invention.
Figure 5 is an illustrative diagram showing an example of operation of smart glasses in running mode according to the third embodiment of the present invention.
Figure 6 is an exemplary diagram showing an example of operation of smart glasses in navigation mode according to the fourth embodiment of the present invention.
Figure 7 is an illustrative diagram showing an example of operation of smart glasses in a star-hunting mode according to the fifth embodiment of the present invention.
Figure 8 is an exemplary diagram showing an example of operation of smart glasses in golf mode according to the sixth embodiment of the present invention.

In describing the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Please note in advance that even if the terms are the same, if the displayed parts are different, the drawing symbols will not match.

The terms described below are terms set in consideration of the functions in the present invention, and since they may vary depending on the intention or custom of operators such as experimenters and measurers, their definitions should be made based on the content throughout the specification.

The terms used herein are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as indicating substantially the same meaning as the meaning they have in the context of the related technology, and unless clearly defined in the present application, they should be interpreted in an ideal or excessively formal sense. It doesn't work.

Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Accordingly, the present invention can be modified in various ways and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all transformations, equivalents, and substitutes included in the spirit and technical scope of the present invention. In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

FIG. 1 is a configuration diagram of smart glasses interoperating with a mobile device according to an embodiment of the present invention, and FIG. 2 is a flowchart of a method of operating smart glasses interoperating with a mobile device according to an embodiment of the present invention.

Referring to FIG. 1, the present invention relates to smart glasses that interact with a mobile device 200 running a specific application for interoperating with smart glasses 100, wherein the mobile device is a movable computer device, This may include, but is not limited to, mobile devices, portable terminal devices, mobile phones such as smartphones, smart watches, personal tablet devices, mobile computers, PDAs/EDAs, personal navigation devices (PNDs), and laptops. , can include all wearable devices.

Accordingly, the mobile device 200 of the present invention is capable of linking various network networks, including the Internet, and has GPS sensor, acceleration sensor, gyro sensor, gravity sensor, temperature/humidity sensor, altitude sensor, barometric pressure sensor, heart rate sensor, and geomagnetism. It refers to a mobile device that is equipped with one or more of a sensor or an image sensor and has a specific application installed that can collect and process sensing data from the sensors in conjunction with smart glasses. Accordingly, the application may include UI/UX that can be operated by the user.

In addition, the smart glasses 100 of the present invention refer to a glasses-type projection HMD that can display image data generated by the mobile device 200 in a form close to the eyeball with multimedia content.

Accordingly, the smart glasses 100 according to an embodiment of the present invention include a display interface, a control device connected to the display interface and including a communication unit, a processor unit, and memory, and an inertial measurement unit (IMU) connected to the control device. Unit) may be provided built-in.

The communication unit provides a means for the smart glasses 100 to interact with the mobile device 200, and generally provides a wireless method, but a wired method may also be provided. The wireless includes, but is not limited to, Bluetooth, and other short-range wireless communication methods may be used.

In addition, the method of operating smart glasses of the present invention referring to FIG. 2 includes an IMU data acquisition step (S100), an IMU data wireless transmission step (S110), a mode image generation step (S120), a mode image transmission step (S130), and a mode image transmission step (S130). It may include a reception step (S140) and a mode image projection step (S150).

The present invention relates to a method of operating smart glasses that link with a mobile device running a specific application for linking with smart glasses. The IMU data acquisition step (S100) involves collecting sensing data through an IMU operating inside the smart glasses. This is the step of acquiring in real time.

Here, the IMU may include one or more of an acceleration sensor, a gyro sensor, a rotational speed sensor, a geomagnetic sensor, an altitude sensor, and an atmospheric pressure sensor built into the smart glasses.

The IMU data wireless transmission step (S110) is a wireless IMU data transmission step that transmits the sensing data to a mobile device.

The mode image generation step (S120) is a step of receiving the IMU data from the mobile device and processing it according to the mode preset in the mode setting step (S200) to generate image data.

In the mode setting step (S200), the mobile sets up a situation suitable for the situation desired by the user wearing the smart glasses, such as hang gliding, cycling, running, skiing, navigation, golf, constellation finding, and weight training. This is the step to set up the device application.

At this time, the mode setting may be additionally set according to the operation situation of the smart glasses in the application of the mobile device. In other words, the user can additionally set the desired situation and change the display type of the mode image generated by the application.

The mode setting step (S200) is set directly by the UI of the application of the mobile device, or transmits a mode setting signal from the smart glasses to the mobile device using voice recognition means or gesture recognition means provided by the smart glasses. You can set it in the application.

To this end, the control device of the smart glasses may be provided with a means for detecting hand movements through image processing and controlling the images visible from the smart glasses based on this.

Additionally, the control device of the smart glasses may be provided with means for identifying the user's voice and controlling the image displayed by the smart glasses based on this.

For example, the control device recognizes a hand in an image frame input for hand motion recognition, detects the image where the hand is located, or recognizes or predicts the direction of hand movement, and places content on the screen or moves the current image accordingly. Alternatively, the current image can be placed overlapping with another image. Hand movement prediction may include recognizing the hand, fingers, or a combination thereof. Such gesture recognition can be performed based on image processing and preset gesture information.

In addition, the control device recognizes the user's voice input and can place content on the screen, move the current image, or arrange the current image to overlap with other images by commands given by the voice. Voice recognition can be performed based on voice processing and preset voice information.

Therefore, the mode image generation step (S120) collects and processes sensing information obtained from IMU sensors mounted on the smart glasses in the application, information on one or more sensors mounted on the mobile device, and information on a network service. This means generating an image of real-time data based on the mode preset in the mode setting step (S200).

The mode image transmission step (S130) is a step of compressing and encrypting the mode image in the mobile device and transmitting it to the smart glasses in a streaming manner. According to the streaming method, encoding, encryption, transmission control, and QoS (Quality of Service) for the mode image can be performed to ensure a certain level of performance in data transmission.

The mode image reception step (S140) is a step of receiving the mode image transmitted from the mobile device in the smart glasses, decoding and decompressing it.

The mode image projection step (S150) is a step of displaying the decompressed mode image in the smart glasses by HMD projection.

The operation method of smart glasses that link with a mobile device running a specific application of the present invention described above will be described with reference to various embodiments as follows.

Figure 3 is an exemplary diagram showing an example of operation of smart glasses in hang gliding mode according to the first embodiment of the present invention.

According to the first embodiment of the present invention with reference to FIG. 3, the hang gliding mode is set in the application of the mobile device according to the mode setting step (S200).

Smart glasses perform a step of acquiring IMU data measured from an acceleration sensor, a rotational speed sensor, a gyro sensor, a geomagnetic sensor, an altitude sensor, and a heavy cloud barometric pressure sensor (IMU data acquisition step; S100).

The smart glasses transmit the acquired IMU data to the mobile device through the wireless transmission step (S110).

And, sensing information obtained from the IMU sensors received by the application of the mobile device and information from one or more sensors mounted on the mobile device are collected, and hang gliding preset in the mode setting step (S200) is performed. Real-time data based on the mode is imaged and generated.

At this time, the three-dimensional direction and posture of the smart glasses can be measured through the acceleration sensor and gyro sensor of the IMU. Rather than providing independent information, the acceleration sensor and gyro sensor are used to generate calculated values that are linked and complementary to each other. Here, technically, pose can generally mean direction and position. In other words, the absolute direction of the smart glasses can be detected through a geomagnetic sensor, and the altitude at which the smart glasses are located can be measured by detecting air pressure through an altitude sensor or barometric pressure sensor.

Additionally, the geographical location can be specified by reflecting measurement data from the GPS sensor of the mobile device.

Based on this sensing data, the application can determine the direction, speed, and altitude of the user's location such as north, south, east, west, etc., and a 3D compass function can be provided, guiding the current location and target point, and detecting the risk of speed or altitude. A mode image that can provide advance notice is created.

Additionally, when the mobile device receives and reflects weather services such as wind direction/speed through the network, the creation of the mode image in the hang gliding mode is completed.

The mode image finally generated in the hang gliding mode is transmitted to smart glasses through the mode image transmission step (S130), and the smart glasses receive it through the mode image reception step (S140) and can decode and decompress it. .

Lastly, the hang gliding mode image can be displayed through HMD projection through the mode image projection step (S150) on the smart glasses.

This has the effect of helping users operate more safely and accurately in hang gliding situations.

Additionally, Figure 4 is an illustrative diagram showing an example of operation of smart glasses in cycling mode according to the second embodiment of the present invention.

According to the second embodiment of the present invention with reference to FIG. 4, the cycling mode is set in the application of the mobile device according to the mode setting step (S200), and then in the IMU data acquisition step (S100) of the present invention, the IMU data wireless The cycling mode image is projected to the HMD in the smart glasses through a transmission step (S110), a mode image generation step (S120), a mode image transmission step (S130), a mode image reception step (S140), and a mode image projection step (S150). It can be displayed in this way.

At this time, reflecting the sensing data of the IMU and the measurement data of the GPS sensor of the mobile device may be similar to the first embodiment, but in the cycling mode of the present invention, a heart mounted on the mobile device is additionally used. By reflecting the sensing data of the heart beat sensor, data identifying the user's heart rate can be additionally reflected in the mode image.

Therefore, by projecting the cycling mode image onto the smart glasses, the climbing angle with respect to the ground, such as a hill, can be determined according to the altitude measurement of the smart glasses, and the remaining distance and completion record can be predicted by determining the current location and speed. You can set a schedule according to your main strategy. Through this, users can use it as a tool to improve their cycling records, and have the effect of being warned of health risks based on their heart rate status.

Figure 5 is an illustrative diagram showing an example of operation of smart glasses in running mode according to the third embodiment of the present invention, in which the running mode is set in the application of the mobile device according to the third embodiment of the present invention, and the running mode is set in the application of the mobile device according to the third embodiment of the present invention. By creating a mode image according to the mode and projecting it on the smart glasses, the user can determine the current location and speed, estimate the remaining distance and completion record, and set a schedule according to his or her main strategy. There is an effect of limiting the appropriate speed depending on the strategy. Additionally, even in the running mode, the user's heart rate state can be reflected in the mode image by reflecting the sensing data of the heart rate sensor mounted on the mobile device and wearable device.

Figure 6 is an exemplary diagram showing an example of operation of smart glasses in navigation mode according to the fourth embodiment of the present invention.

As the navigation mode is set in the application of the mobile device according to the fourth embodiment of the present invention, a mode image according to the navigation mode is generated in the mode image generation step (S120) of the present invention and the mode image projection step (S150) ), the navigation mode image can be displayed in the smart glasses through HMD projection. Through this, the user can receive guidance on driving speed, current location, and direction through the smart glasses, and can determine the expected arrival time and remaining distance.

Figure 7 is an illustrative diagram showing an example of operation of smart glasses in a star-hunting mode according to the fifth embodiment of the present invention.

The user's geographic location information (latitude, longitude) is reflected as measurement data from the GPS sensor of the mobile device through the creation step of the mode image according to the Byulchari search mode setting according to the fifth embodiment shown in FIG. 7. By using the IMU of smart glasses to determine the direction (viewpoint) of the stars that the user is looking at, and additionally receiving and reflecting the current date and time and constellation information via the network through the mobile device, looking at the night sky You can create a constellation search mode image that allows you to easily find the desired constellation.

In addition, Figure 8 is an illustrative diagram showing an example of operation of smart glasses in golf mode according to the sixth embodiment of the present invention, including a step of generating a mode image according to golf mode settings, and a step of wirelessly transmitting the IMU data ( Through S110), the mode image generation step (S120), the mode image transmission step (S130), the mode image reception step (S140), and the mode image projection step (S150), the user of smart glasses can determine the hitting direction and angle, This has the effect of helping to update records by identifying wind direction/speed, etc.

At this time, in the mode image creation step (S120), the mobile device reflects not only weather services such as wind direction/wind speed through the network, but also map information services, so that the golf course topography can be displayed more realistically in the form of a 3D wireframe. You can create a mode image.

In addition, since the mode setting step (S200) of the present invention can be additionally set according to the operation situation of the smart glasses in the application of the mobile device, the mode image generated in the application is displayed by additionally setting the situation desired by the user. You can also change the type.

For example, if you additionally set it to weight training mode, you can determine the number of times and degree of completion of each exercise to enable effective training. Therefore, you can change the mode image to achieve the same effect as training with a professional personal trainer. It is obvious that it can be generated and displayed through HMD projection on the smart glasses.

Even though all the components constituting the embodiments of the present invention described above are described as being combined or operated in combination, the present invention is not necessarily limited to these embodiments. That is, as long as it is within the scope of the purpose of the present invention, all of the components may be operated by selectively combining one or more of them. In addition, although all of the components may be implemented as a single independent hardware, some or all of the components may be selectively combined to form a program or program that performs some or all of the combined functions in one or more pieces of hardware. It can be implemented by a computing device on which a software module is mounted. The program or software module includes modules that perform, control, or manage each step in FIG. 2 (IMU data acquisition module, IMU data wireless transmission module, mode image reception module, mode image projection module, mode setting module, mode image generation module, mode image transmission module) and may be stored in computer readable media such as semiconductor memory, CD disk, flash memory, etc. Recording media may include magnetic recording media, optical recording media, carrier wave media, etc.

Although the present invention has been described above with reference to preferred embodiments, those skilled in the art can make various modifications and changes to the present invention without departing from the spirit and scope of the present invention as set forth in the claims. You will understand.

100: Smart Glasses 200: Mobile Device

Claims (5)

In the method of operating smart glasses in conjunction with a mobile device running a specific application for linking with smart glasses,
An IMU data acquisition step of acquiring sensing data in real time through an IMU (Inertial Measurement Unit) operating inside the smart glasses;
An IMU data wireless transmission step of transmitting the sensing data to a mobile device;
Receiving the IMU data from the mobile device, collecting and processing sensing information obtained from the IMU data based on the mode preset in the mode setting step, information on one or more sensors mounted on the mobile device, and information on network services A mode image generation step of generating a mode image suitable for the situation;
A mode image transmission step of compressing and encrypting the mode image in the mobile device and transmitting it to the smart glasses;
A mode image reception step of receiving, decoding, and decompressing the mode image from the smart glasses; and
A mode image projection step of displaying the mode image on the smart glasses by HMD projection,
The IMU includes one or more of an acceleration sensor, a gyro sensor, a geomagnetic sensor, an altitude sensor, and an atmospheric pressure sensor built into the smart glasses,
In the mode setting step, one of hang gliding, cycling, running, skiing, navigation, golf, constellation finding, and weight training is set in the application of the mobile device,
The setting mode in the mode setting step is additionally set according to the operation situation of the smart glasses in the application of the mobile device,
The mode setting step is set directly by the UI of the application of the mobile device, or is set in the application by generating a mode setting signal using a voice recognition means or a gesture recognition means in the smart glasses and transmitting it to the mobile device,
The control device of the smart glasses is provided with means for identifying the user's voice and controlling the image displayed by the smart glasses based on this,
The control device recognizes the user's input voice and can place content on the screen, move the current image, or arrange the current image to overlap with another image by commands given by the voice, and recognize the voice. A method of operating smart glasses in conjunction with a mobile device, characterized in that it is performed based on voice processing and preset voice information. delete delete delete delete