KR102337908B1 - Method and device for assisting work - Google Patents

Abstract

The present disclosure relates to a method for assisting work executed on at least one processor of augmented reality smart glasses. The method for assisting work comprises: a step of receiving at least one front-facing image from at least one front-facing camera; a step of recognizing a visually coded pattern in the received at least one front-facing image, wherein the visually coded pattern is associated with specific equipment; and a step of displaying an image guiding a method of operating specific equipment associated with the recognized visually coded pattern in augmented reality on a transparent display. The user checks the external situation and the image together through the transparent display. In various embodiments of the present invention, the user can quickly identify his/her current location and escape route using a display, a mini map, and the like displayed on smart glasses and evacuate.

Description

Methods and devices for assisting work {METHOD AND DEVICE FOR ASSISTING WORK}

The present disclosure relates to a method and apparatus for assisting a task, and more particularly, to a method and apparatus for assisting a task using smart glasses.

Recently, with the development of technology, the demand for an augmented reality device or a mixed reality device that provides various types of information by superimposing a virtual image on an image or background of the real world is rapidly increasing. The demand for such augmented reality devices or mixed reality devices is continuously increasing for devices for special tasks and vehicles as well as entertainment areas. In particular, various types of wearable devices that can be worn on the body are emerging, and among them, an augmented reality glass (AR glass) device is a wearable device worn on a user's head, and provides visual information through a display to the user. Augmented reality services may be provided.

Currently, most augmented reality glasses devices are configured to support a user's entertainment such as games and video calls, or simple office tasks such as writing documents. That is, the current augmented reality glasses device is not suitable for supporting the work of a worker in a work space such as a smart factory.

The present disclosure provides a method for assisting work, a computer program stored in a recording medium, and an apparatus (system) for solving the above problems.

The present disclosure may be implemented in various ways including a method, an apparatus (system), or a computer program stored in a readable storage medium.

According to an embodiment of the present disclosure, a method for assisting a task executed in at least one processor of augmented reality smart glasses includes: receiving at least one front image from at least one front camera; Recognizing a visually coded pattern - the visually coded pattern is associated with a specific equipment - and displaying an image guiding a method of operating a specific equipment associated with the recognized visually coded pattern in augmented reality on a transparent display including the steps of The user checks the external situation and the image together through the transparent display.

According to an embodiment of the present disclosure, the method further includes extracting a plurality of manipulation areas of a specific device included in the at least one front image. An image for guiding a method of operating a specific device includes a guide related to an order of operating a plurality of manipulation areas.

According to an embodiment of the present disclosure, the steps of analyzing at least one front image and extracting motion information for the user to operate a specific equipment, comparing the extracted motion information with a reference motion stored in advance, and the operation information stored in advance In response to determining that there is a difference between the reference action and the first threshold or more, displaying the first warning message in augmented reality on the transparent display.

According to an embodiment of the present disclosure, the step of extracting the motion information for the user to operate a specific equipment includes the step of extracting the user's skeletal information from at least one front image to determine the user's hand motion information according to time include

According to an embodiment of the present disclosure, the steps of transmitting the user's hand motion information as the user's work details to an external device, receiving the result of inspecting the user's work details from the external device, and displaying the inspection results on a transparent display The method further includes displaying in augmented reality.

According to an embodiment of the present disclosure, extracting the user's skeletal information from at least one front image to determine the user's hand motion information over time, based on the time-dependent hand motion information, the possibility of injury of the user Calculating and when the possibility of injury of the user is greater than or equal to the second threshold, further comprising the step of displaying a second warning message in augmented reality on the transparent display.

According to an embodiment of the present disclosure, extracting the user's skeletal information from at least one front image to determine the user's hand motion information over time, and determining the user's workload based on the time-dependent hand motion information The method further includes the steps of: determining a reward to be provided to the user based on the determined amount of work, and displaying the determined reward in augmented reality on the transparent display.

According to an embodiment of the present disclosure, extracting the user's skeletal information from at least one front image to determine the user's hand motion information over time, and determining the user's proficiency based on the time-dependent hand motion information and displaying the customized tutorial to the user in augmented reality on the transparent display based on the determined proficiency level.

A computer program stored in a computer-readable recording medium is provided for executing the above-described method according to an embodiment of the present disclosure in a computer.

Augmented reality smart glasses according to an embodiment of the present disclosure are connected to at least one front camera configured to photograph an external situation, at least one transparent display configured to overlay and display an image, a communication module, a memory and a memory, and to the memory and at least one processor configured to execute at least one included computer readable program. The at least one program is configured to receive at least one front image from the at least one front camera, recognize a visually coded pattern in the received at least one front image, and configure the specific equipment associated with the recognized visually coded pattern. and instructions for displaying an image guiding a manipulation method in augmented reality on a transparent display. The user checks the external situation and the image together through the transparent display.

In various embodiments of the present disclosure, a user may quickly complete a necessary task and escape the workspace by using an escape route intuitively displayed through smart glasses, a route to a specific equipment, and the like.

In various embodiments of the present disclosure, a user simply recognizes an object using smart glasses worn without using or manipulating a separate device for recognizing an object and displays an external situation and an image related to the recognized object through a transparent display can be checked together.

In various embodiments of the present disclosure, the user may quickly identify his/her current location and escape route using a display displayed on smart glasses, a mini map, and the like, and evacuate.

In various embodiments of the present disclosure, an image associated with an object necessary for an emergency situation is adaptively displayed according to the user's gaze direction and region, so that the user can effectively recognize the image associated with the object.

In various embodiments of the present disclosure, even when the user does not know in advance how to operate the specific device, the user can effectively manipulate the specific device by using information displayed on the smart glasses.

In various embodiments of the present disclosure, the user can effectively check whether the user is operating a specific device normally.

In various embodiments of the present disclosure, smart glasses may determine the risk of injury, etc. in real time by using the user's skeletal information and provide it to the user who performs the task. Accordingly, the user can effectively maintain an accurate and low risk of injury posture while working.

In various embodiments of the present disclosure, the user may be provided with reasonable compensation as much as the amount of work performed by the user through the work amount and work compensation automatically determined using smart glasses without separately generating his/her work history.

In various embodiments of the present disclosure, even when a user lacks skill for a task, a task can be effectively performed using a customized tutorial intuitively displayed through smart glasses.

The effect of the present disclosure is not limited to the above-mentioned effects, and other effects not mentioned are clear to those of ordinary skill in the art (referred to as "person of ordinary skill") from the description of the claims. can be understood clearly.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals denote like elements, but are not limited thereto.
1 is a plan view illustrating an augmented reality smart glasses device according to an embodiment of the present disclosure.
2 is a block diagram illustrating a detailed configuration of an augmented reality smart glasses device according to an embodiment of the present disclosure.
3 is a flowchart illustrating a method of controlling a display according to a user's eye condition of an augmented reality smart glasses device according to an embodiment of the present disclosure.
4 is a flowchart illustrating a method of determining a user's dominant eye according to an embodiment of the present disclosure.
5 illustrates an example in which a user's dominant eye is determined in the augmented reality smart glasses device according to an embodiment of the present disclosure.
6 is a schematic diagram illustrating an example in which a display of an augmented reality smart glasses device is alternately reproduced according to an embodiment of the present disclosure.
7 is a diagram illustrating an example of an emergency situation occurring in a work space according to an embodiment of the present disclosure.
8 is a diagram illustrating an example in which a user wearing smart glasses recognizes an object according to an embodiment of the present disclosure.
9 is a diagram illustrating an example in which a route to an emergency exit is displayed on smart glasses according to an embodiment of the present disclosure.
10 is a diagram illustrating an example in which information related to a method of operating a specific device is displayed on smart glasses according to an embodiment of the present disclosure.
11 is a flowchart illustrating an emergency situation control method according to an embodiment of the present disclosure.
12 is a diagram illustrating an example in which a method of operating a specific device is displayed on smart glasses according to an embodiment of the present disclosure.
13 is a diagram illustrating an example in which a warning message is displayed based on a user's action according to an embodiment of the present disclosure.
14 is a diagram illustrating an example in which motion information of a user is extracted through smart glasses according to an embodiment of the present disclosure.
15 is a diagram illustrating an example in which a reward to be provided to a user is displayed through smart glasses according to an embodiment of the present disclosure.
16 is a diagram illustrating an example in which a customized tutorial is displayed through smart glasses according to an embodiment of the present disclosure.
17 is a flowchart illustrating a work assistance method according to an embodiment of the present disclosure.

Hereinafter, specific contents for carrying out the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the gist of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, the same or corresponding components are assigned the same reference numerals. In addition, in the description of the embodiments below, overlapping description of the same or corresponding components may be omitted. However, even if descriptions regarding components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the present disclosure to be complete, and the present disclosure provides those skilled in the art with the scope of the invention. It is provided for complete information only.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. Terms used in this specification have been selected as currently widely used general terms as possible while considering the functions in the present disclosure, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the content throughout the present disclosure, rather than the simple name of the term.

References in the singular herein include plural expressions unless the context clearly dictates the singular. Also, the plural expression includes the singular expression unless the context clearly dictates the plural. In the entire specification, when a part includes a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

In addition, the term 'module' or 'unit' used in the specification means a software or hardware component, and 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not meant to be limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium or configured to reproduce one or more processors. Thus, as an example, a 'module' or 'unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays or at least one of variables. Components and 'modules' or 'units' are the functions provided therein that are combined into a smaller number of components and 'modules' or 'units' or additional components and 'modules' or 'units' can be further separated.

According to an embodiment of the present disclosure, a 'module' or a 'unit' may be implemented with a processor and a memory. 'Processor' should be construed broadly to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some contexts, a 'processor' may refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. 'Processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configuration. You may. Also, 'memory' should be construed broadly to include any electronic component capable of storing electronic information. 'Memory' means random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM); may refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor is capable of reading information from and/or writing information to the memory. A memory integrated in the processor is in electronic communication with the processor.

Throughout this specification, the term 'on a network' or 'on a network' may refer to a state that is searchable or accessible through any electronic device capable of wired/wireless communication. For example, 'on a network' or 'on a network' may indicate a state in which any content and/or information related thereto can be searched for or accessed in any device connected to any electronic device by wire or wirelessly.

In the present disclosure, a 'visually coded pattern' may refer to a code in which data or information is displayed in a machine-readable visual form. In an embodiment, a pattern reader, a smartphone, an application program of a device with a built-in camera, such as smart glasses, may be used to scan/read a visually coded pattern. The visually coded pattern includes a visually coded one-dimensional pattern and a visually coded two-dimensional pattern. The visually coded one-dimensional pattern is made of parallel lines of different widths and spacings, and may include a Universal Product Code (UPC) or the like. The visually coded two-dimensional pattern consists of geometric patterns such as rectangles, dots, and hexagons, and may include Data Matrix, MaxiCode, Quick Response (QR) codes, and the like. In this specification, a 'visually coded pattern' may also be referred to as a 'pattern' or a 'marker'.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. The present application may be embodied in several different forms and is not limited to the embodiments described herein.

In the accompanying drawings, the same or corresponding components are assigned the same reference numerals. In addition, in the description of the embodiments below, overlapping description of the same or corresponding components may be omitted. However, even if descriptions regarding components are omitted, it is not intended that such components are not included in any embodiment.

1 is a plan view illustrating an augmented reality smart glasses device 100 according to an embodiment of the present disclosure. As shown, the augmented reality smart glasses device 100 includes at least one front camera (122, 124) configured to photograph an external situation, and at least one rear camera (132, 134) configured to photograph the user's left and right eyes. , a first transparent display 112 configured to overlay a first image, a second transparent display 114 configured to overlay a second image, and at least one front camera (122, 124) and at least one rear surface and a controller configured to receive and process at least one image from the cameras 132 , 134 and control operation of the first transparent display 112 and the second transparent display 114 .

The front cameras 122 and 124 are cameras configured to be exposed to the outside or the outside of the augmented reality smart glasses device 100 to photograph an external situation, and include the front camera 122 on the left and the front camera 124 on the right. . The rear cameras 132 and 134 are cameras configured to photograph both eyes of the user by being exposed to the inside or the inside of the augmented reality smart glasses device 100, and the rear camera 132 on the left for photographing the user's left eye 192 and and a right rear camera 134 for capturing the user's right eye 194 . The rear cameras 132 and 134 are configured to photograph visual information such as movement of the user's eyes, blinking, pupil size, whether or not tears are generated, and gaze. Additionally or alternatively, the rear camera 132 , 134 may be disposed next to the transparent display 112 , 114 . In an embodiment, a higher-precision security function may be provided by photographing the user's left eye 192 and/or right eye 194 using the rear cameras 132 and 134 to perform user authentication through iris recognition. .

The transparent displays 112 and 114 are transparent organic light-emitting diode (OLED) displays, which are installed on the front of the augmented reality smart glass device to reproduce text or images recognized by the user's naked eye. In one embodiment, the transparent displays 112 and 114 may be micro displays. The transparent displays 112 and 114 may display text or images when the user recognizes an external object through the transparent displays 112 and 114 to display information. The transparent displays 112 and 114 may lower transmittance so that the user can more clearly recognize text or images. For example, the first transparent display 112 may be configured to display the first image by overlaying it, and the second transparent display 114 may be configured to display the second image by overlaying it. 1 , the first transparent display 112 may be disposed in front of the user's left eye 192 , and the second transparent display 114 may be disposed in front of the user's right eye 194 .

In an embodiment, the transparent displays 112 and 114 may be implemented in such a way that an image is projected on a lens instead of a transparent OLED. In this case, a beam is emitted from the projectors installed on both sides of the frame of the augmented reality smart glasses device 100 toward the transparent display, and the user can check the image projected on the transparent display.

The control unit receives and processes at least one image from at least one front camera (122, 124) and at least one rear camera (132, 134), the first transparent display 112 and the second transparent display (114) configured to control the operation. Also, the controller may control the operation of the augmented reality smart glasses device 100 based on a communication signal exchanged with the central server 240 . For example, the controller may display text or an image on the transparent displays 112 and 114 . Also, when a command signal is received through the microphones 142 and 144 , the controller may display text or an image corresponding to the command signal on the transparent displays 112 and 114 . In one embodiment, the control unit sets the direction of the sound source to the main direction, and when the image is displayed, outputs an arrow or a compass image in the main direction.

The user of the augmented reality smart glasses device 100 according to an embodiment of the present disclosure checks the external situation and the first image with the left eye 192 through the first transparent display 112, and the user sees the second transparent display ( Through 114 , the external situation and the second image can be checked with the right eye 194 . In this case, the controller may detect the user's eye condition through at least one image received from the at least one rear camera 132 and 134 . In response to determining that the user's eye condition is abnormal, the control unit may selectively reproduce the first transparent display 112 or the second transparent display 114 .

The controller of the augmented reality smart glasses device 100 according to an embodiment of the present disclosure determines a dominant eye of the user based on at least one image received from at least one rear camera 132 and 134 . can In response to determining the dominant eye, the controller may reproduce a display corresponding to the user's dominant eye among the first transparent display 112 or the second transparent display 114 for a first time period. In this case, the controller may stop reproduction of the display corresponding to the user's non-dominant eye for the first time period.

The controller of the augmented reality smart glasses apparatus 100 according to an embodiment of the present disclosure may determine the fatigue degree of the user's main eye based on at least one image received from the at least one rear camera 132 and 134 . According to the fatigue degree of the primary eye, the control unit may dynamically adjust the first time period. Further, after the first time period (or the adjusted first time period) has elapsed, the control unit reproduces the display corresponding to the non-focusing eye of the user for a second time period, and reproduces the display corresponding to the user's gaze eye. A second period of time may be stopped. In this case, the controller determines the fatigue level of the non-focused eye of the user based on at least one image received from the at least one rear camera 132 and 134, and sets a second time period based on the fatigue level of the non-focused eye of the user. It can be dynamically adjusted.

The controller of the augmented reality smart glasses device 100 according to an embodiment of the present disclosure may track the user's eye movement based on at least one image received from the at least one rear camera 132 and 134 . According to the movement of the pupil, the controller may correct an image to be displayed on the first transparent display 112 and/or the second transparent display 114 or correct an output position. With this configuration, it is possible to provide a more realistic augmented reality experience.

In one embodiment, the control unit displays on the first transparent display 112 and/or the second transparent display 114 based on the object in the at least one image captured by the at least one front camera 122 , 124 . It is configured to decide which image to play. For example, when a QR code is recognized in the image taken by the front cameras 122 and 124 , the control unit displays an image corresponding to the QR code on the first transparent display 112 and/or the second transparent display 114 . ) can be displayed on the

The augmented reality smart glasses device 100 may include a first microphone 142 on the left side and a second microphone 144 on the right side. The microphones 142 and 144 are configured to recognize the user's voice and external sounds. In one embodiment, the microphones 142 and 144 may be mounted on the left and right of the front cameras 122 and 124 and the rear cameras 132 and 134 . In this case, the microphones 142 and 144 may be installed in different directions. For example, any one of the microphones 142 and 144 may be installed to face the front, and the other microphones to face the rear. The direction in which the sound is heard may be detected due to the microphones 142 and 144 installed in different directions. Signals received from the microphones 142 and 144 are provided to the control unit. The controller may determine an image to be displayed on at least one of the first transparent display 112 and the second transparent display 114 based on a sound or a voice (voice command) received from the microphones 142 and 144 .

The frame 180 is configured to have a shape of glasses. The frame 180 is provided with two legs and may be formed to be worn by a user. Two detection sensors 152 and 154 are installed on each leg of the frame 180 to detect the user's wearing of the augmented reality smart glasses device 100 . In addition, a lens or tempered glass 170 to protect the augmented reality smart glasses device 100 may be mounted on the front of the frame 180 if necessary. The transparent displays 112 , 114 may be disposed on or within the protective lens or tempered glass 170 . The augmented reality smart glasses device 100 may replace the role of safety glasses and safety equipment at the work site according to the environment or purpose in which it is used. Additionally, since touch sensors are installed on both sides of the frame, the augmented reality smart glasses device 100 may be controlled by a user's touch input. Through this configuration, the user may control the augmented reality smart glasses device 100 through various input methods including a voice command, a gesture command, an eye pupil command, a blink command, a touch input, and the like. For example, the user may perform various controls, such as starting/stop recording an image, through a touch input.

In addition, a connection port for connecting to an external device through a connection cable may be installed in the frame. For example, by connecting the augmented reality smart glasses device 100 to an external device such as a mobile phone, tablet, or PC using a connection cable, the external device performs some tasks (eg, high-load tasks) with the augmented reality smart glasses device (100) may be dealt with on behalf of or in conjunction with.

The IMU sensor 160 is positioned between the first transparent display 112 and the second transparent display 114 and may sense the direction of the augmented reality smart glasses device 100 by detecting a user's motion. The controller may control image transmission based on the sensed degree of change in the direction of the augmented reality smart glasses device 100 . The degree of change may be determined according to a change angle of the augmented reality smart glasses device 100 sensed through the IMU sensor 160 . For example, the degree of change may be determined by the amount of change in the head of a user wearing the augmented reality smart glasses device 100 . It may correspond to the degree of shaking, the speed of movement, or the degree of fast rotation in the up, down, left, and right directions.

2 is a block diagram illustrating a detailed configuration of the augmented reality smart glasses device 100 according to an embodiment of the present disclosure. As shown, the augmented reality smart glasses device 100 includes a camera module 212 , a display module 214 , a communication module 216 , a voice processing module 218 , an image processing module 220 , and an IMU sensor module ( 222 ), a detection sensor module 224 , a storage module 226 , and a control unit 210 . The augmented reality smart glasses device 100 may communicate with the central server 240 through the network 230 .

The camera module 212 may include at least one front camera and at least one rear camera. The external image captured by the camera module 212 and the image of both eyes of the user may be transmitted to the controller 210 .

The display module 214 may include at least one transparent display. The controller 210 may control transparency, brightness control, reproduction, etc. of the display in consideration of ambient illuminance and user fatigue.

The communication module 216 provides a communication function so that the augmented reality smart glasses device 100 can communicate with the central server 240 through the network 230 . For example, the augmented reality smart glasses device 100 may transmit data to the central server 240 through the communication module 216 . Similarly, the augmented reality smart glasses device 100 may receive data from the central server 240 via the communication module 216 . The communication module 216 may communicate with the outside using a communication method such as Bluetooth, Wi-Fi, or Zigbee.

The voice processing module 218 may include at least one microphone. The voice processing module 218 may record an external sound and the user's voice and transmit it to the controller 210 , or process the recorded data and transmit a processed result to the controller 210 . The controller 210 may determine an image to be displayed on at least one of the first transparent display 112 and the second transparent display 114 based on data received from the microphones 142 and 144 .

The image processing module 220 may be configured to output a pre-processed image by performing image enhancement processing on the photographed image in order to improve the recognition rate of an object included in the image photographed by the front camera and the rear camera. For example, image enhancement processing of a photographed image includes Denoise operation processing, Deblur operation processing, High Dynamic Range operation processing, Color Tone Mapping operation processing, Defog operation processing, Brightness operation processing, Contrast operation processing, Auto White Balance operation processing, It may include Back Light Compensation operation processing, Decompression operation processing, and the like. Additionally or alternatively, the image processing module 220 may be configured to perform image enhancement processing of the captured image by adjusting parameters of the captured image. Although the audio processing module 218 and the image processing module 220 are shown as separate components from the control unit 210 in FIG. 2 , the present invention is not limited thereto, and the control unit 210 includes the audio processing module 218 and the image processing module ( 220) may be performed.

The IMU sensor module 222 may sense a direction of the augmented reality smart glasses device 100 by detecting a user's motion. Information sensed by the IMU sensor module 222 may be transmitted to the controller 210 . The detection sensor module 224 may detect whether the user wears the augmented reality smart glasses device 100 . Information sensed by the detection sensor module 224 may be transmitted to the control unit 210 .

The storage module 226 may store a program for controlling the operation of the augmented reality smart glasses device 100 . The storage module 226 may include at least one instruction for controlling the operation of the augmented reality smart glasses device 100 . Programs stored in the storage module 226 may be classified into a plurality of modules according to their functions. The storage module 226 includes a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, SD, micro SD, XD memory). etc.), RAM (Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic It may include a memory, a magnetic disk, an optical disk, or the like. In an embodiment, the augmented reality smart glasses device 100 may store an image captured by the camera module 212 in the storage module 226 . In this case, the storage module 226 may be a non-volatile storage device. For example, the augmented reality smart glasses device 100 may store an image captured in real time in the storage module 226 or record an event image.

The augmented reality smart glasses device 100 may include more components than those of FIG. 2 . In an embodiment, the augmented reality smart glasses device 100 may include a depth sensor. The depth sensor may be implemented as a stereo camera, a LiDAR sensor, a ToF sensor, or the like. In an embodiment, the augmented reality smart glasses device 100 measures a distance to an object (eg, an object recognized in an image captured by the front camera) using a depth sensor, and based on the measured distance Thus, it is possible to determine a position to display an image on the display module 214 . Additionally, the augmented reality smart glasses device 100 may perform eye tracking (ie, tracking movements of the left and right eyes) using the rear camera and correct the image based on the result of the motion tracking. For example, an image may be corrected (eg, an image tilting angle, etc.) so that a sense of depth can be best felt according to the position of the user's pupil. With this configuration, it is possible to provide an augmented reality experience in which a sense of depth is reflected, thereby increasing a sense of reality. Additionally, the augmented reality smart glasses device 100 may include a battery. The battery may supply power to the control unit 210 and various modules 212 to 226 .

The network 230 may remotely connect the central server 240 and the augmented reality smart glasses device 100 . The network 230 is a wireless network such as WLAN (Wireless LAN), Bluetooth and ZigBee and/or Ethernet, a wired home network, a power line communication network, a telephone line communication network, and an RS- according to an installation environment. A wired network such as serial communication may be variously selected and configured.

3 is a flowchart illustrating a method 300 for controlling a display according to a user's eye condition of an augmented reality smart glasses device according to an embodiment of the present disclosure. The method 300 may be performed by at least one processor (eg, at least one processor in the controller 210 ). The method 300 may be initiated by the processor reproducing the first transparent display and the second transparent display (S310).

The processor may detect the user's eye condition while the first transparent display and the second transparent display are reproduced ( S320 ). For example, the processor may receive at least one image from at least one rear camera to detect the user's eye condition. Thereafter, the processor may determine whether the user's eye condition is normal ( S330 ).

In response to determining that the user's eye condition is normal, the processor may continue to reproduce the first transparent display and the second transparent display ( S310 ). On the other hand, in response to determining that the user's eye condition is abnormal, the processor reproduces a display (eg, a first transparent display) corresponding to the user's gazing eye, and a display (eg, a non-focused display) corresponding to the user's non-focused eye , the second transparent display) may not be reproduced. With this configuration, when the user's eyes are tired, the user's discomfort can be reduced by reproducing the augmented reality image only for one eye.

4 is a flowchart illustrating a method 400 of determining a user's dominant eye according to an embodiment of the present disclosure. The method 400 may be performed by at least one processor (eg, at least one processor in the controller 210 ). The method 400 may be initiated by the processor recognizing a dominant object in at least one image received from at least one front camera ( S410 ).

Thereafter, the processor may track the movement of the user's left eye and right eye based on at least one image received from at least one rear camera ( S420 ). As a result of tracking the movements of the left and right eyes, the processor may determine the eye that first looks at the recognized external dominant object as the dominant eye ( S430 ). For example, when a new object appears outside and the user first gazes at the newly appeared object with the right eye, the processor may determine the user's right eye as the dominant eye.

5 illustrates an example in which a user's dominant eye is determined in the augmented reality smart glasses device according to an embodiment of the present disclosure. As shown, the first rear camera 132 and the second rear camera 134 of the augmented reality smart glasses device may photograph the left eye 192 and the right eye 194 of the user, respectively. In order to determine the user's dominant eye, the first object 510 and the second object 520 may be displayed on the first transparent display 112 and the second transparent display 114 of the augmented reality smart glasses device, respectively. have.

When the user's left eye 192 is the dominant eye, when looking at the first object 510 , the perceived distance R of the dominant eye 192 may be shorter than the perceived distance R′ of the non-dominant eye. Accordingly, when looking at the first object 510 , the focus of the pupil of the dominant eye 192 may be relatively faster than the focus of the pupil of the non-primary eye 194 . Accordingly, in the augmented reality smart glasses device, when the first object 510 is displayed on the first transparent display 112 , the movement of the pupil of the left eye 192 and the movement of the pupil of the right eye 194 , and the second transparent display When the second object 520 is displayed on 114 , the pupil movement of the left eye 192 and the pupil movement of the right eye 194 may be checked to determine the user's dominant eye.

Additionally or alternatively, the augmented reality smart glasses device displays the first object 510 and the second object 520 on the first transparent display 112 and the second transparent display 114, respectively, and provides more information to the user. It can guide you to select objects that look sharper or darker. In this case, when the user makes a gesture 530 to select the direction of the first object 510 by hand, the augmented reality smart glasses device may determine the left eye 192 as the user's main eye. Alternatively, the user may be provided with an interface for selecting the dominant eye, and the user may directly select the left eye or the right eye as the principal eye.

6 is a schematic diagram illustrating an example 600 in which the display of the augmented reality smart glasses device is alternately reproduced according to an embodiment of the present disclosure. The augmented reality smart glasses device according to an embodiment of the present disclosure may determine a dominant eye of a user based on at least one image received from a rear camera. When the user's dominant eye is determined, the augmented reality smart glasses device _{reproduces the first display corresponding to the user's dominant eye for a first time period T 1} , and displays the user's non-dominant eye Playback of the corresponding second display may be stopped _{for the first time period T 1 .} After the first time period T ₁ has elapsed, the augmented reality smart glasses device reproduces a second display corresponding to the user's non-gazing eye for a second time period T ₂ , and a second display corresponding to the user's gaze eye 1 Reproduction of the display may be stopped for a second time period T _{2 .} In general, the display reproduction time (T ₁ ) corresponding to the gaze familiar to the user is preferably set to be longer, but is not limited thereto, and the first time period (T ₁ ) and the second time period (T ₂ ) are the same can be set.

In an embodiment, the augmented reality smart glasses device 100 may dynamically adjust _{the first time period T 1} and the second time period T _{2 according to the user's eye fatigue level.} For example, during playback of the first display, when the degree of fatigue in the watch user is determined to be greater than or equal to the first threshold value, the eye for a period of time (T ₁₎ that adjusts the first period of time (T ₁ ') reducing the The first display corresponding to the inside may be reproduced. On the other hand, during playback of the second display, and user non-involved in when fatigue is determined to be less than or equal to the second threshold, the second time period ratio for adjusting the second time period (T ₂ ') is increased to (T ₂₎ - A second display corresponding to the main eye can be reproduced.

7 is a diagram illustrating an example of an emergency situation occurring in the work space 700 according to an embodiment of the present disclosure. As shown, the work space 700 is a space in which arbitrary work by one or more users is made, and may be any indoor space including a plurality of emergency exits 730, but is not limited thereto, and the outdoor space is not limited thereto. may include Also, the user may be a worker wearing the smart glasses described above.

In one embodiment, an emergency situation may occur within the workspace 700 . Here, the emergency situation is an arbitrary situation in which escape or evacuation of the user is required, and may include, for example, a fire in the work space 700 , leakage of hazardous chemicals, a decrease in oxygen saturation, and the like. When an emergency situation occurs, the captured image of the smart glasses worn by users in the work space 700 may be automatically stored in a non-volatile memory and transmitted to the central control system in real time. That is, the central control system may receive and store the on-site image in real time from the smart glasses of the user (or all users) in the vicinity of the emergency situation occurrence area. The captured or stored images in this way can be used for the purpose of analyzing a cause-in for an emergency situation and submitting insurance proof later. When an emergency situation occurs, the user may receive a guide message related to the emergency situation through the smart glasses. For example, the guide message may be transmitted from a central control system that manages the workspace 700 .

In one embodiment, the received guidance message may be displayed in augmented reality on the transparent display of the smart glasses. That is, the received guidance message may be displayed in an overlapping manner with an actual scene where the user gazes through the smart glasses. For example, the smart glasses worn by the user receive at least one front image from at least one front camera, recognize an object (eg, an identification code such as a QR code, etc.) in the received at least one front image, and , an image associated with the recognized object may be displayed in augmented reality on the transparent display. In this case, the user can check the external situation and the image related to the recognized object together through the transparent display.

When receiving the guidance message, the user may transmit a response message indicating that the user has recognized the emergency situation. In this case, the user may transmit the response message using eye movements, voice commands, gestures (eg, hand gestures, facial expressions, pupil gestures, etc.) indicating a response. For example, when a guide message is displayed and the user blinks for a predetermined time, a response message may be transmitted. Here, the user's operation (eye movement, voice, gesture, etc.) for transmitting the response message may be predetermined in advance, or a necessary operation may be included in the received guidance message. With this configuration, the user can quickly recognize the emergency situation through the guide message, and the central control system can quickly check whether the emergency situation is recognized by receiving a response message from each user.

In an embodiment, when an emergency situation occurs, the central control system may determine the current location of each user by being connected to each user's smart glasses by communication, etc. Then, the central control system may remotely instruct each user's task based on the current location of each user and the point of occurrence of an emergency situation. That is, in an emergency situation, the central control system may support control of all users in the work space 700 .

In an embodiment, when an emergency situation occurs, a route to the optimal emergency exits 730_1 and 730_2 for each user may be determined based on the current location of each user. For example, the routes to the emergency exits 730_1 and 730_2 may be determined based on a current location of each user, a relative location between users, a location where an emergency occurs, and equipment requiring an emergency operation nearby. In this case, the routes to the emergency exits 730_1 and 730_2 may be determined by an arbitrary algorithm or a machine learning model. In the illustrated example, the optimal exit at the current location of the first user 710 may be determined as the second emergency exit 730_2 , and the optimal emergency exit at the current location of the second user 720 is the first emergency exit It may be determined as (730_1).

In an embodiment, the first user 710 may move according to the determined path to the second emergency exit 730_2 . For example, a path to the second emergency exit 730_2 may be displayed in augmented reality on the smart glasses of the first user 710 . That is, the first user 710 may move while checking the route to the second emergency exit 730_2 in real time.

While the user is moving to the emergency exit, the smart glasses may determine the user's blind spot. In addition, the smart glasses may display a warning message in augmented reality on the transparent display when there is a dangerous object in the blind spot. In the illustrated example, when the work vehicle 740 approaches from the blind spot while the first user 710 is moving to the second emergency exit 730_2, the smart glasses display a warning message in augmented reality on the transparent display. can do.

In an embodiment, the smart glasses may determine a blind spot using an IMU sensor, a camera, or the like. For example, the smart glasses receive IMU data from the IMU sensor, and based on the recognized object (a code such as a QR code; each code may be associated with an indoor location) and the IMU data, the user's current location and direction of movement can be estimated. Here, the IMU sensor is a device that measures the speed, direction, gravity, acceleration, etc. of a pedestrian or a moving object, and the IMU data may be any data related to the movement situation of a pedestrian or a moving object measured or estimated by the IMU sensor. have. That is, the smart glasses can estimate the moving speed, the moving direction, etc. according to the user's movement using the IMU sensor. In an embodiment, the smart glasses may estimate the approximate current location and movement direction of the user by using the IMU data. In addition, the smart glasses recognize a marker, etc. photographed by the camera, and then use the recognized information to correct the user's current position and/or movement direction to more precisely correct the user's current position and/or movement direction. can be estimated. Additionally, the smart glasses may determine the user's serial position and/or movement direction using only the marker captured by the camera.

Together with the user's current location and movement direction, the smart glasses may receive at least one rear image of the user's pupils from at least one rear camera, and detect the user's gaze direction based on the at least one rear image. have. Here, the rear camera may be configured to photograph the user's pupil, and the smart glasses may estimate the user's gaze direction and viewing angle based on the direction the user's pupil is facing. Then, the smart glasses may determine a blind spot outside the user's viewing angle based on the user's current location, moving direction, and user's gaze direction. In this case, an arbitrary algorithm, a machine learning model, etc. for determining a blind spot outside the user's viewing angle using the user's current location, movement direction, and user's gaze direction may be used.

In one embodiment, when an emergency situation occurs, it may be necessary to control the equipment according to the emergency situation (eg, emergency operation such as cutting off power in the work space 700 ). In this case, a path to a specific device and/or specific device requiring control may be provided to a specific user. For example, the specific user may be determined as the user with the shortest distance from the current location to the specific device among users. In the illustrated example, the second user 720 having the closest distance from the current location to the specific device 750 may be provided with a specific device and/or a route to the specific device. In this case, the specific device and/or the path to the specific device may be displayed in augmented reality on the transparent display of the smart glasses worn by the second user 720 . That is, the second user 720 may escape through the first emergency exit 730_1 after manipulating the specific equipment 750 . With such a configuration, the user can quickly complete a necessary task and escape the work space 700 by using an escape route intuitively displayed through smart glasses, a route to a specific equipment, and the like.

8 is a diagram illustrating an example in which a user 810 wearing smart glasses 820 recognizes an object 830 according to an embodiment of the present disclosure. In an embodiment, the smart glasses 820 may receive at least one front image from at least one front camera. In this case, the smart glasses 820 may recognize the object 830 in the received at least one front image, and display an image associated with the recognized object 830 in augmented reality on the transparent display. Here, the object 830 may be attached to a specific location in the work space or attached to specific equipment.

In an embodiment, the smart glasses 820 may estimate the current location of the user 810 based on the recognized object 830 . For example, the smart glasses 820 may estimate the current location of the user 810 based on an ID and location information of the recognized object 830 . Here, the object 830 used for estimating the position of the user 810 is an indicator used to determine the position or posture of the object, and a pre-registered fiducial marker including a specific pattern, a visual marker (visual marker), signage (signage), may include a QR code, and the like. That is, the smart glasses 820 may estimate the current location of the user 810 based on the object 830 using an arbitrary algorithm or machine learning model. In this case, the current location of the user may be estimated based on the size, location, shape, etc. of the object 830 included in the front image.

In this way, when the current location of the user 810 is estimated, the smart glasses 820 may calculate a route from the current location of the user 810 to the emergency exit. In this case, the route to the emergency exit may be displayed in augmented reality on the transparent display, and the user 810 may check the external situation and the route to the emergency exit through the transparent display.

In an embodiment, the smart glasses 820 may provide a manipulation method of a specific device as an image on the transparent display based on the recognized object 830 . In other words, the object 830 may include a visually coded pattern associated with the specific equipment, and the image associated with the recognized object 830 may include information related to the manipulation method of the specific equipment. For example, the object 830 may be displayed on the surface of a specific device, etc., and when the user 810 gazes at the object 830 through the smart glasses 820 , the manipulation method of the specific device is displayed on the transparent display. may be displayed in augmented reality. In this case, the user can check the external situation and the operation method of the specific equipment together through the transparent display.

Although the smart glasses 820 are illustrated to recognize one object 830 in FIG. 8 , the present invention is not limited thereto, and the smart glasses 820 may recognize a plurality of objects 830 . With this configuration, the user 810 simply recognizes the object 830 by using the smart glasses 820 being worn without using or manipulating a separate device for recognizing the object 830 and displays a transparent display. Through this, an external situation and an image/image associated with the recognized object 830 may be checked together.

9 is a diagram illustrating an example in which a path to an emergency exit is displayed on the smart glasses 910 according to an embodiment of the present disclosure. As described above, the smart glasses 910 may display the route to the emergency exit in augmented reality on the transparent display. That is, the user wearing the smart glasses 910 can check the external situation and the route to the emergency exit through the transparent display. In this case, the route to the emergency exit may include a direction 920 in which the user should move, a distance 930 in which the user needs to move, and a mini map 940 in which the movement path is displayed, which are indicated by arrows.

In an embodiment, the route to the emergency exit may include a direction 920 in which the user should move indicated by an arrow, a distance 930 in which the user should move, and a minimap 940 in which the movement path is displayed. For example, the direction 920 to move may include an arrow image indicating a straight forward, a right turn, a left turn, a U-turn, and the like. In addition, the mini-map 940 may be displayed in augmented reality on the transparent display, and the mini-map includes the current location of the worker in the workspace, the location of equipment, etc., and a visualized route from the current location to the emergency exit. It may be a simplified map. With such a configuration, the user can quickly find out his or her current location and escape route by using a display displayed on the smart glasses 910, a mini map, and the like, and evacuate.

In an embodiment, the smart glasses 910 may receive at least one rear image obtained by photographing the user's pupil from at least one rear camera, and detect the gaze direction of the user based on the at least one rear image. When the gaze direction is detected, a position at which an image associated with the recognized object is displayed on the transparent display may be determined based on the gaze direction of the user. For example, the smart glasses 910 may display a direction 920 in which the user should move, a distance 930 in which the user should move, etc. in the direction in which the user gazes.

In an embodiment, the indicator, the minimap, etc. displayed in the direction in which the user gazes may be displayed in augmented reality by adjusting the size, angle, etc., based on the area in the direction in which the user gazes. For example, when the user is gazing at a wall in the work space, an indicator, a mini map, etc. may be displayed as if they were actually attached to the wall. That is, the smart glasses 910 extract the size, angle, display area, etc. for correcting the image related to the object based on the area in the direction in which the user gazes, and use the extracted size, angle, display area, etc. to obtain the object. It is possible to correct the image associated with

9 shows that the path to the emergency exit is displayed on the right lens of the smart glasses 910, but is not limited thereto. The route to the emergency exit may be displayed on the left lens of the smart glasses 910 , displayed on both lenses simultaneously, or alternately displayed on both lenses according to an arbitrary criterion. In addition, when moving to the left, the indicator may be displayed on the left lens, and when moving to the right, the indicator may be displayed on the right lens. Alternatively, the indicator may be displayed on one lens and the minimap may be displayed on the opposite lens. With this configuration, an image associated with an object necessary for an emergency situation is adaptively displayed according to the user's gaze direction and region, so that the user can effectively recognize the image associated with the object.

10 is a diagram illustrating an example in which information 1020 related to a method of operating a specific device is displayed on the smart glasses 1010 according to an embodiment of the present disclosure. As described above, the smart glasses 1010 may display information 1020 related to a method of operating a specific device in augmented reality on the transparent display. That is, the user wearing the smart glasses 1010 may check the information 1020 related to the external situation and the operation method of the specific equipment together through the transparent display.

In an embodiment, the smart glasses 1010 may recognize a visually coded pattern (eg, a QR code, etc.) associated with a specific device, and provide information 1020 related to an operation method of the specific device. Here, the visually coded pattern may be an indicia associated with a particular equipment, for example, may be attached or displayed on a surface of the particular equipment and/or any other area associated with the particular equipment. In addition, the information 1020 related to the operation method of the specific equipment may include text, images, images, etc. indicating the operation order of the specific equipment, the operation area, and the like.

In an embodiment, the smart glasses 1010 may extract a plurality of manipulation areas associated with a specific device from at least one front image. In this case, the information related to the manipulation method of the specific equipment may include information on the order of manipulating the plurality of manipulation areas. In the illustrated example, the smart glasses 1010 may extract operation areas corresponding to the A button 1030 , the B button 1040 , and the C button 1050 in a specific device. Then, the smart glasses 1010 display the A button 1030 as shaded 1032, etc. according to the operation sequence of the specific device, and the "emergency situation" associated with the operation of the A button 1030. Press the A button A message 1034 such as "You are welcome." may be displayed. That is, the display position of the information on the manipulation area and/or the manipulation sequence may be determined differently depending on a direction in which the user gazes at a specific device.

In FIG. 10 , information 1020 related to a method of operating a specific device is illustrated as being displayed on the right lens of the smart glasses 1010 , but is not limited thereto. Information 1020 related to a method of operating a specific device may be displayed on the left lens of the smart glasses 1010 , displayed on both lenses simultaneously, or alternately displayed on both lenses according to an arbitrary criterion. With this configuration, even when the user does not know in advance how to operate the specific device, the user can effectively manipulate the specific device by using the information displayed on the smart glasses 1010 .

11 is a flowchart illustrating an emergency situation control method 1100 according to an embodiment of the present disclosure. The emergency situation control method 1100 may be performed by smart glasses (eg, at least one processor of augmented reality smart glasses). In the emergency situation control method 1100, the processor may receive at least one front image from at least one front camera (S1110).

The processor may recognize an object in the received at least one front image ( S1120 ). Also, the processor may display an image associated with the recognized object in augmented reality on the transparent display (S1130). In this case, the user can check the external situation and the image related to the recognized object together through the transparent display.

The processor may receive a message related to an emergency situation from an external device, and estimate a current location of the user based on the recognized object. In addition, the processor may calculate a route from the user's current location to the emergency exit. In this case, the image associated with the recognized object may include an indicator indicating a direction in which the user should move according to the calculated path. In addition, the image associated with the recognized object may further include a mini-map indicating a path from the user's current location to the emergency exit according to the calculated path. Additionally or alternatively, the processor may receive IMU data from an IMU sensor. Here, the user's current location may be estimated based on the recognized object and IMU data.

The processor may receive at least one rear image obtained by photographing the user's pupil from at least one rear camera, and detect the gaze direction of the user based on the at least one rear image. In this case, a position at which an image associated with the recognized object is displayed on the transparent display may be determined based on the user's gaze direction.

The processor may estimate the user's current location and movement direction based on the recognized object and IMU data. In addition, the processor may receive at least one rear image obtained by photographing the user's pupil from at least one rear camera, and detect the gaze direction of the user based on the at least one rear image. The processor may determine a blind spot outside the user's viewing angle based on the user's current location, moving direction, and user's gaze direction. In this case, the image associated with the recognized object may include a warning message associated with the dangerous object within the blind spot.

In one embodiment, the object may include a visually coded pattern associated with the specific equipment, and the image associated with the recognized object may include information related to the operation method of the specific equipment. The processor may extract a plurality of manipulation areas associated with specific equipment from the at least one front image. In this case, the information related to the manipulation method of the specific equipment may include information on the order of manipulating the plurality of manipulation areas.

12 is a diagram illustrating an example in which a method of operating a specific device is displayed on the smart glasses 1210 according to an embodiment of the present disclosure. In an embodiment, the smart glasses 1210 may display an image 1220 guiding a method of operating a specific device in augmented reality on the transparent display. In this case, the user may check the external situation and the image 1220 together through the transparent display.

In an embodiment, the smart glasses 1210 may receive at least one front image from at least one front camera. In addition, the smart glasses 1210 may recognize a visually coded pattern in the received at least one front image. Here, the visually coded pattern may be associated with specific equipment. In this case, based on the recognized visually coded pattern, an image guiding a method of operating specific equipment, etc. may be provided. For example, a method of operating a specific device may include an operating sequence of the specific device, an operating area, and the like.

In an embodiment, the smart glasses 1210 may extract a plurality of manipulation areas of a specific device included in at least one front image. In this case, the image for guiding a method of operating specific equipment may include guidance related to an order of operating a plurality of manipulation areas. In the illustrated example, the smart glasses 1210 may recognize a visually coded pattern and extract operation areas corresponding to the A button 1230 , the B button 1240 , and the C button 1250 in specific equipment. . Then, the smart glasses 1210 display the A button 1230, the B button 1240, and the C button 1250 in shaded colors, etc. according to the operation sequence of the specific device, and display "Start operation. Press the A button A message 1232 such as "Press." may be displayed. Here, the display position of the manipulation area, the message, etc. displayed on the smart glasses 1210 by shading or the like may be determined differently depending on the direction in which the user gazes at a specific device.

13 is a diagram illustrating an example in which a warning message 1342 is displayed based on a user's action according to an embodiment of the present disclosure. As described above, the smart glasses 1310 may display an image 1320 guiding a method of operating a specific device in augmented reality on the transparent display. In this case, the user can check the external situation and the image 1320 together through the transparent display.

As described above in FIG. 12 , the smart glasses 1310 recognize the visually coded pattern, and extract the operation areas corresponding to the A button 1330 , the B button 1340 , and the C button 1350 in specific equipment. can do. In addition, the A button 1330 , the B button 1340 , the C button 1350 , etc. may be shaded according to the operation sequence, and a message regarding the operation method may be displayed.

In an embodiment, the user may perform a necessary operation according to a method of operating a specific device displayed through the smart glasses 1310 . For example, based on a message displayed on the smart glasses 1310 (eg, 1232 in FIG. 12 ), the user first selects the A button 1330 by a touch input or the like, and controls a specific device or can make it work In this case, the smart glasses 1310 may analyze at least one front image to extract operation information for the user to operate a specific device.

In an embodiment, the smart glasses 1310 may analyze at least one front image to extract motion information for the user to operate a specific device, and compare the extracted motion information with a pre-stored reference motion. In addition, the smart glasses 1310 may display a warning message in augmented reality on the transparent display in response to determining that there is a difference between the reference operation in which the operation information is stored in advance and a threshold or more. Here, the pre-stored reference operation may be generated by combining operation information according to the user's past work or by combining operation information of workers who normally operate specific equipment. In addition, the threshold may be arbitrarily determined for each specific device or may be pre-stored together with a reference operation.

In an embodiment, the smart glasses 1310 may determine a difference between the user's motion information and a pre-stored reference motion using an arbitrary algorithm, a machine learning model, or the like. For example, the smart glasses 1310 may determine a difference between a hand gesture and the like by comparing a pre-stored image associated with a reference gesture with an image associated with the user's gesture information. In this case, in order to generate the user's motion information, the user's skeleton information (eg, hand skeleton information, etc.) may be extracted from the image. In another example, the smart glasses 1310 may determine a difference by comparing the time at which the manipulation of the specific device is completed based on the reference motion stored in advance with the time at which the manipulation of the specific device is completed based on the user's motion information. have.

In the illustrated example, the user may select or attempt to select the B button 1340 by a touch input or the like without first selecting the A button 1330 differently from the operation sequence of the specific equipment. In this case, the smart glasses 1310 extract the motion information 1360 of the user who selects or wants to select the B button 1340 by a touch input, etc., and the difference between the reference motion in which the motion information 1360 is stored in advance and the threshold is greater than or equal to a threshold. In response to determining that there is, an alert message 1342 may be displayed in augmented reality on the transparent display. For example, a warning message 1342 such as “※Caution※ A button was not pressed.” may be displayed in augmented reality on the transparent display. In this case, the operation method of specific equipment together with the warning message 1342 A separate voice, image, video, etc. indicating that this is incorrect may be displayed.

In FIG. 13 , when a user abnormally manipulates a specific device, a warning message is displayed, but the present invention is not limited thereto. For example, a warning message may be displayed when the user's motion differs from a predetermined reference motion by a working method, working speed, hand motion, etc. by more than a threshold value. With this configuration, the user can effectively check whether he or she is operating a specific equipment normally/efficiently.

14 is a diagram illustrating an example in which motion information of a user is extracted through smart glasses 1410 according to an embodiment of the present disclosure. As described above, the smart glasses 1410 analyze at least one front image 1420 to extract motion information for a user to operate a specific device, compare the extracted motion information with a pre-stored reference motion, and motion information may display a warning message in augmented reality on the transparent display in response to determining that there is a difference greater than or equal to the threshold value from the pre-stored reference operation.

In an embodiment, the smart glasses 1410 may determine the user's hand motion information according to time by extracting the user's skeleton information 1440 from the at least one front image 1420 . For example, the smart glasses 1410 may detect an area 1430 corresponding to the user's hand from at least one front image 1420 . In this case, an arbitrary algorithm, a machine learning model, etc. for detecting the region 1430 corresponding to the user's hand may be used.

In an embodiment, extracting the user's skeleton information 1440 from at least one front image may be performed according to the following process. (i) converting at least one foreground image to a gray scale image, (ii) performing histogram segmentation on the gray scale image to generate a binary image, (iii) morphological enhancement to the binary image ( Perform morphological enhancement processing and hole filling, (iv) recognize a hand region, (v) generate a chamfer distance image, and from the gradient distance image to an axis of the median value of the hand region ( median axis) can be extracted (local maxima), and the resulting set can be concatenated to generate a hand topology. That is, the user's skeleton information 1440 may be performed using a chamfer distance transformation (one of the distance transformation-based methods).

Then, the smart glasses 1410 may extract the user's skeleton information 1440 based on the detected area 1430 corresponding to the user's hand. Here, the skeleton information may be information indicating the size, length, direction, etc. of the user's hand using straight lines and curves. For example, the smart glasses 1410 may extract the user's skeleton information 1440 based on the area 1430 corresponding to the user's hand using an arbitrary algorithm, machine learning model, or the like. In this case, one machine learning model for detecting the region 1430 and extracting the skeletal information 1440 may exist.

In an embodiment, the smart glasses 1410 may transmit the user's hand motion information as the user's work details to an external device, and receive a result of examining the user's work details from the external device. As such, by extracting the user's skeletal information 1440, determining the user's hand motion information according to time, and transmitting it to an external device, the data transmission amount is reduced compared to transmitting images captured by the front camera to an external device as it is. can be greatly reduced.

Then, the smart glasses 1410 may display the inspection result in augmented reality on the transparent display. In the example shown, when the user's wrist is bent at an angle greater than or equal to the threshold compared to the reference motion stored in advance, a message 1450 such as "※Caution※ Wrist is too bent" may be displayed as a result of the inspection. Additionally or Alternatively, the smart glasses 1410 may directly inspect the user's work history, and display the inspection result in augmented reality on the transparent display.

In an embodiment, the smart glasses 1410 may extract the user's skeleton information from the at least one front image 1420 to determine the user's hand motion information according to time. Also, the smart glasses 1410 may calculate the possibility of injury of the user based on the time-dependent hand motion information. Then, the smart glasses 1410 may display a warning message in augmented reality on the transparent display when the possibility of injury of the user is greater than or equal to the threshold. For example, the likelihood of injury may be calculated to be higher as the angle at which the user's wrist or finger is bent is greater, and may be calculated to be higher as the duration at which the wrist or finger is bent is longer. In addition, the threshold may be determined for each user or may be predetermined for each specific device.

In the illustrated example, when the probability of injury of the user calculated based on the user's skeleton information 1440 is greater than or equal to a predetermined threshold, a warning message 1460 may be displayed in augmented reality on the transparent display. For example, if the user's wrist is broken for more than a predetermined standard, the possibility of injury may be determined to be higher than the threshold, and a warning message 1460 such as "※Caution※ There is a possibility of wrist injury" is displayed can be

14 illustrates that two warning messages 1450 and 1460 are simultaneously displayed on the transparent display, but the present invention is not limited thereto, and each message 1450 and 1460 may be displayed separately. With such a configuration, the smart glasses 1410 may determine the risk of injury in real time using the user's skeletal information and provide it to the user performing the task. Accordingly, the user can effectively maintain an accurate and low risk of injury during work, thereby effectively preventing industrial accidents.

15 is a diagram illustrating an example in which a reward 1530 to be provided to a user is displayed through the smart glasses 1510 according to an embodiment of the present disclosure. As described above, the smart glasses 1510 may determine the user's hand motion information according to time by extracting the user's skeleton information from at least one front image. Also, the smart glasses 1510 may determine a reward 1530 to be provided to the user based on the extracted information on the user's hand motion according to time.

In an embodiment, the smart glasses 1510 may determine the user's workload 1520 based on time-dependent hand motion information. Here, the amount of work 1520 may indicate the total amount of work performed by the user for a predetermined time (eg, one day, one week, one month, etc.). The smart glasses 1510 may capture a user's hand motion related to a task in real time or based on a predetermined period, and determine the amount of work 1520 performed by the user using the captured image. For example, the smart glasses 1510 may determine the amount of work 1520 performed by the user based on a captured image using an arbitrary algorithm, a machine learning model, or the like. In another example, the smart glasses 1510 may transmit a captured image to an external system or the like, and receive the determined amount of work 1520 .

The smart glasses 1510 may determine a reward 1530 to be provided to the user based on the determined amount of work 1520 . Also, the smart glasses 1510 may display the determined reward 1530 in augmented reality on the transparent display. Here, the reward 1530 represents a salary/bonus to be provided to the user, and may be determined according to a predetermined criterion based on the degree of completion of the task according to the task. For example, when 100% of the first task is completed, an amount of 100,000 won may be provided. Accordingly, when the user 89% of the first task is completed, an amount of 89,000 won is a reward for the first task can be determined as As described above, all of the rewards calculated based on the degree of completion of each task may be summed and determined as the user's task reward 1530 .

In FIG. 15 , the amount of work 1520 is shown to be displayed along with the degree of work completion for each task, but the present invention is not limited thereto, and the amount of work may be configured as a list of completed tasks. Also, although FIG. 15 illustrates that the summed rewards 1530 for all tasks are displayed, the present invention is not limited thereto, and rewards determined for each task may be displayed. With such a configuration, the user does not need to separately create his/her work history, and the amount of work 1520 automatically determined using the smart glasses 1510 and the work compensation 1530 are reasonable enough for the user to perform the work. compensation may be provided.

16 is a diagram illustrating an example in which a customized tutorial is displayed through smart glasses 1610 according to an embodiment of the present disclosure. As described above, the smart glasses 1610 may determine the user's hand motion information according to time by extracting the user's skeleton information from the at least one front image 1620 .

In an embodiment, the smart glasses 1610 may determine the user's skill level based on time-dependent hand motion information. Here, the user's proficiency indicates the degree to which the user is familiar with the operation of specific equipment, and may be displayed as a numerical value or in an arbitrary grade. For example, the smart glasses 1610 may determine the user's skill level based on the user's operation speed, operation accuracy, and the like. In this case, the smart glasses 1610 may determine the user's skill level based on the user's motion speed, motion accuracy, etc. using an arbitrary algorithm, machine learning model, or the like.

The smart glasses 1610 may display a customized tutorial to the user in augmented reality on the transparent display based on the determined skill level. Here, the customized tutorial may indicate guide information for assisting the user in performing the task. In addition, the guide information may include a guide for the user's work pattern, and may be stored at all times according to a designated task. For example, the customized tutorial may include a guide area 1630 and a guide message 1632 for inducing a user's hand motion. Here, the guide area 1630 may be posture correction data output through real-time AI learning or the like. In an embodiment, when the user performs a specific operation according to the guide area 1630 , a new guide area and a guide message for assisting the next operation may be displayed. Additionally or alternatively, the guide area 1630 may change dynamically over time.

In an embodiment, the guide area and the guide message may be continuously displayed based on the user's skill level, or may disappear after being displayed for a predetermined time. Additionally or alternatively, all of the plurality of guide areas and the guide message associated with the operation may be displayed, or only some of the plurality of guide areas and the guide message may be displayed based on the user's skill level.

16 shows that the customized tutorial is displayed on the right lens of the smart glasses 1610, but is not limited thereto. The customized tutorial may be displayed on the left lens of the smart glasses 1610 , displayed on both lenses simultaneously, or alternately displayed on both lenses according to an arbitrary criterion. In addition, although it has been described above in FIG. 16 that the smart glasses 1610 provide a customized tutorial by determining the user's skill level, the present invention is not limited thereto. For example, the smart glasses 1610 may transmit at least one front image 1620 to the central control system. In this case, the central control system (or the manager of the central control system) checks the user's hand gestures included in the at least one front image 1620 , and displays text, voice, and image for posture correction to the smart glasses 1610 . ) can be transmitted. With such a configuration, even when the user lacks skill in the operation, the operation can be effectively performed using the customized tutorial intuitively displayed through the smart glasses 1610 .

17 is a flowchart illustrating a work assistance method 1700 according to an embodiment of the present disclosure. The task assistance method 1700 may be performed by smart glasses (eg, at least one processor of augmented reality smart glasses). The task assistance method 1700 may be started when the processor receives at least one front image from at least one front camera (S1710).

The processor may recognize a visually coded pattern in the received at least one front image (S1720). Here, the visually coded pattern may be associated with specific equipment. In addition, the processor may display an image guiding a method of operating a specific device associated with the recognized visually coded pattern in augmented reality on the transparent display (S1730). In this case, the user can check the external situation and the image together through the transparent display.

The processor may extract a plurality of manipulation areas of a specific device included in the at least one front image. Here, the image for guiding a method of operating a specific device may include guidance related to an order of operating a plurality of manipulation areas.

The processor may analyze at least one front image to extract operation information for the user to operate a specific device. For example, the processor may determine the user's hand motion information according to time by extracting the user's skeleton information from the at least one front image. Also, the processor may compare the extracted motion information with a reference motion stored in advance. In response to determining that there is a difference between the reference operation in which the operation information is stored in advance and the first threshold or more, the processor may display the first warning message in augmented reality on the transparent display.

The processor may transmit the user's hand motion information as the user's work details to the external device, and receive a result of checking the user's work details from the external device. In addition, the processor may display the inspection result in augmented reality on the transparent display. Additionally or alternatively, the processor may directly inspect the user's work history.

The processor may extract the user's skeletal information from the at least one front image to determine the user's hand motion information over time, and calculate the possibility of injury of the user based on the time-dependent hand motion information. In addition, when the possibility of injury of the user is greater than or equal to the second threshold, the processor may display a second warning message in augmented reality on the transparent display.

The processor may extract the user's skeleton information from the at least one front image to determine the user's hand motion information over time, and determine the user's workload based on the time-dependent hand motion information. Also, the processor may determine a reward to be provided to the user based on the determined amount of work. In this case, the processor may display the determined reward in augmented reality on the transparent display.

The processor may extract the user's skeletal information from the at least one front image to determine the user's hand motion information over time, and determine the user's skill level based on the time-dependent hand motion information. In addition, the processor may display the customized tutorial to the user in augmented reality on the transparent display based on the determined proficiency level.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution by a computer. The medium may continuously store a computer executable program, or may be a temporary storage for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributedly on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store for distributing applications, sites supplying or distributing other various software, and servers.

The method, operation, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those of ordinary skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logic blocks, modules, and circuits described in connection with this disclosure are suitable for use in general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or the present disclosure. It may be implemented or performed in any combination of those designed to perform the functions described in A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In firmware and/or software implementations, the techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on computer-readable media such as programmable read-only memory), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, and the like. It may be implemented as stored instructions. The instructions may be executable by one or more processors, and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the embodiments described above have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in connection with any computing environment, such as a network or distributed computing environment. . Still further, aspects of the subject matter in this disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across the plurality of devices. Such devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in connection with some embodiments herein, various modifications and changes may be made without departing from the scope of the present disclosure that can be understood by those skilled in the art to which the present disclosure pertains. Further, such modifications and variations are intended to fall within the scope of the claims appended hereto.

100: augmented reality smart glasses device
112, 114: transparent display
122, 124: front camera
132, 134: rear camera
142, 144: microphone
152, 154: detection sensor
160: IMU sensor
170: tempered glass
180: frame

Claims (10)

delete A method of assisting a task running on at least one processor of an Augmented Reality smart glass, comprising:
receiving at least one front image from at least one front camera;
recognizing a visually coded pattern in the received at least one foreground image, wherein the visually coded pattern is associated with a particular piece of equipment;
displaying an image guiding a method of operating specific equipment associated with the recognized visually coded pattern in augmented reality on a transparent display; and
extracting a plurality of operation areas of the specific equipment included in the at least one front image
including,
The user checks the external situation and the image together through the transparent display,
The image for guiding the operation method of the specific equipment includes a guide related to a sequence of operating the plurality of operation areas, the operation assistance method.
delete A method of assisting a task running on at least one processor of augmented reality smart glasses, the method comprising:
receiving at least one front image from at least one front camera;
recognizing a visually coded pattern in the received at least one foreground image, wherein the visually coded pattern is associated with a particular piece of equipment;
displaying an image guiding a method of operating specific equipment associated with the recognized visually coded pattern in augmented reality on a transparent display;
extracting operation information for a user to operate the specific equipment by analyzing the at least one front image;
comparing the extracted motion information with a reference motion stored in advance; and
Displaying a warning message in augmented reality on the transparent display in response to determining that the operation information is different from the previously stored reference operation by more than a predetermined threshold
including,
The user confirms the external situation and the image together through the transparent display,
The step of extracting the operation information for the user to operate the specific equipment,
and determining the user's hand motion information according to time by extracting the user's skeleton information from the at least one front image.
5. The method of claim 4,
transmitting the user's hand motion information as the user's work details to an external device;
receiving a result of examining the user's work history from the external device; and
Displaying the inspection result in augmented reality on the transparent display
Further comprising, a work assistant method.
A method of assisting a task running on at least one processor of augmented reality smart glasses, the method comprising:
receiving at least one front image from at least one front camera;
recognizing a visually coded pattern in the received at least one foreground image, wherein the visually coded pattern is associated with a particular piece of equipment;
displaying an image guiding a method of operating specific equipment associated with the recognized visually coded pattern in augmented reality on a transparent display;
determining the user's hand motion information according to time by extracting the user's skeletal information from the at least one front image;
Calculating the possibility of injury of the user based on the time-dependent hand motion information; and
When the possibility of injury of the user is greater than or equal to a predetermined threshold, displaying a warning message in augmented reality on the transparent display
including,
The user confirms the external situation and the image together through the transparent display, work assistance method.
delete A method of assisting a task running on at least one processor of augmented reality smart glasses, the method comprising:
receiving at least one front image from at least one front camera;
recognizing a visually coded pattern in the received at least one foreground image, wherein the visually coded pattern is associated with a particular piece of equipment;
displaying an image guiding a method of operating specific equipment associated with the recognized visually coded pattern in augmented reality on a transparent display;
determining the user's hand motion information according to time by extracting the user's skeletal information from the at least one front image;
determining the skill level of the user based on the time-dependent hand motion information; and
displaying a customized tutorial to the user in augmented reality on the transparent display based on the determined proficiency level
including,
The user confirms the external situation and the image together through the transparent display, work assistance method.
A computer program stored in a computer-readable recording medium for executing the method according to any one of claims 2, 4 to 6 and 8 in a computer.
delete

Images