KR102445496B1 - Wearable apparatus and operating method thereof and server for reconizing situation - Google Patents

Abstract

A wearable device is disclosed. In one embodiment, the user's surroundings are recognized based on at least one of image data generated by one or more of the image capturers and sound data generated by one or more of the sound receivers, and the recognition result is outputted to a sound output device. one or more of them and guide the user through the display.

Description

WEARABLE APPARATUS AND OPERATING METHOD THEREOF AND SERVER FOR RECONIZING SITUATION

The embodiments below relate to a wearable device and a context-aware server.

Recently, a device for displaying a virtual reality (VR) image, an augmented reality (AR) image, or a mixed reality (MR) image appears.

As a related prior art, there is Korean Patent Application Laid-Open No. 10-2019-0098925 (Title of the Invention: XR Device and Control Method Thereof, Applicant: LG Electronics Co., Ltd.). The patent publication discloses a transparent display, a sensing unit for sensing the user's relative position and gaze direction with respect to the transparent display, and based on the relative position and gaze direction sensed through the sensing unit, the user is viewing through the transparent display. An extended reality (XR) device including a processor for recognizing an external object in the real world located in front of the transparent display is disclosed. Here, XR is an expression referring to VR, AR, and MR.

A wearable device according to one side includes: a plurality of image capturers; a plurality of sound receivers; a plurality of sound output devices; display; and recognizing the surrounding situation of the user based on at least one of image data generated by one or more of the image capturers and sound data generated by one or more of the sound receivers, and outputting the recognition result to the sound output device one or more of these and a controller for guiding the user through the display.

The controller may output a visual feedback corresponding to the recognition result to the display, and output an auditory feedback corresponding to the recognition result through one or more of the sound output devices.

The display may correspond to a transparent display, and the controller may display an augmented reality image including the recognition result on the transparent display.

When the controller recognizes a situation in which an object makes sound in a specific direction based on at least one of the image data and the sound data, the controller matches the specific direction or through a sound output device matching the specific direction among the sound output devices A voice included in the recognized situation may be output through a plurality of sound output devices as much as possible, and a message corresponding to the recognized situation and a graphic object expressing the specific direction may be exposed on the display.

The message may be tilted and exposed in a region matching the specific direction among regions of the display. The slope of the message may indicate the specific direction.

The display may display an image corresponding to the direction in which the user is looking through the image data, and when the controller recognizes a situation in which an object in the image displayed on the display produces a sound, the A sound included in the recognized situation can be output through a sound output device matching the direction or through a plurality of sound output devices to match the direction, and a message corresponding to the recognized situation can be exposed around the object in the image. can

When the controller recognizes a situation in which an object approaches the user in a specific direction based on the image data and the sound data, a sound included in the recognized situation is matched with the specific direction among the sound output devices. A graphic expressing the specific direction and at least one of a message corresponding to the recognized situation and a message indicating the approach of the object on the display, which can be output through an output device or through a plurality of sound output devices to match the specific direction objects can be exposed.

When the controller fails to perform processing for recognizing the surrounding situation, the controller transmits the image data and the sound data to a context-aware server through a communication interface, and from the context-aware server, the context-aware server sends the image data and A result of recognizing the surrounding situation based on the sound data may be received through the communication interface, and an augmented reality image including the received result may be displayed on the display.

The wearable device may include a center frame in the form of spectacles, temples, and a spectacle band connected to the temples.

The imagers are located in one or more of the front camera located in the center frame to photograph the front image of the user, the rear camera located in the spectacle band to photograph the rear image of the user, and located in one of the temples. It may include a first side camera for photographing a first side image of the user, and a second side camera located on the other one of the temples to photograph a second side image of the user.

The sound receivers may include one or more microphones corresponding to the one or more front cameras, a microphone corresponding to the rear camera, a microphone corresponding to the first side camera, and a microphone corresponding to the second side camera.

The sound output devices may include one or more speakers corresponding to the one or more front cameras, a speaker corresponding to the rear camera, a speaker corresponding to the first side camera, and a speaker corresponding to the second side camera.

The operating method of the wearable device according to one side is to recognize the surrounding situation of the user based on at least one of image data generated by one or more of the imagers and sound data generated by one or more of the sound receivers step; and guiding the recognition result to the user through one or more of the sound output devices and a display.

The guiding may include outputting a visual feedback corresponding to the recognition result to the display and outputting an auditory feedback corresponding to the recognition result through one or more of the sound output devices.

The display may correspond to a transparent display, and the step of guiding may include displaying an augmented reality image including the recognition result on the transparent display.

In the guiding step, when a situation in which an object makes a sound in a specific direction is recognized based on at least one of the image data and the sound data, a sound output device matching the specific direction among the sound output devices or the specific direction It may include outputting a voice included in the recognized situation through a plurality of sound output devices to match the direction, and exposing a message corresponding to the recognized situation and a graphic object expressing the specific direction on the display. .

The message may be tilted and exposed in a region matching the specific direction among regions of the display. Also, the slope of the message may indicate the specific direction.

In the guiding step, when an image corresponding to the direction in which the user is looking is displayed on the display through the image data and a situation in which an object in the image displayed on the display produces a sound is recognized, one of the sound output devices outputting the voice included in the recognized situation through a sound output device matching the direction or through a plurality of sound output devices to match the direction, and exposing a message corresponding to the recognized situation around the object in the image may include

In the guiding step, when a situation in which an object approaches the user in a specific direction is recognized based on the image data and the sound data, a voice included in the recognized situation is matched with the specific direction among the sound output devices A graphic expressing the specific direction and at least one of a message corresponding to the recognized situation and a message indicating the approach of the object on the display and output through a plurality of sound output devices to match the specific direction It may include exposing the object.

The method of operating the wearable device may include transmitting the image data and the sound data to a context recognition server through a communication interface when processing for recognizing the surrounding situation is not performed; receiving, by the context-aware server, a result of recognizing the surrounding context based on the image data and the sound data from the context-aware server through the communication interface; and displaying an augmented reality image including the received result on the display.

Context-aware server according to one side is a communication interface; and communicating at least one of image data generated by one or more of imagers included in the wearable device and sound data generated by one or more of sound receivers included in the wearable device from a wearable device. and a controller for receiving through an interface, recognizing the surrounding situation of the user based on at least one of the image data and the sound data, and transmitting the recognition result to the wearable device through the communication interface.

The recognition result is guided to the user through one or more of the sound output devices of the wearable device and a display of the wearable device.

1 to 3 are views for explaining a wearable device according to an embodiment.
4 to 5 are diagrams for explaining the surrounding situation guidance of the wearable device according to an embodiment.
6 is a view for explaining a processing and communication module included in an example of a wearable device according to an embodiment.
7 is a diagram for explaining that the context recognition server recognizes the surrounding context of the user of the wearable device according to an embodiment.
8 is a flowchart illustrating a method of operating a wearable device according to an exemplary embodiment.
9 is a block diagram illustrating a context-aware server according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

Terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 to 3 are views for explaining a wearable device according to an embodiment.

Referring to FIG. 1 , a wearable device 100 according to an embodiment includes a plurality of image capturers 110-1 to 110-a, a plurality of sound receivers 120-1 to 120-b, and a plurality of It includes sound output devices 130 - 1 to 130 - c , a display 140 , a communication unit 150 , and a controller 160 .

The number of imagers 110-1 to 110-a, the number of sound receivers 120-1 to 120-b, and the number of sound output devices 130-1 to 130-c are shown in Figs. It may be the same as the example described with reference to 3 . Depending on the implementation, the number of imagers 110-1 to 110-a, the number of sound receivers 120-1 to 120-b, and the number of sound output devices 130-1 to 130-c are may be different.

Each of the image capturers 110 - 1 to 110 - a may generate image data by photographing the surroundings. Each of the image capturers 110 - 1 to 110 - a may include, for example, a camera.

Each of the sound receivers 120 - 1 to 120 - b may receive sound and generate sound data. Each of the sound receivers 120 - 1 to 120 - b may include, for example, a microphone.

Each of the sound output devices 130-1 to 130-c may output a sound. Each of the sound output devices 130-1 to 130-c may include, for example, a speaker.

The display 140 may correspond to a transparent display and may output augmented reality images and content.

The communication unit 150 communicates with a context-aware server, which will be described later. The communication unit 150 performs, for example, 3G communication, 4G communication, or 5G communication. Also, the communication unit 150 may perform short-range wireless communication (eg, Bluetooth) and may perform Wi-Fi communication. In particular, the wearable device 100 and the context-aware server can be better applied in 5G communication due to the conditions for transmitting and receiving a large number of high-speed multimedia traffic such as video and voice, real-time situational awareness, and low-latency situation recognition results.

The controller 160 may include at least one of image data generated by one or more of the image capturers 110-1 to 110-a and sound data generated by one or more of the sound receivers 120-1 to 120-b. Based on one, the user's surrounding situation is recognized. For example, the controller 160 may recognize at least one of a front situation, a rear situation, a left situation, and a right situation of the user.

The controller 160 may guide the user through the display 140 and at least one of the sound output devices 130 - 1 to 130 -c based on the recognition result of the surrounding situation. As an example, the controller 160 may output visual feedback (or content) (eg, a message and/or a graphic object) corresponding to the recognition result of the surrounding situation to the display 140 and match the recognition result. The resulting auditory feedback (eg, voice) may be output through one or more of the sound output devices 130 - 1 to 130 -c.

In an embodiment, the wearable device 100 may be a glasses type. A wearable device 100 of the glasses type is shown in FIGS. 2 and 3 . For convenience of explanation, a part of the temples is shown in FIG. 3 .

Referring to FIG. 2 , the glasses-type wearable device 100 is connected to a center frame 250 , temples 260-1 and 260-2, and temples 260-1 and 260-2. It includes a spectacle band 280 that is.

In addition, the glasses-type wearable device 100 shown in FIG. 2 includes a processing and communication module 240 and a battery 270 . The processing and communication module 240 includes the controller 160 and the communication unit 150 described with reference to FIG. 1 , and the battery 270 supplies power to the wearable device 100 .

One or more front cameras for capturing a user's front image, and a microphone and a speaker corresponding to the one or more front cameras may be located in the center frame 250 . For example, as in the example shown in FIGS. 2 and 3 , the front camera 210-1, the microphone 220-1, and the speaker 230-1 are located in the center frame 250 to support the user's nose. ) may be located and a right front camera 210-2, a microphone 220-2, and a speaker 230-2 for photographing the right front in the vicinity of the right side of the center frame 250 may be located, and the center A left front camera 210 - 3 , a microphone 220 - 3 , and a speaker 230 - 3 for photographing the left front of the frame 250 may be located near the left side of the frame 250 .

The controller 160 may recognize a forward situation based on image data generated by the front camera 210-1 photographing the front and sound data generated by the microphone 220-1 receiving a sound. In an embodiment, the corresponding image data may include an index (or identifier) indicating that it is based on the front camera 210-1, and the corresponding sound data may include an index indicating that it is based on the microphone 220-1. Due to this, the controller 160 can know which camera the corresponding image data is generated by and can know which microphone the corresponding sound data is generated by.

The controller 160 may output visual feedback to one or more of the transparent displays 290-1 and 290-2 based on the recognition result of the forward situation, and output an auditory feedback through the speaker 230-1. can More specifically, the controller 160 recognizes the front situation based on the image data received from the front camera 210-1, but when recognizing a situation occurring to the left of the reference line 310, the left side based on the recognition result Visual feedback can be output to the transparent display 290 - 2 , and when a situation occurring on the right side of the reference line 310 is recognized, visual feedback can be output to the right transparent display 290 - 1 based on the recognition result. can For example, when the controller 160 recognizes a situation in which the vehicle approaches the user from the front left of the reference line 310 through the image data received from the front camera 210-1, a message indicating the approach of the vehicle is displayed. It may be exposed to the left transparent display 290 - 2 , and a voice notifying the approach of the car may be output through the speaker 230 - 1 .

The controller 160 may recognize the right front situation based on the image data generated by the right front camera 210 - 2 photographing the right front and the sound data generated by the microphone 220 - 2 receiving the sound. . In an embodiment, the corresponding image data may include an index indicating that it is based on the right front camera 210 - 2 , and the corresponding sound data may include an index indicating that it is based on the microphone 220 - 2 .

The controller 160 may output a visual feedback to the right transparent display 290-1 based on the recognition result of the right front side, and may output an auditory feedback through the speaker 230-2. Accordingly, the wearable device 100 may visually and audibly guide the user to the right situation.

The controller 160 may recognize the left front situation based on the image data generated by the left front camera 210-3 photographing the left front and the sound data generated by the microphone 220-3 receiving the sound. . In an embodiment, the corresponding image data may include an index indicating that it is based on the left front camera 210 - 3 , and the corresponding sound data may include an index indicating that it is based on the microphone 220 - 3 .

The controller 160 may output a visual feedback to the left transparent display 290 - 2 based on the recognition result of the left front and may output an auditory feedback through the speaker 230 - 3 . Accordingly, the wearable device 100 may visually and audibly guide the user about the left side situation.

The temples 260-1 have a first side camera 210-5 for photographing the first side of the user, a microphone 220-5 corresponding to the first side camera 210-5, and a speaker 230-5 ) can be located.

The controller 160 recognizes the first side situation based on the image data generated by the first side camera 210-5 photographing the first side and the sound data generated when the microphone 220-5 receives the sound can do. In an embodiment, the corresponding image data may include an index indicating that it is based on the first side camera 210 - 5 , and the corresponding sound data may include an index indicating that it is based on the microphone 220 - 5 .

The controller 160 may output a visual feedback to the left transparent display 290 - 2 based on the recognition result of the first side situation and may output an auditory feedback through the speaker 230 - 5 . Accordingly, the wearable device 100 may visually and audibly guide the user about the left side situation.

The temples 260-2 have a second side camera 210-4 for photographing the user's second side, a microphone 220-4 corresponding to the second side camera 210-4, and a speaker 230-4. ) can be located.

The controller 160 recognizes the second side situation based on the image data generated by the second side camera 210-4 photographing the second side and the sound data generated when the microphone 220-4 receives the sound can do. In an embodiment, the corresponding image data may include an index indicating that it is based on the second side camera 210 - 4 , and the corresponding sound data may include an index indicating that it is based on the microphone 220 - 4 .

The controller 160 may output a visual feedback to the right transparent display 290-1 based on the recognition result of the second side situation and may output an auditory feedback through the speaker 230-4. Accordingly, the wearable device 100 may visually and audibly guide the user to the right situation.

The glasses band 280 may include a rear camera 210-6 for photographing the rear of the user, a microphone 220-6 corresponding to the rear camera 210-6, and a speaker 230-6.

The controller 160 may recognize the rear situation based on the image data generated by the rear camera 210-6 photographing the rear and the sound data generated by the microphone 220-6 receiving the sound. In an embodiment, the corresponding image data may include an index indicating that it is based on the rear camera 210 - 6 , and the corresponding sound data may include an index indicating that it is based on the microphone 220 - 6 .

The controller 160 may output visual feedback to one or more of the transparent displays 290-1 and 290-2 based on the recognition result of the rear situation. For example, when the controller 160 recognizes the situation for the left rear side based on the image data of the rear camera 210 - 6 , the controller 160 may output a visual feedback to the left transparent display 290 - 2 , and the right rear side When the situation is recognized, a visual feedback may be output to the right transparent display 290-1. Also, the controller 160 may output an auditory feedback through the speaker 230 - 6 based on the recognition result of the rear situation. Accordingly, the wearable device 100 may visually and audibly guide the user about the rear situation.

In an embodiment, the controller 160 receives an image input from a plurality of the cameras 210-1 to 210-6 and/or a sound input from a plurality of the microphones 220-1 to 220-6. It is possible to recognize a situation in a specific direction by analyzing, visual feedback corresponding to the recognition result may be output to the displays 290-1 and 290-2, and a voice corresponding to the recognition result may be outputted to the speakers 230-1 to 230-1 to 230-6) can be output through a plurality of.

For example, the controller 160 may include images input from the cameras 210-1, 210-2, and 210-3 and/or images inputted from the microphones 220-1, 220-2, and 220-3. By analyzing the sound, it is possible to recognize the situation of the right front (or left front, or front). Accordingly, the controller 160 may more accurately determine the direction of the situation recognized by the controller 160 . The controller 160 may output a voice corresponding to the recognition result of the right front (or left front, or front) situation through the speakers 230 - 1 , 230 - 2 , and 230 - 3 . Accordingly, the controller 160 may more accurately express the direction of the recognized situation.

As another example, the controller 160 analyzes the image input from the cameras 210-2 and 210-4 and/or the sound input from the microphones 220-2 and 220-4 between the right front and the right side. can recognize the situation in In this case, the controller 160 may output a voice corresponding to the recognition result of the situation between the right front side and the right side through the speakers 230 - 2 and 230 - 4 . Accordingly, the controller 160 may more accurately express the direction of the recognized situation.

As another example, the controller 160 analyzes the image input from the cameras 210-5 and 210-6 and/or the sound input from the microphones 220-5 and 220-6 to be between the rear and the left. can recognize the situation in The controller 160 may output a voice corresponding to the recognition result of the situation between the rear and the left side through the speakers 230 - 5 and 230 - 6 . Accordingly, the controller 160 may more accurately express the direction of the recognized situation.

According to an embodiment, the wearable device 100 may perform 360-degree situational awareness by utilizing the cameras 210-1 to 210-6 and the microphones 220-1 to 220-6, , the situation recognition result may be notified to the user through the displays 290-1 and 290-2 and the speakers 230-1 to 230-6.

In addition, the wearable device 100 may recognize a situation given to a user in an unfamiliar space, an unfamiliar environment, or a foreign country based on images and sounds for 360-degree surroundings, and transparent displays 290-1 and 290- In 2), a translation of a human voice may be displayed or an interpretation may be provided to the user through the speakers 230-1 to 230-6. Accordingly, the wearable device 100 can improve user convenience by supporting the user so that there are no problems such as communication and accident prevention.

4 to 5 are diagrams for explaining the surrounding situation guidance of the wearable device according to an embodiment.

Referring to FIG. 4 , an augmented reality image may be displayed on the display 140 . The display 140 may correspond to a transparent display. For example, an augmented reality image including an image captured by the front camera 210-1 and a result of recognizing a surrounding situation by the controller 160 may be displayed on the display 140 . According to the implementation, the controller 160 has an image captured through the front camera 210 - 1 , an image captured through the right front camera 210 - 2 , and an image captured through the left front camera 210 - 3 . may be combined, and an augmented reality image including an image generated by the combination and a result of recognizing the surrounding situation by the controller 160 may be displayed on the display 140 .

In the example shown in FIG. 4 , a bicycle approaches from the rear of the user of the wearable device 100 , and the person riding the bicycle shouts “Please get out of the way”, and on the left side of the wearable device 100 . Let's say a person says "Hey, there's a bike behind you", and a person from the front right waves towards you and says "hello".

The rear camera 210-6 may transmit image data generated by photographing the rear to the controller 160, and the microphone 220-6 receives the voice of a person riding the rear bicycle and controls the generated sound data. It can be transmitted to (160).

The first side camera 210-5 may transmit image data generated by photographing the left side to the controller 160, and the microphone 220-5 may receive the voice of the left person and transmit the generated sound data to the controller 160 ) can be transmitted.

The right front camera 210-2 may transmit image data generated by photographing the right front to the controller 160, and the microphone 220-2 may receive the voice of the right front person and control the sound data generated by the controller. It can be transmitted to (160).

The controller 160 may recognize a rear situation based on image data received from the rear camera 210 - 6 and sound data received from the microphone 220 - 6 . In other words, the controller 160 may recognize a situation in which the bicycle is approaching from the left rear and asks to get out of the way. Based on the recognition result, the controller 160 may output the message 410 - 1 and the graphic object 42 - 1 representing the rear on the display 140 or 290 - 2 . In this case, the message 410 - 1 and the graphic object 42 - 1 may be output to an area corresponding to the rear of the entire area of the display 140 (ie, the lower left of the display 140 ).

In the example shown in FIG. 4 , the message 410 - 1 is turned over and output to the display 140 , but this is merely exemplary and may be expressed without being turned over. Also, according to an embodiment, an image captured by the rear camera 210 - 6 may be displayed on a partial area of the display 140 .

The controller 160 may recognize the left side situation based on the image data received from the first side camera 210 - 5 and the sound data received from the microphone 220 - 5 . In other words, the controller 160 may recognize a situation on the left in which a person shouts to get out. Based on the recognition result, the controller 160 may output the message 410-2 and the graphic object 420-2 representing the left side on the display 140 or 290-2. In this case, the message 410 - 2 and the graphic object 420 - 2 may be output to the area corresponding to the left side of the entire area of the display 140 (ie, the left side of the reference line 310 ). In addition, the message 410-2 may be output at an angle (eg, 45 degrees) corresponding to the left direction as shown in the example shown in FIG. 4 . The slope of the message 410-2 may indicate or express a specific direction. If the controller 160 recognizes a situation in which a person shouts to get away from the right side, the controller 160 may expose a message inclined by an angle corresponding to the right direction (eg, -45 degrees) to the right side of the display 140 . can

The controller 160 may recognize the right front situation based on the image data received from the right front camera 210 - 2 and the sound data received from the microphone 220 - 2 . In other words, the controller 160 may recognize a right front situation in which a person greets. Based on the recognition result, the controller 160 may output a message 410-3 around the human object 430 on the display 140 or 290-1.

As shown in the example shown in FIG. 4 , the wearable device 100 provides contents (eg, messages and graphic objects) generated based on the recognition result of the surrounding situation and the user sees through the display 140 . An augmented reality image including an image corresponding to the viewing direction may be displayed on the display 140 . Accordingly, the wearable device 100 may visually guide the surrounding situation.

FIG. 5 shows an example in which the wearable device 100 audibly guides the surrounding situation described with reference to FIG. 4 .

The controller 160 may output a voice of “Please get out of the way while approaching the bicycle” through the speaker 230 - 6 based on the recognition result of the rear situation. Accordingly, the wearable device 100 may guide the user with a sound of a situation in which a bicycle is approaching from the rear and a situation in which a bicycle is requested to move away.

The controller 160 may output a voice of “Please get out of the bicycle because it is behind you” through the speaker 230 - 5 based on the recognition result of the upper left corner. Accordingly, the wearable device 100 may guide the user to a situation on the left side of a bicycle coming from behind with a sound.

The controller 160 may output a voice of “Hello. Someone greets” through the speaker 230 - 2 based on the recognition result of the right front situation. Accordingly, the controller 160 may guide the user to a situation in which a person greets from the front with a sound.

If the same situation described with reference to FIG. 4 occurs to the user in a foreign country, the messages 410-1, 410-2, and 410-3 may be translated and output, and each voice included in the recognized situation may be displayed. A machine interpretation service may be provided through the speakers 230 - 2 , 230 - 5 , and 230 - 6 . Accordingly, even if the user does not know the language of the corresponding country well, the user can be provided with information about the surrounding situation through the wearable device 100, thereby improving user convenience.

6 is a view for explaining a processing and communication module included in an example of a wearable device according to an embodiment.

Referring to FIG. 6 , the processing and communication module 240 includes a communication unit 150 and a controller 160 .

The controller 160 includes a management unit 610 and a data processing unit 620 .

The data processing unit 620 includes an input unit 621 and an output unit 622 .

The input unit 621 collects data for situation recognition, and the output unit 622 expresses the situation recognition result.

The input unit 621 includes an image data processing unit and sound receivers 120 for processing image data transmitted from the image capturers 110-1 to 110-a (eg, cameras 210-1 to 210-6). -1 to 120-b) (eg, the microphones 220-1 to 220-6) may include a sound data processing unit for processing sound data.

The output unit 622 is an image output unit for displaying a message and a graphic object corresponding to the situation recognition result on the display 140 and a voice corresponding to the situation recognition result to the sound output devices 130-1 to 130-c (an example). For example, it may include an audio output unit outputting through the speakers 230-1 to 230-6).

The management unit 610 includes a data processing control unit that controls to process each data of an input image, an input sound, an output image, and an output voice handled by the data processing unit 620 in real time, and a data management unit that stores the data or prepares for server communication may include.

The communication unit 150 receives the transmission unit 151 for transmitting image data and sound data input to the input unit 621 to the context recognition server, and messages, voices, and graphic objects corresponding to the context recognition results from the context recognition server. It includes a receiving unit 152 and a communication control unit 153 that controls communication with the context recognition server.

According to an embodiment, the communication unit 150 transmits image data and sound data input to the input unit 621 so that the controller 160 can perform real-time situation recognition processing through the situation recognition server when it is difficult for the controller 160 to perform an operation for situation recognition. It can be sent to a context-aware server.

7 is a diagram for explaining that the context recognition server recognizes the surrounding context of the user of the wearable device according to an embodiment.

Referring to FIG. 7 , the wearable device 100 transmits image data and sound data for a surrounding situation to the context recognition server 700 ( 710 ). As an example, the wearable device 100 includes image data and sound receivers ( Sound data generated by each of 120-1 to 120-b (eg, the microphones 220-1 to 220-6) may be transmitted to the context recognition server 700 .

In the embodiment, the image data of each of the imagers 110-1 to 110-a (eg, the cameras 210-1 to 210-6) has an index assigned to each imager (or each camera). may be included. In addition, the sound data of each of the sound receivers 120-1 to 120-b (eg, the microphones 220-1 to 220-6) may include an index assigned to each sound receiver (or each microphone). have. Accordingly, the context recognition server 700 may distinguish or distinguish between which camera the image data is generated and which microphone the sound data is generated by.

For example, the image data of the front camera 210 - 1 may include index “1”, the image data of the right front camera 210 - 2 may include index “2”, and the left front camera 210 - The image data of 3) may include an index “3”. In addition, the image data of the first side camera 210-4 may include an index “4”, the image data of the second side camera 210-5 may include an index “5”, and the rear camera 210 The image data of -6) may include an index “6”. Similarly, sound data of each of the microphones 220 - 1 to 220 - 6 may include indices "1" to "6", respectively.

The above-mentioned index may mean a direction. Index "1" may mean forward, index "2" may mean right front, index "3" may mean left front, index "4" may mean left, The index “5” may mean the right side, and the index “6” may mean the rear side.

The context recognition server 700 recognizes the user's surroundings based on the image data and sound data received from the wearable device 100 (operation 720 ). In other words, the context recognition server 700 may perform voice-based context recognition and image-based context recognition. For example, the context recognition server 700 may recognize a forward situation based on image data and sound data including index “1”, and a rear situation based on image data and sound data including index “6” can be recognized

Since the description of the surrounding situation recognition of the wearable device 100 may be applied to the surrounding situation recognition of the context recognition server 700 , a detailed description thereof will be omitted.

The context recognition server 700 transmits a recognition result of the surrounding situation of the wearable device 100 to the wearable device 100 ( 730 ), and the wearable device 100 receives the surrounding situation based on the received recognition result. guides the user ( 740 ). As an example, the context recognition server 700 may recognize a rear situation in which a bicycle is approaching from the rear and a rear situation in which a bicycle is approaching from the rear, based on image data and sound data including the index “6”. In this case, the context recognition server 700 may transmit direction information (eg, the index "6" described above), visual information, and voice information indicating in which direction the surrounding situation was recognized to the wearable device 100 . have. Corresponding visual information may include, for example, text of "Please get out of the way (bicycle approaching)" and a graphic object indicating the rear, and the corresponding audio information may include, for example, "bicycle approaching, get out of the way." A voice of "please" may be included. The wearable device 100 may expose visual information to an area corresponding to direction information (eg, index "6") in the entire area of the display 140 . Accordingly, as in the example described with reference to FIG. 4 , the message 410 - 1 and the graphic object 42 - 1 are to be exposed on the lower left side of the display 140 (or the lower end of the left transparent display 290 - 2 ). can Also, the wearable device 100 may select a speaker 230-6 corresponding to direction information (eg, index “6”) from among the speakers 230-1 to 230-6, and the speaker 230 Voice information can be output through -6). Accordingly, as described with reference to FIG. 5 , a voice of “Please get out of the way while approaching a bicycle” may be output through the speaker 230 - 6 .

According to an embodiment, the wearable device 100 may not be able to perform processing for recognizing a surrounding situation. For example, the wearable device 100 may temporarily suspend a processing function for recognizing a surrounding situation in order to reduce power consumption because the state of charge of the battery 270 is less than a predetermined standard. In this case, the wearable device 100 may transmit image data and sound data to the context recognition server 700 to request context recognition from the context recognition server 700 , and receive a recognition result from the context recognition server 700 . can do. Depending on the implementation, the wearable device 100 may not have a context recognition function. Even in this case, the wearable device 100 may request the context recognition server 700 to recognize the surrounding situation as described with reference to FIG. 7 , and by receiving the recognition result from the context recognition server 700 , guidance can be provided.

Since the matters described with reference to FIGS. 1 to 6 may be applied to the matters described with reference to FIG. 7 , a detailed description thereof will be omitted.

8 is a flowchart illustrating a method of operating a wearable device according to an exemplary embodiment.

Referring to FIG. 8 , the wearable device 100 includes image data generated by one or more of the imagers 110-1 to 110-a and one or more of the sound receivers 120-1 to 120-b. The user's surrounding situation is recognized based on at least one of the sound data generated by 810 .

The wearable device 100 guides the recognition result to the user through one or more of the sound output devices 130 - 1 to 130 -c and the display 140 ( 820 ).

According to an embodiment, in step 820 , the wearable device 100 may output visual feedback corresponding to the recognition result to the display 140 , and may output auditory feedback corresponding to the recognition result to the sound output devices 130- 1 to 130-c) may be used for output.

According to an embodiment, in step 820 , the wearable device 100 may move an object (eg, a person, an animal, a car, a bicycle, etc.) in a specific direction based on at least one of image data and sound data. ), when recognizing a situation in which a sound is made, a voice included in the recognized situation may be output through a sound output device matching a specific direction among the sound output devices 130-1 to 130-c, and the display 140 It is possible to expose a message corresponding to the recognized situation and a graphic object expressing a specific direction. In this case, the message may be tilted and exposed to a region matching a specific direction among regions of the display 140 . As an example of the embodiment, since the left situation guidance and the rear situation guidance have been described with reference to FIGS. 4 and 5 , a detailed description of the corresponding embodiment will be omitted.

According to an embodiment, in step 820 , in the wearable device 100 , an image corresponding to the direction the user is looking through is displayed on the display 140 through the image data, and an object in the image displayed on the display 140 is sound. When it is recognized that a situation of making A message corresponding to the recognized situation can be exposed. Since the forward situation guidance has been described with reference to FIGS. 4 and 5 as an example of the embodiment, a detailed description of the embodiment will be omitted.

According to an embodiment, when the wearable device 100 recognizes a situation in which an object approaches the user from a specific direction based on the image data and the sound data in step 820 , a voice included in the recognized situation is output to the sound output device. It can be output through a sound output device matching a specific direction among the ones 130-1 to 130-c, and at least one of a message corresponding to a situation recognized on the display 140 and a message indicating the approach of an object and a specific direction You can expose a graphic object representing Since the rear situation guidance has been described with reference to FIGS. 4 and 5 as an example of the embodiment, a detailed description of the embodiment will be omitted.

According to an embodiment, when the wearable device 100 fails to perform processing for recognizing the surrounding situation, the wearable device 100 may transmit image data and sound data to the context recognition server 700 through a communication interface. Here, the communication interface may correspond to the above-described communication unit 150 . The wearable device 100 may receive the result of recognizing the surrounding situation based on the image data and the sound data from the context recognition server 700 by the context recognition server 700 through the communication interface, including the received result The augmented reality image may be displayed on the display 140 .

The wearable device 100 informs the user of the result in the situation-recognized direction by tilting the text or additional content information or displaying an arrow based on the direction in which the user looks to express the 360-degree situation on the display 140 . can

In addition, the wearable device 100 may use a surround speaker in a 360-degree situation to hear a voice corresponding to the situation recognition result in the corresponding direction.

The wearable device 100 may recognize a specific direction with one camera. Also, the wearable device 100 may recognize a specific direction by collecting and analyzing images coming from multiple cameras.

The wearable device 100 may recognize a specific direction by analyzing sound coming from a plurality of microphones as well as a single microphone input.

The wearable device 100 may output a voice through one speaker. In addition, the wearable device 100 may output a voice through a plurality of speakers so that the voice from a specific direction with higher accuracy can be heard.

Since the matters described with reference to FIGS. 1 to 7 may be applied to the matters described with reference to FIG. 8 , a detailed description thereof will be omitted.

9 is a block diagram illustrating a context-aware server according to an embodiment.

Referring to FIG. 9 , the context-aware server 700 includes a communication interface 910 and a controller 920 .

The controller 920 transmits image data generated by one or more of the imagers 110-1 to 110-a included in the wearable device 100 and the wearable device 100 from the wearable device 100 . At least one of sound data generated by one or more of the included sound receivers 120 - 1 to 120 - b is received through the communication interface 910 .

The controller 920 recognizes the user's surroundings based on at least one of image data and sound data, and transmits the recognition result to the wearable device 100 through the communication interface 910 .

Since the matters described with reference to FIGS. 1 to 8 may be applied to the matters described with reference to FIG. 9 , a detailed description thereof will be omitted.

In one embodiment, it is possible to analyze a situation based on video and sound information for 360 degrees and change it to text or additional information, and output text or arrows inclined at an angle to fit the direction on the display, and through a surround speaker A sound may be heard in the corresponding direction as a result of situation recognition in the corresponding direction. Accordingly, an embodiment may recognize the situation for 360-degree omnidirectional and inform the user.

In addition, an embodiment may provide convenience and convenience to the user through communication including automatic translation based on situational awareness in an unfamiliar space or environment such as a foreign country. In addition, one embodiment can help the hearing and visually impaired to recognize situations and communicate according to a 360-degree direction, such as an activity place such as a ski resort, a diving environment, a mountainous highland and rain forest area, outer space, etc. It can be applied to an environment that requires 360-degree situational awareness to help users safety and respond to abnormal situations.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (19)

In the wearable device,
a plurality of imagers;
a plurality of sound receivers;
a plurality of sound output devices;
display; and
Recognizing the user's surrounding situation based on at least one of image data generated by one or more of the image capturers and sound data generated by one or more of the sound receivers that operate independently of the image capturers,
When a situation for a specific direction is recognized based on at least one of the image data and the sound data, through a sound output device matching the specific direction among the sound output devices or through a plurality of sound output devices to match the specific direction output the voice included in the recognized situation,
A controller that exposes a message corresponding to the recognized situation and a graphic object expressing the specific direction on the display, wherein the voice and the message are converted according to a set language
containing,
wearable device.
According to claim 1,
The controller is
outputting a visual feedback corresponding to the recognition result to the display, and outputting an auditory feedback corresponding to the recognition result through one or more of the sound output devices,
wearable device.
According to claim 1,
The display is
It corresponds to a transparent display,
The controller is
Displaying an augmented reality image including the recognition result on the transparent display,
wearable device.
delete According to claim 1,
The message is tilted to be exposed to an area matching the specific direction among the areas of the display or to indicate a specific direction according to the degree of inclination,
wearable device.
According to claim 1,
The display is
Displaying an image corresponding to the direction in which the user is looking through the image data,
The controller is
When a situation in which an object in the image displayed on the display emits a sound is recognized, a sound output device matching the direction among the sound output devices is included in the recognized situation through a plurality of sound output devices to match the direction Outputting a voice and exposing a message corresponding to the recognized situation around the object in the image,
wearable device.
According to claim 1,
The controller is
When a situation in which an object approaches the user in a specific direction is recognized based on the image data and the sound data, a sound included in the recognized situation is transmitted through a sound output device matching the specific direction among the sound output devices or outputting through a plurality of sound output devices to match the specific direction, and exposing at least one of a message corresponding to the recognized situation and a message indicating an approach of the object and a graphic object expressing the specific direction on the display,
wearable device.
According to claim 1,
The controller is
If the processing for recognizing the surrounding situation cannot be performed, the image data and the sound data are transmitted to a context-aware server through a communication interface, and the context-aware server sends the image data and the sound data from the context-aware server Receiving a result of recognizing the surrounding situation based on the communication interface, and displaying the augmented reality image including the received result on the display,
wearable device.
According to claim 1,
The wearable device is
a center frame in the form of spectacles, temples, and a spectacle band connected to the temples,
The video cameras are
One or more front cameras positioned in the center frame to capture a front image of the user, a rear camera positioned in the spectacle band to capture a rear image of the user, and one of the temples to capture the user's first A first side camera for photographing a side image, and a second side camera located on the other one of the temples to photograph a second side image of the user,
wearable device.
10. The method of claim 9,
The sound receivers include one or more microphones corresponding to the one or more front cameras, a microphone corresponding to the rear camera, a microphone corresponding to the first side camera, and a microphone corresponding to the second side camera,
The sound output devices include one or more speakers corresponding to the one or more front cameras, a speaker corresponding to the rear camera, a speaker corresponding to the first side camera, and a speaker corresponding to the second side camera,
wearable device.
In the method of operating a wearable device,
recognizing a user's surroundings based on at least one of image data generated by one or more of the image capturers and sound data generated by one or more of sound receivers that operate independently of the image capturers; and
guiding the recognition result to the user through one or more of the sound output devices and a display
including,
The guiding step is
When a situation for a specific direction is recognized based on at least one of the image data and the sound data, through a sound output device matching the specific direction among the sound output devices or through a plurality of sound output devices to match the specific direction outputting a voice included in the recognized situation, and exposing a message corresponding to the recognized situation and a graphic object expressing the specific direction on the display - the voice and the message are converted according to a set language
containing,
How a wearable device works.
12. The method of claim 11,
The guiding step is
outputting a visual feedback corresponding to the recognition result to the display, and outputting an auditory feedback corresponding to the recognition result through one or more of the sound output devices;
containing,
How a wearable device works.
12. The method of claim 11,
The display is
It corresponds to a transparent display,
The guiding step is
Displaying an augmented reality image including the recognition result on the transparent display
containing,
How a wearable device works.
delete 12. The method of claim 11,
The message is tilted to be exposed to an area matching the specific direction among the areas of the display or to indicate a specific direction according to the degree of inclination,
How a wearable device works.
12. The method of claim 11,
The guiding step is
When an image corresponding to the direction in which the user is looking is displayed on the display through the image data and a situation in which an object in the image displayed on the display produces a sound is recognized, a sound matching the direction among the sound output devices Outputting a voice included in the recognized situation through an output device or a plurality of sound output devices to match the direction, and exposing a message corresponding to the recognized situation around the object in the image
containing,
How a wearable device works.
12. The method of claim 11,
The guiding step is
When a situation in which an object approaches the user in a specific direction is recognized based on the image data and the sound data, a sound included in the recognized situation is transmitted through a sound output device matching the specific direction among the sound output devices or outputting through a plurality of sound output devices to match the specific direction, and exposing on the display at least one of a message corresponding to the recognized situation and a message indicating an approach of the object and a graphic object expressing the specific direction
containing,
How a wearable device works.
12. The method of claim 11,
transmitting the image data and the sound data to a context recognition server through a communication interface when processing for recognizing the surrounding situation cannot be performed;
receiving, by the context-aware server, a result of recognizing the surrounding context based on the image data and the sound data from the context-aware server through the communication interface; and
displaying an augmented reality image including the received result on the display
further comprising,
How a wearable device works.
communication interface; and
Image data generated by one or more of the imagers included in the wearable device from a wearable device and generated by one or more of sound receivers included in the wearable device and operating independently of the imagers Receives at least one of the received sound data through the communication interface, recognizes the user's surroundings based on at least one of the image data and the sound data, and transmits the recognition result to the wearable device through the communication interface controller that transmits
including,
When the controller recognizes a situation in a specific direction based on at least one of the image data and the sound data,
The recognition result is that the voice included in the recognized situation is output through a sound output device matching the specific direction among sound output devices of the wearable device or through a plurality of sound output devices to match the specific direction, A message corresponding to the recognized situation and a graphic object representing the specific direction are exposed on the display of the device, wherein the voice and the message are converted according to a set language;
Context-Aware Server.

Images