WO2024076039A1

WO2024076039A1 - Haptic feedback device and method

Info

Publication number: WO2024076039A1
Application number: PCT/KR2023/013887
Authority: WO
Inventors: 위대한; 최재순; 문영진
Original assignee: 재단법인 아산사회복지재단; 울산대학교 산학협력단
Priority date: 2022-10-06
Filing date: 2023-09-15
Publication date: 2024-04-11
Also published as: KR20240048409A

Abstract

A haptic feedback device according to one embodiment can capture, through a first vision sensor, first image data of a first viewing angle corresponding to a head direction of a user, determine a virtual plane on the basis of a position detected through second image data captured by a second vision sensor of one or more haptic units at a time of request when a vibration feedback for a scene corresponding to the head direction of the user is requested, and provide, through the one or more haptic units, vibration feedback determined on the basis of the first image data and a position detected while the position of the one or more haptic units is detected on the virtual plane.

Description

Haptic feedback device and method

Technology related to a haptic feedback device and method is provided.

Haptic devices can be divided into detachable and wearable types. The separate method allows you to feel the image with your fingertips that are sensitive to stimulation by touching your hand to a cluster of sensory replacement elements such as a tablet. The wearable method can deliver images through vibration, etc. by contacting the user's body, such as the abdomen, back, and arms, with a collection of sensory replacement elements consisting of a vibration motor or electrical stimulation in the form of a pad.

Separate haptic devices inevitably limit the user's movement because the user must touch the medium with their fingertips to feel alternative sensory elements, and may have restrictions on outdoor use. A wearable haptic device that solves this problem has a small medium and can be used while moving. However, when a vibration motor is used in a wearable haptic device, since it is worn on a contact area with low stimulation sensitivity, only a large degree of direction indication can be recognized, making it impossible to transmit images related to the user's peripheral vision. When using electrical stimulation in a wearable haptic device, a moist environment is required to improve sensitivity, so the device's human body contact area may be limited to the oral environment. When the device touches the oral environment, the number of electrodes that transmit electrical stimulation is limited, which can cause problems with very low resolution.

The background technology described above is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and cannot necessarily be said to be known technology disclosed to the general public before this application.

A haptic feedback device according to an embodiment can provide an image representing a scene in peripheral vision to a visually impaired person based on haptic feedback.

However, technical challenges are not limited to the above-mentioned technical challenges, and other technical challenges may exist.

A haptic feedback device according to an embodiment includes a housing configured to be worn on the user's head; a first vision sensor positioned in the housing, the first vision sensor configured to capture first image data at a first viewing angle corresponding to the direction of the user's head to collect visual information about the field of view; a second vision sensor located in the housing and capturing second image data at a second viewing angle corresponding to a direction lower than the head direction to collect visual information about the user's manipulated body part; a processor that determines vibration feedback based on a result of analyzing the first image data and the second image data; and one or more haptic units configured to be wearable on the manipulated body part of the user and providing the vibration feedback, wherein the processor detects the position of the one or more haptic units based on the second image data. And, in response to requesting vibration feedback for the scene corresponding to the direction of the user's head, determining a virtual plane based on the detected position of the one or more haptic units at the time point at which the vibration feedback was requested, and , While detecting the position of the one or more haptic units in the virtual plane, vibration feedback determined based on the detected position and the first image data may be provided through the one or more haptic units.

The processor maps pixel values of the first image data to the determined coordinates of the virtual plane, and provides a haptic unit in contact with the virtual plane among the one or more haptic units to provide a pixel value corresponding to the mapped pixel value to the virtual plane. Vibration feedback can be provided.

The manipulating body part includes a hand, and the processor determines a plane corresponding to the bottom surface of the hand at the time when the vibration feedback is requested as the virtual plane, and the vibration according to the movement of the hand in the virtual plane. Feedback can be provided.

The processor may determine that the vibration feedback is requested based on at least one of the user's voice command or gesture and generate the virtual plane.

The haptic feedback device may be formed such that an angle difference between the center direction of the first viewing angle and the center direction of the second viewing angle exceeds a critical angle.

The processor extracts visual information about at least one of an object or a background appearing in a scene corresponding to the first viewing angle by analyzing the first image data, and the haptic feedback device guides the extracted visual information. It may further include a sound output unit that outputs a sound.

The haptic feedback device further includes a sound collection unit that collects the user's voice, and the processor selects an object of interest from one or more objects appearing in the scene corresponding to the first viewing angle based on the collected voice, Vibration feedback for a focusing area corresponding to the selected object of interest among the first image data may be provided through the one or more haptic units on the virtual plane.

The processor maps the highlighted pixel value for the focusing area in the first image data to coordinates of a virtual plane, and provides vibration feedback corresponding to the highlighted pixel value mapped to the virtual plane to the haptic unit in contact with the virtual plane. can be provided.

The processor may map pixel values of the focusing area to coordinates of a virtual plane in the first image data and exclude pixel values of the remaining areas.

The vision sensor may include at least one of a camera sensor, a radar sensor, a LiDAR sensor, an ultrasonic sensor, or an infrared sensor.

A haptic feedback method performed by a processor according to an embodiment includes capturing first image data at a first viewing angle corresponding to the direction of the user's head through a first vision sensor, and capturing first image data at a first viewing angle corresponding to the direction of the user's head. Capturing second image data with a second vision sensor at a requested time when vibration feedback is requested; determining a virtual plane based on the positions of one or more haptic units detected through the second image data; and providing vibration feedback determined based on the detected position and the first image data through the one or more haptic units while detecting the position of the one or more haptic units in the virtual plane.

Capturing the first image data may include extracting visual information about at least one of an object or a background appearing by analyzing the first image data; Outputting a sound guiding the extracted visual information through a sound output unit; Collecting the user's voice speaking based on the guiding sound through a sound collection unit; and selecting a focusing area corresponding to an object of interest among one or more objects appearing in the first image data based on the collected user's voice.

The first image data maps pixel values highlighted for the focusing area to coordinates of the virtual plane, excludes pixel values for the remaining areas, and provides the highlighted pixels mapped to the virtual plane to a haptic unit in contact with the virtual plane. Vibration feedback corresponding to the value can be provided.

A computer-readable recording medium storing one or more computer programs according to an embodiment may include instructions for performing a method.

1 shows an exemplary embodiment of a haptic feedback device according to one embodiment.

Figure 2 shows a block diagram of a haptic feedback device according to one embodiment.

FIG. 3A is a side view of a haptic feedback device according to one embodiment, and FIG. 3B is a front view of the haptic feedback device.

Figure 4 shows a perspective view of a second wearable device among the haptic feedback devices according to an embodiment.

Figure 5 shows an example of vibration feedback of a second wearable device among the haptic feedback devices according to an embodiment.

Figure 6 shows a flowchart of a haptic feedback method according to one embodiment.

FIG. 7 shows a flowchart illustrating an operation for selecting a focusing area of first image data according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

The haptic feedback device 100 according to one embodiment may include a first wearable device 110 (e.g., a sensing unit) and a second wearable device 120 (e.g., a feedback unit). You can.

The first wearable device 110 may be wearable on the head of a user (eg, a visually impaired person). The first wearable device 110 may capture image data based on a vision sensor mounted on the device. For example, the first wearable device 110 may capture image data at the viewing angles 130 of the mounted vision sensor. The first wearable device 110 may separately capture first image data corresponding to the first viewing angle 131 and second image data corresponding to the second viewing angle 132. The first viewing angle 131 may be a viewing angle corresponding to the front (eg, head direction) of the user, and the second viewing angle 132 may be a viewing angle lower than the first viewing angle 131. The first wearable device 110 includes first image data about a scene corresponding to the first viewing angle 131 and a user's hand moving within the second viewing angle 132 or a second wearable device 120 mounted on the hand. ) can be acquired together with second image data including. Additionally, the first wearable device 110 may capture a plurality of image data in consecutive frames. The haptic feedback device 100 may synchronize the frames of the first image data and the frames of the second image data. In other words, the first wearable device 110 can identify first image data and second image data captured simultaneously for each frame.

The first wearable device 110 may analyze first image data and second image data. For example, the first wearable device 110 may map pixel values of the first image data to a virtual plane. The virtual plane may be set as a plane included within the second viewing angle 132 while the user's hand or the pointing position of the second wearable device 120 can be located. The virtual plane is described later. The first wearable device 110 determines a point matching the pointing position (e.g., tip portion of a finger) of the second wearable device 120 in the virtual plane based on the second image data. You can. The first wearable device 110 may generate vibration data based on pixel values mapped to a point matching the tip portion. For example, the first wearable device 110 may generate vibration data including at least one of vibration intensity or vibration time corresponding to the pixel value. The first wearable device 110 may transmit or share the results of processing image data (eg, vibration data) to the second wearable device 120.

The second wearable device 120 may be worn on the user's hand or finger. The second wearable device 120 may generate vibration feedback based on a result of processing the captured image data. For example, the second wearable device 120 may receive data for vibration feedback (eg, vibration data including at least one of vibration intensity or vibration time) from the first wearable device 110. The second wearable device 120 may generate vibration feedback based on the received vibration data. For example, the second wearable device 120 may generate vibration according to at least one of the vibration intensity or vibration time indicated in the vibration data.

However, in this specification, an example in which processing of image data and generation of data for vibration feedback is performed by the first wearable device 110 is mainly described, but is not limited thereto. The first wearable device 110 and the second wearable device 120 may cooperate to analyze image data and generate data for vibration feedback. Additionally, a separate electronic device (eg, a smartphone) may process data by establishing communication with at least one of the first wearable device 110 or the second wearable device 120. For example, the electronic device may receive image data from the first wearable device 110, analyze it, and transmit the analysis result to the second wearable device 120.

As described above with reference to FIG. 1 , the haptic feedback device 100 may include a first wearable device 110 and a second wearable device 120 . The first wearable device 110 and the second wearable device 120 may form a communication channel through direct communication. The haptic feedback device 100 may be connected to the electronic device 200 through a communication channel of the first wearable device 110 and the second wearable device 120.

The first wearable device 110 may include a first vision sensor 111, a second vision sensor 112, a first processor 113, and a first communication unit 114.

The first vision sensor 111 may capture first image data of the first viewing angle 131. The first viewing angle 131 may have a range and angle corresponding to the direction of the user's head. The first vision sensor 111 may collect visual information about the field of view corresponding to the first viewing angle 131. For example, the first vision sensor 111 may capture objects and/or backgrounds in front of the user when going out.

The second vision sensor 112 may capture second image data of the second viewing angle 132 . The second viewing angle 132 may have a range and angle corresponding to a direction lower than the direction of the user's head. The second vision sensor 112 may collect visual information about the manipulated body part. For example, the second vision sensor 112 may capture the position of the user's hand as he or she walks sauntering when going out. A detailed description of the viewing angle and sensing range of the first vision sensor 111 and the second vision sensor 112 will be described later with reference to FIG. 3 .

The first processor 113 may capture a plurality of image data in consecutive frames through the first vision sensor 111 and the second vision sensor 112. In other words, the first vision sensor 111 may capture a plurality of consecutive first image data, and the second vision sensor 112 may capture a plurality of consecutive second image data. The first processor 113 may synchronize the frames of the first image data and the frames of the second image data. That is, the first processor 113 can identify first image data and second image data captured simultaneously for each frame. For example, the first processor 113 may synchronize first image data of an object and/or background in front of the user captured in the first frame with second image data representing the user's hand position captured in the first frame. You can.

The first processor 113 may determine vibration feedback based on a result of analyzing the first image data and the second image data. The first processor 113 may extract visual information about an object and/or background appearing in a scene corresponding to the first viewing angle 131 by analyzing the first image data. The first processor 113 transfers the extracted visual information to the sound output unit, so that the sound output unit can guide the user to the visual information through sound.

As described above, the first processor 113 may detect the position of one or more haptic units based on the second image data corresponding to the second viewing angle 132. The first processor 113 may form a virtual plane in response to vibration feedback being requested. Vibration feedback may be requested based on first image data, which is a scene corresponding to the user's head direction. The virtual plane may be determined based on the detected position of one or more haptic units at the time point at which vibration feedback is requested. For example, when the manipulated body part on which one or more haptic units can be worn is the hand, the first processor 113 may determine the virtual plane as a plane corresponding to the bottom surface of the hand at the time when vibration feedback is requested. That is, the first processor 113 may determine that vibration feedback is requested based on at least one of the user's voice command or gesture and generate a virtual plane.

The first processor 113 may provide vibration feedback through one or more haptic units. Vibration feedback may be determined based on the first image data and the detected position while detecting the position of one or more haptic units in the virtual plane. The first processor 113 may map pixel values of the first image data to the determined coordinates of the virtual plane. The pixel value may be an intensity value of pixels included in the first image data. When the first vision sensor includes a radar sensor, LiDAR sensor, or ultrasonic sensor, the pixel value of the first image data is a value indicating the distance to the physical location corresponding to the pixel (e.g., time of flight (ToF)) value). When the first vision sensor includes an infrared sensor, the pixel value of the first image data may be a value representing the intensity of infrared light reflected from a physical location corresponding to the pixel.

When the first vision sensor includes a camera sensor, the first image data may include color values for each pixel. For example, each pixel of the first image data may have a color vector including color values for each color channel according to the color space. In a representative RGB color space, each pixel of the first image data may include color values (eg, RGB values) including a red value, a green value, and a blue value.

The haptic feedback device can convert the color values of each pixel into a value (e.g., brightness value) that follows gray scale. For example, the haptic feedback device may convert the first image data from a color image to a gray scale image (eg, downgrade conversion). Each pixel value of a gray scale image may, for example, be a brightness value. The brightness value can be expressed as a value between 0 and 255, where 0 indicates the darkest color and 255 indicates the brightest color. The brightness value can be mapped to the location of the corresponding pixel in the virtual plane. As will be described later, the haptic feedback device can provide haptic feedback of the vibration intensity corresponding to the brightness value mapped to the position for each position indicated by the user in the virtual plane.

However, this is not limited to excluding the color values of each pixel from analysis. For example, the haptic feedback device may transmit the color values of each pixel to the first processor 113. The first processor 113 can more easily extract visual information from the first image data based on the color values of each transmitted pixel.

Additionally, the haptic feedback device can generate vibration feedback indicating the color of the object of interest using the color values of each pixel. For example, a haptic feedback device can store distinct vibration patterns for each of a plurality of colors. Each of the plurality of colors may be illustratively defined by a color value range of the RGB color space (eg, a red value range, a green value range, and a blue value range). When the color value of a pixel falls within the color value range of one color among a plurality of colors, the haptic feedback device may determine that the pixel is of the corresponding color. Information defining the color value range of each of a plurality of colors may be referred to as color classification information. Each of the vibration patterns may have a unique pattern that is distinct from other vibration patterns, and the number of vibration patterns and the number of colors classified as described above may be the same. In other words, each color can be assigned a unique vibration pattern. The haptic feedback device can determine the color classification to which the pixel of the object of interest belongs using pre-stored color classification information. The haptic feedback device may generate a vibration pattern assigned to the determined color classification. For example, color classification information may include color value ranges for red, orange, yellow, green, blue, indigo, and purple. When the pixel of the object of interest is determined to be red, the haptic feedback device can generate a unique vibration pattern corresponding to red as vibration feedback.

The first processor 113 may provide vibration feedback to a haptic unit in contact with the virtual plane among one or more haptic units. Vibration feedback may correspond to pixel values mapped to a virtual plane. That is, the first processor 113 may provide vibration feedback according to the movement of the hand in the virtual plane, as described above. A specific method of providing vibration feedback corresponding to the haptic unit will be described later with reference to FIGS. 4 and 5.

The first processor 113 may select an object of interest from one or more objects appearing in the scene corresponding to the first viewing angle. The object of interest may be selected based on the user's voice collected through the sound collection unit. The first processor 113 may provide vibration feedback to the focusing area corresponding to the object of interest selected among the first image data through one or more haptic units 122 on the virtual plane. The first processor 113 may map pixel values of the focusing area to coordinates of a virtual plane in the first image data and exclude pixel values of the remaining areas. That is, since the first processor 113 provides vibration feedback only for the focusing area, the load on calculations for generating vibration feedback can be reduced.

The haptic feedback device 100 may further include a sound output unit that outputs a sound to guide the user and a sound collection unit that collects the user's voice. The sound output unit and the sound collection unit may be configured together or separately in the first wearable device 110 and/or the second wearable device 120.

For reference, the above-described first processor 113 is shown as one processor in the first wearable device 110, but is not limited thereto. The first processor 113 may include a plurality of processors, and may distribute and process tasks performed by each processor.

The first communication unit 114 may be connected to a wired or wireless communication channel. For example, the first communication unit 114 may wirelessly form a Bluetooth or Wi-Fi communication channel with the second wearable device 120. For example, the first communication unit 114 may form a wired communication channel in the form of a telecommunication network (eg, LAN (Local Area Network)) with the second wearable device 120. The first communication unit 114 may be connected to the electronic device 200 by wire or wirelessly. The electronic device 200 may be connected to the first wearable device 110 through the first communication unit 114 and perform the function of the first processor 113 described above. For example, the electronic device 200 may receive image data from the first wearable device 110 and perform an operation to generate vibration data.

The second wearable device 120 may include a second processor 121, a haptic unit 122, and a second communication unit 123.

The second processor 121 may only perform the functions of the first processor 113 described above or control the haptic unit 122. For example, the first processor 113 may generate a pixel value by analyzing the first image data. The second processor 121 may generate final vibration data by mapping the generated pixel values to a virtual plane and analyzing the second image data. For another example, the first processor 113 may analyze image data acquired through the first vision sensor 111 and the second vision sensor 112. The second processor 121 may control the haptic unit 122 based on vibration data generated by the first processor 113 through image data analysis.

In this specification, it is explained that a processor is configured in each of the first wearable device 110 and the second wearable device 120, but it is not limited to this. The operations of the above-described first processor 113 and second processor 121 may be performed in an integrated processor, and the integrated processor may be used in the first wearable device 110, the second wearable device 120, or another external device. Can be included in electronic devices.

One or more haptic units 122 may be wearable on the user's manipulated body part. One or more haptic units 122 may provide vibration feedback through the user's manipulated body part. One or more haptic units may be worn on one or more fingers of each hand. For example, if the haptic feedback device includes 10 haptic units, it can be worn on each of 10 fingertips. Accordingly, the user can freely move the haptic unit 122 worn on the fingertips. The detailed configuration of the haptic unit will be described later in Figures 4 and 5.

The second communication unit 123 may be connected to a wired or wireless communication channel like the first communication unit 114 described above. The second communication unit 123 is also connected to the electronic device 200 and can receive the results of functions performed in the electronic device 200. For example, the electronic device 200 may directly transmit vibration data generated by performing the function of the first processor to the second communication unit 123.

A first vision sensor 111, a second vision sensor 112, a first processor 113, and a communication unit 114 may be located in the housing 115 of the first wearable device 110. The housing 115 may be configured to be worn on the user's head.

The first vision sensor 111 may have a first viewing angle 131 and the second vision sensor 112 may have a second viewing angle 132 . The central direction of the first viewing angle 131 may be formed such that the angle difference from the central direction of the second viewing angle 132 exceeds a critical angle. The first sensing range 141 of the first vision sensor 111 may not overlap with the second sensing range 142 of the second vision sensor 112 or may overlap less than a critical range. In other words, the overlapping range of the first sensing range 141 and the second sensing range 142 can be minimized. That is, when the first vision sensor 111 captures the first image data, the first viewing angle 131 of the capture range may not be disturbed. Additionally, even when the second vision sensor 112 captures second image data, the second viewing angle 132 of the capture range may not be disturbed. For example, the first vision sensor 111 may capture the front view corresponding to the direction of the user's head. The first vision sensor 111 may be placed on the glasses bridge, but is not limited to this. In the example shown in FIGS. 3A and 3B, the central direction of the first viewing angle 131 of the first vision sensor 111 may be depicted as a straight line A. The center direction of the second viewing angle 132 of the second vision sensor 112 may be depicted as a straight line B. The first vision sensor 111 and the second vision sensor 112 may be arranged so that the center direction of the first viewing angle 131 and the center direction of the second viewing angle 132 intersect (e.g., are perpendicular to each other). That is, the second vision sensor 112 may capture second image data including the user's hand positioned between the ground and the front direction. The second vision sensor 112 may capture not only the user's hand but also the virtual plane 140 formed on the bottom of the hand. The virtual plane may be determined based on the location of one or more haptic units detected through the second image data. For example, a virtual plane may be formed around the haptic unit worn on the index finger and thumb, but the present invention is not limited to this, and the virtual plane may be formed based on one or more of the other fingers.

However, in this specification, examples in which the position and viewing angle of the vision sensor are static are mainly described, but are not limited thereto. Depending on the design, the second vision sensor 112 rotates and/or moves according to the movement of the user's hand so that the user's hand 120 can be included within the second viewing angle 132. can be performed. Accordingly, the second vision sensor 112 can continuously capture the user's moving hands when going out. The first vision sensor 111 may rotate and/or move according to the rotation and/or movement of the second vision sensor 112 so that the angle between the center directions of the mutual viewing angles exceeds the critical angle.

For reference, one second vision sensor 112 is shown in FIG. 3A, but it is not limited thereto. The plurality of second vision sensors may include various types of sensors and may be located on any part of the temple or rim.

The first wearable device 110 may be in the form of a headset, helmet, or goggles that can secure a critical angle between the first viewing angle 131 and the second viewing angle 132 as well as glasses.

The first vision sensor 111 and the second vision sensor 112 may include at least one of a camera sensor, a radar sensor, a LiDAR sensor, an ultrasonic sensor, or an infrared sensor. For example, the camera sensor may be implemented as a stereo camera sensor and configured to measure depth.

The second wearable device 400 (eg, second wearable device 120) may be mounted on a user's body part (eg, finger). The second wearable device 400 may be one or more mounting units 410 . One or more mounting units 410 include a control unit 411 (e.g., second processor 121), a vibration unit 412 (e.g., haptic unit 122), and a signal transceiver 413 (e.g., : A second communication unit 123) may be formed. The control unit 411, the vibration unit 412, and the signal transmitting/receiving unit 413 may be interconnected for power supply and mutual control. For example, the control unit 411, the vibration unit 412, and the signal transmitting/receiving unit 413 may be electrically connected through a separate connection member. For reference, in FIG. 4, one or more mounting units 410 are shown in a boneless shape, but the present invention is not limited thereto. Additionally, the second wearable device 400 may form the control unit 411 using a flexible printed circuit board (FPCB). Through this, the one or more mounting units 410 may be boneless or armored connected to the FPCB, or may be attached to the body at any location on the user's body.

The control unit 411 may perform the functions of the second processor 121 described above. As an example, the control unit 411 may directly generate vibration feedback based on the second image data or may control the vibration unit 412 by simply receiving a vibration feedback control signal.

The vibration unit 412 may perform the function of the haptic unit 122 described above. The vibration unit 412 is composed of a vibration motor and can create various vibration patterns. For example, the vibration intensity and vibration time of the vibration unit 412 may be set to vary for each finger. The vibrating unit 412 can deliver a virtual tactile sensation to the user through various vibration patterns.

The signal transceiving unit 413 can perform the functions of the second communication unit 123 described above. That is, the signal transmitting and receiving unit 413 may form a communication channel with the first wearable device 110 and/or the electronic device 200 to transmit and receive control signals. Additionally, the signal transceiver 413 may also receive power for driving from the first wearable device 110 and/or the electronic device 200.

A virtual plane 420 may be formed in the second wearable device 400. The virtual plane 430 may be determined as a plane formed at the tip of a finger on which one or more mounting units 410 are mounted. One or more mounting units 410 may move on the virtual plane 420 . One or more mounting units 410 may move and transmit vibration feedback that changes according to pixel values mapped on the virtual plane 430.

Since one or more mounting units 410 can be mounted on both hands, two virtual planes 420 formed based on one or more mounting units 410 can also be formed. As an example, pixel values mapped to each virtual plane 420 may be mapped by dividing the focusing area of the same first image data. That is, vibration feedback for the left part of the focusing area can be delivered to the left hand, and vibration feedback for the right part of the focusing area can be delivered to the right hand. In conclusion, if the user specifies an object for which recognition is desired through voice, the first processor 113 and/or the second processor 121 can divide parts of the object into left and right based on the center coordinates of the designated object. there is. The first processor 113 and/or the second processor 121 may map the left and right sides of the divided object by corresponding them to right-hand coordinates and left-hand coordinates, respectively.

As another example, pixel values mapped to each virtual plane 420 may be mapped to a first focusing area and a second focusing area among the first image data. That is, pixel values mapped to the virtual plane 420 can be mapped to each virtual plane 420 by selecting a plurality of focusing areas. The first focusing area is mapped to the right hand, and the second focusing area is mapped to the left hand, so that corresponding vibration feedback can be delivered.

The control unit 411 may provide vibration feedback to the vibration unit 412 in contact with the virtual plane 420 among the one or more mounting units 410. Vibration feedback may be generated based on pixel values corresponding to the first image data 510 mapped to the virtual plane. That is, the vibrating unit 412 of the first finger located at the first position 521 may provide vibration based on the first pixel value corresponding to the first position 521 in the first image data. Likewise, the vibrating units 412 of the second, third, fourth, and fifth fingers are respectively positioned at the second position 522, third position 523, fourth position 524, and fifth position. It will be located at (525). Accordingly, the vibrating units 412 of the second, third, fourth, and fifth fingers are respectively positioned at the second position 522, third position 523, and fourth position 524 in the first image data. ), vibration may be provided based on the second pixel value, third pixel value, fourth pixel value, and fifth pixel value corresponding to the fifth position 525. For example, as shown in Figure 5, when the focusing area of the first image data displayed in light and dark corresponds to the eye part of the human face, the first position 521 of the first finger corresponds to the front part of the eye. It may have a first pixel value (e.g., mid-value brightness). Depending on the first pixel value, vibration feedback of medium intensity may be provided to the first finger. The second position 522 of the second finger may have a second pixel value (eg, low value of contrast) corresponding to the eyehole portion. Depending on the second pixel value, strong vibration feedback may be provided to the second finger. The third position 523, fourth position 524, and fifth position 525 of the third, fourth, and fifth fingers correspond to the third pixel, the fourth pixel corresponding to the forehead and the skin between the eyes and eyebrows. pixel, may have a fifth pixel value (e.g., high value contrast). Vibration feedback with a weak intensity may be provided to the third, fourth, and fifth fingers according to the third, fourth, and fifth pixel values. Here, the intensity of vibration feedback is simply expressed as weak, medium, and strong, but it is not limited to this. For example, each pixel value expressed as a value between 0 and 255 may have a step-by-step difference in the intensity of vibration feedback for each value, enabling delicate haptic feedback to express an image. In other words, if the pixel value corresponding to the position of the finger is 1, it can have a vibration feedback intensity expressing 1, and if the pixel value is 2, it can have a vibration feedback intensity expressing 2, which is slightly different from 1. make it possible

In step 610, the processor may capture first image data at a first viewing angle corresponding to the direction of the user's head through the first vision sensor. For example, when the processor detects that the time the head direction was maintained (eg, the time the user maintained the direction without moving) exceeds the threshold time, the processor may acquire first image data through the first vision sensor. For example, the processor may identify a request for visual field information from the user's voice collected through the sound collection unit. The processor may initiate capture of first image data through the first vision sensor based on identifying the request for field of view information.

In step 620, the processor may capture second image data with the second vision sensor at the requested time when vibration feedback is requested. Information in the first image data at the requested viewpoint may be a scene corresponding to the direction of the user's head.

At step 630, the processor may determine the virtual plane based on the positions of one or more haptic units detected through the second image data.

In step 640, the processor may provide vibration feedback determined based on the location where the haptic unit is detected and the first image data through one or more haptic units. The position where the haptic unit is detected may detect the position of one or more haptic units in a virtual plane.

In step 710, the processor may analyze the first image data to extract visual information about at least one of the appearing object or the background. For example, a person's face can be extracted by analyzing the first image data.

In step 720, the processor may output a sound guiding the extracted visual information through a sound output unit. For example, the processor may output guidance information (e.g., guidance voice) about a recognized object (e.g., a human face) through a speaker.

In step 730, the processor may collect the user's voice through a sound collection unit. The user may speak in response to a sound (e.g., guidance information) output by the haptic feedback device. The processor may collect the user's spoken voice in response to the guidance information. For example, the processor may collect the user's spoken voice in at least one of the time sections while outputting the guidance information or after the output of the guidance information. For example, the microphone may collect the user's spoken voice indicating the eye portion of the person's face during the above-described section.

In step 740, the processor may select a focusing area corresponding to the object of interest among one or more objects appearing in the first image data based on the collected user's spoken voice. By analyzing the collected speech sounds, the processor can identify the object of interest indicated by the speech sounds among objects provided to the user through guidance information. The processor may extract a focusing area corresponding to the identified object of interest from the first image data. The processor may generate data (e.g., focusing data) based on the extracted focusing area.

For example, the processor may apply weight to the first image data to emphasize pixel values included in the focusing area compared to pixel values in other areas. The processor may apply a first weight (eg, a weight exceeding 1) to pixel values included in the focusing area. The processor may apply a second weight (eg, a weight of less than 1) to pixel values included in other areas.

For another example, the processor may divide a focusing area from the first image data and generate a focusing image including pixel values corresponding to the divided focusing area. The processor may generate a focused image by preserving pixel values corresponding to the focusing area and removing remaining pixel values. As another example, the processor may enlarge the focusing image by fitting it to the same size as the first image data. For reference, as described above, the pixel value is a value determined by converting the above-mentioned RGB value into a gray scale value, and may be, for example, a brightness value.

In step 750, processing to downscale the focusing area of the first image data may be performed.

In step 760, the highlighted pixel value for the focusing area of the first image data may be mapped to coordinates of a virtual plane through the processor. As described above, pixel values of areas other than the focusing area may be reduced or excluded. A haptic unit in contact with the virtual plane may provide vibration feedback corresponding to the highlighted pixel value mapped to the virtual plane. For example, the haptic feedback device may provide vibration feedback for each pixel within the focusing area of the first image data to a user equipped with the haptic unit when the haptic unit is moved in the virtual plane. Accordingly, the user can quickly recognize the two-dimensional scene for the front view at high resolution (e.g., the resolution of the first image data) while freely moving the body part (e.g., hand) equipped with the haptic unit.

The haptic feedback device can guide rough visual information through voice, focus on the area desired by the user, and provide intensive haptic feedback for that area. Therefore, the haptic feedback device can improve the recognition speed of the user's surrounding visual information.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. It may be possible. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims

In the haptic feedback device,

A housing configured to be worn on the user's head;

a first vision sensor positioned in the housing, the first vision sensor configured to capture first image data at a first viewing angle corresponding to the direction of the user's head to collect visual information about the field of view;

a second vision sensor located in the housing and capturing second image data at a second viewing angle corresponding to a direction lower than the head direction to collect visual information about the user's manipulated body part;

a processor that determines vibration feedback based on a result of analyzing the first image data and the second image data; and

One or more haptic units configured to be wearable on the manipulated body part of the user and providing the vibration feedback

Including,

The processor,

Detecting the location of the one or more haptic units based on the second image data,

In response to requesting vibration feedback for the scene corresponding to the direction of the user's head, determine a virtual plane based on the detected position of the one or more haptic units at the time point at which the vibration feedback was requested,

providing vibration feedback determined based on the detected position and the first image data through the one or more haptic units while detecting the position of the one or more haptic units in the virtual plane,

Haptic feedback device.
According to paragraph 1,

The processor,

Map pixel values of the first image data to coordinates of the determined virtual plane,

Providing vibration feedback corresponding to a pixel value mapped to the virtual plane to a haptic unit in contact with the virtual plane among the one or more haptic units,

Haptic feedback device.
According to paragraph 1,

The manipulated body part includes a hand,

The processor,

Determining a plane corresponding to the bottom surface of the hand at the time when the vibration feedback is requested as the virtual plane,

Providing the vibration feedback according to the movement of the hand in the virtual plane,

Haptic feedback device.
According to paragraph 1,

The processor,

Based on at least one of the user's voice command or gesture, determining that the vibration feedback is requested and generating the virtual plane,

Haptic feedback device.
According to paragraph 1,

Formed so that the angle difference between the center direction of the first viewing angle and the center direction of the second viewing angle exceeds a critical angle.

Haptic feedback device.
According to paragraph 1,

The processor,

By analyzing the first image data, visual information about at least one of an object or a background appearing in a scene corresponding to the first viewing angle is extracted,

The haptic feedback device,

A sound output unit that outputs a sound to guide the extracted visual information

A haptic feedback device further comprising:
According to paragraph 1,

Sound collection unit that collects the user's voice

It further includes,

The processor,

Selecting an object of interest from one or more objects appearing in a scene corresponding to the first viewing angle based on the collected voice,

Providing vibration feedback for a focusing area corresponding to the selected object of interest among the first image data through the one or more haptic units on the virtual plane,

Haptic feedback device.
In clause 7,

The processor,

Mapping the highlighted pixel value for the focusing area in the first image data to coordinates of a virtual plane, and providing vibration feedback corresponding to the highlighted pixel value mapped to the virtual plane to a haptic unit in contact with the virtual plane.

Haptic feedback device.
In clause 7,

The processor,

In the first image data, pixel values of the focusing area are mapped to coordinates of a virtual plane and pixel values of the remaining areas are excluded.

Haptic feedback device.
According to paragraph 1,

The vision sensor is,

Containing at least one of a camera sensor, radar sensor, lidar sensor, ultrasonic sensor, or infrared sensor,

Haptic feedback device.
In a haptic feedback method performed by a processor,

Capturing first image data at a first viewing angle corresponding to the direction of the user's head through a first vision sensor;

Capturing second image data with a second vision sensor at a requested time when vibration feedback for a scene corresponding to the direction of the user's head is requested;

determining a virtual plane based on the positions of one or more haptic units detected through the second image data; and

providing vibration feedback determined based on the detected position and the first image data through the one or more haptic units while detecting the position of the one or more haptic units in the virtual plane.

containing

Haptic feedback method.
According to clause 11,

The step of capturing the first image data includes:

extracting visual information about at least one of an object or a background appearing by analyzing the first image data;

Outputting a sound guiding the extracted visual information through a sound output unit;

Collecting the user's voice speaking based on the guiding sound through a sound collection unit; and

Selecting a focusing area corresponding to an object of interest among one or more objects appearing in the first image data based on the collected user's voice.

containing more

Haptic feedback method.
According to clause 12,

The first image data maps pixel values highlighted for the focusing area to coordinates of the virtual plane, excludes pixel values for the remaining areas, and provides the highlighted pixels mapped to the virtual plane to a haptic unit in contact with the virtual plane. Provides vibration feedback corresponding to the value

Haptic feedback method.
A computer-readable recording medium storing one or more computer programs including instructions for performing the method of any one of claims 11 to 13.