KR102654176B1 - Computer device for visual-based tactile output using machine learning model, and method of the same - Google Patents

Abstract

Various embodiments relate to a computer device and method for visual-based tactile output using a machine learning model, using a machine learning model to obtain a visual image, extract tactile information from the visual image, and obtain tactile information. Based on this, it can be configured to generate a tactile map. According to various embodiments, a computer device may provide an interface for a user using a tactile map.

Description

COMPUTER DEVICE FOR VISUAL-BASED TACTILE OUTPUT USING MACHINE LEARNING MODEL, AND METHOD OF THE SAME}

Various embodiments relate to computer devices and methods for vision-based tactile output using machine learning models.

With the advancement of technology, electronic devices perform various functions and provide various services. Accordingly, the electronic device can provide augmented reality. Augmented reality is a technology that superimposes virtual content on the real environment. In other words, users can view virtual content overlaid on the real environment through an electronic device.

However, such electronic devices do not provide a flexible interface between the electronic device and the user. In other words, the electronic device only provides designated content in a designated environment, but does not provide appropriate content according to the user's situation. For example, electronic devices simply provide an interface based on pre-specified tactile information for a specific object. Because of this, there are problems in that user convenience of electronic devices is low and use efficiency of electronic devices is low.

Korean Patent Publication No. 10-2011-0130469 (2011.12.05.) US Patent Application Publication No. US2021/0397260 (2021.12.23.) Korean Patent Publication No. 10-2021-0143891 (2021.11.29.)

Various embodiments provide computer devices and methods for vision-based tactile output using machine learning models.

A method of a computer device according to various embodiments may include acquiring a visual image, extracting tactile information from the visual image, and generating a tactile map based on the tactile information.

A computer device according to various embodiments includes a memory, and a processor connected to the memory and configured to execute at least one command stored in the memory, wherein the processor generates a visual image using a machine learning model. It may be configured to acquire, extract tactile information from the visual image, and generate a tactile map based on the tactile information.

According to various embodiments, a computer device may provide an appropriate interface depending on the user's situation. In other words, rather than providing pre-designated tactile information to the user, the computer device may provide tactile information to the user by generating a tactile map for a visual image in the direction the user is facing. Through this, the user can feel the tactile sensation of the object without actually approaching or touching the object. At this time, the computer device generates a tactile map using a machine learning model, so user convenience for the computer device can be improved. Accordingly, utilization efficiency for the computer device can be improved.

1 is a diagram illustrating a computer device according to various embodiments.
FIG. 2 is a diagram for explaining the characteristics of the machine learning model of FIG. 1.
FIG. 3 is a diagram illustrating the machine learning model of FIG. 2 by way of example.
FIG. 4 is a diagram for explaining the operation of the machine learning model of FIG. 3.
5 is a diagram illustrating a method of a computer device according to various embodiments.

Hereinafter, various embodiments of this document are described with reference to the attached drawings.

FIG. 1 is a diagram illustrating a computer device 100 according to various embodiments. FIG. 2 is a diagram for explaining the characteristics of the machine learning model 160 of FIG. 1. FIG. 3 is a diagram illustrating the machine learning model 160 of FIG. 2 by way of example. FIG. 4 is a diagram for explaining the operation of the machine learning model 160 of FIG. 3.

Referring to FIG. 1 , various embodiments provide a computer device 100 for visual-based tactile output. At this time, the computer device 100 may be implemented as a near-eye display (NED) device that can be worn on the user's face or head. For example, the near-eye display device may include at least one of smart glasses or a head mounted display (HMD) device. In some embodiments, computer device 100 may provide augmented reality (AR).

According to various embodiments, the computer device 100 includes at least one of a camera module 110, an input module 120, an output module 130, a memory 140, or a processor 150. may include. In some embodiments, at least one of the components of computer device 100 may be omitted and at least one other component may be added. In some embodiments, at least two of the components of computer device 100 may be implemented as one integrated circuit.

The camera module 110 can capture images. At this time, when the computer device 100 is implemented as a near-eye display device, the camera module 110 may capture a front image of the user wearing the computer device 100. For example, the camera module 110 may include at least one lens, at least one image sensor, at least one image signal processor, and at least one flash.

The input module 120 may input a signal to be used in at least one component of the computer device 100. The input module 120 is configured to receive a signal from an input device configured to allow the user to directly input a signal into the computer device 100, a sensor device configured to generate a signal by detecting changes in the surroundings, or an external device. It may include at least one of the receiving devices. For example, the input module 120 may include a microphone, at least one physical button, touch circuitry configured to detect a touch, or a sensor configured to measure the intensity of force generated by the touch. It may include at least one of the circuits.

The output module 130 may output information to the outside of the computer device 100. The output module 130 may include at least one of a display device configured to visually output information, an audio output device capable of outputting information as an audio signal, or a transmission device capable of transmitting information wirelessly. . At this time, when the computer device 100 is implemented as a near-eye display device, the output module 130 may be placed in front of the eyes of the user wearing the computer device 100. The output module 130 may be displayed differently depending on the type of computer device 100. According to one embodiment, when the computer device 100 is an optical see-through type, at least a portion of the output module 130 is made of a transparent or translucent material, and the user can use the output module 130 to You can see the real environment directly. According to another embodiment, when the computer device 100 is a video see-through type, the user can view images of the actual environment captured through the camera module 110 through the output module 130. As an example, the display device may be implemented as a touch screen by being assembled with at least one of the touch circuit or the sensor circuit of the input module 120. For example, an audio output device may include at least one of a speaker or a receiver.

According to some embodiments, the receiving device and the transmitting device may be implemented as communication modules. The communication module may perform communication with an external device in the computer device 100. The communication module may establish a communication channel between the computer device 100 and an external device and perform communication with the external device through the communication channel. Here, the external device may include at least one of a vehicle, satellite, base station, server, or other computer system. The communication module may include at least one of a wired communication module or a wireless communication module. The wired communication module is connected to an external device by wire and can communicate by wire. The wireless communication module may include at least one of a short-range communication module or a long-distance communication module. The short-range communication module can communicate with external devices using short-range communication. For example, the short-range communication method may include at least one of Bluetooth, WiFi direct, or infrared data association (IrDA). The long-distance communication module can communicate with external devices through long-distance communication. Here, the long-distance communication module can communicate with external devices through a network. For example, the network may include at least one of a cellular network, the Internet, or a computer network such as a local area network (LAN) or a wide area network (WAN).

Memory 140 may store various data used by at least one component of computer device 100. For example, the memory 140 may include at least one of volatile memory and non-volatile memory. Data may include at least one program and input or output data related thereto. The program may be stored in the memory 140 as software including at least one command, and may include at least one of an operating system, middleware, or an application.

The processor 150 may execute a program in the memory 140 to control at least one component of the computer device 100. Through this, the processor 150 can process data or perform calculations. At this time, the processor 150 may execute instructions stored in the memory 140.

According to various embodiments, the processor 150 may generate a tactile map 220 for the visual image 210. At this time, the visual image 210 may include at least one of an image captured through the camera module 110 or an image input through the input module 120. Here, the visual image 210 may include at least one of a two-dimensional image or a three-dimensional image. To this end, the processor 150 may include a machine learning model 160. As shown in FIG. 2 , the machine learning model 160 may extract tactile information from the visual image 210 and generate a tactile map 220 based on the tactile information. At this time, the machine learning model 160 may perform tasks such as object detection, semantic segmentation, etc. This machine learning model 160 may be trained in advance using data collected in advance. Here, the collected data may include a plurality of visual images and a plurality of tactile maps each generated correspondingly. Meanwhile, when the visual image 210 consists of a background 211 and at least one object 213, 215, the background 211 and the objects 213, 215 can be distinguished in the tactile map 220. there is.

In some embodiments, machine learning model 160 may be implemented as a deep learning model based on an artificial neural network. As shown in FIG. 3, this machine learning model 160 has three layers 361, 363, and 365, namely, the first layer 361, the second layer 363, and the third layer 365. ) can be achieved. For example, the first layer 361 is composed of a plurality of convolutional layers, the second layer 363 is composed of a plurality of residual blocks, and the third layer 365 ) may be composed of a plurality of deconvolutional layers.

The first layer 361 may extract visual information from the visual image 210 based on at least one predetermined input parameter. For example, visual information may represent surface texture. At this time, the first layer 361 may extract visual information for each pixel of the visual image 210.

The second layer 363 may detect tactile information from visual information based on at least one output parameter mapped to an input parameter. At this time, the second layer 363 can detect tactile information for each pixel of the visual image 210. And, the second layer 363 can detect tactile information for each output parameter. Here, the output parameter may include at least one of frequency, magnitude, or period. For example, as shown in FIG. 4, the second layer 363 may detect tactile information, that is, tactile information for at least one of frequency, size, or period, according to visual information.

The third layer 365 can generate the tactile map 220 using tactile information. Here, the third layer 365 can generate the tactile map 220 by applying tactile information to each pixel of the visual image 210. At this time, the tactile map 220 may be composed of one tactile parameter map 321, 323, and 325, or may be composed of a combination of a plurality of tactile parameter maps 321, 323, and 325. Each tactile parameter map 321, 323, and 325 corresponds to each output parameter and can be generated from tactile information detected for the corresponding output parameter. When tactile information is detected for a plurality of output parameters, the third layer 365 generates a plurality of tactile parameter maps 321, 323, and 325, respectively, and the tactile map 220 is generated by combining these. You can.

For example, the first layer 361 may extract visual information from the visual image 210. Here, the visual image 210 includes a background 211, a first object 213, and a second object 215, and the first layer 361 includes a background 211, a first object 213, and a second object 215. and visual information may be extracted for each of the second objects 215. Here, for each of the background 211, the first object 213, and the second object 215, different visual information can be detected. The second layer 363 may detect tactile information for each of the first output parameter (frequency), second output parameter (size), and third output parameter (period) from visual information. Here, for each of the background 211, the first object 213, and the second object 215, different tactile information can be detected. The third layer 365 applies tactile information to each pixel of the visual image 210, corresponding to the first output parameter (frequency), the second output parameter (size), and the third output parameter (period), respectively. A first tactile parameter map (frequency) 321, a second tactile parameter map (size) 323, and a third tactile parameter map (period) 325 can be generated, respectively. Through this, the tactile map 220 is generated as a combination of the first tactile parameter map (frequency) 321, the second tactile parameter map (size) 323, and the third tactile parameter map (period) 325. It can be.

Accordingly, the processor 150 can use the tactile map 220 to provide an interface for the user. At this time, the processor 150 may provide tactile feedback for the user. Here, the tactile feedback may include at least one of tactile feedback generated while in contact with the user or non-contact feedback generated without contact with the user. For example, tactile feedback can be generated by vibration. Specifically, when the tactile map 220 consists of one tactile parameter map 321, 323, and 325, the processor 150 may provide tactile feedback according to the tactile information of the tactile map 220. Meanwhile, when the tactile map 220 consists of a plurality of tactile parameter maps 321, 323, and 325, the processor 150 provides tactile feedback as a combination of the tactile information of the tactile parameter maps 321, 323, and 325. can be provided.

FIG. 5 is a diagram illustrating a method of using the computer device 100 according to various embodiments.

Referring to FIG. 5, the computer device 100 may acquire a visual image 210 in step 510. At this time, the visual image 210 may include at least one of an image captured through the camera module 110 or an image input through the input module 120. Here, the visual image 210 may include at least one of a two-dimensional image or a three-dimensional image. That is, the processor 150 may acquire the visual image 210 through the camera module 110 or the input module 120.

Next, the computer device 100 may extract tactile information from the visual image in step 520. The processor 150 includes a machine learning model 160, and uses the machine learning model 160 to extract tactile information from the visual image 210 and generate a tactile map 220 based on the tactile information. You can. At this time, the machine learning model 160 may perform tasks such as object detection, semantic segmentation, etc. This machine learning model 160 may be trained in advance using data collected in advance. Here, the collected data may include a plurality of visual images and a plurality of tactile maps each generated correspondingly. Meanwhile, when the visual image 210 consists of a background 211 and at least one object 213 or 215, the background 211 and the objects 213 or 215 may be distinguished in the tactile map 220.

Specifically, the processor 150 may extract visual information from the visual image 210 based on at least one predetermined input parameter. For example, visual information may represent surface texture. At this time, the processor 150 may extract visual information for each pixel of the visual image 210. Afterwards, the processor 150 may detect tactile information from visual information based on at least one output parameter mapped to an input parameter. At this time, the processor 150 may detect tactile information for each pixel of the visual image 210. Additionally, the processor 150 can detect tactile information for each output parameter. Here, the output parameter may include at least one of frequency, magnitude, or period.

Finally, the computer device 100 may generate a tactile map based on tactile information in step 530. The processor 150 may generate the tactile map 220 using tactile information. Here, the processor 150 may generate the tactile map 220 by applying tactile information to each pixel of the visual image 210. At this time, the tactile map 220 may be composed of one tactile parameter map 321, 323, and 325, or may be composed of a combination of a plurality of tactile parameter maps 321, 323, and 325. Each tactile parameter map 321, 323, and 325 corresponds to each output parameter and can be generated from tactile information detected for the corresponding output parameter. When tactile information is detected for a plurality of output parameters, the processor 150 generates a plurality of tactile parameter maps 321, 323, and 325, respectively, and the tactile map 220 can be generated by combining these. .

Accordingly, the processor 150 can use the tactile map 220 to provide an interface for the user. At this time, the processor 150 may provide tactile feedback for the user. Here, the tactile feedback may include at least one of tactile feedback generated while in contact with the user or non-contact feedback generated without contact with the user. For example, tactile feedback can be generated by vibration. Specifically, when the tactile map 220 consists of one tactile parameter map 321, 323, and 325, the processor 150 may provide tactile feedback according to the tactile information of the tactile map 220. Meanwhile, when the tactile map 220 consists of a plurality of tactile parameter maps 321, 323, and 325, the processor 150 provides tactile feedback as a combination of the tactile information of the tactile parameter maps 321, 323, and 325. can be provided.

According to various embodiments, the computer device 100 may provide an appropriate interface depending on the user's situation. That is, the computer device 100 does not provide pre-designated tactile information to the user, but may provide tactile information to the user by generating a tactile map 220 for the visual image 210 in the direction toward which the user is facing. . Through this, the user can feel the tactile sensation of the object without actually approaching or touching the object. At this time, the computer device 100 generates the tactile map 220 using the machine learning model 160, so user convenience for the computer device 100 can be improved. Accordingly, use efficiency for the computer device 100 can be improved.

A method of the computer device 100 according to various embodiments includes acquiring a visual image 210 (step 510), extracting tactile information from the visual image 210 (step 520), and extracting tactile information. Based on this, a step (530) of generating a tactile map (220) may be included.

According to various embodiments, the step of extracting tactile information (step 520) includes extracting visual information from the visual image 210 based on at least one predetermined input parameter, and mapping to the input parameter. It may include detecting tactile information from visual information based on at least one output parameter.

According to various embodiments, visual information is extracted for each of the pixels of the visual image 210, and the step of generating the tactile map 220 (step 530) involves applying tactile information to each of the pixels. , may include generating a tactile map 220.

According to various embodiments, detecting tactile information may include detecting tactile information for each of a plurality of output parameters.

According to various embodiments, the step of generating the tactile map 220 (step 530) includes generating a plurality of tactile parameter maps 321 and 323 corresponding to each of the output parameters, based on tactile information for each of the output parameters. , 325), respectively, and generating the tactile map 220 by combining the tactile parameter maps 321, 323, and 325.

According to various embodiments, the output parameter may include at least one of frequency, magnitude, or period.

According to various embodiments, the method of the computer device 100 may be implemented with a machine learning model 160.

According to various embodiments, the tactile map 220 may be utilized to provide tactile feedback to a user.

According to various embodiments, the computer device 100 may be implemented as a near-eye display device.

The computer device 100 according to various embodiments may include a memory 140 and a processor 150 connected to the memory 140 and configured to execute at least one instruction stored in the memory 140. .

According to various embodiments, the processor 150 acquires the visual image 210 using the machine learning model 160, extracts tactile information from the visual image 210, and based on the tactile information, It may be configured to generate a tactile map 220 .

According to various embodiments, the processor 150 extracts visual information from the visual image 210 based on at least one predetermined input parameter and extracts visual information from the visual image 210 based on at least one output parameter mapped to the input parameter. , may be configured to detect tactile information from visual information.

According to various embodiments, visual information is extracted for each of the pixels of the visual image 210, and the processor 150 applies tactile information to each of the pixels to generate a tactile map 220. It can be configured.

According to various embodiments, the processor 150 detects tactile information for each of a plurality of output parameters, and based on the tactile information for each of the output parameters, a plurality of tactile parameters corresponding to each of the output parameters. It may be configured to generate the maps 321, 323, and 325, respectively, and to generate the tactile map 220 by combining the tactile parameter maps 321, 323, and 325.

According to various embodiments, the output parameter may include at least one of frequency, magnitude, or period.

According to various embodiments, the computer device 100 may be implemented as a near-eye display device.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general-purpose or special-purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for the purpose of being interpreted by or providing instructions or data to the processing device. there is. Software may be distributed over networked computer systems and thus stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

Methods according to various embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. At this time, the medium may continuously store a computer-executable program, or temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

The various embodiments of this document and the terms used herein are not intended to limit the technology described in this document to a specific embodiment, and should be understood to include various changes, equivalents, and/or replacements of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar components. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. In this document, expressions such as “A or B”, “at least one of A and/or B”, “A, B or C” or “at least one of A, B and/or C” refer to all of the items listed together. Possible combinations may be included. Expressions such as "first", "second", "first" or "second" can modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another. The components are not limited. When a component (e.g. a first) component is said to be "connected (functionally or communicatively)" or "connected" to another (e.g. a second) component, it means that the component is connected to the other component. It may be connected directly to a component or may be connected through another component (e.g., a third component).

The term “module” used in this document includes a unit comprised of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrated part, a minimum unit that performs one or more functions, or a part thereof. For example, a module may be comprised of an application-specific integrated circuit (ASIC).

According to various embodiments, each component (eg, module or program) of the described components may include a single entity or a plurality of entities. According to various embodiments, one or more of the components or steps described above may be omitted, or one or more other components or steps may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to integration. According to various embodiments, the steps performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the steps may be executed in a different order, omitted, or , or one or more other steps may be added.

Claims (15)

In the method of the computer device,
Obtaining a visual image;
extracting tactile information from the visual image; and
Generating a tactile map based on the extracted tactile information.
Including,
The step of extracting the tactile information is,
extracting visual information from the visual image based on at least one predetermined input parameter; and
Detecting tactile information from the visual information based on at least one output parameter mapped to the input parameter.
Including,
The step of detecting the tactile information is,
Detecting tactile information for each of a plurality of output parameters.
Including,
The step of generating the tactile map is,
generating a plurality of tactile parameter maps corresponding to each of the output parameters based on tactile information for each of the output parameters; and
Generating the tactile map by combining the tactile parameter maps.
Including,
method.
delete According to claim 1,
The visual information is,
Extracted for each pixel of the visual image,
The step of generating the tactile map is,
Generating the tactile map by applying the extracted tactile information to each of the pixels.
Including,
method.
delete delete According to claim 1,
The output parameters are,
Containing at least one of frequency, magnitude, or period,
method.
The method of claim 1, 3, or 6,
The above method is,
Implemented as a machine learning model,
method.
According to claim 1,
The tactile map is,
Utilized to provide tactile feedback to the user,
method.
According to claim 1,
The computer device,
Implemented as a near-eye display device,
method.
In computer devices,
Memory; and
A processor coupled to the memory and configured to execute at least one instruction stored in the memory
Including,
The processor uses a machine learning model,
Acquire visual images,
Extracting tactile information from the visual image,
Configured to generate a tactile map based on the extracted tactile information,
The processor,
Extracting visual information from the visual image based on at least one predetermined input parameter,
configured to detect tactile information from the visual information based on at least one output parameter mapped to the input parameter,
The processor,
Detect tactile information for each of the plurality of output parameters,
Based on the tactile information for each of the output parameters, generate a plurality of tactile parameter maps corresponding to each of the output parameters, respectively,
configured to generate, by a combination of the tactile parameter maps, the tactile map,
computer device.
delete According to claim 10,
The visual information is,
Extracted for each pixel of the visual image,
The processor,
configured to generate the tactile map by applying the extracted tactile information to each of the pixels,
computer device.
delete According to claim 10,
The output parameters are,
Containing at least one of frequency, magnitude, or period,
computer device.
According to claim 10,
The computer device,
Implemented as a near-eye display device,
computer device.