KR101935543B1 - Method, Device, and Computer-Readable Medium for Controlling Tactile Interface Device - Google Patents

Abstract

The present invention relates to a method, an apparatus and a computer-readable medium for controlling a tactile interface apparatus, and more particularly, to a method and apparatus for controlling a tactile interface apparatus in which a blind person uses a tactile sense more intuitively and efficiently, To a method, apparatus and computer-readable medium for controlling a tactile interface device of a new concept allowing interaction with a tactile interface device.

Description

METHOD, APPARATUS, AND COMPUTER-READABLE MEDIUM FOR CONTROLLING Tactile Interface Device [

The present invention relates to a method, an apparatus and a computer-readable medium for controlling a tactile interface apparatus, and more particularly, to a method and apparatus for controlling a tactile interface apparatus in which a blind person uses a tactile sense more intuitively and efficiently, To a method, apparatus and computer-readable medium for controlling a tactile interface device of a new concept allowing interaction with a tactile interface device.

In the information society, it is essential to acquire and use information using computers, and this tendency is the same for the general public as well as the visually impaired.

In order to improve the level of informatization of the visually impaired, they should also be able to acquire and use information at a level similar to that of the normal person. This is important because it makes the daily life of the visually impaired more convenient, Do. In addition, it can be said that it is very important because it can ultimately improve the welfare of the visually impaired by providing various education opportunities and expanding opportunities for social advancement and participation.

However, the conventional method of using the present computer mainly uses a method of visually recognizing visual information output through a monitor and inputting information using an input tool such as a keyboard, a mouse, or a touch pad. Therefore, there is a great limitation in visually impaired persons who can not use the time to perceive the output information of the computer at the same level as the general person and to interact such as immediate information input. As a result, the visual impairment significantly degrades the efficiency of computer use, thereby greatly depriving the computer of opportunities for information acquisition and utilization.

Various techniques have been developed to recognize the visual information and to interact with the computer through the use of the hearing, tactile sense, etc., in order to solve the difficulties in using the computer of the visually impaired. As a representative technology, there is a screen reader which assists the visually impaired to use the computer through the hearing. This is a device or software that helps users to use the computer by outputting the contents displayed on the computer screen and the keyboard information inputted by the user.

However, the screen reader has difficulty in finding the screen output information because it searches Graphic User Interface (GUI) elements of the output screen by only one line of linearized information without two-dimensional spatial position information on the output screen. Especially, The more information there is, the greater the difficulty. In addition, since screen readers provide only a simple textual explanatory description for various graphic information such as pictures and diagrams in addition to characters and GUI elements, the visually impaired can understand and interact with graphic information It will have great difficulty.

Another related art is a Braille information terminal which transmits character information through a braille cell by a tactile sense. This is a method that is used as an independent device by providing some functions of a computer useful for a visually impaired person and a method of using as a screen output auxiliary device for outputting text information of a computer screen analyzed by a screen reader in braille. Instead of acting as an interface for efficient interaction with the computer, both of them serve as substitute devices that perform only a few limited functions of the computer as a substitute for the computer, or as output auxiliary devices that output text information in braille do. Particularly, the Braille information terminal specialized in the output of braille has a problem that it can not display the graphic information like the screen reader.

Korean Published Patent No. 10-2012-0063982

It is an object of the present invention to provide a method, an apparatus, and a computer-readable medium for controlling a new concept tactile interface device that enables a visually impaired person to interact with a computer more intuitively and efficiently using a tactile sense in a manner similar to that of a general- Readable medium.

CLAIMS What is claimed is: 1. A method for controlling a tactile interface device, the tactile interface device being implemented as a computing device and capable of interacting with a user connected to the computing device, the method comprising: An input information generating step of generating information; And an output information generation step of generating output information to the tactile interface device based on an output of a focus area out of outputs from the application being executed in the computing device, A method for controlling a tactile interface device comprising data capable of implementing tactile graphics comprising tactile pixels.

10. A computer-readable medium having stored thereon instructions for causing a computing device to perform the following steps, the steps of: receiving input from a tactile interface device capable of interacting with a user An input information generating step of generating input information to an application being executed in the computing device, based on the input information; And an output information generation step of generating output information to the tactile interface device based on an output of a focus area out of outputs from the application being executed in the computing device, A computer-readable medium comprising data capable of implementing tactile graphics comprising tactile pixels.

A computing device that includes at least one processor and at least one memory and that is capable of controlling a tactile interface device, the computing device comprising: an input conversion unit that generates input information to an application executing in the computing device, ; And an output conversion unit for generating output information to the tactile interface device based on an output of a focus area out of outputs from the application being executed in the computing device, the output information including a plurality of two-dimensional tactile pixels A tactile display device, and a tactile display device.

According to embodiments of the present invention, it is possible to visually impaired persons to intuitively recognize tactile display items in real time.

According to the embodiment of the present invention, it is possible to achieve the effect that the visually impaired persons are able to efficiently input information based on the perceived information at the same time while tactually recognizing the information.

According to an embodiment of the present invention, a blind person recognizes the screen output information of a computer in the form of braille, voice, and tactile graphics, and promptly and intuitively inputs information thereon, It is possible to achieve an effect of making it usable.

According to the embodiments of the present invention, various functions of the OS and various application software can be easily expanded to use.

According to the embodiment of the present invention, it is possible to provide a blind person with a remarkably improved work execution speed and convenience in comparison with a conventional apparatus such as a screen reader.

According to the embodiment of the present invention, it is possible to greatly improve the ability of the visually impaired to use the computer, and to expand the market of computer hardware and software for the visually impaired.

1 is a schematic view illustrating an internal structure of a tactile interface device and a user terminal according to an exemplary embodiment of the present invention.
FIG. 2 is a schematic view illustrating an internal structure of a tactile display device according to an exemplary embodiment of the present invention. Referring to FIG.
FIG. 3 is a three-dimensional view illustrating a tactile interface device according to an embodiment of the present invention.
4 is a three-dimensional view illustrating a tactile display unit according to an embodiment of the present invention.
5 is a view illustrating an example of pin drive modules according to an embodiment of the present invention.
6 is a cross-sectional view of a pin drive module according to an embodiment of the present invention.
7 is a view schematically showing detailed steps of a tactile display step according to an embodiment of the present invention.
8 is a view schematically showing a configuration of a computing device and a tactile-sensing interface device for controlling a tactile-sensing interface device according to an embodiment of the present invention.
9 is a diagram schematically illustrating an internal structure of a computing device for controlling a tactile interface device according to an embodiment of the present invention.
10 is a view schematically showing an internal structure of a part of a configuration of a computing device controlling a tactile interface device according to an embodiment of the present invention.
11 is a diagram schematically illustrating steps of a method for controlling a tactile interface device according to an embodiment of the present invention.
12 is a view schematically showing the steps of input information generating step according to an embodiment of the present invention.
FIG. 13 is a diagram schematically illustrating processes of generating output information according to an embodiment of the present invention.
FIG. 14 is a view schematically showing processes of a tactile graphic conversion step according to an embodiment of the present invention.
15 is a view schematically showing a tactile graphics according to an embodiment of the present invention.
16 is a table showing tactile icon icons according to an embodiment of the present invention.
17 is a diagram showing a first example of a display screen of the computing device.
Fig. 18 is a diagram showing a first example of the display screen shown in Fig. 21, which is converted tactile graphics according to an embodiment of the present invention.
19 is a diagram showing a second example of a display screen of the computing device.
FIG. 20 is a diagram showing a second example of the display screen shown in FIG. 19, which is converted tactile graphics according to an embodiment of the present invention.
21 to 27 are diagrams showing outputs of a tactile interface device according to an operation of a user according to an embodiment of the present invention.
28 is a diagram showing an example of the internal configuration of a computing device according to an embodiment of the present invention.

In the following, various embodiments and / or aspects are now described with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it will also be appreciated by those of ordinary skill in the art that such aspect (s) may be practiced without these specific details. The following description and the annexed drawings set forth in detail certain illustrative aspects of one or more aspects. It is to be understood, however, that such aspects are illustrative and that some of the various ways of practicing various aspects of the principles of various aspects may be utilized, and that the description set forth is intended to include all such aspects and their equivalents.

In addition, various aspects and features will be presented by a system that may include multiple devices, components and / or modules, and so forth. It should be understood that the various systems may include additional devices, components and / or modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in connection with the drawings Must be understood and understood.

As used herein, the terms "an embodiment," "an embodiment," " an embodiment, "" an embodiment ", etc. are intended to indicate that any aspect or design described is better or worse than other aspects or designs. . The terms 'component', 'module', 'system', 'interface', etc. used in the following generally refer to a computer-related entity, And a combination of software and software.

It is also to be understood that the term " comprises "and / or" comprising " means that the feature and / or component is present, but does not exclude the presence or addition of one or more other features, components and / It should be understood that it does not.

Also, terms including ordinal numbers such as first, second, etc. may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, are intended to be inclusive in a manner that is generally understood by those of ordinary skill in the art to which this invention belongs. Have the same meaning. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and, unless explicitly defined in the embodiments of the present invention, are intended to mean ideal or overly formal .

Tactile interface device

Hereinafter, an exemplary method and apparatus for controlling a tactile-sensing interface device according to the present invention and an exemplary tactile-sensing interface device to be controlled in a computer-readable medium will be described.

1 is a schematic view illustrating an internal structure of a tactile-sensing device 1000 and a user terminal according to an exemplary embodiment of the present invention.

As shown in FIG. 1, the tactile interface device 1000 is connected to the user terminal A in a pneumatic or wireless manner, and is configured to transmit and receive data. For example, And may receive the graphic signal from the terminal A to provide tactile information.

However, unlike FIG. 1, the present invention differs from FIG. 1 in that the tactile interface apparatus 1000 itself operates by a proprietary operating system without an external user terminal, and is operated by a tactile interface apparatus (1000) may be operated. However, also in this case, the tactile-sensing interface apparatus 1000 may be provided with a communication function.

Meanwhile, the user terminal A includes a smart phone, a tablet, a personal computer (PC), a mobile phone, a video phone, an electronic book An e-book reader, a desktop PC, a laptop PC, a netbook PC, a personal digital assistant (PDA) A portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, a wearable device (for example, a portable multimedia player (PMP) A head-mounted device (HMD), an electronic garment, an electronic bracelet, an electronic necklace, an electronic app apparel, an electronic tattoo, or a smart watch or a smart watch or the like.

1 or 3, the tactile sensing apparatus 1000 according to an exemplary embodiment of the present invention includes an external device connection unit 1100, a tactile display unit 1200, a direction key unit 1300, a keyboard unit 1400 A control unit 1500, and a speaker unit 1600. The tactile display unit 1200 converts a visual graphic output from a computer to a monitor into a tactile graphic and outputs the visual graphic.

Meanwhile, the user performs input to the tactile interface apparatus through the direction key unit 1300 and the keyboard unit 1400, and the tactile interface apparatus converts the inputted command or information, Lt; / RTI >

Preferably, the keyboard unit 1400 preferably includes a braille keyboard that converts a braille character generally used by the visually impaired to general characters and transmits a character input signal to the computer.

The keyboard unit 1400 receives the braille and transmits it to a tactile interface apparatus 1000 or a user terminal connected to the tactile interface apparatus 1000. The keyboard unit 1400 preferably includes a point key 1410, a shortcut key 1420, and an execution or space key 1430. Since braille embeds a number of dots to form a single letter, it is possible to transmit the information of the pressed button at the same time. The transmitted braille information is transferred to the general character through the software in the tactile interface apparatus 1000 or the software of the user terminal.

For this reason, the tactile display unit 1200 plays the same role as a monitor of a general computer, and the direction key unit 1300 and the keyboard unit 1400 function as a keyboard, a mouse, etc. of a general computer And serves as an input device.

FIG. 2 is a schematic view illustrating an internal structure of a tactile display device 1000 according to an embodiment of the present invention.

The tactile interface apparatus 1000 according to an embodiment of the present invention provides tactile information to a user through a plurality of pins based on data received from a connected external device or data generated in the tactile interface apparatus 1000 A tactile display unit 1200; A direction key unit 1300 for changing the position of the input coordinate or detail to be focused; An external device connection unit 1100 that can be connected to an external device; A keyboard unit 1400 for receiving information from a user in a key input form; A speaker unit 1600 capable of outputting sounds to the outside; And a control unit 1500 for controlling operations of the tactile display unit 1200, the direction key unit 1300, the external device connection unit 1100, the keyboard unit 1400, and the speaker unit 1600 do.

The tactile display unit 1200 provides tactile information with at least one dimensional tactile pixel, and the tactile pixel is constituted by a plurality of pins moving up and down by applying power to a transducer including a piezoelectric ceramics and an elastic body . Preferably, the tactile pixel provides tactile information in two dimensions.

The tactile display unit 1200 includes a display data receiving unit 1210 that receives data received from an external user terminal or receives data generated in the tactile interface apparatus 1000; A tactile data converter 1220 for converting the data into tactile display data; And a plurality of pin driving modules (1230) driven by the tactile display data; And a driving power supply unit 1240 which is supplied with power for driving the tactile display unit 1200, and provides tactile information or tactile graphics based on the input data.

Meanwhile, the direction key unit 1300 changes the position of the input coordinate or detail to be focused.

The external device connection unit 1100 includes at least one of a communication module unit 1110 capable of performing wireless communication and a wired connection unit 1120 capable of wiredly connecting with the external device. The communication module 1110 includes a Bluetooth communication module, a Zigbee communication module, an infrared communication module, a Bluetooth low energy (BLE) communication module, an audio communication module, an LTE (long term evolution) A communication module, an IrDA-based infrared communication module, a wireless LAN (WLAN), a WiBro module, and a wireless USB (Wireless USB) module. The wired connection unit 1120 may include a connection module using a universal serial bus (USB) interface, and may be a wired connection module capable of transmitting and receiving data.

Meanwhile, the speaker unit 1600 performs a function of voice-reading the information displayed in response to the tactile-displayed information, character information, or information input by the direction-key unit 1300 in the tactile- can do.

The controller 1500 controls the overall operation of the tactile interface device, the external device connection unit 1100; The keyboard unit 1400; Or controls the operation of the speaker unit 1600. For example, when data is received from an external user terminal, the data is transmitted to the display data receiving unit 1210 of the tactile display unit 1200, and the operation of the tactile display unit 1200 is commanded. The control unit 1500 processes the input or transmits the input signal to the user terminal A connected through the external device connection unit 1100. In the case where the user inputs the input from the direction key unit 1300,

If the tactile display device is not connected to an external user terminal but operates with its own OS and applications, it can operate independently including a CPU and a memory device of a level that can perform its own operation. Even if it is operable independently, the tactile display device may perform communication with the outside.

FIG. 3 is a three-dimensional view of a tactile display device 1000 according to an embodiment of the present invention.

3, the tactile display apparatus 1000 according to the present invention includes components of the tactile display unit 1200 exposed to the outside, in particular, a plurality of pin drive modules 1230, Lt; / RTI > With the tactile display unit 1200, the user can tactilely recognize the plurality of pin drive modules 1230 of the tactile display unit 1200.

Meanwhile, the user can input information to the tactile interface apparatus 1000 through the keyboard unit 1400, for example, to input text information in order to input specific information.

In addition, in order to output voice or sound information of text to the user, the tactile sensing interface apparatus 1000 may output voice information to the user through the speaker unit 1600. [

3, the tactile sensing apparatus 1000 may be connected to an external user terminal through a wired connection unit 1120 or may be connected to an external user terminal by wireless communication through a communication module unit 1110. [ In this case, the GUI elements in the user terminal are tactually displayed in the tactile interface apparatus 1000, and the user inputs the response or input to the GUI elements through the direction key unit 1300 or the keyboard unit 1400 . In addition, according to the input coordinate change command inputted by the user through the direction key unit 1300, the tactile display device 1000 displays the changed input coordinates in the tactile display unit 1200, Based PC or the like can be more intuitively used.

In addition, the tactile interface apparatus 1000 is not limited to the tactile information recognition by the visually impaired person, but may recognize the tactile information and perform the input of commands or the like to the tactile interface apparatus 1000 accordingly, As a result, it is possible to use a user terminal such as a PC based on a graphic screen as a convenience to the general public.

4 is a three-dimensional view illustrating a tactile display unit 1200 according to an embodiment of the present invention.

The tactile display unit 1200 may display or provide tactile pixels of one or more dimensions. FIG. 4 shows an example of a tactile display unit 1200 capable of providing a two-dimensional tactile pixel. The tactile pixel is constituted by a plurality of pins moving up and down by applying power to a transducer including a piezoelectric ceramic and an elastic body.

Specifically, the tactile pixel is represented by a plurality of pin driving module groups, and each of the pin driving module groups is made up of a plurality of pin driving modules 1230. Alternatively, the entire pin drive module may constitute one pin drive module group.

Specifically, the tactile display unit 1200 includes a master board 1250; A display data receiving unit (1210) electrically connected to the master board (1250) and receiving primary data for tactile display; A tactile data converter 1220 for converting the primary data received from the display data receiver 1210 into data for tactile display; A plurality of slave boards 1260 electrically connected to the master board 1250; And a plurality of pin driving modules (1230) disposed on an outer surface of each of the plurality of slave boards (1260); And a driving power supply unit 1240 for supplying driving power to the above components through the master board 1250.

The driving power supply unit 1240 preferably receives power supplied from the outside to the master board 1250.

The tactile data converting unit 1220 includes a first tactile data converting unit 1221 arranged on the master board 1250, And a second tactile data converter 1222 disposed in the slave board 1260. The first tactile data converter 1221 converts the primary data into tactile data for each slave board 1260, And the second tactile data converting unit 1222 converts the secondary data into third data for driving a group of the plurality of pin driving modules 1230 disposed in each slave board 1260, Data.

The tactile display unit 1200 converts a visual graphic into a tactile graphic form by vertically moving a pin of the pin driving module 1230 having a plurality of arrangements. Since the tactile display unit 1200 according to the present invention does not require the configuration of an additional module as compared with the relay mechanism based on the piezoelectric bimorph in which the pin rises when the voltage is applied and the pin is lowered when the voltage is not applied , It is possible to miniaturize and has a structure in which multiple arrangements are easy.

Therefore, the tactile display unit 1200 as in the present invention can exhibit the effect of providing a high-resolution tactile display at the same time while reducing the size of the apparatus.

The master board 1250 receives the tactile graphic data input through the display data receiving unit 1210 and transmits the tactile graphic data to the first tactile data converting unit 1221. The display data receiving unit 1210 may be a universal asynchronous receiver / transmitter (UART), and the first tactile data converting unit 1221 may be an MCU (Micro Controller Unit).

The first tactile data converting unit 1221 analyzes the tactile graphic data input through the display data receiving unit 1210 and transmits the tactile graphic data to a plurality of slave boards 1260. In such a transmission, the first tactile data converter 1221 may convert data based on the information of the plurality of slave boards 1260, and then transmit the converted data to the slave board 1260.

Alternatively, the first tactile data converter 1221 may convert the tactile graphic data into data corresponding to the respective slave boards 1260, and may transmit the converted data to the respective slave boards 1260 It is possible.

Alternatively, the tactile graphic data transmitted through the display data receiving unit 1210 may be directly transmitted to the slave board 1260 without the first tactile data converting unit 1221.

Then, the second tactile data converter 1222 incorporated in the slave board 1260 converts the data received from the first tactile data converter 1221 or the data received from the first tactile data converter 1221 using a previously programmed CPLD (Complex Programmable Logic Device) The data received through the display data receiving unit 1210 is divided according to each pin driving module array composed of the pin driving module 1230 or the plurality of pin driving modules 1230, Or to a pin drive module array composed of a plurality of pin drive modules 1230.

According to the operation signal, the pin of the pin driving module 1230 can move up and down. Preferably, the pin interval and the rising height are in the range of 1 mm to 10 mm.

5 is a diagram illustrating examples of the pin driving module 1230 according to an embodiment of the present invention.

Hereinafter, an embodiment of the pin drive module 1230 shown in Figs. 5A and 5B will be described.

As shown in Fig. 5 (A), the pin drive module 1230 includes a transducer 1231 that converts a vibration motion to a linear motion; A shaft 1232 connected to the transducer; A moving body 1233 which can move up and down in the axial direction of the shaft 1232 on the shaft 1232; And a pin 1234 that can move according to the movement of the moving body 1233. [

Preferably, the transducer 1231 includes a plurality of piezoelectric ceramics and elastic bodies. When the power is applied, the transducer 1231 linearly converts vibrations due to expansion / contraction of the piezoelectric ceramics to move the moving body 1233 up and down , And the movement of the moving body 1233 can be controlled by controlling the transducer.

Hereinafter, an embodiment of the pin drive module 1230 shown in Figs. 5 (C) and 5 (D) will be described.

As shown in FIG. 5C, the pin drive module 1230 includes a transducer 1231, which converts the oscillating motion into a linear motion; A shaft 1232 connected to the transducer; A moving body 1233 which can move up and down in the axial direction of the shaft 1232 on the shaft 1232; And a pin 1234 movable according to the movement of the moving body 1233; And a casing 1235 surrounding the shaft 1232 and having one end of the pin 1234 inserted therein.

Preferably, the transducer 1231 includes a plurality of piezoelectric ceramics and elastic bodies. When the power is applied, the transducer 1231 linearly converts vibrations due to expansion / contraction of the piezoelectric ceramics to move the moving body 1233 up and down , And the movement of the moving body 1233 can be controlled by controlling the transducer.

6 is a cross-sectional view of an example of a pin drive module 1230 according to an embodiment of the present invention.

6, the transducer 1231 of the pin drive module 1230 includes a housing 1231.1, a first piezoelectric ceramic 1231.2, a second piezoelectric ceramic 1231.4 disposed inside the housing 1231.1; And an elastic member 1231.3 disposed between the first piezoelectric ceramic and the second piezoelectric ceramic.

On the other hand, the moving body 1233 includes a moving body 1233.1 and a rubber ring 1233.2 disposed inside the moving body 1233.1.

By controlling the voltages applied to the first piezoelectric ceramic 1231.2 and the second piezoelectric ceramic 1231.4 of the transducer 1231 in this structure, the vertical movement of the moving body 1233 can be efficiently controlled.

7 is a view schematically showing detailed steps of a tactile display step according to an embodiment of the present invention.

The tactile display step S10 provides tactile information by tactile pixels represented by a plurality of pins moving up and down by applying power to a transducer including a piezoelectric ceramic and an elastic body.

Specifically, the tactile display step includes a primary data receiving step (S11) of receiving primary data transmitted from the connected external device or primary data generated in the tactile interface device (1000);

A slave board information loading step S12 for loading slave board 1260 information such as the number of slave boards 1260 and the number of tactile pixels that can be expressed by each slave board 1260;

A secondary data conversion step (S13) of converting the primary data into secondary data according to the loaded slave board (1260) information;

A slave board transmitting step (S14) of transmitting the converted secondary data to each slave board (1260);

Each of the slave boards 1260 converts the secondary data into tertiary data as a driving signal of a pin driving module array composed of each pin driving module 1230 or a plurality of pin driving modules 1230, Step S15;

And a pin driving step S16 for driving each pin driving module 1230 with the generated driving signal.

The tactile display step converts the visual graphic into a tactile graphic form through the up and down movement of pins of the pin driving module 1230 composed of multiple arrangements and transmits the tactile graphic. The tactile display step as in the present invention requires no additional module configuration as compared with the relay mechanism using the piezoelectric bimorph structure in which the pin rises when a voltage is applied and the pin is lowered when a voltage is not applied, And can be implemented in a structure in which multiple arrangements are easy.

Therefore, the tactile display step as in the present invention can exhibit the effect of providing a high-resolution tactile display at the same time while reducing the size of the apparatus.

Preferably, the primary data receiving step S11 is performed in the master board 1250 and is performed through a universal asynchronous receiver / transmitter (UART) module, which receives external tactile graphic data.

The slave board information loading step S12 and the secondary data converting step S13 may be performed by an MCU (Micro Controller Unit) electrically connected to the master board 1250. [

Preferably, the secondary data conversion step S13 analyzes tactile graphic data input through the primary data reception step S11. Thereafter, the slave board transmission step S14 transmits the analyzed tactile graphic data to the slave board 1260. [

The secondary data conversion step S13 and the slave board transmission step S14 may be performed by converting the data based on the information of the plurality of slave boards 1260 and transmitting the data to the slave board 1260 Do.

Alternatively, the secondary data conversion step S13 may convert the tactile graphic data into data corresponding to the respective slave boards 1260, and may transmit the converted data to the respective slave boards 1260 have.

Alternatively, the tactile graphic data transmitted through the primary data receiving step S11 may be directly transmitted to the slave board 1260 without the secondary data converting step S13.

Thereafter, the tertiary data conversion step S15 is performed by the processing unit built in the slave board 1260. [ In the tertiary data conversion step, the data received through the received data using a pre-programmed CPLD (Complex Programmable Logic Device) is transmitted to each pin drive module 1230 or each of the plurality of pin drive modules 1230 According to the pin drive module array, and transmits an operation signal to the pin drive module array constituted by the respective pin drive module 1230 or the plurality of pin drive modules 1230. [

The plurality of slave boards 1260 are disposed on a single master board 1250 and are preferably electrically connected to the master board 1250.

The above-described tactile interface apparatus is described as a tactile interface apparatus that can be used as one example in a method, apparatus, and computer-readable medium for controlling a tactile interface apparatus described later. The description of the method, apparatus, and computer-readable medium is not limited to the above description, but should be interpreted in the best mode according to its own invention.

Method, apparatus and computer-readable medium for controlling a tactile interface apparatus

Hereinafter, a method, an apparatus, and a computer-readable medium for controlling the tactile interface device according to the present invention will be described.

8 is a view schematically showing a configuration of a computing device and a tactile-sensing interface device for controlling a tactile-sensing interface device according to an embodiment of the present invention.

The tactile interface device 1000 connected to the computing device B is shown as having the same components as the tactile interface device described above in FIG. 13, but this is merely an example, and the present invention is not limited thereto. The tactile interface device connected to the computing device B controlling the tactile interface device of the present invention described below includes any interface device capable of providing tactile graphics, a tactile interface device capable of interacting with the user, And a computing device (B) capable of outputting graphics.

As shown in Fig. 8, the computing device B is connected to a tactile interface device, which is a connection that includes both a wired connection and a wireless connection.

The computing device B may correspond to the above-described user terminal A shown in Fig. The computing device B includes a smart phone, a tablet personal computer (PC), a mobile phone, a videophone, an electronic book reader an e-book reader, a desktop PC, a laptop PC, a netbook PC, a personal digital assistant (PDA) A portable medical device, a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, a wearable device (for example, a head- An electronic bracelet, an electronic necklace, an electronic app apparel, an electronic tattoo, or a smart (smart) device, such as a head-mounted device (HMD) watch and so on.

Such a computing device B includes at least one processor and memory, and may be optionally connected to a display device such as a monitor or may have its own display module.

Alternatively, the computing device B may be in a form coupled to the tactile interface device. In this case, the user may recognize the tactile interface device and the computing device B that controls the tactile interface device as a single device. In the case of such a combined device, the computing device (B) and the tactile interface device may be in the form of sharing memory with the processor.

9 is a diagram schematically illustrating an internal structure of a computing device for controlling a tactile interface device according to an embodiment of the present invention.

A computing device for controlling the tactile interface device according to the present embodiment may include a processor, a bus (corresponding to a bi-directional arrow between the processor, memory, and the network interface unit), a network interface, and a memory. The memory may include an operating system 5100, an input / output control routine 5200, and an application 5300. The processor may include an input / output control unit 2000 and an application execution unit 3000. The computing device controlling the tactile interface device in other embodiments may include more components than the components of FIG.

The memory may be a computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive. In addition, program codes for the operating system 5100, the input / output control routine 5200, and the application 5300 may be stored in the memory. These software components may be loaded from a computer readable recording medium separate from the memory using a drive mechanism (not shown). Such a computer-readable recording medium may include a computer-readable recording medium (not shown) such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. In other embodiments, the software components may be loaded into the memory via the network interface portion 4000 rather than a computer-readable recording medium.

The bus may enable communication and data transfer between components of a computing device controlling the tactile interface device. The bus may be configured using a high-speed serial bus, a parallel bus, a Storage Area Network (SAN), and / or other suitable communication technology.

The network interface unit 4000 may be a computer hardware component for connecting a computing device that controls the tactile interface device to a computer network. The network interface 4000 may connect a computing device that controls the tactile interface device to a computer network via a wireless or wired connection. A computing device that controls the tactile interface device through the network interface unit 4000 may be wirelessly or wirelessly connected to the tactile interface device.

The processor may be configured to process instructions of a computer program by performing an input / output operation of a computing device that controls basic arithmetic, logic, and tactile interface devices. The command may be provided to the processor by the memory or network interface portion 4000 and via the bus. The processor may be configured to execute the program code for the input / output control unit 2000 and the application execution unit 3000. [ Such program codes may be stored in a recording device such as a memory.

The input / output control unit 2000 and the application execution unit 3000 may be configured to perform a method of controlling the tactile interface device to be described below. The above-mentioned processor may omit some components according to the method of controlling the tactile interface device, or may further include additional components not shown, or two or more components may be combined.

10 is a view schematically showing an internal structure of a part of a configuration of a computing device controlling a tactile interface device according to an embodiment of the present invention.

The computing device (B) according to the present invention includes an application execution unit (3000) and an input / output control unit (2000) for controlling mutual input / output of the tactile interface device.

The application executing unit 3000 may include an application (e.g., a searcher in WINDOWS ^TM ) for performing a function or the like basically provided in the OS, an application (e.g., Notepad in WINDOWS ^TM ), an application that is installed separately from the OS and performs its own functions (for example, MS WORD ^TM ), an application that performs its own functions that run on the web, an app on a smartphone, It is a term referring to a module that executes an application that can operate. That is, the application execution unit 3000 refers to a module that executes programs having executable instructions stored in the memory.

A general user can execute the application in the computing device B and interact with the application with a mouse, a keyboard, a monitor, or a touchpad. However, such general devices may include graphic information .

In the present invention, in order to use an application executed by the application executing section 3000 of the computing device (B), the interface or input / output of the application executed by the application executing section (3000) Conversion control is available.

In addition, the control method of the tactile interface device of the present invention is not limited to the input / output system using tactile information in other devices such as vehicles and flight devices, and the input / output system . ≪ / RTI >

According to an embodiment of the present invention, the input / output controller 2000 converts input / output of an application executed in the application executing unit 3000 into a form that can be used by a visually impaired person, implements the input / output in a tactile interface device, The application can be more intuitively and conveniently used through the tactile interface device.

Particularly, since the tactile interface device 1000 is a device that transmits information to a user using tactile information and can receive information or commands from a user through tactile key input or the like, The application execution unit can be used.

10, the input / output control unit 2000 includes an input conversion unit 2100 for converting an input from the tactile interface device into an input to the application execution unit 3000; And an output conversion unit 2200 for converting an output from the application executing unit 3000 into an output of the tactile interface device.

Preferably, in an apparatus for controlling a tactile interface apparatus according to an embodiment of the present invention, in addition to an operation of displaying output information from the application execution unit 3000 in a tactile display form, It is possible to perform an operation of converting into a form usable in the execution unit 3000 and inputting it into the application execution unit 3000. [

That is, according to the present invention, the user does not receive information manually but inputs information based on the recognized information, and the input / output control unit 2000 can input the input information into the application executing unit 3000 So that a user who is visually impaired can use an application executed in the application executing section 3000 at a level similar to that of a general user.

The input conversion unit 2100 includes a key conversion unit 2110 for converting a key input input from the tactile interface device into a form that can be input to the application execution unit 3000; And a coordinate mapping unit 2120 for transforming input coordinates inputted or changed by a keypad or the like in the tactile interface apparatus into input coordinates in the application executing unit 3000. [

Preferably, the input received from the tactile interface device and converted by the key converting unit 2110 is information input by braille characters, direction keys, shortcut keys, and the like.

The coordinate mapping unit 2120 converts the input coordinates or the cursor position in the tactile interface device into the input coordinates in the application execution unit 3000. The input coordinates or the position of the cursor may be generated or changed by inputting a direction key of the tactile interface device.

As will be described later, when the electronic device is a PC, tactile icons are implemented in the tactile graphics outputted by reconfiguring the GUI elements of the monitor display screen of the PC. Since the tactile icon is not matched with the monitor display screen in a spatial 1: 1 manner, the tactile interface device performs detailed focusing (focusing is performed more finely in the focusing area described later), or when the cursor is located When the input coordinates are used as they are, the application executing unit 3000 of the electronic device can not determine which GUI element event has occurred.

Preferably, the coordinate mapping function of the coordinate mapping unit 2120 includes a function of mapping a tactile icon object currently being output as a tactile graphics and a GUI element outputting on a monitor display screen when driving the actual application execution unit 3000 And maps the recognized coordinates.

The tactile graphics displayed on the tactile interface device and the display screen when the application executing part 3000 is generally executed may be different in resolution, configuration, and position. In this case, In order to be input to the application execution unit 3000, input coordinates in the tactile interface must be converted or mapped to input coordinates in the application execution unit 3000, and this is performed by the coordinate mapping unit 2120. With the coordinate mapping unit 2120, the user can use the function of the application executing unit 3000 at a general user level without reducing or simplifying the function of the application executing unit 3000. [

For example, in order to execute an instruction 'A' in the application execution unit 3000 (for example, when a user opens a document folder on a Windows desktop) The coordinate mapping unit 2120 maps the coordinates of the point in the tactile interface device to the coordinates of the icon 'A' of the connected application execution unit 3000, The 'A' command in the application execution unit 3000 can be executed.

As described above, the input conversion unit 2100 includes a coordinate mapping function for informing the application execution unit of a GUI element corresponding to an input coordinate or a cursor position, and a reverse mapping function for converting braille information input from the point character into general characters Can be performed.

The output conversion unit 2200 includes a tactile graphic conversion unit 2210 that converts a focus area corresponding to a part or the whole of the display screen output from the application execution unit 3000 into tactile graphic data, The tactile graphics provided by the tactile interface device may be implemented from the tactile graphics data.

In the present specification, the " display screen " refers to a concept including visual elements included in a visual screen or visual screen, which the application outputs to a general visual display device such as a monitor.

The output conversion unit 2200 further includes a TTS conversion unit 2220 that converts information output from the application execution unit 3000 into voice information. Preferably, the TTS converting unit 2220 performs conversion of the detailed information of the textual form of the tactile element referred to by the input from the tactile interface apparatus or the input coordinates to speech.

On the other hand, when the actual application execution unit 3000 is driven, when all the GUI elements output to the monitor display screen are converted into tactile graphics and outputted to the tactile interface device, it may be difficult for the user to understand the tactile graphics .

Therefore, according to an embodiment of the present invention, the focus area of the display screen output from the application execution unit 3000 is converted into tactile graphic data. That is, only the focus area corresponding to the area focussed by the user or the focus area is converted into tactile graphic data, and the tactile graphic data is displayed on the tactile graphic data so that the user can recognize the current focused area only quickly and conveniently. The error in the input can be eliminated.

Meanwhile, the focus area is determined by at least one of the previous inputs by the user. For example, when the user moves the input coordinate in the application with the direction key input of the tactile interface device, or when the area indicated by the input coordinate is focused by the key input such as the execution key input, .

Meanwhile, the haptic graphic converter 2210 divides the focus area into a plurality of focus area and converts the display elements of each focus area into tactile graphic elements according to a predetermined conversion rule Tactile graphic data.

Fig. 16 shows an example of a predetermined conversion rule for converting such a display element into a tactile graphic element.

For example, when the display element (GUI element) corresponds to 'Button' in FIG. 16, the form displayed on the display screen of the computing device by the application execution unit 3000 is a form Of the GUI. Then, according to the conversion rule shown in FIG. 16, the display element for the 'Button' is converted into a tactile icon, which is a tactile graphic element that can be represented by a tactile graphic, and the converted tactile icon is converted into a tactile graphic .

On the other hand, the key input of the user's tactile interface device, preferably the key element of the display element referred to by the input coordinates or the text information of the tactile icon in a sound form, or the text information of the display element referred to by the input coordinates, The detailed information on the display element or the tactile icon can be output as a braille tactile graphic of the tactile interface device or a voice by the TTS conversion.

15 is a view schematically showing a tactile graphics (TG) according to an embodiment of the present invention. The tactile graphics include a plurality of layers. In the embodiment shown in FIG. 15, the plurality of layers include a first layer L1, a second layer L2, and a third layer L3.

The plurality of focus partial areas correspond to the plurality of layers, respectively. That is, the focus area including a part of the display screen output from the application executing unit 3000 is composed of a plurality of focus partial areas, and each of the plurality of focus partial areas corresponds to each of the plurality of layers . Accordingly, the tactile graphics converting unit 2210 converts the display screen into a tactile graphics for each focus region or layer.

The first layer L1 is a screen control area. The first layer L1 is an icon for indicating a title of a tactile graphic output from the current tactile interface device, and a screen for switching a screen such as 'previous', 'confirm', 'cancel' And tactile icons of the buttons. That is, the first layer L 1 includes a tactile icon for executing a command to change the screen of the display screen output from the application executing unit 3000.

The second layer L2 is a main screen area in which tactile graphics for buttons, menus, folders, graphic information, etc. are provided. The user can perform text input, recognize graphic information, recognize GUI elements, or perform user's control input through the second layer (L2).

The third layer L3 is a layer for displaying detailed information on tactile graphic elements output from the application executing unit 3000 in braille form. The user can confirm detailed information about the tactile elements by recognizing the braille output to the third layer L3. As described above, confirmation of such detailed information can also be performed by the voice converted by the TTS converting unit 2220. [

Although the tactile-sensing interface device and the computing device for controlling the tactile-sensing interface device described above have been described in terms of the tactile-sensing interface device capable of outputting and inputting and the control thereof, the present invention is not limited thereto, But also includes inventions for control methods, apparatus, and computer-readable storage media for tactile interface devices that are only capable of being used.

11 is a diagram schematically illustrating steps of a method for controlling a tactile interface device according to an embodiment of the present invention.

As shown in Figure 11, a method of controlling a tactile interface device of the present embodiment is a method of controlling a tactile interface device that is implemented in a computing device and is capable of interacting with a user connected to the computing device, An input information generating step (SlOO) for generating input information to an application executing in the computing device, based on the input from the interface device; And an output information generation step (S200) for generating output information to the tactile interface device based on an output of a focus area out of an output from the application being executed in the computing device. The output information includes data capable of implementing a tactile graphic composed of a plurality of two-dimensional tactile pixels.

The tactile interface device may provide tactile graphics by a tactile sense, and the input information generation step (S100) and the output information generation step (S200) may be performed in real-time. Specifically, the user recognizes the tactile graphics output by the output information generating step (S200), performs the input according to the tactile graphics, and generates the input information in a form capable of inputting the user's input to the application executing unit (3000) It is preferable that the conversion or input information is generated in step S100.

Here, the computing device includes a smart phone, a tablet personal computer (PC), a mobile phone, a videophone, an e-book reader e a notebook PC, a netbook PC, a personal digital assistant (PDA), a portable personal computer (PC) A mobile multimedia device, a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, a wearable device (e.g., a head- Electronic devices such as a head-mounted device (HMD), an electronic apparel, an electronic bracelet, an electronic necklace, an electronic app apparel, an electronic tattoo, or a smart watch ), Etc. Such a computing device may have more than one A processor and a memory, and may be optionally connected to a display device such as a monitor or may have its own display module.

Alternatively, the computing device may be in a form coupled to the tactile interface device. In this case, the user may recognize the tactile interface device and the computing device that controls the tactile interface device as a single device. In the case of such a combined device, the computing device and the tactile interface device may be in the form of sharing memory with the processor.

The application may be an application (e.g., a searcher in WINDOWS ^TM ) for performing a function or the like provided fundamentally in an OS, an application for performing a function supplementarily provided in the OS (for example, a notepad in WINDOWS ^TM ), An application that is installed separately from the OS and performs its own functions (for example, MS WORD ^TM ), an application that performs its own functions that run on the Web, an application on a smartphone, or an application . That is, the application collectively refers to programs having executable instructions.

Preferably, the output information to the tactile interface device includes information on a display screen displayed in the focus area and input coordinates in the focus area.

FIG. 12 is a diagram schematically illustrating processes of an input information generating step (S100) according to an embodiment of the present invention.

The input information generating step (S100) includes an input receiving step (S110) of receiving an input from the tactile interface device; An input category classifying step (S120) of classifying the category of the input information; An application input generating step (S130) of generating input information in a form that can be input to the application based on the input, or converting the input information into input information in a form that can be input to the application, .

The category of the input includes braille information by a point key input, input coordinate movement information by a direction key input, and command information by command key input.

The point key input is input by a braille keyboard that converts a braille character generally used by visually impaired people into a normal character and transmits a character input signal to a computer. The Braille keyboard transmits the Braille to the user terminal connected to the tactile interface device or the tactile interface device as well as the conventional braille keyboard. Since a braille character is composed of a plurality of dots to form a single character, the braille keyboard can transmit information of a pressed button at the same time.

That is, when the category of the input is braille information, the application input creating step (S130) can generate text information input in the input coordinates in the application according to the braille information.

The direction key input is an input for changing the input coordinates. The direction keys may be up, down, left, and right, and as the user enters a direction key, the input coordinates in the application and / or the input coordinates or cursors in the display elements displayed in the cursor and tactile graphics have.

In one embodiment of the present invention, the input coordinates in the application are changed according to the input of the direction key, and the input coordinates in the changed application can be implemented and displayed as the tactile graphics. Here, the actual coordinates in the application executed in the computing device may be referred to as input coordinates, and the input coordinates in the tactile graphics implemented in the tactile interface device may be referred to as tactile input coordinates.

In one embodiment of the present invention, the tactile input coordinates that designate any one of the display elements of the tactile graphic or adjacent groups are changed according to the input of the direction key, and a tactile input Coordinates may be mapped to input coordinates in the application.

Alternatively, in an embodiment of the present invention, the tactile input coordinates that designate any one of the display elements of the tactile graphic or adjacent groups are changed according to the input of the direction key, and the display element or the adjacent group The input coordinates of any one of the display elements corresponding to the finally changed tactile input coordinate or the adjacent group may be mapped to the input coordinates in the application.

That is, when the input category is input coordinate movement information, the application input creating step (S130) can generate new input coordinates in the application according to the input coordinate movement information.

When the electronic device is a PC, tactile icons are implemented in the tactile graphics outputted by reconfiguring the GUI elements of the monitor display screen of the PC. Since the tactile icon is not matched with the monitor display screen in a spatial 1: 1 manner, when the input coordinates at which the touch input occurs in the tactile interface device are used as they are, It can not be discriminated by the application executing section 3000. [ The application input creation step (S130) for key input with input coordinate movement information substantially corresponds to the coordinate mapping function. Preferably, the mapping table is configured to map a tactile icon object currently output as a tactile graph to a GUI element output on a monitor display screen when driving the actual application execution unit 3000, Mapping.

The tactile graphics displayed on the tactile interface device and the display screen when the application executing part 3000 is generally executed may be different in resolution, configuration, and position. In this case, In order to be input to the application executing unit 3000, the input coordinates in the tactile interface must be converted or mapped to the input coordinates in the application executing unit 3000, and this is performed by the application input generating step S130. The user can use the function of the application executing unit 3000 at a general user level without reducing or simplifying the function of the application executing unit 3000 at this stage.

For example, in order to execute an instruction 'A' in the application execution unit 3000 (for example, when a user opens a document folder on a Windows desktop) The coordinate mapping unit 2120 maps the coordinates of the point in the tactile interface device to the coordinates of the icon 'A' of the connected application execution unit 3000, The 'A' instruction in the execution unit 3000 can be executed.

Preferably, the at least one tactile cell included in the tactile pixel region in which the tactile input coordinates are located may cyclically move up or down or vibrate. Such up-down movement or vibration of the tactile pixel allows the user to grasp which display element the cursor lies on or tactile input coordinates with the tactile angle.

The command information by the command key input includes at least one of an execution command, a space command, and a predetermined shortening command. More preferably, the command information may further include an instruction for performing TTS execution or braille output for the detailed text information of the display element on which the input coordinates or tactile input coordinates are placed.

FIG. 13 is a diagram schematically showing processes of generating output information (S200) according to an embodiment of the present invention. The output information generating step S200 is a step of converting a focus area including a part of a display screen output from the application executing unit 3000 into tactile graphic data.

The output information generating step (S200) includes receiving (S210) output information of an application executed by the application executing unit (3000); A change determining step (S220) of determining whether a display screen displayed in a focus area in an application executed by the application executing unit (3000) is changed; A tactile graphic generation step (S230) of generating tactile graphic data based on a display screen displayed in a focus area in an application executed by the application execution unit (3000); And transmitting the tactile graphic data to the tactile interface device (S240).

Preferably, when it is determined that the display screen has been changed in the change determination step, the tactile graphics conversion step is performed.

FIG. 14 is a view schematically showing processes of tactile graphic conversion step S230 according to an embodiment of the present invention.

When the application is executed by the actual application execution unit 3000, when all the GUI elements output on the monitor display screen are converted into the tactile graphics and output to the tactile interface device, it may be difficult for the user to understand the tactile graphics .

Therefore, according to an embodiment of the present invention, the tactile graphic generation step S230 may convert the focus area including a part of the display screen output from the application execution unit 3000 into tactile graphic data. That is, only the focus area corresponding to the area focussed by the user or the focus area is converted into tactile graphic data, and the tactile graphic data is displayed on the tactile graphic data so that the user can recognize the current focused area only quickly and conveniently. And errors in the input can be eliminated.

Meanwhile, the focus area is determined by at least one of the previous inputs by the user. For example, when a user moves a tactile input coordinate in an application or an input coordinate in an application or a tactile input coordinate in an application with a directional key input of a tactile interface device, or moves an input coordinate or a tactile input coordinate, When the area referred to by the input coordinate or tactile input coordinate is focused, the focusing area is determined.

According to an embodiment of the present invention, the tactile graphic generation step S230 may include dividing the focus area into a plurality of focus part areas, and displaying the display elements of each focus part area according to a predetermined conversion rule as a tactile graphic element Thereby generating tactile graphics data. The tactile graphic element may represent a tactile icon.

That is, the tactile graphic generation step S230 for converting the display screen displayed by the application execution unit 3000 into the tactile graphic data includes a focus area including a part of the display screen output from the application execution unit 3000 A focus area determination / reception step S231 of determining or receiving;

A focus partial area dividing step (S232) of dividing the focus area into a plurality of focus partial areas;

A tactile graphic element conversion step (S233) of converting a display element of each focus partial area into a tactile graphic element according to a predetermined conversion rule; And

And tactile graphic data generation step S234 for generating tactile graphic data including the tactile graphic element.

Fig. 16 shows an example of a predetermined conversion rule for converting such a display element into a tactile graphic element.

For example, when the display element (GUI element) corresponds to 'Button' in FIG. 16, the form displayed on the display screen of the computing device by the application execution unit 3000 is a form Of the GUI. The display element for the 'Button' is converted into a tactile icon, which is a tactile graphic element that can be represented by a tactile graphic, according to the conversion rule shown in FIG. 20, and the converted tactile icon is converted into a tactile graphic .

On the other hand, the key input of the user's tactile interface device, preferably the key element of the display element referred to by the input coordinates or the text information of the tactile icon in a sound form, or the text information of the display element referred to by the input coordinates, The detailed information on the display element or the tactile icon can be output as a braille tactile graphic of the tactile interface device or a voice by the TTS conversion.

15 is a view schematically showing a tactile graphics (TG) according to an embodiment of the present invention. The tactile graphics include a plurality of layers. In the embodiment shown in FIG. 15, the plurality of layers include a first layer L1, a second layer L2, and a third layer L3.

The plurality of focus partial areas correspond to the plurality of layers, respectively. That is, the focus area including a part of the display screen output from the application executing unit 3000 is composed of a plurality of focus partial areas, and each of the plurality of focus partial areas corresponds to each of the plurality of layers . Accordingly, the tactile graphics converting unit 2210 converts the display screen into a tactile graphics for each focus region or layer.

The first layer L1 is a screen control area. The first layer L1 is an icon for indicating a title of a tactile graphic output from the current tactile interface device, and a screen for switching a screen such as 'previous', 'confirm', 'cancel' And tactile icons of the buttons. That is, the first layer L 1 includes a tactile icon for executing a command to change the screen of the display screen output from the application executing unit 3000.

The second layer L2 is a main screen area in which tactile graphics for buttons, menus, folders, graphic information, etc. are provided. The user can perform text input, recognize graphic information, recognize GUI elements, or perform user's control input through the second layer (L2).

The third layer L3 is a layer for displaying detailed information on tactile graphic elements output from the application executing unit 3000 in braille form. The user can confirm detailed information about the tactile elements by recognizing the braille output to the third layer L3. As described above, confirmation of such detailed information can also be performed by the voice converted by the TTS converting unit 2220. [

Hereinafter, embodiments of the display screen output from the application executing unit 3000, which is converted by the method of controlling the tactile display device according to the present invention, will be described. In the following description, the predetermined conversion rule for converting the display element of the display screen shown in Fig. 16 into the tactile graphic element is applied.

17 is a diagram showing a first example of a display screen of the computing device B;

17, the focus area corresponds to a window in which the application is executed, and the focus partial area includes a first focus partial area including (1), (2), (3), and (4) And a second focus partial area.

The above (1) corresponds to the title of this application, (2) corresponds to the menu bar of this application, (3) corresponds to the 'minimize' button of this application, Quot; close " button, and (5) corresponds to a window for displaying objects displayed in the application.

FIG. 18 is a diagram showing a tactile graphics output by converting a first example of the display screen shown in FIG. 17 into a tactile graphics according to an embodiment of the present invention.

The first focus partial area including the above-mentioned (1), (2), (3) and (4) is converted or mapped to the upper layer (first layer Ll) Is converted or mapped to a layer (second layer L2) of interruption of tactile graphics. As described above, the conversion rule at the time of mapping is in accordance with the rule shown in FIG.

On the other hand, the lower layer of the tactile graphics corresponds to a layer representing the text information of the tactile icon in the first layer L1 and the second layer L2 in braille form.

When the input coordinate or the tactile input coordinate is changed by the keypad, the lower layer automatically displays the text information of the tactile icon corresponding to the input coordinate or the tactile input coordinate in a braille form, The text information of the tactile icon corresponding to the current input coordinate or tactile input coordinate can be represented in braille form.

For example, (1) area corresponds to a title, which is converted into a predetermined tactile icon by the conversion rule shown in FIG. 16, and is implemented in the first layer L1 of the tactile graphics. On the contrary, when the user places an input coordinate or a tactile input coordinate on a tactile icon corresponding to the area (1) and inputs a shortcut key for displaying braille information, the third layer (L3) The "Haptic Editor" can be output in braille. Alternatively, if the user changes the input coordinates or the tactile input coordinates to the tactile icon corresponding to the area (1), the third layer (L3) automatically displays the "Haptic Editor" corresponding to the detailed information of the title in braille Lt; / RTI >

Fig. 19 is a diagram showing a second example of a display screen of the computing device B; Fig.

19 is a display screen when a menu bar corresponding to the area (2) is selected in the application shown in Fig. 17, and 'Page Control' is selected or selected.

In this case, the focus area corresponds to an area referred to as " Focused area " in FIG. 19, and the focus part area includes a first focus partial area including (1) and a second focus partial area including (2) And can be divided into a focus partial area.

The above (1) corresponds to the title selected in the menu bar, and the above (2) to (6) correspond to the submenu that can be selected within the title.

FIG. 20 is a diagram showing a tactile graphics output by converting a second example of the display screen shown in FIG. 19 into a tactile graphics according to an embodiment of the present invention.

The first focus partial area including the above-mentioned (1) and (7) is converted or mapped to the upper layer (the first layer Ll) of the tactile graphics, and the second focus partial area including the (2) The focus partial area is converted or mapped to the layer of interruption of the tactile graphic (second layer L2). As described above, the conversion rule at the time of mapping is in accordance with the rule shown in FIG. (7) is a tactile icon for the 'backward' button, which is not actually included in the display screen of the application, but corresponds to an icon generated by itself in order to expand the convenience of the user in the control method according to the present invention.

On the other hand, the lower layer of the tactile graphic corresponds to a layer that represents text information of the tactile icon corresponding to the input coordinates or tactile input coordinates in the first layer (L1) and the second layer (L2) in braille form.

21 to 27 are diagrams showing outputs of a tactile interface device according to an operation of a user according to an embodiment of the present invention.

A method of controlling a tactile interface device in accordance with an embodiment of the present invention is a method of controlling a tactile interface device that is implemented in a computing device and is connected to the computing device and capable of interacting with a user, An input information generating step of generating input information to an application executing in the computing device based on the input of the input information; And an output information generation step of generating output information to the tactile interface device based on an output of a focus area out of outputs from the application being executed in the computing device, And data capable of implementing tactile graphics composed of tactile pixels.

This method of controlling the tactile interface device enables the visually impaired to operate the operating system and applications, for example, to control and input information more efficiently based on the haptic display.

 According to an embodiment of the present invention, the user can perform control and information input to the tactile interface device primarily by key input of a point key, a direction key, and the like, Lt; / RTI > may be entered into the operating system or application being executed on the network.

Meanwhile, in an embodiment of the present invention, when a blind person controls the operation of an operating system (Windows / Android / iOS / Linux, etc.) and an application used by a general person based on a haptic display and inputs information, The user must be able to move the input coordinates to the desired location in the current application and to know which display element the current input coordinate is located in.

The movement of the input coordinates may be performed by key input of a direction key or the like as described above. Meanwhile, in one embodiment of the present invention, which display element is currently positioned with input coordinates is shown as a tactile graphic.

That is, the output information generating step includes a tactile graphic generation step of generating tactile graphic data based not only on the display screen displayed in the focus area in the application but also on the input coordinates in the focus area, And is implemented based on graphic data.

The tactile graphic data includes data for embodying a tactile graphic element transformed according to a predetermined conversion rule and a graphic of the display screen or a graphical element formed by simplifying the graphic.

Here, the display element and the graphic element correspond to the GUI element shown in Fig. 16 described above. On the other hand, a graphic or graphic element of a display screen corresponds to a line, a circle, a triangle, etc. constituting a picture, not an icon or a folder.

In one embodiment of the present invention, the user can grasp where the current input coordinates are located on the tactile graphics. More preferably, the input coordinates correspond to respective positions of the tactile graphic element or the graphic element.

That is, when the first tactile graphic element, the second tactile graphic element, the first graphic element, the third tactile graphic element are output from the tactile interface device and the current input coordinate is located in the first tactile graphic element, When the moving direction key is pressed, the input coordinate is moved to the second tactile graphic element and the input coordinate moved by the user using the tactile angle can be recognized. Similarly, when the input coordinate is located in the second tactile graphic element, the user can move to the first graphic element if the user further presses the direction key to move to the right. That is, the input coordinates may be discontinuously input so as to correspond to respective positions of the tactile graphic element or graphic element. That is, when the user presses the direction key, it is preferable that the input coordinates in the application are moved but not moved continuously in units of one pixel, but move based on the tactile graphic element or graphic element displayed at the output of the tactile interface device .

In a preferred embodiment of the present invention, tactile graphics are output in real time so that the user can tactically recognize the current input coordinates. More specifically, when the input coordinate is placed on the tactile graphic element or the graphic element, the tactile graphic generation step may include generating the tactile graphic element based on the tactile graphic element or one or more tactile pixels of the detailed area of the tactile graphic corresponding to the graphic element Thereby generating tactile graphic data to be moved up and down. That is, in the case of a tactile graphic element or a graphic element in which input coordinates are placed, a tactile pixel of a corresponding area moves up and down, and the user senses a tactile pixel periodically changed even though he does not input, It is possible to grasp which icon, graphic element, and the like the input coordinates of which are located.

Further, in order to allow the visually impaired person to recognize detailed information of the tactile graphic element in more detail, the output information generating step may include a step of generating, based on the detailed information of the tactile graphic element corresponding to the input coordinate, And outputting information to the tactile interface device may include the braille information or the sound information.

Hereinafter, tactile graphic display screens of a method for controlling a tactile interface device according to an embodiment of the present invention will be described with reference to FIGS. 21 to 27.

The configuration of the apparatus of the tactile interface used in the method of controlling the tactile interface apparatus of the embodiment shown in Figs. 21 and 27 is as follows.

(Output section)

1. Tactile pixels consisting of multiple arrays: embedded in operating system folders and icons, graphic user interface (GUI) such as icons and application menus and buttons, graphic elements (images), and operating system folders, icons, application menus, or buttons Text Braille information is output in a tactile graphic form that can be recognized by the visually impaired.

(Input section)

2. Braille Keyboard / Direction Key-based Control / Input Interface: Based on the above-mentioned directional keys, braille keyboard, and shortcut keys, the visually impaired performs the same operation functions as operating the operating system and applications by using the keyboard.

(1) A direction key composed of up / down / left / right button that allows the visually impaired to control the operating system folders and icons or application menus and buttons in the same way as the direction keys of the general keyboard (focus shift for the components).

(2) Braille keyboards (including Run and Space keys; Run-Double-Click / Space-Click) and a number of hotkey buttons that allow operating systems and applications to enter and execute information the same as a regular keyboard.

Figure 21 illustrates a tactile graphics of a control / input interface in accordance with an embodiment of the present invention.

In Figure 21, the input coordinates are located in the first tactile graphic element (hatched area) in the second layer (interrupt layer). Here, the tactile graphic generation step generates tactile graphic data for moving the tactile pixels in the center (2,2) in real time in the detail area of the 3x3 tactile pixel representing the first tactile graphic element in real time, By outputting such tactile graphic data, the interface device can identify the current input coordinate or the component being finely focused.

That is, the tactile graphics of FIGS. 21A and 21B are periodically repeatedly switched so that the user can confirm which tactile graphic element the current input coordinate lies in.

On the other hand, the lowermost third layer can output braille information of the tactile graphic element in which the current input coordinate is placed.

22 illustrates a tactile graphics of a control / input interface in accordance with an embodiment of the present invention.

FIG. 22 corresponds to a tactile graphics after the user inputs the right direction key in the tactile graphic display state of FIG. In Fig. 22, the input coordinates are located in a second tactile graphic element (hatched area) in the second layer (interruption layer). Here, the tactile graphics generation step generates tactile graphic data for moving the tactile pixels in the center (2,2) in real time in the detailed area of the 3x3 tactile pixel representing the second tactile graphic element in real time, By outputting such tactile graphic data, the interface device can identify the current input coordinate or the component being finely focused.

That is, the tactile graphics of FIGS. 22A and 22B are cyclically repeatedly changed so that the user can check which tactile graphic element the current input coordinate is located after inputting the direction key

23 illustrates a tactile graphics of a control / input interface in accordance with an embodiment of the present invention.

FIG. 23B corresponds to a tactile graphic after the user inputs the execution key in the tactile graphic display state of FIG. FIG. 23B is a tactile graphic after a command such as execution of a second tactile graphic element in which the input coordinates are located or folder opening in FIG. 23A is executed. That is, FIG. 23B corresponds to the tactile graphic of the execution screen or detail folder for the second tactile graphic element in which the input coordinates are located in FIG. 23A.

On the other hand, in this embodiment, when the input coordinates are displayed immediately (for example, (2, 2) cells are not moved up and down) after a new tactile graphic element is executed and the user inputs an additional key such as a direction key The input coordinates can be displayed.

24 illustrates a tactile graphics of a control / input interface in accordance with an embodiment of the present invention.

24B corresponds to a tactile graphics after the user places the input coordinates in the third tactile graphic element and inputs the execution key in the tactile graphic display state shown in FIG. 24A. FIG. 24B corresponds to an execution screen of an application for displaying a graphic element (image) with a tactile graphic. Here, the tactile graphic of the inverted U character corresponds to the tactile graphic element designating the text input, and FIG. 24B corresponds to the tactile graphic of the state in which the text can be input in the currently given image.

25 illustrates a tactile graphics of a control / input interface in accordance with an embodiment of the present invention.

25 (A) and 25 (B) are repeatedly switched. Here, the tactile graphic of the inverted U character corresponds to a tactile graphic element designating a text input, and FIG. 25 (B) corresponds to a tactile graphic in a state where text can be input in a currently given image. In this case, the tactile graphic of the inverted U character is periodically moved up and down as a whole, and the user senses the movement of the tactile cell to find that the current input coordinate is located at the top of the square. Here, the user can perform a function such as a text memo for the graphic (image) currently expressed by the tactile graphics by inputting the braille information.

26 illustrates a tactile graphics of a control / input interface in accordance with an embodiment of the present invention.

26 (A), the user inputs text information using the point key. Thereafter, the tactile graphic is changed as shown in Fig. 26 (B). Here, an inverted U-shaped object (tactile graphic element) is temporarily or permanently stored in the corresponding image, and the presence of such textual information is represented by a tactile graphic element of the inverted U character.

27 illustrates a tactile graphics of a control / input interface in accordance with an embodiment of the present invention.

Fig. 27 (A) corresponds to the screen after the user depresses the downward direction key in the state of Fig. 26 (B). That is, when the user performs a text input for a place where the current input coordinate is located as shown in FIG. 26 (B), and then changes the input coordinate as shown in FIG. 27 (A) The same tactile graphics are expressed. Here, since the text is input, the tactile graphic element of the inverted U character remains as a tactile graphic.

Thereafter, as shown in FIG. 27 (B), a graphic (image) of a quadrangle in which the current input coordinate is located, periodically performs up and down movement of all the tactile cells representing the entire image. With such real-time tactile graphics, the user can recognize elements of the tactile graphics where the current input coordinates are located.

28 is a block diagram for explaining an example of the internal configuration of a computing device in an embodiment of the present invention.

28, computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, an input / output subsystem I / Osubsystem) 11400, a power circuit 11500, and a communication circuit 11600. At this time, the computing device 11000 may correspond to the user terminal A connected to the tactile interface device or the computing device B described above.

Memory 11200 can include, for example, a high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory or non-volatile memory. have. The memory 11200 may include software modules, a set of instructions, or various other data required for operation of the computing device 11000.

At this point, accessing memory 11200 from other components, such as processor 11100 or peripheral device interface 11300, may be controlled by processor 11100.

Peripheral device interface 11300 may couple the input and / or output peripheral devices of computing device 11000 to processor 11100 and memory 11200. The processor 11100 may execute a variety of functions and process data for the computing device 11000 by executing a software module or set of instructions stored in the memory 11200.

The input / output subsystem 11400 may couple various input / output peripherals to the peripheral interface 11300. For example, input / output subsystem 11400 may include a controller for coupling a peripheral, such as a monitor, keyboard, mouse, printer, or a touch screen or sensor, as needed, to peripheral interface 11300. According to another aspect, the input / output peripheral devices may be coupled to the peripheral device interface 11300 without going through the input / output subsystem 11400.

Power circuitry 11500 may provide power to all or a portion of the components of the terminal. For example, the power circuit 11500 may include one or more power supplies, such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, And may include any other components for creation, management, distribution.

Communication circuitry 11600 may enable communication with other computing devices using at least one external port.

Or as described above, communication circuitry 11600 may, if necessary, enable communications with other computing devices by sending and receiving RF signals, also known as electromagnetic signals, including RF circuitry.

28 is merely an example of the computing device 11000, and the computing device 11000 may be configured so that some of the components shown in FIG. 28 are omitted, or have additional components not shown in FIG. 28, Lt; RTI ID = 0.0 > components. ≪ / RTI > For example, in addition to the components illustrated in FIG. 18, a computing device for a mobile communication terminal may further include a touch screen, a sensor, and the like. The communication device 1160 may be connected to various communication methods (WiFi, 3G, LTE , Bluetooth, NFC, Zigbee, etc.). The components that may be included in computing device 11000 may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing or application specific integrated circuits.

The methods according to embodiments of the present invention may be implemented in the form of a program instruction that can be executed through various computing devices and recorded in a computer-readable medium. In particular, the program according to the present embodiment can be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied can be installed in a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transfer unit (not shown) for transferring the file according to a request from the user terminal.

As described above, according to the embodiment of the present invention, it is possible to visually impaired people to intuitively recognize tactile display items in real time.

According to the embodiment of the present invention, it is possible to achieve the effect that the visually impaired persons are able to efficiently input information based on the perceived information at the same time while tactually recognizing the information.

According to an embodiment of the present invention, a blind person recognizes the screen output information of a computer in the form of braille, voice, and tactile graphics, and promptly and intuitively inputs information thereon, It is possible to achieve an effect of making it usable.

According to the embodiments of the present invention, various functions of the OS and various application software can be easily expanded to use.

According to the embodiment of the present invention, it is possible to provide a blind person with a remarkably improved work execution speed and convenience in comparison with a conventional apparatus such as a screen reader.

According to the embodiment of the present invention, it is possible to greatly improve the ability of the visually impaired to use the computer, and to expand the market of computer hardware and software for the visually impaired.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For the sake of understanding, it is assumed that one processing device is used.

It will be appreciated by those skilled in the art that the processing apparatus may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (5)

CLAIMS What is claimed is: 1. A method of controlling a tactile interface device implemented as a computing device, the tactile interface device being connectable to and being capable of interacting with a user,
An input information generating step of generating input information to an application being executed in the computing device based on the input in the tactile interface device; And
And an output information generation step of generating output information to the tactile interface device based on an output of a focus area out of outputs from the application being executed in the computing device,
Wherein the output information includes data capable of implementing a tactile graphic composed of a plurality of two-dimensional tactile pixels,
Wherein the output information to the tactile interface device comprises:
A display screen displayed in the focus area, and information on an input coordinate in the focus area,
Wherein the output information generating step includes a tactile graphic converting step,
Wherein the tactile graphic conversion step comprises:
A focus partial area dividing step of dividing the focus area in the display screen output from the application into a focus partial area; And
And a tactile graphic element conversion step of converting the display element of each focus partial area into a tactile graphic element according to a predetermined conversion rule,
Wherein the tactile graphic includes a plurality of layers,
Wherein the focus partial area corresponds to the plurality of layers.
delete delete The method according to claim 1,
Wherein the output information generating step comprises:
A tactile graphic generation step of generating tactile graphic data based on a display screen displayed in a focus area in the application and input coordinates in the focus area,
Wherein the tactile graphic data includes data for embodying a tactile graphic element transformed according to a predetermined transformation rule and a graphic of the display screen or a graphical element formed by simplifying the graphic,
Wherein the input coordinates correspond to respective positions of the tactile graphic element or the graphic element,
The tactile graphics generation step may include moving the tactile graphic element or at least one of the tactile pixels of the tactile graphics corresponding to the graphic element up and down when the input coordinate is laid on the tactile graphic element or the graphic element And generating tactile graphics data.
22. A computer-readable medium,
The computer-readable medium storing instructions that cause a computing device to perform the steps of:
An input information generating step of generating input information to an application being executed in a computing device based on an input in a tactile interface device capable of interacting with a user; And
And an output information generation step of generating output information to the tactile interface device based on an output of a focus area out of outputs from the application being executed in the computing device,
Wherein the output information includes data capable of implementing a tactile graphic composed of a plurality of two-dimensional tactile pixels,
Wherein the output information to the tactile interface device comprises:
A display screen displayed in the focus area, and information on an input coordinate in the focus area,
Wherein the output information generating step includes a tactile graphic converting step,
Wherein the tactile graphic conversion step comprises:
A focus partial area dividing step of dividing the focus area in the display screen output from the application into a focus partial area; And
And a tactile graphic element conversion step of converting the display element of each focus partial area into a tactile graphic element according to a predetermined conversion rule,
Wherein the tactile graphic includes a plurality of layers,
And the focus partial area corresponds to the plurality of layers.

Images