KR20210096945A - Cognitive Assistance System and Method for Visually Impaired - Google Patents

Abstract

The present invention relates to a cognitive assistance system and a method for the visually impaired, more specifically, to the cognitive assistance system and the method for the visually impaired that photograph an information target that a visually impaired person wants to perceive through a smart glass, recognize and analyze the photographed image based on an artificial intelligence technology to provide the analyzed information through a braille pad that can be tactilely recognized by the visually impaired and in case that a photographed image is book information comprising a book text, the contents of the book can be provided in Braille.

Description

Cognitive Assistance System and Method for Visually Impaired

The present invention relates to a cognitive assistance system and method for the visually impaired, and more particularly, information obtained by photographing an information object to be recognized by the visually impaired through smart glasses and recognizing and analyzing an image captured based on artificial intelligence technology. can be provided through a braille pad that the visually impaired can tactilely recognize, and if the photographed image is book information containing book text, Auxiliary systems and methods.

Recently, the wearable market is accelerating due to the expansion of the wearable market due to the abundance of Internet of Things and network resources due to the research of many technologies to ensure the free walking of the visually impaired and the transportation weak. Devices for free walking include smart wands and smart glasses-related technologies using video cameras, ultrasonic sensors, and acceleration sensors. Technology that recognizes symbol information and provides information about the surrounding environment, and technologies such as machine-to-machine (M2M)-based intelligent walking assistance systems are being researched and developed to enable the visually impaired to walk more safely and conveniently. there is a trend However, most of these technologies require smart devices that are difficult for visually impaired people to use, and most of the information provided is provided to the user through a method of outputting a voice. There is an increasing demand for guided walking, which intuitively provides detailed information necessary for walking for the visually impaired. Although these technologies are being developed, blind people are still reluctant to go out alone in an environment unfriendly to the visually impaired, and the current situation is that basic life is not easy without someone's help.

In addition, there are cases where the visually impaired want to visit a library or the like to read books or papers, and in places such as a library for the visually impaired, Braille books are provided to increase convenience. However, there are many cases of producing and distributing these Braille books only for a limited number of books. It is difficult to access.

In order to solve this problem, information about the surrounding environment of the visually impaired can be filmed and transmitted, and the analyzed information is provided, and general smart devices (smartphones, smart pads), etc., can be easily understood by the visually impaired in Braille. By translating not only Braille books but also general books into easy-to-understand Braille through the Braille pad, the visually impaired can acquire, recognize, and utilize various image information, their daily life and educational environment. It is necessary to provide services that can improve the daily life of the visually impaired as differentiated application technologies for , safety, and quality of life, but there is no such conventional technology.

The present invention recognizes and analyzes an image captured by photographing an information object to be recognized by the visually impaired through smart glasses based on artificial intelligence technology, and through a braille pad that enables the visually impaired to tactilely recognize the analyzed information. The purpose of the present invention is to provide a cognitive assistance system and method for the visually impaired that can be provided, and if the photographed image is book information including book text, the book content can be translated into Braille and provided.

In order to solve the above problems, in the present invention, there is provided a cognitive assistance system for the visually impaired, comprising: a smart glass including a photographing module and a communication module, and transmitting an image photographed by the photographing module to a user terminal; a braille pad capable of providing tactile information to a user by performing braille output based on data received from the user terminal; and a server that transmits and receives data to and from the user terminal, wherein the server detects an object in the image, and accordingly, braille output information to be output from the braille pad or the user terminal, braille pad, or smart glass an object detection unit generating audio output information to be output from any one of the following; And when the object detection unit detects a book image from the image, a book reader unit for extracting the book text information and the book image information of the book to be output in Braille from the braille pad; Containing, a cognitive assistance system for the visually impaired to provide.

In one embodiment of the present invention, the object detection unit, determining the number of objects detected in the image; and when the number of detected objects is two or more, recognizing the detected object through a learned artificial neural network, and generating the voice output information for the recognized object.

In one embodiment of the present invention, the object detection unit, determining the number of objects detected in the image; and when the number of the detected objects is two or more, the detected object is recognized through a learned artificial neural network, and coordinate information for the two or more recognized objects and the mapped based on a preset criterion for the two or more recognized objects are mapped. generating braille output information including icon information; may be performed.

In one embodiment of the present invention, the object detection unit, determining the number of objects detected in the image; recognizing the object through a learned artificial neural network when the detected object is one; It is possible to perform a step of generating braille output information in which the shape of an object is displayed based on the object information on the recognized object and the feature points extracted from the recognized object.

In one embodiment of the present invention, the object detection unit, determining the number of objects detected in the image; recognizing the object through a learned artificial neural network when the detected object is one; When the shape of the recognized object is not all included in the image, generating audio output information for the recognized object and guide audio output information for adjusting the position of the smart glass; may be performed.

In one embodiment of the present invention, the object detection unit, determining the number of objects detected in the image; recognizing the object through a learned artificial neural network when the detected object is one; and when the corresponding object is not recognized through the artificial neural network, generating guide voice output information for position adjustment of the smart glass.

In an embodiment of the present invention, the book reader unit, an image pre-processing step of pre-processing by changing one or more of the gradient and color of the book image detected in the image according to a preset standard; a text image area deriving step of deriving at least one text area and at least one image area based on the image pre-processed in the image pre-processing step; A text recognition step of determining an output order of the derived one or more text areas, recognizing texts included in the one or more text areas according to the output order, and deriving the book text information to be output in Braille from the Braille pad; can be done

In an embodiment of the present invention, one or more text areas and one or more image areas are extracted from the book image detected in the image, and the one or more texts are extracted from the screen on the braille pad provided by the book reader. Text included in the region may be sequentially arranged and displayed, and a list of one or more book image information detected by the book reader may be provided according to a user input.

In an embodiment of the present invention, the book reader unit further performs an image recognition step of recognizing the book image information extracted from the derived one or more image areas, and in the braille pad provided by the book reader unit. The screen may provide a list of the one or more book image information according to a user's input, including image information recognized in the image recognition step.

In order to solve the above problems, in one embodiment of the present invention, as a method of providing a cognitive assistance service for the visually impaired implemented in a cognitive assistance system for the visually impaired, the cognitive assistance system for the visually impaired includes a photographing module and a communication module, , smart glasses for transmitting the image photographed by the photographing module to the user terminal; a braille pad capable of providing tactile information to a user by performing braille output based on data received from the user terminal; and a server that transmits and receives data to and from the user terminal, wherein the server detects an object in the image, and accordingly, braille output information to be output from the braille pad or the user terminal, braille pad, or smart glass an object detection step of generating audio output information to be output in any one; and a book reader step of extracting book text information and book image information of the book to be output in braille from the braille pad when the object detection step detects a book image from the image; provide a method for providing

According to an embodiment of the present invention, by analyzing images acquired through smart glasses and providing information about the surrounding environment as braille information that the visually impaired can understand more accurately, the daily life of the visually impaired who cannot see the surroundings directly It can exert the effect of improving the discomfort in

According to an embodiment of the present invention, by analyzing the image acquired through smart glass and providing information about the surrounding environment as braille output information including a mapped icon that the visually impaired can more intuitively understand, The blind people who cannot see can more easily recognize the surrounding environment through the Braille information.

According to an embodiment of the present invention, by analyzing an image acquired through smart glasses and providing information about the surrounding environment or information about objects as voice information, inconvenience in the life of the visually impaired who cannot see the surroundings directly It can have the effect of improving the performance.

According to an embodiment of the present invention, the text information and graphic information of the corresponding book are extracted from the book image information obtained through the smart glass, the text information is provided in transliterated Braille, and the corresponding graphic information is provided in a list so that the user can By providing a way to directly touch the shape with Braille information, it is possible to effectively provide cognitive assistance for the visually impaired.

According to one embodiment of the present invention, even if the corresponding book image information is not set differently, by sequentially arranging and translating the text according to the order of the text, the user intuitively provides information such as general books or pages through transliterated Braille. It can have the effect of understanding the content.

According to an embodiment of the present invention, by not only translating and providing information on objects in the surrounding environment in braille with a linguistic meaning, but also displaying Braille information according to the shape of the object, the user can use the An effect that can be confirmed more intuitively with respect to the form can be exhibited.

1 schematically shows the overall form of a cognitive assistance system for the visually impaired according to an embodiment of the present invention.
2 schematically illustrates an operation of an object detection unit according to an embodiment of the present invention.
3 schematically illustrates mapped icon information in a process of the object detection unit generating braille output information according to an embodiment of the present invention.
4 schematically illustrates braille output information generated according to icon information mapped by an object detection unit according to an embodiment of the present invention.
5 schematically illustrates braille output information generated according to icon information mapped by an object detection unit according to an embodiment of the present invention.
6 schematically shows a screen in a user terminal and a braille pad provided by an object detecting unit according to an embodiment of the present invention.
7 schematically illustrates an image captured by a user through smart glasses according to an embodiment of the present invention.
8 schematically illustrates the execution steps of a book reader according to an embodiment of the present invention.
9 schematically shows a screen in a user terminal and a braille pad provided by a book reader according to an embodiment of the present invention.
10 schematically shows the form of book image information according to the operation of the book reader according to an embodiment of the present invention.
11 schematically shows a screen in a user terminal and a braille pad provided by a book reader according to an embodiment of the present invention.
12 schematically shows a screen in a user terminal and a braille pad provided by a book reader according to an embodiment of the present invention.
13 exemplarily shows a computing device according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that other components may be further included rather than excluding other components unless otherwise stated.

Also, terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware. Meanwhile, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, '~' denotes components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

The "user terminal" referred to below may be implemented as a computer or portable terminal that can access a server or other terminal through a network. Here, the computer includes, for example, a laptop, a desktop, and a laptop equipped with a web browser (WEB Browser), and the portable terminal is, for example, a wireless communication device that ensures portability and mobility. , PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code) Division Multiple Access)-2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (Wibro) terminals, and the like may include all types of handheld-based wireless communication devices. In addition, "network" refers to a wired network such as a local area network (LAN), a wide area network (WAN), or a value added network (VAN), or a mobile radio communication network or satellite. It may be implemented as any kind of wireless network, such as a communication network.

1 schematically shows the overall form of a cognitive assistance system for the visually impaired according to an embodiment of the present invention.

The cognitive assistance system for the visually impaired of the present invention includes a smart glass 2000 , a user terminal 3000 , a server 1000 , and a braille pad 4000 . The user terminal 3000 or the smart glass 2000 of FIG. 1 may correspond to the aforementioned user terminal, and the server 1000 corresponds to a computing device including one or more processors and one or more memories.

The user terminal 3000 communicates with the smart glass 2000, the server 1000 and the braille pad 4000, transmits the image photographed by the smart glass 2000 to the server 1000, and receives the image. The server 1000 receives the information on the image analyzed based on the artificial intelligence technology, and transmits it to the smart glass 2000 or the braille pad 4000 so that the voice output information and the braille output information can be output.

The smart glass 2000 includes a photographing module, speakers 2200 and 2200 , a communication module 2300 , and a button unit 2400 . The smart glass 2000 is the external environment according to the user's point of view by the built-in photographing module 2100 according to the user's input through the button unit 2400 provided by the user wearing the glasses in the same form as wearing glasses; You can take pictures of objects, people, etc. Also, the photographed image is transmitted to the user terminal 3000 by the communication module 2300 . The smart glass 2000 may communicate with the user terminal 3000 including the communication module 2300 . The speaker 2200 may output audio output information received from the user terminal 3000 . In addition, the button unit 2400, according to the user's input, executes the photographing through the photographing module so that the external environment can be photographed.

On the other hand, the server 1000, the object detection unit 1100 for detecting an object in the image received from the smart glass 2000, and when the object detection unit 1100 detects a book image, the book text information and and a book reader unit 1200 for extracting book image information.

The object detection unit 1100 includes: an object detection unit 1110 for detecting an object in the image received from the smart glass 2000; And according to the object extracted from the image, the braille output information to be output from the braille pad 4000 or the voice output information to be output from any one of the user terminal 3000, the braille pad 4000, or the smart glass 2000 and an object information generating unit 1120 to generate it. Preferably, the braille output information may be braille output information in which information on the detected object is transliterated, or braille output information in which the shape of the detected object is expressed.

The book reader unit 1200 includes an image pre-processing unit 1210 for pre-processing a book image, a text image area deriving unit 1220 for deriving a text area and an image area from the pre-processed image, and text included in the derived text area. and a text recognition unit 1230 for deriving book text information output from the braille pad 4000 and an image recognition unit 1240 for recognizing book image information;

The image pre-processing unit 1210 pre-processes by changing one or more of the gradient and color of the book image extracted from the object detected in the image according to a preset criterion. The main users of the present invention may be those who have a visual impairment or have significantly reduced visual acuity due to reasons such as farsightedness, myopia, astigmatism, presbyopia of the elderly, etc. may not be included in the image or may be photographed in an inclined form. Therefore, in order to extract accurate book text information from the book image captured according to the user's input, the image pre-processing unit 1210 may change the inclination of the book image according to a preset standard when the photographed image is tilted. there is. In addition, in the present invention, the image area and the text area can be distinguished by analyzing the image based on the artificial intelligence technology and binarizing the image into black and white in order to more reliably distinguish the image area and the text area.

The text image area deriving unit 1220 derives one or more text areas and one or more image areas from the book image based on the image pre-processed by the image pre-processing unit 1210 . The text image area deriving unit 1220 may derive a text area and an image area from the pre-processed image, and the derived text area and image area may be converted into signals outputable by the braille pad 4000 and output.

The text recognition unit 1230 determines an output order of one or more text areas derived from the text image area extractor 1220, recognizes texts included in the one or more text areas according to the output order, and recognizes the Braille The pad 4000 derives the book text information to be output in Braille.

The image recognition unit 1240 recognizes one or more image areas derived from the text image area deriving unit 1220 through the learned artificial neural network, and derives the book image information to be output in Braille from the braille pad 4000. .

Meanwhile, the braille pad 4000 may perform braille output based on data received from the user terminal 3000 and provide tactile information to the user. It may include any interface device capable of providing tactile graphics, a tactile interface device capable of interacting with a user, and a computing device capable of outputting tactile graphics.

Hereinafter, a detailed operation of the server 1000 will be described in more detail.

2 schematically illustrates the operation of the object detection unit 1100 according to an embodiment of the present invention.

As described above, the object detecting unit 1100 of the present invention includes an object detecting unit 1110 and an object information generating unit 1120 . The object detection unit 1110 detects an object from the image received from the smart glass 2000 . FIG. 2A shows an example of an image received from the smart glass 2000. As shown in FIG. The user can input the smart glasses 2000 through the button unit 2400 to take pictures while walking while wearing the smart glasses 2000 in an external environment to obtain information about their surroundings. The photographed image may be transmitted to the user terminal 3000 by the communication module 2300 and transmitted to the server 1000 . The object detection unit 1110 of the object detection unit 1100 of the server 1000 receives the image transmitted from the smart glass 2000 in this way and detects an object in the image through the learned artificial neural network, and detects the detected object. can recognize In the image received as shown in (b) of FIG. 2 by the operation of the object detection unit 1110, a store on the street where the user walked, a toilet, a car on the road, a tree on the roadside, several people walking on the road, etc. An object on the image may be detected. In this way, the object detection unit 1110 of the object detection unit 1100 detects an object from the image. Thereafter, the object information generating unit 1120 of the object detecting unit 1100 may output braille output information to be output from the braille pad 4000 or any one of the user terminal 3000 , the braille pad 4000 , or the smart glass 2000 . Generates audio output information to be output in The object detection unit 1100 and the book reader unit 1200 of the present invention detect an object according to each operation including the learned artificial neural network model, and analyze information on the object according to the number of the object to provide the user with can provide

Specifically, the object detection unit 1100 of the present invention includes an object information generation unit 1120, and the object information generation unit 1120 of the object detection unit 1100 includes the object detection unit 1110 in the image. determining the number of detected objects (S110); and when the number of the detected objects is two or more, recognizing the detected object through a learned artificial neural network and generating the voice output information for the recognized object (S120); and, When the number of the detected objects is two or more, the detected object is recognized through a learned artificial neural network, and coordinate information for the two or more recognized objects and icon information mapped according to a preset criterion for the two or more objects are included. A step of generating the braille output information (S130); is performed.

Specifically, in step S110, the object detection unit 1110 of the object detection unit 1100 determines the number of objects detected in the image.

In step S120, when a plurality of objects are detected in the image, the object detection unit 1100 generates audio output information for the object, and any one of the user terminal 3000, the braille pad 4000, or the smart glass 2000 output from one For example, the object detection unit 1100 that has received the image as shown in FIG. 2 may say, “A bus is running on the road, there are three people and two trees on the street across the road, and there are shops and toilets. The voice output information for the object detected in the image is generated as shown in the figure, "This may be transmitted and outputted to any one of the smart terminal, the braille pad 4000, or the smart glass 2000.

In step S130, braille output information including coordinate information of the two or more objects generated by the object detection unit 1100 and icon information mapped according to a preset criterion for the two or more objects is generated. This will be described in more detail in the description of FIGS. 3 to 5 which will be described later.

3 schematically shows mapped icon information in a process in which the object detection unit 1100 generates braille output information according to an embodiment of the present invention, and FIGS. 4 and 5 are diagrams according to an embodiment of the present invention. The braille output information generated according to the icon information mapped by the object detection unit 1100 is schematically illustrated.

The object detection unit 1100, when two or more objects are detected from the image as in the above description, recognizes the objects detected through the learned artificial neural network, and coordinates information and the recognized two or more objects. A step (S130) of generating braille output information including icon information mapped according to a preset criterion for two or more objects is performed. 3 schematically shows such mapped icon information. As shown in FIG. 3, the object information generating unit 1120 of the object detecting unit 1100 may display an object detected in an image through mapped icon information according to a preset criterion in the form of a mapped icon in the braille output information. there is. Although some mapped icons are shown in FIG. 3, the mapped icon information included in the braille output information output by the object detection unit 1100 of the present invention is larger than the number shown in FIG. may be stored. As described above, the object detecting unit 1100 generates braille output information based on coordinate information of an icon and an object mapped according to a preset criterion. 4 and 5 show braille output information generated according to the coordinate information and the mapped icon information. Figure 4 (a) is an image received from the smart glass 2000, Figure 4 (b) is a braille output including coordinate information and mapped icon information of two or more objects according to the image of Figure 4 (a) A screen on the braille pad 4000 on which information is output is shown. As shown in (b) of Figure 4, shops, toilets, buses on the driveway, trees on the roadside, a number of people walking on the road, etc. shown in the image of Figure 4 (a) are mapped to the object of Figure 3 According to the icon mapped to each object in the table representing the icon, it is converted into coordinate information in the braille pad 4000 based on the coordinate information of each object, and the coordinates of the object are mapped to the coordinates in the braille pad 4000 Icons are displayed in Braille. In addition, the object detection unit 1100 derives an expected distance between the user and the two or more objects based on the coordinate information of the two or more objects, and the mapped icon based on the derived expected distance between the two or more objects. The braille output information is generated by changing the size of ? according to a preset standard. As shown in (a) of FIG. 4 , it is shown that the person in front of the tree from the user's point of view is closer than other people from the user's point of view. Accordingly, as shown in (b) of FIG. 4 , the screen of the braille pad 4000 provided by the object detecting unit 1100 is an icon representing the person shown in front when compared with the icon representing the two people shown in the back. is displayed in a larger size.

Preferably, the object detection unit 1100 derives icon coordinate information in the braille pad 4000 on which the mapped icon is to be displayed according to a preset criterion based on the coordinate information of the object in the image, It generates braille output information in which icon information mapped to the icon coordinate information is displayed.

Preferably, the object detection unit 1100 derives the predicted coordinates of the user based on the coordinate information on the two or more objects, and derives the predicted distances between the user and the two or more objects according to the predicted coordinates of the user, , to generate the braille output information by changing the size of the mapped icon according to a preset standard based on the derived expected distance to the object.

5 also shows a screen on the braille pad 4000 on which braille output information including mapped icon information is similarly output. FIG. 5A shows an image containing a landscape of a park, and the object detection unit 1100 receiving the image detects objects such as trees, animals, people, and benches from the image and maps them to the corresponding objects. By generating the braille output information including the selected icon, the generated braille output information may be output on the screen of the braille pad 4000 as shown in FIG. 5( b ).

In this way, the object detection unit 1100 generates and provides braille output information including the mapped icon, so that visually impaired people who cannot directly see the surroundings can more easily recognize the surrounding environment through the braille information. can have an effect.

6 schematically shows screens of the user terminal 3000 and the braille pad 4000 provided by the object detection unit 1100 according to an embodiment of the present invention.

The object detection unit 1100 of the present invention includes the steps of determining the number of objects detected in the image (S110); When the detected object is one, recognizing the object through the learned artificial neural network (S140); and generating braille output information indicating the shape of the object based on the object information on the recognized object and the feature points extracted from the recognized object.

Specifically, in step S110, the object information generating unit 1120 of the object detecting unit 1100 determines the number of objects detected in the image.

In step S140, when the detected object is one, the object is recognized through the learned artificial neural network. The object detection unit 1100 or the object information generation unit 1120 of the object detection unit 1100 may include a learned artificial neural network model, and provide identification information and identification information for one object detected through the artificial neural network. It is possible to derive object information including voice output information according to the

In step S150, braille output information indicating the shape of the object is generated based on the object information on the recognized object and the feature points extracted from the recognized object.

Fig. 6 (a) shows a screen showing an object detected from an image by the object detection unit 1100 provided by the object detection unit 1100 displayed on the user terminal 3000, and Fig. 6 (b) is The screen on the braille pad 4000 provided by the object detection unit 1100 is shown. When there is one detected object as shown in (a) of FIG. 6 , the object detection unit 1100 performs object information on the recognized object as shown in (b) of FIG. 6 and a feature point extracted from the recognized object. It creates braille output information that displays the shape of an object. In this way, by generating the shape of the object as Braille output information based on the characteristic points of the recognized object, the user can more intuitively check the shape of the object through Braille, and it is possible to learn the shape. .

7 schematically illustrates an image captured by a user through the smart glasses 2000 according to an embodiment of the present invention.

As described above, the cognitive assistance system for the visually impaired of the present invention efficiently delivers image information, which is essential information throughout education and daily life, to the visually impaired, who are mainly recognizing and utilizing only limited information centered on Braille or voice. It has a purpose to be recognized and used. As such, the cognitive assistance system for the visually impaired of the present invention shows an example of a photographed image when the visually impaired take an image to recognize a certain surrounding environment or object because the main user is a visually impaired person in Fig. 7 (a) and (b). For example, a user who does not recognize the shape of the cup may photograph the cup using the smart glasses 2000 to obtain information about the cup. At this time, when an image as shown in (a) of FIG. 7 is taken, the object detection unit 1100 of the present invention may include: determining the number of objects detected in the image (S110); When the detected object is one, recognizing the object through the learned artificial neural network (S160); If the image does not include all the shapes of the recognized object, generating audio output information for the recognized object and guide audio output information for position adjustment of the smart glass 2000 (S170); is performed. .

In step S110, the number of objects detected in the image is determined.

In step S160, when the detected object is one, the object is recognized through the learned artificial neural network. The trained artificial neural network can recognize the object even if the provided image is not complete. As shown in (a) of FIG. 7 , even if the shape of the detected object is a half-photographed image, it can be recognized that the object is a cup.

In step S170, the object detection unit 1100, which has recognized the object through the execution of step S160, as shown in FIG. Voice output information for an object and guide voice output information for position adjustment of the smart glass 2000 are generated. The object information generation unit 1120 of the object detection unit 1100, as described above in the description of FIG. 6, displays the shape of the object based on the object information on the recognized object and the feature points extracted from the recognized object. Information may be generated and provided to the user, which may require a complete image of the detected object. Accordingly, the object detection unit 1100 determines the position of the smart glass 2000 in which a complete image can be taken based on the received image and generates guide voice output information for the position adjustment of the smart glass 2000. there is. In the case of an image as shown in (a) of FIG. 7 , for example, voice output information and guide voice output information for the corresponding object are generated, such as “Move down and shoot again”, and send to the user terminal 3000 . and to be output from the user terminal 3000 , the braille pad 4000 , or the smart glass 2000 .

On the other hand, when an image as shown in (b) of FIG. 7 is taken and the object detected from the image is not recognized, the object detection unit 1100 and the object detection unit 1100 detect the number of objects detected in the image. determining (S110); When the detected object is one, recognizing the object through the learned artificial neural network (S180); and when the corresponding object is not recognized through the artificial neural network, generating guide voice output information for the position adjustment of the smart glass 2000 (S190).

In step S110, the number of objects detected in the image is determined.

In step S180, if there is one detected object, the object is recognized through the learned artificial neural network. In the case of (b) of FIG. 7 , although an object is detected in the image, it may be inappropriate to recognize the object according to the learned artificial neural network. As such, when the corresponding object is not recognized, the object detecting unit 1100 may perform step S190.

In step S190, when the object is not recognized through the artificial neural network, guide voice output information for the position adjustment of the smart glass 2000 is generated. When the object extracted from the image is unclear as shown in FIG. 7B and the object is not recognized, the object is recognized, and braille output information indicating the shape of the recognized object is generated and provided to the user. The detection unit 1100 may determine the position of the smart glass 2000 at which a complete image can be captured based on the received image and generate guide voice output information for adjusting the position of the smart glass 2000 . In the case of an image as shown in (b) of FIG. 7, for example, guide audio output information for the object is generated in order to take a complete image of the object, for example, “Please move to the right and down and shoot again”. to be transmitted to the user terminal 3000 , and output from the user terminal 3000 , the braille pad 4000 , or the smart glass 2000 .

In this way, the object detection unit 1100 recognizes the object detected from the image, generates audio output information for the recognized object and braille output information indicating the shape of the object, and provides the received image to the user. When it is difficult to recognize the detected object itself or to display the shape of the object, it provides guide voice output information to the user and receives an image that can be recognized through the learned artificial neural network again to provide more accurate object information. It can exert an effect that can be provided to users.

FIG. 8 schematically illustrates the execution steps of the book reader unit 1200 according to an embodiment of the present invention.

The cognitive assistance system for the blind of the present invention detects an object from an image captured by the user through the smart glasses 2000 and provides information on the external environment, objects, and people around the user according to the user's point of view, When a book image obtained by photographing the contents of a book that the user wants to read is detected, book text information and book image information of the corresponding book may be extracted and provided from the corresponding book image. At this time, when the object detected from the image is, for example, an unopened book, the text information corresponding to the content of the book cannot be recognized, so the book itself is detected as an object and information that the object is a book Object information including the may be provided to the user. The book image analyzed by the book reader unit 1200, which will be described later, preferably means an image of a photographed book including text content that can identify the content of the book.

As shown in Figure 8, the book reader unit 1200, the image pre-processing step (S210) of pre-processing by changing one or more of the inclination and color of the book image detected in the image according to a preset standard; a text image area deriving step (S220) of deriving one or more text areas and one or more image areas based on the image pre-processed in the image pre-processing step (S210); and determining an output order of the derived one or more text areas, recognizing texts included in the one or more text areas according to the output order, and deriving the book text information to be output in braille on the braille pad 4000 A recognition step (S230); is performed.

Specifically, in step S210, the image pre-processing unit 1210 of the book reader unit 1200 pre-processes by changing one or more of the inclination and color of the book image detected in the image according to a preset criterion. The book image photographed according to the input of the blind user, who is the main user, may be photographed in an inclined form. Therefore, in order to extract accurate book text information from the book image captured according to the user's input, the image pre-processing unit 1210 may change the inclination of the book image according to a preset standard when the photographed image is inclined. there is. In addition, by binarizing the image to black and white in order to extract the corresponding book text information, image preprocessing can be performed to extract the book text information of the corresponding book.

In step S220, the text image area deriving unit 1220 of the book reader 1200 derives one or more text areas and one or more image areas based on the image pre-processed in the image pre-processing step S210. A text area and an image area are derived from the pre-processed image, and the derived text area and image area are converted into signals that can be output by the braille pad 4000 and are output as braille, which is easy for the user to recognize.

In step S230, the text recognition unit 1230 of the book reader unit 1200 determines an output order of one or more text areas derived from the text image area extracting unit 1220, and according to the output order, the 1 By recognizing the text included in the above text area, the book text information to be output in braille is derived from the braille pad 4000 . The text recognition unit 1230 derives book text information by recognizing the text included in the text area when one text area is derived from the corresponding book image, but when the derived text area is 1 or more, According to a preset standard, the output order according to the contents of the book is determined, and texts included in one or more text areas are recognized according to the output order, so that the contents of the book can be sequentially translated and provided.

In this way, the book reader unit 1200 extracts and provides the book text information and the book image information of the book to be output in braille from the braille pad 4000 when the object detection unit 1100 detects a book image, The user can exert the effect of reading not only limited Braille books but also general books in Braille that is easy to understand.

9 schematically shows screens of the user terminal 3000 and the braille pad 4000 provided by the book reader unit 1200 according to an embodiment of the present invention.

Specifically, FIG. 9( a ) shows an example of a book image received from the smart glass 2000 . As shown in (a) of FIG. 9 , the book image may be divided into an image area L1 and a text area L2 of the corresponding book. The text image area deriving unit 1220 of the book reader unit 1200 explains the part containing the picture of a boa constrictor expressed like a hat in the book image as shown in FIG. 9(a) as the image area L1. A paragraph containing the text to be read is drawn into the text area (L2). In the thus derived text area L2, as shown in FIG. 9(b) , the text included in the corresponding text area L2 is displayed on the screen of the user terminal 3000 . FIG. 9C shows a screen of the braille pad 4000 on which book text information derived from the text area L2 by the text recognition unit 1230 is displayed. As shown in (c) of FIG. 9 , the book text information derived by the text recognition unit 1230 is outputted in braille on the braille pad 4000 and displayed.

As described above, by translating and providing the text of the corresponding book included in the book image in Braille, which is easy for the visually impaired to understand, the user is not limited to the Braille book, and it is possible to exert the effect of reading various types of books.

10 schematically shows the form of book image information according to the operation of the book reader unit 1200 according to an embodiment of the present invention.

Specifically, (a) and (b) of Figure 10 shows an example of the book image received from the smart glass (2000). Although the text area L2 of FIG. 9(a) is one, the book image shown in FIGS. 10(a) and (b) may include a plurality of paragraphs and may be divided into a plurality of text areas L2. In some cases, the book image as shown in (a) of FIG. 10 may be pre-processed by the image pre-processing unit 1210, and then, as shown in FIG. 10 (b) by the text image area deriving unit 1220. Similarly, one or more text areas L2 and one or more image areas L1 are derived. When one or more text areas L2 are derived as described above, the text recognition unit 1230 determines an output order of one or more text areas. In (b) of FIG. 10, preferably, the output order is, for example, from left to right, and when one or more text areas exist on the same vertical line, in the order from top to bottom, etc. According to this, the output order of the text area may be determined.

10( c ) shows a screen on which text information included in one or more text areas L1 is recognized and recognized text information is displayed on the user terminal 3000 according to the output order determined by the text recognition unit 1230 . and FIG. 10( d ) shows a screen on which book text information generated by the text recognition unit 1230 is displayed on the braille pad 4000 . The text recognition unit 1230 determines the output order of each text area L2 as shown in FIG. In the user terminal 3000, the transmitted book text information is displayed according to the output order as shown in FIG. 10(c). The user terminal 3000 may transmit the received book text information to the braille pad 4000, and the book text information may be displayed on the braille pad 4000 according to the output order as shown in FIG. . Preferably, the screen of the braille pad 4000 provided by the book reader unit 1200 is obtained by extracting one or more text areas and one or more image areas from the book image detected in the image, Texts included in the text area are sequentially arranged and displayed.

In this way, the book reader unit 1200 derives the text area of the received book image and derives and provides the book text information to be output in Braille according to the output order, thereby allowing the user to select more types of books than Braille books. You can directly read and interact with it, and you can have the effect of getting a variety of information.

11 schematically shows screens of the user terminal 3000 and the braille pad 4000 provided by the book reader 1200 according to an embodiment of the present invention.

Specifically, FIGS. 11 (a) and (c) show a screen of the user terminal 3000 provided by the book reader unit 1200, and FIGS. 11 (b) and (d) show the book. A screen of the braille pad 4000 provided by the reader unit 1200 is schematically shown. The screen on the braille pad 4000 provided by the book reader 1200 provides a list of one or more book image information detected by the book reader 1200 according to a user input. The text area L2 and the image area L1 are detected by the text image area extracting unit 1220 of the book reader unit 1200, and the image area is input by the user (eg, a braille pad 4000). A list of one or more book image information detected by the book reader unit 1200 may be provided according to a shortcut key of . As shown in (a) of FIG. 10, there is one book image information detected in the image area L1 detected in the book image, and FIG. 11 shows a screen provided with a list of such book image information. 11 (a) is a screen on the user terminal 3000 provided by the book reader unit 1200, and a list of book image information defined as “image 01” is provided, and FIG. 11 (b) is a picture of a book As a screen on the braille pad 4000 provided by the reader unit 1200, it is shown that the list of image 01 is transliterated and displayed in braille.

Meanwhile, the book reader unit 1200 of the present invention further performs an image recognition step of recognizing the book image information extracted from the derived one or more image areas, and the book reader unit 1200 provides the The screen of the braille pad 4000 provides a list of the one or more book image information according to a user's input, including the image information recognized in the image recognition step. The book reader unit 1200 further includes an image recognition unit 1240, and the image recognition unit 1240 recognizes the book image information extracted from the derived one or more image areas through a learned artificial neural network. Perform the image recognition step. Accordingly, the screen of the braille pad 4000 provided by the book reader unit 1200 is not “image 01” as shown in FIG. It is shown that a list of book image information is provided. 11 (d) also shows that the Braille corresponding to the fire extinguisher is displayed in a dotted pattern. As such, the book reader unit 1200 may provide the book text content and book image information through Braille by dividing the text area and the image area of the book image, and the book image information may be provided through a list according to the user's input. By providing and providing the recognition information of the corresponding image, the user can exert the effect of more accurately acquiring the contents of the book.

12 schematically shows screens of the user terminal 3000 and the braille pad 4000 provided by the book reader 1200 according to an embodiment of the present invention.

Fig. 12 (a) shows a screen of the user terminal 3000 provided by the book reader unit 1200, and Fig. 12 (b) is a braille pad 4000 provided by the book reader unit 1200. ) schematically shows the screen. When a user provided with a screen as shown in FIG. 11 makes a user input (eg, pressing a button on the braille pad 4000) to check the book image information, the book reader unit 1200 receives the input Information on one book image may be transmitted to the user terminal 3000 and provided on the screen of the user terminal 3000 as shown in FIG. 12( a ). Also, the book reader unit 1200 may generate braille output information in which the form of the selected and input book image is displayed, as shown in FIG. 12( b ), and transmit it to the user terminal 3000 . The transmitted braille output information is displayed on the screen of the braille pad 4000 in which the form of the corresponding book image information is displayed as shown in FIG. 12(b). Preferably, the book reader unit 1200 generates braille output information in which the form of the book image information is displayed based on the feature points extracted from the book image information detected according to a user's input. In this way, the book reader unit 1200 generates and provides Braille output information in which the form of the book image information derived from the image area L1 is displayed, and thus the user can obtain an image through the Braille in which the form of the book image information is displayed. It is possible to exert the effect of intuitively understanding the specific form of

13 exemplarily shows an internal configuration of a computing device according to an embodiment of the present invention.

13, the computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, an input/output subsystem ( I/O subsystem) 11400 , a power circuit 11500 , and a communication circuit 11600 may be included at least. In this case, the computing device 11000 may correspond to the smart glass 2000 , the user terminal 3000 , the server 1000 , and the braille pad 4000 .

The memory 11200 may include, for example, a high-speed random access memory, a magnetic disk, an SRAM, a DRAM, a ROM, a flash memory, or a non-volatile memory. there is. The memory 11200 may include a software module required for the operation of the computing device 11000 , an instruction set, or other various data included in the learned embedding model.

In this case, access to the memory 11200 from other components such as the processor 11100 or the peripheral interface 11300 may be controlled by the processor 11100 .

Peripheral interface 11300 may couple input and/or output peripherals of computing device 11000 to processor 11100 and memory 11200 . The processor 11100 may execute a software module or an instruction set stored in the memory 11200 to perform various functions for the computing device 11000 and process data.

The input/output subsystem 11400 may couple various input/output peripherals to the peripheral interface 11300 . For example, the input/output subsystem 11400 may include a controller for coupling a peripheral device such as a monitor, keyboard, mouse, printer, or a touch screen or sensor as required to the peripheral interface 11300 . According to another aspect, input/output peripherals may be coupled to peripheral interface 11300 without going through input/output subsystem 11400 .

The power circuit 11500 may supply power to all or some of the components of the terminal. For example, the power circuit 11500 may include a power management system, one or more power sources such as batteries or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or a power source. It may include any other components for creation, management, and distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, if necessary, the communication circuit 11600 may include an RF circuit to transmit and receive an RF signal, also known as an electromagnetic signal, to enable communication with other computing devices.

This embodiment of FIG. 13 is only an example of the computing device 11000 , and the computing device 11000 may omit some components shown in FIG. 13 , or further include additional components not shown in FIG. 13 , or 2 It may have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor in addition to the components shown in FIG. 13 , and various communication methods (WiFi, 3G, LTE) are provided in the communication circuit 1160 . , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components that may be included in the computing device 11000 may be implemented in hardware, software, or a combination of both hardware and software including an integrated circuit specialized for one or more signal processing or applications.

Methods according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded in a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. The application to which the present invention is applied may be installed in the user terminal through a file provided by the file distribution system. As an example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to a request of the user terminal.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computing devices, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

As a cognitive assistance system for the visually impaired,
a smart glass including a photographing module and a communication module, and transmitting an image photographed by the photographing module to a user terminal;
a braille pad capable of providing tactile information to a user by performing braille output based on data received from the user terminal; and
A server that transmits and receives data with a user terminal; includes;
The server is
an object detection unit that detects an object in the image, and accordingly generates braille output information to be output from the braille pad or audio output information to be output from any one of the user terminal, braille pad, or smart glass; and
When the object detection unit detects a book image from the image, a book reader unit for extracting book text information and book image information of the book to be output in Braille from the braille pad; Containing, a cognitive assistance system for the visually impaired.
The method according to claim 1,
The object detection unit,
determining the number of objects detected in the image; and
When the number of the detected objects is two or more, recognizing the detected object through a learned artificial neural network, and generating the voice output information for the recognized object; Cognitive assistance system for the visually impaired to perform.
The method according to claim 1,
The object detection unit,
determining the number of objects detected in the image; and
When the number of the detected objects is two or more, the detected object is recognized through the learned artificial neural network, and coordinate information for the two or more recognized objects and an icon mapped according to a preset criterion for the two or more recognized objects A cognitive assistance system for the visually impaired to perform; generating braille output information including the information.
The method according to claim 1,
The object detection unit,
determining the number of objects detected in the image;
recognizing the object through a learned artificial neural network when the detected object is one;
A cognitive assistance system for the visually impaired that performs a step of generating braille output information in which the shape of an object is displayed based on object information on the recognized object and feature points extracted from the recognized object.
The method according to claim 1,
The object detection unit,
determining the number of objects detected in the image;
recognizing the object through a learned artificial neural network when the detected object is one;
When the shape of the recognized object is not all included in the image, generating audio output information for the recognized object and guide audio output information for adjusting the position of the smart glass; performing a cognitive assistance system for the visually impaired .
The method according to claim 1,
The object detection unit,
determining the number of objects detected in the image;
recognizing the object through a learned artificial neural network when the detected object is one; and
When the object is not recognized through the artificial neural network, generating guide voice output information for the position adjustment of the smart glass; performing a cognitive assistance system for the visually impaired.

The method according to claim 1,
The book reader unit,
an image pre-processing step of pre-processing by changing one or more of the gradient and color of the book image detected in the image according to a preset standard;
a text image area deriving step of deriving at least one text area and at least one image area based on the image pre-processed in the image pre-processing step;
A text recognition step of determining the output order of the derived one or more text areas, recognizing the text included in the one or more text areas according to the output order, and deriving the book text information to be output in Braille from the Braille pad; A cognitive assistance system for the visually impaired.
The method according to claim 1,
The screen on the braille pad provided by the book reader is,
One or more text areas and one or more image areas are extracted from the book image detected in the image, and the texts included in the extracted one or more text areas are sequentially arranged and displayed,
A cognitive assistance system for the visually impaired that provides a list of one or more book image information detected by the book reader according to a user's input.
8. The method of claim 7,
The book reader unit,
performing an image recognition step of recognizing the book image information extracted from the derived one or more image areas;
The screen on the braille pad provided by the book reader is,
A cognitive assistance system for the visually impaired that provides a list of the one or more book image information according to a user's input, including image information recognized in the image recognition step.
As a method of providing a cognitive assistance service for the visually impaired implemented in the cognitive assistance system for the visually impaired,
The cognitive assistance system for the visually impaired,
a smart glass including a photographing module and a communication module, and transmitting an image photographed by the photographing module to a user terminal;
a braille pad capable of providing tactile information to a user by performing braille output based on data received from the user terminal; and
A server that transmits and receives data with a user terminal; includes;
The server is
an object detection step of detecting an object in the image, and generating braille output information to be output from the braille pad or audio output information to be outputted from any one of the user terminal, braille pad, or smart glass according to the object detection step; and
In the case of detecting a book image from the image in the object detection step, a book reader step of extracting the book text information and the book image information of the book to be output in Braille from the braille pad; Containing, providing a cognitive assistance service for the visually impaired method.

Images