WO2019221416A1

WO2019221416A1 - Method for providing service for guiding visually impaired person by using real-time on-site video relay broadcast

Info

Publication number: WO2019221416A1
Application number: PCT/KR2019/004901
Authority: WO
Inventors: 문경진
Original assignee: Moon Kyong Jin
Priority date: 2018-05-18
Filing date: 2019-04-23
Publication date: 2019-11-21
Also published as: KR101929875B1

Abstract

A method for providing a service for guiding a visually impaired person by using real-time on-site video relay broadcast is provided. The method comprises the steps of: when a help request event is received from a request terminal, receiving streams of real-time image data from the request terminal or a photographing device interconnected to the request terminal; selecting a providing terminal capable of providing guidance, from at least one pre-registered providing terminal, and transmitting the streamed real-time image data to the selected providing terminal; establishing a channel for real-time image data streaming between the selected providing terminal and the request terminal; and when a voice corresponding to a description of the real-time image data is received from the providing terminal, transmitting the voice to the request terminal in real time.

Description

How to provide information service for the blind using real-time video relay

The present invention relates to a method for providing a guide service for the visually impaired by using real-time on-site video relaying, and provides a method for remotely performing a response to a request requested by a guide requester by streaming a screen shot by a camera in real time.

Recently, many technologies for ensuring the free walking of the visually impaired and the disabled have been accelerated by the expansion of the wearable market and the enrichment of the Internet of Things and network resources. The devices for free walking are related to smart wand and smart glasses using image camera, ultrasonic sensor, and acceleration sensor. Typical technologies include technology to identify obstacles and extract walking area and landmark symbol. Technology such as technology that recognizes information and gives surrounding environment information and intelligent walking aid system based on Machine-to-Machine (M2M) has been researched and developed so that blind people can walk safely and conveniently.

At this time, a technology for photographing the visually impaired visually impaired and analyzing the information has been developed. In this regard, the prior art Korean Patent Publication No. 2012-0059752 (published on June 11, 2012) moves with the visually impaired. In order to recognize the emotions that can inform the visually impaired information or location information to the visually impaired, the situation information necessary for the visually impaired is transmitted and received, the background image around the visually impaired is photographed to obtain image information, and the location of the visually impaired. To recognize the, and to recognize the background image based on the image information signal, to provide information on the situation information or background image to the blind, and to guide the direction is disclosed.

In recent years, however, the demand for guided walks that provide detailed information necessary for walking the traffic vulnerable beyond the detection of obstacles that hinder the traffic vulnerable to walking is increasing and artificial intelligence is still being developed. There is a need to provide detailed information on walking because it is a stage, and blind people are still reluctant to go out alone in an environment unfriendly to the blind, and it is not even basic life in the house unless someone is there. Income was not enough to hire a carer or caregiver.

One embodiment of the present invention, the smart glasses based on the camera, microphone and speaker interworking with the smart terminal, when the help request event of voice recognition or button recognition occurs available that can help among a plurality of help providers Selecting a terminal and sharing the real-time video collected from the smart glasses from the terminal providing help to guide the requester to approach the object or destination desired, independent living inside and outside the home even if there is no worker or caregiver It is possible to provide a blind hand guide service providing method using real-time on-site video relaying. However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention, when the help request event is received from the requesting terminal, the step of receiving real-time shooting data from the requesting terminal or from the photographing device linked to the requesting terminal Selecting and transmitting the streamed real-time shooting data from among the at least one pre-registered provisioning terminal, the provisioning capable of providing the guide, forming a channel for real-time streaming of the photographing data between the selected provisioning terminal and the requesting terminal, and When the voice corresponding to the description of the real-time shooting data is received from the providing terminal, and transmitting to the requesting terminal in real time.

According to any one of the problem solving means of the present invention described above, through the change and improvement in the pedestrian environment for the traffic vulnerable ultimately can play a role in carrying out the same fair and convenient traffic right to the general public who is the traffic vulnerable. In addition to providing a variety of guidance services or assistance services in addition to the road guidance has the effect of eradicating direct discrimination as well as indirect discrimination of the disabled.

1 is a view for explaining a system for providing a guide service for the visually impaired using real-time on-site video relaying according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a server for providing a guide service for the visually impaired included in the system of FIG. 1.

3 is a view for explaining an embodiment of the blind people guide service using the real-time on-site video relay according to an embodiment of the present invention.

FIG. 4 is a diagram illustrating a process of transmitting and receiving data between components included in a system for providing a visually-impaired person using the real-time on-site video relay of FIG. 1 according to an embodiment of the present invention.

5 is an operation flowchart for explaining a method for providing a visually impaired guide service using real-time on-site video relaying according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, which means that it may further include other components, except to exclude other components, unless specifically stated otherwise, one or more other features It is to be understood that the present disclosure does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, parts, or combinations thereof.

As used throughout the specification, the terms "about", "substantially", and the like, are used at, or in close proximity to, numerical values when manufacturing and material tolerances inherent in the meanings indicated are intended to aid the understanding of the invention. Accurate or absolute figures are used to assist in the prevention of unfair use by unscrupulous infringers. As used throughout the specification of the present invention, the term "step of" or "step of" does not mean "step for".

In the present specification, the term 'unit' includes a unit realized by hardware, a unit realized by software, and a unit realized by both. In addition, one unit may be realized using two or more pieces of hardware, or two or more units may be realized by one piece of hardware.

Some of the operations or functions described as being performed by the terminal, the apparatus, or the device may be performed instead in a server connected to the terminal, the apparatus, or the device. Similarly, some of the operations or functions described as being performed by the server may be performed by the terminal, apparatus or device connected to the server.

In the present specification, some of the operations or functions described as mapping or matching with a terminal mean that a unique number of a terminal or identification information of an individual, which is identification data of the terminal, is mapped or matched. Can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a view for explaining a system for providing a guide service for the visually impaired using real-time on-site video relaying according to an embodiment of the present invention. Referring to FIG. 1, the blind person guide service providing system 1 using real-time on-site video relaying includes a request terminal 100, a blind person guide service providing server 300, a providing terminal 400, and a companion device 500. And at least one CCTV 600. However, since the visually impaired guide service providing system 1 using the real-time on-site video relay of FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1.

In this case, each component of FIG. 1 is generally connected through a network 200. For example, as shown in FIG. 1, the request terminal 100 may be connected to the blind person guide service providing server 300 through the network 200, and the request terminal 100 may be connected through the network 200. It may be connected to the companion device 500. In addition, the request terminal 100 may be connected to the providing terminal 400 directly or via the guide service providing server 300 for the visually impaired through the network 200. The blind person guide service providing server 300 may be connected to the request terminal 100, the providing terminal 400, the companion device 500, and the CCTV 600 to the network 200. In addition, the providing terminal 400 may be connected to the request terminal 100 through the network 200, or may be connected to the CCTV 600. In addition, the companion device 500 may be connected to the requesting terminal 100 through the network 200, and the CCTV 600 may be connected to the visual service guide server 300 through the network 200.

Here, the network refers to a connection structure capable of exchanging information between respective nodes such as a plurality of terminals and servers. Examples of such a network include RF, 3rd Generation Partnership Project (3GPP) network, and long term (LTE). Evolution network, 5th Generation Partnership Project (5GPP) network, World Interoperability for Microwave Access (WIMAX) network, Internet, Local Area Network (LAN), Wireless Local Area Network (WLAN), Wide Area Network (WAN) , PAN (Personal Area Network), Bluetooth (Bluetooth) network, NFC network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network and the like, but is not limited thereto.

In the following description, the term “at least one” is defined as a singular and plural term, and each component may exist in the singular or plural, even though the term “at least one” does not exist, and may mean the singular or plural. It will be self explanatory. In addition, the singular or plural elements may be changed according to embodiments.

The request terminal 100 may be a terminal of a requester who receives a guide service using a web page, an app page, a program, or an application related to a visually impaired guide service using real-time on-site video relaying. The request terminal 100 may be a terminal that generates a help request event by voice recognition or button input and transmits the generated help request event to the blind guide service providing server 300. In addition, when the request terminal 100 is connected to the providing terminal 400 through the guide service providing server 300 for the visually impaired, video recording or voice input data of the request terminal 100 itself, or from the companion device 500. Provides input so that the input video data or audio data is streamed in real time to the visually impaired guide service providing server 300 to provide the request terminal 100 in real time to transmit the information or a response to the query to the requesting terminal 100 in real time. It may be a terminal. At this time, any help event is possible, and its use or purpose will be obvious if any of the configuration of the patent is the same.

Here, the request terminal 100 may be implemented as a computer that can access a server or a terminal in a remote place through a network. Here, the computer may include, for example, a navigation, a laptop equipped with a web browser, a desktop, a laptop, and the like. In this case, the request terminal 100 may be implemented as a terminal that can access a server or terminal in a remote place through a network. The request terminal 100 is, for example, a wireless communication device that ensures portability and mobility. The request terminal 100 may include navigation, a personal communication system (PCS), a global system for mobile communications (GSM), a personal digital cellular (PDC), and a personal PHS (PHS). Handyphone System (PDA), Personal Digital Assistant (PDA), International Mobile Telecommunication (IMT) -2000, Code Division Multiple Access (CDMA) -2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (WBRO) terminal, All types of handheld based wireless communication devices such as smartphones, smartpads, tablet PCs, and the like may be included.

The blind person guide service providing server 300 may be a server for providing a blind person guide service web page, an app page, a program, or an application using real-time on-site video relaying. In addition, when the visually impaired guide service providing server 300 receives a help request event from the requesting terminal 100, the visually impaired guide service providing server 300 connects to any one of the at least one providing terminal 400 and the providing terminal 400. If the provider of 400 provides help, it may be a server transmitting the request to the requesting terminal 100 in real time. At this time, the visually impaired guide service providing server 300, before the providing terminal 400 is connected with the requesting terminal 100, the photographing data photographed at the providing terminal 400 or photographed at the requesting terminal 100 mechanically It may be a server that provides information in real time by analyzing the information, and after connecting, the server may transmit a text or voice input from the providing terminal 400 to the requesting terminal 100. In addition, the visually impaired guide service providing server 300, by searching the CCTV 600 based on the location of the request terminal 100 to transmit to the providing terminal 400 to provide a wider field of view to provide assistance. It can be a server.

Here, the guide service providing server 300 for the visually impaired may be implemented by a computer that can access a server or a terminal in a remote place through a network. Here, the computer may include, for example, a navigation, a laptop equipped with a web browser, a desktop, a laptop, and the like.

The providing terminal 400 may be a terminal of a help provider using a web page, an app page, a program, or an application related to a blind person guide service using real-time on-site video relaying. In addition, when the connection signal with the request terminal 100 is received from the visually-impaired guide service providing server 300 and the photographing data and the query data are received, the providing terminal 400 sends a response thereto as a voice signal. It may be a terminal transmitting to the request terminal 100 in real time. In addition, the providing terminal 400 may be provided with the image of the CCTV 600 from the guide service providing server 300 for the visually impaired, and may be a terminal that receives and outputs the screen.

Here, the providing terminal 400 may be implemented as a computer that can access a server or a terminal in a remote place through a network. Here, the computer may include, for example, a navigation, a laptop equipped with a web browser, a desktop, a laptop, and the like. In this case, the providing terminal 400 may be implemented as a terminal that can access a server or terminal in a remote place through a network. The providing terminal 400 is, for example, a wireless communication device that ensures portability and mobility, and includes a navigation, a personal communication system (PCS), a global system for mobile communications (GSM), a personal digital cellular (PDC), and a personal PHS (PHS). Handyphone System (PDA), Personal Digital Assistant (PDA), International Mobile Telecommunication (IMT) -2000, Code Division Multiple Access (CDMA) -2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (WBRO) terminal, All types of handheld based wireless communication devices such as smartphones, smartpads, tablet PCs, and the like may be included.

The companion device 500 may be a device including a microphone, a speaker (or earbud), and a camera to transmit the input voice or photographing data to the requesting terminal 100. In addition, the companion device 500 may be an input device for transmitting the requester's voice, etc. collected through the microphone to the requesting terminal 100 and transmitting the information to the visually impaired guide service providing server 300. In addition, the companion device 500 may be, for example, Electronic Travel Aids (ETA) and Robotic Travel Aids (RTA). The ETA may use a distance sensor such as an ultrasonic / laser and a front obstacle using a camera. It can be a device that detects the location and converts it into a voice or tactile signal, and scans the surrounding environment to detect obstacles by using the distance sensor and vision sensor, and further measures to determine the size of the detected obstacle. It may be necessary. Meanwhile, the RTA is a walking aid device that combines various sensor technologies and mobile robot technology with the ETA to improve convenience of use, and may be implemented as a device combined with the mobile robot technology using outdoor GPS information. Of course, in addition to the camera, microphone, earphones, and the device including only the interworking function between the request terminal 100 will be apparent that various devices can be used without being limited to those listed above.

The at least one CCTV 600 may be a photographing apparatus that transmits a screen sharing request from the visually impaired guide service providing server 300 to the visually impaired guide service providing server 300. The at least one CCTV 600 may identify and track an object when an object tracking request exists, and may be a device capable of tilting or rotating up, down, left, and right. In addition, the at least one CCTV (600) by storing the location of the point photographed by the at least one CCTV (600) as an identification code to transmit to the blind guide service providing server 300, the blind person guide service providing server 300 ) May be a device that provides this identification code when searching for the CCTV 600 based on the location of the request terminal 100.

Here, the at least one CCTV 600 may be implemented as a computer that can be connected to a remote server or terminal through a network. Here, the computer may include, for example, a navigation, a laptop equipped with a web browser, a desktop, a laptop, and the like. In this case, the at least one CCTV 600 may be implemented as a terminal that can be connected to a remote server or terminal through a network. At least one CCTV 600 is, for example, a wireless communication device that ensures portability and mobility, navigation, Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication) -2000, Code Division Multiple Access (CDMA) -2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (Wibro) It may include all kinds of handheld based wireless communication devices such as a terminal, a smart phone, a smart pad, a tablet PC, and the like.

2 is a block diagram illustrating a server for providing a guide service for the visually impaired included in the system of FIG. 1, and FIG. 3 is a view for implementing the guide service for the visually impaired using real-time on-site video relay according to an embodiment of the present invention. It is a figure for demonstrating an Example.

Referring to FIG. 2, the blind person guide service providing server 300 may include a request receiving unit 310, a selecting unit 320, a forming unit 330, a transmitting unit 340, and an object tracking unit 350. Can be.

The blind person guide service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with the visually impaired using the real-time field video relay to the request terminal 100, and the providing terminal 400 In the case of transmitting a guide service application, a program, an app page, a web page, and the like, the request terminal 100 and the providing terminal 400 include a visually impaired guide service application, a program, an app page, a web page using real-time on-site video relaying. Can install or open the back. In addition, the service program may be driven in the request terminal 100 and the provision terminal 400 using a script executed in a web browser. Here, the web browser is a program that enables the use of the World Wide Web (WWW) service, and refers to a program that receives and displays hypertext described in HTML (hyper text mark-up language), for example, Netscape. , Explorer, chrome and the like. In addition, an application means an application on a terminal, and includes, for example, an app running on a mobile terminal (smartphone).

In one embodiment of the present invention, the type of help request object recognition, document recognition, learning program, location information notification, emergency call, safety notification, schedule management, daily guide, face-to-face situation interpretation, text message, telephone, navigation, Environmental control, medication time notifications, games, exercise, etc., but is not limited thereto.

Referring to FIG. 2, when the help request event is received from the request terminal 100, the request receiving unit 310 streams real-time photographing data from the request terminal 100 or from a photographing device linked with the request terminal 100. I can receive it. In this case, the help request event may be input and generated from the request terminal 100 through various interfaces such as voice recognition, button input recognition, motion or touch recognition of the touch panel. For example, in the case of speech recognition, it may be words such as "where is it?" Or "help", but the words may be learned to derive similar words to make a database or to learn a user's tone. The present invention may not be limited to the above-described words or preset words. Alternatively, the help request event may be generated by a method of making a call in the request terminal 100, that is, a call origination and received by the request receiving unit 310.

In addition, when the request receiving unit 310 receives the help request event from the request terminal 100, when the real-time shooting data is streamed from the requesting terminal 100 or from the photographing apparatus linked with the requesting terminal 100, the request is received. When the terminal 100 100 matches a previously stored word with a word converted into a text through voice recognition, a call call for a help request event may be executed. For example, it may be almost impossible for a blind person to take out a smart terminal, unlock a lock screen, and find and execute a required app among a large number of apps. Accordingly, convenience can be provided by allowing help to be requested by a button or voice.

In this case, the requesting terminal 100 includes or interlocks with any one or a combination of at least one of a microphone, a speaker, and a camera, and the photographing apparatus linked with the requesting terminal 100 is attached to the wearable glass to photograph the opposite direction. The photographing data photographed by interworking with the request terminal 100 by wire or wirelessly may be transmitted to the request terminal 100.

In addition, when a request for help event is received, consent for sharing the location information of the request terminal 100 may be included, and the location information of the request terminal 100 may be updated and collected in real time until the blind guide service ends. have. For example, the visually impaired person may make various requests for help even if it is not necessarily a road guide, and may require location information but may not. Accordingly, the request for help is determined to agree to the collection of location information, but collecting the location, if it is not necessary in the providing terminal 400 dispose of it or stop collecting any more, agree to the urgent situation To prevent the delay caused by

The selector 320 may select and transmit the streamed real-time photographing data from the at least one pre-registered provision terminal 400 to which the provision terminal may be provided. In this case, the selection unit 320 may be a method of transmitting a help request event by broadcasting to the providing terminal 400 and connecting to a terminal, that is, an available terminal, in which an Ack is received. It may also be connected by going through the steps or by immediately connecting the available terminal.

When the real-time photographing data streamed by the selecting unit 320 is selected and transmitted from among at least one registered terminal 400 that is provided for guidance, the providing terminal 400 and the requesting terminal 100 are transmitted. Analysis of real-time photographing data received from the requesting terminal 100, binarization of real-time photographing data, extraction of features of objects, classification of objects through clustering, and pre-stored descriptions of classified objects The data may be voice converted and transmitted to the request terminal 100 in real time. For example, if all of the providers are busy or connected and the requester is in a hurry and wants to know what is going on right now, it may be inconvenient to wait for an agent or provider. In this case, by notifying the situation that is mechanically identified during the connection time, it is possible to prevent accidents in an emergency.

In this case, various methods may be used as a method of mechanically identifying the screen, in addition to the above-described method. For example, the landmark area may be recognized and classified from the ROI obtained by detecting the landmark area from the original image. In other words, the recognition method using an artificial neural network is a representative technique of machine learning, which is a method of extracting feature vectors from a sign DB and landmark DB image previously detected through neural networks, and learning, classifying, and recognizing them through a hierarchical structure of neural networks. Artificial neural network is a system that recognizes and classifies images by extracting features that are robust to change by forming multi-layered structure and passing through each layer. Among artificial neural networks, a general multilayer structure of a convolutional neural network (CNN), which is widely used in image processing, may be possible. Another symbol recognition method is a recognition technique using feature matching. As a basic feature matching method, SIFT is widely used, but it is the most common method but requires a large amount of computation. In addition, there is a method of extracting and identifying geometric features based on CPT (Central Projected Transformation), which shows a good recognition rate compared to the constant moment methods such as Hu, Tchebichef, and Zernike. In addition, there is a method of generating feature points using SURF (Speed Up Feature Transform) and ORB (Oriented FAST and Rotated BRIEF) that can be matched at high speed, and matching with template of DB, but are not limited to the above-described methods.

The forming unit 330 may form a channel for real-time streaming of photographing data between the selected providing terminal 400 and the requesting terminal 100. In order to achieve real-time streaming, a channel between two

terminals

100 and 400 may be formed. For delay-free streaming, the bandwidth and network resources are determined to determine the specification of transcoding, and the real-time streaming is performed through the formed channel. Can provide.

When the voice corresponding to the description of the real-time photographing data is received from the providing terminal 400, the transmitting unit 340 may transmit to the requesting terminal 100 in real time. For example, if the requester wants to find a black suit and needs help, the provider will explain to the requestor while viewing the situation, such as the number of drawers or the number of hangers, while viewing the screen received from the request terminal 100. Can be.

In this case, the real-time photographing data may be streamed in real time with the received voice data inputted from the request terminal 100 or a microphone linked with the request terminal 100 added thereto. That is, the photographing data may include voice, but if not, the voice data inputted from the microphone is also transmitted, so that the requestor may transmit the request or inquiry to the provider, so as to determine exactly what the provider wants to request. Provide information.

When the request tracking unit 350 receives a help request event from the request terminal 100, the object tracking unit 350 may capture real-time photographing data from the requesting terminal 100 or from a photographing apparatus linked with the requesting terminal 100. After receiving the stream, search for at least one CCTV 600 photographing the direction in which the request terminal 100 is located based on the position of the request terminal 100 and stream the real-time photographing data of the searched at least one CCTV 600. In response, the indicator of the user of the request terminal 100 may be extracted from at least one photographing object captured by the at least one CCTV 600.

For example, it is assumed that the visually impaired person needs to take the bus 311, and because the blind person stands behind the direction of the bus coming, the provider does not know where the bus stop is only by the screen shot by the companion device 500. lets do it. At this time, even when trying to figure out while looking at a map or the like, the report may not be able to grasp only the direction of the interlocking device 500 or the map. Accordingly, the object tracking unit 350 may share the screen of the CCTV 600 to the providing terminal 400, and display a tag or an image for identifying the visually impaired and transmit the screen to the providing terminal 400. have. To this end, the CCTV 600 may store the GPS coordinates of the shooting location, not the place where the CCTV 600 is located, as an identification code or identification number. Accordingly, the CCTV 600 having the same identification code as the corresponding coordinates can be detected based on the GPS coordinates of the requesting terminal 100, and thus has a wider viewing angle than the image captured by the requesting terminal 100. Allow the screen to be provided. Accordingly, the providing terminal 400 turns around, and there is a stop when a few more steps are taken, and the bus 311 may be informed that it has now arrived. However, it is obvious that the present invention is not limited to the above example.

In addition, the visually impaired people have difficulty walking due to various problems caused by the invisibility of the eyes. The visually impaired people are limited in their daily activities by limiting the most dependent of the five senses of human beings. As a tool used to assist walking, there are various things such as a walkway block, a guide dog, a cane, etc., but since only a part of information about the surroundings is obtained and there are many limitations in recognizing the whole, in one embodiment of the present invention, the providing terminal ( In addition to connecting 400, a service may be provided to allow walking more conveniently. In other words, although there is a problem with the visually impaired walking, the fact is that the ability to walk is not a problem, but it is difficult to recognize the direction while walking. To this end, first, Obstacle Recognition is performed. When an image is input from the companion device 500, an object and a background are separated, and the object is recognized using a shape feature for the object area. Then, after binarizing the image, the feature is extracted. The length and size of the object are determined using the circumferential length and the area size, the feature is selected, and the object is classified to find out what the object is. To this end, clustering is to be achieved, and when the selected features are collected and tied together, the object can be known through the process of clustering. In addition to this understanding, how to deliver information on the road is also important. By providing voice information on the way to the visually impaired, users can reduce the burden of spatial constraints. Use GPS to determine your current location, then use your compass to estimate your direction, then to estimate your distance and direction to your destination, go straight to intuitive words, e.g. to the left of the tricycle, etc. Use words that are easy to recognize together. In other words, when spatial information is expressed in language, confusion often occurs due to the difference in the standard of each person. In the state facing each other, the front of the two people are opposite, so the direction should be determined from the user's standards to guide.

Hereinafter, an operation process according to the configuration of the above-described visually impaired guide service providing server of FIG. 2 will be described in detail with reference to FIG. 3. However, the embodiment is only any one of various embodiments of the present invention, but is not limited thereto.

Referring to FIG. 3, it is assumed that (a) the requester tries to feed the guide dog but cannot find where the feed is. In this case, the request terminal 100 generates a help request event using a call origination, a button press, or a voice recognition as an input value, and transmits the help request event to the blind person guide service providing server 300. And, (b) if there is a waiting time with the providing terminal 400, the visually impaired guide service providing server 300, first, the shape or logo (text of the object depicted by the requester on the screen taken by the companion device 500) Perception, and (c) calculate and measure in which direction how many steps in front or behind, based on the direction the requestor is looking.

On the other hand, (d) if the requestor is standing with the ticket machine, and there is a request that asks where the ticket machine is located, the visually impaired guide service providing server 300 does not have a ticket machine on the screen photographed by the companion device 500. In addition, the CCTV 600 may be searched for photographing the location where the request terminal 100 is located to identify the requestor and may indicate in which direction the ticket is issued. In this case, (e) when the providing terminal 400 and the requesting terminal 100 are connected by the visually impaired guide service providing server 300, the answer transmitted from the providing terminal 400 may be transmitted to the requesting terminal 100. The requester will then be able to issue a ticket.

The description of the method for providing the visually impaired guide service using the real-time on-site video relay of FIGS. 2 and 3 is described above with reference to the method for providing the visually-impaired guide service using the real-time on-site video relay. Since it can be easily inferred from the same or described content it will be omitted.

FIG. 4 is a diagram illustrating a process of transmitting and receiving data between components included in a system for providing a visually-impaired person using the real-time on-site video relay of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process in which data is transmitted and received between each component will be described with reference to FIG. 4, but the present disclosure is not limited to the above-described embodiments, and is illustrated in FIG. 4 according to the various embodiments described above. It is apparent to those skilled in the art that the process of transmitting and receiving data may be changed.

Referring to FIG. 4, when the requesting terminal 100 and the companion device 500 are interlocked (S4100), the requesting terminal 100 confirms whether help is requested through voice monitoring or input monitoring (S4200) (S4300). If the help is requested, the help service event is sent to the blind guide service providing server 300 (S4400).

In addition, the visually-impaired guide service providing server 300 checks the availability of the providing terminal 400 (S4500), streams the photographing data from the request terminal 100 during the waiting time and receives a query (S4600). Until the connection with the providing terminal 400 (S4700), the voice guidance is automatically performed through image analysis (S4710).

In addition, the visually impaired guide service providing server 300 forms a channel when the connection is made (S4730), and if the voice is transmitted from the providing terminal 400 and delivers it to the requesting terminal 100 (S4740), if necessary The CCTV 600 is searched (S4800) to receive the screen shot from the CCTV 600 is shared (S4810), and transmits to the providing terminal 400 (S4820).

Finally, the visually impaired guide service providing server 300 returns to S4730 and loops until the guide is finished (S4900).

The order between the above-described steps S4100 to S4900 is merely an example, and is not limited thereto. That is, the order between the above-described steps (S4100 to S4900) may be mutually changed, and some of the steps may be executed or deleted at the same time.

The matters that are not described with respect to the method for providing the visually impaired guide service using the real-time on-site video relay of FIG. 4 are as described above with reference to the method for providing the visually impaired guide service using the real-time on-site video relay through FIGS. 1 to 3. Since it can be easily inferred from the same or described content it will be omitted.

5 is an operation flowchart for explaining a method for providing a visually impaired guide service using real-time on-site video relaying according to an embodiment of the present invention. Referring to FIG. 5, when a help request event is received from a requesting terminal, the visually impaired guide service providing server receives real-time photographing data from the requesting terminal or from a photographing apparatus linked with the requesting terminal (S5100).

In addition, the guide service providing server for the visually impaired, selects and transmits the streamed real-time shooting data from the at least one pre-registered providing terminal capable of providing the guide (S5200), and the shooting data between the selected providing terminal and the requesting terminal A channel for real time streaming is formed (S5300).

In addition, when the visually impaired guide service providing server receives a voice corresponding to the description of the real-time photographing data from the providing terminal, it transmits to the requesting terminal in real time (S5400).

Not described about the method for providing the visually impaired guide service using the real-time on-site video relay of FIG. 5 is described above with respect to the method for providing the visually impaired guide service using the real-time on-site video relay through FIGS. 1 to 4. Since it can be easily inferred from the same or described content it will be omitted.

The blind person guide service providing method using a real-time on-site video relay according to an embodiment described with reference to FIG. 5 is in the form of a recording medium including instructions executable by a computer, such as an application or a program module executed by a computer. May also be implemented. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The method for providing a visually impaired user guide service using real-time on-site video relaying according to an embodiment of the present invention described above may include an application basically installed in a terminal (this may include a program included in a platform or an operating system, which is basically installed in the terminal). ), Or may be executed by an application (ie, a program) that is directly installed on the master terminal by an application providing server such as an application store server, an application, or a web server associated with the corresponding service. In this sense, the method for providing a visually-impaired guide service using real-time on-site video relaying according to an embodiment of the present invention described above is implemented as an application (ie, a program) basically installed in a terminal or directly installed by a user, and the like. Can be recorded on a computer readable recording medium.

The foregoing description of the present invention is intended for illustration, and it will be understood by those skilled in the art that the present invention may be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

Claims

In the blind visual guide service providing server executed in the server providing method,

When the help request event is received from the requesting terminal, receiving real-time photographing data from the requesting terminal or from a photographing apparatus linked with the requesting terminal;

Selecting and transmitting the streamed real-time photographing data from a pre-registered at least one provisioning terminal to which a provisional terminal can be provided;

Forming a channel for real-time streaming of photographing data between the selected providing terminal and the requesting terminal;

Transmitting a voice corresponding to the description of the real-time photographing data from the providing terminal to the requesting terminal in real time;

Searching for at least one CCTV photographing a direction in which the requesting terminal is located based on the position of the requesting terminal after the streaming of the real-time photographing data;

Receiving real-time photographing data of the searched at least one CCTV; And

And extracting a user of the requesting terminal from at least one photographing object captured by the at least one CCTV, and tagging an indicator.
The method of claim 1,

The requesting terminal includes or cooperates with any one or at least one of a microphone, a speaker, and a camera.

The photographing apparatus interlocked with the request terminal captures a direction opposite to the wearable glass and interlocks with the request terminal in a wired or wireless manner to transmit the photographed photographing data to the request terminal. How to provide information service for the blind using relay.
The method of claim 1,

The real-time shooting data is a visually impaired guide service providing method using a real-time on-site video relay, characterized in that the audio received from the request terminal or the microphone associated with the request terminal is added and streamed in real time.
The method of claim 1,

When the help request event is received, consent for sharing the location information of the requesting terminal is included, and the location information of the requesting terminal is updated in real time until the visually impaired guide service is terminated. How to provide guide service for the blind using on-site video relay.
The method of claim 1,

When the help request event is received from the requesting terminal, the step of receiving real-time shooting data from the requesting terminal or from a photographing apparatus linked with the requesting terminal,

If a call word for the help request event is executed if the previously stored word and the text word converted by the conversion through the voice recognition in the requesting terminal comprises the step of executing a real-time on-site video relay How to provide information service for the blind.
The method of claim 1,

Selecting and transmitting the streamed real-time photographing data to a provision terminal capable of providing guidance among at least one registered terminal,

Analyzing real-time photographing data received from the requesting terminal until the connection between the providing terminal and the requesting terminal is established;

Binarizing the real-time photographing data to extract a feature of an object and classifying the object through clustering;

And visually converting previously stored description data with respect to the classified objects to the requesting terminal in real time.